Page created: 2013-01-27
This page is being linked to by various *BSD news sources and user groups, (Patrick !McEvoy et. al.), we'll try to keep this page up to date with the current state of this issue. Eventually, I hope this page exists merely to announce it's own deprecation- once there is a resolution accepted the FreeBSD community at large.
=== UPDATE 2 : 2015-08-06 ===
Most of the contents of this page are seriously out of date.
There is now an in-base version of svn, as /usr/bin/svnlite. This is a stripped down but fully functional client but has the bulky API / language bindings removed. It has a relatively lightweight footprint and no external dependencies. svnlite will make use of the ca_root_nss package in order to validate https certificates, if it is present.
If you do not have svnlite, or prefer to use the official packaged version, it can be securely obtained:
{{{{
# pkg install ca_root_nss
# pkg install subversion
}}}}
If you wish to use secure https urls with svnlite, you should install the ca_root_nss package to obtain the current root certificate lists.
{{{{
# pkg install ca_root_nss
}}}}
You should use https://svn.freebsd.org/ where practical and so long as ca_root_nss is installed and up to date, the certificate should be automatically checked and securely verified.
svn.freebsd.org connects you to the closest project-operated mirror. The old legacy mirror names should not be used.
Official documentation is here: [[https://www.freebsd.org/doc/handbook/svn.html]]
As of writing:
* cvs and cvsup is fully removed and should be ignored.
* ctm is obsolete and is thought to be dangerous. Its use should be discouraged.
* svnup can now use https to access svn.freebsd.org, but it needs the ca_root_nss package.
* Any remaining references to 'pkg_add -r' should be replaced with 'pkg add'.
* svnup is not expected to be added to base but will remain in ports/packages.
* ctm is expected to be removed from base and moved to ports/packages.
* The FreeBSD project encourages the use of svn or svnlite as the preferred source access method.
Partially obsolete information follows:
=== UPDATE ===
There is now a working '''svnup(1)''' utility, which is tested functional!
Fetch ports sources, then:
{{{{
# pkg install svnup
# svnup stable -h svn.freebsd.org
}}}}
Currently it's a port- but it's such a small binary, it's absolutely a perfect candidate to replace csup(1) in base! Please note, however, that this port does not yet support fetching over a secure channel (i.e. https), and therefore should only be used on trusted networks.
==== Legacy cvsup Status, (updated 2015-05-26): ====
* source, ports, cvsup access fully removed.
----
=== The src/base Fetch Problem ===
At the time of this writing, there is no ''in-base'' solution for fetching canonical RELEASE sources, (to build and maintain production grade systems).
'''UPDATE: no longer true; svnlite is in-base'''
For FreeBSD users, (particularly systems administrators and integrators), fetching source has always been easy - via either c[v]sup or CTM. Until recently, these tools could also be used to fetch the ports collection. CTM has not been heavily promoted and has not been as widely used as c[v]sup, but has a strong user base.
[[CvsIsDeprecated|CVS has been deprecated in FreeBSD development for several years now]], and all FreeBSD development has moved to the more modern [[http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/svn.html|SVN]]. While [[http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/svn.html|SVN]] is a great leap forward for the development process, c[v]sup has been heavily used by server administrators and system integrators for source management - and a replacement has fallen behind as the CVS infrastructure is being phased out.
=== Other methods of fetching source ===
The objective: ability to fetch sources for everything using just the tools in base.
1. svnup(1) a tiny utility which can pull deltas from SVN servers:
'''UPDATE: An early version is functional, but not in base. (2013 02 23)'''
The work has been added as a port, '''net/svnup'''.
[[http://lists.freebsd.org/pipermail/freebsd-stable/2013-February/072026.html|John Mehr is currently working on "a "lightweight, dependency-free, BSD licensed program to pull source using the svn protocol]]. svnup is still being optimized, but is tested functional. The original (very small) sources are available [[http://jcm.dsl.visi.com/freebsd/svnup/|on his website]].
This feels like the obvious solution, to put a small program in base. [[http://www.freebsd.org/cgi/man.cgi?query=portsnap&apropos=0&sektion=0&manpath=FreeBSD+9.1-RELEASE&arch=default&format=html|portsnap(8)]] is in base, for similar reason
[[http://lists.freebsd.org/pipermail/freebsd-stable/2013-January/071909.html|Arrigo Marchiori discusses DES's svnup]]
[[http://lists.freebsd.org/pipermail/freebsd-stable/2013-January/071836.html} I personally (maintainer of subversion port!) prefer csup over all other methods for "non-developers" systems too, and I'll be happy to see "svnup" when (if?) it will be created...]]
- Black Lion AKA Lev Serebryakov <lev@FreeBSD.org>
An early [[http://svnweb.freebsd.org/base/user/des/svnsup/|svnsup]] was started by [[DagErlingSmørgrav|DES]], but not completed.
2. [[http://www.freebsd.org/cgi/man.cgi?query=freebsd-update&apropos=0&sektion=0&manpath=FreeBSD+9.1-RELEASE&arch=default&format=html|freebsd-update(8)]] can be configured, via freebsd-update.conf(5), to download src only. Updates are signed and automatically verified by freebsd-update.
* freebsd-update(8) man states the tool is designed to: ''fetch and install binary updates to FreeBSD'', (awesome approach and possibilities, but opposite approach for what we're after here)
* Can only mirror -release branches
* Disheartening PR: [[http://www.freebsd.org/cgi/query-pr.cgi?pr=bin/147430|freebsd-update(8) not updating /usr/src/UPDATING]]
3. [[http://www.freebsd.org/cgi/man.cgi?query=ctm&sektion=1&apropos=0&manpath=FreeBSD+9.1-RELEASE|ctm(1)]] provides updates to all stable branches (4 through 9) and -current as well as the entire SVN repository. It can function in either push mode (subscribe to the relevant mailing list and receive deltas) or pull mode (fetch deltas from a FTP mirror).
* Can only currently mirror -stable branches and -current. Mirroring -release branches requires a different tool
* Pull mode is not secured out of the box, (c[v]sup shares this problem, though it validates checksums). Verifying signatures (in email, directly from shell, or by other means) requires installing PGP (one of the GPG ports, or source).
4. subversion-static port, just added to the ports tree early Feb 2013. This port aims to be a nearly dependency-free SVN build, built statically, as not to bring unwanted dependencies onto a system. While this is an awesome leap foreword for developers, it still carries a lot of the problems inherent in merely fetching base/src. The subversion-static port appears to be a lightweight and feature complete svn client for developers needs, as well.
5. CTMup '''UPDATE: stopped work, now that svnup(1) works''' [[http://lists.freebsd.org/pipermail/ctm-users/2013-January/000399.html|Isaac (.ike) Levy was working on a c[v]sup workalike in shell, which uses ctm(1)]]. Peculariaties of the ctm(1), particularly error/exit codes, made this a time consuming approach to implement. This 'ctmup' utility was meant to get a c[v]sup workalike out there fast, but it is not at all meant to replace the svnup work.
6. No in-base solution - let users fetch/package/distribute sources themselves. (possibly the worst solution). [[http://www.freebsd.org/cgi/man.cgi?query=fetch&apropos=0&sektion=0&manpath=FreeBSD+9.1-RELEASE&arch=default&format=html|fetch(1)]] exists. Leave users to their own devices- to download src.tbz tarballs via ftp/http infrastructure. Sorting out REL, RELENG, and STABLE from these tarballs is *quite* the trick- and obviously could potentially soak more bandwidth from Project online resources. Checksum/mtree userland tricks will pop up, hunting down signature files will become a new hassle. Feels like it's back to the stone age with this.
=== FAQ (from lists) ===
==== Q. Why is cvsup/CVS deprecated? ====
A. The FreeBSD project has been migrating development from CVS to SVN, since 2008. The CVS repository no longer exists.
==== Q. Why is installing SVN such a big deal for some users? ====
A. Several answers: This one tool is big, its license is not BSD.
* SVN carries the Apache 2 license, which is pretty BSD friendly, but it is not a BSD license. A number of its dependencies are extremely unfriendly to the BSD license, (GNU bits, bsddb, etc...) At best, a stripped down version, if it were sucked into base, would live in src/contrib. (And the chances of SVN being imported into the base system are negligible).
* svn is big:
* Source + dependencies for SVN: 496M (+)
* Source for FreeBSD 9.1: 746M (actual)
Stripping down the SVN dependencies, its still '''7.3%''' the size of the entire FreeBSD base/src:
* no-deps SVN Source alone: 55M
* Source for FreeBSD 9.1: 746M
That's a pretty big utility, to want a home in /usr/bin. SVN is also quite fat for many applied uses, e.g. when automating system installs, (with custom kernels, stripped down userland, etc...)
==== Q. Why would users/admins want to install from sources (e.g. buildworld/makeworld)? ====
A. There are many reasons, but the simplest are:
* custom kernel builds (more features, stripped features, tuning or problem solving)
* custom source patches for systems integrators, (things which the project will not accept, or are use privately)
* ability to pach any security problems *quickly*, this has made FreeBSD such a great driver of core internet infrastructure
* In coming years, t will be interesting to see how fast FreeBSD users actually dive into using/building/distributing binary updates using freebsd-update(8).
* FreeBSD users/admins typically '''adore''' C, UNIX.
==== Q. Why would users/admins want to install anything other than RELEASE? ====
A. Administrators run "production" environments using all of RELEASE, RELENG, STABLE, CURRENT.
* '''RELEASE''' - most folks live here, it is the conservative/stable choice
* '''RELENG''' -
* sometimes for a security fix
* sometimes for a bugfix
* '''STABLE'''
* prepping an environment for upcoming releases
* contain a necessary bugfix, new feature or other reason
(often A/B testing style, running on selected/safe machines etc...)
* '''CURRENT''' - yes, it happens in "prod"
(typically if CURRENT hits production, some extreme bug/security fix is being "back-patched" for temporary resolution)
==== Q. Why is this sort of trivial source fetching a big deal? ====
A. Many FreeBSD "users" come to the platform to try things like carp(4), not pidgin. They tend toward rational, clean, and honest systems- not user-friendly systems. While it is a great boon for FreeBSD developers to use SVN, it is now a great burden for users with different usage needs. Many *BSD enviornments keep their own source control systems, for other software which runs on *top* of the OS- so svn is cumbersome and in some cases, unwelcome based on problems described on this page.
The CVS/cvsup infrastructure is, in the meantime, falling into disrepair- ''and could even considered a security liability''. The FreeBSD developers are being spread even more thinly with growth pains right now, supporting CVS just for cvsup is quite a burden. It's got to turn off sometime... (And for the sake of time wasted in the FreeBSD project, the sooner- the better...)
----
== Here is a comparison between OLD/NEW source fetching ==
Users face a daunting change when cvsup / csup(1) is eventually end-of-lifed, the current state of SVN for users poses some seriously time consuming 'ouroboros' problems, evident below:
----
==== Existing source fetching via CTM (ongoing) ====
Reference: [[http://www.freebsd.org/doc/handbook/ctm.html]]
''It is unclear at this time weather or not CTM support will continue when the CVS servers are eventually turned off, (CTM is not tied to CVS directly, but the CTM deltas need to be generated regularly).''
* Email Push mode:
1. Subscribe to the appropriate ctm mailing list (using a dedicated address that forwards to '|ctm_rmail ...')
2. Follow instructions for [[http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/makeworld.html|buildworld/buildkernel dance]]...
* Pull mode:
1. Fetch new deltas from a FTP mirror
2. Run ctm_rmail on the new deltas
3. Follow instructions for [[http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/makeworld.html|buildworld/buildkernel dance]]...
Example based on notes from [[http://lists.freebsd.org/pipermail/freebsd-stable/2013-January/071921.html|Stephen Montgomery-Smith]]:
{{{{
## fetch 'xEmpty' file from FTP servers:
# fetch ftp://ftp.freebsd.org/pub/FreeBSD/CTM/src-9/src-9.0400xEmpty.gz
## src dir must be empty, to start
# mkdir -p /usr/src
# cd /usr/src && ctm /path/to/src-9.0400xEmpty.gz
## fetch the rest of the CTM patch whose number is greater than
## the xEmpty file you downloaded, and do:
## ftp://ftp.freebsd.org/pub/FreeBSD/CTM/src-9/
# cd /usr/src && ctm /path/to/src-9.04*
}}}}
Now in /usr/src, you will have a reasonably up to date version of
9.x-stable.
You can keep it up to date by getting more files, either from the ftp
site, or by email, and doing again
cd /usr/src && ctm the-rest-of-the-files*
It will automatically ignore the files already applied.
Similar instructions for all the other stable/currents and ports. Main
thing to remember - start with an empty directory.
Also making local changes is not permitted. If ctm tries to modify a
file whose md5 checksum has changed, it will quit with an error message.
(But it won't leave your system in an unusable state - if you put that
file back to its original state, then ctm will work again.)
Now, if you want something not offered by ctm (e.g. 8.2-release), then
you need to use svn. You can get svn via ctm. But you (1) need to
apply the patch, (2) install the svn port, and (3) install the xz port
if your FreeBSD is really old.
----
==== OLD source fetching via c[v]sup (deprecated): ====
1. csup one-liner:
{{{ # csup /path/to/ports-sup }}}
2. {{{ # move on to file }}}
or, from muscle memory,
{{{ # csup -h cvsup14.us.freebsd.org /usr/share/examples/standard-supfile }}}
''note: need to replace CHANGE_THIS.FreeBSD.org in the config file, or use -h flag when calling the csup utility''
2. # move on to [[http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/makeworld.html|buildworld/buildkernel dance]]...
----
==== NEW canonical source fetching: ====
1. Ensure ca_root_nss is installed
{{{{
# pkg install ca_root_nss
}}}}
2. If svnlite is not installed, or you prefer the official svn build
{{{{
# pkg install subversion
}}}}
3. Finally, use svnlite or svn to check out base, (typically to /usr/src)
{{{{
# svnlite checkout https://svn.FreeBSD.org/base/head /usr/src
}}}}
or:
{{{{
# svn checkout https://svn.FreeBSD.org/base/head /usr/src
}}}}
4. # move on to [[http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/makeworld.html|buildworld/buildkernel dance]]...
----
=== Related References, Constructive Rants ===
''A Generation Lost in the Bazaar''
Poul-Henning Kamp, 2012
http://queue.acm.org/detail.cfm?id=2349257
''FreeBSD, csup/cvsup, portsnap, and svn''
Jeremy Chadwick, 2013
http://koitsu.wordpress.com/2013/01/02/freebsd-csupcvsup-portsnap-and-svn/
Similar situation, before cvsup
Kent S. Gordon, 1997
http://www.mavetju.org/mail/view_message.php?list=freebsd-current&id=209027
----
CategoryHistorical