Page created: 2013-01-27

This page is being linked to by various *BSD news sources and user groups, (Patrick McEvoy et. al.), we'll try to keep this page up to date with the current state of this issue. Eventually, I hope this page exists merely to announce it's own deprecation- once there is a resolution accepted the FreeBSD community at large.

UPDATE 2 : 2015-08-06

Most of the contents of this page are seriously out of date.

There is now an in-base version of svn, as /usr/bin/svnlite. This is a stripped down but fully functional client but has the bulky API / language bindings removed. It has a relatively lightweight footprint and no external dependencies. svnlite will make use of the ca_root_nss package in order to validate https certificates, if it is present.

If you do not have svnlite, or prefer to use the official packaged version, it can be securely obtained:

# pkg install ca_root_nss
# pkg install subversion

If you wish to use secure https urls with svnlite, you should install the ca_root_nss package to obtain the current root certificate lists.

# pkg install ca_root_nss

You should use https://svn.freebsd.org/ where practical and so long as ca_root_nss is installed and up to date, the certificate should be automatically checked and securely verified.

svn.freebsd.org connects you to the closest project-operated mirror. The old legacy mirror names should not be used.

Official documentation is here: https://www.freebsd.org/doc/handbook/svn.html

As of writing:

Partially obsolete information follows:

UPDATE

There is now a working svnup(1) utility, which is tested functional!

Fetch ports sources, then:

# pkg install svnup
# svnup stable -h svn.freebsd.org

Currently it's a port- but it's such a small binary, it's absolutely a perfect candidate to replace csup(1) in base! Please note, however, that this port does not yet support fetching over a secure channel (i.e. https), and therefore should only be used on trusted networks.

Legacy cvsup Status, (updated 2015-05-26):


The src/base Fetch Problem

At the time of this writing, there is no in-base solution for fetching canonical RELEASE sources, (to build and maintain production grade systems).

UPDATE: no longer true; svnlite is in-base

For FreeBSD users, (particularly systems administrators and integrators), fetching source has always been easy - via either c[v]sup or CTM. Until recently, these tools could also be used to fetch the ports collection. CTM has not been heavily promoted and has not been as widely used as c[v]sup, but has a strong user base.

CVS has been deprecated in FreeBSD development for several years now, and all FreeBSD development has moved to the more modern SVN. While SVN is a great leap forward for the development process, c[v]sup has been heavily used by server administrators and system integrators for source management - and a replacement has fallen behind as the CVS infrastructure is being phased out.

Other methods of fetching source

The objective: ability to fetch sources for everything using just the tools in base.

  1. svnup(1) a tiny utility which can pull deltas from SVN servers:
  2. freebsd-update(8) can be configured, via freebsd-update.conf(5), to download src only. Updates are signed and automatically verified by freebsd-update.

    • freebsd-update(8) man states the tool is designed to: fetch and install binary updates to FreeBSD, (awesome approach and possibilities, but opposite approach for what we're after here)

    • Can only mirror -release branches
    • Disheartening PR: freebsd-update(8) not updating /usr/src/UPDATING

  3. ctm(1) provides updates to all stable branches (4 through 9) and -current as well as the entire SVN repository. It can function in either push mode (subscribe to the relevant mailing list and receive deltas) or pull mode (fetch deltas from a FTP mirror).

    • Can only currently mirror -stable branches and -current. Mirroring -release branches requires a different tool
    • Pull mode is not secured out of the box, (c[v]sup shares this problem, though it validates checksums). Verifying signatures (in email, directly from shell, or by other means) requires installing PGP (one of the GPG ports, or source).
  4. subversion-static port, just added to the ports tree early Feb 2013. This port aims to be a nearly dependency-free SVN build, built statically, as not to bring unwanted dependencies onto a system. While this is an awesome leap foreword for developers, it still carries a lot of the problems inherent in merely fetching base/src. The subversion-static port appears to be a lightweight and feature complete svn client for developers needs, as well.
  5. CTMup UPDATE: stopped work, now that svnup(1) works Isaac (.ike) Levy was working on a c[v]sup workalike in shell, which uses ctm(1). Peculariaties of the ctm(1), particularly error/exit codes, made this a time consuming approach to implement. This 'ctmup' utility was meant to get a c[v]sup workalike out there fast, but it is not at all meant to replace the svnup work.

  6. No in-base solution - let users fetch/package/distribute sources themselves. (possibly the worst solution). fetch(1) exists. Leave users to their own devices- to download src.tbz tarballs via ftp/http infrastructure. Sorting out REL, RELENG, and STABLE from these tarballs is *quite* the trick- and obviously could potentially soak more bandwidth from Project online resources. Checksum/mtree userland tricks will pop up, hunting down signature files will become a new hassle. Feels like it's back to the stone age with this.

FAQ (from lists)

Q. Why is cvsup/CVS deprecated?

  1. The FreeBSD project has been migrating development from CVS to SVN, since 2008. The CVS repository no longer exists.

Q. Why is installing SVN such a big deal for some users?

  1. Several answers: This one tool is big, its license is not BSD.
  2. SVN carries the Apache 2 license, which is pretty BSD friendly, but it is not a BSD license. A number of its dependencies are extremely unfriendly to the BSD license, (GNU bits, bsddb, etc...) At best, a stripped down version, if it were sucked into base, would live in src/contrib. (And the chances of SVN being imported into the base system are negligible).
  3. svn is big:
    • Source + dependencies for SVN: 496M (+)
    • Source for FreeBSD 9.1: 746M (actual)

      Stripping down the SVN dependencies, its still 7.3% the size of the entire FreeBSD base/src:

    • no-deps SVN Source alone: 55M
    • Source for FreeBSD 9.1: 746M That's a pretty big utility, to want a home in /usr/bin. SVN is also quite fat for many applied uses, e.g. when automating system installs, (with custom kernels, stripped down userland, etc...)

Q. Why would users/admins want to install from sources (e.g. buildworld/makeworld)?

  1. There are many reasons, but the simplest are:
    • custom kernel builds (more features, stripped features, tuning or problem solving)
    • custom source patches for systems integrators, (things which the project will not accept, or are use privately)
    • ability to pach any security problems *quickly*, this has made FreeBSD such a great driver of core internet infrastructure
    • In coming years, t will be interesting to see how fast FreeBSD users actually dive into using/building/distributing binary updates using freebsd-update(8).
    • FreeBSD users/admins typically adore C, UNIX.

Q. Why would users/admins want to install anything other than RELEASE?

  1. Administrators run "production" environments using all of RELEASE, RELENG, STABLE, CURRENT.
    • RELEASE - most folks live here, it is the conservative/stable choice

    • RELENG -

      • sometimes for a security fix
      • sometimes for a bugfix
    • STABLE

      • prepping an environment for upcoming releases
      • contain a necessary bugfix, new feature or other reason (often A/B testing style, running on selected/safe machines etc...)
    • CURRENT - yes, it happens in "prod"

      • (typically if CURRENT hits production, some extreme bug/security fix is being "back-patched" for temporary resolution)

Q. Why is this sort of trivial source fetching a big deal?

  1. Many FreeBSD "users" come to the platform to try things like carp(4), not pidgin. They tend toward rational, clean, and honest systems- not user-friendly systems. While it is a great boon for FreeBSD developers to use SVN, it is now a great burden for users with different usage needs. Many *BSD enviornments keep their own source control systems, for other software which runs on *top* of the OS- so svn is cumbersome and in some cases, unwelcome based on problems described on this page.

    The CVS/cvsup infrastructure is, in the meantime, falling into disrepair- and could even considered a security liability. The FreeBSD developers are being spread even more thinly with growth pains right now, supporting CVS just for cvsup is quite a burden. It's got to turn off sometime... (And for the sake of time wasted in the FreeBSD project, the sooner- the better...)


Here is a comparison between OLD/NEW source fetching

Users face a daunting change when cvsup / csup(1) is eventually end-of-lifed, the current state of SVN for users poses some seriously time consuming 'ouroboros' problems, evident below:


Existing source fetching via CTM (ongoing)

Reference: http://www.freebsd.org/doc/handbook/ctm.html

It is unclear at this time weather or not CTM support will continue when the CVS servers are eventually turned off, (CTM is not tied to CVS directly, but the CTM deltas need to be generated regularly).

Example based on notes from Stephen Montgomery-Smith:

## fetch 'xEmpty' file from FTP servers:
# fetch ftp://ftp.freebsd.org/pub/FreeBSD/CTM/src-9/src-9.0400xEmpty.gz

## src dir must be empty, to start
# mkdir -p /usr/src
# cd /usr/src && ctm /path/to/src-9.0400xEmpty.gz

## fetch the rest of the CTM patch whose number is greater than
## the xEmpty file you downloaded, and do:
## ftp://ftp.freebsd.org/pub/FreeBSD/CTM/src-9/
# cd /usr/src && ctm /path/to/src-9.04*

Now in /usr/src, you will have a reasonably up to date version of 9.x-stable.

You can keep it up to date by getting more files, either from the ftp site, or by email, and doing again cd /usr/src && ctm the-rest-of-the-files* It will automatically ignore the files already applied.

Similar instructions for all the other stable/currents and ports. Main thing to remember - start with an empty directory.

Also making local changes is not permitted. If ctm tries to modify a file whose md5 checksum has changed, it will quit with an error message.

file back to its original state, then ctm will work again.)

Now, if you want something not offered by ctm (e.g. 8.2-release), then you need to use svn. You can get svn via ctm. But you (1) need to apply the patch, (2) install the svn port, and (3) install the xz port if your FreeBSD is really old.


OLD source fetching via c[v]sup (deprecated):

  1. csup one-liner:
    •  # csup /path/to/ports-sup 

  2.  # move on to file 

    • or, from muscle memory,

       # csup -h cvsup14.us.freebsd.org /usr/share/examples/standard-supfile 

      note: need to replace CHANGE_THIS.FreeBSD.org in the config file, or use -h flag when calling the csup utility

  3. # move on to buildworld/buildkernel dance...


NEW canonical source fetching:

  1. Ensure ca_root_nss is installed
    •     # pkg install ca_root_nss
  2. If svnlite is not installed, or you prefer the official svn build
    •     # pkg install subversion
  3. Finally, use svnlite or svn to check out base, (typically to /usr/src)
    •     # svnlite checkout https://svn.FreeBSD.org/base/head /usr/src
      or:
          # svn checkout https://svn.FreeBSD.org/base/head /usr/src
  4. # move on to buildworld/buildkernel dance...


A Generation Lost in the Bazaar Poul-Henning Kamp, 2012 http://queue.acm.org/detail.cfm?id=2349257

FreeBSD, csup/cvsup, portsnap, and svn Jeremy Chadwick, 2013 http://koitsu.wordpress.com/2013/01/02/freebsd-csupcvsup-portsnap-and-svn/

Similar situation, before cvsup Kent S. Gordon, 1997 http://www.mavetju.org/mail/view_message.php?list=freebsd-current&id=209027

UsersFetchingSource (last edited 2015-08-06 21:06:48 by PeterWemm)