Page created: 2013-01-27

This page is being linked to by various *BSD news sources and user groups, (Patrick McEvoy et. al.), we'll try to keep this page up to date with the current state of this issue. Eventually, I hope this page exists merely to announce it's own deprecation- once there is a resolution accepted the FreeBSD community at large.

UPDATE

There is now a working svnup(1) utility, which is tested functional!

Fetch ports sources, then:

# cd /usr/ports/net/svnup && make install
# svnup stable -h svn0.us-west.freebsd.org

Currently it's a port- but it's such a small binary, it's absolutely a perfect candidate to replace csup(1) in base! Please note, however, that this port does not yet support fetching over a secure channel (i.e. https), and therefore should only be used on trusted networks.

Legacy cvsup Status, (updated 2013-03-10):


The src/base Fetch Problem

At the time of this writing, there is no in-base solution for fetching canonical RELEASE sources, (to build and maintain production grade systems).

For FreeBSD users, (particularly systems administrators and integrators), fetching source has always been easy - via either c[v]sup or CTM. Until recently, these tools could also be used to fetch the ports collection. CTM has not been heavily promoted and has not been as widely used as c[v]sup, but has a strong user base.

CVS has been deprecated in FreeBSD development for several years now, and all FreeBSD development has moved to the more modern SVN. While SVN is a great leap forward for the development process, c[v]sup has been heavily used by server administrators and system integrators for source management - and a replacement has fallen behind as the CVS infrastructure is being phased out.

Other methods of fetching source

The objective: ability to fetch sources for everything using just the tools in base.

  1. svnup(1) a tiny utility which can pull deltas from SVN servers:
  2. freebsd-update(8) can be configured, via freebsd-update.conf(5), to download src only. Updates are signed and automatically verified by freebsd-update.

    • freebsd-update(8) man states the tool is designed to: fetch and install binary updates to FreeBSD, (awesome approach and possibilities, but opposite approach for what we're after here)

    • Can only mirror -release branches
    • Disheartening PR: freebsd-update(8) not updating /usr/src/UPDATING

  3. ctm(1) provides updates to all stable branches (4 through 9) and -current as well as the entire SVN repository. It can function in either push mode (subscribe to the relevant mailing list and receive deltas) or pull mode (fetch deltas from a FTP mirror).

    • Can only currently mirror -stable branches and -current. Mirroring -release branches requires a different tool
    • Pull mode is not secured out of the box, (c[v]sup shares this problem, though it validates checksums). Verifying signatures (in email, directly from shell, or by other means) requires installing PGP (one of the GPG ports, or source).
  4. subversion-static port, just added to the ports tree early Feb 2013. This port aims to be a nearly dependency-free SVN build, built statically, as not to bring unwanted dependencies onto a system. While this is an awesome leap foreword for developers, it still carries a lot of the problems inherent in merely fetching base/src. The subversion-static port appears to be a lightweight and feature complete svn client for developers needs, as well.
  5. CTMup UPDATE: stopped work, now that svnup(1) works Isaac (.ike) Levy was working on a c[v]sup workalike in shell, which uses ctm(1). Peculariaties of the ctm(1), particularly error/exit codes, made this a time consuming approach to implement. This 'ctmup' utility was meant to get a c[v]sup workalike out there fast, but it is not at all meant to replace the svnup work.

  6. No in-base solution - let users fetch/package/distribute sources themselves. (possibly the worst solution). fetch(1) exists. Leave users to their own devices- to download src.tbz tarballs via ftp/http infrastructure. Sorting out REL, RELENG, and STABLE from these tarballs is *quite* the trick- and obviously could potentially soak more bandwidth from Project online resources. Checksum/mtree userland tricks will pop up, hunting down signature files will become a new hassle. Feels like it's back to the stone age with this.

FAQ (from lists)

Q. Why is cvsup/CVS deprecated?

  1. The FreeBSD project has been migrating development from CVS to SVN, since 2008. Active development using the CVS repositories is over, and cvs service is currently becoming extra management overhead. It needs to go, calmly. The requirement to support the CVS exporter prevents use of some SVN functionality.

Q. Why is installing SVN such a big deal for some users?

  1. Several answers: This one tool is big, its license is not BSD.
  2. SVN carries the Apache 2 license, which is pretty BSD friendly, but it is not a BSD license. A number of its dependencies are extremely unfriendly to the BSD license, (GNU bits, bsddb, etc...) At best, a stripped down version, if it were sucked into base, would live in src/contrib. (And the chances of SVN being imported into the base system are negligible).
  3. svn is big:
    • Source + dependencies for SVN: 496M (+)
    • Source for FreeBSD 9.1: 746M (actual)

      Stripping down the SVN dependencies, its still 7.3% the size of the entire FreeBSD base/src:

    • no-deps SVN Source alone: 55M
    • Source for FreeBSD 9.1: 746M That's a pretty big utility, to want a home in /usr/bin. SVN is also quite fat for many applied uses, e.g. when automating system installs, (with custom kernels, stripped down userland, etc...)

Q. Why would users/admins want to install from sources (e.g. buildworld/makeworld)?

  1. There are many reasons, but the simplest are:
    • custom kernel builds (more features, stripped features, tuning or problem solving)
    • custom source patches for systems integrators, (things which the project will not accept, or are use privately)
    • ability to pach any security problems *quickly*, this has made FreeBSD such a great driver of core internet infrastructure
    • In coming years, t will be interesting to see how fast FreeBSD users actually dive into using/building/distributing binary updates using freebsd-update(8).
    • FreeBSD users/admins typically adore C, UNIX.

Q. Why would users/admins want to install anything other than RELEASE?

  1. Administrators run "production" environments using all of RELEASE, RELENG, STABLE, CURRENT.
    • RELEASE - most folks live here, it is the conservative/stable choice

    • RELENG -

      • sometimes for a security fix
      • sometimes for a bugfix
    • STABLE

      • prepping an environment for upcoming releases
      • contain a necessary bugfix, new feature or other reason (often A/B testing style, running on selected/safe machines etc...)
    • CURRENT - yes, it happens in "prod"

      • (typically if CURRENT hits production, some extreme bug/security fix is being "back-patched" for temporary resolution)

Q. Why is this sort of trivial source fetching a big deal?

  1. Many FreeBSD "users" come to the platform to try things like carp(4), not pidgin. They tend toward rational, clean, and honest systems- not user-friendly systems. While it is a great boon for FreeBSD developers to use SVN, it is now a great burden for users with different usage needs. Many *BSD enviornments keep their own source control systems, for other software which runs on *top* of the OS- so svn is cumbersome and in some cases, unwelcome based on problems described on this page.

    The CVS/cvsup infrastructure is, in the meantime, falling into disrepair- and could even considered a security liability. The FreeBSD developers are being spread even more thinly with growth pains right now, supporting CVS just for cvsup is quite a burden. It's got to turn off sometime... (And for the sake of time wasted in the FreeBSD project, the sooner- the better...)


Here is a comparison between OLD/NEW source fetching

Users face a daunting change when cvsup / csup(1) is eventually end-of-lifed, the current state of SVN for users poses some seriously time consuming 'ouroboros' problems, evident below:


Existing source fetching via CTM (ongoing)

Reference: http://www.freebsd.org/doc/handbook/ctm.html

It is unclear at this time weather or not CTM support will continue when the CVS servers are eventually turned off, (CTM is not tied to CVS directly, but the CTM deltas need to be generated regularly).

Example based on notes from Stephen Montgomery-Smith:

## fetch 'xEmpty' file from FTP servers:
# fetch ftp://ftp.freebsd.org/pub/FreeBSD/CTM/src-9/src-9.0400xEmpty.gz

## src dir must be empty, to start
# mkdir -p /usr/src
# cd /usr/src && ctm /path/to/src-9.0400xEmpty.gz

## fetch the rest of the CTM patch whose number is greater than
## the xEmpty file you downloaded, and do:
## ftp://ftp.freebsd.org/pub/FreeBSD/CTM/src-9/
# cd /usr/src && ctm /path/to/src-9.04*

Now in /usr/src, you will have a reasonably up to date version of 9.x-stable.

You can keep it up to date by getting more files, either from the ftp site, or by email, and doing again cd /usr/src && ctm the-rest-of-the-files* It will automatically ignore the files already applied.

Similar instructions for all the other stable/currents and ports. Main thing to remember - start with an empty directory.

Also making local changes is not permitted. If ctm tries to modify a file whose md5 checksum has changed, it will quit with an error message.

file back to its original state, then ctm will work again.)

Now, if you want something not offered by ctm (e.g. 8.2-release), then you need to use svn. You can get svn via ctm. But you (1) need to apply the patch, (2) install the svn port, and (3) install the xz port if your FreeBSD is really old.


OLD source fetching via c[v]sup (deprecated):

  1. csup one-liner:
    •  # csup /path/to/ports-sup 

  2.  # move on to file 

    • or, from muscle memory,

       # csup -h cvsup14.us.freebsd.org /usr/share/examples/standard-supfile 

      note: need to replace CHANGE_THIS.FreeBSD.org in the config file, or use -h flag when calling the csup utility

  3. # move on to buildworld/buildkernel dance...


NEW canonical source fetching (work in progress behind SVN move):

  1. Install subversion - at the time of this writing, choose your own adventure

  2.  #noop# pkg_add -r subversion 

  3.  #noop# pkg install devel/subversion }

    • (noop note: At the time of this writing, due to the security incident cleanup, no binaries available)

  4. fetch ports using portsnap, and familiarize yourself with the new pkg(8) tool pkgng or pkg(8), and don't forget there are still old tools necessary, like pkg_info

  5. Finally install subversion,
    •     # cd /usr/ports/devel/subversion
          # make install clean

      Now that we have subversion, we just acquired this on our system too:

    • SQLite3 (not the SQLite in base
    • APR
    • Expat 2.x (not the Expat/libbsdxml now in base)
    • Neon or Serf
    • GNU gettext and libintl
    • libiconv Any of these could be a problem/conflict/undesirable for system integrators. APR for example could be a big problem if you're next steps are to load up a particular version of the Apache web server, (like a great number of FreeBSD users do).
  6. Finally, use svn to check out base, (typically to /usr/src)
    •     # svn checkout https://svn0.us-east.FreeBSD.org/base/head /usr/src
          Error validating server certificate for 'https://svn0.us-east.freebsd.org:443':
          - The certificate is not issued by a trusted authority. Use the
            fingerprint to validate the certificate manually!
          - The certificate hostname does not match.
          Certificate information:
          - Hostname: svnmir.nyi.FreeBSD.org
          - Valid: from Aug 12 23:01:31 2012 GMT until Aug 12 23:01:31 2013 GMT
          - Issuer: clusteradm, FreeBSD.org, (null), CA, US (clusteradm@FreeBSD.org)
          - Fingerprint: 06:D1:23:DE:5E:7A:F7:2B:7A:7E:74:95:5F:54:8D:5C:B0:D6:2E:8F
          (R)eject, accept (t)emporarily or accept (p)permanently?
  7. # move on to buildworld/buildkernel dance...


A Generation Lost in the Bazaar Poul-Henning Kamp, 2012 http://queue.acm.org/detail.cfm?id=2349257

FreeBSD, csup/cvsup, portsnap, and svn Jeremy Chadwick, 2013 http://koitsu.wordpress.com/2013/01/02/freebsd-csupcvsup-portsnap-and-svn/

Similar situation, before cvsup Kent S. Gordon, 1997 http://www.mavetju.org/mail/view_message.php?list=freebsd-current&id=209027

UsersFetchingSource (last edited 2013-08-12 06:11:15 by ColinPercival)