Subversion primer
NOTE: in addition to this, you should read http://people.freebsd.org/~peter/svn_notes.txt, which documents some of the gorier and more obscure details.
NOTE: The "subversion book" is an excellent reference. See the first few chapters of http://svnbook.red-bean.com/. At the time of writing, you want the 'nightly' build for svn-1.5.
Overview
The FreeBSD src repo switched from CVS to Subversion on 2008-06-03.
There are mechanisms in place to automatically merge changes back from the Subversion repository to the CVS repository, so regular users shouldn't notice any difference, but developers most definitely will.
Subversion is not so different from CVS in daily use, but there are differences. In addition, Subversion has a number of features that should make developers' lives easier. The most important advantage to Subversion (and the reason why we switched) is that it handles branches and merging much better than CVS does. Some of the principal differences are:
- Commits are atomic
- Revision numbers apply across the repository - all files that were modified in the same commit have the same revision number
- Branching and tagging are namespace operations
- Directories are versioned
Files and directories can have arbitrary, versioned metadata (properties) attached to them
- Files and directories can be copied, with full history tracking
- No more contortions due to cvs weaknesses, such as applying patch(1) files at compile time in order to avoiding touching of vendor branch code.
- No more repo-copies.
Getting started
There are basically three ways to get a working copy from Subversion.
Direct checkout
The first is to check out directly from the main repo:
$ svn checkout svn+ssh://svn.freebsd.org/base/head /usr/src
This will check out a CURRENT source tree as /usr/src. Replace /usr/src with whatever you want. If you omit the final argument, the working copy will be named head, but you can safely rename it to whatever you want.
svn+ssh means the SVN protocol tunneled over ssh; svn.freebsd.org is the name of the server, base is the path to the repo, head is the subdirectory within the repo.
This is the simplest method, but it's hard to tell just yet how much load it will place on the repo. Subversion is much faster than CVS (significantly less disk I/O), so it might not be a problem.
Note that svn diff - probably the most frequently used command - does not require access to the server, as SVN stores a reference copy of every file in your working copy (which unfortunately means that SVN working copies are very big)
Checkout from a mirror
You can check out a working copy from a mirror by simply substituting the mirror's URL for svn+ssh://svn.freebsd.org/base. This can be an official mirror, or a mirror you maintain yourself using svnsync or similar.
There is a serious disadvantage to this method: every time you want to commit, you have to svn switch --relocate to the master repo, then svn switch back to the mirror after you've committed. Also, since svn switch only works between repos that have the same UUID, you have to hack the UUID of your local repo before you can start using it.
Unlike with cvs and cvsup, you probably don't want the hassle of a local svnsync mirror unless your network connectivity situation or other considerations demand it. See section on setting up a mirror at the end.
Checkout from a local mirror using SVK
The third alternative is to use SVK to maintain a local mirror. Svk is a version control system built on top of the Subversion storage engine. It is identical to SVN in most respects, except that it allows you to set up parts of your repo as mirrors of other repos, and to keep local branches which you can merge back to the upstream repo. There are extensions that allows SVK to mirror CVS and Perforce repos in addition to Subversion repos.
There are disadvantages to SVK as well - one of them being that local revision numbers will not match upstream revision numbers, which makes it difficult to svk log or svk diff or svk update to an arbitrary upstream revision.
To set up a mirror of the FreeBSD repo:
$ svk mirror svn+ssh://svn.freebsd.org/base //freebsd/base
Your local SVK repo will be stored in ~/.svk/local; you can move it to wherever you like, but you have to update ~/.svk/config manually to reflect the move.
You can use any path you want instead of //freebsd/base. One common pattern is to place mirrors under //mirror (e.g. //mirror/freebsd/base) and local branches under //local.
Then you have to pull down the contents:
$ svk sync //freebsd/base
Note that svk sync will take a very long time (up to several days depending on your network connection). Peter Wemm has a tarball (freefall:/home/peter/dot_svk_r179527.tbz2) you can use to jumpstart your SVK mirror, but only if you don't already have one...
$ cd ~ $ scp freefall:/home/peter/dot_svk_r179646.tbz2 . $ tar xf dot_svk_r179646.tbz2
Then edit ~/.svk/config and replace /scratch/tmp/peter/.svk/local with the expanded equivalent of ~/.svk/local (e.g. /home/jarjar/.svk/local)
You can check out directly from your mirror:
$ svk checkout //freebsd/base/head /usr/src
Unlike SVN, SVK does not store metadata or reference copies in the working copy. All metadata is recorded in ~/.svk/config; reference copies aren't used at all since SVK always operates on a local repo.
When you commit from this working copy, SVK will commit directly to the upstream repo, then sync the mirror.
However, the "killer app" for SVK is the ability to work without a network connection. To do that, you have to set up a local branch:
$ svk mkdir //local/freebsd $ svk copy //freebsd/base/head //local/freebsd/head
Again, you can use any path you like instead of //local/freebsd/head.
You have to synchronize your local branch before you can use it:
$ svk pull //local/freebsd/head
Now you can check out from your local branch:
$ svk checkout //local/freebsd/head /usr/src
The point of this exercise is that you can commit work-in-progress to your local branch, and only push it to the upstream repo when you're done. The easy way to push is with svk push, but there is a serious disadvantage to it: it will push every single commit you made to your local branch incrementally instead of lumping them all into a single commit. Therefore, you should use svk smerge instead. For instance, if you've been working on the bge(4) driver:
$ cd /usr/src $ svk smerge -f sys/dev/bge
You'll know what to do from there.
RELENG_* branches and general layout
Given svn+ssh://svn.freebsd.org/base, base refers to the src tree. There will presumably be others. Expect things like /ports, /scratch, /proj, etc. These are separate repositories with their own change number sequences, access controls and commit mail.
For the "base" repository, /head refers to the -current tree. /head/bin/ls is what would go in /usr/src/bin/ls in a release.
The other key locations are:
/stable/<n> corresponds to RELENG_<n>.
/releng/<n.n> corresponds to RELENG_<n_n>
/release/<n.n.n> corresponds to RELENG_<n_n_n>_RELEASE
/vendor* vendor branch import work area
/projects and /user feature branch work area, like we have in perforce.
Daily use
This section will show you how to perform common day-to-day operations with Subversion. There should be no difference between SVN and SVK in daily use, except for the revision renumbering mentioned earlier.
Note that SVN / SVK commands that have direct CVS equivalents usually have the same name, including abbreviations (for instance, checkout and co', update and up', commit and ci).
Help
Both SVN and SVK have built-in documentation:
$ svn help $ svn help checkout
Checkout
As seen earlier:
$ svn checkout svn+ssh://svn.freebsd.org/base/head /usr/src
At some point, you will probably need more than just head - for instance, you may want to merge changes to stable/7. Therefore, it may be useful to have a partial checkout of the complete tree (a full checkout would be very painful).
First, check out the root of the repo:
$ svn checkout --depth=immediates svn+ssh://svn.freebsd.org/base
This will give you base with all the files it contains (currently just ROADMAP.txt) and empty subdirectories (head, stable, vendor etc.)
You can expand your working copy by changing the depth of the various subdirectories:
$ svn up --set-depth=infinity base/head $ svn up --set-depth=immediates base/release base/releng base/stable
This will give you a full copy of head, and empty copies of every release tag and every releng and stable branch.
If, at a later point, you need to merge something to the 7-STABLE, you can expand your working copy again:
$ svn up --set-depth=infinity base/stable/7
Note that you don't have to expand a complete subtree - for instance, you can expand only stable/7/sys, then later expand the rest of stable/7:
$ svn up --set-depth=infinity base/stable/7/sys $ svn up --set-depth=infinity base/stable/7
You can svn update the root of your working copy without fear that it will pull down the entire tree. It will only update what you previously asked for - in this case, head and stable/7.
The flip side is that you can only increase the depth of a working copy, never decrease it.
Update
Update a working copy to the latest revision, or to a specific revision:
$ svn update $ svn update -r12345
Status
See what local changes you have in your working copy:
$ svn status
This has no direct equivalent in CVS: the cvs up -N command will not only show local changes, but also indicate files that are out-of-date. This corresponds to the following SVN command:
$ svn status --show-updates
Edit and commit
Like CVS (and unlike Perforce), SVN and SVK don't need to be told in advance that you intend to edit a file.
The svn commit command works pretty much just like its CVS equivalent.
Commit all changes in the current directory and all subdirectories:
$ svn commit
Commit all changes in the lib/libfetch and usr.bin/fetch directories in a single operation:
$ svn commit lib/libfetch usr.bin/fetch
Add and remove
NOTE: Before adding files, get a copy of http://people.freebsd.org/~peter/auto-props.txt and edit it into your ~/.subversion/config file according to the instructions in the file. If you added something before you've read this, you may use svn rm --keep-local for just added files, fix your config file and re-add them again. The initial config file is created when you first run a svn command, even something as simple as: svn help
Files are added with svn add, just like with CVS:
$ touch foo $ svn add foo
and removed with svn remove:
$ svn remove foo
Unlike CVS, Subversion does not require you to rm the file before you svn rm it; in fact, it will complain if you do, but you can shut it up with --force.
Directories can also be added with svn add:
$ mkdir bar $ svn add bar
However, if the directory does not already exist, it is simpler to just use svn mkdir:
$ svn mkdir bar
In CVS, the directory is immediately created in the repository when you cvs add it; this is not the case in Subversion. Furthermore, unlike CVS, Subversion allows you to remove directories using svn rm (there is no separate svn rmdir command):
$ svn rm bar
Copy and move
The following (obviously) creates a copy of foo.c named bar.c:
$ svn copy foo.c bar.c
This has no equivalent in CVS.
Moving or renaming a file is also supported:
$ svn move foo.c bar.c
This is actually the exact equivalent of
$ svn copy foo.c bar.c $ svn remove foo.c
Log and annotate
The svn log command works pretty much like cvs log. The most important difference is that svn log on a directory will show all revisions affecting the directory or files in it, in reverse chronological order, while cvs log will show the complete log for each file in the directory, including duplicate entries for revisions that affect multiple files.
The svn annotate command (or, depending on your mood svn blame or svn praise) is equivalent to cvs annotate, with some differences in output format.
Diff
As in CVS, the svn diff command displays changes to your working copy. The main differences are that it uses the unified diff format by default (similar to cvs diff -u), and it always shows diffs for added files (similar to cvs diff -N).
Like cvs diff, svn diff can show the changes between two revisions of the same file:
$ svn diff -r179453:179454 ROADMAP.txt
It can also show all changes for a specific changeset. The following will show what changes were made to the current directory and all subdirectories in changeset 179454:
$ svn diff -c179454 .
Revert
Local changes (including additions and deletions) can be reverted using svn revert. Unlike cvs up -C, it does not update out-of-date files - it just replaces them with pristine copies of the version you already have.
Conflicts
If an svn update resulted in a merge conflict, Subversion will remember which files have conflicts and refuse to commit any changes to those files until you've explicitly told it that the conflicts have been resolved. The simple (but deprecated) procedure is the following:
$ svn resolved foo
The preferred procedure is to use svn resolve (no d) and specify the correct action:
$ svn resolve --accept=working foo
This is equivalent to the previous example. Possible values for --accept are:
working: use the version in your working directory (which one presumes has been edited to resolve the conflicts)
base: use a pristine copy of the version you had before svn update, discarding your own changes, the conflicting changes, and possibly other intervening changes as well.
mine-full: use what you had before svn update, including your own changes, but discarding the conflicting changes, and possibly other intervening changes as well.
theirs-full: use the version that was retrieved when you did svn update, discarding your own changes.
Advanced use
Sparse checkouts
The equivalent to cvs checkout -l (to check out a directory without its subdirectories) is svn checkout -N. However, unlike CVS, SVN remembers the -N, so you can't just svn update later to get the subdirectories. In Subversion 1.5 and newer, -N has been deprecated in favor of the --depth option, which allows precise control of this feature:
$ svn checkout -N svn+ssh://svn.freebsd.org/base ~/freebsd
is equivalent to
$ svn checkout --depth=empty svn+ssh://svn.freebsd.org/base ~/freebsd
Valid arguments to --depth are:
empty: the directory itself without any of its contents
files: the directory and any files it contains
immediates: the directory, any files and directories it contains, but none of the subdirectories' contents
infinity: everything
The --depth option applies to many other commands as well, including svn commit, svn revert and svn diff.
Since --depth is sticky, there is a --set-depth option for the update command that will change the selected depth. Thus, given the working copy produced by the previous example:
$ cd ~/freebsd $ svn update --set-depth=immediates .
You will now have ROADMAP.txt and a bunch of empty subdirectories, but if you svn update any of these, nothing will happen. However,
$ svn update --set-depth=infinity head
will set the depth for head to infinity, and fully populate it.
Direct operation
Certain operations can be performed directly on the repo, without touching your working copy. Specifically, this applies to any operation that does not require editing a file, including:
log, diff
mkdir
remove, copy, rename
propset, propdel, propedit
merge
The most important consequence is that branching is very fast. The following command:
$ svn copy svn+ssh://svn.freebsd.org/base/head svn+ssh://svn.freebsd.org/base/stable/8
would be used to branch what RELENG_8, and is equivalent to the following sequence:
$ svn checkout --depth=immediates svn+ssh://svn.freebsd.org/base $ cd base $ svn update --depth=infinity head $ svn copy head stable/8 $ svn commit stable/8
except that the former takes seconds, while the latter may take minutes to hours, depending on your network connection.
Merging
Vendor imports
See SubversionPrimer/VendorImports
Reverting a commit
Reverting to a previous revision is fairly easy:
$ svn merge -r179454:179453 ROADMAP.txt $ svn commit
or, use change number syntax. Negative means a reverse change.
$ svn merge -c -179454 ROADMAP.txt $ svn commit
This can also be done directly in the repo:
$ svn merge -r179454:179453 svn+ssh://svn.freebsd.org/base/ROADMAP.txt
Reverting the deletion of a file is slightly different. You need to copy a version of the file that predates the deletion (typically, to restore a file that was deleted in revision N, you would copy back revision N-1):
$ svn copy svn+ssh://svn.freebsd.org/base/ROADMAP.txt@179454 . $ svn commit
or
$ svn copy svn+ssh://svn.freebsd.org/base/ROADMAP.txt@179454 svn+ssh://svn.freebsd.org/base
Do not simply recreate the file manually and svn add it - this will lose history.
Fixing mistakes
While we can do surgery in an emergency, do not plan on having mistakes fixed behind the scenes. Plan on mistakes remaining in the logs forever. Be sure to check the output of svn status and svn diff before committing.
Mistakes will happen. But, unlike CVS, they can generally be fixed without disruption.
Take a case of adding a file in the wrong location. The right thing to do is to 'svn mv' the file to the correct location and commit. This causes just a couple of lines of metadata in the repository journal. The logs are all linked up correctly.
The wrong thing to do is to delete the file and then 'svn add' an independent copy in the correct location. Instead of a couple of lines of text, the repository journal grows an entire new copy of the file. This is a waste.
Setting up a svnsync mirror
You probably do not want to do this unless there is a good reason for it. Such reasons might be to support many multiple local read-only client machines, or if your network bandwidth is limited. Starting a fresh mirror from empty would take a very long time. Expect a minimum of 10 hours for high speed connectivity. If you have international links, expect this to take 4 to 10 times longer.
A far better option is to grab a seed file. It is large (~1GB) but will consume less network traffic and take less time to fetch than a svnsync will.
$ rsync -va --partial --progress freefall:/home/peter/svnmirror-base-r179637.tbz2 .
or:
$ rsync -va --partial --progress rsync://repoman.freebsd.org:50873/svnseed/svnmirror-base-r179637.tbz2 .
or by ftp, presumably a local mirror:
$ fetch ftp://ftp.freebsd.org/pub/FreeBSD/development/subversion/svnmirror-base-r179637.tbz2
Once you have the file, extract it to somewhere like /home/svnmirror/base. You then update it so that it fetches the changes after rev 17963. In the example location above:
$ svnsync sync file:///home/svnmirror/base
You can then set that up to run from cron, do checkouts locally, set up a svnserve server for your local machines to talk to, etc.
The seed mirror is set to fetch from svn://svn.freebsd.org/base. The configuration for the mirror is stored in revprop 0 on the local mirror. To see the configuration, try:
$ svn proplist -v --revprop -r 0 file:///home/svnmirror/base
Use propset to change things.
Tips
- In commit logs etc. "rev 179872" should be spelled "r179872" per convention.
- You can speed up your checkouts and minimize network traffic a bit. Here's a recipe:
$ svn co --depth=empty svn+ssh://svn.freebsd.org/base fbsvn $ cd fbsvn $ svn up --depth=empty stable $ svn up head $ cd stable $ cp -r ../head/ 7 $ cd 7 $ svn switch svn+ssh://svn.freebsd.org/base/stable/7 $ cd .. $ cp -r 7/ 6 $ cd 6 $ svn switch svn+ssh://svn.freebsd.org/base/stable/6
What this bit of evil does is check out head, stable/7 and stable/6. We create the empty checkout directories under svn's control. In svn, subtrees are self identifying, like in cvs. We check out head and clone it as stable/7. Except we don't want the head version so we "switch" it to the 7.x tree location. Svn downloads diffs to convert the "head" files to "stable/7" instead of doing a fresh checkout. The same goes for stable/6. This definitely counts as "abuse" of the working copy client code so there is no warranty. 