• SubversionPrimer
• VendorImports

Vendor imports with Subversion

This page describes the vendor import procedure.

PLEASE read the entire page before you start your own vendor import!

A note about patches

Patches to vendor code fall into two categories:

• Vendor patches: these are patches that have been issued by the vendor, or that you have extracted from the vendor's version control system, which address issues which in your opinion can't wait until the next vendor release.
• FreeBSD patches: these are patches that modify the vendor code to address FreeBSD-specific issues.

The nature of a patch dictates where it should be committed:

• Vendor patches should be committed to the vendor branch, and merged from there to head. If the patch addresses an issue in a new release that you are currently importing, do not commit it along with the new release: import and tag the release, then apply and commit the patch. There is no need to re-tag the vendor sources after committing the patch.

• FreeBSD patches should be committed directly to head.

Preparing the tree

If this is the first import you do after the switch to Subversion, you will have to flatten and clean up the vendor tree, and bootstrap merge history in the main tree.

Flattening

During the conversion from CVS to Subversion, vendor branches were imported with the same layout as the main tree. This means that the pf vendor sources ended up in vendor/pf/dist/contrib/pf. This is pointless and inconvenient. What you really want is to have the vendor source directly in vendor/pf/dist.

Here's how you flatten the pf tree:

% cd vendor/pf/dist/contrib/pf
% svn mv $(svn list) ../..
% cd ../..
% svn rm contrib
% svn propdel -R svn:mergeinfo .
% svn commit

The propdel bit is necessary because starting with 1.5, Subversion will automatically add svn:mergeinfo to any directory you copy or move. In this case, since you're not going to merge anything from the tree you deleted, they will just get in the way.

You may want to flatten the tags as well (3.4, 3.5 etc); the procedure is exactly the same, with s/dist/3.4/ or whatever. If you do this, put off the svn commit until the end.

Cleaning up

Look through your dist tree and perform any cleanup you deem necessary. One thing you may want to do is disable keyword expansion, as it makes no sense on unmodified vendor code. In some cases, it can even be harmful (OpenSSH includes two files that originated with FreeBSD and still contain the original version tags).

% svn propdel svn:keywords -R .
% svn commit

Bootstrapping merge history

If this is the first import you do after the switch to Subversion, you need to bootstrap svn:mergeinfo on the target directory (in the main tree) to the revision that corresponds to the last change you made to the vendor tree prior to importing new sources.

% cd head/contrib/pf
% svn merge --record-only $FSVN/vendor/pf/dist@180876 .
% svn commit

(on my system, $FSVN expands to svn+ssh://svn.freebsd.org/base)

Importing new sources

If you feel like it, you can repeat this step for every upstream release between the last one you imported and the one you're importing now. You will have to commit twice for every release - once for the import itself and once for the tag.

Preparing the vendor sources

Prepare a full, clean tree of the vendor sources. In CVS, we used to only import the parts we needed, but with Subversion, we can keep a full distribution in the vendor tree without bloating the main tree. We import everything, but merge only what we need.

Note that you will need to svn add any files that were added since the last vendor import, and svn rm any that were removed. To facilitate this, you should prepare sorted lists of the contents of the vendor tree and of the sources you are about to import.

% cd vendor/pf/dist
% svn list -R | grep -v '/$' | sort >../old
% cd ../pf-4.3
% find . -type f | cut -c 3- | sort >../new

With these two files, comm -23 ../old ../new will list removed files (files only in old) while comm -13 ../old ../new will list added files (files only in new).

Importing into the vendor tree

Now copy the new sources into dist, and svn add / svn rm as needed:

% cd vendor/pf/pf-4.3
% tar cf - . | tar xf - -C ../dist
% cd ../dist
% comm -23 ../old ../new | xargs svn rm
% comm -13 ../old ../new | xargs svn --parents add

If any directories were removed, you will have to svn rm them manually. Nothing will break if you don't, but the directories will remain in the tree.

Check properties on any new files. All text files should have svn:eol-style set to native. All binary files should have svn:mime-type set to application/octet-stream, unless there is a more appropriate media type. Executable files should have svn:executable set to *. There should be no other properties on any file in the tree.

You are now ready to commit, but you should first svn stat and svn diff to make sure everything is OK.

Tagging

Once you've committed the new vendor release, you should tag it for future reference. The best and quickest way is to do it directly in the repository:

% svn cp $FSVN/vendor/pf/dist $FSVN/vendor/pf/4.3

Once that's done, you can svn up your working copy of vendor/pf to get the new tag (or not; you probably won't need it anytime soon)

If you choose to create the tag in your working copy instead, don't forget to remove the resulting svn:mergeinfo:

% cd vendor/pf
% svn cp dist 4.3
% svn propdel svn:mergeinfo -R 4.3

Merging to head

% cd head/contrib/pf
% svn up
% svn merge --accept=postpone $FSVN/vendor/pf/dist .

(The --accept=postpone bit tells Subversion not to bother you with merge conflicts; you will take care of them manually)

Resolve any conflicts. This bit is no different than it was with CVS.

Make sure that any files that were added or removed in the vendor tree have been properly added or removed in the main tree.

Check diffs against the vendor branch:

% svn diff --no-diff-deleted --old=$FSVN/vendor/pf/dist --new=.

The --no-diff-deleted part tells Subversion not to bother you with files that are in the vendor tree but not in the main tree: things that you would previously have removed before the vendor import, like the vendor's makefiles and configure scripts and whatnot.

With CVS, once a file was off the vendor branch, you couldn't put it back. With Subversion, there is no concept of on or off the vendor branch. If a file that previously had local modifications no longer does, just remove any left-over cruft, such as FreeBSD version tags, so it no longer shows up in diffs against the vendor tree. Yet another thing that will make your life easier in the long run.

If any changes are required for the world to build with the new sources, make them now - and test, test and retest until you're satisfied that everything builds and runs correctly.

Commit

You are now ready to commit. Make sure you get everything in one go. Ideally, you would have done step 5 in a clean tree, in which case you can just svn commit from the top of that tree; that's the best way to avoid surprises. If you do it properly, the tree will move atomically from a consistent state with the old code to a consistent state with the new code - no pesky race conditions like we had with CVS.

Congratulations, you just completed your first vendor import.

Vendor imports from scratch with Subversion

Importing into vendor

First you need to prepare the directory in vendor:

% svn co --depth immediates $FSVN/vendor
% cd vendor
% svn mkdir byacc
% svn mkdir byacc/dist

Now you can import your sources into the dist directory. Once the files are in place do not forget to svn add all those new files.

You can now just svn commit and after that tag the version you import, to save time and bandwith you can to it directly remotely:

% svn cp -m "Tag byacc 20120115" $FSVN/vendor/byacc/dist $FSVN/vendor/byacc/byacc-20120115 

Merging to head

Since this is the first time it entering, you would need to copy it:

% svn cp -m "Import byacc to contrib $FSVN/vendor/byacc/dist $FSVN/head/contrib/byacc

You can continue to work "normally" with on your freshly imported sources