Note: The first part of this page aims to explain the conventions behind Subversion repository organisation, and use various simplified repository structures to explain this. The recommended repository layout can be found at the end of this page.

In CVS, branches and tags are first class entities -- CVS contains several commands for manipulating these. Subversion does not. Instead, Subversion's support for cheap file/directory copies within the repository is used to support the notions of tagging and branching.

A number of conventions have developed to support this. These conventions are not enforced by any of the base Subversion tools, but they can be enforced by pre-commit hooks in the repository (if it is felt that these decisions need to be programmatically enforced).

$REPO and commits

Subversion repositories are referred to with URLs. In the examples that follow, $REPO is assumed to refer to the root URL for the given repository.

Any Subversion command that specifies the full URL operates against the repository, which often means a simultaneous commit. Commands that just specify local files or directories in the working copy change the working copy, but they must be explicitly committed.

For example, this example copies one file to another in the repository -- the data in the file does not need to travel to the client, so the operation is quite quick, as it happens entirely on the repository host.

  % svn copy $REPO/file1 $REPO/file2

The user is then prompted for the commit message for this commit (or it can be specified with a -m option on the command line) and the commit completes.

Alternatively, the user could do this:

  % svn checkout $REPO/file1
  % svn copy file1 file2
  % svn commit file2

As you can see, that process takes more commands, and involves transferring the file data to the client.

trunk/, tags/, and branches/

This is the most pervasive convention. Code that corresponds to CVS HEAD is placed in the trunk/ directory. So in the ordinary course of operation, if a user

When a tag needs to be made a copy is made of trunk/ to an appropriately named directory under tags/.

  % svn copy $REPO/trunk $REPO/tags/name-of-tag

There is nothing in Subversion that directly prevents someone from checking out the tags/name-of-tag/ directory, and then committing changes. This would appear to violate the spirit of tags, in that they should be immutable. However, CVS supports this operation too (c.f., "sliding a tag"), and an appropriately written pre-commit-hook script that interrogate the commit, and refuse to allow it if the commit includes a directory under tags/, perhaps with some additional checks (e.g., members of the Release Engineering group would be allowed to create/change files under tags/, but no one else).

It follows, therefore, that a branch is just a tag to which people are allowed to commit. And that is exactly how Subversion operates. To create a branch the command is:

  % svn copy $REPO/trunk $REPO/branches/name-of-branch

The user can then check files out from the branches/name-of-branch/ directory, and commit as normal.

Since these are just directories in the repository there is nothing inherent that prevents someone from, for example, creating a branch from a tag. For example,

  % svn copy $REPO/tags/name-of-tag $REPO/branches/name-of-branch

Multi-project repositories

There is nothing that enforces the idea that a repository can contain a single project. In fact, it's commonplace not to. A typical multi-project repository layout might look like this:

  $REPO/project1/trunk/...
  $REPO/project1/tags/...
  $REPO/project1/branches/...
  $REPO/project2/trunk/...
  $REPO/project2/tags/...
  $REPO/project2/branches/...
  $REPO/project3/trunk/...
  $REPO/project3/tags/...
  $REPO/project3/branches/...

That layout is common if branching and tagging operations are normally applied to one project at a time, or if the projects are somewhat independent. For an example of this see the Apache Subversion repository -- http://svn.apache.org/viewvc/. This contains multiple independent projects that branch and tag at different times. If you drill down to the individual projects you'll see per-project trunk/, tags/, and branches/ directories.

An alternative layout, more appropriate for multiple projects that are tightly coupled and that normally branch and tag at the same time would be:

 $REPO/trunk/project1/...
  $REPO/trunk/project2/...
  $REPO/trunk/project3/...
  $REPO/tags/project1/...
  $REPO/tags/project2/...
  $REPO/tags/project3/...
  $REPO/branches/project1/...
  $REPO/branches/project2/...
  $REPO/branches/project3/...

A good example of this is the KDE Subversion repository at http://websvn.kde.org/. As you can see, there are top level trunk/, tags/, and branches/ directories, and every KDE sub-project sits under there. This allows the KDE team to produce a consistent snapshot of the entire project when they plan a new release.

Vendor code

A separate set of conventions exist for managing vendor code that will form part of the main project.

Typically, unmodified vendor code is imported in to sub-directories on a special branch. Instead of being a sub-directory of branches/ this is commonly called vendor/. The very first time the code is imported it is copied to the project's trunk/. After subsequent imports the differences between the previous version of the vendor code are merged in to the code on trunk/.

A worked example may make this clearer.

Consider a project that includes, as part of it's release, the code to Sendmail.

The project exists in $REPO/trunk.

The source code to Sendmail 8.13.7 is imported in to $REPO/vendor/sendmail/current. This directory is then copied to $REPO/vendor/sendmail/8.13.7.

The Sendmail source code is copied to an appropriate subdirectory of the project's trunk/. For example;

  % svn copy $REPO/vendor/sendmail/8.13.7 $REPO/trunk/contrib/sendmail

The code on $REPO/trunk/contrib/sendmail can then be modified as necessary to suit the needs of the project.

Now suppose that Sendmail 8.13.8 is released. The source code for this version is then imported in to the repository, and placed in $REPO/vendor/sendmail/current. As part of this process any differences between 8.13.7 and 8.13.8 are merged. The current/ directory is then copied to 8.13.8/. At this point the vendor/sendmail tree contains three subdirectories, for the two different versions of Sendmail, and current/.

The svn merge command can then be used to merge the changes between the vendor 8.13.7 and 8.13.8 releases on to the project's trunk.

  % svn merge $REPO/vendor/8.13.7 $REPO/vendor/8.13.8 trunk/contrib/sendmail

This takes the changes between 8.13.7 and 8.13.8 and merges them on to trunk/contrib/sendmail. The committer can then review these changes, resolve any conflicts between the imported code and any local modifications, and then commit the merged code.

That probably sounds more complicated than it really is. There's a more detailed write up at http://jc.ngo.org.uk/trac-bin/trac.cgi/wiki/ImportingIntoSubversion -- note that that assumes certain conventions that I'm following in my personal repo which we probably wouldn't follow in the FreeBSD Subversion repo.

Proposed layout

So, after all that, here's a proposal for a layout. This does not need to be set in stone, and I'm sure we can bikeshed the directory names to death. Still, it's a start.

{{{ / freebsd /

}}}

Lets walk through the rationale for this.

The core FreeBSD distribution is kept under a single freebsd/ root. This makes it easy for third parties to easily mirror the bits of FreeBSD that they want. For example, mirroring freebsd/src would be equivalent to using CVSup to fetch src-all. Checking out from freebsd/src/trunk would be equivalent to checking out CVS HEAD.

Mirroring freebsd/ would be like fetching src-all, ports-all, doc-all, and www using CVSup.

Now, at the moment we don't branch trees like ports/. That's (I think) more to do with the time and effort this takes to manage branches under CVS than because we couldn't benefit from branched ports. However, even if we decide not to branch ports (or the other trees) right now, we should probably still use the trunk/ subdirectory. It keeps things like directory depth consistent, which makes it much easier to automate things, or describe the repo in general terms instead of having to include special cases.

This scheme allows the four component parts of the project to branch and tag at different times. This is common during the release process, where, for example, ports/ will have a change freeze that has different dates to the src/ code freeze.

So, for the moment, freebsd/src/trunk/ corresponds to CVS HEAD. What about releases, 6-stable, and 5-stable?

Major releases

A major release (and focusing just on the src/ tree for the moment) is one that is cut from trunk/ (aka CVS HEAD). Here's how it would work for FreeBSD 8.0.

  # First, snapshot the src tree on to a branch from which the release will be cut.

  % svn copy $REPO/freebsd/src/trunk $REPO/freebsd/src/branches/8.x

  # Now prepare 1..n release candidates.  In CVS we do not tag release candidates.
  # With Subversion's cheap tagging it becomes possible to do that.  This step, is,
  # of course, optional.

  % svn copy $REPO/freebsd/src/branches/8.x $REPO/freebsd/src/tags/8.0rc1

  # A similar process takes place on the ports/ and doc/ trees to tag their own
  # files for 8.0rc1.

  # The release engineer(s) prepare a release from (src,doc,ports)/tags/8.0rc1
  # If the release candidate has problems, changes are made to src/branches/8.x
  # to rectify them, and copies of this directory are made to tags/8.0rcN as
  # necessary.

  # Eventually (say, after 3 attempts, so we're at 8.0rc3) the release is deemed
  # ready to go out the door.  So the RE copies the rc3 tag to a release tag.

  % svn copy $REPO/freebsd/src/tags/8.0rc3 $REPO/freebsd/src/tags/8.0-release

  # and carries out similar commands for the doc/ and ports/ trees

  # The final step is to create a branch just for this release -- this it the branch
  # to which security errata (or similar) commits would go

  % svn copy $REPO/freebsd/src/tags/8.0rc3 $REPO/freebsd/branches/8.0

MFCs

Once release has been cut the tree looks like this (again, focusing just on src/ for the time being.

  freebsd / src / trunk
                  branches / 8.x
                           / 8.0
                  tags     / 8.0-release
                           / 8.0rc1
                           / 8.0rc2
                           / 8.0rc3

Those rc1, rc2, and rc3 tags can be removed -- but that's not necessary, and since it's deleting project history it might not be desirable. They take up minimal extra space in the repository (although they do slightly clutter up the output of svn ls $REPO/freebsd/src/tags, which may or may not be a concern).

In this structure, what we would currently call 8-stable corresponds to the branches/8.x/ directory. It's entirely possible that we would want to call this directory 8-stable/ instead of 8.x/. I'm not married to the name -- however, given the occasional outburst from the userbase on the -stable mailing list when -stable turns out not to be "stable", perhaps we should consider taking the opportunity to migrate away from the name.

Anyway, I digress. As is the nature of things, work continues on src/trunk/. Work that will go in to 8.1 is merged from src/trunk/ in to src/branches/8.x.

Minor releases

At some point it becomes time to cut a minor release. This process is almost identical to that for a major release. The difference is the source of the copy for the release candidate. This is just like a major release, only without the initial copy from trunk. All the release candidates are all taken from the 8.x branch.

    % svn copy $REPO/freebsd/src/branches/8.x $REPO/freebsd/src/tags/8.0rc1

These go through the same release candidate tagging cycle as before.

Security releases

Should it become necessary to cut a security release the process is very similar. Recall that the code for each release is also tagged.

So suppose that 8.1 has been released, and it only took one release candidate to do so. That portion of the tree is going to look like this:

  freebsd / src / trunk
                  branches / 8.x
                           / 8.0
                           / 8.1
                  tags     / 8.0-release
                           / 8.0rc1
                           / 8.0rc2
                           / 8.0rc3
                           / 8.1-release
                           / 8.1rc1

The security officer commits their changes to branches/8.1. This means that anyone who is mirroring branches/8.1 (to track security and other errata changes to otherwise frozen code) automatically picks up the fix.

The SO can then release 8.1.1, following the same process as before. The only difference is that the source of the intial copy is branches/8.1, instead of trunk/ (for major releases) or branches/8.x, for minor releases.

Recap

The structure so far replicates (as far as I can see) the same features that we get from CVS.

CVS Tag/branch name

Subversion Path

HEAD

trunk/

RELENG_8_0_0_RELEASE

tags/8.0

RELENG_8

branches/8.x

RELENG_8_1

branches/8.1

There's no need for the *_BP tags, as they're not necessary under Subversion's model. A branch (i.e., a copy) carries with it the information about the path and revision from which it was copied, so this information does not need to be recorded separately.

Refinements

Some possible refinements that we might want to make.

Vendor

As already discussed, the convention is to keep unmodified vendor code in its own space. It's only the modified code that makes it in to the freebsd/ tree.

Projects

The top level projects/ directory is reserved for long-lived projects. The sort of thing that would currently be kept in the Perforce repository. I imagine that each project would have a top-level directory under here, with at least a trunk/ directory underneath. If the project is particularly long lived, and perhaps cuts its own releases, then tags/ and branches/ subdirectories would also make sense.

Under this model we could (if we wanted) move projects like PicoBSD under the main FreeBSD repo.

I have no opinion on how space under projects/ is handed out to requestors.

User

The top level user/ tree is where per-user directories can go. Given that we have something like 300+ comitters at the moment I've shown the directory space subdivided by first character of the username. That may or may not be warranted, I don't especially care.

This is the space for users to conduct their own experiments before bringing them in to the main tree. For example, a committer with 8 ports to their name might want to use this space to do development on their ports. Once the ports are ready the changes can be merged in to the main tree.

One repo? Several repos? Impact on mirrors

A decision to be made is whether we want one big repo that has everything, or several smaller repos.

One repo pros:

One repo cons:

Invert those pros and cons for the multi-repo case.

SVN_Repo_Layout (last edited 2008-06-17 21:37:17 by localhost)