Introduction

There are a couple of FreeBSD git repositories that have been converted with various programs and various parameters. This page describes the "official" git-branches of the FreeBSD project that should be used as common repositories to base other work on.

The repositories are provided as a read-only mirror of the state of the Subversion src, doc and, ports repositories at http://svn.FreeBSD.org/base, http://svn.FreeBSD.org/doc, and http://svn.FreeBSD.org/ports respectively. Using git to commit changes to these repositories is not officially supported (but doable), instead it is supposed to serve as a collaboration point by using additional tools like Github, Gitorious, or Gerrit, etc.

Bear in mind that git will not replace Subversion for use in FreeBSD and the repository is only offered on a best-effort basis. The steps to redo the conversion are outlined below, so that people can move the process in-house should they see the need for it.

CAVEAT EMPTOR: It is possible (but highly unlikely) that the repositories might need to be re-done due to bugs in the converter. This would result in changed commit IDs for the impacted branches. As the actual tree objects for a certain SVN revision will remain identical, it will always be possible to rebase or merge your work on these new branches and not run into any merge conflicts. Advance notice will be provided if this proves necessary, along with advice on the required git commands.

The Repositories

The beta repository-mirrors are updated at least hourly, should they lag the svn repository by more than a day, please contact <uqs@FreeBSD.org> (monitoring still needs to be put into place, help wanted btw).

The obsolete, non-compatible repositories are:

Known Problems

Quick Start

$ git clone --config remote.origin.fetch='+refs/notes/*:refs/notes/*' git://github.com/freebsd/freebsd.git
$ cd freebsd && git pull   # For some reason, the notes will not be fetched during the clone operation ...
$ hack, hack, hack
$ git commit

Advanced, useful examples to show the diff between a subdir of a branch, or the commits that introduced these diffs. It is basically what svn mergeinfo --show-revs=eligible does. See git-log(1) for more info.

$ git diff origin/stable/9 master -- usr.sbin/acpi
$ git log --graph --format=oneline --right-only --cherry-pick --no-merges origin/stable/9...origin/master -- usr.sbin/acpi

See any other git howto or documentation if you're stuck on the basics. A useful reference for the issue with notes is http://git-scm.com/2010/08/25/notes.html.

Adding SVN Revision Notes to an Existing Clone

The git notes in the Github repositories contain useful information, and show up in git-log if present. git-clone without the --config ... line from above will default to not cloning refs/notes. You can configure the local git repo to fetch notes like so:

$ cd freebsd
$ git config --add remote.origin.fetch '+refs/notes/*:refs/notes/*'
$ git fetch   # It's many MB for src, will take some time

It will add lines to your git log output like this:

Notes:
    svn path=/head/; revision=277815

Collaboration

There is no endorsed way yet of how users should collaborate with developers on changes they have made. Please bear in mind that git is strictly ancillary at this point and not many committers are using it.

Having your patches/feature-branches published on github for a FreeBSD committer to comment on and pull from seems like the first sensible step. Sending a patch-bomb with git send-email might be an alternative to github or for developers that do not want to use git. When preparing patches consider using --no-prefix flag, so committer won't have to use -p1 flag with patch or --strip 1 with svn patch. Another pitfall is $FreeBSD$ keyword which is not expanded in git and will cause conflicts if present in the patch.

Receiving and integrating these patches can be done in various ways, see below.

Integration

Integrating the changes back into svn is not as trivial as using git-svn(1), as the current repository doesn't have the required meta-information. Instead patches can be applied to a subversion workspace just like any other regular patch.

A more sophisticated approach would be to share the svn and the git workspaces, which can be done by putting .svn in .gitignore and vice versa. This requires subversion 1.7 and you need to know how to recover from out-of-sync workspaces in both subversion and git, or be extra careful to avoid this situation.

Using git-svn (FreeBSD committers only)

The repositories on github also have a branch that is the result of a conversion using git-svn, i.e. the necessary conversion has already been made. To use this, you simply need to:

# for ports committers:
$ GIT_REPO=git://github.com/freebsd/freebsd-ports.git
$ GIT_SVN_URI=svn.freebsd.org/ports

# for source committers:
$ GIT_REPO=git://github.com/freebsd/freebsd.git
$ GIT_SVN_URI=svn.freebsd.org/base

# do the cloning
$ git clone $GIT_REPO freebsd
$ cd freebsd

# init git-svn to point to the subversion repo:
$ git svn init -Thead svn+ssh://$GIT_SVN_URI .

# Now you'll want to look in .git/config and make sure that 
# it has a config section that looks like this:
#[svn-remote "svn"]
#        url = svn+ssh://svn.freebsd.org/base   (or /ports, if cloning those)
#        fetch = head:refs/remotes/origin/trunk
#
# People w/o write access to SVN can use this instead:
#   $ git svn init -Thead --rewrite-root=svn+ssh://$GIT_SVN_URI svn://$GIT_SVN_URI .

# Now what we have to do is get 'master' to be tracking 'svn_head':

# Now make a git branch 'trunk' for git-svn to follow.  What we want to do it set it to point to the final commit in origin/svn_head.
$ git update-ref refs/remotes/origin/trunk `git show-ref origin/svn_head|cut -d" " -f1` # git-svn really needs this branch

# Sync with the FreeBSD svn repo and reconstruct the necessary metadata.  
# This should be pretty fast and output looks like a list of revisions that start above 100,000.
$ git svn fetch  

# Arrange to have 'master' reference 'trunk'
$ git checkout trunk; git branch -D master # delete master
$ git checkout -b master trunk # and have master hang off trunk

# Get pull requests from the repos:
$ git config --add remote.origin.fetch '+refs/pull/*:refs/remotes/origin/pull/*'
$ git fetch

# Finally do a final update from head.
$ git svn rebase

Now you can periodically sync to svn using git svn rebase and you can push small changes back to svn using git svn dcommit, but read on.

Things to keep in mind:

Merging pull requests from github using git-svn (FreeBSD committers only)

#
# Begin single port submit via pull request:
#

# Get most up to date version of everything from github (all the new pull requests)
$ git fetch

# Make sure you local copy is up to date
$ git svn rebase

# Now get the user's change onto the git-svn branch:
# ...an editor will pop up right after you run this command, see the next section for what to do.
$ git rebase -i --onto master origin/master origin/pull/21/head

# Inside the editor you'll want to "squash" the commits so that SVN only gets a single commit.
# If there is only a SINGLE "pick" line like:
#     "pick 251f162 Try to fix @sample usage"
# then just save+quit out of your editor.
#
# However if there are multiple "pick" lines you will want to flatten (squash) all of the changes into a single commit.
#
# Example, if you see this:
#
#   pick 251f162 Try to fix @sample usage.
#   pick 6c96c06 using VAR as both OPTION and PLIST_SUB doesn't seem to work.
#   pick c259e63 Use auto PLIST_SUB stuff.
#
# Change it to this:
#
#   pick 251f162 Try to fix @sample usage.
#   s 6c96c06 using VAR as both OPTION and PLIST_SUB doesn't seem to work.
#   s c259e63 Use auto PLIST_SUB stuff.
#
# Save+quit the editor, then another editor will pop up for you to combine the commit messages.
# Then cleanup the commit message and save+quit.
#

# Now you are ready for commit... almost, you must add freebsd properties.
#
# Now add FreeBSD required properties.
#
# XXX: at this point you'll need to add svn properties to the files, I have proposed modification to the
# freebsd port for git that supports this propset.  It is here: https://reviews.freebsd.org/D1154
#
# this only works if you have git port compiled with these patches:
# https://reviews.freebsd.org/D1154?download=true
# $ git svn propset REQUIRED_PROPNAME REQUIRED_PROPVAL files...
#
# Probably something like:
$ git svn propset svn:keywords "FreeBSD=%H" devel/py-tipper/Makefile
$ git svn propset fbsd:nokeywords yes devel/py-tipper/pkg-descr
$ git svn propset fbsd:nokeywords yes devel/py-tipper/distinfo 

# make sure the commits look OK:
$ git svn dcommit -n

# Show the actual diff.
$ git diff `git svn dcommit -n | grep '^diff-tree'| cut -f 2,3 -d" "`

# actually commit it...
$ git svn dcommit

Experimental git-svn bootstrap

There is an experimental git-svn bootstrap for quickly getting a complete git-svn enabled repository of your own.

Visit https://github.com/idaemons/freebsd and follow the instructions described in the README file.

FAQ

Everything done is so that everybody can verify the integrity of the conversion using the tools mentioned in this article. Hence the focus on repeat-ability of the process.

Verification

Content

Verifying that the git export is identical to the SVN tree (in content), can be done like this:

$ svn export --ignore-keywords svn://svn.freebsd.org/base/head freebsd.svn
...
Exported revision 243246.
$ git clone --config remote.origin.fetch='+refs/notes/*:refs/notes/*' git://github.com/freebsd/freebsd.git freebsd.git
$ cd freebsd.git && git pull
$ git log
<We see that revision 243246 is git commit f9ebae3>
$ git checkout f9ebae3
$ cd ../freebsd.svn
$ git --git-dir ../freebsd.git/.git diff
<There should be no output, i.e. no diff>

History

To verify the history and integrity of the conversion, you are advised to run such a conversion yourself and compare (and report) any discrepancies that you find.

To redo this at a later stage, you simply need to

Further Reading

It is *really* recommended, that you read Git for computer scientists and skim GitTalkDevSummit, although it's really outdated.

It really helps to understand the data structure of a git commit, because then you know how merging/rebasing works and can fix snafus easily.

A good visual git reference can be found at A Visual Git Reference.

This is a decent book on getting started with Git ($16 USD for ebook) Pragmatic Guide to Git

Implementation Details

The software used for the conversion is a slightly modified fork of svn2git, as used by the KDE project. It can be found at https://gitorious.org/~uqs/svn2git/uqs-svn2git

It requires a rules file to map svn trees and/or revisions to git branches or tags. The current rules don't make use of tags but simply store them as branches.

See the project branch at http://svn.freebsd.org/base/user/uqs/git_conv/ for extra patches, rules and the scripts used to do these conversions.

GitWorkflow (last edited 2015-01-28 14:44:49 by ConradMeyer)