Package Building for the FreeBSD.org Cluster

The Big Picture

The basic idea is that we need to make certain that when port upgrades happen on cluster machines, we need to be able to somewhat reasonably guarantee that the ports being upgraded are compatible with the existing ports, and nothing is missed. poudriere handles this quite nicely and reliably.

Poudriere

Local Setup

The cluster package build repository is separate from the default clusteradm subversion repository. It is located at: svn+ssh://pb00.FreeBSD.org/var/svn/pkgbuild/keg

This repository contains the /usr/local/etc/poudriere.d/ directory on pb00.FreeBSD.org.

This directory contains all the magic needed for FreeBSD.org cluster machine package builds as noted in the Local Configuration and Local Scripts sections.

As we already know, pre-built packages were never one-size-fits-all. What I have tended to do when setting up such package build systems is to do package builds based on either a per-machine basis, or based on a set of machine roles. Both of these have their pros and cons:

My preference has always been per-machine package builds.

Local Configurations

/usr/local/etc/poudriere.d/ is set up according to what poudriere expects to find, and where it will look. These files are all under revision control, for the inevitable case of needing to look back to see what changed with a port build options, or needing to revert an entire set of changed options.

Each build has its own:

These are linked to a particular build by prefixing the jail name and target machine name, for example:

In this case, 10-amd64 is the name of the jail used for the port build and pb00 is the FreeBSD.org cluster machine that will be the consumer of the resulting packages.

The idea for jail names is effectively ${BRANCH_VERSION}-${ARCH}

The idea for the "sets" poudriere will build is effectively the machine hostname, less "freebsd.org". In cases where there are multiple machines per site, or a machine hostname is site specific, the "." is replaced with "-", for example: cook-isc

Local Scripts

There are two scripts right now to help automate the maintenance of package builds:

There are a few configuration files that are used by the scripts:

There is also a pkgbootstrap.sh script that is intended to fetch the sample pkg.conf file and update as necessary, as well as the public pkg.cert file from pbdist.FreeBSD.org, and put them into the correct locations. After fetching the files, the SHA256 is calculated and verified against known values.

Package Distribution

Local Setup

pb00.FreeBSD.org has a single jail for web services, pbdist.FreeBSD.org. Its web root directory is a read-only nullfs(5) mount from pb00.FreeBSD.org, containing an empty index.html file, a sample pkg.conf file, and the public certificate file (pkg.cert) for the SSL key used to sign the packages.

Note: If ever changing the pkg.cert (or the signing key) or pkg.conf, the SHA256 sums must be updated in pkgbootstrap.sh.

Package Repository Creation

poudriere will automatically create the repo.txz using the signing key after a package build completes. In general, there should be nothing to do here, but if manually signing a package build is needed, this can be done as:

 # pkg repo /poudrieredata/data/packages/${JAIL}-cluster-${TARGET_HOST}/ /root/ssl/pkg.key

Pkgng

Cluster Server Bootstrapping

A "bootstrapping a system for cluster pkgng builds" example:

 root@pb00:~ # fetch http://pbdist.freebsd.org/pkgbootstrap.sh
 pkgbootstrap.sh                               100% of 1465  B 1346 kBps 00m00s
 root@pb00:~ # sh ./pkgbootstrap.sh 
 /tmp/pkg.cert                                 100% of 1491  B 4282 kBps 00m00s
 /tmp/pkg.conf                                 100% of  479  B 1480 kBps 00m00s
 SHA256 for pkg.conf and pkg.cert match.
 Creating /etc/ssl/ directory for pkg.cert...
 Installing the public certificate file...
 Done.
 Running FreeBSD:10 amd64
 This hostname is: pb00
 Installing pkg.conf...
 Done.
 Cleaning up.
 Done.
 root@pb00:~ # grep ^PACKAGESITE /usr/local/etc/pkg.conf
 PACKAGESITE         : https://pbdist.FreeBSD.org/10-amd64-cluster-pb00
 root@pb00:~ # pkg update
 Updating repository catalogue
 digests.txz
 100% 2280     2.2KB/s   2.2KB/s   00:00    
 packagesite.txz
 100% 7580     7.4KB/s   7.4KB/s   00:00    
 Incremental update completed, 0 packages processed:
 0 packages updated, 0 removed and 21 added.

Teams/clusteradm/Packages (last edited 2013-08-28T18:08:12+0000 by GlenBarber)