• VladimirKrstulja
• Guides
• Poudriere

Poudriere: Getting Started - Tutorial

To put it simply, Poudriere is a tool that builds pkg repositories. It's the recommended tool for building ports, and it's used by the FreeBSD project to build the official pkg repositories.

Because of its flexibility and the fact it doesn't build a port and install it right away, but builds a repository first, it's perceived as complex and heavy. The reality is, though, it's a simple collection of shell scripts that automate port building in isolated environments using jails, and is applicable for simple systems as well as for complex deployments where a variety of arch, ABI or options sets are required.

Getting started: configuration

While Poudriere almost works out of the box (plus all the setup required for jails, ports trees and package lists, done in this tutorial), it is very much configurable, and not just for some advanced use. The main config file is located at /usr/local/etc/poudriere.conf, and the repo-specific configuration (done by Poudriere automatically for options etc) is located at /usr/local/etc/poudriere.d/.

It almost works out of the box, because it requires at least one option be explicitly set in poudriere.conf, and that is the FREEBSD_HOST config that for some reason does not default to the recommended default (see comments in the config file). All other options needn't be modified to just get started using Poudriere.

The config file is extensively self-documenting, so see the file for more information, but we'll mention a few worth taking a special look at, before we begin:

• USE_TMPFS -- Define where to use tmpfs in the build process, default being yes, which means Poudriere will build each port (having the wrkdir) in a tmpfs. For memory-constrained systems that might not be preferable.

• CCACHE_DIR -- Define the path to (and thus enable) CCACHE files. Default being off, it is very useful to enable CCACHE (create a directory for it, and specify the path to CCACHE_DIR var). The reason for this is that Poudriere will rebuild any reverse dependency of any port that changed the version string. This is important for isolated (re-)builds guaranteeing some build integrity for a dependency graph of any port. The side-effect is frequent rebuilds of ports that have no change in source code. CCACHE thus helps a lot with this, as it allows re-use of previously compiled objects to be re-used without compilation, and drastically improves compilation times.

• PARALLEL_JOBS -- define how many ports to build in parallel, default being the number of detected "cpus", via hw.ncpu sysctl.

• ALLOW_MAKE_JOBS -- define makefile parallelism, per port build process, default off.

The overall default is such that ports are built in parallel, but with no parallelism within each build process. Since not all ports can be built in parallel, especially with dependencies where a port must wait until all build-deps are built first, the production configuration depends on many things. The author's "sweet spot" is to split those in two, so eg. for a 4-core system, build 2 ports in parallel, allow make parallelism, and set MAKE_JOBS_NUMBER in make.conf.

Per-repository configuration and make.conf

A bit advanced topic at this stage in the tutorial, but it's important to mention that the main poudriere configuration can be made repository-specific. As the poudriere(8) manpage shows, Poudriere is using configuration files (and make.conf) in a specific order of patterns tried, so repository-specific configurations are possible with repository-specific naming patterns for those files under /usr/local/etc/poudriere.d/. The manpage explains how.

Basic workflow

The most simple use case for Poudriere -- to build a package repository for use on a single machine -- consists of the following simple steps.

1. Create a builder jail

Poudriere builds ports in isolated environment using jails. It manages jails on its own, so no previous jail configuration is needed for it. It is also possible to run Poudriere in a jail itself.

Pretty much all the steps below require the user to specify which builder jail is used. That is because Poudriere builds repositories, but differentiates them according to:

• Which builder jail is used
• Which ports tree is used
• Which user-defined set is used

Poudriere uses this information to build a directory where the repository is stored, directory the name of which includes the above information. So for example, if we have:

• Builder jail called "111x64" (arbitrary name for a 11.1 64-bit base jail)
• Ports tree called "default"

Poudriere will build the directory called 111x64-default. If a set named "py35" is used, then the directory will be 111x64-default-py35. This allows complex combinations of repositories, but for a simple use case, we need just one jail, one tree and probably no sets to get started.

Poudriere thus requires a builder jail, so we define one, for 11.1 64-bit base, and call it "111x64" (the name is arbitrary, but must be consistently used to identify that jail, for options, bulk and other commands):

poudriere jail -c -j 111x64 -v 11.1-RELEASE

The version is important because that's the FreeBSD base, with libs against which ports will be built (so it defines libc, base openssl and other base facilities that ports might need).

2. Create the ports tree

The second requirement is of course the ports tree. Poudriere manages ports tree on its own, so it's possible to have several, using various methods to track them: svn, ftp or, very important, a local directory which is useful if you already have a /usr/ports populated and don't want it managed again elsewhere by Poudriere. Let's assume we want Poudriere to manage the ports tree (so we don't have a system-wide /usr/ports populated), and let's give it a name "default":

poudriere ports -c -p default

But wait, what if we do want to use our existing /usr/ports tree? Say, we're tracking it with svn and adding our own patches or modifications. Easy, let's then instruct Poudriere to "create" a tree using a local directory. Let's call it "local" (the names are really arbitrary, but must be consistently used to identify that tree, for options, bulk and other commands):

poudriere ports -c p local -F -M /usr/ports -f none

However, in that case poudriere will not manage the tree with the ports command, and a separate command is needed to update it, like running portsnap or svn up, depending on how the tree is managed by the user.

3. Create the list of ports wanted in the repo

Poudriere can build individual ports, by giving the port origin to the bulk command, but its more useful option is to define a persistent list of packages we want built, and have Poudriere manage dependencies, updates and upgrades automatically. Let's say we want Firefox in our repo. All that's needed is a file containing just the Firefox origin:

# /usr/local/etc/poudriere.d/packages-default

www/firefox

The file location, and name, is completely arbitrary, but it would be wise to follow the naming patterns Poudriere uses for make.conf, options and other things (see below), as they're important in the order at which Poudriere selects them to build a repo. See the poudriere(8) manpage, section CUSTOMIZATION on how Poudriere selects correct config files for jail-ports_tree-set combinations.

The list of packages is the only file that Poudriere does not automatically source based on a pattern, so its location and name really is at the choice of user.

4. Optionally change default port options

If we don't change any ports options, Poudriere will automatically use default options, in that it will use defaults to create a specific options directory, the poudriere-managed options directory under /usr/local/etc/poudriere.d/, which is also subject to the above-mentioned naming patterns. So if we do nothing, defaults will be used, but then that's probably not why we're building from ports.

Ideally, we want Poudriere to go through all the options recursively, and give us an options dialog for any port that has new or changed options. Poudriere has the options command for this, which (because it can differentiate options per jail, ports tree or set), requires us to specify jail and ports tree, and of course the list of ports we're configuring. So, simply:

poudriere options -j 111x64 -p default -f /usr/local/etc/poudriere.d/packages-default

See the manpage on how to do more advanced things with options including resetting or forcing a dialog just for one port, etc... If sets are used, options will require a -z flag too.

5. Configure pkg to use the repo built by Poudriere locally

As mentioned before, Poudriere really builds a package repository, so the actual installation, removal or update/upgrade of packages built by Poudriere is done with pkg(8).

To access that repository, we have to configure pkg to use it. The pkg.conf(5) manpage tells us we can either modify the system configs under /etc/pkg/ or have a local config (preferable!) under /usr/local/etc/pkg/repos/. Now, pkg requires a (arbitrarily named) .conf files at those paths. While the file names are not important, the repository names configured in them are important. This allows us to set a local repository configuration and disable the default official repo configured in /etc/pkg/FreeBSD.conf:

# /usr/local/etc/pkg/repos/Local.conf

Poudriere: {
  url: "file:///usr/local/poudriere/data/packages/111x64-default"
}

FreeBSD: {
  enabled: no
}

The path given above is constructed from Poudriere's configured location for repositories, and the repo we're configuring in this tutorial: based on "111x64" named jail and "default" named ports tree. Poudriere stores repositories under ${POUDRIERE_DATA}/packages directory, where POUDRIERE_DATA is configured in /usr/local/etc/poudriere.conf, and the default is /usr/local/poudriere.

6. Build the repository

With all the previous steps done to initialize our Poudriere builder, the actual building is as simple as:

poudriere bulk -j 111x64 -p default -f /usr/local/etc/poudriere.d/packages-default

As mentioned before for other commands, the bulk command requires us to specify which builder jail is used, which ports tree, which list of packages, at minimum. Poudriere will build all the ports we specified in the file given by -f, as well as all the dependencies required to build and run Firefox.

7. Use the repository

Once configured, we really just use pkg to install, update, upgrade or remove the packages. Our configured and locally built repository is essentially not different from the official pkg repositories, except that we've used our own list of ports we want (the official repos build all 30k+ ports in the tree), and we used our own options (the official repos use default options). So to continue the above example with Firefox, we just:

pkg install firefox

Production use workflow

So far we've covered the steps required to set up and build a pkg repository. However, what next? How do we keep our packages updated so we can simply update our systems with pkg upgrade ? The (production) workflow is then as simple as: updating the ports tree, handling any changed/new options and simply re-building the repo for any changes. By that we mean building packages that have updated including any dependencies:

poudriere ports -u
poudriere options -j 111x64 -p default -f /usr/local/etc/poudriere.d/packages-default
poudriere bulk -j 111x64 -p default -f /usr/local/etc/poudriere.d/packages-default

These three commands are all that's needed. Now, it must be noted that the options step might require user interaction. Poudriere will go through options of each port and if there's anything new, will present the dialog for us. It will also present a dialog for any port that became a new dependency, or any port that doesn't have an entry in the poudriere-managed options directory.

One extra step may be needed, and that's the package cleanup. Any package previously built in the repository that is no longer needed -- because it wasn't specified in the -f package list, and is suddenly no longer a dependency -- will remain in the repo unless we explicitly clean it with pkgclean command, so the typical workflow might require that additional step:

poudriere pkgclean -j 111x64 -p default -f /usr/local/etc/poudriere.d/packages-default

It is very important to give the -f list of packages that we do want installed, so it can determine which package is no longer needed in that dependency graph. The -j and -p options are needed because, like we said, Poudriere defines a repository based on jail + ports tree used. It's also worth noting that, like the options step, this requires user interaction as Poudriere will collate the list of packages to remove, but ask the user for confirmation, unless -y option is given to pkgclean.

Additionally, if the ports tree repo was configured not to be managed by poudriere (like in step #2 above when the tree was defined to be the already existing /usr/ports), it must be then updated separately (in place of poudriere ports -u given in the above example).

Updating jails

Running poudriere bulk will always automatically update and upgrade any package in the list given by -f (as well as any dependencies). But, what about builder jails? The builder jails have to be updated separately, just like a base system needs to be updated running freebsd-update fetch install. With Poudriere, updating a jail named "111x64" is as simple as:

poudriere jail -u -j 111x64

However, unlike with, say, freebsd-update -r 12.0-RELEASE upgrade, Poudriere does not upgrade jails ot new major version, and it would be against the idea of it, and designed workflow if it did. To "upgrade", one merely has to create a jail for the new major base version, and use the new jail name for -j in all poudriere commands that need to specify the (new) jail. This means, of course, that poudriere will create a new repository, named after the new jail.

Note: Updating jails (to a new patch level) will trigger rebuild of all packages defined in the repo (in the package list given by -f), the next time bulk is used for the -j jail that has been updated.

Sets

A more advanced feature of Poudriere are sets, or the -z flag given to most poudriere commands. Sets allow building different repositories based on some variation. Since poudriere will already build a repository based on jail and ports tree, namely -j and -p flags, it is further possible to vary things for the same jail and/or for the same ports tree.

As an example, the user might want to build two repositories, one with Python 3.x in DEFAULT_VERSIONS, and one with Python 2.7 in DEFAULT_VERSIONS. Let's ignore the fact that right now 2.7 is default and no specific configuration is needed for it, and so let's create two make.conf files, for those two sets, both of which will use the same builder jail -j 111x64 and same default port tree -p default. Given the filename patterns tried first, we can thus define two make.conf files:

# /usr/local/etc/poudriere.d/py27-make.conf
DEFAULT_VERSIONS= python=2.7

# /usr/local/etc/poudriere.d/py36-make.conf
DEFAULT_VERSIONS= python=3.6

Then in all the options, bulk and pkgclean commands we supply sets "py27" and "py36" in order to build two repositories. For example bulk:

poudriere bulk -j 111x64 -p default -f /usr/local/etc/poudriere.d/packages-default -z py27
poudriere bulk -j 111x64 -p default -f /usr/local/etc/poudriere.d/packages-default -z py36

With the set name of "py36", poudriere will source /usr/local/etc/poudriere.d/py36-make.conf (as it tries <setname>-make.conf), and similarly for the set name of "py27". The created pkg repositories will thus be at:

• ${POUDRIERE_DATA}/packages/111x64-default-py27/

• ${POUDRIERE_DATA}/packages/111x64-default-py36/

Which is at ${POUDRIERE_DATA}/packages/<jailname>-<ports_tree>-<setname>.

Of course, we've used here one builder jail and one ports tree, but all these could be combined to create a really complex and flexible array of individual package repositories. Each is at a path that can then be given to pkg.conf in various (optional) priorities.

Note: With set-specific make.conf as given in this example, or in other words with multiple make.conf existing on the system, Poudriere will source each in the order explained in the manpage. That means it will use multiple at the same time. It will first try /usr/local/etc/poudriere.d/make.conf and source it if found. It will then try /usr/local/etc/poudriere.d/<setname>-make.conf, thatis for example /usr/local/etc/poudriere.d/py27-make.conf and source it too, additionally. Which means that individual repository-specific make.conf files should only override whatever is given in the make.conf files sourced first.

CategoryHowTo CategoryPorts