Developing Ports - Tips, Tricks and Gotchas
This document describes working with the ports tree in a non-standard location, primarily for the purpose of developing or maintaining them. The process can be broken in several steps, each explained more thoroughly below.
It is assumed that the Porter's Handbook has been read or at least perused, so you know where to look up information with regards to developing/maintaining ports.
It is also assumed that you know what Poudriere is, because it is a crucial tool in developing/maintaining ports. The poudriere(8) manpage has a lot of info about using Poudriere.
TODO: Add a link to the Poudriere Guide.
1. Preparing the local tree via SVN
First, we need to fetch the ports tree for local modification. We will be modifying the tree and ports, and from that tree we'll build and test build ports. Finally we will create patches against the SVN tree with all the changes we wanted done.
We assume an unprivileged user and a path in its homedir is used to work with ports. Unless noted otherwise, all the commands in this document are given under the unprivileged user portsvn (ie. not as root). For example, lets set it up like this, as root:
adduser portsvn zfs create -o mountpoint=/home/portsvn/ports -o compression=on zroot/ports
The user name and ports dir location, and the fact we're using ZFS, is just an example which will be used throughout this document. Now, as user portsvn, we check out the ports tree via SVN. We can either use svnlite in the base, or you have to install Subversion from ports.
svnlite checkout https://svn.freebsd.org/ports/head /home/portsvn/ports
The FreeBSD handbook has more info on Using Subversion.
In addition to checking out the actual tree, some environment vars need to be set in order to work with the local tree and with an unprivileged user. You can add them to the rc file of your choice shell. Here we set them on the fly for tcsh:
mkdir -p /home/portsvn/var/db/ports setenv PORTSDIR "/home/portsvn/ports" setenv PORT_DBDIR "/home/portsvn/var/db/ports"
This is needed because the framework otherwise assumes the tree is placed under /usr/ports and is run by root with write privilege into /var/db/ports. So with these env vars, we override that for all ports commands and Makefile tasks run under this local tree. This also means you can have this tree completely separate from the default /usr/ports tree you use to build software for the computer you're working on.
In addition to checking out the tree from SVN for the first time, we occasionally need to update it, fetching modifications done by other contributors. This is important as it would also fetch changes that someone else has done to the ports we want to modify.
# Run in /home/portsvn/ports svn update
2. Preparing a Poudriere jail to work with your local ports tree
Next we prepare the Poudriere environment for testing ports from our local tree. Depending on how and where you've set up Poudriere, this might require running as root.
poudriere jail -c -j 110x64 poudriere ports -c -F -M /home/portsvn/ports -f none -p portsvn
Now, the above is just an example and assumes too much, namely the jail name and ports tree name. However, what is not just example is how we set up the portsvn tree, and that is from a local directory, so Poudriere would not go and fetch the tree into its own file hierarchy, and this is done with the poudriere ports ... command given above.
Note that the ports tree name given with -p is arbitrary and in this example only happens to coincide with the username. There is no dependency between port tree name given to Poudriere with -p and the path to that tree.
Optimally, we will have created a jail for each supported FreeBSD version to test the changes for all of them. We can also vary options with sets (-z for poudriere actions). To test against multiple FreeBSD versions we only need separate jails. The ports tree can be just one, named portsvn above.
3. Working with ports
Depending what we need to do, here are a few notable tasks:
Run make extract in the ports dir (eg. /home/portsvn/ports/lang/python35) to fetch the tarball and extract it locally. This is useful when we need to apply or create patches. If we don't need to modify or inspect the source files, we don't need to extract the tarball. The tarball is extracted into ./work/$DISTNAME in the port's dir.
If we need to modify source files in order to create port patches, first we copy the file you modify into <filename>.orig
Don't forget to first apply already existing patches with make patch
Run make makepatch to create patches from such modified files. This command will remove or overwrite the existing patches from the files/ directory, so running make patch first is required
Run make makesum to fetch new distfiles and update the distfiles file, needed when the port version is changed/updated
Sometimes it happens with, for example, GitHub tags that the upstream updates the source and re-tags under same tag, effectively modifying the distfile hashes, so running make makesum will rehash those
Run portlint in the port's dir to check it with the linter
Run make test if the port has a unit test
Modify the port's Makefile as needed. For example:
Bump PORTREVISION if the modifications will result with a changed package (eg. when applying security or bugfix patches for the same PORTVERSION).
If you upgrade PORTVERSION, you can delete any existing PORTREVISIONs or set it to 0.
DO NOT REMOVE PORTEPOCH. See the Porter's Handbook: 5.2 Naming why this is very important to remember.
Consult /home/portsvn/ports/Mk/ files for explanation of various Makefile variables and situations. USES are also well documented and found under /home/portsvn/ports/Mk/Uses . The Porter's Handbook also explains a lot of such cases and components of the framework.
Testing and building (modified) ports from the local tree with Poudriere, for example lang/python35. Before we build it, we set options if we want to deviate from defaults:
poudriere options -p portsvn lang/python35
Now, Poudriere will set up these options permanently under /usr/local/etc/poudriere.d/. The options' dir name depends on factors like jail used, or sets used, but not on ports tree used, which means you can vary options depending on which jail or set you need. Consult the section CUSTOMIZATION of the poudriere(8) manpage to see how it checks for relevant file names. By combining jail names and set names you can build different test cases and scenarios. For example you might wish to vary DEFAULT_VERSIONS.
Default bulk build:
poudriere bulk -j 110x64 -p portsvn -c lang/python35
Testport build, we include -i so we're automatically chrooted into the builder jail after the building is done, to inspect the installed ports or do run tests:
poudriere testport -j 110x64 -p portsvn -i lang/python35
Note that the testport command doesn't have -c to clean up existing packages, so remember to do that when needed. For example, it's needed if we previously testport'ed another port, so we clean up the environment. We don't need it when the port build fails and we want to rebuild just the target port, leaving all the dependencies already built and installed.
When upgrading ports, observe the log for complaints about invalid pkg-plist entries. As ports upgrade or change, naturally some files will be removed, some added. The framework very helpfully explains what is added (present in the staging dir, but not in the pkg-plist) and what is removed (present in the pkg-plist but not in the staging dir).
4. Creating and submitting patches
After we're done with modifications and it builds/runs fine, we prepare the patch against the SVN tree to be committed.
First, make SVN aware of any new files (like patches you created with make makepatch ) added to the port. We run from the port's directory:
svn add files/patch-src_some_path.patch
SVN will pick up modifications and deletions, so we just need to explicitly mark additions. Finally, we create the patch against the tree itself:
# Do this from the ports root directory cd /home/portsvn/ports svn patch --ignore-properties lang/python35 > lang_python35.patch
The actual patch name is arbitrary. The patch is then attached to an issue filed at FreeBSD Bugzilla.
TODO: Adding a VuXML entry for security vulnerabilities.