Packaging FreeBSD base
Contents
Overview
PkgBase, as the name implies, is the packaging of the FreeBSD base system. With it, users can upgrade their FreeBSD system using pkg(8) rather than using freebsd-update(8) or building from source.
Status
Packaged base (PkgBase) work is happening in HEAD.
A call for testing was made in March 2016. Major blockers and critical issues appear to be resolved. Please note, as with any development branch, this is not yet intended for production environments. Use of virtual machines or dedicated testing machines is strongly encouraged.
PkgBase comprises around 300 packages with the default build (empty src.conf(5) and make.conf(5)) for amd64. The number depends on several factors, but for most cases a runtime binary is split into several components. In particular, most shared libraries are individually packaged, in addition to debugging symbols, and 32-bit packaged separately.
To aid exploration and adoption, Evilham provides an unofficial pacakge repository.
Outstanding
There is currently no method to notify users when new base packages are created (packages that did not previously exist), and they are not installed during a pkg upgrade. This is particularly problematic if files are moved from an existing package to a new dedicated package, which will result in them being deleted during the next pkg upgrade. This was noticed after r299840, which moves /etc/rc.d/[zfs|zvol] from the existing FreeBSD-runtime package to a new FreeBSD-zfs package, and resulted in them being deleted during the next pkg upgrade.
r337991 fixes an issue where a user has defined DISTDIR in make.conf; the error will be that awk can't find [....]/METALOG during make packages. 11.2-RELEASE needs this patch (or wrap the DISTDIR definition with .if ! ${.CURDIR:M/usr/src*})
- certutil rehash needs to be manually run after upgrades
Please follow up on the freebsd-pkgbase@ mailing list with problems (and successes).
Initial setup
Ensure your pkg(8) version is at least 1.10.5_3.
- Update your FreeBSD source tree to HEAD.
Build the userland and kernel as normal with the buildworld and buildkernel targets. Afterward, packages can be created with the packages Makefile target.
# cd /usr/src # make [make flags] buildworld # make [make flags] buildkernel # make packages
The package repository will be created within /usr/obj/usr/src/repo by default.
To enable the repository for use by the local machine, create /usr/local/etc/pkg/repos/FreeBSD-base.conf with the following contents:
# FreeBSD base system repository FreeBSD-base: { url: "file:///usr/obj/usr/src/repo/${ABI}/latest", mirror_type: "none", enabled: yes }
To enable the repository for use on remote machines, you can setup an NGINX webserver on the machine that built the packages, adding the following lines to /usr/local/etc/nginx/nginx.conf:
location /FreeBSD-base { alias /usr/obj/usr/src/repo; autoindex on; }
On remote machines that will use the PkgBase repository, create /usr/local/etc/pkg/repos/FreeBSD-base.conf with the following contents:
# FreeBSD base system repository FreeBSD-base: { url: "http://URL.TO.PKGBASE.MACHINE/FreeBSD-base/${ABI}/latest", enabled: yes }
If installing inside a jail, do not forget to create that jail with allow.chflags set to 1. Without this option, pkg(8) will be unable to overwrite files with the schg flag set, such as /lib/libc.so.7 or /sbin/init
To initially bootstrap the 'FreeBSD-*' packages, they must be forcibly installed. Package registration is not performed during installworld or installkernel, and there are no immediate plans to do this. This can be done by running:
# pkg update -r FreeBSD-base # pkg install -g 'FreeBSD-*'
Caution: There is currently no built-in mechanism for bootstrapping pkg info from the existing system, and files (such as /etc/master.passwd) will be replaced with default copies from the newly installed packages. Existing files will be renamed with a .pkgsave extension. Files that may be of particular interest include:
- /etc/aliases
- /etc/fstab
- /etc/group
- /etc/hosts
- /etc/mail.rc
- /etc/master.passwd
- /etc/ssh/ssh_config
- /etc/ssh/sshd_config
- /etc/sysctl.conf
- /etc/syslog.conf
- /etc/ttys
- /root/.cshrc
- /root/.k5login
- /root/.login
- /root/.profile
- /root/.shrc
WARNINGS:
Fixing missing user
After upgrading, if you have the following message from su
su: who are you?
It means that you have replaced your master.passwd file during the upgrade, in order to restore your user and rebuild the passwd database, you can do the commands below, then login again:
# cp /etc/master.passwd.pkgsave /etc/master.passwd # /usr/sbin/pwd_mkdb -p /etc/master.passwd
Kernel packages
It is possible to build multiple kernel packages by listing the kernel configuration names in the KERNCONF variable (separated by spaces) defined in make.conf(5), as long as NO_INSTALLEXTRAKERNELS is also set to no. The first listed kernel in KERNCONF is installed as /boot/kernel, and subsequent kernels in KERNCONF are installed as /boot/kernel.${KERNEL}.
Building GENERIC is not required, as each kernel package is named with the kernel name included.
For example, if make.conf(5) contains:
NO_INSTALLEXTRAKERNELS=no KERNCONF=MYKERNEL1 MYKERNEL2
The following kernel packages will be created:
- FreeBSD-kernel-mykernel1-release
- FreeBSD-kernel-mykernel1-debug
- FreeBSD-kernel-mykernel2-release
- FreeBSD-kernel-mykernel2-debug
Upgrading
After the first time the source tree was built and installed with PkgBase, an upgrade involves the following steps:
Update the FreeBSD source tree and build the userland and kernel as normal with the buildworld and buildkernel targets. Afterward, packages can be created with the packages Makefile target.
# cd /usr/src # svn up # make [make flags] buildworld # make [make flags] buildkernel # make update-packages
Note that for make update-packages working correctly you need WITH_REPRODUCIBLE_BUILD=yes in /etc/src.conf as it compares the internal package data.
- Upgrade the base packages:
# pkg upgrade -r FreeBSD-base
- Check if any new base packages were created, during the build process, that would not be installed during an upgrade; and check if any base packages no longer exist that would not have been removed during an upgrade:
$ diff <(pkg rquery -r FreeBSD-base %n) <(pkg query -e '%o = base' %n) | grep "^[<>]"
Lines beginning with < are base packages that were built and exist in the repo but are not installed on your system. Lines beginning with > are base packages that are installed on your system but were not built and do not exist in the repo. Consider whether (a) you need to install or remove these packages; or (b) there is a deliberate difference between the built repo and your system.
Major version upgrades
When attempting a major upgrade of FreeBSD (e.g. from 11 to 12), you may find the following error:
# pkg upgrade -r FreeBSD-base Updating FreeBSD-base repository catalogue... Fetching meta.txz: 100% 268 B 0.3kB/s 00:01 Fetching packagesite.txz: 100% 52 KiB 52.9kB/s 00:01 Processing entries: 0% pkg: wrong architecture: freebsd:12:x86:64 instead of FreeBSD:11:amd64 pkg: repository FreeBSD-base contains packages with wrong ABI: freebsd:12:x86:64 Processing entries: 100% Unable to update repository FreeBSD-base All repositories are up-to-date. pkg: Repository FreeBSD-base cannot be opened. 'pkg update' required Checking for upgrades (0 candidates): 100% Processing candidates (0 candidates): 100% Checking integrity... done (0 conflicting) Your packages are up to date.
For this case, the following command will allow a successful upgrade:
# env ABI=freebsd:12:x86:64 pkg-static upgrade -r FreeBSD-base
Project goals and additional unresolved issues
The following items need to be addressed before PkgBase can be considered for production use:
bsdinstall(8) needs to be updated to use pkg(8) during installation (this needs to be optional, otherwise the installworld target will need to be pkg(8)-aware, and must be able to handle non-networked systems) (manu@ : working on it)
- pkg(8) needs to be able to be cross-built for non-amd64 architectures to be able to install on non-networked systems
- Infrastructural implementation for publishing packages, similar to pkg.FreeBSD.org
- In addition to infrastructural support, package signing and other related items needs to be implemented
A wrapper to support the functionality provided by freebsd-update(8) needs to be written, and needs to be 1:1 compatible (manu@ : working on it)
- Handbook updates for installation and upgrading
- Updates to the Release Engineering article
- A strategy must be well thought out and implemented on how to handle Security Advisories and Errata Notices, from when a change gets committed to a releng branch to how updated packages are made available on mirrors
- pkg(8) must continuously be backward-compatible against previous versions
- Generated files, such as mandoc.db, must be excluded from packages
- Conflicting files cannot exist, and known conflicting files must be addressed; for example, the FreeBSD-runtime and FreeBSD-runtime-manuals packages both try to install the tree(3) manual page (note: this particular issue is fixed, but still conceptually unresolved)
How to handle /boot/kernel and /boot/kernel.$KERNCONF for multi-kernel support needs to be addressed (Note: while both kernels can be installed simultaneously as packages, the one that gets installed to /boot/kernel is a dependency of the runtime package. Additionally, changing from make KERNCONF="GENERIC GENERIC-NODEBUG" buildkernel to make KERNCONF="GENERIC-NODEBUG GENERIC" does properly replace /boot/kernel and /boot/kernel.GENERIC-NODEBUG with /boot/kernel and /boot/kernel.GENERIC.) (manu@ : working on it)
How to handle ensuring the kernel package is always installed and the system is rebooted before the new userland is installed (Note: Committed fixes for this as r322327, with followup commits r322352 and r322358. brd pointed out that this dependency as-is is inherited by jail(8), so might need some tweaking.) (Note 2: this was reverted. See the commit message for r322545.)
- The kernel pkg will be marked as vital and thus can only be upgraded.
Reboot is not always required, but we will leave that up to the user as is handled currently. Post install script that will notify the user to reboot. Does require a new pkg feature to handle this: pkg#1699
- How to handle removed files/packages (i.e., rcmds removal in 12-CURRENT)
- manu@: pkg autoremove should be just what's needed here
Conversion script to pkgbase to handle registering pkgs without forcing pkg install -f or similar.
IDEA: script shipped with METALOG from a full build that walks the METALOG and does pkg register with the version from freebsd-version as it finds files that belong to pkgs.
- IDEA: Ship the plist and ucl files from the build, since that is what pkg register would need.
Metapkgs: The current is to implement a pkg feature called groups. See: pkg#1697
Glen Barber's notes from the packaging session at the 2017-08 BSDCam devsummit
Userland packages sets – can someone clarify what exactly this means?
TODO:
Package containing the source tree that a [base] package set was built from. Note: while I think this is a good idea, I have concerns about behavioral consistency between pkg upgrade and svn up ''/usr/src''. Some things that come to mind immediately are:
locally-patched source trees and losing patches on pkg upgrade;
svn up changing checksums recorded in the pkg(8) database;
ultimately "Which one wins?" in an update of /usr/src?
Add SVN rev of source tree to pkg metadata (or equivalent, plus indicate if the tree has been modified similarly to uname(1)) (Note: committed as r322412.)
RDBMS-backed web application to collect file checksums from every official package and allow users to verify files on-line.
Using pkgdb as an IDS?