Packaging FreeBSD base
Contents
About PkgBase
PkgBase is the FreeBSD base operating system (OS) packaged using pkg(8), the default FreeBSD utility for installing and updating ported software.
PkgBase aims to replace .tgz distribution sets for OS installation and freebsd-update(8) for OS updating. It can also complement or replace building and installing from source.
Status
Packages are published for all arches provided they are listed in
make targets
branch |
publication frequency |
url |
main |
twice a day at 12:00 and 00:00 UTC |
|
main |
weekly on sunday at 12:00 UTC |
|
stable/14 |
twice a day at 12:00 and 00:00 UTC |
|
stable/14 |
weekly on sunday at 12:00 UTC |
|
releng/14.1 |
twice a day at 12:00 and 00:00 UTC |
|
releng/14.0 |
twice a day at 12:00 and 00:00 UTC |
publication only happens if there was actually new packages built since the latest publication, releng are usually quite low update branches
Packages will be provided from FreeBSD 14.0 onwards:
as a technology preview of sorts, for experimentation and use by projects such as Doug Rabson's FreeBSD container images
not yet for things such as bsdinstall(8)
- not yet intended for production environments.
In addition, to aid exploration and adoption, an unofficial repository https://pkg.freebsd.evilham.com/images is provided by Evilham (Andrés).
In Bugzilla:
Use of a virtual machine, or a dedicated test machine, is encouraged.
Overview
PkgBase comprises a few hundred packages with the default build (empty src.conf(5) and make.conf(5)) for amd64. The number of packages depends on several factors.
In most cases, a runtime binary is split into multiple components. Debugging symbols and most shared libraries are packaged individually. 32-bit is bit packaged separately.
Initial setup
Ensure your pkg(8) version is at least 1.20.9
- Use FreeBSD 14.0 at minimum (14.0 will be assumed in the steps below)
- Create a directory for pkg repositories and add the pkgbase one
# mkdir -p /usr/local/etc/pkg/repos/ # cat <<EOF >/usr/local/etc/pkg/repos/base.conf > base: { > url: "pkg+https://pkg.FreeBSD.org/\${ABI}/base_release_0", > mirror_type: "srv", > signature_type: "fingerprints", > fingerprints: "/usr/share/keys/pkg", > enabled: yes > } > EOF
- Update the repositories
# pkg update Updating FreeBSD repository catalogue... FreeBSD repository is up to date. Updating base repository catalogue... Fetching meta.conf: 100% 163 B 0.2kB/s 00:01 Fetching packagesite.pkg: 100% 33 KiB 33.9kB/s 00:01 Processing entries: 100% base repository update completed. 421 packages processed. All repositories are up to date.
- Install all base packages
# pkg install -r base -g 'FreeBSD-*' New packages to be INSTALLED: FreeBSD-acct: 14.rc4.20231103061237 [base] FreeBSD-acct-dbg: 14.rc4.20231103061237 [base] FreeBSD-acpi: 14.rc4.20231103061237 [base] FreeBSD-acpi-dbg: 14.rc4.20231103061237 [base] ... FreeBSD-zfs-dev-lib32: 14.0p2 [base] FreeBSD-zfs-lib32: 14.0p2 [base] FreeBSD-zoneinfo: 14.rc4.20231103061237 [base] Number of packages to be installed: 421 The process will require 4 GiB more space. 1 GiB to be downloaded. Proceed with this action? [y/N]:y
- Deal with .pkgsave
If a file that was already present on the disk differs from what is inside the package holding this file, pkg(8) will backup the old one as .pkgsave This is the same for config files, pkg(8) will not be able to merge them all if they were modified.
# cp /etc/master.passwd.pkgsave /etc/master.passwd # cp /etc/group.pkgsave /etc/group # pwd_mkdb -p /etc/master.passwd # cp /etc/sysctl.conf.pkgsave /etc/sysctl.conf
Those files above are the minimum that you should check and restore, other ones might exists.
You can run the following to find all .pkgsave files and copy them interactively:
# find / -name "*.pkgsave" -type f -exec sh -c "f='{}'; echo '==== OLD ===='; ls -l \${f}; md5sum \${f}; echo '==== NEW ===='; ls -l ${f%.pkgsave}; md5sum \${f%.pkgsave}; cp -vi \${f} \${f%.pkgsave}" \;
# find / -name \*.pkgsave -delete # rm /boot/kernel/linker.hints (linker.hints was recreated at kernel install and we had the old modules as .pkgsave so we need to recreate it, this will be done at the next reboot)
Upgrades
To upgrade just simply pkg update && pkg upgrade
Major version upgrades
When attempting a major upgrade of FreeBSD (e.g. from 11 to 12), you may find the following error:
# pkg upgrade -r FreeBSD-base Updating FreeBSD-base repository catalogue... Fetching meta.txz: 100% 268 B 0.3kB/s 00:01 Fetching packagesite.txz: 100% 52 KiB 52.9kB/s 00:01 Processing entries: 0% pkg: wrong architecture: freebsd:12:x86:64 instead of FreeBSD:11:amd64 pkg: repository FreeBSD-base contains packages with wrong ABI: freebsd:12:x86:64 Processing entries: 100% Unable to update repository FreeBSD-base All repositories are up-to-date. pkg: Repository FreeBSD-base cannot be opened. 'pkg update' required Checking for upgrades (0 candidates): 100% Processing candidates (0 candidates): 100% Checking integrity... done (0 conflicting) Your packages are up to date.
For this case, the following command will allow a successful upgrade:
# env ABI=freebsd:12:x86:64 pkg-static upgrade -r FreeBSD-base
Building
To build base and kernel packages and install them to a jail/boot environment directory (14.1 host, 32 cores available):
root@freebsd:~ # cd /usr/src root@freebsd:~ # make -j32 buildworld && make -j32 buildkernel && make -j32 packages root@freebsd:~ # mkdir /root/myjail root@freebsd:~ # mkdir -vp /root/myjail/usr/share/keys/pkg/trusted /root/myjail/usr /root/myjail/usr/share /root/myjail/usr/share/keys /root/myjail/usr/share/keys/pkg /root/myjail/usr/share/keys/pkg/trusted root@freebsd:~ # cp -av /usr/share/keys/pkg/trusted/* /root/myjail/usr/share/keys/pkg/trusted/ /usr/share/keys/pkg/trusted/ -> /root/myjail/usr/share/keys/pkg/trusted /usr/share/keys/pkg/trusted/pkg.freebsd.org.2013102301 -> /root/myjail root@freebsd:~ # mkdir -vp /root/myjail/etc/pkg /root/myjail/etc /root/myjail/etc/pkg root@freebsd:~ # cat /root/myjail/etc/pkg/FreeBSD.conf FreeBSD-base-14.1 { url = file:////usr/obj/usr/src/repo/FreeBSD:14:amd64/14.1 mirror_type = none enabled = yes } Update the repository (full paths required): root@freebsd:~ # pkg --rootdir /root/myjail --repo-conf-dir /root/myjail/etc/pkg update --repository FreeBSD-base-14.1 Find all packages by name and display to standard out: root@freebsd:~ # pkg --rootdir /root/myjail --repo-conf-dir /root/myjail/etc/pkg rquery --repository FreeBSD-base-14.1 --evaluate '%n ~ FreeBSD-*' %n Install to the jail root: root@freebsd:~ # pkg --rootdir /root/myjail --repo-conf-dir /root/myjail/etc/pkg install --repository FreeBSD-base-14.1 --glob --yes '*' root@freebsd:~ # chroot /root/myjail root@freebsd:/ # ls .cshrc COPYRIGH boot etc libexec rescue sbin usr .profile bin dev lib root tmp var
Caveat: /usr/obj/usr/src/repo/FreeBSD:14:amd64/14.1 will not be valid from within the chroot/jail/boot environment and must be populated from the build host in some way.
Outstanding
No method to notify users when new base packages are created (packages that did not previously exist), and they are not installed during a pkg upgrade. This is particularly problematic if files are moved from an existing package to a new dedicated package, which will result in them being deleted during the next pkg upgrade. This was noticed after r299840, which moves /etc/rc.d/[zfs|zvol] from the existing FreeBSD-runtime package to a new FreeBSD-zfs package, and resulted in them being deleted during the next pkg upgrade.
Please follow up on the freebsd-pkgbase@ mailing list with problems (and successes).
Goals, and additional unresolved issues
The following items need to be addressed before PkgBase can be considered for production use:
bsdinstall(8) needs to be updated to use pkg(8) during installation (this needs to be optional, otherwise the installworld target will need to be pkg(8)-aware, and must be able to handle non-networked systems) (manu@ : working on it)
- pkg(8) needs to be able to be cross-built for non-amd64 architectures to be able to install on non-networked systems
- Infrastructural implementation for publishing packages, similar to pkg.FreeBSD.org
- In addition to infrastructural support, package signing and other related items needs to be implemented
A wrapper to support the functionality provided by freebsd-update(8) needs to be written, and needs to be 1:1 compatible (manu@ : working on it)
- Handbook updates for installation and upgrading
- Updates to the Release Engineering article
- A strategy must be well thought out and implemented on how to handle Security Advisories and Errata Notices, from when a change gets committed to a releng branch to how updated packages are made available on mirrors
- pkg(8) must continuously be backward-compatible against previous versions
- Generated files, such as mandoc.db, must be excluded from packages
- Conflicting files cannot exist, and known conflicting files must be addressed; for example, the FreeBSD-runtime and FreeBSD-runtime-manuals packages both try to install the tree(3) manual page (note: this particular issue is fixed, but still conceptually unresolved)
How to handle /boot/kernel and /boot/kernel.$KERNCONF for multi-kernel support needs to be addressed (Note: while both kernels can be installed simultaneously as packages, the one that gets installed to /boot/kernel is a dependency of the runtime package. Additionally, changing from make KERNCONF="GENERIC GENERIC-NODEBUG" buildkernel to make KERNCONF="GENERIC-NODEBUG GENERIC" does properly replace /boot/kernel and /boot/kernel.GENERIC-NODEBUG with /boot/kernel and /boot/kernel.GENERIC.) (manu@ : working on it)
How to handle ensuring the kernel package is always installed and the system is rebooted before the new userland is installed (Note: Committed fixes for this as r322327, with followup commits r322352 and r322358. brd pointed out that this dependency as-is is inherited by jail(8), so might need some tweaking.) (Note 2: this was reverted. See the commit message for r322545.)
- The kernel pkg will be marked as vital and thus can only be upgraded.
Reboot is not always required, but we will leave that up to the user as is handled currently. Post install script that will notify the user to reboot. Does require a new pkg feature to handle this: pkg#1699
- How to handle removed files/packages (i.e., rcmds removal in 12-CURRENT)
- manu@: pkg autoremove should be just what's needed here
Conversion script to pkgbase to handle registering pkgs without forcing pkg install -f or similar.
IDEA: script shipped with METALOG from a full build that walks the METALOG and does pkg register with the version from freebsd-version as it finds files that belong to pkgs.
- IDEA: Ship the plist and ucl files from the build, since that is what pkg register would need.
Metapkgs: The current is to implement a pkg feature called groups. See: pkg#1697
Retrospective
A call for testing was made in March 2016.
Major blockers and critical issues appeared to be resolved.
Glen Barber's notes from the packaging session at the 2017-08 BSDCam devsummit
Userland packages sets – can someone clarify what exactly this means?
TODO:
Package containing the source tree that a [base] package set was built from. Note: while I think this is a good idea, I have concerns about behavioral consistency between pkg upgrade and svn up ''/usr/src''. Some things that come to mind immediately are:
locally-patched source trees and losing patches on pkg upgrade;
svn up changing checksums recorded in the pkg(8) database;
ultimately "Which one wins?" in an update of /usr/src?
Add SVN rev of source tree to pkg metadata (or equivalent, plus indicate if the tree has been modified similarly to uname(1)) (Note: committed as r322412.)
RDBMS-backed web application to collect file checksums from every official package and allow users to verify files on-line.
Using pkgdb as an IDS?
Document FreeBSD 14.1 the source package.