Release Testing

This page contains notes from gjb@ on what testing is done during a release cycle. Once complete, this page will be integrated in to the FreeBSD Releng article in the Handbook.

Testing Environment

For testing BETA, RC, and RELEASE builds, I use a combination of real hardware, VirtualBox, and QEMU. There is no real reason I use VirtualBox over bhyve or other hypervisors other than I find it easier to manipulate virtual machines.

ISO Image Testing

Image	What to test	Notes
bootonly.iso	That it boots.	Since the installer uses FTP to fetch the distribution sets which have not yet propagated to the mirrors, there is not much more to check here.
disc1.iso	It boots, burns to a CD-ROM without problems, and installs without issues.	[1]
dvd1.iso	Install packages provided on-disc.	[2]

[1] I test the installer by doing a few variations of options. Of note:

amd64: mirrored ZFS, mirrored ZFS with GELI, mirrored ZFS with encrypted swap, ZFS with GELI with a space in the passphrase
i386: UFS, default options
all default options selected
miscellaneous options enabled/disabled (especially any new options since the last release)
adding user accounts with non-default pw(8) options, such as the login shell, additional GIDs, password containing a space
When booting into the installed system, verify 'uname -a' looks sane (i.e., correct Subversion revision, $USER from the build machine did not leak, correct architecture, etc.)
Note: Provided the installer does not change during the release cycle, the different variations are not necessarily needed each build unless there are changes to something directly used by bsdinstall(8) (i.e., a ZFS merge, GELI update, pw(8) update, etc.)
Run the Kyua test suite on the freshly-installed system:

  # pkg install -y kyua
  # cd /usr/tests   # Note: /usr/tests is not built or installed by default on 10.x and earlier
  # script /var/log/kyua.log kyua test -k /usr/tests/Kyuafile
  # kyua report-html --output ~/public_html/tests

[2] Generally, I make sure dvd1.iso boots, but re-use the disc1.iso virtual machine for testing package installation. Since bsdconfig(8) has not supported installing packages from dvd1.iso, packages are installed as:

  # mkdir -p /dist
  # mount -t cd9660 /dev/cd0 /dist
  # setenv REPOS_DIR /dist/packages/repos
  # pkg bootstrap
  # pkg update
  # pkg install xorg xorg-server gnome3
  # pkg del -ay
  # pkg install xorg xorg-server kde

After ensuring the desktop environments start properly, install the emulators/virtualbox-ose-additions{,nox11} package from the upstream pkg(8) mirrors and enable the rc(8) scripts in rc.conf(5):

  # unsetenv REPOS_DIR
  # pkg update
  # pkg install emulators/virtualbox-ose-additions-nox11
  # echo 'vboxservice_enable="YES"' >> /etc/rc.conf
  # echo 'vboxguest_enable="YES"' >> /etc/rc.conf

Virtual Machine Testing

Ensure each image format boots (Note: I use QEMU for this, as it is faster than VirtualBox to set up, and QEMU supports all four image formats we provide)
Ensure 'pkg bootstrap' works
Ensure 'uname -a' looks sane

ARM SoC Testing

Verify the images boot on boards I have
Verify SSH access works as the 'freebsd' user
Ensure su(1) to the 'root' user works
Ensure 'pkg bootstrap' works