This page is part of the TestSuite project and provides a description of the basic concepts involved in the test suite, its build infrastructure and the structure of the installed products.

IMPORTANT: Please do not take anything for granted, ESPECIALLY if you have previously worked with and/or have seen src/tools/test/ and src/tools/regression/. Some details differ in important ways.

This document is based on a similar tutorial in NetBSD and a lot of text has been copied form there.

Contents

Concepts
Structure of the installed test suite
Build infrastructure for the test suite

Concepts

Kyua

Kyua is the runtime engine of choice to run the FreeBSD test suite. This utility implements a modular system to safely run test programs of various kinds and that collates their results into a unified database for later report generation and debugging.

Kyua is available from the ports system under devel/kyua/ and its homepage is at http://code.google.com/p/kyua/

Test programs vs. test cases

The definitions in here are tricky given that the line between each concept is very thin.

A test case is a piece of code that exercises a specific functionality of another piece of code. Commonly, test cases validate the outcome of a particular function or class method, or the validity of the execution of a command with a particular combination of flags/arguments. Test cases are supposed to be very concise and should test just one behavior.

A test is more specific than a test case in the sense that a test is the actual "comparison" being performed. If you are familiar with "checks" and "requires" in the popular *Unit frameworks, a single line invoking these functions is a test. For example: in a calculator application, one could have a test case for additions, another for subtractions... and each of these could invoke different tests by using different inputs (corner cases) to the tested operations. That said, a well-designed test suite will often have one single test per test case and thus the test case and test terms refer to the same thing.

There are two types of tests: CHECKs and REQUIREs. A check-style test raises an error in the test case but does not cause the test case to terminate immediately. On the other hand, a require-style test terminates the test case right away. You should use require-style tests whenever it is impossible for the test to continue and yield any meaningful results if that particular test fails.

A test program is a binary that collects and exposes a group of test cases. Typically, these test programs expose conceptually-related tests or all the tests for a particular source file.

In general, having many test programs with just one test case in them is wrong. Consider some some other organization. Please note that this is extremely common in (almost?) all other test frameworks and, when used wisely, becomes an invaluable structure for your test suite.

For example, suppose you have the following fictitious source files for the ls tool:

bin/ls/fs.c: Provides the list_files() and stat_files() functions.
bin/ls/ui.c: Provides the format_columns() function.
bin/ls/main.c: The main method for ls.

Then, you could define the following test programs and test cases:

bin/ls/fs_test.c: Provides test cases for list_files and stat_files. These would be named list_files__empty_directory, list_files__one_file, list_files__multiple_files, stat_files__directory, stat_files__symlink, etc.
bin/ls/ui_test.c: Provides test cases for the format_columns function. These would be named format_columns__no_files, format_columns__multiple_files, etc.
bin/ls/integration_test.sh: Provides "black box" test cases for the binary itself. These would be named lflag, lflag_and_Fflag, no_flags, no_files, etc.

Try to keep your test case names as descriptive as possible so that they do not require comments to explain what they intend to test. The test case names are exposed to the user.

Test program types

Kyua supports the execution of both plain (aka legacy) test programs as well as ATF-based test programs. (Support for other test program types is possible and easy, but there are no specific plans at the moment to implement them.)

Plain test programs are simple binaries that perform one or more tests from their main() function and return 0 on success and non-0 on failure. While plain test programs can implement various tests, they only implement a single test case (the main() function). Other than their return code, these test programs have no mechanism to communicate information to the runtime engine.

ATF-based test programs are binaries that link against the ATF libraries. These test programs: are able to provide separate test cases that the run-time engine can execute individually; have access to rich APIs to simplify the coding of common testing patterns; and can report feedback to the run-time engine on the granularity of each test case.

The recommendation is to write ATF-based test programs wherever possible given their versatility. Plain test programs should only be used as a transitional step while incorporating existing tests into the new FreeBSD test suite or, more importantly, to support plugging tests of third-party software over which FreeBSD has no control.

ATF test case parts

Each ATF test case is split in three parts: the body, which is required; and the head and the cleanup, which are optional.

The head

The head is used for the sole purpose of defining meta-data properties for the test case.

The full list of available properties is provided in kyua-atf-interface(1), and these are the most common ones:

descr: A textual description of the purpose of the test case.
require.user: Set to 'root' to tell the tests runtime that this test requires root privileges. The test will later be skipped if you are running Kyua as non-root, and the test will be executed otherwise. Please do not use this unless absolutely necessary! You can most likely make your tests run as a regular user.
use.fs: Set to 'true' if the test case creates temporary files in the "current directory". If set to false, the tests runtime will set the "current directory" to an unwritable directory, which will disallow the creation of the temporary files and will make your test fail if it attempts to do so. Disabling access to the file system is generally good practice to ensure that test cases don't "leak" files unexpectedly, but is also beneficial for performance reasons.

The body

The body is the actual meat of the test case. This is just a regular function that executes any code you want and calls special ATF functions to report failures; see below.

Test case isolation

Kyua isolates the execution of every test case to prevent side-effects (such as temporary file leftovers, in-memory data corruption, etc.). In particular:

A test case is always executed as a subprocess that is separate from the test runner program. This implies that you cannot pass any in-memory state between the parts.
The current working directory of a test case is changed to a temporary location that gets cleaned up later on automatically. (Set the use.fs property to true in the head if you need to write to this temporary directory.)
The environment of the test case is "sanitized" to get rid of variables that can cause side-effects; e.g. LC_ALL, TZ, etc.

Structure of the installed test suite

The FreeBSD test suite lives in /usr/tests/ and is available out of the box on any system (assuming the system was built with WITH_TESTS, which defaults to disabled at the moment).

Test suite definitions

The whole /usr/tests/ tree represents the full FreeBSD test suite and executing the test suite from that directory will cause all tests to be run. However, every subdirectory of /usr/tests/ can be considered a test suite of its own. For example, the /usr/tests/bin/cp/ subdirectory holds the test suite of the cp(1) binary and it can be executed on its own without having to run all other tests.

This is an important property: any slice of the whole test suite is a test suite of its own and must be able to work independently.

Directory structure

The test suite in /usr/tests/ is intended to mimic the layout of the src tree as much as possible. For example:

/usr/src/bin/cp/ -> /usr/tests/bin/cp/
/usr/src/lib/libc/ -> /usr/tests/lib/libc/
/usr/src/usr.bin/cut/ -> /usr/tests/usr.bin/cut/

Kyuafile scripts

The way the above is implemented is by having a Kyuafile in every subdirectory of /usr/tests/. A Kyuafile file is a script that defines the test suite hanging from that subdirectory onwards and does so by either registering the test programs that exist within a directory and/or by including Kyuafile files from immediate subdirectories.

Here come a set of fictitious Kyuafiles for a test suite containing some tests for cp(1) and libc:

==== /usr/tests/Kyuafile ====
syntax(2)
include('bin/Kyuafile')
include('lib/Kyuafile')

==== /usr/tests/bin/Kyuafile ====
syntax(2)
include('cp/Kyuafile')

==== /usr/tests/bin/cp/Kyuafile ====
syntax(2)
atf_test_program{name='fast_test'}
atf_test_program{name='huge_test'}

==== /usr/tests/lib/Kyuafile ====
syntax(2)
include('libc/Kyuafile')

==== /usr/tests/lib/libc/Kyuafile ====
syntax(2)
plain_test_program{name='close_test'}
plain_test_program{name='open_test', require_progs='some-helper-tool'}
plain_test_program{name='read_test'}
plain_test_program{name='write_test'}

In reality, however, things do not work in this exact way. The top-level Kyuafile and the Kyuafile of any top-level subdirectory of /usr/tests contain a smart script that is able to auto-discover any */Kyuafiles. The reasons for this are given below in the section about the build infrastructure.

For actual sample Kyuafile code, please refer to src/share/examples/tests/.

Build infrastructure for the test suite

Basic test suite components

The src/tests/ directory of the source tree provides basic, generic machinery to build the contents of /usr/tests/. This tree may eventually also include cross-functional tests that do not fit any other directory of the source tree.

In particular, the contents of src/tests/ work this way:

src/tests/Kyuafile is a generic Kyua script that scans the current directory and automatically includes any */Kyuafiles that are found. This file is installed into /usr/tests/ and also into any top-level subdirectory in /usr/tests/ (e.g. /usr/tests/bin/' and /usr/tests/lib/').
src/tests/Makefile installs /usr/tests/Kyuafile and recurses into all other subdirectories of src/tests/ to do their thing.
src/tests/bin/Makefile (and any other Makefile for top-level subdirectories) installs the corresponding Kyuafile and may recurse into other subdirectories.

Makefiles for test programs

Test programs are spread throughout the tree as the goal for the code of the test programs is to live next to the code they are testing. All testing code should be placed in a tests subdirectory of the corresponding tool or library. For example: src/lib/libcrypt/tests/.

These directories should be conditionally built and installed via the MK_TESTS knob. On 12.x branches and newer, this should be done as follows:

==== snippet from src/lib/libcrypt/Makefile that handles integration ====

HAS_TESTS=
SUBDIR.${MK_TESTS}+= tests # `${MK_TESTS}` requires src.opts.mk or bsd.own.mk

On 11.x branches and older, this should be done as follows:

==== snippet from src/lib/libcrypt/Makefile that handles integration ====

# `${MK_TESTS}` requires src.opts.mk on 11.x branches; it requires bsd.own.mk on other versions.
.if ${MK_TESTS} != "no"
SUBDIR+= tests
.endif

Every directory containing tests must have a Makefile that includes bsd.test.mk. The Makefile must also set TESTSDIR to the directory in which these tests will be installed, if it does not fit the hierarchy sans the last component, e.g., /usr/tests/lib/libcrypt is the default value for src/lib/libcrypt. For example:

==== src/lib/libcrypt/tests/Makefile ====

# Optional: bsd.own.mk needs to be included when referencing `${TESTSBASE}`, etc.
#.include <bsd.own.mk>

# TESTSDIR: implied value after r289158 shown below.
#TESTSDIR= ${TESTSBASE}/lib/libcrypt

ATF_TESTS_C= crypt_test

.include <bsd.test.mk>

For actual sample Makefile code, please refer to src/share/examples/tests/.

Test program autodiscovery

As mentioned above, the top-level Kyuafiles auto-discover the test programs that appear in the directories they are located. The reason for this is to simplify the definition of the test suite by decoupling the scripts that define the test suite from the actual test programs being installed.

Consider, for example, the build machinery to construct /usr/tests/lib/. The various libraries in src/lib/ will have their tests in src/lib/*/tests and will install them into subdirectories of /usr/tests/lib/. However, the /usr/tests/lib/Kyuafile file is installed from src/tests/lib/, which has no knowledge of (and shouldn't have) src/lib/*/tests/.