• TestSuite
• DeveloperHowTo

This page is part of the TestSuite project and provides a tutorial on how to add new tests to the test suite and a reference guide for developers. It is assumed that you have already gone over TestSuite/Structure, especially the section detailing the naming scheme for the tests.

Before reading this page, make sure to read tests(7) and src/share/mk/bsd.README for details.

This document is based on a similar tutorial in NetBSD and a lot of text has been copied from there.

Adding a new test case

To add a new test case to the source tree, check if any existing test program can assimilate your test case. What you are looking for here is a test program with a generic-enough name that can encapsulate the behavior of the new test you want to add. If test programs use the right naming scheme, and if the code you are modifying already has tests, it is very likely that you will find a match.

If you find such a program, bingo! Just add the test case to it: no other changes are required. Otherwise, you will have to create a new test program.

Adding a new test program

If you need to add a new test program to the source tree, you first need to consider the type of test program you will be creating. The recommendation is that you create an ATF-based test program; the rationale being that these are more versatile both from a development perspective and the user's experience. Only resort to adding a plain test program as a transitional step or, especially, if you do not own the code of the test (e.g. you are just hooking a test from a third-party package bundled in the base system).

The procedure to add a new test program is the following:

1. Locate the appropriate subdirectory in which to put your test program. In general this just means the tests subdirectory located where the code you are interested in testing lives. It is OK and expected to have multiple test programs into the same directory. Avoid creating a directory per test program!

If the tests subdirectory exists:

1. Choose a sane name for the test program. Avoid naming the test program the same as your test case! If you find yourself in this situation, it probably means that you are choosing a bad name for either of them.

2. Create the test program source file. Refer to share/examples/tests/ for test program sample code.

3. Add the new test program to the existing Makefile.

If the tests subdirectory does not exist:

1. Create an empty tests subdirectory.

2. Do the same as above.

3. Create the Makefile for the directory. Refer to share/examples/tests/ for Makefile sample code.

4. Edit the parent Makefile to recurse into the new subdirectory.

5. Edit etc/mtree/BSD.tests.dist to register the new subdirectory. Note that if you are adding tests to, say, usr.bin/du/tests/, the directory you register in the mtree file should be /usr/tests/usr.bin/du/; i.e. the layout under /usr/tests/ must match the layout of /usr/src.

Adding a new top-level directory to /usr/tests/

If you need to add a test program to a top-level directory within /usr/tests/ that does not exist yet, you will have to populate such directory first with a corresponding Kyuafile.

To do this, follow these steps (which assume you would like to register a tests subdirectory for usr.bin):

1. Edit etc/mtree/BSD.tests.dist to register the new top-level directory.

2. Edit usr.bin/Makefile and:

   1. Add src.opts.mk (after r265419), bsd.init.mk or bsd.own.mk at the top of the Makefile if needed. Make sure that the relevant .mk file is included after any override variables (LIBDIR, etc), otherwise including the .mk file might break the install path, e.g. r271241.

   2. Add the contents below at the end of the file (before any file inclusions). Note that this may not work for tricky Makefiles that mess around with SUBDIR in strange ways. All of these Makefiles already deal with subdirectories, so check how they do so and register tests using the existing format. The standard way, however, is:

      .if ${MK_TESTS} != "no"
      SUBDIR+= tests
      .endif

   Some complete examples can be found here:

   • lib/libnv

   • usr.bin/tr

3. Create usr.bin/tests/Makefile and add the lines below to it. Make sure to replace usr.bin with the name of your new directory. These lines cause the generic Kyuafile from tests/Kyuafile to be installed in your new directory so that tests living in subdirectories can automatically be discovered.

   # $FreeBSD$
   
   .include <bsd.own.mk>
   
   TESTSDIR=       ${TESTSBASE}/usr.bin
   
   .PATH:          ${SRCTOP}/tests # This is required for tests/Kyuafile
   KYUAFILE=       yes
   
   .include <bsd.test.mk>

4. Add your tests within usr.bin/tests/ as usual.

Adding other intermediate directories

If you are trying to add tests to subdirectories that are more than two levels deep from the /usr/tests/ root, then the above instructions are insufficient and you will need to add more intermediate Kyuafiles. Kyua requires every directory in a path to have a single Kyuafile that tells it how to recurse into the next directory, so the source tree must provide all these. For any given subdirectory that lacks a Kyuafile, you will have to provide it.

As an example, consider the case where we want to install /usr/tests/usr.bin/foo/bar/baz_test. If you follow the previous instructions, you would end up with Kyuafiles in /usr/tests/usr.bin/ and /usr/tests/usr.bin/foo/bar/, but not in /usr/tests/usr.bin/foo/. To install the missing file:

1. Edit usr.bin/foo/Makefile and tell it to recurse into tests (same as above).

2. Create tests/usr.bin/foo/tests/Makefile with the same procedure as above. The only difference is that you will have to tweak the .PATH: stanza to add additional :H modifiers for every other subdirectory you are traversing and adjust TESTSDIR accordingly:

   # $FreeBSD$
   
   # This isn't required if the test program component is under usr.bin/foo/tests ; see
   # https://svnweb.freebsd.org/base?view=revision&revision=289158
   #TESTSDIR=       ${TESTSBASE}/usr.bin/foo
   
   .PATH:          ${SRCTOP}/tests
   KYUAFILE=       yes
   
   .include <bsd.test.mk>

FAQ for ATF test programs

Where can I find sample code?

Refer to share/examples/tests/tests/atf/ for the sample C/shell test program code and the supporting build code. Refer to tests/examples/ for the sample python code.

Where are the APIs documented?

See atf-c-api(3), atf-c++-api and atf-sh-api(3) for details on the C, C++ and shell APIs respectively. These manual pages are part of the base system.

Python interface is not documented yet. Examples of the common use cases can be found in the tests/examples/ directory. ATF pytest interface sources can be found in tests/atf_python.

Do I need to remove any temporary files I create?

No, you don't if the files/directories are created relative to the starting directory.

"kyua test" creates a sandbox for test program when the test is executed, so each test case is executed as its own process with its own temporary subdirectory. Simply put, removing files relative to the test sandbox isn't required and is discouraged because it adds unnecessary complexity to test programs.

When do I use ATF_CHECK and when ATF_REQUIRE?

ATF_CHECK logs errors but does not abort the execution of the test program. ATF_REQUIRE logs errors in a similar way but immediately terminates the execution.

You can use this distinction in the following way: use ATF_REQUIRE to check the code that "prepares" your test case. Use ATF_CHECK to do the actual functionality tests once all the set up has been performed.

FAQ for GoogleTest test programs

Where can I find sample code?

Refer to contrib/googletest/googletest/samples for the sample test programs' code; refer to share/examples/tests/tests/googletest for the supporting build code.

FAQ for plain test programs

Where can I find sample code?

Refer to share/examples/tests/tests/plain for sample test program code and the supporting build code.

FAQ for TAP test programs

Where can I find sample code?

Refer to share/examples/tests/tests/tap for sample test program code and the supporting build code.

CategoryHowTo