Integrating TET

FreeBSD Testing Today

We are missing a capable test framework that is part of our main source tree.

This means that building-in testing when working on FreeBSD's base system requires extra steps, and so is harder than should be.

We currently keep our unit tests and regression test cases under /usr/src/tools/regression/. These tests use ad-hoc ways to build and execute their test cases. Test case reporting is also not standardized (though some tests use the Perl 'Test Anything Protocol', see below). Running these tests in an automated way (a test 'tinderbox') is not always possible.

At this point of time we do not archive test logs. Even if we did so, analysis of historical test data would be tedious due to the ad-hoc nature of the test reports.

The desirables from a FreeBSD test framework

What is TET

TET is a test execution framework from the Open Group. Its features include:

TET: Pros

TET: Cons

TET makes a set of assumptions about its execution environment that make integrating it in a BSD-compatible way into the base system complex.

  1. Source layout:

    1. We would want to keep imported TET and TET-based test suites under /usr/src/contrib/, as hier(7) dictates.

    2. We would want to control test suite builds using make-based infrastructure under say /usr/src/test/, in a manner similar to the way other contributed software is built in FreeBSD, see src.conf(5), and build(7). Regular TET assumes that tcc (the test case controller) would control builds.

  2. Cross-architecture builds:

    1. We would want our test suites to be cross-buildable, like the rest of /usr/src/.

    2. This implies that test suite binaries would get built under /usr/obj/${ARCH}/.

  3. Test suite 'Installation':

    1. The TET API's support files (tcm.o, libapi.a, etc.) would need to be installed under some standard location (say, ${DESTDIR}/usr/test/tet3/?).

    2. We would want to 'install' test suite binaries under a location like ${DESTDIR}/usr/test/. This installation process should be controllable by the knobs in src.conf(5).

    3. The TET configuration files under ${DESTDIR}/usr/test/ would need to be configured to only run tests from this location (like a binary test suite).

  4. TET Configuration:

    1. TET itself can be built in many flavours, e.g., supporting distributed testing or in non-distributed form, supporting thread-aware APIs or not, or built with shared-libraries for the API. We'll need to handle test suites that require one or the other form appropriately.
    2. 'Distributed TET' supports multi-machine operation, but requires additional configuration on the base system (entries in /etc/services, a startup script for daemons in /etc/rc.d/ etc.).

  5. Running the tests:

    1. We would want the test journal from a test run to be placed under /var/ by default (say /var/tet/journals/?). This would allow the /usr/test/ hierarchy to be read-only and sharable across multiple systems.

    2. Handling tcc -c (clean) is tricky. For test suites under ${DESTDIR}/usr/test/ we only want temporary files to be removed.

Stock TET assumes that the directory named by environment variable ${TET_ROOT} contains all the control files needed for a TET run. Test suites are assumed be located under ${TET_ROOT}, and tcc, (the test case controller program) takes charge of all steps of test building, execution and cleaning. Some, but not all, of these assumptions are configurable using the command-line and configuration files.

We'll need FreeBSD-specific documentation to cover the places where we diverge from TET's documented defaults.

TET Compared to other Test Tools

STAF/STAX:

TAP:

Experimental Source Tree

TetIntegration (last edited 2008-06-17 21:38:20 by localhost)