NOTE This page is obsolete; please see TestSuite for the current FreeBSD project test effort.
This wiki page will (or link to sections that) describe the following items:
- The goals of testing and test infrastructures and how it relates to FreeBSD.
- Note how testing has been traditionally done in FreeBSD.
- Note the problems faced as a developer or tester when writing new testcases, and the problems faced as a release engineer or tester when executing existing testcases.
Then the wiki page will proceed to note:
- The work taking place to bring ATF to FreeBSD
- Why you as a developer/tester should use ATF.
- Some of the non-ATF items that will be available after the work.
- Known Issues with ATF/non-ATF components added.
The wiki page will finally provide some examples on how to use ATF and how to invoke the new test hooks when testing FreeBSD.
Goals of Testing and Test Infrastructure Comparison
More information about this effort can be found here.
Status of Work
More information about the effort of porting testcases over to ATF, integrating testcases into the build and install, etc can be found here.
More details on how to write example testcases can be found here.
The layout of test code within the tree was agreed to be as follows:
- All test directories must contain a Makefile which .includes the appropriate *.test.mk (atf.test.mk, prove.test.mk, etc) make snippet.
- Ideally, tests should be localized next to features as much as possible, e.g. $SRCTOP/lib/libc/tests should be where the testcases for libc should live.
- Testcases for third-party components like atf should exist under tests/ e.g. $SRCTOP/tests/contrib/atf/ .
- Testcases for kernel components should go under a logical directory/functional group, e.g. pjdfstest should go under $SRCTOP/tests/sys/pjdfstest/ .
In part, because of how ATF is designed and the Kyua project has evolved, it's not the end-all-be-all tool for executing testcases when compared with several other options; it is however an evolving infrastructure that combined with Kyua will serve to become a solid basis for testing on FreeBSD (and hopefully in related projects).
Some known caveats with ATF are as follows:
- In general:
- cleanup fixture support only exists per testcase. I'm working on adding global setup and cleanup as well as per-testcase setup fixtures for compatibility with libcheck and other frameworks to atf-c, atf-c++, and atf-sh.
- There doesn't seem to be a clean way to pass in arguments to testcases on the command line, e.g. tweak the execution to run this test as opposed to this test via getopt(3) [but I could have missed this feature somehow].
- The ATF_REQUIRE* macros don't have ATF_CHECK* analogs like in the atf-c API (inconsistency).
- Exception test support is limited; in particular the exception types must be classes as opposed to base types (ints, floats, etc).
- atf-sh does not work with functions; you must provide it externally visible commands to evaluate.
- If in the event that a test program is missing, not installed, or outputs something via standard out that's unexpected, it will confuse atf-report.
Empty/malformed/missing Atffiles result in confusing output, like atf-run: ERROR: Cannot open Atffile. This wouldn't be so bad if it noted which Atffile caused the issue, and where in the Atffile the error occurred.
- atf.test.mk / bsd.test.mk:
- You can't have testcases that live in non-leaf subdirectories, e.g. if I had a directory structure a/b/c/, you cannot have a test d that lives under a/b/, etc. This is probably an issue with bsd.progs.mk or missing features from bsd.subdir.mk; more investigation is required.
The freebsd-testing mailing list is where interested parties can get in touch to obtain more information about testing on FreeBSD, including topics like testing with ATF, using the test infrastructure in FreeBSD, collaborating with other developers in order to get test code committed to FreeBSD, some basic testing best practices, other test-related topics, etc.
- Google -- sponsored Giorgos's work.
- EMC/Isilon -- sponsored my work.
- Juniper -- sponsored Marcel and Simon's work.
- Julio Merino -- for developing ATF and Kyua, and for working with both Giorgos and myself to help port ATF to FreeBSD.
- Giorgos Keramidas -- for the initial porting work and for working with me to complete the first major hurdles of porting ATF to FreeBSD.
- Marcel and Simon -- provided much appreciated review, code ushering, and brainstorming with the build infrastructure pieces.