The FreeBSD Test Suite
Welcome to FreeBSD Test Suite project!
This project aims to equip FreeBSD with a comprehensive test suite that is easy to run out of the box and during the development of the system. The goal of the test suite is to assist both developers and users in assessing the quality of FreeBSD.
The runtime of choice to execute the test suite and generate reports is Kyua. The tests themselves can be written in a variety of languages and are not restricted to using a specific framework; the recommendation is to use the ATF libraries bundled into the base tree, but other supported options exist.
The following goals are specific to the FreeBSD test suite:
- Provide a comprehensive set of tests that can be used to assess the quality of the tree at any given point in time. The release engineering team should be able to confidently use the results of the test suite to vet a release (including alphas, betas and release candidates).
- Provide a comprehensive set of tests that exercises a release on a given hardware configuration. System administrators should be able to use this right out of the box after deploying FreeBSD to a new machine to ensure the release works as advertised on their particular hardware/software combination. Of course, the release engineering team should also use this on reference hardware.
- Provide a mechanism to implement tests for the kernel. Writing test code for the kernel should not be significantly more difficult than writing test code for user-space applications.
- Minimize divergence with NetBSD's test suite. Both systems should use the same testing tools and have a similar setup so that sharing test code is easy and possible.
The following goals are desirable for the FreeBSD test suite but they mostly describe work required on Kyua:
- Provide a per-release matrix in which the rows describe test cases and the columns describe every supported configuration (any desired combination of hardware architecture, compiler, etc.). Each cell shows the result of the latest execution, be it a test result if the build succeeded or a link to a log if the build failed.
- Provide historical evaluation of tests so that clicking on any cell of the matrix yields a list of results, their timestamps and any information associated to them. The idea is to permit automatically spotting flaky tests and to allow pinpoint which commit is at fault in the face of a test failure.
- Asynchronously notify developers (e.g. by sending an email to a mailing list) of test failures.
- The test suite exists in both HEAD and stable/10.
The building of the test suite is disabled by default at this point. Set WITH_TESTS=yes in src.conf(5) to enable.
The continuous testing cluster is up and running. Visit http://kyua1.nyi.freebsd.org/ for its front page.
See the Project planning spreadsheet for the roadmap of this project. The first sheet in the document contains an overview of the structure of the other sheets.
tests(7): User-level introduction to the FreeBSD test suite.
TestSuite/Structure: Description of the basic concepts involved in the test suite, its build infrastructure and the structure of the installed products.
TestSuite/DeveloperHowTo: Tutorial on how to add new tests and quick API reference.
TestSuite/Cluster: Structure of the FreeBSD-provided testing cluster and procedures to interact with it.
TestingFreeBSD: Old project page. Relevant information should be moved here.
Use the freebsd-testing mailing list for inquires about this project.
Kyua: Project page for the testing framework.
autotest: Source code of the automation scripts and configuration used in the test cluster.
Julio Merino (jmmv): Owner of this project, of these pages and author of Kyua. Also owner of the same project in NetBSD.
Rui Paulo (rpaulo): Reviewer of patches, ex-mentor of jmmv.
Garrett Cooper (ngie): Original driver of this project. Imported ATF into the src tree and has ported lots of tests into their own private repository. Working on integrating tests from tools/regression and NetBSD into FreeBSD.
Simon Gerraty (sjg) and others at Juniper: Ported many NetBSD tests to FreeBSD. Work on the share/mk infrastructure.
Alan Somers (asomers): Maintainer of the devel/kyua port.