Introduction
This page notes what things are currently completed/in-progress for testing in FreeBSD.
Eventually this page will also note coverage in source code as a meaningful rubric for progress.
ATF import status
- ATF 0.16 has been imported into FreeBSD CURRENT and apart from some minor patches, all of the needed support has been added to FreeBSD (gcooper, marcel).
- ATF and Kyua ports have been created (asomers, bdrewery).
- MFCing ATF/bmake support (bmake is required by ATF) has been requested by developers (bapt, et al) to help regression test components on 9-STABLE; MFCing to 8.x was considered a bad idea, but all involved parties (bapt, bdrewery, davide, gcooper) agreed that having ports of ATF/bmake for 8.x was a good idea.
Build infrastructure status
- Major build infrastructure pieces (atf.test.mk, bsd.test.mk) have been imported into FreeBSD in order to provide minimal overhead for developers when writing testcases, testing in-tree and out-of-tree. Minor usability improvements are queued for commit to FreeBSD current.
- A Test::Harness testcase shim (prove.test.mk) is queued for commit into FreeBSD.
- Official documentation (BSD.readme) is queued for commit.
- Automatic test "discovery"/handling logic is queued for commit.
- buildworld/installworld integration is queued for commit.
Building a test release package and being able to install it via bsdinstall is incomplete, but work has been done on GitHub to make this come true.
- Getting the test infrastructure to run on stable/8 and stable/9 is possible, and all of the necessary bits have been pushed back to stable/8 and stable/9.
Test Infrastructure Status
ATF
- Test fixtures (global cleanup/setup, per-testcase setup) need to be added to ATF to achieve feature parity with other test infrastructures (GNU libcheck, Python unittest testcases, Junit, etc).
As JulioMerino has noted on his GoogleCodeIn page for ATF/Kyua, ATF was just the starting point, and Kyua is the evolution of ATF. So, with that in mind, the work that's being done here is just to provide some degree of feature parity with NetBSD, etc, and as time progresses, FreeBSD will catch up with Kyua (which is more evolved than ATF) and will use the test infrastructure layering that Julio Merino has proposed.
RUMP
Rump is an extension to the NetBSD kernel that runs kernel code as a standalone user process for development and testing purposes. It is a great technology for isolating code and testing it using a whitebox method of testing, as opposed to having to use fuzzing techniques, failpoints, blackbox/graybox testing, etc, which ultimately allows developers and testers to write more code that isolates API calls and helps ensure that KPI contracts are sane without panicing a system or corrupting system state.
Testcase Status
ATF Testcases
Directory |
Repository |
Owner |
Status |
Notes |
bin/cat |
gcooper |
complete |
Ported from NetBSD. |
|
bin/date |
gcooper |
complete |
Ported from tools/regression. Research adding coverage from NetBSD. |
|
bin/dd |
gcooper |
complete |
Ported from NetBSD. Needs greater coverage/porting. Throws false positives. |
|
bin/expr |
gcooper |
complete |
Ported from NetBSD. |
|
bin/sh |
gcooper |
complete |
Ported from tools/regression. Need to get jilles@ to stop committing files to tools/regression :P. |
|
bin/sleep |
gcooper |
complete |
Ported from NetBSD. |
|
gnu/usr.bin/diff |
gcooper |
complete |
Ported from NetBSD. |
|
lib/libc |
gcooper |
in-progress |
Ported from NetBSD. Needs greater coverage/porting. Throws false positives. |
|
lib/libcrypt |
gcooper |
complete? |
|
|
lib/librt |
gcooper |
in-progress |
Ported from NetBSD. Needs greater coverage/porting. Throws false positives. |
|
lib/libthr |
gcooper |
in-progress |
Ported from NetBSD. Needs greater coverage/porting. Throws false positives. |
|
lib/msun |
gcooper |
in-progress |
Ported from NetBSD. Needs greater coverage/porting. Throws false positives. |
|
sbin/ifconfig |
gcooper |
complete |
Ported from NetBSD. Needs greater coverage/porting. Throws false positives. |
|
usr.bin/apply |
gcooper |
complete |
Ported from tools/regression. Research adding coverage from NetBSD. |
|
usr.bin/basename |
gcooper |
complete |
Ported from NetBSD. |
|
usr.bin/calendar |
gcooper |
complete |
Ported from tools/regression. |
|
usr.bin/comm |
gcooper |
complete |
Ported from tools/regression. |
|
usr.bin/cut |
gcooper |
complete |
Ported from tools/regression. |
|
usr.bin/dirname |
gcooper |
complete |
Ported from NetBSD. |
|
usr.bin/file2c |
gcooper |
complete |
Ported from tools/regression. |
|
usr.bin/grep |
gcooper |
complete |
Ported from NetBSD. |
|
usr.bin/gzip |
gcooper |
complete |
Ported from NetBSD. |
|
usr.bin/join |
gcooper |
complete |
Ported from tools/regression. |
|
usr.bin/jot |
gcooper |
complete |
Ported from tools/regression. |
|
usr.bin/m4 |
gcooper |
complete |
Ported from tools/regression. |
|
usr.bin/ncal |
gcooper |
complete |
Ported from tools/regression. |
|
usr.bin/printf |
gcooper |
complete |
Ported from tools/regression. |
|
usr.bin/sed |
gcooper |
complete |
Ported from tools/regression. |
|
usr.bin/tr |
gcooper |
complete |
Ported from tools/regression. |
|
usr.bin/uudecode |
gcooper |
complete |
Ported from tools/regression. |
|
usr.bin/uuencode |
gcooper |
complete |
Ported from tools/regression. |
|
usr.bin/xargs |
gcooper |
complete |
Ported from tools/regression. |
|
usr.bin/yacc |
gcooper |
complete |
Ported from tools/regression. |
|
usr.sbin/nmtree |
gcooper |
complete |
Ported from NetBSD. Adjusted slightly to match sha256sum string. |
|
usr.sbin/tcpdump |
gcooper |
complete |
Ported from NetBSD. Needs greater coverage/porting. Throws false positives. |
|
tests/contrib/atf |
gcooper |
complete |
Ported from NetBSD. All test issues have been pushed back via GoogleCodeIn page for the contrib/atf issues; the build issues and adjustments are all remaining local to FreeBSD for now.. |
Test::Harness Testcases
Directory |
Repository |
Owner |
Status |
Notes |
bin/mv |
gcooper |
in-progress |
From tools/regression. Convert to ATF. |
|
bin/pax |
gcooper |
in-progress |
From tools/regression. Convert to ATF. |
|
bin/pkill |
gcooper |
in-progress |
From tools/regression. Convert to ATF. |
|
bin/test |
gcooper |
in-progress |
From tools/regression. Convert to ATF. |
|
sbin/geom |
gcooper |
in-progress |
From tools/regression. Global tests, mirror, and raid3 testcases have been ported over. Convert to ATF. |
|
tests/sys/pjdfstest |
gcooper |
in-progress |
From tools/regression. Need to shim in support for ATF. |
|
usr.bin/lastcomm |
gcooper |
in-progress |
From tools/regression. Convert to ATF. |
|
usr.sbin/newsyslog |
gcooper |
in-progress |
From tools/regression. Convert to ATF. Need to investigate failures. |
|
usr.sbin/sa |
gcooper |
in-progress |
From tools/regression. Convert to ATF. Need to investigate failures. |