Introduction

This page will go into some detail to discuss some of the design concerns and goals considered when evaluating several test infrastructures, as well as provide a comparison of several infrastructures used when evaluating the "best-fit" infrastructure for FreeBSD.

The Goals of Testing and Test Infrastructures

The goal of testing is simply put, to prove that code works according to a set of requirements and within a certain set of bounds.

Simply put, investing in testing is worthwhile a) to ensure that there aren't any bugs (or if there are any, find them before your customers/users do) and if properly written, b) testcases can reduce or possibly eliminate the need for manual testing, thus freeing up your [developers'] time to do more developing (and maybe even testing :)..!).

The golden rule for any script or infrastructure is simplification of a given, repeatable process.

The Goals of Testing on FreeBSD

The requirements for testing in FreeBSD is a bit different: FreeBSD's unique requirements with respect to how code can be freely distributed and having a sourcebase that builds and installs on multiple platforms from various host targets requires more than some testing infrastructures can permit for other technical and business reasons.

Furthermore, due to resource count on the FreeBSD project versus some other opensource or commercial projects (Linux, Microsoft), documentation and stability are a must for any consumers of the chosen infrastructure.

Testing in FreeBSD

As it stands, testing in FreeBSD today is a manual process. Some of the tests can be invoked via prove (perl Test::Harness wrapper), and others can be invoked with make, sh, etc. Most of the tests live under tools/regression, but there are also some testcases provided by third party packages that live elsewhere in the tree and have their own set of unique requirements (be they tools, how they're executed, etc).

The depth and breadth of the test suites vary depending on the component, but there are a hodgepodge of userland and kernel testcases that live in the FreeBSD source tree.

Problem Statement

Test content and structure in FreeBSD currently presents the following problems:

Test Infrastructure Comparison

When looking at testing infrastructures employed in FreeBSD (and in projects that consume FreeBSD), there are many flavors to choose from. Each infrastructure has their pluses and minuses, and I'll highlight some of the major points of some of the more major players I have dealt with when testing FreeBSD.

ATF

Pros:

Neutral:

Cons:

CTest

Pros:

Neutral:

Cons:

Libcheck

Pros:

Cons:

Test::Harness

Pros:

Cons:

Some other test infrastructures [for C] are noted here: http://check.sourceforge.net/doc/check_html/check_2.html#SEC2 . Some other non-C supporting test infrastructures that were completely off the table as far as technology requirements or licensing are concerned that are worth mentioning that the author has dealt with (if for nothing more than perspective) are as follows:

Conclusion

After some consideration, ATF was chosen as the defacto testing infrastructure for FreeBSD, as:

  1. It is a solid base.
  2. Could be extended upon.
  3. Pieces could be pollenated between the FreeBSD and NetBSD projects, thus creating a more solid, tested BSD family.

TestingFreeBSDGoalsAndComparison (last edited 2013-05-18 18:52:48 by GarrettCooper)