Introduction
This page will go into some detail with examples of how one might write testcases in the test infrastructure formats supported by FreeBSD, which currently is:
- ATF (defacto format)
- Test::Harness (deprecated format)
ATF Examples
There are a handful of github.com/yaneurabeya/freebsd/tree/master/share//atf/tests that demonstrate how the infrastructure works. These examples demonstrate how to write ATF-based testcases.
These examples touch on the important points that other documentation didn't go over in great detail, or merely highlight how one would convert standalone assertions and test code to ATF. As always, please see the relevant ATF documentation [2], [3], [4] for more details on how you should formulate testcases, as they go into the general flow and organization in a more in-depth manner than I do here.
example directory
This might be a lowlight, but it's also important to note how one might layout the examples directory with the C, C++ and bourne shell example testcases:
# $FreeBSD$ .include <bsd.own.mk> TESTSDIR= ${TESTSBASE}/examples ATF_TESTS_SUBDIRS+= c_test ATF_TESTS_SUBDIRS+= cxx_test ATF_TESTS_SUBDIRS+= sh_test .include <atf.test.mk>
.include <bsd.own.mk> provides the definition of ${TESTSBASE}.
TESTSDIR describes where the testcase needs to be installed (in this case ${TESTSBASE}/examples, which is generally /usr/tests/examples)
- ATF_TESTS_SUBDIRS describes what subdirectories should be traversed when building/installing looking for ATF testcases, in this case c_test, cxx_test, and sh_test.
.include <atf.test.mk> generates the necessary subdir rules from <bsd.subdir.mk>
C example
The github.com/yaneurabeya/freebsd/tree/master/share//atf/tests/c_test/c_test.c demonstrates how one might write a test application to test out an API that implements the Pythagorean Theorem:
double pythagorean_theorem(double a, double b)
which as one expects would return the square root of a2 + b2, and rejects all values of a <= 0 and b <= 0, returning -1 and setting errno to ERANGE.
The Test Application
Enumerating the requirements the following cases need to be tested:
Case 1
The first positive case is square root of 32 + 42 == square root of 52 == 5, which is equivalent to the following API call:
pythagorean_theorem(3.0, 4.0) == 5.0
In order to verify that this check is sane, we use the ATF_CHECK_EQ macro, like so:
ATF_CHECK_EQ(pythagorean_theorem(3.0, 4.0), 5.0);
The first parameter being the expression being evaluated, and the second parameter being the value to test for.
Case 2
The second positive case ensures that the square root of 12 + 12 == square root of 2. Seeing how there can be rounding errors in the math, we can't test that the exact value returned is sane. So instead we verify that the values are within 1% of one another (this is just a random confidence value, for the sake of illustrating how this would be done). This is equivalent to the following expression:
-0.01 <= pythagorean_theorem(1.0, 1.0) - sqrt(2.0) <= 0.01
or simply,
fabs(pythagorean_theorem(1.0, 1.0) - sqrt(2.0)) < 0.01
In order to verify that this check is indeed sane, we use the ATF_CHECK macro, like so:
ATF_CHECK(fabs(pythagorean_theorem(1.0, 1.0) - sqrt(2.0)) < 0.01);
The first (and only) parameter is the expression that needs to be tested for boolean truth.
Case 3
The final, negative case helps ensure that if one of the values is set to a value less than or equal to 0, that the returned value will be -1 (although the Pythagorean theorem doesn't exclude this case, our program only allows positive floating point values). This is equivalent to the following
pythagorean_theorem(1.0, -1.0) == -1.0 and errno == ERANGE is set.
In order to verify that this is indeed sane, we use the ATF_CHECK_ERRNO macro, like so:
ATF_CHECK_ERRNO(ERANGE, pythagorean_theorem(1.0, -1.0) == -1.0);
The first parameter is the value errno is set to to test for (in this case errno == ERANGE), and the second parameter is the equality that must be tested to ensure that the API indeed meet our desired error state.
The Makefile
The Makefile needed to build our c_test application is as follows:
# $FreeBSD$ .include <bsd.own.mk> TESTSDIR= ${TESTSBASE}/share/examples/atf/c_test TESTS_C= c_test LDADD+= -lm .include <atf.test.mk>
.include <bsd.own.mk> provides the definition of ${TESTSBASE}.
TESTSDIR describes where the testcase needs to be installed (in this case ${TESTSBASE}/share/examples/atf/c_test, which is generally /usr/tests/share/examples/atf/c_test)
- TESTS_C describes the name of the test application that will be installed (think of it like PROG with bsd.prog[s].mk).
- The SRCS variable assumes that there's a c_test.c file in the directory.
- LDADD is needed because c_test uses sqrt(3) from libm.so.
.include <atf.test.mk> generates the necessary build and tests rules from <bsd.test.mk> and <bsd.progs.mk>
C++ example
The github.com/yaneurabeya/freebsd/tree/master/share//atf/tests/cxx_test/cxx_test.c demonstrates how one might write a test application to test out an API that computes factorials:
int sample_test::factorial(int n)
which computes n!, where n must be greater than 0 or equal to 0. When n < 0, an exception of type negative_number is returned.
The Test Application
Enumerating the requirements the following cases need to be tested:
Case 1
By definition, 0! == 1. Thus we need to test the following case:
sample_test::factorial(0) == 1
In order to verify that this check is sane, we use the ATF_REQUIRE_EQ macro, like so:
ATF_REQUIRE_EQ(sample_test::factorial(0), 1)
The first parameter being the expression being evaluated, and the second parameter being the value to test for.
Case 2
By definition, 1! == 1. Thus we need to test the following case:
sample_test::factorial(1) == 1
In order to verify that this check is sane, we use the ATF_REQUIRE_EQ macro, like so:
ATF_REQUIRE_EQ(sample_test::factorial(1), 1)
The first parameter being the expression being evaluated, and the second parameter being the value to test for.
Case 3
By definition, 5! == 120. Thus we need to test the following case:
sample_test::factorial(5) == 120
In order to verify that this check is sane, we use the ATF_REQUIRE_EQ macro, like so:
ATF_REQUIRE_EQ(sample_test::factorial(5), 120)
The first parameter being the expression being evaluated, and the second parameter being the value to test for.
Case 4
The final case to test for would be to ensure that any negative values passed in throw a sample_test::negative_number exception. Thus we need to test the following case:
passed = 0; try { sample_test::factorial(-1); } catch (sample_test::negative_number) { passed = 1; } assert(passed);
In order to verify that this check is sane, we use the ATF_REQUIRE_THROW macro, like so:
ATF_REQUIRE_THROW(sample_test::negative_number, sample_test::factorial(-1));
The first parameter is the exception that will be thrown, and the second parameter is the expression that will be tested.
The Makefile
The Makefile needed to build our cxx_test application is as follows:
# $FreeBSD$ .include <bsd.own.mk> TESTSDIR= ${TESTSBASE}/share/examples/atf/cxx_test TESTS_CXX= cxx_test .include <atf.test.mk>
.include <bsd.own.mk> provides the definition of ${TESTSBASE}.
TESTSDIR describes where the testcase needs to be installed (in this case ${TESTSBASE}/examples/atf/cxx_test, which is generally /usr/tests/share/examples/atf/cxx_test)
- TESTS_CXX describes the name of the test application that will be installed (think of it like PROG_CXX with bsd.prog[s].mk).
- The SRCS variable assumes that there's a cxx_test.cc file in the directory.
.include <atf.test.mk> generates the necessary build and tests rules from <bsd.test.mk> and <bsd.progs.mk>
shell example
The github.com/yaneurabeya/freebsd/blob/atf-tools-regression-convert/share/atf/tests/sh_test/sh_test.sh demonstrates how one might write a test application to test out a script called github.com/yaneurabeya/freebsd/blob/atf-tools-regression-convert/share//atf/tests/sh_test/sigma_sum.sh that implements the simple Sigma Sum Series, and is executed like so:
sigma_sum i n
The script exits with 1 and prints out "n <= i" if n <= i on standard error.
The script exits with 2 and prints out "the efficient method only works with natural (>0) numbers" if i <= 0 or n <= 0 on standard error.
- Otherwise, the script exists with 0 and prints out the result of the sigma summation from i to n on standard out.
The Test Application
Enumerating the requirements the following cases need to be tested:
Case 1
The first case to test for is 1 + 2 ... 100, which by definition is 5050. This we'd need to test the following case:
output=$(sigma_sum 1 100) if [ $? -ne 0 -o "$output" != 5050 ] then exit 1 fi
This can be done more eloquently with atf_check, like so:
err_str="n <= i" atf_check -s exit:0 -o "match:5050" "$(atf_get_srcdir)/sigma_sum" 1 100
The -s argument tests that the script exits with 0, the -o argument ensures that the string "5050" is matched exactly on standard out, and the final parameter is the command that will be executed.
Case 2
The second case to test for is to ensure that if i == -1 and n == -1 the script exits with 1 and prints out "n <= i" if n <= i on standard error. This we'd need to test the following case:
err_str="n <= i" output=$(sigma_sum -1 -1 2>&1 >/dev/null) if [ $? -ne 1 -o "$output" != "$err_str" ] then exit 1 fi
This can be done with atf_check, like so:
err_str="n <= i" atf_check -s exit:1 -e "match:$err_str" "$(atf_get_srcdir)/sigma_sum" -1 -1
The -e argument tests that the script exits with 1, the -e argument ensures that the string "n <= i" is matched exactly on standard error, and the final parameter is the command that will be executed.
Case 3
err_str="n <= i" output=$(sigma_sum 0 -1 2>&1 >/dev/null) if [ $? -ne 1 -o "$output" != "$err_str" ] then exit 1 fi
This can be done with atf_check, like so:
err_str="n <= i" atf_check -s exit:1 -e "match:$err_str" "$(atf_get_srcdir)/sigma_sum" 0 -1
The -s argument tests that the script exits with 1, the -e argument ensures that the string "n <= i" is matched exactly on standard error, and the final parameter is the command that will be executed.
Case 4
The final case to test for would be to ensure that any negative values passed in throw a sample_test::negative_number exception. Thus we need to test the following case:
err_str="the efficient method only works with natural (>0) numbers" output=$(sigma_sum -1 0 2>&1 >/dev/null) if [ $? -ne 2 -o "$output" != "$err_str" ] then exit 1 fi
This can be done with atf_check, like so:
err_str2="the efficient method only works with natural \(>0\) numbers" atf_check -s exit:2 -e "match:$err_str2" "$(atf_get_srcdir)/sigma_sum" -1 0
The -s argument tests that the script exits with 2, the -e argument ensures that the string "the efficient method only works with natural \(>0\) numbers" is matched exactly on standard error (please note that the characters in the string tested are escaped as they're meta characters in regular expression), and the final parameter is the command that will be executed.
The Makefile
The Makefile needed to build our sh_test application is as follows:
# $FreeBSD$ .include <bsd.own.mk> TESTSDIR= ${TESTSBASE}/share/examples/atf/sh_test TESTS_SH= sh_test BINDIR= ${TESTSDIR} SCRIPTS= sigma_sum .include <atf.test.mk>
.include <bsd.own.mk> provides the definition of ${TESTSBASE}.
TESTSDIR describes where the testcase needs to be installed (in this case ${TESTSBASE}/examples/atf/c_test, which is generally /usr/tests/share/examples/atf/c_test)
- TESTS_C describes the name of the test application that will be installed (think of it like PROG with bsd.prog[s].mk).
- The SRCS variable assumes that there's a sh_test.sh file in the directory.
- BINDDIR tells bsd.prog.mk where sigma_sum will be installed (in this case, ${TESTSDIR}).
- SCRIPTS tells bsd.prog.mk what script will be installed in ${BINDIR} (in this case, sigma_sum).
.include <atf.test.mk> generates the necessary build and tests rules from <bsd.test.mk> and <bsd.progs.mk>
Compiling and running the examples
Compiling and running the examples is trivial; this section will describe how one can do it from the source tree and how one can do it from the installed tests directory.
From the Source Tree
The method of doing this from the source tree is as follows:
1. First you must install the testcases
cd tests/examples make obj make depend make all install
2. Then you must run make test in order to run a wrapped version of atf-run:
make test
Example:
# cd tests/examples # make obj # make depend # make all install /usr/obj/usr/src/tests/examples created for /usr/src/tests/examples ===> c_test (obj) /usr/obj/usr/src/tests/examples/c_test created for /usr/src/tests/examples/c_test ===> cxx_test (obj) /usr/obj/usr/src/tests/examples/cxx_test created for /usr/src/tests/examples/cxx_test ===> sh_test (obj) /usr/obj/usr/src/tests/examples/sh_test created for /usr/src/tests/examples/sh_test ... ===> cxx_test (install) install -o root -g wheel -m 444 Atffile /usr/tests/examples/cxx_test/Atffile install -s -o root -g wheel -m 555 cxx_test /usr/tests/examples/cxx_test ===> sh_test (install) install -o root -g wheel -m 444 Atffile /usr/tests/examples/sh_test/Atffile install -o root -g wheel -m 555 sh_test /usr/tests/examples/sh_test/sh_test # make test *** WARNING: make test is experimental *** *** Using this test does not preclude you from running the tests *** installed in /usr/tests. This test run may raise false *** positives and/or false negatives. c_test/c_test (1/3): 3 test cases pythagorean_01: [0.015521s] Passed. pythagorean_02: [0.015134s] Passed. pythagorean_03: [0.014249s] Passed. [0.046747s] cxx_test/cxx_test (2/3): 4 test cases factorial_0: [0.017003s] Passed. factorial_1: [0.020745s] Passed. factorial_5: [0.016368s] Passed. factorial_negative: [0.015882s] Passed. [0.072463s] sh_test/sh_test (3/3): 4 test cases sigma_sum_01: [0.124699s] Passed. sigma_sum_02: [0.117029s] Passed. sigma_sum_03: [0.121133s] Passed. sigma_sum_04: [0.128607s] Passed. [0.493479s] Summary for 3 test programs: 11 passed test cases. 0 failed test cases. 0 expected failed test cases. 0 skipped test cases. *** The verbatim output of atf-run has been saved to /usr/obj/usr/src/tests/examples/obj/atf-run.log *** Once again, note that make test is unsupported.
From the Tests Directory
The method of doing this from the tests directory is similar to doing it from the source tree:
1. First you must install the testcases
cd tests/examples make obj make depend make all install
2. Then you must go to the /usr/tests/examples directory and run atf-run | atf-report manually:
cd /usr/tests/examples # Assumption is that you have TESTSBASE set to /usr/tests -- which is the default. atf-run | atf-report
Example:
# cd tests/examples # make obj # make depend # make all install ... ===> cxx_test (install) install -o root -g wheel -m 444 Atffile /usr/tests/examples/cxx_test/Atffile install -s -o root -g wheel -m 555 cxx_test /usr/tests/examples/cxx_test ===> sh_test (install) install -o root -g wheel -m 444 Atffile /usr/tests/examples/sh_test/Atffile install -o root -g wheel -m 555 sh_test /usr/tests/examples/sh_test/sh_test # cd /usr/tests/share/examples/atf/ # atf-run | atf-report c_test/c_test (1/3): 3 test cases pythagorean_01: [0.014689s] Passed. pythagorean_02: [0.015899s] Passed. pythagorean_03: [0.015703s] Passed. [0.048616s] cxx_test/cxx_test (2/3): 4 test cases factorial_0: [0.017060s] Passed. factorial_1: [0.016793s] Passed. factorial_5: [0.016403s] Passed. factorial_negative: [0.016488s] Passed. [0.069197s] sh_test/sh_test (3/3): 4 test cases sigma_sum_01: [0.136310s] Passed. sigma_sum_02: [0.136014s] Passed. sigma_sum_03: [0.136203s] Passed. sigma_sum_04: [0.139116s] Passed. [0.550156s] Summary for 3 test programs: 11 passed test cases. 0 failed test cases. 0 expected failed test cases. 0 skipped test cases.