Introduction

This page will go into some detail with examples of how one might write testcases in the test infrastructure formats supported by FreeBSD, which currently is:

ATF Examples

There are a handful of github.com/yaneurabeya/freebsd/tree/master/share//atf/tests that demonstrate how the infrastructure works. These examples demonstrate how to write ATF-based testcases.

These examples touch on the important points that other documentation didn't go over in great detail, or merely highlight how one would convert standalone assertions and test code to ATF. As always, please see the relevant ATF documentation [2], [3], [4] for more details on how you should formulate testcases, as they go into the general flow and organization in a more in-depth manner than I do here.

example directory

This might be a lowlight, but it's also important to note how one might layout the examples directory with the C, C++ and bourne shell example testcases: 

# $FreeBSD$

.include <bsd.own.mk>

TESTSDIR=       ${TESTSBASE}/examples

ATF_TESTS_SUBDIRS+= c_test
ATF_TESTS_SUBDIRS+= cxx_test
ATF_TESTS_SUBDIRS+= sh_test

.include <atf.test.mk>

C example

The github.com/yaneurabeya/freebsd/tree/master/share//atf/tests/c_test/c_test.c demonstrates how one might write a test application to test out an API that implements the Pythagorean Theorem:

which as one expects would return the square root of a2 + b2, and rejects all values of a <= 0 and b <= 0, returning -1 and setting errno to ERANGE.

The Test Application

Enumerating the requirements the following cases need to be tested:

Case 1

The first positive case is square root of 32 + 42 == square root of 52 == 5, which is equivalent to the following API call:

In order to verify that this check is sane, we use the ATF_CHECK_EQ macro, like so:

The first parameter being the expression being evaluated, and the second parameter being the value to test for.

Case 2

The second positive case ensures that the square root of 12 + 12 == square root of 2. Seeing how there can be rounding errors in the math, we can't test that the exact value returned is sane. So instead we verify that the values are within 1% of one another (this is just a random confidence value, for the sake of illustrating how this would be done). This is equivalent to the following expression:

or simply,

In order to verify that this check is indeed sane, we use the ATF_CHECK macro, like so:

The first (and only) parameter is the expression that needs to be tested for boolean truth.

Case 3

The final, negative case helps ensure that if one of the values is set to a value less than or equal to 0, that the returned value will be -1 (although the Pythagorean theorem doesn't exclude this case, our program only allows positive floating point values). This is equivalent to the following

In order to verify that this is indeed sane, we use the ATF_CHECK_ERRNO macro, like so:

The first parameter is the value errno is set to to test for (in this case errno == ERANGE), and the second parameter is the equality that must be tested to ensure that the API indeed meet our desired error state.

The Makefile

The Makefile needed to build our c_test application is as follows: 

# $FreeBSD$

.include <bsd.own.mk>

TESTSDIR=       ${TESTSBASE}/share/examples/atf/c_test

TESTS_C=        c_test

LDADD+=         -lm

.include <atf.test.mk>

C++ example

The github.com/yaneurabeya/freebsd/tree/master/share//atf/tests/cxx_test/cxx_test.c demonstrates how one might write a test application to test out an API that computes factorials:

which computes n!, where n must be greater than 0 or equal to 0. When n < 0, an exception of type negative_number is returned.

The Test Application

Enumerating the requirements the following cases need to be tested:

Case 1

By definition, 0! == 1. Thus we need to test the following case:

In order to verify that this check is sane, we use the ATF_REQUIRE_EQ macro, like so:

The first parameter being the expression being evaluated, and the second parameter being the value to test for.

Case 2

By definition, 1! == 1. Thus we need to test the following case:

In order to verify that this check is sane, we use the ATF_REQUIRE_EQ macro, like so:

The first parameter being the expression being evaluated, and the second parameter being the value to test for.

Case 3

By definition, 5! == 120. Thus we need to test the following case:

In order to verify that this check is sane, we use the ATF_REQUIRE_EQ macro, like so:

The first parameter being the expression being evaluated, and the second parameter being the value to test for.

Case 4

The final case to test for would be to ensure that any negative values passed in throw a sample_test::negative_number exception. Thus we need to test the following case: 

passed = 0;
try {
    sample_test::factorial(-1);
} catch (sample_test::negative_number) {
    passed = 1;
}
assert(passed);

In order to verify that this check is sane, we use the ATF_REQUIRE_THROW macro, like so:

The first parameter is the exception that will be thrown, and the second parameter is the expression that will be tested.

The Makefile

The Makefile needed to build our cxx_test application is as follows: 

# $FreeBSD$

.include <bsd.own.mk>

TESTSDIR=       ${TESTSBASE}/share/examples/atf/cxx_test

TESTS_CXX=      cxx_test

.include <atf.test.mk>

shell example

The github.com/yaneurabeya/freebsd/blob/atf-tools-regression-convert/share/atf/tests/sh_test/sh_test.sh demonstrates how one might write a test application to test out a script called github.com/yaneurabeya/freebsd/blob/atf-tools-regression-convert/share//atf/tests/sh_test/sigma_sum.sh that implements the simple Sigma Sum Series, and is executed like so:

The Test Application

Enumerating the requirements the following cases need to be tested:

Case 1

The first case to test for is 1 + 2 ... 100, which by definition is 5050. This we'd need to test the following case: 

 output=$(sigma_sum 1 100)
 if [ $? -ne 0 -o "$output" != 5050 ]
 then
     exit 1
 fi

This can be done more eloquently with atf_check, like so: 

err_str="n <= i"
atf_check -s exit:0 -o "match:5050" "$(atf_get_srcdir)/sigma_sum" 1 100

The -s argument tests that the script exits with 0, the -o argument ensures that the string "5050" is matched exactly on standard out, and the final parameter is the command that will be executed.

Case 2

The second case to test for is to ensure that if i == -1 and n == -1 the script exits with 1 and prints out "n <= i" if n <= i on standard error. This we'd need to test the following case: 

 err_str="n <= i"
 output=$(sigma_sum -1 -1 2>&1 >/dev/null)
 if [ $? -ne 1 -o "$output" != "$err_str" ]
 then
     exit 1
 fi

This can be done with atf_check, like so: 

err_str="n <= i"
atf_check -s exit:1 -e "match:$err_str" "$(atf_get_srcdir)/sigma_sum" -1 -1

The -e argument tests that the script exits with 1, the -e argument ensures that the string "n <= i" is matched exactly on standard error, and the final parameter is the command that will be executed.

Case 3

 err_str="n <= i"
 output=$(sigma_sum 0 -1 2>&1 >/dev/null)
 if [ $? -ne 1 -o "$output" != "$err_str" ]
 then
     exit 1
 fi

This can be done with atf_check, like so: 

err_str="n <= i"
atf_check -s exit:1 -e "match:$err_str" "$(atf_get_srcdir)/sigma_sum" 0 -1

The -s argument tests that the script exits with 1, the -e argument ensures that the string "n <= i" is matched exactly on standard error, and the final parameter is the command that will be executed.

Case 4

The final case to test for would be to ensure that any negative values passed in throw a sample_test::negative_number exception. Thus we need to test the following case: 

 err_str="the efficient method only works with natural (>0) numbers"
 output=$(sigma_sum -1 0 2>&1 >/dev/null)
 if [ $? -ne 2 -o "$output" != "$err_str" ]
 then
     exit 1
 fi

This can be done with atf_check, like so: 

err_str2="the efficient method only works with natural \(>0\) numbers"
atf_check -s exit:2 -e "match:$err_str2" "$(atf_get_srcdir)/sigma_sum" -1 0

The -s argument tests that the script exits with 2, the -e argument ensures that the string "the efficient method only works with natural \(>0\) numbers" is matched exactly on standard error (please note that the characters in the string tested are escaped as they're meta characters in regular expression), and the final parameter is the command that will be executed.

The Makefile

The Makefile needed to build our sh_test application is as follows: 

# $FreeBSD$

.include <bsd.own.mk>

TESTSDIR=       ${TESTSBASE}/share/examples/atf/sh_test

TESTS_SH=   sh_test

BINDIR=         ${TESTSDIR}

SCRIPTS=        sigma_sum

.include <atf.test.mk>

Compiling and running the examples

Compiling and running the examples is trivial; this section will describe how one can do it from the source tree and how one can do it from the installed tests directory.

From the Source Tree

The method of doing this from the source tree is as follows:

1. First you must install the testcases 

cd tests/examples
make obj
make depend
make all install

2. Then you must run make test in order to run a wrapped version of atf-run:

make test

Example: 

# cd tests/examples
# make obj
# make depend
# make all install
/usr/obj/usr/src/tests/examples created for /usr/src/tests/examples
===> c_test (obj)
/usr/obj/usr/src/tests/examples/c_test created for /usr/src/tests/examples/c_test
===> cxx_test (obj)
/usr/obj/usr/src/tests/examples/cxx_test created for /usr/src/tests/examples/cxx_test
===> sh_test (obj)
/usr/obj/usr/src/tests/examples/sh_test created for /usr/src/tests/examples/sh_test

...

===> cxx_test (install)
install -o root  -g wheel -m 444  Atffile  /usr/tests/examples/cxx_test/Atffile
install -s -o root -g wheel  -m 555  cxx_test  /usr/tests/examples/cxx_test
===> sh_test (install)
install -o root  -g wheel -m 444  Atffile  /usr/tests/examples/sh_test/Atffile
install -o root  -g wheel  -m 555  sh_test  /usr/tests/examples/sh_test/sh_test
# make test
*** WARNING: make test is experimental
***
*** Using this test does not preclude you from running the tests
*** installed in /usr/tests.  This test run may raise false
*** positives and/or false negatives.

c_test/c_test (1/3): 3 test cases
    pythagorean_01: [0.015521s] Passed.
    pythagorean_02: [0.015134s] Passed.
    pythagorean_03: [0.014249s] Passed.
[0.046747s]

cxx_test/cxx_test (2/3): 4 test cases
    factorial_0: [0.017003s] Passed.
    factorial_1: [0.020745s] Passed.
    factorial_5: [0.016368s] Passed.
    factorial_negative: [0.015882s] Passed.
[0.072463s]

sh_test/sh_test (3/3): 4 test cases
    sigma_sum_01: [0.124699s] Passed.
    sigma_sum_02: [0.117029s] Passed.
    sigma_sum_03: [0.121133s] Passed.
    sigma_sum_04: [0.128607s] Passed.
[0.493479s]

Summary for 3 test programs:
    11 passed test cases.
    0 failed test cases.
    0 expected failed test cases.
    0 skipped test cases.

*** The verbatim output of atf-run has been saved to /usr/obj/usr/src/tests/examples/obj/atf-run.log
*** Once again, note that make test is unsupported.

From the Tests Directory

The method of doing this from the tests directory is similar to doing it from the source tree:

1. First you must install the testcases 

cd tests/examples
make obj
make depend
make all install

2. Then you must go to the /usr/tests/examples directory and run atf-run | atf-report manually: 

cd /usr/tests/examples # Assumption is that you have TESTSBASE set to /usr/tests -- which is the default.
atf-run | atf-report

Example: 

# cd tests/examples
# make obj
# make depend
# make all install

...

===> cxx_test (install)
install -o root  -g wheel -m 444  Atffile  /usr/tests/examples/cxx_test/Atffile
install -s -o root -g wheel  -m 555  cxx_test  /usr/tests/examples/cxx_test
===> sh_test (install)
install -o root  -g wheel -m 444  Atffile  /usr/tests/examples/sh_test/Atffile
install -o root  -g wheel  -m 555  sh_test  /usr/tests/examples/sh_test/sh_test
# cd /usr/tests/share/examples/atf/
# atf-run | atf-report
c_test/c_test (1/3): 3 test cases
    pythagorean_01: [0.014689s] Passed.
    pythagorean_02: [0.015899s] Passed.
    pythagorean_03: [0.015703s] Passed.
[0.048616s]

cxx_test/cxx_test (2/3): 4 test cases
    factorial_0: [0.017060s] Passed.
    factorial_1: [0.016793s] Passed.
    factorial_5: [0.016403s] Passed.
    factorial_negative: [0.016488s] Passed.
[0.069197s]

sh_test/sh_test (3/3): 4 test cases
    sigma_sum_01: [0.136310s] Passed.
    sigma_sum_02: [0.136014s] Passed.
    sigma_sum_03: [0.136203s] Passed.
    sigma_sum_04: [0.139116s] Passed.
[0.550156s]

Summary for 3 test programs:
    11 passed test cases.
    0 failed test cases.
    0 expected failed test cases.
    0 skipped test cases.

TestingFreeBSDExamples (last edited 2020-10-22T22:41:14+0000 by SashaVigole)