Smoke testing of all base utilities

• Student: Shivansh Rai (<shivansh@freebsd.org>)

• Mentors:

  • Alan Somers (<asomers@freebsd.org>)

  • Brooks Davis (<brooks@freebsd.org>)

  • Ngie Cooper (<ngie@freebsd.org>)

Overview

Smoke testing is a set of light tests which are done for checking basic functionalities of a software to ascertain if the crucial functions work correctly.
FreeBSD currently has a set of tests which are run using the kyua framework. These tests need to be first installed individually before they can be used for testing. This proves to be problematic in cases when direct testing of some newly installed or updated utility has to be performed. It makes testing changes to utilities and libraries difficult as one would like to perform tests (e.g. to ensure a proper build environment) before proceeding for installation.

Project Description

This project aims to develop a test infrastructure and automation tool along with basic tests to verify if all the base utilities in FreeBSD are linked properly. The testing framework is supposed to ease the process of writing "trivial" test cases which will be run in a completely automated and developer friendly manner without need for any prior installation. Once integrated, the tool will also facilitate further development of tests.

Implementation Details

The following steps will be performed in order to understand the problem at hand and the approaches which can be used -

• Note how testing has been traditionally done in FreeBSD.

• Study the kyua framework and the approaches currently being used for testing.

• Study the current problems faced as a developer or tester when writing new test cases, and the problems faced as a release engineer or tester when executing existing test cases.

Test Plan

The file functional_test.c is a simple test file which checks whether the ls utility is properly linked by running trivial commands. The options supported by ls are stored in short_options[] and long_options[] present in functional_test.h.

The smoke tests can be run using the following commands -

>> make
>> make test

It should be noted that finally the tests will be automated with appropriately passed options. The above commands will not have to be run for testing individual programs. The test file uses getopt() and getopt_long() for testing the validity of the passed options. If a valid option is passed, the command <utility> --<option(i)> is executed. In case the command fails to execute for a valid option, this will imply that the utility under test is not properly linked.

Populating short_options[] and long_options[]

short_options[] and long_options[] need to be initially populated with a few supported options for all the base utilities. This can be done by using either one or both of the following available approaches -

• Parse man pages for each utility to get the supported options.
• Pass an unsupported option to the utility. This might generate a usage message which can then be parsed. The unsupported option will be chosen experimentally.

An automation script will be written which will populate short_options[] and long_options[] by following the above mentioned approaches and will generate relevant test files. The initial version of the generated test scripts will only test trivial functionalities.

Although the test plan does not check the entire state of application under test, it suffices for the initial purposes of smoke testing.

Expected Results

• Set of test programs to verify basic functionality of all the base utilities in FreeBSD.
• An automated tooling to generate test scripts which confirm minimal set of functionalities of a utility to check whether it is properly linked.

(To be updated)

Milestones

Table design adapted from NonBSMtoBSMConversionTools#Milestones

Deliverables

Deliverables for first evaluation

• Add kyua based test programs for a subset of base utilities which are currently without any test coverage. The maximum number of utilities possible in the duration will be covered.

• Prepare a basic design of the automation tool. Cover generation of a limited number of test scripts.

(To be updated)

Weekly Reports

Bugs reported

The Code

github/smoketestsuite - Contains a rough implementation sketch of an in-progress smoke-testing tool.

Differentials