Smoke testing of all base utilities


Overview

Smoke testing is a set of light tests which are done for checking basic functionalities of a software to ascertain if the crucial functions work correctly.
FreeBSD currently has a set of tests which are run using the kyua framework. These tests need to be first installed individually before they can be used for testing. This proves to be problematic in cases when direct testing of some newly installed or updated utility has to be performed. It makes testing changes to utilities and libraries difficult as one would like to perform tests (e.g. to ensure a proper build environment) before proceeding for installation.

Project Description

This project aims to develop a test infrastructure and automation tool along with basic tests to verify if all the base utilities in FreeBSD are linked properly. The testing framework is supposed to ease the process of writing "trivial" test cases which will be run in a completely automated and developer friendly manner without need for any prior installation. Once integrated, the tool will also facilitate further development of tests.

Implementation Details

The following steps will be performed in order to understand the problem at hand and the approaches which can be used -


Initially Proposed Test Plan

The file functional_test.c is a simple test file which checks whether the ls utility is properly linked by running trivial commands. The options supported by ls are stored in short_options[] and long_options[] present in functional_test.h.

It should be noted that finally the tests will be automated with appropriately passed options. The above commands will not have to be run for testing individual programs. The test file uses getopt() and getopt_long() for testing the validity of the passed options. If a valid option is passed, the command <utility> --<option(i)> is executed. In case the command fails to execute for a valid option, this will imply that the utility under test is not properly linked.

Populating short_options[] and long_options[]

short_options[] and long_options[] need to be initially populated with a few supported options for all the base utilities. This can be done by using either one or both of the following available approaches -

  • Parse man pages for each utility to get the supported options.
  • Pass an unsupported option to the utility. This might generate a usage message which can then be parsed. The unsupported option will be chosen experimentally.

An automation script will be written which will populate short_options[] and long_options[] by following the above mentioned approaches and will generate relevant test files. The initial version of the generated test scripts will only test trivial functionalities.

Although the test plan does not check the entire state of application under test, it suffices for the initial purposes of smoke testing.


Revised test plan

The test plan for the project has been completely revamped (a brief overview available here). Comments and suggestions are most welcome!


Expected Results

Milestones

Week

Duration

Description

Status

Start of Coding

{*}

1

May 30

Start of coding.

(./)

Add atf-sh tests

{*}

1

May 30 - June 5

Start adding atf-sh tests to utilities without test coverage.

(./)

2

June 6 - June 12

3

June 13 - June 19

Update smoke testing tool.

(./)

Fix a bug in the implementation of ln(1).

4

June 20 - June 26

Add functionality to generate trivial tests.

(./)

First evaluations

{*}

Start generating testcases to demonstrate tool behavior in different cases.

{*}

5

June 27 - July 3

Demonstrate negative testcases for ln(1). Updates

(./)

Support for utilities with no supported options.

Collation of common usage message.

6

July 4 - July 10

Demonstrate the tests which the tool can generate using date(1), ln(1) and stdbuf(1). Updates

(./)

7

June 11 - July 17

Add functionality for reading annotations. Updates

(./)

Second evaluations

{*}

Bug fixes and code cleanup

{*}

8

June 18 - July 24

Add functionality of dynamically generating annotations. Updates

(./)

9

June 25 - Aug 1

Miscellaneous fixes. Updates

(./)

10

Aug 2 - Aug 8

Updates to scripts. Updates

(./)

11

Aug 9 - Aug 15

12

Aug 16 - Aug 22

Add functionality to stop test generation during runtime. Updates

(./)

13

Aug 23 - Aug 29

Update documentation

{*}

Final evaluations

{*}

Table design adapted from NonBSMtoBSMConversionTools#Milestones

Deliverables

Deliverables for first evaluation

Deliverables for second evaluation

Deliverables for third evaluation

Weekly Reports

Note: While visiting links in the following reports to https://shivrai.github.io, they should be renamed to https://shivansh.github.io.

Week

Report

1

2017-June/001060

2

2017-June/001064

3

2017-June/001070

4

2017-July/001079

5

2017-July/001086

6

2017-July/001089

7

2017-July/001094

8

2017-July/001098

The test generation tool was ready at this point

12

Final Report

Bugs reported

PR Number

Fix

219943

D11167

The Code

Github

Contains the implementation of test generation tool. Also contains the implementation of the initially-proposed test plan.

Architecture overview

Differentials Submitted

Week

Differential

Status

1

D11020

Merged (./)

D11036

Merged (./)

2

D11084

Merged (./)

D11158

Merged (./)

3

D11159

Merged (./)

D11167

Merged [1] [2] (./)

4

D11315

Open

D11318

Merged (./)

5

D11529

Merged (./)

10

D12036

Merged (./)

The differential submitted on week 10 lays the basic structure of the generated tests

12

D12249

Open


SummerOfCode2017/SmokeTestingOfBaseUtilities (last edited 2017-09-30 06:17:44 by ShivanshRai)