TCP/IP Regression Test Suite - GSoC '16

The Code

Available on Github

Overview

Regression testing is one of the most critical elements of the test artifacts and proves to be one of the most preventive measures for testing a software. Currently, within FreeBSD, there is no such tool to perform regression testing of the TCP/IP network stack. The purpose of this project is to develop tests using a regression testing tool which can then be integrated with FreeBSD. Once integrated, the tool will also facilitate further development of such tests. The regression testing tool of choice here is packetdrill.

Project description

packetdrill currently supports testing multiple scenarios for TCP/IP protocol suite within Linux. This project aims to design and implement a wire level regression test suite for FreeBSD using packetdrill. The test suite will exercise various states in the TCP/IP protocol suite, with both IPv4 and IPv6 support. Besides Linux, the packetdrill tool works on {Free, Net, Open} BSD. The existing Linux test suite implemented within packetdrill will provide a basis for understanding, and implementation of the FreeBSD test suite. For the current scope of the project, only a subset of the existing test scenarios will be implemented.

Why Packetdrill?

While valuable for measuring overall performance, TCP regression testing with netperf, application load tests, or production workloads can fail to reveal significant functional bugs in congestion control, loss recovery, flow control, security, DoS hardening and protocol state machines. Such approaches suffer from noise due to variations in site/network conditions or content, and a lack of precision and isolation, thus bugs in these areas can go unnoticed. Since netperf is supposed to be more for benchmarking purposes and what we are trying to do is measure correctness, packetdrill, which was built with the same mindset, seemed an apt choice for this project.

Installation

The testsuite is available as a freebsd port and can be installed using the following command if you already have packetdrill installed and configured -

pkg install tcptestsuite

For installing the entire suite with packetdrill -

pkg install packetdrill

Now proceed with the steps for configuring packetdrill.

Test Plan

packetdrill supports two modes of testing - local and remote. A TUN virtual network device is used in the local testing and a physical NIC is used for remote testing. Local testing is relatively easier to use because there is less timing variation and the users need not coordinate access to multiple machines.

The following tests will be done in order to ensure proper functioning of the tests as desired -

Local mode testing

Local mode is the default mode, and hence the user need not specify any special command line flags.

>> ./packetdrill -v <test-script.pkt>

Executing the above command will give the information about the inbound injected and outbound sniffed packets which can be studied and checked whether in accordance with the expected behaviour. The TUN virtual network device will be used as a source and sink for packets in this case.

Remote mode testing

On the system under test (i.e the “client” machine), a command line option to enable remote mode (acting as a client) and a second option to specify the IP address of the remote server machine to which the client packetdrill instance will connect must be specified.

client>> ./packetdrill --wire_client --wire_server_ip=<server_ip> <test-script.pkt>

On the remote machine, using the same layer 2 broadcast domain (same hub/switch), a packetdrill process acting as a “wire server” daemon to inject and sniff packets remotely on the wire will be started.

server>> ./packetdrill --wire_server

The client instance will connect to the server (using TCP), and will send command line options and contents of the script file. Then, the two packetdrill instances will work in coherence to execute the script and test the client machine’s network stack.

IPv4 and IPv6 protocol testing

packetdrill supports IPv4, IPv6 and dual-stack modes. The modes can be specified by the user with --ip_version command line flag. To get FreeBSD to allow using ipv4-mapped-ipv6 mode, the kernel must be notified with the following command -

>> sysctl -w net.inet6.ip6.v6only = 0

For testing using AF_INET6 sockets with IPv4 traffic -

>> ./packetdrill --ip_version=ipv4-mapped-ipv6 <test-script.pkt>

For testing using AF_INET6 sockets with IPv6 traffic -

>> ./packetdrill --ip_version=ipv6 --mtu=1520 <test-script.pkt>

Since the IPv6 headers are 20 bytes larger than the IPv4 headers, the MTU has to be set to 1520 to address the extra 20 bytes, rather than the standard size of 1500 bytes.

Using the script to automate

The script run-tests can be used to automate the tests. The value of the $packetdrill variable should be set to the location of the packetdrill binary on your machine before running the script. The default path is /usr/local/bin/packetdrill in FreeBSD and /usr/bin/packetdrill in Linux. The following command should be used for executing all the tests -

sudo sh run-tests <directory/file>

You can specify the directory for which you want to run the tests. Note: If no directory/file is specified, then pwd is taken into consideration for generating the list of tests (including all the subdirectories).

Note: sudo is required only for running the packetdrill binary.

Update: Logging functionality is now added in the automation script. After a single run, all the errors are placed in error.log in a clean format for easy debugging.

Scenarios covered

Scenario

Number of tests

Result (FreeBSD)

ICMP

5

Passed

Blocking system calls

2

Passed

Fast Retransmit

1

Passed

Early Retransmit

1

Failed

Fast Recovery

1

Passed

init_rto

1

Passed

Initial window

1

Passed

PMTU discovery

1

Passed

Retransmission Timeout

2

Passed

Socket Shutdown

3

Passed

Undo

2

Passed

Connect

1

Passed

TCP options establishment

5

Passed

AIMD

1

Passed

TIME-WAIT configuration

1

Passed

Selective Acknowledgements

1

Passed

Connection Close

3

Passed

Simultaneous Close

1

Passed

RESET from synchronized and
non-synchronized states

6

Passed

MSS

8

6/8 Passed

Receiver RTT

2

Passed

TCP timestamps

-

Passed

Updated directory structure

Scenario

Number of tests

Result (FreeBSD)

ICMP

1

Passed

Socket API

17

Failed: 1

TCP Mechanisms

12

Failed: 1

TCP State Machine

17

Failed: 3

Possible Scenarios

Milestones

Start

End

Task

23 May

Start of coding

(./)

23 May

24 May

Checking for compatibility of previously developed tests for Linux with FreeBSD

(./)

24 May

19 June

Manual development of tests based on TCP, considering all the scenarios covered in Linux tests

(./)

20 June

27 June

Mid-term Evaluations

(./)

28 June

31 July

Attempt at developing new tests covering additional scenarios and socket based tests for FreeBSD

(./)

1 Aug

11 Aug

Attempt at adding support for tcp_info() options in order to use assertions successfully in packetdrill

(./)

12 Aug

14 Aug

Code review

(./)

15 Aug

End of coding (soft)

(./)

15 Aug

23 Aug

Clean up and refactor the entire codebase, fix minor issues

(./)

23 Aug

End of coding (hard)

(./)

Future Plans

There is a huge scope for work yet to be done in this project, and I am not stopping anywhere in the near future. The final goal is to make this test suite exhaustive so that it can be easy for FreeBSD developers to check the authenticity of the network stack in a rigorous manner, and that occurrence of any misbehavior can be found out and rectified easily. The number of scenarios that can be added are innumerable, and the existing implemented set will be kept expanding and perfected.
Some of the tasks which can be listed as of now are -

All the current bugs and enhancements in the project can be tracked via the issues on Github.

Acknowledgements

I cannot thank my mentor Hiren Panchasara enough for all the help and support which he has given in the entire duration of the project. Working in his guidance was full of fun and challenging. I would be extremely grateful to receive his guidance in the future too while I continue to work on this project.

I also thank Michael Tuexen for all his help and guidance during the project.

I would also like to thank the FreeBSD community for accepting this project, which gave me a chance to gain a lot of knowledge about the community.

SummerOfCode2016/TCP-IP-RegressionTestSuite (last edited 2017-06-16 23:17:46 by ShivanshRai)