Pf Testing

Project description

The aim of the project is to create a framework for testing the network packet filter pf in FreeBSD. Testing would cover both rule parsing (in pfctl, the parser and loader, user space) and actual packet processing (netpfil/pf, kernel space). The aim is to also put a system in place for automatically and regularly running these tests, and integrate it with appropriate build and release tools. It shall also be easy to extend the framework and add new test cases.

Such a tool exists already for OpenBSD, and part of the effort would be in investigating the possibility of porting that to FreeBSD as much as possible, to share the effort and make use of existing tests.

Such a framework would be good for FreeBSD since there currently is no such testing system in place. That would help with regression testing among others, and generally increase the productivity of the FreeBSD project by quickly notifying developers of new errors in the codebase.

Approach to solving the problem

An initial way to do rule parsing will be to create a testable version of ​pfctl​ , one that reports the commands to be sent to the kernel instead of actually sending them. Then, different rulesets will be created and the testable version of ​pfctl​ will be run on them.

An initial way to test the packet processing will be to run the test target as a guest in a virtual machine (using ​bhyve​ or similar) with multiple network interfaces, and load different rulesets with ​pfctl​, after it has itself been tested for correct parsing and loading of rulesets. The possibility of loading rulesets via a separate testing method will also be investigated. Other necessary configuration files, such as network configuration, will also be loaded. The host will then generate different types of traffic for the guest, or create more guests that act as network traffic producers and consumers, and monitor the guest behaviour and output. Different kinds of behaviour can be recorded, such as correctness of packet processing, and performance and stability of the test target.

Moreover, further testing can be done with cases of multiple interacting test targets, or targets running different network services and generating their own network traffic. The test cases will ideally be in the form of network traffic-generating shell scripts or small compiled programs. The possibility of creating or adopting a special scripting language for network testing will also be investigated. The test system will be designed to make it easy to load different targets with different versions, rulesets and network topologies.

Deliverables

Testable version of ​pfctl

Investigate testability of pfctl​, perhaps using the ​-n​ flag. Investigate how OpenBSD does it, and port or adapt as necessary.

Parser tests for ​pfctl​ using testable version

Create parser tests and run them using the testable version of pfctl​. Investigate what types of tests are important and in demand. Borrow test cases from OpenBSD when appropriate. Investigate other possibilities for testing, e.g. by looking at other similar projects.

Test setup allowing network tests (using​ bhyve​ or similar)

To test the actual packet processing, the main approach will be not to make any changes in the kernel code, but to create an appropriate setup (in a virtual machine using bhyve​, or similar) that easily allows testing. Different ways to make this as simple and well-performing as possible will be investigated.

Network tests (using test setup)

Create network tests, borrowing from OpenBSD and other projects when appropriate. Investigate what types of tests are important and in demand. Investigate different ways of loading and logging in the test setup, which will probably be a host and guest virtual machine setup.

Integration with build and release system

Investigate how to best integrate the framework with existing FreeBSD build and release tools. Create a target for make​ or hooks for the version control system as appropriate.

System to run tests regularly

Install a system that runs tests regularly and reports results to the developers. That can be a virtual machine installed and running somewhere, or a module for an existing system. Investigate ways to make it as productive as possible, possibly by communicating with interested groups and mailing lists of the FreeBSD project.

Integration with Jenkins/CI/​ATF

Jenkins is a continuous integration (CI) tool used by FreeBSD. Investigate if it would be appropriate to integrate the framework there, and communicate with the Jenkins people of FreeBSD about it. Integrate with ​ ATF​ as appropriate.

Documentation

Produce documentation to make it easy for other to use and extend the framework with new tests. Documentation will be continuously delivered with a refactoring and cleanup phase at the end.

Milestones

May 15-30

May 30-June 20

June 20-July 28

July 28-August 21

August 21-29

August 29-September 15

Test Plan

Since this is a testing tool, a separate test plan would be redundant. The test plan for the test tool is to run the test against what we’re testing :)

Results

(Current as of 20170829.)

We have managed to deliver almost all of our deliverables. The pfctl tests have been included in FreeBSD, while the kernel pf tests are on the way.

The deliverable that was skipped was System to run tests regularly, as focus went on integrating the tests into the regular test suite. Such a system can still be built, and there is room for optimization for such an installation.

The Code

The code is on GitHub at https://github.com/paggas/freebsd, under the 'paggas' branch, forked off of FreeBSD's GitHub mirror at https://github.com/freebsd/freebsd. Most, if not all, of the work will be under tests/.

For easier review, we have gathered the work in two patches:

Patch D11322 is already included in FreeBSD, while D11401 is on the way (as of 20170829).

This project is part of SummerOfCode2017.

SummerOfCode2017/PfTesting (last edited 2017-08-29T11:37:28+0000 by PanagiotesMousikides)