• DevSummit
• 201708
• Testing

https://hackmd.io/KYJhGMA4CMFYBMC0BDAbARgMyICwAZ1IV151FSRJUBOScWCHIA==?both

Test all the things! http://goo.gl/Xcocrp

Last Working Group: DevSummit/201706/Testing

Current Status

• CI system:

  • Jenkins/TODO

  • DTrace test pipeline: https://ci.freebsd.org/job/FreeBSD-head-amd64-dtrace_test/

What to test

• user space utilities

  • SummerOfCode2017/SmokeTestingOfBaseUtilities

• kernel features

  • in src/tests/sys, many skipped due to test environment (ex: no zfs)

• performances
  • network
  • filesystem
• fuzzing
• non-x86 architectures
  • use qemu user-mode to test cross platform
    • slow, some tests are racy
  • there seems a arm-cloud provider willing to sponsor
• Tracing tools
• Possibly finding a way to test regressions while deploying security mitigations?
• How others do?
  • NetBSD

    • http://releng.netbsd.org/test-results.html

  • OpenBSD

    • http://bluhm.genua.de

      • has test for sys/
      • has performance tests

      • https://github.com/bluhm/regress-all

  • Microsoft, FreeBSD on Azure

    • https://github.com/clovertrail/rocket/tree/master/shell/performance

How to test

• Automated analysis of fuzzing results?
• Ideas to generate an analysis data structure (Crash Graph ideas)
• QEMU fuzzing?
• real hardware testing (arm/mips/... boards)
• unit testing of kernel functions/modules/units (requires framework)?
• user-space versions of kernel code (examples: SCTP library, TCP library, etc.)?
• Create "culture of testing"?
  • Should reviewers ask for tests for big features just like they ask for man pages?

Quality Assurance of FreeBSD

• criteria of "good to ship"
  • no regression
  • feature
    • experimental feature should be marked explicitly

TODO list

• go through cases in test suite
  • need list of "skipped", "know to fail" cases
    • can jenkins list them?
• integrate more test suites
  • DTrace

    • needs WITH_DTRACE_TESTS=yes in src.conf

    • can be enabled after fixing all bugs (in DTrace or test cases)
  • Capsicum

    • https://github.com/google/capsicum-test

  • pf

    • https://lists.freebsd.org/pipermail/freebsd-hackers/2017-July/051261.html

  • TCP
    • Using Packet Drill tests from Michael Tuexen (gnn@)

      • Test suite is available here https://github.com/freebsd-net/tcp-testsuite

  • ZFS Test suite (brd@)

  • Have we converted all things under tools/regression?

• setup performance testing cluster
  • location? NYI
  • hardware requirements? (brd@ / gnn@)
• generalize freebsd-ci codes for downstream adoption
  • a package to setup everything?
• Some form of automated testing for security mitigation deployment (W^X, ASLR, PAN, ...) (brooks@, Domagoj)
• Talk to Backtrace (gnn@)