Debug with CI.FreeBSD.org
Contents
TL;DR:
- "Failure" and "Unstable" - please check your changes (or other's change in the same run).
- "Still *" - your change has not been tested, please watch the future tests.
*-build jobs
"Failure" - the build is suspected being broken by the following changes
"Still Failing" - the build has not been fixed by the following changes and
this is a notification to note that these changes have
not been fully tested by the CI system
*-test jobs
"Unstable" - some tests are suspected being broken by the following changes
"Still Unstable" - the failing test cases have not been fixed by these
following changes and this is a notification to note that
these changes have not been fully tested by the CI system
Reproduce
Get the commit hash in the parentheses of the first line of the notification mail.
FreeBSD-main-amd64-test - Build #21603 (becd9908beb8f1b47ddc6628cb005185a26ec85c) - Still Unstable
And from the body to see the failing case:
The failed test cases: 1 tests failed. FAILED: sys.netpfil.pf.killstate.multilabel
Get the VM image (replace the commit hash in the URL):
Or use base.txz, kernel.txz, etc. to install by yourself
Run tests manually (replace commit hash in the URL):
note: you may need to comment out test_suites.FreeBSD.ci = 'true' in /etc/kyua/kyua.conf
sh /usr/share/examples/bhyve/vmrun.sh -m 4096m -c 2 -d disk-test.img test sysctl -w kern.corefile=/tmp/%N.core (will be added to /etc/sysctl.conf) cd /usr/tests/lib/libarchive kyua debug functional_test:test_write_filter_zstd
Debugging hints
An assorted collection of potentially helpful hints for debugging failing test cases.
- Most tests can be executed outside kyua:
cd /usr/tests/sys/netpfil/pf && sudo ./names names
- atf-sh(1) tests will issue a warning about this, but will execute as expected.
atf-sh(1) tests are shell scripts, so adding set -x to the test body can be helpful.
atf-sh(1) tests typically fail at an atf_test call, which will be logged as 'Executing command [ ... ]'. There may be multiple atf_test calls in a single test, so note carefully which one is actually failing.
Console Output
Some useful tips when reading console output
On a failed (not completed) test run, search for panic: (include ':') for the panic point
For -test job, search for + fetch https://artifact for the test vm disk image download URL.