Tinderbox
This article describes the FreeBSD source tinderbox. To learn more about what it does, take a look at the current FreeBSD src build status.
The tinderbox ecosystem consists of:
A build script, tinderbox that automates checking out a specific version of the FreeBSD src tree and building it
A supervisor script, tbmaster, that runs and monitors individual tinderbox instances, logs their output, and mails out failure notices
A CGI script, index.cgi, that reads a set of tbmaster logs and presents an easy-to-read HTML summary
- A set of build servers that continuously test the tip of the most important FreeBSD branches
- A web server that keeps a complete set of tinderbox logs and displays an up-to-date summary
Contents
History
tinderbox scripts were developed and are maintained by DagErlingSmørgrav. The original version was a shell script that ran on bento.freebsd.org once a day, after one Alpha-breaking commit too many. It was replaced with the current Perl implementation in early 2003.
All scripts and configuration files are kept in base/user/des/tinderbox.
Dashboard & Build Status
Note: The service described here is no longer operational and now redirects to the FreeBSD Jenkins Continuous Integration service: Tinderbox-style view.
Summaries and logs from the official build servers are made available online at http://tinderbox.freebsd.org/.
Currently, this site is hosted by DagErlingSmørgrav. The site is set up as follows:
A cron job checks the build servers at regular intervals and downloads any new log files using rsync.
Apache is set up to use index.cgi as DirectoryIndex.
A Varnish instance in front of Apache ensures that index.cgi does not need to run more than once every two minutes.
tinderbox script
This section has not yet been written. See the tinderbox(1) manual page for details.
tbmaster script
This section has not yet been written. See the tbmaster(1) manual page for details.
Configuration Files
This section has not yet been written.
index.cgi script
The index.cgi script generates the HTML summary. Although intended to be used as a CGI script (as the name indicates), it can also be run from the command line, or from a cron job, in which case it will look for logs in the directory in which the script itself is located. It will automatically detect the context, and output HTTP headers only if run as a CGI script.
The script outputs strict XHTML 1.0 code and uses CSS for styling.
MAIN
The script's entry point is in the named block MAIN.
The script starts by attempting to verify that it is running on the official tinderbox web site. If it isn't, the page it produces will include text to that effect, and a link to the official site.
It then scans the log directory to get an inventory of configurations, branches and architectures for which log files exist, to avoid hard-coding a list into the script and potentially ending up with blank rows or columns. This information is derived from the names of the log files, which match the following pattern:
tinderbox-$config-$branch-$arch-$machine.{brief,full}The configurations used on the official tinderbox build servers are named for the branches they build; for instance, the releng_6 configuration is used to build RELENG_6 as well as all still-supported release branches.
Once this has been determined, the script generates a prologue, calls do_config() for each configuration, and generates an epilogue.
do_config()
The do_config() subroutine generates HTML code for a single tinderbox configuration.
It first generates a header row, then iterates over each branch build with this particular configuration, producing a single row of results for each in the following manner:
- For each architecture,
- For each machine within that architecture,
If a brief log file exists:
Call success() to determine the outcome of the build
- Output the modification date
Output the size of the brief log file with a link to the file itself
If a full log file also exists:
Output the size of the full log file with a link to the file itself
- Otherwise:
Output n/a
- For each machine within that architecture,
success()
The success() subroutine scans a brief log file to determine whether the build was successful. This is done quite simply by searching for the string "tinderbox run completed".
Build Sorting
Configurations and branches are sorted according to their branch rank, which is computed as follows:
HEAD and CURRENT have rank 9999
RELENG_x has rank xx99
RELENG_x_y has rank xxyy
In practical terms, this means that HEAD always ranks highest, and that RELENG branches are ranked in numerical order, with each STABLE branch ranking higher than the release branches forked off of it. For instance, for FreeBSD 6, the order will be, from highest-ranking to lowest-ranking:
RELENG_6 (branch rank 699)
RELENG_6_3 (branch rank 603)
RELENG_6_2 (branch rank 602)
RELENG_6_1 (branch rank 601)
RELENG_6_0 (branch rank 600)
Colors
The summary script uses CSS to apply different colors to each cell in the table. Successful builds are displayed with green text, failed builds with red text. The color fades as time passes since the corresponding build, with every half-hour bringing it one step closer to gray.
Official Build Servers
The official tinderbox build servers are hosted by Sentex Data Communications, who also host the FreeBSD netperf cluster.
There are currently three build servers:
- freebsd-current.sentex.ca
- builds
HEAD for amd64, arm, i386, pc98, ia64, mips, powerpc, and sparc64; this takes about four hours.
RELENG_8 for amd64, arm, i386, pc98, ia64, mips, powerpc and sparc64; this takes about three and a half hours.
- freebsd-stable.sentex.ca
- builds RELENG_7 and supported 7.x branches for amd64, i386, pc98, ia64, powerpc and sparc64; each branch takes about six hours.
- freebsd-legacy.sentex.ca
- builds RELENG_6 and supported 6.x branches for amd64, i386, pc98 and sparc64; each branch takes about three hours.
Future Work
Suggested improvements:
Improve the logic tbmaster uses to extract the error message from the full log to put in the brief log.
- When a build finishes, keep the source and object tree along with the log file, and filter the log file through a CGI script that adds markup linking every file name to the corresponding file in the source or object tree, and every compiler error or warning with the file and line it references.
- Instead of checking out a separate source tree for each platform, use a single source tree per iteration
- Store results (though perhaps not full logs) in a database
- Implement a better build scheduling algorithm which
- avoids building branches that haven't changed since the last build
- prioritizes some branches over others, e.g. security branches over stable branches over head
- Implement a patch-testing service where people with appropriate credentials can submit a patch and schedule a build run with that patch applied
One weakness of the current summary script is that it relies on colors to distinguish successful from failed builds. I have tried to find a better way—adding "FAIL" or "OK" to each table cell depending on the outcome of the corresponding build, for instance—but everything I've come up with has increased the size of the summary page to the point where it won't fit in a browser without scrolling, even on 1600x1200.
See DagErlingSmørgrav/LightMyFire for more information on this subject.