Light My Fire
Light My Fire or LMF is the code name for a planned rewrite of the Tinderbox.
The goal of this page is to summarize LMF concepts and features, flesh them out, and track their progress.
Terminology
- build
make buildworld followed by make buildkernel for all selected kernels
- builder
- a machine that runs builds
- iteration
- a single build of every selected branch for every selected architecture.
- supervisor
- a machine that plans an iteration, parcels out work to builders, and gathers and presents the results. The supervisor may or may not be a builder.
Features
The following is a list of planned features, more or less in the order in which they will be implemented. Features which are present in the current implementation are omitted.
Not restricted to FreeBSD
LMF should be sufficiently general and modular to be usable for other projects as well (including projects which use neither cvsup nor cvs).
Single-tree iterations
Currently, the tinderbox checks out a separate source tree for every build immediately before that build starts. This is a bad idea for several reasons:
- Results are not comparable across architectures, since each iteration can take several hours, during which many commits may take place. Currently, it is possible for a build-breaking bug to be committed while an iteration is already under way, so some builds will succeed while others will fail, even if the bug is not architecture-specific.
- Performance: a multi-core builder can run multiple parallel builds much faster if the builds all share the same source tree, since more of the tree will be cached.
Therefore
- A single copy of the source tree should be shared between all builds running on the same builder.
If several builders run builds for the same branch, they must all check out the same version of the source tree. This is easily done by explicitly specifying a date to cvs or cvsup, but since the supervisor needs a copy of the source tree anyway, the builders might as well rsync their trees from the supervisor.
Build history
Record every iteration and every build in a database, with the exact source version used, all build parameters, and the outcome. Provide an interface to this database which allows people to compare successive iterations.
Cross-reference logs with source
Augment the build logs with HTML markup, linking every file name in the log to the corresponding file in the source or object tree. Link each compiler warning or error to the exact line of code it references.
Obviously, we can't keep old source trees around forever, but the last couple of iterations should be doable.
Remember rel="nofollow"!
Avoid unnecessary builds
Branches that haven't changed since the last build should not be rebuilt. The easiest way to determine this is probably to capture cvs or cvsup output.
Resilient build
Through creative application of make -V, it is possible to build a complete dependency graph for the source tree (if it isn't complete, heckle ru@ until he fixes it). This means that even if part of the tree is broken, LMF can identify and attempt to build those parts of the tree that are independent of the broken bits.
This is actually a lot easier than it sounds; porteasy already does this to avoid unnecessarily updating unused parts of the ports tree.
Optional features
The following features fall in the "nice to have" category.
Images
After a successful build, install it into an empty directory and package up the result as a tarball and / or ISO image.
RDF
Provide an Atom feed of the build results.
Source cross-reference
Using the dependency information gathered for the resilient build feature, provide an lxr-like source cross-reference which (unlike lxr) is able to bridge the gap between programs and the libraries they use, without confusing unrelated but similarly-named symbols used in separate programs and libraries.