FreeBSD-ng
We are at a crossroad. Server "Unix" is going in different directions, some of which we are not equipped to support.
We do a complete, usable by default, "Heavyweight" Unix fairly well. However, there are lots of new models reaching critical mass. Eg: trivial, massively replicable containers.
We are also suffering from a very heavyweight build process. Yes, it is self contained and integrated, but damn.. we pay the price for it.
For phk's "be the best unix around" goal, we have done a reasonable job for a while. However, an increasing number of people don't want that any more.
How could we do it better?
The first clue is that we have a serious contender for a decent package system now.
The second clue is that over time, more and more functionality has been moving to build options or completely to ports. There is clearly an acceptance that a mega-unix isn't for everyone.
However, the "ports tree" has plenty of its own problems and isn't exactly suitable as-is for augmenting the base system. For starters, ports were designed for installing into a working system and packaging was added as an afterthought. We've been dancing around that issue for decades, and in spite of heroic efforts to improve packaging support, ports are still a wrapper around a 'configure; make all; make install' into a live system.
Solution? Burn it all to the ground!
- Step 0: define FreeBSD-ng as a pkg(8) based system.
- Step 1: Make a "base" package template with nothing in it but life-cycle definitions of what's supposed to be in it by default. An illustration: base 9.x tracks openssl openssl-0.98, 10.x tracks openssl-1.0.x, etc.
- Step 2: Partition the traditional base tree into logical components where logical dependency tracking can happen. eg: libc, openssh, coreutils, rc-startup, etc.
- Step 3: Burn down 'make world'. Use a staging build system (think: poudriere) that installs toolchain, prerequisites etc and produces a pkg as output.
At this point, the parts of base that are worth packaging are avalable for pkg to install. A new style 'buildworld' is a staged rebuild of core packages and an 'installworld' is pkg upgrade -f.
Once the basics are viable, the tough issues need to be dealt with:
- Step 4: Rethink how we do ports, maintainers etc. MAINTAINERS as it exists in classic ports as a hard lock is untenable for base system packages. The whole base / ports / maintainers division of labor and responsibility needs a re-think for this approach to work.
- Step 5: Classic ports where a set of patches exist to a third party release doesn't suit the model where *we* are the upstream. This means designing a scalable, sustainable system for producing packages from source code.
- Step 6: Figure out how we might do -stable, releases. With this many more moving parts the old way won't be possible. If developers are doing a pourdriere style build for producing their systems, this should translate well to binary release models.
- Step 7: Teach the binary installers how to build fresh systems the new way.
- Step 8: deal with errata/security releases. If we do this right, an errata/security release should be a simple case of updating package manifests and have downstream binary consumers do a 'pkg upgrade'.
This is definitely a disruptive long term project to complete.
If it is done right, the benefits will be apparent..
For example:
- Everybody deals with binary packages - they are no longer an afterthought or the second class option. The difference is how they are acquired (self built or binary distribution), and where from (base packages, ports packages, whatever we want).
- Every developer has to focus on usability of packages. Shipping useless-by-default packages won't cut it.
- Build toolchain is separate and wouldn't have to be in the base system at all. (Presumably it would be, but the choice becomes easy)
- We get to have a badly needed "spring cleaning" of our base system.
- We get to have a badly needed "spring cleaning" of the ports system.
- With branch/stable/release/patch profiles we don't have to duplicate work for each branch. eg: one version of fortune for all versions of freebsd. We can do branch development but in many cases it wouldn't be needed.
- contrib goes away - we build things their native way where possible.
- /usr/local becomes the property of the system owner, without requiring "mandatory" things be installed there. (That might be a good differentiation between ports and base packages - things that are designed to be installed in / rather than /usr/local.)
- A functional jail/container might be able to be just a handful of files.
Flavors become build profiles. We define stable or long-term-support branches in terms of ABI lockdown and feature sets.
Since partitioning of the base system would have to be on the table, we wouldn't need to be locked into a single VCS. Since the parts would be smaller, they become well within the comfort zone of the many dvcs systems out there. When the parts become smaller, then that makes CI workflows simpler.
I am NOT talking about .....
- NOT suggesting a source package based build system where a source package is submitted to a build system - keep it source based, please!
- NOT suggesting ports-style maintainer single threading - its got to be a community effort still.
Known problems..
- freebsd-update goes away in this design, but pkg would likely need a bit more hardening for remote updates.
- We'd need to rethink the way we do pkg "sets" and pools etc. We'd need to be able to pull different revisions of the same package within an abi set and I'm not sure we can do that yet.
- Culture shock would be high. This is a very different approach.
- Need to find a bulk supplier of SPF-1-billion sunblock and some bikeshed-proof-sunglasses.
- A million more edge cases and what-ifs..
Can we even do it?
I know we have the capability, but I'm not sure about the willpower. As a whole, we are a very change-averse group and many of us are here because we liked having an integrated system.