• SummerOfCode2019Projects
• PortsSeparatedBuild

Separation of Ports Build Process from Local Installation

GSoC 2019 project

• Student: TheronTarigo

• Mentor: BakulShah

Introduction

The FreeBSD ports framework currently makes it possible, in a single command, to build a third-party software package and all of its dependencies. However, that dependency installation mechanism has the limitation that dependencies must be fully installed onto the build system before building of ports that require them can proceed.

In general, there is no way for the mainstream ports framework to complete all work within a temporary location without frequently interrupting the process to modify the build host to install these dependencies.

Modifying the build host in the middle of a build is often undesirable and unsafe. To avoid this, existing port-building tools such as Poudriere and Synth use jails to accomplish isolation. However, due to their extensive scope of features, these tools are not designed for nor appropriate for integration into Mk/bsd.port.mk as a standard feature of the ports tree.

The Ports Separated Build project enables building of ports in isolation by staging all tools, include files, and shared objects into a temporary location used exclusively by the ports build process, eliminating the need to install intermediate dependencies onto the build host. It enables this through changes to the scripts and makefiles provided with the ports tree in /usr/ports/Mk/ rather than through a separate tool. It does not rely on Jails or other privileged features.

In this way, it is made to behave more like the FreeBSD src tree, in which "buildworld" is a self-contained and self-sufficient process requiring no special preparation of the build host.

Currently, this works for 97% of the ~4200 ports tested. Remaining obstacles to a general solution working for 100% of ports are discussed later in this document.

Code

The changes to the ports tree to enable this are published at:

• https://github.com/therontarigo/freebsd-ports/tree/separated

• Difference with respect to latest pull from upstream: https://github.com/freebsd/freebsd-ports/compare/b1282523e949c12df991cd746e053b4de8b6dfc9...therontarigo:separated

The only major *new* piece of software involved is a userspace library for manipulating an unsuspecting program's filesystem namespace, redirecting file paths as configured by an environment variable. It is published in its own repository:

• https://github.com/therontarigo/freebsd-user-namespace

It is available as a port: devel/userns (found in the freebsd-ports repository above)

Working status

It is possible to perform the following (without root):

• $ git clone https://github.com/therontarigo/freebsd-ports -b separated --depth 1

  $ cd freebsd-ports

  $ make PORTS_SEPARATED_BUILD=1 PORTSDIR=$PWD PORTBLDBASE=$HOME/ports -C <category>/<portname> config-recursive package-recursive 

It will be more telling of actual success if this is done on a system or jail that doesn't already have any ports installed to /usr/local (or instead use a different LOCALBASE) since otherwise there is a chance that something accessed the real /usr/local when looking for a dependency (only a controlled location, defaulting to ${PORTBLDBASE}/depsroot, is meant to be accessible for dependencies in PORTS_SEPARATED_BUILD mode).

To the extent that building succeeds (the scheme is known to be compatible with some ports and not with others), the resulting packages are placed into $HOME/ports/packages/All. They can then be installed (as superuser) onto the system.

From the user's perspective, the most significant difference is that if this were done from the upstream ports framework, the process would be interrupted several times to install dependencies into the real /usr/local after requesting permission.

Background and Motivations

(this section will be of little interest to anyone already familiar with the history and established behavior of the Ports framework).

• Want: Given only a POSIX-compliant environment (of any architecture), without root or any other special privilege, a large collection of software is buildable from sources for any supported architecture without manual intervention and configuration. Actual usability of software is expected to vary depending on target environment.
• Have: FreeBSD ports collection (as-is):
  • Large collection of software buildable from sources: yes
  • Without manual intervention: generally yes
  • Without manual configuration: yes
  • Given only a POSIX-compliant environment: NO
  • Without root or any other special privilege: NO
  • For another architecture (or otherwise binary incompatible system): generally, no

Now, much work has gone into compatibility with BSD environment (not strictly POSIX) and to make use of BSD/FreeBSD extensions where they are useful. Therefore the POSIX-compatibility goal can be reasonably relaxed to a goal of requiring only FreeBSD base. (A POSIX-compatible bootstrapper to set up the BSD tools would bridge this gap).

Most ports require more than just a FreeBSD-base environment: they need some additional tools to be installed and configured. Special privileges must be granted by the OS to accomplish this modification to the environment: Installation of packages (Ports tree default) or management of Jails (Poudriere and similar tools).

POSIX-compliant environment (or FreeBSD base) is only a starting point; build systems (including ports framework itself) can create a modified environment when needed. It is the kind of modification that needs to be made that determines whether special system privileges and features are required.

Compare this limitation of the existing ports framework to the base system source tree:

• Tools executed: found in base system installation.
• Files read: /usr/src (can be changed)
• Files created: /usr/obj (can be changed)
• For another architecture: YES
• It is sufficiently self-contained that adding the capability to build the base system under a different host OS is a reasonable possibility.

Implementation details

Ports building scheme overview

• Ports are built using PORTS_SEPARATED_BUILD=1 set (otherwise normal ports behavior is left alone as the default).
• Port building is started from unprivileged user. All compilation work is run as this user.
• Ports built as dependencies are installed to a working directory, not system-wide.
• Dependency resolution is patched to use this working directory.
• All port-building commands (configure scripts, compilers, interpreters) run under file redirection to see dependency install area instead of local system.

Section outline

• PORTBLDROOT
• UserNS

• PortEnv

• bmake
• sub-make
• ldconfig
• pkg install scripts
• bootstrap

PORTBLDROOT

Required dependency ports are installed to a temporary location rather than into the root of the build host. The temporary location is specified by the PORTBLDROOT make variable.

UserNS

To address the need of ported software to find dependencies in expected locations under ${LOCALBASE} without performing the real installation to this location, a library providing a virtual filesystem namespace has been developed as the first phase of this project.

"UserNS" intercepts an unsuspecting program's C library API calls to rewrite pathnames in file operations, giving the appearance to the application of a filesystem namespace other than that of the host system.

It is included in the modified ports tree as devel/userns.

The most comparable previous work is the Linux-only "fakechroot" library. The "fake chroot" functionality is only a special case of UserNS's capability, which allows creating a namespace as an aggregate of arbitrarily many locations in the real filesystem.

UserNS is controlled through an environment variable. Its basic usage within the ports framework is:

• FILEPATHMAP=${LOCALBASE}%${PORTBLDROOT}${LOCALBASE}:/%/

This is a colon-separated list of virtual%real entries:

• ${LOCALBASE} is resolved to ${PORTBLDROOT}${LOCALBASE}
• All other file accesses are passed through as-is.

Actual usage involves further remappings.

PORTENV

A new Make variable is introduced, PORTENV, to expand to the command-line prefix needed to run a command inside the virtual file namespace facilitated by UserNS.

${PORTENV} defaults to ${SETENV} when separated-build mode is not used.

bmake

FreeBSD's make, /usr/bin/make, is a statically linked executable and therefore cannot utilize UserNS. This is not a problem for the ports framework itself but when ports internally require a Make utility for building, UserNS should be used so that Makefile file-existence checks work correctly.

FreeBSD's built-in make is imported from NetBSD Make. The FreeBSD port devel/bmake is a dynamically-linked alternative, with the only functional difference being that it uses its own (NetBSD-style) makefile includes in /usr/local/share/mk instead of FreeBSD's /usr/share/mk. This is easily corrected by a wrapper script in Mk/Exec/make. UserNS redirects /usr/bin/make to this wrapper so that it is always used instead by any port attempting to use "make".

Many ports use devel/gmake (GNU Make) instead, which is also compatible with UserNS.

sub-make

Ports themselves do not know about ${PORTENV}. Make does not provide a reliable mechanism for running all target commands with an alternate shell. Usage of ${SETENV} by ports' targets is inconsistent, and MAKE_ENV / CONFIGURE_ENV etc. do not capture all cases.

Instead, any port-defined targets (typically pre-* post-* do-* ... *-configure *-build *-install etc.) are run in a Make subprocess which itself executes under the PORTENV environment.

ldconfig

One of the requirements for PORTENV is that it enables execution of programs installed into PORTBLDROOT. This requires a ld-elf.so.hints file that is correct for that environment.

FreeBSD uses /etc/rc.d/ldconfig to maintain the system-default /var/run/ld-elf.so.hints.

A new script, ${PORTSDIR}/Mk/Scripts/ldconfig.sh provides similar functionality, processing the library directories under ${PORTBLDROOT} to maintain ${PORTBLDROOT}/var/run/ld-elf.so.hints. PORTENV sets LD_ELF_HINTS_PATH to use this hints file.

pkg install scripts

Some packages contain post-install scripts which must be run for the package to be properly installed. PORTENV is used to run these scripts when packages are installed to ${PORTBLDROOT} such that the operations of the scripts apply to files within ${PORTBLDROOT}.

Use of /usr/sbin/service from post-install scripts is specially handled. It is remapped by UserNS to ${PORTSDIR}/Mk/Exec/service, which supports only the services which are deemed relevant to proper installation of port-building tools. Currently, the only service is "ldconfig", which executes ${PORTSDIR}/Mk/Scripts/ldconfig.sh described above.

bootstrap

The ports tree is expected to install any tools it needs which do not come from the base system. The official ports tree as-is installs ports-mgmt/pkg before building any port, since it is an essential part of the ports build system's ability to check dependencies and to install ports once they are build.

This ability to bootstrap the pkg tool is extended to also build devel/bmake and devel/userns and prepare them within ${PORTBLDROOT}, since these are essential components of separated-build mode functionality.

Causes of incompatibilities with the separated-build scheme

• Failure to use PORTENV. If a port somehow executes a command outside of this environment, UserNS is not used at all and virtual file accesses will fail. Generally this is always fixable by changes within Mk/ files.
• Breakage of PORTENV environment. LD_PRELOAD and other variables must be set correctly for UserNS to function. If a port's script attempts to execute programs after changing these variables, those programs may fail to run or fail to resolve file locations as expected.
• Incompatibility with UserNS. UserNS relies on LD_PRELOAD to override LibC functions. Its handling of paths works in many cases but remains buggy for special cases. Therefore UserNS is not always a working substitute for actual installation of files into a location in which a port expects to find them.
• Usage of statically linked executables. UserNS has no effect on these, so when a port uses such an executable to access files from the port's dependencies, it fails.
• Usage of full paths in C Preprocessor includes. The base toolchain, including the preprocessor, is statically linked. Command-line include and library paths are rewritten to use the temporary dependency location but absolute #include paths bypass this. In practice there is not a good reason for ports to use these paths and they can be corrected to avoid doing this.
  • If lookup of UserNS-provided paths from the compiler is absolutely required, there are a few possibilities:
    • Including a dynamically-linked compiler toolchain in the basic FreeBSD install requires a change to FreeBSD base.
    • A dynamically-linked toolchain can be built from ports (as is done with devel/bmake), however: Clang is a large piece of software. Rebuilding it from ports is unacceptable for some users, and would be a major annoyance.
    • Developing a Clang extension integrating UserNS file lookup into the compiler is a potential solution.
    • Use one of the previous two solutions only for ports which are marked as requiring it.

Needed work

The current status of the project is that it is usable for some but not all ports. Much work remains before it can be adopted officially into the ports tree:

• Verification of non-breakage of conventional ports-build process
  • Before any changes can be merged into the official ports tree, they must be verified to not break or change the existing and well-established behavior.
    • Wherever possible, modification of existing Mk/ files are either:
      • Insertions of references to variables which default to being empty.
      • Insertions of code blocks guarded by conditionals which default to not using enabling the added code.
    • This is intended to eliminate chances of breakages when the separated-build mode is *not* used. The intention is that it can coexist with the conventional system as an optional, sometimes-useful mechanism, available as a tool for ports that are already compatible, and as a starting point for future work on expanding its compatibility.
• UserNS bugs
  • Relative paths are not translated.
  • "/../" in pathnames prevent correct translation of paths.
  • Symbolic links to virtual locations in the namespace do not work.
    • Note that realpath(3) does work as expected within UserNS; symlinks and

      ".." are resolved correctly. Utilizing realpath(3) before all virtual -> real path translations appears to provide correct behavior for basic file operations, but introduces a bug that symlinks become indistinguishable from normal files and directories, and the extra system interaction required creates a performance regression.

  • UserNS works generally only for conventional POSIX API file operations provided with absolute paths.
    • Luckily, tools used in the building of software use this kind of file access almost exclusively.
    • Rather than bring UserNS up to full general correctness, it may be more sensible to rebase it on the global namespace emulation provided by libpreopen. This potentially has the very desirable side-effect that all building work currently working under UserNS can be sandboxed with Capsicum.

      • libpreopen: https://github.com/musec/libpreopen

  • UserNS requires /usr/lib/debug/libexec/ld-elf.so.1.debug to exist so that the address of the open(...) function within ld-elf.so.1 is known, which is used to live-patch it to use the UserNS-provided open(...) call. This requires FreeBSD to be installed with debugging information included. One way to relax this limitation would be to maintain a table of ld-elf.so.1 file hashes to appropriate addresses for the versions of this file from all recent FreeBSD releases.
• Ports tree
  • A large block of variable definitions was added to the top of Mk/bsd.port.mk. This needs to be moved to its own file. Most of these additions are processed more than once due to their misplacement within Mk/bsd.port.mk, but this can be more easily fixed once the additions are moved to a separate file. Another run of ports building is needed to verify that this refactoring doesn't break the progress that has been made.

  • make install

    • Presently this target is overridden to provide the means of preparing dependencies in the PORTBLDROOT location. It should be restored to its original functionality of actually installing the built port onto the host system, requesting superuser privileges as needed.
  • Concurrent building
    • By using a different PORTBLDROOT location for each port, it is now possible to safely build multiple ports concurrently. However, the core dependency-handling scripts and makefiles likely need to be rewritten to take advantage of this.
  • Patching all Mk/Uses/*.mk
    • By design of the "portenv" mechanism, port-defined targets are always executed within the prepared environment, while target commands of makefile-includes in Mk/ must explicitly use ${PORTENV} where needed. Some Mk/Uses/*.mk have been amended accordingly; several remain to be done.

What could have been done differently

Possible alternatives to UserNS approach:

• Build all build tools for the host machine, using temporary location as prefix (but then existing packages can't be used for these since they use default prefixes)
  • This supposes there is only one location in the ports-building process used for installing build tools. What if two unrelated ports' build tools conflict? What if the presence of one build tool subtlely influences the behavior of another in a way which is difficult to debug?
  • Depending on user needs, each build tool may need to be built twice: once for port-building system (alternative prefix) and once for direct installation onto system (default prefix).
• Rebuild and repackage any build tool (any port which needs to run to build another port) to be relocatable i.e. can be installed to and run from any location.
  • This has additional benefit of easing the process of debugging conflicts between a piece of software and changes to the tools it requires.
  • The resulting package is appropriate both for usage on the build machine and for final installation on the target machine.
  • However:
    • Adding this capability may be infeasible for some tools.
    • Requires much work rewriting port Makefiles for these tools.
    • Maintenance burden is increased if patches must be used.
• Use cross-sysroot for locating all dependency files (not tools)

Future direction: cross-compilation support

Although only tangential to the primary results of the project, one of the deliverables as a side-effect is the following collection of notes on adding a generally working cross-compilation capability to the ports framework.

When cross compiling, there is an important distinction between two different kinds of build-time dependency:

• Library dependencies: provide files needed by the cross-toolchain to build the port.
• Tool dependencies: provide executables which need to be run on the build machine to do the work of building the port.

FreeBSD's ports tree did not initially record these distinctions in ports' build-time dependency information.

NetBSD's pkgsrc system, similar to and inspired by FreeBSD's ports, has slowly been solving this problem by adding a new type of dependency, TOOL_DEPENDS. This is similar to BUILD_DEPENDS except that it identifies a port providing commands which must be executed on the build host.

FreeBSD porting convention already provides a way to record this information, however in a way which might not be as obvious: the indication that a port depends on a program for building is an entry of the form:

• BUILD_DEPENDS+=  progname:category/portname

Many ports already make use of this. Unfortunately this convention is not always adhered to. By enforcing it as a rule, a port's attempts to use a port-provided program at build-time without specifying it appropriately in BUILD_DEPENDS can be considered as a bug in the port and fixed accordingly.

One way to enforce this is to remove ${LOCALBASE} from PATH and replace it with the path to a directory created by the ports framework containing only symlinks to the programs which have actually been registered correctly as dependencies. This needs to be done recursively so that run-depends of those programs are also satisfied.

PORTBLDROOT from the ports-separated-build project will need to be split into two different kinds of root, each with a specific purpose:

• Executable tool root. Contains the host-architecture executables required, and all support files needed to run them.
• Target system root. Contains the library objects, headers, etc. for the target architecture. An overlay of this root (containing only what is supplied by ported dependencies) on top of a FreeBSD installation root for the target architecture is used as the sysroot of the compiler toolchain for cross-building of ports.

GSoC 2019

The original goals of the GSoC project, followed by weekly updates made during the project, are preserved below.

Project Objectives

This project aims to provide the capability of the FreeBSD ports infrastructure to safely and cleanly build ports and all their dependencies without superuser privileges, jails, or touching the installed system in any way, in the interest of improving the safety, reliability, and repeatability of ports building without the administrative and resource overhead of a separated build host or jail.

This will bring the building capability of ports tree in line with what is already possible for FreeBSD base system: separate build vs. install workflow.

Currently this level of separation can only be accomplished in practice through chroot or Jail. This project will eliminate the need for cooperation of the root user since /usr/local will not need to be touched. To enable this, file accesses will be redirected to a location controlled by the ports build process through use of a userspace library to intercept file accesses.

This implementation takes the possibility of cross-building of ports into consideration as a long-term goal (far beyond end of GSoC due to the significant challenges and history of ports not taking this into consideration).

Where possible, the need for the userspace file interception hack should be eliminated, by switching to use of build tools supporting sysroot (this is analogous to desirability of sysroot-capable compiler vs. user-mode emulation for cross-arch compilation). The interception tool serves to cover otherwise uncooperative software.

Goal #1 - working isolated build

User (no special privileges, no ahead-of-time preparation by system admin) may do:

make -C /usr/ports/<category>/<port> package

-> The port and its dependencies are built and installed within a pristine environment (File access attempts to /usr/local never occur).

-> The packaged port is identical (with exception of dates and hostnames) to the package from official FreeBSD repository, assuming the same SVN revision has been used.

Goal #2 - efficiency & performance

The building as done according to the first goal is done efficiently.

Goal #3 - concurrency

Independent ports are built concurrently.

Project Deliverables

• File path redirection library, capable of emulating chroot behavior on any dynamically-linked executable (BSD replacement for "fakechroot") (Nominally completed for amd64 arch; bugs may be found)
• bsd.port.mk coordinates usage of temporary dependency location (Completed)
• Most ports are buildable without any interaction with real ${PREFIX} (In progress)
• Packages built by modified process are identical to packages build by existing ports framework

Technical Challenges

Running the entire ports framework in a chroot (as is trivially possible) defeats the purpose of the project, since it only moves the problem of isolating each port's build environment from a physical host to a virtual one: Ports framework itself, not just a blanket chroot created by system administrator, should configure the build-environment of each port.

Weekly updates

06/03

• Familiarized myself with the port dependency mechanisms in
  • /usr/ports/Mk/Scripts.
• Hacked on bsd.port.mk to check assumptions and to get an idea of what
  • additions will be needed.
• Studied /usr/src/lib/libc to understand chain of userspace calls between C
  • API and system call. Implemented proof-of-concept redirection library intercepting open(...).

06/10

• Continued to work on the file access interception library. It is working
  • well enough to redirect file accesses from shell and basic tools: sh, cat, ls - more thorough testing will be needed.

06/18

• Modified bsd.port.mk to behave as follows:
  • All built dependencies of a port are installed to ${PORTBLDROOT}
    • (=${PORTSDIR}/build on my test system).
  • Ports resolve dependencies to ${PORTBLDROOT} instead of to local system.
  • All port-building operations (configure, make, make install, etc.) are run
    • in a chroot, in which ${LOCALBASE} is mapped to ${PORTBLDROOT}${LOCALBASE} and base system is mapped as-is.
• Added hooks to several Mk/Uses/* and ports to allow switch to chroot where
  • appropriate (far from complete).
• Successfully compiled handful of ports (including C libraries, GNU build
  • tools, perl modules, and python modules) under this modified system.
• Next stage:
  • Replace the chroot with an equivalent solution in userspace (filesystem
    • namespace redirection), to eliminate requirement of superuser intervention.

06/24

• Switched bsd.port.mk from using a chroot for namespace manipulation to using
  • the userspace namespace tool.
• Enabled namespace tool to read path mappings from environment, supplied by
  • bsd.port.mk
• Re-ran port builds. Several problems which had not occurred when using
  • chroot became introduced.
• Studied interactions between executables, namespace tool, dynamic linker,
  • and standard library to understand the cases in which file accesses fail to be intercepted.
• Next stage:
  • Research, design, implement, and test appropriate solutions for the
    • problems encountered with userspace file access interception.

07/01

• Filesystem namespace intercept tool now redirects paths to shared object
  • files read by rtld.
• Fixed several problems with running programs under the namespace tool.
• Developed a plan for providing correct ldconfig behavior for programs run as
  • part of port building.
• Next:
  • Implement ldconfig fix.
  • Create a port of freebsd-user-namespace to be used by ports framework,
    • similarly to how ports-mgmt/pkg is used.
  • Set up automated testing of ports to assess progress and catch
    • regressions.

07/09

• Ldconfig hints file is maintained at each dependency port installation,
  • allowing build tools to run as expected.
• Fixed several more problems with running programs under the namespace tool:
  • Some ports contain /bin/sh scripts with no shebang. FreeBSD allows this;
    • Namespace tool's execve implementation is changed accordingly.
  • Fixed a bug in execve implementation concerning argv[0]
  • Debugging output, when requested, now does not get written to wrong file
    • after (v)fork and/or exec.
• Have ports use devel/bmake instead of /usr/bin/make, where possible, as the
  • latter is statically linked and cannot handle ports Makefiles' dependencies on installed files.
• Confirmed that devel/llvm60 and all its dependencies build successfully.
  • This selection of ports represents many build tools commonly used in building of other ports. To see for yourself: make -C /usr/ports/devel/llvm60 all-depends-list

07/15

• Created a port of freebsd-user-namespace: devel/userns.
• Integrated devel/userns into PORTS_SEPARATED_BUILD mode of ports framework.
• freebsd-user-namespace looks up addresses from /usr/lib/debug to live-patch
  • ld-elf.so.1 where needed.
• Improved integration of devel/userns and devel/bmake into bsd.ports.mk such
  • that these tools are created when required by PORTS_SEPARATED_BUILD mode.
• Confirmed that devel/llvm60 and all its dependencies build successfully,
  • this time in a fresh FreeBSD 12.0-RELEASE installation with

    https://github.com/therontarigo/freebsd-ports -b separated as /usr/ports. This selection of ports represents many build tools commonly used in building of other ports: python, perl, lua, GNU, CMake, ...

07/22

• Eliminated the need for any special setup to be able to use
  • PORTS_SEPARATED_BUILD. The mode may be enabled by this one variable, at the command line or in /etc/make.conf.
• Created a lightweight framework for testing large numbers of ports under the
  • separated build mode. It does the following:
    • Handles FLAVORS.
    • Saves success/failure/skipped status.
    • Skips ports which have failed dependencies.
    • Builds each port in its own pristine environment. (This allows
      • concurrent building of unrelated ports)
    • Saves success/failure/skipped status.
    • Skips ports which have failed dependencies.
    • Builds each port in its own pristine environment. (This allows
      • parallel (concurrent) port build to work)
• Attempted build of 2000 ports, selected at random, and their dependencies.
• Identified (and applied temporary fixes for) some ports causing highest
  • number of skipped dependents.
• Retried the skipped ports.
• Developed a plan of a general fix for the most common cause of failure:
  • custom target definitions in port Makefiles running build tools (also from ports) without the userns wrapper.

Latest summary of port building success

• 5613 ports tried
• 2958 (53%) succeeded
• 177 (3%) failed
• 2478 (44%) skipped due to failed dependencies
• Success rate: 94%

Timeline

(Not updated)