What do to about /usr/local in toolchains

The question about what to do about adding /usr/local/{include,lib} to various toolchains' search paths. A poll about what the project and its users desires can be found here. It shows there's much interest in changing the status quo.

Two main things are driving this effort. First, there's a desire to make things more consistent. /usr/local/lib is added to the dynamic library search path, but not to the static libraries. The system compiler doesn't add /usr/local to its paths, but the ports-built compilers add it. Ports-build compilers add it before the system defaults, but ports-built binutils add them after. We're also inconsistent with other systems that add it.

Second, there's a desire for the programming environment to be consistent with other systems. There's a desire to have the following steps work like other systems:

pkg install foo
cc prog-using-libfoo.c -lfoo
./a.out

On other systems, this just works. Either because all packages are installed in /usr, or because the system toolchain looks in /usr/local (or where ever foo winds up).

People have been talking about this issue in the project for literally decades. The time is right to stop talking and pick a path forward.

It should be noted that LOCALBASE and /usr/local will be used interchangeably in this document. They are the same thing and reviewers shouldn't nit-pick unless it actually matters. I'll note when they aren't actually the same thing.

Status Quo

The current arrangement of how the toolchains in the system is a bit of a dog's breakfast.

gcc and clang both have similar include search paths. /usr/include is in both of them by default. /usr/local/include is not.
ld and clang both have /lib and /usr/lib in their search paths. /usr/local/lib is not searched by default.
ld.so searches /lib and /usr/lib by default. /etc/rc.d scripts adds /usr/compat/lib, /usr/local/lib and /usr/local/lib/compat/pkg. These libraries are added after /usr/lib.
The gcc port adds ${LOCALBASE}/include before /usr/include.
The binutils port adds ${LOCALBASE}/lib after /usr/lib
Something, and I've not found where, is also adding libraries added by compiler parts.
LOCALBASE is /usr/local. All packages are installed relative to this root.
pkgconf looks for data in /usr/libdata/pkgconfig and /usr/local/libdata/pkgconfig. Other systems look for it in /usr/lib/pkgconfig and /usr/local/lib/pkgconfig.

Detailed Overview of the Problem

A number of problems exist with the current status quo.

Static libraries are handled differently than dynamic ones.
System compiler's include path is inconsistent with ports-built compiler's include path.
System library path in ld is inconsistent with the ports-built ld.
The search order for includes in the ports-built compilers is different than the search order for the libraries in the base system and the ports-built binutils.
Needing to add -I/usr/local/include and -L/usr/local/lib to compilation lines for programs not using the ports system is a major friction point for some users.
The ports system can redefine LOCALBASE to any path, which adds an additional layer of complexity for some external build systems.
pkgconfig looks in /usr/local for its data, but has slightly different paths than Linux

Static vs Dynamic Libraries

In FreeBSD, by default, we include /usr/local/lib (not influenced by LOCALBASE) in the dynamic library search path. The default path is not set in the dynamic linker itself, but via the ld_config_paths series of variables. So dynamic libraries are found by default, but static libraries are not. And the behavior is different at runtime (when they are) and ld time (when they aren't) because ld doesn't use the ldconfig generated cache on FreeBSD. But the very similar LD_LIBRARY_PATH environment variable, when set, is used by both.

Include paths

When building with a ports-built compiler, ${LOCALBASE}/include is searched before /usr/include. When building with the system compiler, /usr/local/include is not searched. This can lead different builds to fail when CC=cc (from the base) versus CC=gcc50 (eg, from ports).

Library paths

Ports-built ld from the binutils port searches ${LOCALBASE}/lib for all libraries, while the system effectively searches just for dynamic libraries. The base system doesn't respect LOCALBASE setting for this. The ports-build ld uses the LOCALBASE setting at the time it was built. It doesn't read LOCALBASE from the environment when it executes.

Includes vs Libraries

There's a disconnect with the ports-built compilers' include path (which is before, so overrides the base system) and the base system and ports-build binutils library path (which is after, so does not override the base system). This is wrong and should be sorted out. There's a longer history of after, but that makes it hard to setup situations where ports override base system components.

Adding options

Other systems don't require adding -I/usr/local/include for packages that are installed. Ours is an odd-man out in requiring it. Some do this by installing into /usr instead of /usr/local. Some do this by including the path in the compiler. We do the worst of both worlds from a convenience point of view. It's a purer environment, but that purity comes at the cost of much friction for independent software developers. In addition, when adding ports, often times this is an extra step required to integrate it into the ports tree.

LOCALBASE

The ports system allows LOCALBASE to be changed from its default of /usr/local to something else. This creates an additional layer of complexity for doing ports. It also poses challenges to integrating /usr/local into the base. LOCALBASE is only evaluated at build-time for ports. The base system would need to grow knowledge of LOCALBASE and people that define it may need to move it from a ports-specific config file to the global config file, or also added a copy to src.conf.

pkgconf

The pkgconf program already looks in /usr/libdata/pkgconfig and /usr/local/libdata/pkgconfig for data to do its job. It is inconsistent with how other systems install their data. Linux packages install it into /usr/local/lib/pkgconfig or /usr/lib/pkgconfig, which our pkgconfig doesn't search. This may be an orthogonal issue, especially since it is fixed by a one line fix to the port's makefile.

Alternatively, though, this could be solved with config.site. Another issue is that --prefix is inconsistently used and honored by ports using autoconf's generated configure script. More research may be needed to determine the actual extent of this issue.

Binaries

/usr/local/bin and /usr/local/sbin aren't in the default path (though to be fair, /usr/sbin and /sbin aren't either). So any programs installed by packages won't be found. This is needed for both the default path and the cron path potentially. Perhaps the best solution to this isn't to change the default path in /bin/sh and /bin/csh, but rather add it to the config files .profile and .login used as a template.

Data from poll / vote

Early results from the polling suggest that there's about an 75/25 split between having this the default behavior vs not having it be the default behavior. There's another 60/40 split between having it be an option you can turn on vs hard coding it.

Feedback from the polling indicates that having it be a BASE option of some flavor would be undesirable from the ports perspective. They would need to make sure that the ports worked with the option turned on and with it turned off, easily doubling the amount of testing work needed to verify new ports and packaging. Having it optional would also mean that configuration files for supporting FreeBSD in upstream projects like ntp, etc would need to contemplate both types of systems, again acting as a friction point for the adaptation of FreeBSD.

There's also some debate as to the proper order. while there's a long history of having /usr/local last in the list, there's a contingent that desires it to be first so we can 'swap out' bits of the base system (like openssl) for bits from ports (maybe even going to libressl globally). While a useful feature, this hasn't been a supported configuration in the system. The defaults of the system always have /lib and /usr/lib before /usr/local/lib. Enabling this feature likely would be much larger effort to do generically. It would be better to wait until we get base system packages, and start to break them down into smaller bits, before exploring this option.

There's also concern that arbitrarily adding stuff from /usr/local/ could taint the build of the base system. While we already disable the default includes and libraries when we're building the system (preferring to build them from the source and then using the fresh, uncontaminated bits), not every build system is as careful as FreeBSD's. In addition, building individual programs from FreeBSD would become riskier since they'd be open to contamination from /usr/local. These scenarios aren't too common, but there's no hard data to show just how uncommon they are.

Julie Mallet reports doing exactly this years ago for a fork of FreeBSD for one of her customers. The issues were relatively small and easy to diagnose. David Chisnel suggests adding it because he's spent a lot of time helping "... non-FreeBSD developers work around this and trying to persuade them that, if they get past the stage of 'FreeBSD can't even find libraries that it installed,' the system has enough positive attributes that they should spend time learning it."

Portsmgr history

Portsmgr consider this in this PR. The notion was to add it in bsd.ports.mk so individual ports would need to cope. At the end of the day, the consensus was that bsd.ports.mk was the wrong place to do this, and individual ports would be corrected as needed.

Path Forward

TDB