Introduction

Link-Time Optimization (LTO) is a compiler toolchain feature that enables additional program optimizations by deferring optimization passes to link time.

This page covers how to enable Link-Time Optimization for software built from the Ports tree, and work to address LTO related build issues in the ports tree globally.

Contents

  1. Introduction
  2. Status
  3. Testing LTO
  4. Major Issue Classes
    1. Ports not respecting variables from framework
    2. ld: error: undefined symbol: *
    3. strip: file format not recognized
    4. libtool: eval: 1: Syntax error: "|" unexpected
  5. References
    1. Toolchain & Hardening

Status

LTO support is at a very preliminary stage and a work-in-progress. There are a number of major issue classes to work through, which are outlined below.

Currently working together on this is:

We'd love people to get involved, help test and coordinate resolution of issues, you can get in touch at:

Testing LTO

The following setup enables LTO globally (for every port), and will produce many build failures. It is possible to disable LTO on a per-port basis using either:

1. Add the following to /etc/make.conf 

# Enable LTO for ports only
.if !empty(.CURDIR:M/*/ports*/*)
USE_LTO= yes
.endif
# Enable use of Mk/bsd.local.mk
USE_LOCAL_MK= yes

2. Until committed, add the following to path/to/ports/tree/Mk/bsd.local.mk 

# Here is where any code that needs to run at bsd.port.pre.mk inclusion
# time should live.
#

# Note: This setup is for testing purposes only and not ready
# for committing to the ports tree in this state.

.if !empty(USE_LTO)

# Gotta figure out how to make this non-recursive/dependency loop
#.if !empty(.CURDIR:M/*/ports*/devel/binutils)
# Need binutils:strip at the moment
#BUILD_DEPENDS+=${LOCALBASE}/bin/strip:devel/binutils
#.endif

AR=     llvm-ar
RANLIB= llvm-ranlib
NM=     llvm-nm
OBJDUMP=llvm-objdump
STRIP=
STRIPBIN=${STRIP}

LTO_TOOLCHAIN=  AR=${AR} \
                RANLIB=${RANLIB} \
                NM=${NM} \
                OBJDUMP=${OBJDUMP} \
                STRIP=${STRIP} \
                STRIPBIN=${STRIPBIN}

MAKE_ENV+=              ${LTO_TOOLCHAIN}
.if defined(GNU_CONFIGURE)
CONFIGURE_ARGS+=        ${LTO_TOOLCHAIN}
.elif defined(HAS_CONFIGURE)
CONFIGURE_ENV+=         ${LTO_TOOLCHAIN}
.endif
.endif

Major Issue Classes

These are the most prevalent issues currently observed when enabling LTO globally in the ports tree.

Ports not respecting variables from framework

Setting the variables above in Mk/bsd.local.mk identified many ports that were not respecting variables, and where replacing = with += assignments resolved the issue.

The following command identified many ports (4457 as of 2021-11-19) that currently do this (intentionally, or otherwise). We have policies in place for respecting CC/CFLAGS/PREFIX among a couple others, and it's likely that these will need to be addressed as a part of the Ports LTO work as well. 

find /usr/ports -type f -and -not -path '*/files/*' -exec grep -HE '^(CONFIGURE_ARGS=|CONFIGURE_ENV=|MAKE_ENV=|MAKE_ARGS=|CFLAGS=|LDFLAGS=|CXXFLAGS=)' {} +

ld: error: undefined symbol: *

Incomplete or incorrect toolchain setup for ar, nm, objdump, ranlib, strip etc al. This is what LTO_TOOLCHAIN above is for.

strip: file format not recognized

Mostly observed on *.a (static libraries) only.

Our strip in base is from elftoolchain, which doesn't like these lto libraries.

We've tested using different strip programs:

Long term, we probably want to address this upstream in the LLVM toolchain.

In the meantime, elftoolchain improvement is possible, but both llvm toolchain and elftoolchain are provided in base, which means a long wait before all users will have these, so we may not be able to rely on these in the short term.

That leaves binutils, llvm or llvm tools, or elftoolchain (with fixes) from ports/packages. Alternatively, we may be able to massage the strip command invocations to be non-fatal.

See Also:

libtool: eval: 1: Syntax error: "|" unexpected

Appears to only involve projects using -export-symbols-regex in their builds.

Libtool contains the following function that returns empty with LTO_TOOLCHAIN: 

# Take the output of nm and produce a listing of raw symbols and C names.
global_symbol_pipe="sed -n -e 's/^.*[    ]\\([BCDEGRST][BCDEGRST]*\\)[   ][      ]*_\\([_A-Za-z][_A-Za-z0-9]*\\)\$/\\1 _\\2 \\2/p' | sed '/ __gnu_lto/d'"

Which is used later in: 

# The commands to list exported symbols.
export_symbols_cmds="\$NM \$libobjs \$convenience | \$global_symbol_pipe | \$SED 's/.* //' | sort | uniq > \$export_symbols"

This results in | | in the libtool command output, which returns the error.

See Also:

References

Toolchain & Hardening

Reference:

Guides:

Software:

CategoryPorts CategoryProject CategoryPerformance

  1. But it means that libraries don't end up stripped, which is technically a ports framework policy issue and POLA violation. (1)

KubilayKocak/Ports/LinkTimeOptimization (last edited 2022-03-21T22:33:26+0000 by KubilayKocak)