Developing and Cross-building for ARM Systems

Developing on the ARM system itself is just like developing FreeBSD for any system using the native tools. For example, to build userland on an ARM system you would just cd /usr/src ; make buildworld. Modern i386 and amd64 systems are often faster than the ARM systems we develop for, which makes cross-building an attractive option. This page focuses on the cross-build process.

Introduction

The FreeBSD build system includes support for cross-building. The only thing needed to start a cross-build instead of a native build is to add TARGET_ARCH= to the make command line, naming an architecture different than the build host. (Some architectures also require you to add TARGET= but doing so is optional for ARM architectures.)

The initial TARGET_ARCH names available for ARM were:

In 10 the only difference is:

As of 12 the latter two are split:

When cross-building, the object files are placed into a separate subdirectory which includes arm.archname as the top level. For example, if you haven't overridden MAKEOBJDIRPREFIX, then the object files from make TARGET_ARCH=armv6 buildworld will be in /usr/obj/arm.armv6/

It is important to consistently specify TARGET_ARCH= on all make commands. If you make buildworld TARGET_ARCH=armv6 and then make installworld DESTDIR=/mnt you will accidentally install the native archicture, not the armv6 world you just built. A simple wrapper script which sets up the environment and command line variables needed to run cross-builds with consistent options can be very helpful; an example appears later in this page.

Where to cross-build

Most people's first exposure to building FreeBSD uses the source installed with the system in /usr/src, but that isn't always the source you want to cross-build for an ARM system. For one thing, FreeBSD development is usually done on the -current branch, and your build host may not be running the latest code. A common option for development is to check out the FreeBSD source into a directory within your home directory, and use the MAKEOBJDIRPREFIX environment variable to place the object files into another directory within your home directory.

If you develop for multiple boards, or have multiple projects going on at once, it can be useful to set up a separate directory that contains src and object and other subdirectories. In effect, each project gets its own separate sandbox to play in. In addition to src and obj directories, an nfsroot directory is useful if you use NFS root filesystems (described later), and a config directory can hold build and kernel options specific to the project. Overall you end up with a sandbox layout that looks something like this: 

  ~/
   projects/
     wandboard/
       config/
       nfsroot/
       obj/
       src/

How to Cross-Build

Shell Game

Please note that all scripts and examples in this article are in /bin/sh script syntax, not csh. Major differences that could cause confusion include the following:

Sudo

Many tasks within the development cycle require root privileges for things like mounting and formatting sdcards, and installing the results of a build. You can easily use the sudo command hundreds of times a day, and before too long you'll find yourself burying sudo in shortcut scripts, and to save your sanity you might consider adding yourself to your sudoers file as a NOPASSWD entry. Unless you're paranoid or just love typing your password a hundred times a day.

Do it

So to get started, create the sandbox and its subdirs, cd into the the sandbox, and get the FreeBSD source code using your favorite version control system: 

svn checkout svn://svn.freebsd.org/base/head src
git clone git://github.com/freebsd/freebsd.git src

After getting a copy of the source you need to cross-build world and kernel. Here's a very minimal build example (which leaves out a few details for now): 

export BASEDIR=$(pwd)
export MAKEOBJDIRPREFIX=$BASEDIR/obj
cd $BASEDIR/src
make buildworld TARGET_ARCH=armv6
make buildkernel TARGET_ARCH=armv6 KERNCONF=IMX6
sudo -E make installworld TARGET_ARCH=armv6 DESTDIR=$BASEDIR/nfsroot
sudo -E make distribution TARGET_ARCH=armv6 DESTDIR=$BASEDIR/nfsroot
sudo -E make installkernelTARGET_ARCH=armv6 KERNCONF=IMX6 DESTDIR=$BASEDIR/nfsroot

The buildworld step starts by building a cross-compiler and a few related tools. It then uses that to do a full world build, where world means basically everything except the kernel and kernel modules; the buildkernel step builds those. The installworld and installkernel steps install all the arm binaries. The distribution step creates directories and installs non-binary files such as the contents of the /etc directory. It's important to do distribution before installkernel.

The DESTDIR in the preceding example points to the nfsroot folder within the sandbox. That is useful when using TFTP to load the kernel and NFS for the root filesystem on a development board. If you want to install directly to an sdcard or USB drive instead, mount that filesystem and point DESTDIR to the mount, for example: 

sudo mount /dev/da0s2a /mnt
sudo -E make installworld distribution installkernel DESTDIR=/mnt
sudo umount /mnt

More information about ARM sdcard images.

Note that you do typically need root privileges to run the installation steps because of some of the commands and file ownership issues involved. There is work in progress to allow creation of images without privileges.

Pesky details

A detail left out of the simple build example above is the UBLDR_LOADADDR=0xnnnnnnnn value which must be provided on the buildworld command line. This is a tedious little detail currently required for ARM systems. The loader(8) flavor used on most ARM systems is 'ubldr' (U-Boot loader), and it currently has to be linked at a fixed address. The address is different for every board or system. Typically the address is whatever the U-Boot on your system has set in its loadaddr environment variable.

Another detail omitted above involves the make.conf and src.conf files used by the FreeBSD build system. Normally it will look for /etc/make.conf and /etc/src.conf and use whatever variables are set in them during the cross-build. The files in the /etc directory contain values for the build machine itself, which may not be appropriate for the cross-build target (CPUTYPE=core2 would cause ARM build errors, for example). To avoid such problems, you must set __MAKE_CONF=<path> and srcconf=<path> on the make command line. The path can be /dev/null or even a non-existant file. If you need to set ARM-related values for your build, you can put them in config/make.conf and config/src.conf in the build sandbox. The values in make.conf apply to all builds (including ports), whereas the values in src.conf apply only to building the FreeBSD source. For this cross-build process, all the source being built is FreeBSD source.

Develop, Compile, Test

Now you have an sdcard image that boots your development system using the shiny new copy of FreeBSD you just built, (or you've set up the NFS root filesystem and load the kernel using TFTP)... you need to start hacking on some code and test your changes! Of course, after you've done some hacking, rebuilt, got the results onto the sdcard, rebooted from it, tested, now you need to make another change... man, this stuff takes too long! Here are some tips to speed up the development cycles.

The cross-build process is completely "-j safe", meaning you can run as many parallel build processes as you have CPU and memory resources for, by adding -jN to the make command, where N is the number of parallel build processes you want to run. If you have very fast (SSD) disk storage, a good value for N is the number of CPUs in your build system (the value of sysctl hw.ncpu). If you use traditional spinning disk drives with their slower seek times, use 1.5 times hw.ncpu. (That's not a mistake -- use more jobs for slower disks; while some jobs are waiting for disk IO others can be using the CPUs.)

By default, the build process for both kernel and world starts by deleting everything in the corresponding area of your object directory, so that it can be built over from scratch. No matter how fast your build machine is, that gets old fast. The solution is to add -DNO_CLEAN to the make command line. NO_CLEAN lets make do the job it was designed to do: figure out which source files have changed since the last build, and rebuild just those files. To do that it must visit every directory, so it still takes a while. If you use a wrapper script that contains the -DNO_CLEAN there is no extra option you can give it on the command line that overrides it so that the cleaning happens. Instead, just use rm -rf obj/* in your sandbox. You should never need root privs to clean out the obj directory unless you accidentally use sudo on a buildworld or buildkernel.

When you are working on one specific area (other than the kernel) you can speed up a rebuild by building in just the directory you're working in. Suppose you're adding a new feature to syslogd... to rebuild just it, add SUBDIR_OVERRIDE=usr.sbin/syslogd to the make command. It still visits many directories, to check whether any tools or libraries need rebuilding, but it saves some time.

When hacking on the kernel, another build bottleneck is regenerating the include file dependencies, even though they rarely change. You can eliminate the time spent on this by adding -DNO_KERNELDEPEND to the make command. If any of the changes you've made added or removed #include statements, do a buildkernel without using that flag so that it regenerates the dependencies once, then go back to using it again.

When you're working on a userland app or library, the fastest way to get a rebuilt copy of it onto your test system might be to use scp(1). You can even skip the installworld step and copy the binary directly from your obj dir to the target system. Some things such as always-running daemons and libraries such as libc may require you to scp to a temp file, drop the target system to single-user mode and mv them to their rightful place/name, then exit single-user mode to test them.

For a kernel module you can scp the module, then kldunload/kldload to load the copy.

For the kernel itself, of course you must fully reboot to test changes. You can replace the kernel on a running system using scp, at least until you make a kernel change that prevents it from booting all the way. At that point your only option is to move the sdcard back to the build system and copy a new kernel onto it. That happens to me enough that I have a little script called updkernel to make it easier: 

#!/bin/sh
#  updkernel [device]
dev=${1:-da0}
if [ -r nfsroot/boot/kernel/kernel ] ; then
    kernel=nfsroot/boot/kernel/kernel
else
    echo "Can't find a kernel file here"
    exit 1
fi
set -x
sudo cp /dev/null /dev/${dev} # force geom retaste
sudo mount /dev/${dev}s2a /mnt || exit 1
sudo cp ${kernel} /mnt/boot/kernel/
sudo umount /mnt

This contains the hard-coded fact that on all the ARM boards/systems I work on, the FreeBSD root filesystem ends up on partition a of slice 2; you may need to adjust for your systems.

A wrapper script to make it all easier

A lot of stuff has to go onto the make command line for a cross-build, and it doesn't change from one run to the next, so a wrapper script can make things easier. I have a ~/bin/mk script which reads a few variables from a config/mk.conf file and runs the cross-build process. You run this script from the sandbox directory (the one containing config/ and src/), it will cd into the src directory and run make for you, supplying all the standard command line stuff plus anything you add on the mk command line.

I generally use this script like this: 

mk buildworld && mk installworld
mk distribution
mk buildkernel -DNO_KERNELDEPEND && mk installkernel

Note that two separate runs joined with && are required to build then install, so that only the install part gets done with sudo. Often I'll add -DNO_MODULES to both the buildkernel and installkernel targets if I'm doing kernel hacking that doesn't involve modules.

The script looks like this: 

#!/bin/sh
# Cross-build and system-build wrapper.
# See https://wiki.freebsd.org/arm/crossbuild

# If the local dir has a custom mk script, run it instead.
if [ -x ./mk ]; then
    ./mk "$@"
    exit $?
fi

# First set all tweakable variables to default values before loading
# config/mk.conf which can override any of these defaults.
mk_arch=$(sysctl -n hw.machine_arch)
mk_insdir="$(pwd)/nfsroot"
mk_jobs="$(sysctl -n hw.ncpu)"
mk_kernel="GENERIC"
mk_makeconf="$(pwd)/config/make.conf"
mk_mkargs=""
mk_nice="nice -10"
mk_objdir="$(pwd)/obj"
mk_srcconf="$(pwd)/config/src.conf"
mk_srcdir="$(pwd)/src"
mk_ubldraddr="0x0"

# Source in config/mk.conf if it exists.
if [ -r config/mk.conf ] ; then
    . config/mk.conf
fi

# Turn target-arch into target.
case ${mk_arch} in
    arm*)     mk_targ=arm;;
    mips*)    mk_targ=mips;;
    powerpc*) mk_targ=powerpc;;
    *)        mk_targ=${mk_arch};;
esac

# If making a target that requires root privs, automatically add sudo.
# This can be overridden by setting mk_sudo="" in config/mk.conf.
case "$*" in
    *installworld* | *installkernel* | *distribution* | *builddtb* )
        is_install="yes"
        mk_sudo=sudo
        ;;
esac

# Turn target arch into target dir for the kernel config file.
case ${mk_arch} in
    arm*)     target_dir=arm;;
    mips*)    target_dir=mips;;
    powerpc*) target_dir=powerpc;;
    *)        target_dir=${mk_arch};;
esac

# If there is a local kernel config file, link it into the source tree.
if [ -r "config/${mk_kernel}" ] ; then
    ln -fs "../../../../config/${mk_kernel}"   "src/sys/${target_dir}/conf/${mk_kernel}"
fi

if [ ! -e "${mk_srcdir}" ]; then
    echo "Source dir '${mk_srcdir}' doesn't exist"
    exit 1
fi

if [ ! -e "${mk_objdir}" ]; then
    mkdir -p "${mk_objdir}"
fi

if [ ! -r "${mk_makeconf}" ]; then
    echo "Warning: Cannot read '${mk_makeconf}', using /dev/null"
    mk_makeconf="/dev/null"
fi

if [ ! -r "${mk_srcconf}" ]; then
    echo "Warning: Cannot read '${mk_srcconf}', using /dev/null"
    mk_srcconf="/dev/null"
fi

if [ "${is_install}" = "yes" ]; then
    case "$*" in
        *DESTDIR* )
            # DESTDIR on command line, honor it, but verify dir exists.
            ;;
        *)
            destdir_arg="DESTDIR=${mk_insdir}"
            ;;
    esac
fi

#if [ ! -e "${DESTDIR}" ]; then
#    echo "Error: Cannot find install dir '${DESTDIR}'"
#    exit 1
#fi
set -x
case "$*" in
  *buildenv*) jobopt="-B" ;;
  *-j*)       jobopt="" ;;
  *)          jobopt="-j ${mk_jobs}"
esac

# MAKEOBJDIRPREFIX must be in the environment, not on the make command line.
export MAKEOBJDIRPREFIX="${mk_objdir}"

# Do it.
set -x
cd ${mk_srcdir} && time ${mk_nice} ${mk_sudo} make ${BWSILENT} ${jobopt} \
    ${destdir_arg} \
    "-DNO_CLEAN" \
    "TARGET_ARCH=${mk_arch}" \
    "__MAKE_CONF=${mk_makeconf}" \
    "SRCCONF=${mk_srcconf}" \
    "KERNCONF=${mk_kernel}" \
    "UBLDR_LOADADDR=${mk_ubldraddr}" \
    ${mk_mkargs} \
    "$@"

All those variables that begin with "mk_" at the start of the script are things that you can customize by putting your own values in config/mk.conf. The only required config value is the name of the kernel config to build. The only values you're likely to ever set in mk.conf are:

A wrapper for the wrapper

So now you've got this cool wrapper script that does the right thing for cross builds, but it's named "mk" and you have years of finger-memory that makes you type "make" every time. If you use bash, you can fix that with a function that figures out what you really meant when you typed make. Just add this to your .bashrc 

make()
{
    if [ -f config/mk.conf ]; then
        mk "$@"
    else
        nice make "$@"
    fi
}

If you don't want to run all makes as nice processes, just replace nice with command.

CategoryStale CategoryHowTo

arm/crossbuild (last edited 2022-06-09T01:07:02+0000 by KubilayKocak)