How to build and use QEMU User Mode on FreeBSD

To build from qemu-1.7.0-rc0 sources and bsd-user patches (

1. Get the source.

% git clone

2. Checkout the bsd-user branch:

% git checkout bsd-user

2. Add Andreas Tobler's patch to fix FreeBSD build breakage:

3. Configure. I just use:

./configure --target-list=mips64-bsd-user,mips-bsd-user,arm-bsd-user --prefix=/usr/local --disable-linux-user --disable-linux-aio --disable-kvm --disable-xen --static

which configures qemu to build statically linked qemu user mode emulators for mips, mips64, and arm perfect to drop in your target cross built area.

Usage Notes:

* Currently only "TARGET=mips TARGET_ARCH=mips64 and "TARGET=arm TARGET=armv6" has enough machine dependent code in place to do everything described below. qemu-mips64 and qemu-arm have been used to cross build 1000's of ports. qemu-mips needs more testing. Use 'qemu-mips64' for mips.mips64 and 'qemu-arm' for arm.armv6. To see the status of these, known bugs, and other missing code see the QemuUserModeToDo page.

* For 64-bit targets (i.e. mips64) use an 64-bit host (i.e. amd64). For 32-bit targets (i.e. armv6) use an 32-bit host (i.e. i386) or compat-32. It may be possible to emulate 32-bit targets on a 64-bit host in the future but currently that is not possible.

* Here are some mips64 static executables to play with.

* To run a statically linked mips64 executable, for example:

% qemu-mips64 ./sh

* To run dynamically linked mips64 executables, install mips64 dynamic libraries and the run-time link-editor in /usr/gnemul/qemu-mips64. (See below about cross building a mips64 root.) To use a different PREFIX instead of /usr/gnemul/qemu-mips64 use the '-L' option..

% qemu-mips64 -L /usr/local/qemu-mips64 ./sh

* QEMU's debugging options are supported. For example:

% qemu-mips64 -strace -d in_asm,out_asm,cpu ./sh

will output strace-like information for system calls and create /tmp/qemu.log with lots additional information. See the QEMU documentation for more information.

To use gdb do something like the following:

% qemu-mips64 -g 4567 ./executable

In another terminal running gdb:

(gdb) target remote 127.1:4567

Cross Building Ports/Packages:

You can use Qemu user mode to help cross build either mips64 or armv6 ports and packages.

1. Build a target root:

For armv6 do

% TARGET=arm
% EMULATOR=qemu-arm
% DESTDIR=/usr/gnemul/${EMULATOR}
% BUILDPATH=arm-bsd-user

For mips64 do

% TARGET=mips
% TARGET_ARCH=mips64
% EMULATOR=${qemu-mips}
% DESTDIR=/usr/gnemul/${EMULATOR}
% BUILDPATH=mips64-bsd-user

Then do a buildworld:

% cd /usr/src
% export MAKEOBJDIRPREFIX=/home/sson/obj
% make -j 8 TARGET=${TARGET} TARGET_ARCH=${TARGET_ARCH} buildworld

2. After a successful build, install in the default image directory as the 'root' user:

If the first time do:

# mkdir -p ${DESTDIR}
# cd /usr/src

If the root has been previously installed then do the following to ensure that no artefacts slip from the previous build:

# chflags -Rf noschg ${DESTDIR}
# rm -Rf ${DESTDIR}
# mkdir -p ${DESTDIR}

Of course, extreme caution is advised doing the above as the 'root' user. For example, misspelling DESTDIR could install the target root into the host environment.

3. Build a statically linked user mode emulator and install it into ${DESTDIR}/usr/local/bin:

Enable the 'STATIC_LINK' port option:

# make config  
(enable both the 'STATIC_LINK' and 'BSD_USER' options)

Configure qemu:

$ make configure

Build and install manually:

$ cd work/qemu-*
$ gmake
$ sudo mkdir -p ${DESTDIR}/usr/local/bin
$ sudo cp ${BUILDPATH}/${EMULATOR} ${DESTDIR}/usr/local/bin/${EMULATOR}

4. Install 'ports' in /usr/gnemul/qemu-mips64/usr/ports:

$ cd /var/tmp
$ fetch
$ cd ${DESTDIR}/usr
$ sudo tar -xpf /var/tmp/ports.tar.gz

5. Mount devfs on ${DESTDIR}/dev

$ sudo mkdir -p ${DESTDIR}/dev
$ sudo mount -t devfs devfs ${DESTDIR}/dev

6. chroot into the target root:

$ sudo chroot -u root ${DESTDIR} /usr/local/bin/${EMULATOR} /bin/sh

Note at this point the host system is running the target executables via user mode emulation. For example:

# file /bin/ls
/bin/ls: ELF 64-bit MSB executable, MIPS, MIPS-III version 1 (FreeBSD), dynamically linked (uses shared libs), FreeBSD-style, for FreeBSD 10.0 (1000027), stripped
# ls
.cshrc          boot            libexec         rescue          tmp
.profile        dev             media           root            usr
COPYRIGHT       etc             mnt             sbin            var
bin             lib             proc            sys

It is recommended, at this point, that you exit the chroot'ed emulated environment and create a tarball snapshot of your clean target root in case you may want to "roll back" to it at some future point:

# exit
$ cd /usr/gnemul/
$ sudo tar -czf ${EMUALTOR}-root.tgz qemu-mips64
$ sudo chroot -u root ${DESTDIR} /usr/local/bin/${EMULATOR} /bin/sh

7. Cross build ports/packages as you would on the native host system:

# cd /usr/ports
# make index  (or 'make fetchindex' for the impatient)
# make search name=lighttpd
Port: lighttpd-1.4.28_4
Path: /usr/ports/www/lighttpd
Info: A secure, fast, compliant, and very flexible Web Server
B-deps: libtool-2.2.10 pcre-8.10 pkg-config-0.25_1
R-deps: pcre-8.10
# cd /usr/ports/www/lighttpd
# make package

(Also create packages for any the runtime dependencies or "R-deps"...)

# cd /usr/ports/devel/pcre
# make package

The packages file can then be copied to the target mips64 system and installed.

Cross Building Using the Host Cross Compiler (and other toolchain friends):

Using the host cross compiler and other toolchain binaries greatly decreases the build time for ports/packages. For example, to build the vim-lite port the following was reported using time(1) and the relative performance compared to the native (amd64) host build:

Pure emulation of mips64 target on amd64 host (using target compiler)

Partial emulation of mips64 target on amd64 host (using host cross compiler)

Host (amd64) build of host binary

1256.73 real

822.33 user

480.20 sys

243.97 real

100.73 user

135.49 sys

63.40 real

44.50 user

19.70 sys

19.82x slower

18.48x slower

24.38x slower

3.84x slower

2.26x slower

6.88x slower




Of course, with most benchmarking, your mileage may vary.

To install a host cross compiler into the chroot area, do the following on the host system:

1. Build the cross compiler toolchain do the following, as 'root':

# cd /usr/src
# mkdir -p /usr/obj
# export MAKEOBJDIRPREFIX=/usr/obj
# make -j 4 toolchain TARGET=${TARGET} TARGET_ARCH=${TARGET_ARCH}

2. Install into the chroot area:

# mkdir -p ${DESTDIR}/usr/obj
# cd /usr/obj
# mv usr ${DESTDIR}/usr/obj

3. chroot into the chroot image area:

# chroot -u root /usr/gnemul/${EMULATOR} /usr/local/bin/${EMULATOR} /bin/sh

4. Change to use local headers:

# cd /usr/obj/${TARGET}.${TARGET_ARCH}/usr/src/tmp/usr
# rm -rf include
# ln -s ../../../../../../include ./include

5. Set up the environment:

# CC=/usr/obj/${TARGET}.${TARGET_ARCH}/usr/src/tmp/usr/bin/cc
# export CC
# CPP=/usr/obj/${TARGET}.${TARGET_ARCH}/usr/src/tmp/usr/bin/cpp
# export CPP
# CXX=/usr/obj/${TARGET}.${TARGET_ARCH}/usr/src/tmp/usr/bin/c++
# export CXX
# AS=/usr/obj/${TARGET}.${TARGET_ARCH}/usr/src/tmp/usr/bin/as
# export AS
# NM=/usr/obj/${TARGET}.${TARGET_ARCH}/usr/src/tmp/usr/bin/nm
# export NM
# RANLIB=/usr/obj/${TARGET}.${TARGET_ARCH}/usr/src/tmp/usr/bin/gnu-ranlib
# export RANLIB
# LD=/usr/obj/${TARGET}.${TARGET_ARCH}/usr/src/tmp/usr/bin/ld
# export LD
# OBJCOPY=/usr/obj/${TARGET}.${TARGET_ARCH}/usr/src/tmp/usr/bin/objcopy
# export OBJCOPY
# SIZE=/usr/obj/${TARGET}.${TARGET_ARCH}/usr/home/src/tmp/usr/bin/size
# export SIZE
# STRIPBIN=/usr/obj/${TARGET}.${TARGET_ARCH}/usr/src/tmp/usr/bin/strip
# export STRIPBIN

You may want to add this to ~/.profile in the chroot area.

6. Cross build ports/packages 5x to 8x faster.

Add the miscellaneous binary image activator patch to FreeBSD

By extending the kernel with this patch it is possible to create an hybrid environment for mixed native and emulated binaries. This could be used in a chroot'ed environment like above to support running /bin/sh and other scripting languages natively for performance or to run foreign binaries transparently on a host of a different architecture.

Once the patch has been installed an image activator can be added for mips64 binaries by using binmiscctl:

# binmiscctl add mips64elf --interpreter "/usr/local/bin/qemu-mips64" \
--magic "\x7f\x45\x4c\x46\x02\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x08" \
--mask  "\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff" \
--size 20 --set-enabled

For arm little endian binaries:

# binmiscctl add armelf --interpreter "/usr/local/bin/qemu-arm" \
--magic "\x7f\x45\x4c\x46\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00" \
--mask  "\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff" \
--size 20 --set-enabled

For arm big endian binaries

# binmiscctl add armebelf --interpreter "/usr/local/bin/qemu-armeb" \
--magic "\x7f\x45\x4c\x46\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28" \
--mask  "\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff" \
--size 20 --set-enabled

For mips32 binaries

binmiscctl add mips32 --interpreter "/usr/local/bin/qemu-mips" \
--magic "\x7f\x45\x4c\x46\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x08" \
--mask "\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff" \
--size 20 --set-enabled

For more information see the binmiscctl(8) man page.

Be sure to see the FreeBSD Qemu User Mode ToDo list for information about missing bits and known bugs.

QemuUserModeHowTo (last edited 2014-04-04 01:45:33 by SeanBruno)