How to build and use QEMU User Mode on FreeBSD

Install statically linked qemu

You just need to install the port emulators/qemu-user-static

Usage Notes:

* Currently only "TARGET=mips TARGET_ARCH=mips64 and "TARGET=arm TARGET_ARCH=armv6" has enough machine dependent code in place to do everything described below. qemu-mips64 and qemu-arm have been used to cross build 1000's of ports. qemu-mips needs more testing. Use 'qemu-mips64' for mips.mips64 and 'qemu-arm' for arm.armv6. To see the status of these, known bugs, and other missing code see the QemuUserModeToDo page.

* For 64-bit targets (i.e. mips64) use an 64-bit host (i.e. amd64). For 32-bit targets (i.e. armv6) use an 32-bit host (i.e. i386) or compat-32. It may be possible to emulate 32-bit targets on a 64-bit host in the future but currently that is not possible.

* Here are some mips64 static executables to play with.

* To run a statically linked mips64 executable, for example:

% qemu-mips64-static ./sh

* To run dynamically linked mips64 executables, install mips64 dynamic libraries and the run-time link-editor in /usr/gnemul/qemu-mips64-static. (See below about cross building a mips64 root.) To use a different PREFIX instead of /usr/gnemul/qemu-mips64-static use the '-L' option..

% qemu-mips64-static -L /usr/local/qemu-mips64 ./sh

* QEMU's debugging options are supported. For example:

% qemu-mips64-static -strace -d in_asm,out_asm,cpu ./sh

will output strace-like information for system calls and create /tmp/qemu.log with lots additional information. See the QEMU documentation for more information.

To use gdb do something like the following:

% qemu-mips64-static -g 4567 ./executable

In another terminal running gdb:

(gdb) target remote 127.1:4567

Cross Building Ports/Packages:

You can use Qemu user mode to help cross build either mips64 or armv6 ports and packages.

1. Build a target root:

For armv6 do

% TARGET=arm
% EMULATOR=qemu-arm-static
% DESTDIR=/usr/gnemul/${EMULATOR}
% BUILDPATH=arm-bsd-user

For mips64 do

% TARGET=mips
% TARGET_ARCH=mips64
% EMULATOR=qemu-mips64-static
% DESTDIR=/usr/gnemul/${EMULATOR}
% BUILDPATH=mips64-bsd-user

Then do a buildworld:

% cd /usr/src
% export MAKEOBJDIRPREFIX=/home/sson/obj
% make -j 8 TARGET=${TARGET} TARGET_ARCH=${TARGET_ARCH} buildworld

2. After a successful build, install in the default image directory as the 'root' user:

If the first time do:

# mkdir -p ${DESTDIR}
# cd /usr/src

If the root has been previously installed then do the following to ensure that no artefacts slip from the previous build:

# chflags -Rf noschg ${DESTDIR}
# rm -Rf ${DESTDIR}
# mkdir -p ${DESTDIR}

Of course, extreme caution is advised doing the above as the 'root' user. For example, misspelling DESTDIR could install the target root into the host environment.

3. Install statically linked user mode emulator into ${DESTDIR}/usr/local/bin:

$ sudo mkdir -p ${DESTDIR}/usr/local/bin
$ sudo cp /usr/local/bin/${EMULATOR} ${DESTDIR}/usr/local/bin/${EMULATOR}

4. Install 'ports' in /usr/gnemul/qemu-mips64/usr/ports:

$ cd /var/tmp
$ fetch
$ cd ${DESTDIR}/usr
$ sudo tar -xpf /var/tmp/ports.tar.gz

5. Mount devfs on ${DESTDIR}/dev

$ sudo mkdir -p ${DESTDIR}/dev
$ sudo mount -t devfs devfs ${DESTDIR}/dev

6. chroot into the target root:

$ sudo chroot -u root ${DESTDIR} /usr/local/bin/${EMULATOR} /bin/sh

Note at this point the host system is running the target executables via user mode emulation. For example:

# file /bin/ls
/bin/ls: ELF 64-bit MSB executable, MIPS, MIPS-III version 1 (FreeBSD), dynamically linked (uses shared libs), FreeBSD-style, for FreeBSD 10.0 (1000027), stripped
# ls
.cshrc          boot            libexec         rescue          tmp
.profile        dev             media           root            usr
COPYRIGHT       etc             mnt             sbin            var
bin             lib             proc            sys

At this point, it is likely that the chroot'ed environment lacks a proper ldconfig setup for runtime-linking shared libraries from non-default locations. This causes, for instance, the installation of pkg(8) from ports to fail. The required hints file /var/run/ can be generated by executing the following command inside the target chroot:

# service ldconfig start

It is recommended, at this point, that you exit the chroot'ed emulated environment and create a tarball snapshot of your clean target root in case you may want to "roll back" to it at some future point:

# exit
$ cd /usr/gnemul/
$ sudo tar -czf ${EMUALTOR}-root.tgz qemu-mips64
$ sudo chroot -u root ${DESTDIR} /usr/local/bin/${EMULATOR} /bin/sh

7. Cross build ports/packages as you would on the native host system:

# cd /usr/ports
# make index  (or 'make fetchindex' for the impatient)
# make search name=lighttpd
Port: lighttpd-1.4.28_4
Path: /usr/ports/www/lighttpd
Info: A secure, fast, compliant, and very flexible Web Server
B-deps: libtool-2.2.10 pcre-8.10 pkg-config-0.25_1
R-deps: pcre-8.10
# cd /usr/ports/www/lighttpd
# make package

(Also create packages for any the runtime dependencies or "R-deps"...)

# cd /usr/ports/devel/pcre
# make package

The packages file can then be copied to the target mips64 system and installed.

Cross Building Using the Host Cross Compiler (and other toolchain friends):

Using the host cross compiler and other toolchain binaries greatly decreases the build time for ports/packages. For example, to build the vim-lite port the following was reported using time(1) and the relative performance compared to the native (amd64) host build:

Pure emulation of mips64 target on amd64 host (using target compiler)

Partial emulation of mips64 target on amd64 host (using host cross compiler)

Host (amd64) build of host binary

1256.73 real

822.33 user

480.20 sys

243.97 real

100.73 user

135.49 sys

63.40 real

44.50 user

19.70 sys

19.82x slower

18.48x slower

24.38x slower

3.84x slower

2.26x slower

6.88x slower




Of course, with most benchmarking, your mileage may vary.

To install a host cross compiler into the chroot area, do the following on the host system:

1. Build the cross compiler toolchain do the following, as 'root':

# cd /usr/src
# mkdir -p /usr/obj
# export MAKEOBJDIRPREFIX=/usr/obj
# env -u TARGET -u TARGET_ARCH make -j 4 toolchain

2. Install into the chroot area:

# mkdir -p ${DESTDIR}/usr/obj
# cd /usr/obj
# mv usr ${DESTDIR}/usr/obj

3. chroot into the chroot image area:

# chroot -u root /usr/gnemul/${EMULATOR} /usr/local/bin/${EMULATOR} /bin/sh

4. Change to use local headers:

# cd /usr/obj/usr/src/tmp/usr
# rm -rf include
# ln -s ../../../../../include ./include

5. Set up the environment:

# export CC=/usr/obj/usr/src/tmp/usr/bin/cc
# export CPP=/usr/obj/usr/src/tmp/usr/bin/cpp
# export CXX=/usr/obj/usr/src/tmp/usr/bin/c++
# export AS=/usr/obj/usr/src/tmp/usr/bin/as
# export NM=/usr/obj/usr/src/tmp/usr/bin/nm
# export RANLIB=/usr/obj/usr/src/tmp/usr/bin/gnu-ranlib
# export LD=/usr/obj/usr/src/tmp/usr/bin/ld
# export OBJCOPY=/usr/obj/usr/src/tmp/usr/bin/objcopy
# export SIZE=/usr/obj/usr/home/src/tmp/usr/bin/size
# export STRIPBIN=/usr/obj/usr/src/tmp/usr/bin/strip

You may want to add this to ~/.profile in the chroot area.

6. Cross build ports/packages 5x to 8x faster.

Add the miscellaneous binary image activator patch to FreeBSD

NOTE: Below patch was committed on HEAD (r264314) and MFC'd for stable/10 (r266272).

By extending the kernel with this patch it is possible to create an hybrid environment for mixed native and emulated binaries. This could be used in a chroot'ed environment like above to support running /bin/sh and other scripting languages natively for performance or to run foreign binaries transparently on a host of a different architecture.

Once the patch has been installed an image activator can be added for mips64 binaries by using binmiscctl:

# binmiscctl add mips64elf --interpreter "/usr/local/bin/qemu-mips64-static" \
--magic "\x7f\x45\x4c\x46\x02\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x08" \
--mask  "\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff" \
--size 20 --set-enabled

For arm little endian binaries:

# binmiscctl add armelf --interpreter "/usr/local/bin/qemu-arm-static" \
--magic "\x7f\x45\x4c\x46\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00" \
--mask  "\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff" \
--size 20 --set-enabled

For arm big endian binaries

# binmiscctl add armebelf --interpreter "/usr/local/bin/qemu-armeb" \
--magic "\x7f\x45\x4c\x46\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28" \
--mask  "\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff" \
--size 20 --set-enabled

For arm64 binaries

/usr/sbin/binmiscctl add arm64 --interpreter "/usr/local/bin/qemu-aarch64" \
--magic "\x7f\x45\x4c\x46\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00" \
--mask  "\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff" \
--size 20 --set-enabled

For mips32 binaries

binmiscctl add mips32 --interpreter "/usr/local/bin/qemu-mips-static" \
--magic "\x7f\x45\x4c\x46\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x08" \
--mask "\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff" \
--size 20 --set-enabled

For powerpc binaries

binmiscctl add powerpc --interpreter "/usr/local/bin/qemu-ppc-static" \
--magic "\x7f\x45\x4c\x46\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x14" \
--mask "\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff" \
--size 20 --set-enabled

For powerpc64 binaries

binmiscctl add powerpc --interpreter "/usr/local/bin/qemu-ppc64-static" \
--magic "\x7f\x45\x4c\x46\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x15" \
--mask "\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff" \
--size 20 --set-enabled

For sparc64 binaries

binmiscctl add sparc64 --interpreter "/usr/local/bin/qemu-sparc64-static" \
--magic "\x7f\x45\x4c\x46\x02\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x2b" \
--mask "\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff" \
--size 20 --set-enabled

For more information see the binmiscctl(8) man page.

Be sure to see the FreeBSD Qemu User Mode ToDo list for information about missing bits and known bugs.

QemuUserModeHowTo (last edited 2015-01-08 17:01:42 by StaceySon)