• QemuUserModeHowTo

How to build and use QEMU User Mode on FreeBSD

Install statically linked qemu

You just need to install the port emulators/qemu-user-static

Usage Notes:

• Currently only "TARGET=mips TARGET_ARCH=mips64 and "TARGET=arm TARGET_ARCH=armv6" has enough machine dependent code in place to do everything described below. qemu-mips64 and qemu-arm have been used to cross build 1000's of ports. qemu-mips needs more testing. Use 'qemu-mips64' for mips.mips64 and 'qemu-arm' for arm.armv6. To see the status of these, known bugs, and other missing code see the QemuUserModeToDo page.

• For 64-bit targets (i.e. mips64) use an 64-bit host (i.e. amd64). For 32-bit targets (i.e. armv6) use an 32-bit host (i.e. i386) or compat-32. It may be possible to emulate 32-bit targets on a 64-bit host in the future but currently that is not possible.

• Here are some mips64 static executables to play with.

• To run a statically linked mips64 executable, for example:

% qemu-mips64-static ./sh
$

• To run dynamically linked mips64 executables, install mips64 dynamic libraries and the run-time link-editor in /usr/gnemul/qemu-mips64-static. (See below about cross building a mips64 root.) To use a different PREFIX instead of /usr/gnemul/qemu-mips64-static use the '-L' option..

% qemu-mips64-static -L /usr/local/qemu-mips64 ./sh
$

• QEMU's debugging options are supported. For example:

% qemu-mips64-static -strace -d in_asm,out_asm,cpu ./sh

will output strace-like information for system calls and create /tmp/qemu.log with lots additional information. See the QEMU documentation for more information.

Using Qemu User-Mode for Cross Building Ports/Packages

You can use Qemu user mode to help cross build either mips64 or armv6 ports and packages.

1. Build a target root:

For armv6 do

% TARGET=arm
% TARGET_ARCH=armv6
% EMULATOR=qemu-arm-static
% DESTDIR=/usr/gnemul/${EMULATOR}
% BUILDPATH=arm-bsd-user

For mips64 do

% TARGET=mips
% TARGET_ARCH=mips64
% EMULATOR=qemu-mips64-static
% DESTDIR=/usr/gnemul/${EMULATOR}
% BUILDPATH=mips64-bsd-user

Then do a buildworld:

% cd /usr/src
% export MAKEOBJDIRPREFIX=/home/sson/obj
% make -j 8 TARGET=${TARGET} TARGET_ARCH=${TARGET_ARCH} buildworld

2. After a successful build, install in the default image directory as the 'root' user:

If the first time do:

# mkdir -p ${DESTDIR}
# cd /usr/src
# make DESTDIR=${DESTDIR} TARGET=${TARGET} TARGET_ARCH=${TARGET_ARCH} installworld
# make DESTDIR=${DESTDIR} TARGET=${TARGET} TARGET_ARCH=${TARGET_ARCH} distribution

If the root has been previously installed then do the following to ensure that no artefacts slip from the previous build:

# chflags -Rf noschg ${DESTDIR}
# rm -Rf ${DESTDIR}
# mkdir -p ${DESTDIR}
# make DESTDIR=${DESTDIR} TARGET=${TARGET} TARGET_ARCH=${TARGET_ARCH} installworld
# make DESTDIR=${DESTDIR} TARGET=${TARGET} TARGET_ARCH=${TARGET_ARCH} distribution

Of course, extreme caution is advised doing the above as the 'root' user. For example, misspelling DESTDIR could install the target root into the host environment.

3. Install statically linked user mode emulator into ${DESTDIR}/usr/local/bin:

$ sudo mkdir -p ${DESTDIR}/usr/local/bin
$ sudo cp /usr/local/bin/${EMULATOR} ${DESTDIR}/usr/local/bin/${EMULATOR}

4. Install 'ports' in /usr/gnemul/qemu-mips64/usr/ports:

$ cd /var/tmp
$ fetch ftp://ftp.freebsd.org/pub/FreeBSD/ports/ports/ports.tar.gz
$ cd ${DESTDIR}/usr
$ sudo tar -xpf /var/tmp/ports.tar.gz

5. Mount devfs on ${DESTDIR}/dev

$ sudo mkdir -p ${DESTDIR}/dev
$ sudo mount -t devfs devfs ${DESTDIR}/dev

6. chroot into the target root:

$ sudo chroot -u root ${DESTDIR} /usr/local/bin/${EMULATOR} /bin/sh
#

Note at this point the host system is running the target executables via user mode emulation. For example:

# file /bin/ls
/bin/ls: ELF 64-bit MSB executable, MIPS, MIPS-III version 1 (FreeBSD), dynamically linked (uses shared libs), FreeBSD-style, for FreeBSD 10.0 (1000027), stripped
# ls
.cshrc          boot            libexec         rescue          tmp
.profile        dev             media           root            usr
COPYRIGHT       etc             mnt             sbin            var
bin             lib             proc            sys

At this point, it is likely that the chroot'ed environment lacks a proper ldconfig setup for runtime-linking shared libraries from non-default locations. This causes, for instance, the installation of pkg(8) from ports to fail. The required hints file /var/run/ld-elf.so.hints can be generated by executing the following command inside the target chroot:

# service ldconfig start
#

It is recommended, at this point, that you exit the chroot'ed emulated environment and create a tarball snapshot of your clean target root in case you may want to "roll back" to it at some future point:

# exit
$ cd /usr/gnemul/
$ sudo tar -czf ${EMUALTOR}-root.tgz qemu-mips64
$ sudo chroot -u root ${DESTDIR} /usr/local/bin/${EMULATOR} /bin/sh

7. Cross build ports/packages as you would on the native host system:

# cd /usr/ports
# make index  (or 'make fetchindex' for the impatient)
# make search name=lighttpd
Port: lighttpd-1.4.28_4
Path: /usr/ports/www/lighttpd
Info: A secure, fast, compliant, and very flexible Web Server
Maint: mm@FreeBSD.org
B-deps: libtool-2.2.10 pcre-8.10 pkg-config-0.25_1
R-deps: pcre-8.10
WWW: http://www.lighttpd.net/
# cd /usr/ports/www/lighttpd
# make package

(Also create packages for any the runtime dependencies or "R-deps"...)

# cd /usr/ports/devel/pcre
# make package

The packages file can then be copied to the target mips64 system and installed.

Cross Building Using the Host Cross Compiler (and other toolchain friends):

Using the host cross compiler and other toolchain binaries greatly decreases the build time for ports/packages. For example, to build the vim-lite port the following was reported using time(1) and the relative performance compared to the native (amd64) host build:

Of course, with most benchmarking, your mileage may vary.

To install a host cross compiler into the chroot area, do the following on the host system:

1. Build the cross compiler toolchain do the following, as 'root':

# cd /usr/src
# mkdir -p /usr/obj
# export MAKEOBJDIRPREFIX=/usr/obj
# env -u TARGET -u TARGET_ARCH make -j 4 toolchain

2. Install into the chroot area:

# mkdir -p ${DESTDIR}/usr/obj
# cd /usr/obj
# mv usr ${DESTDIR}/usr/obj

3. chroot into the chroot image area:

# chroot -u root /usr/gnemul/${EMULATOR} /usr/local/bin/${EMULATOR} /bin/sh

4. Change to use local headers:

# cd /usr/obj/usr/src/tmp/usr
# rm -rf include
# ln -s ../../../../../include ./include

5. Set up the environment:

# export CC=/usr/obj/usr/src/tmp/usr/bin/cc
# export CPP=/usr/obj/usr/src/tmp/usr/bin/cpp
# export CXX=/usr/obj/usr/src/tmp/usr/bin/c++
# export AS=/usr/obj/usr/src/tmp/usr/bin/as
# export NM=/usr/obj/usr/src/tmp/usr/bin/nm
# export RANLIB=/usr/obj/usr/src/tmp/usr/bin/gnu-ranlib
# export LD=/usr/obj/usr/src/tmp/usr/bin/ld
# export OBJCOPY=/usr/obj/usr/src/tmp/usr/bin/objcopy
# export SIZE=/usr/obj/usr/home/src/tmp/usr/bin/size
# export STRIPBIN=/usr/obj/usr/src/tmp/usr/bin/strip

You may want to add this to ~/.profile in the chroot area.

6. Cross build ports/packages 5x to 8x faster.

Add the miscellaneous binary image activator patch to FreeBSD

NOTE: Below patch was committed on HEAD (r264314) and MFC'd for stable/10 (r266272).

By extending the kernel with this patch it is possible to create an hybrid environment for mixed native and emulated binaries. This could be used in a chroot'ed environment like above to support running /bin/sh and other scripting languages natively for performance or to run foreign binaries transparently on a host of a different architecture.

Once the patch has been installed an image activator can be added for mips64 binaries by using binmiscctl:

# binmiscctl add mips64elf --interpreter "/usr/local/bin/qemu-mips64-static" \
--magic "\x7f\x45\x4c\x46\x02\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x08" \
--mask  "\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff" \
--size 20 --set-enabled

For arm little endian binaries:

# binmiscctl add armelf --interpreter "/usr/local/bin/qemu-arm-static" \
--magic "\x7f\x45\x4c\x46\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00" \
--mask  "\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff" \
--size 20 --set-enabled

For arm big endian binaries

# binmiscctl add armebelf --interpreter "/usr/local/bin/qemu-armeb" \
--magic "\x7f\x45\x4c\x46\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28" \
--mask  "\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff" \
--size 20 --set-enabled

For arm64 binaries

/usr/sbin/binmiscctl add arm64 --interpreter "/usr/local/bin/qemu-aarch64" \
--magic "\x7f\x45\x4c\x46\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00" \
--mask  "\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff" \
--size 20 --set-enabled

For mips32 binaries

binmiscctl add mips32 --interpreter "/usr/local/bin/qemu-mips-static" \
--magic "\x7f\x45\x4c\x46\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x08" \
--mask "\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff" \
--size 20 --set-enabled

For powerpc binaries

binmiscctl add powerpc --interpreter "/usr/local/bin/qemu-ppc-static" \
--magic "\x7f\x45\x4c\x46\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x14" \
--mask "\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff" \
--size 20 --set-enabled

For powerpc64 binaries

binmiscctl add powerpc --interpreter "/usr/local/bin/qemu-ppc64-static" \
--magic "\x7f\x45\x4c\x46\x02\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x15" \
--mask "\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff" \
--size 20 --set-enabled

For sparc64 binaries

binmiscctl add sparc64 --interpreter "/usr/local/bin/qemu-sparc64-static" \
--magic "\x7f\x45\x4c\x46\x02\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x2b" \
--mask "\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff" \
--size 20 --set-enabled

For more information see the binmiscctl(8) man page.

Using Qemu User-Mode for Cross Development

Qemu user-mode will write a target compatible core that can be used with either target or cross debugging tools such as gdb(1), lldb(1), and procstat(1). For example, using the following simple program, dumpcore.c:

#include <stdlib.h>
#include <stdio.h>

int
main(int argc, char **argv)
{
        printf("Hello %s\n", argv[1]);

        abort();
}

Compile, create coredump, and debug:

# chroot arm
# cd tmp
# cc -g -o dumpcore dumpcore.c
# ./dumpcore arg1
Hello arg1
qemu: uncaught target signal 6 (Abort trap) - core dumped
Abort
# gdb dumpcore qemu_dumpcore.core 
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "armv6-marcel-freebsd"...

Core was generated by `./dumpcore'.
Program terminated with signal 6, Aborted.
Reading symbols from /lib/libc.so.7...done.
Loaded symbols for /lib/libc.so.7
Reading symbols from /libexec/ld-elf.so.1...done.
Loaded symbols for /libexec/ld-elf.so.1
#0  0x420ba314 in thr_kill () from /lib/libc.so.7
(gdb) list
1       #include <stdlib.h>
2       #include <stdio.h>
3       
4       int
5       main(int argc, char **argv)
6       {
7               printf("Hello %s\n", argv[1]);
8       
9               abort();
10      }
(gdb) bt
#0  0x420ba314 in thr_kill () from /lib/libc.so.7
#1  0x420ba300 in raise () from /lib/libc.so.7
#2  0x420ba278 in abort () from /lib/libc.so.7
#3  0x00008668 in main (argc=2, argv=0x9fffecdc) at dumpcore.c:9
(gdb) quit
# procstat -l qemu_dumpcore.core
  PID COMM             RLIMIT                  SOFT             HARD     
53221 dumpcore         cputime             infinity         infinity     
53221 dumpcore         filesize            infinity         infinity     
53221 dumpcore         datasize               32768 MB         32768 MB  
53221 dumpcore         stacksize             524288 KB        524288 KB  
53221 dumpcore         coredumpsize        infinity         infinity     
53221 dumpcore         memoryuse           infinity         infinity     
53221 dumpcore         memorylocked        infinity         infinity     
53221 dumpcore         maxprocesses           15872            15872     
53221 dumpcore         openfiles             352773           352773     
53221 dumpcore         sbsize              infinity         infinity     
53221 dumpcore         vmemoryuse          infinity         infinity     
53221 dumpcore         pseudo-terminals    infinity         infinity     
53221 dumpcore         swapuse             infinity         infinity     
53221 dumpcore         kqueues             infinity         infinity     
# procstat -c qemu_dumpcore.core
  PID COMM             ARGS                                                 
53221 dumpcore         ./dumpcore arg1
# procstat -e qemu_dumpcore.core
  PID COMM             ENVIRONMENT                                          
53221 dumpcore         LANG=en_US.UTF-8 USER=sson LOGNAME=sson HOME=/root MAIL=/var/mail/sson PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/root/bin TERM=xterm-256color BLOCKSIZE=K SHELL=/bin/csh SSH_CLIENT=XXX.XXX.XXX.XXX 53512 22 SSH_CONNECTION=XXX.XXX.XXX.XXX 53512 XXX.XXX.XXX.XXX 22 SSH_TTY=/dev/pts/6 HOSTTYPE=FreeBSD VENDOR=acorn OSTYPE=FreeBSD MACHTYPE=arm32 SHLVL=3 PWD=/tmp GROUP=devel HOST=crack.ysv.freebsd.org REMOTEHOST=XXX.XXX.XXX.XXX EDITOR=vim PAGER=less

Since the ptrace() system call is not currently supported attempting to using a debugger like gdb directly will not work. Instead, do something like the following:

% qemu-mips64-static -g 4567 ./executable

In another terminal running gdb:

(gdb) target remote 127.1:4567

Be sure to see the FreeBSD Qemu User Mode ToDo list for information about missing bits and known bugs.

CategoryVirtualization

CategoryHowTo