Porting FreeBSD to Android Emulator

Project description

Here I will describe the progress of porting FreeBSD to run on top of Android Emulator. Android Emulator runs on a lot of platforms and is easy to set up. It can be therefore used as a unified platform for hacking FreeBSD on ARM.

What's Android Emulator?

It's a fork of Qemu emulator with a new emulated SoC and a board called Goldfish. A somewhat nicer UI is also included.



The Code

The code is based off the 'HEAD' version of FreeBSD cloned around the start of June. You can check it out via

svn co https://socsvn.freebsd.org/socsvn/soc2014/astarasikov

I'm now preparing to merge it upstream. I keep a github branch which I regularly rebase against master and push cleaned-up code

git clone https://github.com/astarasikov/freebsd.git
cd freebsd
git checkout android_goldfish_arm_master

http://benno.id.au/blog/2007/11/29/android-qemu - Description of the emulated board (Goldfish). Oh, and the up-to-date emulator does not contain the mentioned bugs :)

https://android.googlesource.com/platform/external/qemu.git/+/master/docs/GOLDFISH-VIRTUAL-HARDWARE.TXT - documentation for the emulated hardware from Android Emulator


Android Emulator supports ARMv5 (ARM926E) and ARM1136 cores, but not the coprocessor register instruction for domain control for these SoCs. Since we want to run on a vanilla emulator, let's just rely on the fact that the emulator also supports emulating ARM Cortex-A8 and only build FreeBSD for ARMv7

The latest version of Android Emulator is identical to QEMU in that it loads the kernel binary to 0x10000 (yes, 64K, not 1M) and jumps there. So let's just do what the versatilepb board does - link FreeBSD to 0x100000 (1M) and set up a trampoline. It should theoretically be possible to save some memory by linking to 0x10000 but I assume the kernel initially maps memory by 1M sections and that will fail unless additional patching is done.

Most importantly - the emulator binaries from Android SDK contain a bug in MMU that does not occur in Linux but leads to data corruption in FreeBSD and being unable to boot kernel. To work around, you can use the prebuilt binaries from https://android.googlesource.com/platform/prebuilts/android-emulator (branch "l-preview").

How to compile Android Emulator on FreeBSD

I have ported the Android Emulator source code to compile natively on FreeBSD. Currently graphics are broken but console works. I think it should be possible to have it fully working with minor patches.

git clone https://github.com/astarasikov/qemu.git
cd cd android_external_gtest
git checkout cm-11.0
cd ../qemu.git
git checkout l-preview-freebsd
bash android-build-freebsd.sh

How to compile

export TARGET=arm
export TARGET_ARCH=armv6
export DESTDIR=/root/handhelds/arm
pushd .
cd /usr/src/
make -j10 buildkernel KERNCONF=GOLDFISH

How to build a qemu-compatible zImage

The same way as Versatile PB.

rm ./header
rm ./flash
/usr/bin/printf "\0\0\240\343" > header
/usr/bin/printf "\0\020\240\343" >> header
/usr/bin/printf "\0\040\240\343" >> header
/usr/bin/printf "\0\060\240\343" >> header
/usr/bin/printf "\001\366\240\343" >> header
dd of=flash bs=1M count=4 if=/dev/zero
dd of=flash bs=1 conv=notrunc if=header
dd of=flash bs=64k oseek=15 conv=notrunc if=$KERN

How to run

Create an Android Virtual device (install the SDK, run 'android'). Be sure to select an ARMv7A target (basically, installing an SDK for Android 4.X will make ARMv7A the default target). Name your AVD (Android Virtual Device) 'FreeBSD'.

Launch it from console specifying the avd name and the kernel file name ('flash' in this case)

emulator -shell-serial stdio -show-kernel -shell -avd FreeBSD  -kernel flash -qemu

You can get a prebuilt kernel from https://drive.google.com/file/d/0B7wcN-tOkdeRN0lRUDJKa2pWM0U/edit?usp=sharing

You can use the rootfs from Raspberry Pi to test. For example, use the following SD Card image: http://ftp4.us.freebsd.org/pub/FreeBSD/snapshots/arm/armv6/ISO-IMAGES/10.0/FreeBSD-10.0-STABLE-arm-armv6-RPI-B-20140729-r269271.img.bz2

To test, add the "sd card" device to the android AVD. Then, go to "~/.android/avd/YOUR_DEV_NAME.avd". Replace sdcard.img with the downloaded image. Also, edit the .ini file and write the correct sd card size in megabytes (you can also do it while configuring the AVD).

Debugging Bootup issues

In this section I describe the approach I used for debugging the initialization process of the FreeBSD kernel in Android Emulator (modified QEMU) on ARM.

VM Setup

I was using Mac OS X 10.9.4 as a Host OS, and I was using FreeBSD 10.0 in a VirtualBox VM for cross-compiling the kernel. I have configured the VM to bridge the virtual network adapter with my WiFi connection. My workflow was the following: I logged into the VM by SSH, launched the compilation, and then pulled the compiled kernel and debugging symbols to a directory in a Host OS. Then, I launched the Android Emulator in the Host OS.

In the subsequent sections, the following conventions are used: VM IP Address -> Working directory in the Host OS -> /Users/alexander/Documents/handhelds/bsd/ VM RootFS mounted to Host OS -> /Users/alexander/Documents/handhelds/bsd/mnt/

GDB Tricks

I have chosen the following approach to debugging: use QEMU's GDB server and a GDB from a cross-compiler toolchain. I confess I was a bit lazy and decided not to build GDB in FreeBSD, but used the Linaro toolchain (on both Linux and Mac OS X).

I also mounted the FreeBSD root file system from the VM via the sshfs. The primary reason to do so was to allow GDB to find sources (note that we need to use substitute-path to convert absolute paths to our local mount path). Besided, it allows to use a GUI text editor which in the Host OS, which may be considered useful by some people.

I used the following script (which I named fbsd.gdb) for the GDB and I invoked GDB as arm-none-eabi-gdb -x fbsd.gdb.

sshfs mnt/

target remote tcp::1234
add-symbol-file ./kernel.debug 0xc0100000
set substitute-path / /Users/alexander/Documents/handhelds/bsd/mnt/

scp . || exit -1
scp . || exit -1
killall -9 emulator || true
emulator -shell-serial stdio -show-kernel -shell -avd FreeBSD  -kernel flash -qemu -s -S

Increasing Boot Verbosity

One problem that I've faced is no matter how many options I enable (DDB, KDB, GDB, VERBOSE_SYSINIT), all loaded modules and init functions were printed as hex addresses instead of symbol name strings.

JFYI, I have pushed the relevant changes to https://github.com/astarasikov/freebsd/tree/android_goldfish_arm_master

Of course, it would be possible to use GDB to symbolize each address, but that would be boring. So, I added a couple hacks to the kernel:

In kern/kern_module.c at function module_register_init, I have added a debugging print to echo the module name.

printf(" module name='%s' ", data->name);

In kern/init_main.c, I have replaced the DB_STGY_PROC with DB_STGY_ANY to make the kernel try to resolve the function name within all symbols.

In case the symbol is not resolved, I used the function db_printsym. Unlike symbol_name (which used db_search_symbol), this function performs a fuzzy lookup, so if the function pointer is not exactly at its entry point, it will still print the function name and the offset unlike db_search_symbol which will fail and result in printing the HEX address as is.

--- kern/init_main.c    (revision 270039)
+++ kern/init_main.c    (working copy)
@@ -263,14 +263,24 @@
                        const char *func, *data;
                        func = symbol_name((vm_offset_t)(*sipp)->func,
-                           DB_STGY_PROC);
+                           DB_STGY_ANY);
                        data = symbol_name((vm_offset_t)(*sipp)->udata,
                        if (func != NULL && data != NULL)
                                printf("   %s(&%s)... ", func, data);
                        else if (func != NULL)
                                printf("   %s(%p)... ", func, (*sipp)->udata);
+                       {
+                           printf("   ");
+                               db_printsym((vm_offset_t)(*sipp)->func, DB_STGY_ANY);
+                               printf("(&");
+                               db_printsym((vm_offset_t)(*sipp)->udata, DB_STGY_ANY);
+                               printf(")... ");
+                       }
+                       if (0)
                                printf("   %p(%p)... ", (*sipp)->func,

So, hex address like 0xc010a27c(0) becomes something like "some_cool_function+0x20"


SummerOfCode2014/PortingToAndroidEmulator (last edited 2018-04-01 01:22:12 by MarkLinimon)