Booting FreeBSD on an ARM SoC chip is a multi-stage process involving several distinct software components. This article describes the boot process using the Atmel AT91RM9200 SoC as an example.

FreeBSD includes all the source code required to boot an AT91RM9200 chip without requiring any "binary blob" from a vendor or source code downloaded from other locations. This makes it an ideal example for explaining the very early parts of the boot process, which are often hidden in such vendor-supplied components when you buy an evaluation board or one of the numerous small ARM systems available.

While the early part of the boot process described here is specific to the Atmel chip, the general principles apply to all systems, and once control is handed to the kernel entry point the process is the same for all ARM chips.

Within The Chip

SoC chips typically contain a small amount of SRAM memory, and a small amount of boot firmware in an internal ROM. The memory is typically on the order of 16-64K. At power-on the ROM code does very minimal hardware setup, enabling processor and peripherals just enough to locate a boot image. Because resources are limited, the ROM program is generally very small and limitted to checking simple busses such as SPI, I2C, NOR flash, and NAND flash, and often as a last resort an RS-232 serial debug port. The datasheet for the SoC usually describes this builtin boot program in detail.

The builtin ROM boot code often doesn't know anything about the board it's attached to. In particular it doesn't know how much and what kind of memory is attached, which of its multifunction pins are actually connected to a bus or device as opposed being used for a status LED, etc. It is the responsibility of the first-level bootloader code to initialize the chip for the attached hardware, at least enough to proceed to the next boot stage.

The AT91RM9200

The Atmel chip has 16K of internal static ram. It will attempt to locate a file on an attached SPI dataflash chip or I2C eeprom by matching a signature pattern in the first few bytes read from the device. If it matches, the code is downloaded into the SRAM, at physical address 0x00000000. The ROM loader uses 4K at the top of the SRAM as buffer and work space while downloading, so the first-level bootloader code for this chip must fit within 12K.

The First-Level Bootloader

In the FreeBSD source code, the first-level bootloader for the AT91RM9200 is in directory src/sys/boot/arm/at91/boot2 with supporting code in nearby directories. This bootloader sets up the hardware enough to load the kernel (or a next-stage bootloader) from an SD card. The following section describes the general steps it takes to do this, with a particular emphasis on what lives where in memory and how control transfers from one step to the next.

Early Boot Steps

Memory Layout

Managing the memory layout during the early boot process is largely a tedious manual process. The absolute location in physical memory of several important items is specified in source code or source config files. To customize boot2 for a particular memory configuration you need to locate these constants and carefully avoid overlaps.

The values shown in this table are appropriate for a 64MB system. They are not the values currently checked in to FreeBSD.

Memory Location Constants


SVC_STACK_USE (boot 2 stack)






used in kernel's locore.S


This article started out as a readme file containing just the information in this table. The path to locating it began with a kernel image with a 25MB filesystem embedded in it, and the fact that SVC_STACK_USE at that time located the stack 16MB into physical ram.

Other Bootloaders


Many modern boards and ARM-based systems come with U-Boot. Something like the first-level bootloader described above often runs first, and gets the low-level hardware (clocks and ram) running enough to load and launch u-boot. U-boot can be customized with support for a wide variety of storage systems, network interfaces, filesystems, and even utilities to test and format hardware devices and copy data to them.

Generally you need a copy of u-boot that has been customized and compiled for your particular chip and board.


The Kernel

Kernel Flavors


Kernel with elf headers


Kernel without elf headers


Kernel with elf headers and full debugging info


Just symbols for debugging a crash dump with gdb


Gzipped kernel+unzipper trampoline with elf headers


Gzipped kernel+unzipper trampoline w/o elf headers


Uncompressed kernel+trampoline with elf headers


Uncompressed kernel+trampoline w/o elf headers

The .bin versions have the elf headers stripped off, and a bootloader would launch such a kernel by jumping to offset zero (the load address).

For a kernel without trampoline code, the entry point is in locore.S. This is described below.

The trampoline entry point is _startC() in sys/arm/elf_trampoline.c. This is specified in sys/conf/Makefile.arm which actually writes a little asm source file named tmphack.S which jumps to the _startC() routine. Tricky.

The stack on entry to _startC() is still the one from boot2. The first thing _startC() does is move the stack to be about 1MB above the end of the loaded kernel. At this point physical addressing is still in effect; the trampoline code is linked to run at the physical address specified in std.tsc4370, not at the virtual address the kernel is linked for.

The trampoline code decompresses the kernel to the ram that immediately follows the end of the loaded kernel, then copies the uncompressed kernel to the physical address specified in the decompressed elf headers. It sets up rudimentary page tables immediately following the kernel (I think this is unecessary, the kernel will do its own mmu setup), and then sets up the initial kernel entry stack to be right after these page tables.

Custom code we've added (not in stock FreeBSD) copies the tsc_bootinfo struct to the new stack so that it will remain accessible to the kernel during the handoff. (It turns out this may be a bit dodgy -- the struct passed by boot2 lives in boot's bss, so it's essentially sitting in the middle of sdram somewhere, and could have been wiped out by uncompressing the kernel.)

The trampoline jumps to the entry point listed in the kernel's elf header.

The kernel entry point listed in the elf header is the label _start in the file sys/arm/locore.S. When entered directly from a bootloader the MMU is typically disabled and the pc is the physical address of the entry point. When entered from the trampoline the MMU is enabled and the pc is the virtual address. The code in locore.S is designed to work right either way.

The entry code crafts simple startup page tables, enables the MMU (but not caches, MD code does that later), and transitions the pc from physical to virtual addresses. The page tables are placed at the location specified by STARTUP_PAGETABLE_ADDR, which comes from sys/arm/at91/std.tsc4370. These have a short lifetime, the machine-dependent initarm() routine is expected to create and install more complete page tables.

The stack pointer is set to a small (2K) stack in bss, and the initarm() function is called, passing the same values in r0-r3 that were present on entry to _start. Each type of arm Soc has its own initarm() routine which does a lot of machine-specific and board-specific setup. It must preserve any incoming parms from the bootloader. Some early parts of kernel init are called from within initarm(), including cninit() which is the point at which printf() begins to work. The initarm() routine returns a pointer to what will become the supervisor mode stack for the kernel.

When initarm() returns, the code in locore.S installs the stack pointer it returned. At this point there is no longer any dependence on magical fixed addresses in memory that were compiled-in or came from a config file.

Next the code in locore.S calls mi_startup(), which is essentially like main() for the kernel. It should not return.

FreeBSD/arm/BootProcess (last edited 2013-01-09 05:51:13 by IanLepore)