NAND Flash Framework Introduction

This page is a user-oriented tutorial for FreeBSD NAND Flash framework, related references and documentation, developed by Semihalf.

By reading this page you will learn how to use directly attached NAND memory with FreeBSD, how the FreeBSD NAND Flash stack is organised and more.

All examples assume the reference platform is the SheevaPlug computer.

Overview

FreeBSD support for NAND Flash memory consists of several distinct components:

NAND Framework main function is to communicate with NAND devices and to provide character and GEOM devices that can be used by other system entities (at the moment only GEOM approach is supported). Each discovered NAND chip is represented in the /dev/ directory and is named according to the following convention:

NANDsim is a simulator that allows imitiating ONFI-compliant NAND devices as if they were attached to the system via a virtual controller. NANDsim is a part of NAND Framework and is controlled by user space program /usr/sbin/nandsim.

NANDFS is a log-structured filesystem dedicated to run on NAND Flash memory devices. It has the following major features and characteristics:

Parts of NANDFS internal design are derived from the new implementation of the log-structured file system (NILFS) and the original (now legacy) BSD log-structured file system (LFS). They were used as a reference for the NANDFS mainly to leverage the existing modulartity and block management for the benefit of NAND-oriented environments. Each block on a NAND chip is equivalent to a NILFS segment, which allows for optimal segment management including erasing re-used segment, marking segment as bad, etc.

Warning: The user must be aware that NANDFS is a new file system and thus highly experimental. It is not recommended for production use at the moment.

Usage

This part will show you step-by-step how to use NAND Flash memory in FreeBSD. Most of the examples are based on using FreeBSD and NAND stack on SheevaPlug computer and creating such configuration that SheevaPlug would run FreeBSD entirely from NAND memory.

Enabling

By default FreeBSD NAND Flash framework code is disabled. In order to enable user space part a FreeBSD world has to be compiled and installed with WITH_NAND make(8) option. The following example shows how to build world for ARM architecture with NAND support enabled:

root@host:/usr/src # make buildworld TARGET_ARCH=arm -DWITH_NAND

Enabling NAND Flash support in FreeBSD kernel is divided into two parts: enabling NAND Framework and/or enabling NANDFS filesystem.

NAND Framework

To include support for the NAND Framework, the following should be added to kernel configuration:

device  nand

NAND controller and NAND devices (chips) will be discovered during system bootstrap and will have their representation in the devices hierarchy:

[...]
nand0: <Marvell NAND controller> mem 0xf9300000-0xf93fffff on localbus0
nandbus0: <NAND bus> on nand0
lnand0: <Hynix NAND 512MiB 3,3V 8-bit> on nandbus0
lnand0: Found BBT table for chip
[...]

root@sheevaplug:/root # devinfo
nexus0
  fdtbus0
    localbus0
      nand0
        nandbus0
          lnand0
    simplebus0
      ic0
      timer0
      gpio0
      rtc0
      mge0
        miibus0
          e1000phy0
      uart0
      uart1
      cesa0
      ehci0
        usbus0
          uhub0
            umass0
  cryptosoft0
root@sheevaplug:/root # ll /dev/*nand*
crw-r-----  1 root  operator  0x2f Dec 31 23:59 /dev/gnand.raw0
crw-r-----  1 root  operator  0x34 Dec 31 23:59 /dev/gnand.raw0s.root
crw-r-----  1 root  operator  0x33 Dec 31 23:59 /dev/gnand.raw0s.u-boot
crw-r-----  1 root  operator  0x2c Dec 31 23:59 /dev/gnand0
crw-r-----  1 root  operator  0x31 Dec 31 23:59 /dev/gnand0s.root
crw-r-----  1 root  operator  0x30 Dec 31 23:59 /dev/gnand0s.u-boot
crw-rw-rw-  1 root  wheel     0x1a Dec 31 23:59 /dev/nand0.0
crw-rw-rw-  1 root  wheel     0x13 Dec 31 23:59 /dev/nandsim.ioctl
root@sheevaplug:/root # 

NAND File System (NANDFS)

NANDFS can be built and used both as an integral (static) kernel code and dynamically loaded kernel module.

In order to include the NANDFS into the kernel the following build option is used:

options         NANDFS

Please note the NANDFS itself is independent of the underlying storage and even though it has been designed towards applications using NAND chips, it can be freely used on any other storage media. In particular the NAND Framework needs to be enabled separately i.e.:

device          nand

Building

NANDFS is also available as a module in the source directory sys/modules/nandfs, and can be built through the following steps:

root@sheevaplug:/root # cd /usr/src/sys/modules/nandfs
root@sheevaplug:/usr/src/sys/modules/nandfs # make -DWITH_NAND
Warning: Object directory not changed from original /usr/src/sys/modules/nandfs
@ -> /usr/src/sys
machine -> /usr/src/sys/arm/include
awk -f @/tools/vnode_if.awk @/kern/vnode_if.src -p
awk -f @/tools/vnode_if.awk @/kern/vnode_if.src -q
awk -f @/tools/vnode_if.awk @/kern/vnode_if.src -h
...
...
awk -f /usr/src/sys/modules/nandfs/../../conf/kmod_syms.awk nandfs.kld  export_syms | xargs -J% objcopy % nandfs.kld
ld -Bshareable  -d -warn-common -o nandfs.ko nandfs.kld
objcopy --strip-debug nandfs.ko
root@sheevaplug:/usr/src/sys/modules/nandfs # make install -DWITH_NAND
install -o root -g wheel -m 555   nandfs.ko /boot/kernel
kldxref /boot/kernel
kldxref: file isn't dynamically-linked
root@sheevaplug:/usr/src/sys/modules/nandfs #

Note: The kldxref: file isn't dynamically-linked message is harmless

Loading

The NANDFS module can be loaded manually using the kldload(8) command:

root@sheevaplug:/root # kldload nandfs
root@sheevaplug:/root # kldstat
Id Refs Address    Size     Name
 1    3 0xc0900000 58996c   elf kernel
 2    1 0xc4479000 25000    nandfs.ko
root@sheevaplug:/root # 

Another way to load the module is implicit via mounting the NANDFS; in this case the module will be loaded automatically:

root@sheevaplug:/root # kldstat
Id Refs Address    Size     Name
 1    1 0xc0900000 58996c   elf kernel
root@sheevaplug:/root # mount -t nandfs /dev/gnand0s.root /mnt/
root@sheevaplug:/root # ll /mnt/
total 2
-rw-r--r--   1 root  wheel    0 Jan  1 00:20 .nandfs
root@sheevaplug:/root # kldstat 
Id Refs Address    Size     Name
 1    3 0xc0900000 58996c   elf kernel
 2    1 0xc4479000 25000    nandfs.ko

NAND chip simulator (NANDsim)

NANDsim can be built and used both as an integral (static) kernel code and dynamically loaded kernel module.

Some NANDsim features rely on the ability to log contents to the file, which is achieved through the alq(9) FreeBSD facility. In order for the NANDsim to work the following option should be added to kernel configuration:

options ALQ
device nandsim

Building

NANDsim is also available as a module in the source directory sys/modules/nandsim, and can be built through the following steps:

root@sheevaplug:/root # cd /usr/src/sys/modules/nandsim/
root@sheevaplug:/usr/src/sys/modules/nandsim # make -DWITH_NAND
Warning: Object directory not changed from original /usr/src/sys/modules/nandsim
@ -> /usr/src/sys
machine -> /usr/src/sys/arm/include
awk -f @/tools/makeobjops.awk @/kern/bus_if.m -h
awk -f @/tools/makeobjops.awk @/kern/device_if.m -h
awk -f @/tools/vnode_if.awk @/kern/vnode_if.src -p
...
...
awk -f /usr/src/sys/modules/nandsim/../../conf/kmod_syms.awk nandsim.kld  export_syms | xargs -J% objcopy % nandsim.kld
ld -Bshareable  -d -warn-common -o nandsim.ko nandsim.kld
objcopy --strip-debug nandsim.ko
root@sheevaplug:/usr/src/sys/modules/nandsim # make install -DWITH_NAND
install -o root -g wheel -m 555   nandsim.ko /boot/kernel
kldxref /boot/kernel
kldxref: file isn't dynamically-linked
root@sheevaplug:/usr/src/sys/modules/nandsim # 

Loading

The NANDsim module can be loaded manually using the kldload(8) command:

root@sheevaplug:/root # kldstat
Id Refs Address    Size     Name
 1    1 0xc0900000 58996c   elf kernel
root@sheevaplug:/root # kldload nandsim.ko
root@sheevaplug:/root # kldstat
Id Refs Address    Size     Name
 1    5 0xc0900000 58996c   elf kernel
 2    1 0xc447c000 10000    nandsim.ko
root@sheevaplug:/root # 

Partitioning NAND

Overview

When using NAND devices for the filesysystem, it is very important to note there are often possible cases when the NAND device cannot be used solely for the NANDFS purposes, for example the firmware (U-Boot) is permanently stored in some area of the NAND device.

For these reasons there must be a way of specifying information about partitions (slices) so that only a specific (unused) region of the NAND chip is subject to the file system management and responsibility.

Warning: The user must be aware of the danger of destroying some crucial contents (e.g. U-Boot) that can be stored on the NAND device, so great care must be taken when specifying the Flash map and subsequently using with the NANDFS.

Part of the NAND stack environment is a partitioning mechanism based on Flattened Device Tree technology, where a user is able to divide any GEOM-compatible device into arbitrary compartments (slices). When slices are not provided the user is able to access the NAND device in its entirety. Slices should be defined in Device Tree Blob (DTB) which is supplied to the kernel at boot time.

Run-time NAND partitioning

TODO

Static NAND partitioning

Let’s assume that a 512MB NAND device gnand0 is available and three partitions are required:

Please note that the above partition sizes don't sum up to full 512MB because of NANDFS metadata which is stored on the same device but hidden from the user.

  1. Edit sys/boot/fdt/dts/sheevaplug.dts file and put in the NAND device section (nand@0,0) the following entries that will define partitions on this device:

     slice@0 {
        reg = <0x0 0x200000>;
        label = "u-boot";
        read-only;
     };
    
     slice@200000 {
        reg = <0x200000 0x600000>;
        label = "fbsd-boot";
     };
    
     slice@800000 {
        reg = <0x800000 0x1f800000>;
        label = "root";
     };
  2. Recompile kernel (make sure not to use -DNO_CLEAN flag in order to force DTB recompilation after changes).
  3. Run target. After it has booted up you should see the following devices under /dev:

    root@sheevaplug:/root # ll /dev/gnand*
    crw-r-----  1 root  operator  0x30 Jan  1 00:00 /dev/gnand.raw0
    crw-r-----  1 root  operator  0x36 Jan  1 00:00 /dev/gnand.raw0s.fbsd-boot
    crw-r-----  1 root  operator  0x37 Jan  1 00:00 /dev/gnand.raw0s.root
    crw-r-----  1 root  operator  0x35 Jan  1 00:00 /dev/gnand.raw0s.u-boot
    crw-r-----  1 root  operator  0x2f Jan  1 00:00 /dev/gnand0
    crw-r-----  1 root  operator  0x33 Jan  1 00:00 /dev/gnand0s.fbsd-boot
    crw-r-----  1 root  operator  0x34 Jan  1 00:00 /dev/gnand0s.root
    crw-r-----  1 root  operator  0x31 Jan  1 00:00 /dev/gnand0s.u-boot

Creating NANDFS on a device with newfs_nandfs(8)

NANDFS structure is created on a device with the newfs_nandfs(8) program on the specified device special file.

Usage:

newfs_nandfs [ -options ] device

where the options are:
        -b block-size
        -B blocks-per-segment
        -L volume label
        -m reserved-segments-percentage

Example:

root@sheevaplug:/root # newfs_nandfs /dev/gnand0s.root 
filesystem parameters:
blocksize: 0x1000 sectorsize: 0x800
erasesize: 0x20000 mediasize: 0x1fe00000
segment size: 0x20000 blocks per segment: 0x20
cannot delete segment 597 (offset 188088320)
cannot delete segment 598 (offset 188219392)
...
...
cannot delete segment 62d (offset 207749120)
filesystem created succesfully
total segments: 0xfea valid segments: 0xfc7
total space: 509 MB free: 504 MB
root@sheevaplug:/root # 

Note 1: Some of the NAND device's blocks cannot be erased and are ommitted by newfs_nandfs(8) with proper info. NAND chip block wear out is normal and a few bad blocks should be considered usual.

Note 2: In case of using the newfs_nandfs(8) with a real NAND device:

  • the NANDFS block size (-b value) should be a multiple of NAND device’s page size

  • the NANDFS segment size (-b value multiplied by -B value) should be a multiple of NAND device’s block size

Note 3: Ambiguous terms are used to describe NAND chip and NANDFS characteristics. The following table should help understand the difference:

NANDFS

NAND chip

block size

page size or a multiple of page size

segment size

block size or a multiple of block size

More detailed description can be found in the tool’s manpage.

Creating NANDFS image with makefs(8)

TODO

Using NANDFS with NANDsim

This example shows how to create NANDFS on a simulated NAND device. Simulation is performed by NANDsim part of the NAND Framework.

NANDsim is controlled by the nandsim(8) user space program which has the following usage:

        nandsim <command> [command params] [params]

nandsim status <ctl_no|--all|-a> [-v]
nandsim conf <filename>
nandsim start <ctrl_no>
nandsim mod [-l <loglevel>] | <ctl_no:cs_no> [-p <prog_time>]
        [-e <erase_time>] [-r <read_time>]
        [-E <error_ratio>] | [-h]
nandsim stop <ctrl_no>
nandsim error <ctrl_no:cs_no> <page_num> <column> <length> <pattern>
nandsim bb <ctl_no:cs_no>  [blk_num1,blk_num2,..] [-U] [-L]
nandsim freeze [ctrl_no]
nandsim log <ctrl_no|--all|-a>
nandsim stats <ctrl_no:cs_no> <pagenumber>
nandsim dump <ctrl_no:cs_no> <filename>
nandsim restore <ctrl_no:chip_no> <filename>
nandsim destroy <ctrl_no[:cs_no]|--all|-a>
nandsim help [-v]

Detailed description can be found in nandsim(8) manual.

Use case example:

  1. Enable NANDsim in kernel as described here.

  2. Create configuration file for the NANDsim simulator:
    root@sheevaplug:/root # cat nandsim_test.conf
    [sim]
    log_level=11
    log_output=none
    
    [ctrl]
    ctrl_num=0
    num_cs=1
    ecc=on
    ecc_layout=[0-53]
    
    [chip]
    chip_ctrl=0
    chip_cs=0
    device_id=0xd3
    manufacturer_id=0xec
    model="k9xxg08uxM:1GiB 3,3V 8-bit"
    manufacturer="SAMSUNG"
    page_size=2048
    oob_size=64
    pages_per_block=64
    blocks_per_lun=1024
    luns=1
    column_addr_cycle=2
    row_addr_cycle=3
    program_time=0
    erase_time=0
    read_time=0
    wear_out=50000
    width=8
    bad_block_map=[100-200]

    The above example is derived from sample configuration file can be found in FreeBSD source tree: usr.sbin/nandsim/sample.conf.

  3. Load configuration into NANDsim:
    root@sheevaplug:/root # nandsim conf nandsim_test.conf 
    root@sheevaplug:/root # nandsim status --all
    
    [Controller info]
            running: no
            num cs: 1
            ecc: 1
            log_filename: 
            ecc_layout:
    0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
    16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
    32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47
    48 49 50 51 52 53
    [Chip info]
            num=0
            device_model="k9xxg08uxM:1GiB 3,3
            manufacturer="SAMSUNG"
            page_size=2048
            write_protect=NO
    Controller#1 is not configured!
    Controller#2 is not configured!
    Controller#3 is not configured!
  4. Start simulation:
    root@sheevaplug:/root # nandsim start 0
    nandsim0: <NAND controller simulator> on motherboard
    nandbus1: <NAND bus> on nandsim0
    onand0: <ONFI compliant NAND> on nandbus1
    onand0: No BBT found. Prescan chip...
    ###########
    root@sheevaplug:/root # 
  5. Create and mount NANDFS filesystem:
    root@sheevaplug:/root # newfs_nandfs /dev/gnand10 
    filesystem parameters:
    blocksize: 0x1000 sectorsize: 0x800
    erasesize: 0x20000 mediasize: 0x7fc0000
    segment size: 0x20000 blocks per segment: 0x20
    cannot delete segment 60 (offset 13107200)
    cannot delete segment 61 (offset 13238272)
    ...
    ...
    cannot delete segment c4 (offset 26214400)
    filesystem created succesfully
    total segments: 0x3f8 valid segments: 0x393
    total space: 127 MB free: 114 MB
    root@sheevaplug:/root # mount -t nandfs /dev/gnand10 /mnt
    root@sheevaplug:/root # mount
    xxx.xxx.xxx.xxx:/nfsroot/sheevaplug/ on / (nfs, noatime)
    devfs on /dev (devfs, local)
    /dev/gnand10 on /mnt (nandfs, local)
    root@sheevaplug:/root # ll /mnt/
    total 0
    -rw-r--r--  1 root  wheel  0 Jan  1 08:40 .nandfs
    root@sheevaplug:/root #
  6. Check contents of the mounted directory:
    root@sheevaplug:/root # mkdir /mnt/test
    root@sheevaplug:/root # touch /mnt/file
    root@sheevaplug:/root # ll /mnt/
    total 4
    -rw-r--r--  1 root  wheel   0 Jan  1 08:40 .nandfs
    -rw-r--r--  1 root  wheel   0 Jan  1 08:42 file
    drwxr-xr-x  2 root  wheel  32 Jan  1 08:42 test/
    root@sheevaplug:/root #
  7. Umount the filesystem:
    root@sheevaplug:/root # umount /mnt/
    root@sheevaplug:/root # mount
    xxx.xxx.xxx.xxx:/nfsroot/sheevaplug/ on / (nfs, noatime)
    devfs on /dev (devfs, local)
    10.2.0.1:/nfsroot/users/jps/src/nandfs/current on /usr/src (nfs)
    10.2.0.1:/nfsroot/users/jps/share on /nfsshare (nfs)
    root@sheevaplug:/root #
  8. Stop simulation:
    root@sheevaplug:/root # nandsim stop 0
    onand0: detached
    nandbus1: detached
    nandsim0: detached
    root@sheevaplug:/root # nandsim destroy --all
    root@sheevaplug:/root #

Using NANDFS with real NAND device

  1. Enable NAND Framework and NANDFS support in kernel as described here.

  2. Create and mount NANDFS filesystem:
    root@sheevaplug:/root # newfs_nandfs /dev/gnand0s.root
    filesystem parameters:
    blocksize: 0x1000 sectorsize: 0x800
    erasesize: 0x20000 mediasize: 0x1f800000
    segment size: 0x20000 blocks per segment: 0x20
    cannot delete segment 567 (offset 181796864)
    cannot delete segment 568 (offset 181927936)
    ...
    ...
    cannot delete segment 5f8 (offset 200802304)
    filesystem created succesfully
    total segments: 0xfba valid segments: 0xf96
    total space: 503 MB free: 498 MB
    root@sheevaplug:/root # mount -t nandfs /dev/gnand0s.root /mnt
    root@sheevaplug:/root # mount
    xxx.xxx.xxx.xxx:/nfsroot/sheevaplug/ on / (nfs, noatime)
    devfs on /dev (devfs, local)
    /dev/gnand0s.root on /mnt (nandfs, local)
    root@sheevaplug:/root # ll /mnt/
    total 0
    -rw-r--r--  1 root  wheel  0 Jan  1 08:40 .nandfs
    root@sheevaplug:/root #
  3. Check contents of the mounted directory:
    root@sheevaplug:/root # mkdir /mnt/test
    root@sheevaplug:/root # touch /mnt/file
    root@sheevaplug:/root # ll /mnt/
    total 4
    -rw-r--r--  1 root  wheel   0 Jan  1 08:40 .nandfs
    -rw-r--r--  1 root  wheel   0 Jan  1 08:42 file
    drwxr-xr-x  2 root  wheel  32 Jan  1 08:42 test/
    root@sheevaplug:/root #
  4. Umount the filesystem:
    root@sheevaplug:/root # umount /mnt/
    root@sheevaplug:/root # mount
    xxx.xxx.xxx.xxx:/nfsroot/sheevaplug/ on / (nfs, noatime)
    devfs on /dev (devfs, local)
    root@sheevaplug:/root #

Using NANDFS with non-NAND device

NANDFS can be used with non-NAND storage devices such as USB memory sticks or HDD.

  1. Enable NANDFS support in kernel as described here.

  2. Check storage device's parameters (it's a 2GB USB memory stick):
    root@sheevaplug:/root # diskinfo -v /dev/da0 
    /dev/da0
            512             # sectorsize
            2000683008      # mediasize in bytes (1.9G)
            3907584         # mediasize in sectors
            0               # stripesize
            0               # stripeoffset
            243             # Cylinders according to firmware.
            255             # Heads according to firmware.
            63              # Sectors according to firmware.
            07A10C003C48EF64        # Disk ident.
    
    root@sheevaplug:/root #
  3. Create and mount NANDFS filesystem:
    root@sheevaplug:/root # newfs_nandfs /dev/da0 
    filesystem parameters:
    blocksize: 0x1000 sectorsize: 0x200
    erasesize: 0x20000 mediasize: 0x77400000
    segment size: 0x20000 blocks per segment: 0x20
    filesystem created succesfully
    total segments: 0x3b9a valid segments: 0x3b9a
    total space: 1907 MB free: 1907 MB
    root@sheevaplug:/root # mount
    /dev/gnand0s.root on / (nandfs, local)
    devfs on /dev (devfs, local)
    root@sheevaplug:/root # mount -t nandfs /dev/da0 /mnt/
    root@sheevaplug:/root # mount
    /dev/gnand0s.root on / (nandfs, local)
    devfs on /dev (devfs, local)
    /dev/da0 on /mnt (nandfs, local)
    root@sheevaplug:/root # ll /mnt/
    total 0
    -rw-r--r--  1 root  wheel  0 Jan  1 08:40 .nandfs
    root@sheevaplug:/root #
  4. Check contents of the mounted directory:
    root@sheevaplug:/root # mkdir /mnt/test
    root@sheevaplug:/root # touch /mnt/file
    root@sheevaplug:/root # ll /mnt/
    total 4
    -rw-r--r--  1 root  wheel   0 Jan  1 08:40 .nandfs
    -rw-r--r--  1 root  wheel   0 Jan  1 08:42 file
    drwxr-xr-x  2 root  wheel  32 Jan  1 08:42 test/
    root@sheevaplug:/root #
  5. Umount the filesystem:
    root@sheevaplug:/root # umount /mnt
    root@sheevaplug:/root # mount
    /dev/gnand0s.root on / (nandfs, local)
    devfs on /dev (devfs, local)
    root@sheevaplug:/root #

Booting from the NAND Flash device

The following two scenarios are real-life examples on how to run FreeBSD from NAND Flash memory. Example target systems are SheevaPlug and a development board for Marvell 88F6281 SoC.

There are two approaches to booting FreeBSD kernel:

Using loader(8)

TODO

Using binary kernel

Booting SheevaPlug from NAND Flash memory requires two SHEEVAPLUG kernels to be compiled: one will be used for intermediate preparation steps whereas the second will be used to boot SheevaPlug from NAND memory. First kernel will mount root file system over NFS, second - from NANDFS.

General procedure is like this: run SheevaPlug with NFS-mounted root FS in order to prepare NAND partitions and put FreeBSD kernel and world on them then reboot target, configure U-Boot and finally boot FreeBSD from NAND. The following steps show the details:

  1. Build FreeBSD ARM world and package it to a tar file. Let's assume it is named arm-world.tar and available in target's /root/ directory.

  2. Prepare NAND partitions as described in here.

  3. Enable NAND Framework and NANDFS support in sys/arm/conf/SHEEVAPLUG kernel config file and recompile kernel.

  4. Copy aside binary kernel compiled in step 3.
    host# cp /path/to/obj/arm.arm/path/to/src/sys/SHEEVAPLUG/kernel.bin /root/kernel.nfsroot.bin
  5. Edit sys/arm/conf/SHEEVAPLUG kernel config file and add the following line that will tell kernel to mount root file system from NAND partition. Recompile kernel afterward.

    options       ROOTDEVNAME=\"nandfs:/dev/gnand0s.root\"
  6. Run SheevaPlug with kernel compiled in step 3. Check that NAND partitions have been properly configured - see [[|here]]

  7. Copy binary kernel compiled in step 5 to target. Let's assume it is copied to /root/kernel.nandboot.bin

  8. Erase gnand0s.fbsd-boot NAND partition:

    root@sheevaplug:/root # nandtool erase dev=/dev/gnand0s.fbsd-boot
  9. Copy kernel from step 7 to gnand0s.fbsd-boot partition. Note that conv=sync dd(1) option has to be used in order for the whole binary to be copied. Please also note that dd(1) block size should be a multiple of NAND's page size. Remember the size of the kernel. It will be required in further steps:

    root@sheevaplug:/root # dd if=/root/kernel.nandboot.bin of=/dev/gnand0s.fbsd-boot bs=2k conv=sync
    1917+1 records in
    1918+0 records out
    3928064 bytes transferred in 5.188410 secs (757084 bytes/sec)
    root@sheevaplug:/root # 
  10. Prepare NANDFS partition to store FreeBSD user space:
    root@sheevaplug:/root # newfs_nandfs /dev/gnand0s.root
    filesystem parameters:
    blocksize: 0x1000 sectorsize: 0x800
    erasesize: 0x20000 mediasize: 0x1f800000
    segment size: 0x20000 blocks per segment: 0x20
    cannot delete segment 7b3 (offset 258867200)
    cannot delete segment 7b4 (offset 258998272)
    ...
    ...
    cannot delete segment 8e9 (offset 299499520)
    filesystem created succesfully
    total segments: 0xfba valid segments: 0xfb2
    total space: 503 MB free: 502 MB
    root@sheevaplug:/root # mount -t nandfs /dev/gnand0s.root /mnt
  11. Untar FreeBSD ARM world to NANDFS gnand0s.root partition:

    root@sheevaplug:/root # tar -tf /root/arm-world.tar
    arm-world/
    arm-world/bin
    arm-world/boot
    arm-world/dev
    ...
    ...
    arm-world/bin/test
    arm-world/bin/[
    arm-world/bin/uuidgen
    root@sheevaplug:/root # tar -xf /root/arm-world.tar -C /mnt --strip-components=1
    root@sheevaplug:/root # ll /mnt
    .cshrc          bin             libexec         root            var
    .nandfs         boot            media           sbin
    .profile        dev             mnt             sys
    COPYRIGHT       etc             proc            tmp
    lib             rescue          usr
    root@sheevaplug:/root # umount /mnt
    root@sheevaplug:/root #
  12. Configure newly installed system:
    root@sheevaplug:/root # echo 'hostname="sheevaplug-from-nand"' > /mnt/etc/rc.conf
    root@sheevaplug:/root #
  13. Reboot target and enter U-Boot prompt:
    root@sheevaplug:/root # reboot
    Jan  1 00:03:47 sheevaplug reboot: rebooted by root
    Jan  1 00:03:47 sheevaplug syslogd: exiting on signal 15
    ...
    ...
    CPU : Marvell Feroceon (Rev 1)
    
    Streaming disabled 
    Write allocate disabled
    
    
    USB 0: host mode
    PEX 0: interface detected no Link.
    Net:   egiga0 [PRIME], egiga1
    Hit any key to stop autoboot:  0 
    Marvell>>
  14. Set U-Boot variables to automatically boot FreeBSD binary kernel. Use kernel size from step 9 converted to hex (hex(3928064) = 0x3bf000):
    Marvell>> set fbsdnandboot 'nand read 0x900000 0x200000 0x3bf000; go 0x900000'
    Marvell>> set bootcmd 'run fbsdnandboot'
    Marvell>> saveenv
    Saving Environment to NAND...
    Erasing Nand...Writing to Nand... done
    Marvell>> 
  15. Reset target and wait for FreeBSD to boot:
    Marvell>> reset
    
             __  __                      _ _
            |  \/  | __ _ _ ____   _____| | |
            | |\/| |/ _` | '__\ \ / / _ \ | |
            | |  | | (_| | |   \ V /  __/ | |
            |_|  |_|\__,_|_|    \_/ \___|_|_|
     _   _     ____              _
    | | | |   | __ )  ___   ___ | |_ 
    | | | |___|  _ \ / _ \ / _ \| __| 
    | |_| |___| |_) | (_) | (_) | |_ 
     \___/    |____/ \___/ \___/ \__| 
     ** MARVELL BOARD: SHEEVA PLUG LE 
    
    U-Boot 1.1.4 (Mar 19 2009 - 16:06:59) Marvell version: 3.4.16
    
    U-Boot code: 00600000 -> 0067FFF0  BSS: -> 006CEE80
    
    Soc: 88F6281 A0 (DDR2)
    CPU running @ 1200Mhz L2 running @ 400Mhz
    SysClock = 400Mhz , TClock = 200Mhz 
    
    DRAM CAS Latency = 5 tRP = 5 tRAS = 18 tRCD=6
    DRAM CS[0] base 0x00000000   size 256MB 
    DRAM CS[1] base 0x10000000   size 256MB 
    DRAM Total size 512MB  16bit width
    Flash:  0 kB
    Addresses 8M - 0M are saved for the U-Boot usage.
    Mem malloc Initialization (8M - 7M): Done
    NAND:512 MB
    
    CPU : Marvell Feroceon (Rev 1)
    
    Streaming disabled 
    Write allocate disabled
    
    
    USB 0: host mode
    PEX 0: interface detected no Link.
    Net:   egiga0 [PRIME], egiga1
    Hit any key to stop autoboot:  0 
    
    NAND read: device 0 offset 0x200000, size 0x3bf000
     3928064 bytes read: OK
    ## Starting application at 0x00900000 ...
     dtbp = 0xc0c98c50
    GDB: no debug ports present
    KDB: debugger backends: ddb
    KDB: current backend: ddb
    Copyright (c) 1992-2012 The FreeBSD Project.
    Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
            The Regents of the University of California. All rights reserved.
    FreeBSD is a registered trademark of The FreeBSD Foundation.
    FreeBSD 10.0-CURRENT #17 r+9fa721a-dirty: Tue Apr 24 09:57:38 CEST 2012
        xxx@xxx:/usr/obj/arm.arm/usr/src/sys/SHEEVAPLUG arm
    ...
    ...
    Trying to mount root from nandfs:/dev/gnand0s.root []...
    ...
    ...
    Starting background file system checks in 60 seconds.
    
    Tue Apr 24 12:42:15 UTC 2012
    
    FreeBSD/arm (sheevaplug-from-nand) (ttyu0)
    
    login: root
    Last login: Tue Apr 24 12:37:38 on ttyu0
    Apr 24 12:44:26  login: ROOT LOGIN (root) ON ttyu0
    FreeBSD 10.0-CURRENT (SHEEVAPLUG) #17 r+9fa721a-dirty: Tue Apr 24 09:57:38 CEST 2012
    
    Welcome to FreeBSD!
    
    Before seeking technical support, please use the following resources:
    
    o  Security advisories and updated errata information for all releases are
       at http://www.FreeBSD.org/releases/ - always consult the ERRATA section
       for your release first as it's updated frequently.
    
    o  The Handbook and FAQ documents are at http://www.FreeBSD.org/ and,
       along with the mailing lists, can be searched by going to
       http://www.FreeBSD.org/search/.  If the doc package has been installed
       (or fetched via pkg_add -r lang-freebsd-doc, where lang is the
       2-letter language code, e.g. en), they are also available formatted
       in /usr/local/share/doc/freebsd.
    
    If you still have a question or problem, please take the output of
    `uname -a', along with any relevant error messages, and email it
    as a question to the questions@FreeBSD.org mailing list.  If you are
    unfamiliar with FreeBSD's directory layout, please refer to the hier(7)
    manual page.  If you are not familiar with manual pages, type `man man'.
    
    Edit /etc/motd to change this login announcement.
    
    root@sheevaplug-from-nand:/root # mount
    /dev/gnand0s.root on / (nandfs, local)
    devfs on /dev (devfs, local)
    root@sheevaplug-from-nand:/root #

    That's it! You have booted FreeBSD and are running it on SheevaPlug entirely from the NAND Flash memory.

cleaner(4) tuning

The garbage collector a.k.a cleaner(4) is a kernel thread that is responsible for recycling of used resources of the NANDFS with out-of-date contents, which can be disposed of (cleaned) and prepared for reuse. The behavior of this thread can be controlled with the following sysctl(8) variables:

Default values are:

root@sheevaplug:/root # sysctl vfs.nandfs | grep cleaner
vfs.nandfs.cleaner_segments: 5
vfs.nandfs.cleaner_interval: 5
vfs.nandfs.cleaner_enable: 1
root@sheevaplug:/root #

The above settings give the following behavior: every 5 seconds at most 5 segments will be cleaned. If more aggressive cleaning policy is required you can set for example the following values:

root@sheevaplug:/root # sysctl vfs.nandfs.cleaner_segments=20
vfs.nandfs.cleaner_segments: 5 -> 20
root@sheevaplug:/root # sysctl vfs.nandfs.cleaner_interval=3
vfs.nandfs.cleaner_interval: 5 -> 3
root@sheevaplug:/root # sysctl vfs.nandfs | grep cleaner
vfs.nandfs.cleaner_segments: 20
vfs.nandfs.cleaner_interval: 3
vfs.nandfs.cleaner_enable: 1
root@sheevaplug:/root # 

Note: Increasing maximum number of segments or shortening interval may cause significant rise in system load thus reducing overall system responsiveness and performance

Tools and utilities

TODO

Using nandtool(8)

Using nandfs(8)

Development notes

TODO

NAND Framework

NAND File System

References

TODO

NAND (last edited 2012-06-13 11:22:44 by JanSieka)