Running CentOS 5.5 in a Jail

Work In Progress

This document is intended to serve as a walk-through for getting a Centos 5.5 jail running on a host running FreeBSD 8-STABLE.

Contents

Running CentOS 5.5 in a Jail

Motivation

I have been working on a project to provide an arbitrary number of dev/test/demo environments for a 3-tier web application. The environments need to be independent, lightweight, easily replicated, network-reachable, and resemble the production environment as closely as possible. Being able to rapidly provision new environments is a plus. In this case the production environment runs CentOS 5, Python, Apache, and PostgreSQL.

Fully virtualized environments as provided by VMware or VirtualBox might have worked, but I wanted something more flexible and lightweight in terms of resource allocation and consumption. The virtual disks required by such VM's are also not ideal since they need to be a fixed size and generally need to be copied in full to be cloned.

I decided that I really wanted to use ZFS and FreeBSD jails. That combination meets all of the requirements except for being similar to the production stack running CentOS. Unless maybe CentOS could run inside a FreeBSD jail with Linux emulation...

It turns out that it can with a modest amount of effort. Now that I've been through that once I'd like to share what I've learned so anyone else who likes this crazy idea can quickly implement it. With a successful proof-of-concept in hand I will be using this solution for my project.

Preparing the Host

Configure ZFS on the host as in the handbook and this wiki.
- This is not a requirement but it makes managing jail filesystems very easy. Each jail can have its own ZFS, and the filesystems can be cloned, snapshotted, backed up, compressed, etc. with very little effort. The remainder of this document assumes you have a zpool named "tank" and a (parent) ZFS named "tank/jails" mounted on /jails. The (first) jail will go into a (child) ZFS called "tank/jails/centos". Adjust as necessary if you use different names or mountpoints.
Enable the Linux kernel module as in the handbook.
For FreeBSD 8.x, patch linprocfs.
- Linprocfs in 8-STABLE needs a hack for openpty (used by e.g. SSH in the jail to get a pty) to work.
- Apply this patch to your kernel source: linprocfs.patch (from http://lists.freebsd.org/pipermail/freebsd-emulation/2010-March/007487.html).
- Rebuild, reinstall and re-load linprocfs.ko.
  There was a (different) commit to 9-CURRENT to address this (http://svn.freebsd.org/viewvc/base?view=revision&revision=204825) but it does NOT work on 8.x (at least I couldn't get fdescfs to behave for me). The fix does work for FreeBSD 9.x but you do need to enable fdescfs in the jail. See: http://lists.freebsd.org/pipermail/freebsd-emulation/2010-March/007493.html (same thread as initial patch above). Reading the whole thread could be enlightening).
Install archivers/rpm4.
- Version 4 works better with the CentOS packages than version 3 (archivers/rpm), which is what you get by default.
Install emulators/linux_base-f10.

Bootstrapping the Jail

To get things rolling we will extract the squashfs environment used by the CentOS installer. We then use that environment to install the selected set of packages.

Manually install Linux squashfs-tools on the host. The link below is for a Fedora 10 package to match the host's linux_base-f10 environment.

cd /tmp
fetch http://archives.fedoraproject.org/pub/archive/fedora/linux/releases/10/Everything/i386/os/Packages/squashfs-tools-3.4-1.i386.rpm
cd /compat/linux
rpm2cpio -q < /tmp/squashfs-tools-3.4-1.i386.rpm | cpio -id

Download the CentOS 5.5 DVD ISO image.
- Remember that even on an amd64 (64-bit) host only 32-bit Linux emulation is supported, so get the "i386" DVD. Use BitTorrent or find a mirror at centos.org that hosts the whole thing on the web (only some of them do). The file I used is called CentOS-5.5-i386-bin-DVD.iso. He also suggested simply downloading the content of centos/5.5/os/i386 from any mirror instead of using an ISO image.

Mount the DVD image on the host.

mdconfig -a -t vnode -f /tmp/CentOS-5.5-i386-bin-DVD.iso
mkdir /media/centos
mount -t cd9660 /dev/md0 /media/centos

Extract the CentOS "stage2" chroot.
- This is the environment that runs when you boot from the DVD. For our purposes it provides Linux-native versions of bash and rpm, along with the libraries they need.
```
cd /jails
/compat/linux/usr/sbin/unsquashfs -dest centos-stage2 /media/centos/images/stage2.img
```

Re-mount the DVD image inside the stage2 chroot.

cd centos-stage2
mkdir dist
umount /dev/md0
mount -t cd9660 /dev/md0 /usr/jails/centos-stage2/dist

Mount the jail's ZFS inside the stage2 chroot.

mkdir stage
zfs set mountpoint=/usr/jails/centos-stage2/stage tank/centos

Chroot into stage2.

chroot /usr/jails/centos-stage2 /usr/bin/bash

Initialize the RPM database in the Jail's ZFS.

mkdir -p /stage/var/lib/rpm
rpm --root /stage --initdb

Identify the complete initial set of RPM's to be installed in the jail.
- This can be tricky without automatic dependency resolution. If someone knows of a good way to provide that at this point let me know. My approach was to iteratively copy the packages I knew I would want or need into a "bootstrap" directory, then add packages to supply the dependencies rpm complained about. Lucky for you, I saved my list of packages. I recommend copying the RPM packages in the list and others you need into the bootstrap directory to eliminate the need for both the RPM list and the DVD image in the future. It also makes the RPM command a lot simpler.
- Save the list of packages (centos5.5_bootstrap_rpm_list.txt) to /usr/jails/centos-stage2.
```
mkdir bootstrap
for rpm in `cat /centos5.5_bootstrap_rpm_list.txt`; do
    cp /dist/CentOS/${rpm} bootstrap
done
```
Install the RPM's.
- ```
cd bootstrap
rpm --root /stage -Uvh *.rpm
```
  You should not have any dependency errors. If you do, check the contents of the bootstrap directory against the list and try again. You can do this repeatedly, since nothing is actually installed until all dependencies are satisfied. You will now have a minimal but reasonable set of packages installed. If you want to add more packages, you can use yum after the jail is running. (Yum of course provides automatic dependency resolution and will use the default CentOS repositories to retrieve the appropriate packages.)
Leave the chroot and un-mount the DVD image.
- ```
exit
umount /dev/md0
```

Configuring the Jail

Mount the Jail's ZFS in its regular location

mkdir -p /usr/jails/centos
zfs set mountpoint=/usr/jails/centos tank/centos

Configure /etc/rc.conf (see also handbook and jail(8) manpage).

If you plan to have non-linux jails then some of these settings will need to be set per-jail instead of globally as below. Replace the interface and IP with the appropriate values for your system.

jail_enable="YES"
jail_list="centos"
jail_sysvipc_allow="YES" # INSECURE (shared memory isn't jailed) but required for e.g. postgresql
jail_interface="em0"
jail_devfs_enable="YES"
jail_fdescfs_enable="YES" # XXX this only works for me on FreeBSD 9.x and newer
jail_exec_start="/bin/sh /etc/rc.d/rc 3"
jail_exec_stop="/bin/sh /etc/rc.d/rc 0"
jail_centos_rootdir="/jails/centos"
jail_centos_hostname="centos.your.domain"
jail_centos_ip="1.2.3.4"
jail_centos_fstab="/jails/fstab_centos" # for linprocfs

Create a jail-specific fstab and add linprocfs to it. Thanks to Stefan Bethke for the hint--much safer this way than modifying the system fstab.
- This must be done for every jail.
```
echo "linproc /jails/centos/proc linprocfs rw 0 0" >> /jails/fstab_centos
```
Edit /jails/centos/etc/sysconfig/syslog to use a log socket not in /dev (replace existing options line).
- Otherwise you have to gesticulate around devfs and you may not get logging from the jail.
  SYSLOGD_OPTIONS="-m 0 -p /var/run/log"

Create master.passwd inside the jail.

This is used by the host to look up the uid of the jail owner, so it must exist. Take care not to clobber the exising (CentOS) etc/passwd.

echo "root::0:0::0:0:Charlie &:/root:/bin/bash" > /jails/centos/etc/master.passwd
pwd_mkdb -d /jails/centos/tmp -p /jails/centos/etc/master.passwd
cd /jails/centos/tmp
mv master.passwd pwd.db spwd.db ../etc

Populate etc/resolv.conf (edit as appropriate).
- ```
cp /etc/resolv.conf /jails/centos/etc
```

Set up hostname and bogus localhost inside the jail.

echo "NETWORKING=yes" >> /jails/centos/etc/sysconfig/network
echo "HOSTNAME=centos.your.domain" >> /jails/centos/etc/sysconfig/network
echo "1.2.3.4 localhost centos centos.your.domain" >> /jails/centos/etc/hosts

Do some final touch-up in a chroot.

Make CentOS authentication happy, silence utilities that look for unneeded files, set the root password.

chroot /jails/centos /bin/bash
cd /etc
pwconv
grpconv
passwd root # this is the password you will use for su, etc. within the jail
touch /etc/fstab
touch /etc/mtab

Silence a few errors in host dmesg and jail logs

cd /sbin
mv consoletype consoletype.orig
ln -s /bin/true consoletype
cd /bin
mv umount umount.prev
ln -s /bin/true umount

Turn off unneeded services

chkconfig --list # (look through the output)
chkconfig gpm off
chkconfig lvm2-monitor off
chkconfig mcstrans off
chkconfig netfs off
chkconfig network off # (really)
chkconfig rawdevices off

In my setup that just leaves crond, sendmail, sshd and syslog. Your needs may vary.

Leave chroot
```
exit
```

Edit /jails/centos/etc/init.d/syslog to not reference klogd.
- My changes are in this patch: initd_syslog.patch
Edit /jails/centos/etc/init.d/halt to not do much.
- My changes are in this patch: initd_halt.patch

Starting the Jail

With the changes to /etc/rc.conf above the jail will start automatically at reboot.

Start the jail manually the first time. Allow a minute for SSH key generation.
- ```
/etc/rc.d/jail start centos
```

Check for potential problems.

tail -n300 /var/log/messages|grep "linux:"
cat /var/log/jail_centos_console.log

With any luck, you should be in business at this point. Use "jexec <num> bash" to look around inside the jail, where <num> is obtained from running "jls". SSH (as root) should work as well.

RPM and yum will be installed and functional, so installing new packages or updates to existing ones is much easier than bootstrapping the initial set.

Most things should work as you would expect.

Known Limitations

Apache (httpd) will not run in the jail unless the installed apr wass built without epoll support (which is not true of the CentOS RPM). I built a new RPM on a (real) 32-bit CentOS machine after renaming /usr/include/sys/epoll.h (the configure script does the right thing when it doesn't find it).
GCC doesn't work reliably in the jail, which means you can't build a custom RPM (like for the apr item above)--you need either a real Linux host or a full virtual machine (a la VMware or VirtualBox). I didn't try too hard to figure out why, so it's quite possible there's a workaround.
BIND utils (host, nslookup) don't work even when built without epoll. Regular name resolution does work, however.
Others to be documented as they are discovered/reported.

Example Cloning Procedure

Snapshot and clone the ZFS:

zfs snapshot tank/centos@somelabel
zfs clone tank/centos@somelabel tank/jails/newjail

Clean up files that shouldn't transfer--ssh host keys, rpm work files.

rm -f /jails/newjail/etc/ssh/ssh_host_*
rm -f /jails/newjail/var/lib/rpm/__*

Populate hostname and IP a few places within the jail.
- Both in /etc/hosts
- Hostname in /etc/sysconfig

Create an fstab for the jail

echo "linproc /jails/newjail/proc linprocfs rw,late 0 0" >> /jails/fstab_newjail

Add jail-specific lines to /etc/rc.conf on the host.

jail_newjail_rootdir="/jails/newjail"
jail_newjail_hostname="newjail.your.domain"
jail_newjail_ip="1.2.3.5"
jail_newjail_fstab="/jails/fstab_newjail"

Add newjail to the jail_list line in /etc/rc.conf.
Start the jail
- ```
/etc/rc.d/jail start newjail
```