bhyve VM usage in Jenkins cluster

Installation of FreeBSD into bhyve VM for use in Jenkins cluster

  1. Make sure that in /boot/loader.conf you have:

    # Include vmm driver for bhyve
    vmm_load="YES"
    # Load nmdm driver to communicate with bhyve VM's
    nmdm_load="YES"
    # For bhyve VM's networking
    if_bridge_load="YES"
    bridgestp_load="YES"
    if_tap_load="YES"
  2. Obtain a release ISO image from FreeBSD download site.

  3. Create a ZFS dataset which will hold the VM
       zfs create zroot/vm
  4. Set ZFS property so that dataset will export ZVOL's as block devices only, and not try to probe them with GEOM:
       zfs set volmode=dev zroot/vm
       zfs get volmode zroot/vm
  5. Create a 200G zvol:
       zfs create -V 200G zroot/vm/disk.img
  6. Boot the release ISO image and install it into the disk image:
     sh /usr/share/examples/bhyve/vmrun.sh -i -I FreeBSD-9.2-RELEASE-amd64-disc1.iso -d /dev/zvol/zroot/vm/disk.img -m 2G vm_test
  7. Do the initial installation, using disk.img as the backing hard drive

  8. Edit /etc/ttys and add this line:

     ttyu0   "/usr/libexec/getty std.9600"   vt100   on secure

    or this, if the guest is running 11-CURRENT newer than r264175

     ttyu0  "/usr/libexec/getty 3wire"  vt100  on  secure
  9. shutdown the VM
  10. Reboot the VM without the ISO image:
     sh /usr/share/examples/bhyve/vmrun.sh -d disk.img -m 2G vm_test
  11. Set up basic networking inside the VM by editing /etc/rc.conf

  12. Reboot the VM.
  13. Install necessary packages such as devel/jenkins

Configure tap devices for bhyve VM's

  1. On the host machine which is running the VM's, pick a network interface which will be used in the bridge. This example will use igb0.

  2. Add the following to /etc/rc.conf on the host machine:

    #####################################################
    # Create tap devices, one tap interface per bhyve VM.
    # Add the tap interfaces to bridge0
    ####################################################
    cloned_interfaces="bridge0 tap0 tap1"
    
    autobridge_interfaces="bridge0"
    autobridge_bridge0="tap* igb0"
  3. Each VM should have a separate tap device, all connected to the same bridge. If you add more VM's, remember to add another tap device to cloned_interfaces

  4. Add the following to /etc/sysctl.conf on the host machine:

    # bhyve needs this for tap interfaces
    net.link.tap.user_open=1
    net.link.tap.up_on_open=1
  5. Refer to: http://www.freebsd.org/doc/handbook/network-bridging.html

Accessing console of VM

  1. Type the following to see the list of VM's running:
     ls /dev/vmm/*
  2. The bhyve VM's are configured to send their console output to the nmdm(4) nullmodem driver. Type the following to see the list of nullmodem devices in use:

     ls /dev/nmdm*
  3. Each nmdm(4) has two endpoints A and B. You must connect to the B endpoint using the cu utility. For example:

     cu -l /dev/nmdm0B
  4. To disconnect, type:
     ~..

Startup scripts

The scripts used to start the bhyve VM's are located at Github: https://github.com/freebsd/freebsd-ci/tree/master/vm

Configure rc.d scripts to boot VM

  1. Read this: README for VM scripts

Jenkins/VMSetup (last edited 2016-07-04T04:18:12+0000 by MichaelDexter)