bhyve Windows Virtual Machines

The bhyve hypervisor supports Microsoft Windows virtual machines on 12.4-RELEASE and later.

Supported Windows Versions

bhyve supports Windows desktop versions Vista, 7, 8/8.1/8.2, 10 and 11, and Windows Server versions 2008/2008R2, 2012/2012R2, 2016, 2019, 2022, and 2025.

Windows Storage Considerations

All supported versions of windows perform well with bhyve NVMe emulation. Alternatively, ahci-hd and ahci-cd can be used, and virtio-blk using the Red Hat VirtIO drivers.

Previous versions of the VirtIO driers may be required but as of January, 2024, bhyve supports Windows Server 2022 with the latest .240 virtio-blk driver.

Windows virtual machines support raw disk images on UFS and ZFS, and ZFS volumes (ZVOLs). Consider a ZVOL volblocksize of 32k or 64k to avoid write amplification with NTFS and ReFS.

Windows Networking Considerations

Windows desktop and Server support the bhyve e1000 emulated network interface, and virtio-net using the Red Hat VirtIO drivers.

Windows 11 TPM Requirement

According to the Microsofts documentation at docs.microsoft.com, a TPM module in version 2.0 is needed. As of 14.0-RELEASE, bhyve supports TPM passthrough and TPM emulation - see bhyve(8) man page and YouTube for details.

To bypass this TPM requirement, you can modify the registry manually during the Windows 11 installation process or create an ISO that omits the check.

To install Windows 11 from an unmodified ISO press Shift+F10 after the graphical environment loads, which opens a command shell. Run regedit to edit the registry, and make the following changes:

1. Navigate to HKEY_LOCAL_MACHINE\SYSTEM\Setup and right click it to add the Key LabConfig

2. Add a DWORDs (32) entry named BypassTPMCheck with a value of 1.

2. Add a DWORDs (32) entry named BypassSecureBootCheck with a value of 1.

And continue the installation as normal. For the simplest installation, use the "e1000" interface for first boot.

In order to get Windows 11 running without modifying the running registry, we can use the tool from github.com/AveYo/MediaCreationTool.bat to create an ISO image that omits to check for the TPM module. As of May 2022, this worked:

1. Use an updated Windows 10 (eg in a bhyve VM) as a base

2. Download MediaCreationTool.bat and start the tool from an Administrator's prompt (may work from a normal user's cmd)

3. When asked for the version ("MCTVersion"), choose "11". As Option "11 MCTPreset" choose "Auto ISO" to generate the ISO file; grab a coffee while waiting for the download.

4. Copy the freshly baked ISO file to the host. Create a new VM with UEFI boot option, virtio-net network, a virtio-blk disk and an additional cdrom drive where you use the virtio-iso from fedora - so you have the virtio ISO in one drive and the Windows 11 ISO in the second drive. Version 0.1.185 worked: choose add drivers when asked for the disk to install Windows 11. Network is being omitted for now.

5. After the install, use virtio 0.1.217 to install the virtio-net interface

Simple alternative install method

The information below is outdated. You can use the how-to of the package vm-bhyve. It offers a very simple and almost effortless installation of Microsoft Windows (tested with 1909). For a full graphical installation see UEFI Graphics (VNC)

Requirements

At this time of writing, the Windows Sever 2016 Technical Previews provide the easiest means of testing Windows support with bhyve. The Technical Previews:

ISO Remastering

Manually copy or GIT clone (requires the git package) a prepared AutoUnattend.xml file for your version of Windows 

git clone https://github.com/nahanni/bhyve-windows-unattend-xml

Fetch the following Windows VirtIO driver ISO 

fetch https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/archive-virtio/virtio-win-0.1.96/virtio-win-0.1.96.iso

Windows 10 requires virtio-win-0.1.118: 

fetch https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/archive-virtio/virtio-win-0.1.118-2/virtio-win-0.1.118.iso

Later releases have various issues with various Windows versions.

Windows Vista requires a previous version of the drivers: 

fetch https://fedorapeople.org/groups/virt/virtio-win/deprecated-isos/archives/virtio-win-0.1-94/virtio-win-0.1-94.iso

With the Windows ISO, XML auto installation file and VirtIO obtained, the ISO is ready to be remastered. 7z is suggested because it is UDF-friendly, unlike the in-base tar(1).

The remastering steps are as follows: 

mkdir -p iso-remaster/virtio

# Note the DOS syntax and replace <windows>.iso with the proper file name:
/usr/local/bin/7z x <windows>.iso -oiso-remaster

# The 2012R2 and later driver:
tar xf virtio-win-0.1.96.iso --include="NetKVM/2k12R2/amd64/*" -C iso-remaster/virtio/

cp bhyve-windows-unattend-xml/files/win2016preview_AutoUnattend.xml iso-remaster/AutoUnattend.xml

mkisofs \
     -b boot/etfsboot.com -no-emul-boot -c BOOT.CAT \
     -iso-level 4 -J -l -D \
     -N -joliet-long \
     -relaxed-filenames -v \
     -V "Custom" -udf \
     -boot-info-table -eltorito-alt-boot -eltorito-platform 0xEF \
     -eltorito-boot efi/microsoft/boot/efisys_noprompt.bin \
     -no-emul-boot \
     -o install.iso iso-remaster

This will result in a remastered Windows ISO named install.iso

You will also require the EFI loader and a disk image: 

pkg install uefi-edk2-bhyve
truncate -s 15GB windows2016.img

Note that 15GB will only leave 4GB free space. Consider far more for production use.

Windows ISO Boot

The install.iso is only required for the first boot of Windows Server and must be removed from the bhyve command after the first boot. Desktop editions of Windows require that a null install.iso file remains and it can be created with touch install.iso

The trick is to have the CD configured only for the first boot. It has to be removed from the bhyve command line for the second and subsequent boots, since there isn't currently a boot selection in UEFI and a CD is automatically given preference, resulting in an endless cycle of CD installs. For Windows-desktop installs, the ahci-cd drive *must* be kept there, and a 0-byte file used instead e.g. 'touch null.iso'.

There are currently some slot limitations with UEFI:

virtio-block devices have an issue with descriptor overflow that is being worked on, so avoid using it for the time being.

A typical bhyve command for Windows is: 

bhyve \
      -c 2 \
      -s 0,hostbridge \
      -s 3,ahci-hd,windows2016.img \
      -s 4,ahci-cd,install.iso \
      -s 10,virtio-net,tap0 \
      -s 31,lpc \
      -l com1,/dev/nmdm0A \
      -l com2,/dev/nmdm1A \
      -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
      -m 2G -H -w \
      windows2016

Only one or two VCPUs should be used during installation but this number can be increased once Windows is installed.

This should produce many pages of EFI debugging information and land you in the Windows SAC: 

Computer is booting, SAC started and initialized.                               
                                                                                
Use the "ch -?" command for information about using channels.
Use the "?" command for general help.

SAC>
EVENT: The CMD command is now available.
SAC>
EVENT:   A new channel has been created.  Use "ch -?" for channel help.
Channel: SACSetupAct
SAC>
EVENT:   A new channel has been created.  Use "ch -?" for channel help.
Channel: SACSetupErr

The initial SAC> prompt will appear immediately, followed by the three EVENTs.

Each SAC channel performs a different role. You can list available channels at any time with: 

SAC>ch
Channel List

(Use "ch -?" for information on using channels)

# Status  Channel Name
0 (AV)    SAC
1 (AV)    SACSetupAct
2 (AV)    SACSetupErr

You can toggle between channels with ESC-TAB.

Some useful SAC commands to shutdown, display the network configuration (once Windows is installed) and set a fixed IP on the 3rd network device are: 

SAC>shutdown
SAC>i
SAC>i 3 192.168.1.230 255.255.255.0 192.168.1.1

The CMD event indicates that the Windows command line is available. Typing cmd will create this channel and should get: 

X:\windows\system32>

If you only see the CMD channel with another version of Windows, your installation has probably had an error with regard to license keys. To exit the CMD channel back to SAC, type exit.

The SACSetupAct is most useful during installation as it will show the installation steps and is your best indication of success or failure. The SACSetupErr channel will provide the specific error should one occur.

The process will move quickly until the main Windows image extraction begins: 

...
2015-10-10 20:58:33, Info       [0x06009e] IBS    DeployWIMImage:Calling IDepWI
MImageResolved::Apply...
2015-10-10 20:58:33, Info       [0x0606cc] IBS    Calling WIMApplyImage (flags 
= 0x180)...

This will cause moderate disk activity on the host and high CPU activity because of the image decompression. These can be monitored with gstat and top to get a sense of a "normal" installation. This knowledge will be useful for blind installations of desktop versions of Windows that do not include the SAC.

If you attempt to boot to a blank disk image, you will see: 

Booting EFI Network
InstallProtocolInterface: 245DCA21-FB7B-11D3-8F01-00A0C969723B BEC7C0E0
.

Windows Hard Disk Boot

Remove or adjust the ahci-cd,install.iso line of the bhyve command as appropriate and allow the VM to boot to the hard disk image. This will involve a SYSPREP boot that will attempt a restart. Boot the system again and you should eventually get a fully-installed system that will default to DHCP networking and be available by CMD in the SAC and RDP.

From the CMD: 

Please enter login credentials.
Username: Administrator
Domain  : 
Password: *******

Note that the delete key does not work and you must type carefully. The default password in the XML file is "Test123"

You can determine the DHCP IP of the VM from the SAC: 

SAC>i
Net: 3, Ip=192.186.1.14  Subnet=255.255.255.0  Gateway=192.168.1.1
Net: 3, Ip=fe80::b423:80d7:e553:8b02

Note that you might not be able to ping the Windows virtual machine because of the default firewall configuration.

RDP Connection

You can initiate an RDP connection to this IP address with: 

xfreerdp /v:192.186.1.14 /u:Administrator /p:Test123

The Windows, OSX, iOS and Android RDP clients from Microsoft will also work.

Note that xfreerdp stores host fingerprints in a manner similar to OpenSSH and will prevent you from connecting to a host whose fingerprint has changed. These keys can be managed in ~/.config/freerdp/known_hosts}

UTC Time

Windows clocks use local time by default. To use UTC instead, add -u to the guest's bhyve command and run the following in the guest. 

reg add "HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\TimeZoneInformation" /v RealTimeIsUniversal /d 1 /t REG_DWORD /f

This can fix timekeeping problems during daylight saving time transitions.

Notes

Windows Server "Essentials" and other variations have one installation image rather than four. These will give a "missing image" error during installation if you do not set the correct "IMAGE/INDEX" value to "1": 

<MetaData wcm:action="add">
   <Key>/IMAGE/INDEX</Key>
   <Value>1</Value>
</MetaData>

You can list the available install images on a given ISO with: 

7z l iso-remaster/sources/install.wim|more
...
Images = 4
...

This page is an expansion of this documentation:

CategoryVirtualization

bhyve/Windows (last edited 2024-03-10T18:54:22+0000 by MichaelDexter)