FreeBSD Bhyve Virtualization

The Bhyve FreeBSD hypervisor (called/spelled ‘beehive’ usually) was created almost 10 years ago. Right now it offers speed and features that other similar solutions provide – such as KVM/VMware/XEN. You can check all the details in the FreeBSD Handbook for details. One of the last things Bhyve lacks is so called live migration between physical hosts but save state and resume from saved state are in the works currently so not long before that live migration. Up until recently I used mostly VirtualBox for my small virtualization needs. Recently I started to evaluate Bhyve and this time I am very pleased – the FreeBSD VirtualBox integration is not perfect anyway – for example – the USB passthru does not work since several years – and even when it worked – it was limited to USB 1.x speeds only. Also because of FreeBSD policy of pkg(8) packages building process – the VirtualBox packages remain broken for 3 months after each *.1 or upper release (*.2/*.3/…). The other impulse that forced me to switch from VirtualBox to Bhyve was the VirtualBox (in)stability. I often needed to restart crashed VirtualBox VMs because they failed for some unspecified reason.

One of the Bhyve features that I especially liked was that by default Bhyve only uses memory that guest system wanted to allocate. For example a FreeBSD virtual machine with 2 GB RAM set will use after boot about 70 MB RAM 🙂

Another great feature I really liked about Bhyve was that I could suspend the host machine with all the VMs started – both on my ThinkPad W520 laptop and AMD Based FreeBSD Desktop and then it all successfully resumed. With VirtualBox you would have to poweroff all VMs because if you suspend with running VirtualBox VMs – it will just crash – its not possible to do suspend/resume cycle with VirtualBox.

The Table of Contents for this article is as follows:

• FreeBSD Bhyve Virtualization
• Bhyve Managers
• Bhyve versus KVM and VMware
• Bhyve libvirt/virt-manager GUI
• vm-bhyve
  • Install/Setup
  • Networking
    • Server/Desktop LAN Bridge
    • Laptop WiFi NAT
    • Networking Restart
  • Datastores
  • Templates
  • NVMe
  • ISO Images
  • Guest OS Install
    • FreeBSD
    • Linux
    • Windows 7
    • Windows 10
      • Force Windows 10 Offline Account
      • Windows 10 Bloat Removers
    • Windows 11
  • Dealing with Locked VMs
  • Disk Resize
• Summary

Bhyve Managers

While VirtualBox has quite usable QT based GUI – the Bhyve does not have anything like that. I once seen some GUI QT prototype for Bhyve but it was very basic – so forget about that currently. There are however several web interfaces such as TrueNAS CORE or CBSD/CloneOS. There are also several terminal managers such as vm-bhyve. The older one iohyve has not been maintained for at least 6 long years. There is also libvirt Bhyve driver but more on that later.

Bhyve versus KVM and VMware

Klara Systems compared Bhyve to KVM and Benjamin Bryan compared it against VMware hypervisor. While Bhyve remains competitive against both of them there are two important aspects from Klara Systems that stand out and are worth repeating here.

First – using nvme driver is a lot faster then using more traditional virtio-blk or ahci-hd backends.

Second – and this one seems strange – using a raw file is faster then using ZFS zvol device.

To summarize these thoughts – just just file on a disk like disk0.img and use nvme as storage backend everytime the guest operating system supports it.

Bhyve libvirt/virt-manager GUI

Theoretically the libvirt virtualization API supports Bhyve as one of its backends and the details about Bhyve driver – https://libvirt.org/drvbhyve.html – are available here. I have tried it with virt-manager and after some basic configuration I was able to start FreeBSD 13.2 installation … but it got frozen at the kernel messages and nothing more happened.

… and the moment it hanged below. I have tried multiple times with the same effect.

I really liked the virtual machine settings window of virt-manager.

vm-bhyve

While You can use Bhyve directly with bhyve(8) and bhyvectl(8) commands – which I was doing in the past – after trying the vm-bhyve both on the desktop and server space – I really liked it and this is what I currently use. I just moved from vm-bhyve package to the newer vm-bhyve-devel one.

The vm(8) command is simple and provides all needed use cases.

host # vm help
vm-bhyve: Bhyve virtual machine management v1.6-devel (rev. 106001)
Usage: vm ...
    version
    init
    set [setting=value] [...]
    get [all|setting] [...]
    switch list
    switch info [name] [...]
    switch create [-t type] [-i interface] [-n vlan-id] [-m mtu] [-a address/prefix-len] [-b bridge] [-p] 
    switch vlan  
    switch nat  
    switch private  
    switch add  
    switch remove  
    switch destroy 
    datastore list
    datastore add  
    datastore remove 
    datastore add  
    list
    info [name] [...]
    create [-d datastore] [-t template] [-s size] [-m memory] [-c vCPUs] 
    install [-fi]  
    start [-fi]  [...]
    stop  [...]
    restart 
    console  [com1|com2]
    configure 
    rename  
    add [-d device] [-t type] [-s size|switch] 
    startall
    stopall
    reset  [-f] 
    poweroff [-f] 
    destroy [-f] 
    passthru
    clone  
    snapshot [-f] 
    rollback [-r] 
    iso [url]
    img [url]
    image list
    image create [-d description] [-u] 
    image destroy 
    image provision [-d datastore]  

Install/Setup

We need only several packages to add.

host # pkg install -y \
         vm-bhyve-devel \
         uefi-edk2-bhyve-csm \
         bhyve-firmware \
         edk2-bhyve \
         dnsmasq \
         grub2-bhyve \
         tigervnc-viewer \
         rdesktop

The setup is pretty easy also.

First we need to add several vm_* settings into the main FreeBSD /etc/rc.conf file.

  vm_enable=YES
  vm_dir="zfs:zroot/vm"
  vm_list=""
  vm_delay=3

Keep in mind that you will later use the vm_list="" for the list of VMs that you would like to be started at boot. Like vm_list="freebsd13 freebsd14uefi" for example. Then the vm list command would place [1] in at the freebsd13 name (as its first) and [2] in the freebsd14uefi name as this one is second on the list. See below.

host # vm list
NAME           DATASTORE  LOADER     CPU  MEMORY  VNC           AUTO     STATE
almalinux8     default    uefi       2    2G      0.0.0.0:5908  No       Running (11819)
freebsd13      default    bhyveload  1    256M    -             Yes [1]  Running (2342)
freebsd14      default    bhyveload  1    256M    -             No       Stopped
freebsd14uefi  default    uefi       2    8G      -             Yes [2]  Running (35394)
windows10      default    uefi       2    2G      -             No       Stopped
windows7       default    uefi       2    2G      -             No       Stopped

We need to create a dedicated ZFS dataset for our VMs. You can also use directory on UFS – check vm-bhyve documentation.

host # zfs create -o mountpoint=/vm zroot/vm

We will also copy the available templates to our new /vm dir.

host # cp -a /usr/local/share/examples/vm-bhyve /vm/.templates

Remember to check /vm/.templates/config.sample as it has the documentation for all available options.

host # head -12 /vm/.templates/config.sample
# This is a sample configuration file containing all supported options
# Please do not try and use this file itself for a guest
# For any option that contains a number in the name, such as "network0_type",
# you can add additional devices of that type by creating a new set of
# variables using the next number in sequence, e.g "network1_type"
#
# Please make sure all option names are specified in lowercase and
# at the beginning of the line. If there is any whitespace before
# the option name, the line will be ignored.
# The '#' character signifies the start of a comment, even within
# double-quotes, and so cannot be used inside any values.

We can now start initialize the vm-bhyve.

host # service vm start

Networking

There as many network setups as many FreeBSD has network capabilities – a lot! I this guide I will cover two most typical network setups for Bhyve. One would be the most server (or desktop) oriented – as it requires a LAN card to be used. The other one I would call a laptop one – that one would provide network connectivity using wlan0 WiFi interface.

No matter which one we will choose – we need to enable port forwarding on our FreeBSD host. Do that with these two commands.

host # sysrc gateway_enable=YES

host # sysctl net.inet.ip.forwarding=1

host # echo net.link.tap.up_on_open=1 >> /etc/sysctl.conf

host # sysctl net.link.tap.up_on_open=1

I assume that our FreeBSD host system would use 10.0.0.10/24 IP address and that 10.0.0.1 would be its default gateway.

Your host system main /etc/rc.conf file can looks as follows then.

host # cat /etc/rc.conf
# NETWORK
  hostname=host
  ifconfig_re0="inet 10.0.0.10/24 up"
  defaultrouter="10.0.0.1"
  gateway_enable=YES

# DAEMONS
  sshd_enable=YES
  zfs_enable=YES

# BHYVE
  vm_enable="YES"
  vm_dir="zfs:zroot/vm"
  vm_list=""
  vm_delay="3"

Server/Desktop LAN Bridge

We will use 10.0.0.0/24 network – the same that our host system uses. We will need one bridge/switch named vm-public with 10.0.0.100/24 address on it. Without that address later the dnsmasq will complain unknown interface vm-public about it. Information about the switches is kept in the /vm/.config/system.conf file. We will also need to add out LAN interface to the public switch. It will be re0 interface in my case.

host # vm switch create -a 10.0.0.100/24 public

host # vm switch add public re0

host # vm switch list
NAME    TYPE      IFACE      ADDRESS        PRIVATE  MTU  VLAN  PORTS
public  standard  vm-public  10.0.0.100/24  no       -    -     re0

host # cat /vm/.config/system.conf
switch_list="public"
type_public="standard"
addr_public="10.0.0.100/24"
ports_public="re0"

To be honest the networking part setup is complete.

When you will be setting up your Bhyve VMs you will either use static 10.0.0.0/24 IP address space or just use DHCP and the one that is already on your network will take care of the rest (assuming you have one).

If you do not have one you may use dnsmasq service to do that easily.

host # cat /usr/local/etc/dnsmasq.conf
port=0
no-resolv
server=1.1.1.1
except-interface=lo0
bind-interfaces
local-service
dhcp-authoritative
interface=vm-public
dhcp-range=10.0.0.69,10.0.0.96

host # service dnsmasq enable

host # service dnsmasq start

That should do.

Now – to access the VMs in this bridged networking mode you can just ssh(1) to their IP address directly.

Laptop WiFi NAT

This is one of the cases where VirtualBox has one more feature over Bhyve. With VirtualBox its possible to use bridge mode over WiFi interface. Its not possible with Bhyve currently. I have submitted a proposal to FreeBSD Foundation to implement such configuration – especially as open source VirtualBox code already exists. Time will tell if it will be implemented or if there would be more important tasks to take care of.

We will use 10.1.1.0/24 network for our VM needs. We will also need only one vm-bhyve switch that we will use – and it will be the vm-public one with 10.1.1.1/24 address – we will be using it as a gateway for our VMs in NAT. Information about the switches is kept in the /vm/.config/system.conf file.

host # vm switch create -a 10.1.1.1/24 public

host # vm switch list
NAME    TYPE      IFACE      ADDRESS      PRIVATE  MTU  VLAN  PORTS
public  standard  vm-public  10.1.1.1/24  no       -    -     -

host # cat /vm/.config/system.conf 
switch_list="public"
type_public="standard"
addr_public="10.1.1.1/24"

Now the NAT part – we will do that with very simple pf(4) config.

host # cat /etc/pf.conf
# SKIP LOOPBACK
  set skip on lo0

# bhyve(8) VMS NAT 
  nat on wlan0 from {10.1.1.1/24} to any -> (wlan0)

# PASS IN/OUT ALL
  pass in all
  pass out all

host # service pf enable

host # service pf start

You can check the stats of that pf(4) rules like that.

host # pfctl -Psn -vv
No ALTQ support in kernel
ALTQ related functions disabled
@0 nat on wlan0 inet from 10.1.1.0/24 to any -> (wlan0) round-robin
  [ Evaluations: 18774     Packets: 362277    Bytes: 352847937   States: 0     ]
  [ Inserted: uid 0 pid 69837 State Creations: 38    ]

Feel free to add all your pf(4) rules into the /etc/pf.conf file.

Now the DHCP server. For simplicity of the setup we will use dnsmasq daemon – but nothing prevents you from setting up a Highly Available DHCP Server instead using isc-dhcp44-server package.

host # cat /usr/local/etc/dnsmasq.conf
port=0
no-resolv
server=1.1.1.1
except-interface=lo0
bind-interfaces
local-service
dhcp-authoritative
interface=vm-public
dhcp-range=10.1.1.11,10.1.1.99

host # service dnsmasq enable

host # service dnsmasq start

Now you should be ready to setup Bhyve VMs on your laptop.

When I was using VirtualBox – it allowed me to use its Port Forwarding feature where I could add as many mappings in the NAT network type as shown below.

… but with Bhyve its even better as no port forwarding in the NAT mode is needed at all! 🙂

You can access the Bhyve VMs in NAT networking mode the same as you can in the bridged mode – with just ssh(1) to their IP address from directly from the 10.1.1.0/24 range.

So to put it bluntly – if a Bhyve VM in NAT mode got 10.1.1.33/24 IP address – then just ssh(1) to that IP address directly from the host system.

Networking Restart

Sometimes – when for example you laptop will boot without network connectivity – the tap(4) interfaces sometimes do not went UP.

There is simple fix tor that problem – bhyve-network-restart.sh script.

Its shown below.

# ADD IP ADDRESS TO EACH vm-bhyve SWITCH
vm switch list \
  | sed 1d \
  | while read NAME TYPE IFACE ADDRESS PRIVATE MTU VLAN PORTS a0 a1 a2 a3 a4 a5 a6 a7 a8 a9
    do
      if [ "${ADDRESS}" != "-" ]
      then
             vm switch address ${NAME} ${ADDRESS}
        echo vm switch address ${NAME} ${ADDRESS}
      fi
    done        

# SET TO 'up' ALL vm-bhyve SWITCH MEMBERS
vm switch list \
  | sed 1d \
  | awk '{print $1}' \
  | while read SWITCH
    do
      ifconfig vm-${SWITCH} \
        | awk '/member:/ {print $2}' \
        | while read INTERFACE
          do
                 ifconfig ${INTERFACE} up
            echo ifconfig ${INTERFACE} up
          done
    done

Execute it everytime you lost connectivity with your VMs and you are done.

Datastores

While vm-bhyve supports multiple datastores – you will only need one – the default one.

host # vm datastore list
NAME            TYPE        PATH                      ZFS DATASET
default         zfs         /vm                       zroot/vm

Snapshots and Clones

The vm-bhyve also supports snapshots and clones of the VMs disks. Generally they are just ZFS snapshots and clones.

Templates

While vm-bhyve comes with several handy templates – they are incomplete – and small several changes makes the game more playable.

NVMe

First – we will implement the things that we know work faster – the nvme type for disk images instead of virt-blk or ahci-hd ones. Of course not all operating systems have support for such devices – for them we will use the latter options.

A fast way to change it to nvme is below.

host # sed -i '' s.virtio-blk.nvme.g /vm/.templates/freebsd.conf

ISO Images

Each VM needs an ISO image from which it will be installed. Of course you can also just create new VM and copy the disk contents from other server or use one of the FreeBSD images.

There are two ways to feed vm-bhyve with ISO images.

One is to fetch them from some URL.

host # vm iso http://ftp.freebsd.org/pub/FreeBSD/releases/ISO-IMAGES/13.2/FreeBSD-13.2-RELEASE-amd64-disc1.iso

host # vm iso
DATASTORE           FILENAME
default             FreeBSD-13.2-RELEASE-amd64-disc1.iso

The other way is to just simple copy ISO file to the /vm/.iso directory.

host # cp /home/vermaden/download/ubuntu-mate-23.04-desktop-amd64.iso /vm/.iso/

host # vm iso
DATASTORE           FILENAME
default             FreeBSD-13.2-RELEASE-amd64-disc1.iso
default             ubuntu-mate-23.04-desktop-amd64.iso

Guest OS Install

Generally each VM install is very similar as shown below.

host # vm create -t TEMPLATE NAME

host # vm install NAME ISO

host # vm console NAME

Example for FreeBSD is below.

host # vm create -t freebsd freebsd13

host # vm install freebsd13 FreeBSD-13.2-RELEASE-amd64-disc1.iso
Starting freebsd13
  * found guest in /vm/freebsd13
  * booting...

host # vm console freebsd13

You will probably see something like that below.

Then you do the installation in the text mode and after reboot you have your running FreeBSD VM.

host # vm list
NAME          DATASTORE  LOADER     CPU  MEMORY  VNC  AUTO     STATE
freebsd13     default    bhyveload  1    256M    -    Yes [1]  Running (85315)

Some more info to display can be shown with info argument.

host # vm info freebsd13
------------------------
Virtual Machine: freebsd13
------------------------
  state: stopped
  datastore: default
  loader: bhyveload
  uuid: a91287a1-39d3-11ee-b73d-f0def1d6aea1
  cpu: 1
  memory: 256M

  network-interface
    number: 0
    emulation: virtio-net
    virtual-switch: public
    fixed-mac-address: 58:9c:fc:0b:98:30
    fixed-device: -

  virtual-disk
    number: 0
    device-type: file
    emulation: nvme
    options: -
    system-path: /vm/freebsd13/disk0.img
    bytes-size: 21474836480 (20.000G)
    bytes-used: 885089280 (844.086M)

  snapshots
    zroot/vm/freebsd13@fresh    85.2M   Mon Aug 14 11:18 2023

host # env EDITOR=cat vm configure freebsd13
loader="bhyveload"
cpu=1
memory=256M
network0_type="virtio-net"
network0_switch="public"
disk0_type="nvme"
disk0_name="disk0.img"
uuid="a91287a1-39d3-11ee-b73d-f0def1d6aea1"
network0_mac="58:9c:fc:0b:98:30"

If you want to edit and not only display the VM config use this.

host # vm configure freebsd13

FreeBSD

FreeBSD can be boot in two ways. One is with bhyveload which may be translated to legacy BIOS boot. You can also of course boot FreeBSD un UEFI mode.

host # cat /vm/.templates/freebsd.conf
loader="bhyveload"
cpu=1
memory=256M
network0_type="virtio-net"
network0_switch="public"
disk0_type="nvme"
disk0_name="disk0.img"

The above will use bhyveload and it mostly works … but sometimes if you want to install a lot newer version under Bhyve the loader may not have all the needed features. I was hit by this problem recently where I used FreeBSD 13.2-RELEASE for the FreeBSD host system and wanted to try 14.0-ALPHA1 version.

I described the details of this problem here – FreeBSD Bug 273099 – in a BUG report.

This is how such error looks like:

| FreeBSD/amd64 User boot lua, Revision 1.2
| ZFS: unsupported feature: com.klarasystems:vdev_zaps_v2
| ERROR: cannot open /boot/lua/loader.lua: no such file or directory.
| 
| Type '?' for a list of commands, 'help' for more detailed help.
| OK 

To overcome that you will need latest (more up to date then 14.0-ALPHA1 version) FreeBSD sources and below commands.

host # pkg install gitup

host # cp /usr/local/etc/gitup.conf.sample /usr/local/etc/gitup.conf

host # gitup current

host # cd /usr/src/stand

host # make

host # find /usr/obj -type f -name userboot_lua.so
/usr/obj/usr/src/amd64.amd64/stand/userboot/userboot_lua/userboot_lua.so

host # cp /usr/obj/usr/src/amd64.amd64/stand/userboot/userboot_lua/userboot_lua.so /vm/userboot_lua.so

Now – we need to add bhyveload_loader="/vm/userboot_lua.so" option to out FreeBSD 14.0-ALPHA1 machine config.

host # cat /vm/freebsd14/freebsd14.conf
loader="bhyveload"
bhyveload_loader="/vm/userboot_lua.so"
cpu=1
memory=256M
network0_type="virtio-net"
network0_switch="public"
disk0_type="nvme"
disk0_name="disk0.img"
uuid="975bca2a-39c4-11ee-b73d-f0def1d6aea1"
network0_mac="58:9c:fc:03:67:47"

Now it will boot properly.

Of course it was very easy to overcome that using UEFI boot instead.

host # cat /vm/freebsd14uefi/freebsd14uefi.conf
loader="uefi"
cpu=1
memory=256M
network0_type="virtio-net"
network0_switch="public"
disk0_type="nvme"
disk0_name="disk0.img"
uuid="35ca42b7-7f28-43eb-afd9-2488c5ec83cf"
network0_mac="58:9c:fc:0a:16:4b"

Linux

By default for Linux the grub way is the proposed way. I do not use it as at it olny allows console access – and even many so called enterprice grade Linux distributions such as AlmaLinux or Rocky have graphical installer that needs/wants graphical display … and that is only available in uefi mode.

Maybe for Alpine or Void Linux such approach may be usable … but uefi will also work very well – thus I do not see ANY advantages of using grub way here.

I will show you the next example based on AlmaLinux 8.x install but the same worked properly with Ubuntu Mate for example.

First the default template.

host # cat /vm/.templates/linux.conf
loader="uefi"
cpu=2
memory=4G
network0_type="virtio-net"
network0_switch="public"
disk0_type="nvme"
disk0_name="disk0.img"
xhci_mouse="yes"
graphics="yes"

The above added xhci_mouse="yes" uses more precise xhci(4) USB 3.x mouse driver and graphics="yes" forces the exposure of VNC connection.

With such template the installation looks like that.

host # cp AlmaLinux-8.8-x86_64-minimal.iso /vm/.iso/

host # vm create -t linux almalinux8

host # vm install almalinux8 AlmaLinux-8.8-x86_64-minimal.iso
Starting almalinux8
  * found guest in /vm/almalinux8
  * booting...

host # vm list
NAME           DATASTORE  LOADER     CPU  MEMORY  VNC           AUTO  STATE
almalinux8     default    uefi       2    4G      0.0.0.0:5900  No    Running (11819)

host % vncviewer -SendClipboard -AcceptClipboard -LowColorLevel -QualityLevel 6 :5900 &

The last vncviewer(1) command is executed as regular user. It comes from net/tigervnc-viewer package.

If you will be connecting to some external server then use IP address in the command.

host % vncviewer -SendClipboard -AcceptClipboard -LowColorLevel -QualityLevel 6 10.0.0.66::5900 &

After the Linux system is installed you may specify the exact VNC port or IP address. Also the screen resolution or enable/disable waiting for the VNC connection.

graphics_port="5900"
graphics_listen="0.0.0.0"
graphics_res="1400x900"
graphics_wait="no"

Windows 7

A lot of people will criticize me for this one – as Windows 7 is not an officially supported version anymore. I do not care about that when I want to use some localhost software … or older software that works better on older version. Not to mention that its one of the last Windows versions that does not force online Microsoft account down your throat. It also uses less resources and is more responsive.

First – the template – similar to the Linux one.

host # cat /vm/.templates/windows7.conf           
loader="uefi"
graphics="yes"
cpu=2
memory=2G
ahci_device_limit="8"
network0_type="e1000"
network0_switch="public"
disk0_type="ahci-hd"
disk0_name="disk0.img"
disk0_opts="sectorsize=512"
utctime="no"
bhyve_options="-s 8,hda,play=/dev/dsp,rec=/dev/dsp"

If you set the xhci_mouse="yes" option with Windows 7 – you will end up without a working mouse in VNC connection and you will have to make all the install and configuration by keyboard only.

One may think about adding xhci_mouse="yes" after installation when you will already have working RDP connection – but that would also reqiure additional drivers. In theory – the device VEN_8086&DEV_1E31 name is recognized as Intel USB 3.0 eXtensible Host Controller … but for some reason anytime I wanted to install it – the Windows 7 system crashed and instantly rebooted.

The other even more imporant thing is having the disk0_opts="sectorsize=512" option. Without it the Windows 7 instaler will fail with the following error.

The last option bhyve_options="-s 8,hda,play=/dev/dsp,rec=/dev/dsp" enables audio.

The install procedure is also similar to Linux.

host # cp win_7_amd64_sp1_en.iso /vm/.iso/

host # vm iso
DATASTORE           FILENAME
default             win_7_amd64_sp1_en.iso

host # vm create -t windows7 -s 40G windows7

host # vm install windows7 win_7_amd64_sp1_en.iso
Starting windows7
  * found guest in /vm/windows7
  * booting...

host # vm list
NAME           DATASTORE  LOADER     CPU  MEMORY  VNC           AUTO  STATE
windows7       default    uefi       2    2G      0.0.0.0:5900  No    Running (11819)

host % vncviewer -SendClipboard -AcceptClipboard -LowColorLevel -QualityLevel 6 :5900 &

After the install we should enable RDP connections for more features. Rememeber to select any version option.

You can add one or more CD-ROM drives with following options in the configure argument.

disk1_type="ahci-cd"
disk1_dev="custom"
disk1_name="/vm/.iso/virtio-drivers.iso"

It would be easier for RDP connections to have static IP instead of a DHCP one.

Now as we have the static 10.1.1.7 IP address we can use RDP connection with rdesktop(1) command.

host % rdesktop -u buser -p bpass -P -N -z -g 1800x1000 -a 24 -r sound:local -r disk:HOME=/home/vermaden 10.1.1.7
Autoselecting keyboard map 'en-us' from locale

ATTENTION! The server uses and invalid security certificate which can not be trusted for
the following identified reasons(s);

 1. Certificate issuer is not trusted by this system.

     Issuer: CN=vbox


Review the following certificate info before you trust it to be added as an exception.
If you do not trust the certificate the connection atempt will be aborted:

    Subject: CN=vbox
     Issuer: CN=vbox
 Valid From: Mon Aug 14 00:58:25 2023
         To: Mon Feb 12 23:58:25 2024

  Certificate fingerprints:

       sha1: 4ad853c40a8aa0560af315b691038202506e07ce
     sha256: 44ec8f7650486aef6261aea42da99caba4e84d7bc58341c0ca1bb8e28b81d222


Do you trust this certificate (yes/no)? yes
Connection established using SSL.

There are several useful options here.

The -u buser and -p bpass will take care of credentials.

The -P option enables caching of bitmaps to disk (persistent bitmap caching). This improves performance (especially on low bandwidth connections) and reduces network traffic.

The -N option enables numlock synchronization between the X11 server and remote RDP session.

The -z enables compression of the RDP datastream.

The -g 1800x1000 and -a 24 specifies resolution and color depth rate.

The -r disk:HOME=/home/vermaden enables transparent sharing of your home directory and additional share is shown in My Computer in the Windows 7 machine – very handy for sharing files between the host and guest VM as chown below.

The last one option -r sound:local specifies that the audio will be realized on the guest VM – this will only work if you added the bhyve_options="-s 8,hda,play=/dev/dsp,rec=/dev/dsp" to the Windows 7 Bhyve config. Alternatively without that hda(4) emulation you can use -r sound:remote option – this would use RDP protocol to transfer audio events from the guest machine to your host machine and then audio will be played then locally on your host machine.

Windows 10

Finally a supported version.

Template is similar to the Windows 7 one.

host # cat /vm/.templates/windows10.conf
loader="uefi"
graphics="yes"
xhci_mouse="yes"
cpu=2
memory=2G
ahci_device_limit="8"
network0_type="e1000"
network0_switch="public"
disk0_type="nvme"
disk0_name="disk0.img"
utctime="no"
bhyve_options="-s 8,hda,play=/dev/dsp,rec=/dev/dsp"

The Windows 10 supports the xhci_mouse="yes" so we enable and keep it all the time.

The Windows 10 does not need the disk0_opts="sectorsize=512" option.

As Windows 10 is newer – the nvme can (and should) be used for performance reasons.

The last option bhyve_options="-s 8,hda,play=/dev/dsp,rec=/dev/dsp" enables audio.

The install procedure is also similar to Windows 7.

host # cp win_10_amd64_en_LTSC.iso /vm/.iso/

host # vm iso
DATASTORE           FILENAME
default             win_10_amd64_en_LTSC.iso

host # vm create -t windows10 -s 40G windows10

host # vm install windows10 win_10_amd64_en_LTSC.iso
Starting windows10
  * found guest in /vm/windows10
  * booting...

host # vm list
NAME           DATASTORE  LOADER     CPU  MEMORY  VNC           AUTO  STATE
windows10      default    uefi       2    2G      0.0.0.0:5900  No    Running (11819)

host % vncviewer -SendClipboard -AcceptClipboard -LowColorLevel -QualityLevel 6 :5900 &

After the install we should enable RDP connections for more features. Remember to select any version option.

You can add one or more CD-ROM drives with following options in the configure argument.

disk1_type="ahci-cd"
disk1_dev="custom"
disk1_name="/vm/.iso/virtio-drivers.iso"

It would be easier for RDP connections to have static IP instead of a DHCP one.

Now as we have the static 10.1.1.8 IP address we can use RDP connection with rdesktop(1) command.

host % rdesktop -u buser -p bpass -P -N -z -g 1600x900 -a 24 -r sound:local -r disk:HOME=/home/vermaden 10.1.1.8
Autoselecting keyboard map 'en-us' from locale

ATTENTION! The server uses and invalid security certificate which can not be trusted for
the following identified reasons(s);

 1. Certificate issuer is not trusted by this system.

     Issuer: CN=DESKTOP-HKJ3H6T


Review the following certificate info before you trust it to be added as an exception.
If you do not trust the certificate the connection atempt will be aborted:

    Subject: CN=DESKTOP-HKJ3H6T
     Issuer: CN=DESKTOP-HKJ3H6T
 Valid From: Mon Aug 14 10:33:41 2023
         To: Tue Feb 13 09:33:41 2024

  Certificate fingerprints:

       sha1: 967d5cdb164e53f7eb4c5c17b0343f2f279fb709
     sha256: c08b732122a39c44d91fac2a9093724da12d2f3e6ea51613245d13cf762f4cd2


Do you trust this certificate (yes/no)? yes

Options are the same as with Windows 7 and they are described in the Windows 7 section.

Force Windows 10 Offline Account

To force creation of local account instead of forced online account you need to boot the Windows 10 without network.

Do the following steps to do that.

host # yes | vm poweroff windows10

host # vm configure windows10
- network0_type="e1000"
- network0_switch="public"

host # vm start windows10

Now create the offline account.

After creating it poweroff the Windows 10 VM.

host # vm configure windows10
+ network0_type="e1000"
+ network0_switch="public"

host # vm start windows10

Now you have local account on Windows 10 system.

Windows 10 Bloat Removers

You may consider using on of the known Windows 10 bloat removers available here:

• https://github.com/Sycnex/Windows10Debloater
• https://github.com/W4RH4WK/Debloat-Windows-10
• https://github.com/LeDragoX/Win-Debloat-Tools
• https://github.com/ChrisTitusTech/winutil

Windows 11

The setup/install of Windows 11 is the same as Windows 10.

Dealing with Locked VMs

Lets assume that our host system crashed.

The vm-bhyve will left run.lock files in the machines dirs.

host # ls -l /vm/freebsd14uefi
total 1389223K
-rw-r--r-- 1 root wheel          32 2023-08-16 23:36 console
-rw------- 1 root wheel 21474836480 2023-08-16 23:46 disk0.img
-rw-r--r-- 1 root wheel         200 2023-08-16 23:35 freebsd14uefi.conf
-rw-r--r-- 1 root wheel          11 2023-08-16 23:36 run.lock
-rw-r--r-- 1 root wheel        5583 2023-08-16 23:36 vm-bhyve.log

host # vm list
NAME           DATASTORE  LOADER     CPU  MEMORY  VNC  AUTO     STATE
almalinux8     default    uefi       2    2G      -    No       Stopped
freebsd13      default    bhyveload  1    256M    -    Yes [1]  Running (19258)
freebsd13alt   default    bhyveload  1    256M    -    No       Stopped
freebsd14      default    bhyveload  1    256M    -    No       Stopped
freebsd14uefi  default    uefi       2    8G      -    No       Locked (w520.local)
windows10ltsc  default    uefi       2    2G      -    No       Stopped
windows7       default    uefi       2    2G      -    No       Stopped

host # rm /vm/freebsd14uefi/run.lock

host # vm list
NAME           DATASTORE  LOADER     CPU  MEMORY  VNC  AUTO     STATE
almalinux8     default    uefi       2    2G      -    No       Stopped
freebsd13      default    bhyveload  1    256M    -    Yes [1]  Running (19258)
freebsd13alt   default    bhyveload  1    256M    -    No       Stopped
freebsd14      default    bhyveload  1    256M    -    No       Stopped
freebsd14uefi  default    uefi       2    8G      -    No       Stopped
windows10ltsc  default    uefi       2    2G      -    No       Stopped
windows7       default    uefi       2    2G      -    No       Stopped

Now you may want to start the locked machine properly.

Disk Resize

By default vm-bhyve will create disks with 20 GB in size.

To resize the Bhyve virtual machine disk we would use truncate(1) command.

host # vm stop freebsd13

host # cd /vm/freebsd13

host # truncate -s 40G disk0.img

host # vm start freebsd13

If you are not sure about that – you may work on a copy instead.

host # vm stop freebsd13

host # truncate -s 40G disk0.img.NEW

host # dd bs=1m if=disk0.img of=disk0.img.NEW conv=notrunc status=progress
  20865613824 bytes (21 GB, 19 GiB) transferred 43.002s, 485 MB/s
20480+0 records in
20480+0 records out
21474836480 bytes transferred in 43.454036 secs (494196586 bytes/sec)

host # mv disk0.img disk0.img.BACKUP

host # mv disk0.img.NEW disk0.img

host # vm start freebsd13

Now we need to resize the filesystem inside the VM.

freebsd13 # lsblk
DEVICE         MAJ:MIN SIZE TYPE                                          LABEL MOUNT
nvd0             0:90   40G GPT                                               - -
  nvd0p1         0:91  512K freebsd-boot                           gpt/gptboot0 -
           -:-   492K -                                                 - -
  nvd0p2         0:92  2.0G freebsd-swap                              gpt/swap0 SWAP
  nvd0p3         0:93   18G freebsd-zfs                                gpt/zfs0 
           -:-   1.0M -                                                 - -

freebsd13 # geom disk list
Geom name: nvd0
Providers:
1. Name: nvd0
   Mediasize: 42949672960 (40G)
   Sectorsize: 512
   Mode: r2w2e3
   descr: bhyve-NVMe
   lunid: 589cfc2081410001
   ident: NVME-4-0
   rotationrate: 0
   fwsectors: 0
   fwheads: 0

freebsd13 # gpart show
=>      40  41942960  nvd0  GPT  (40G) [CORRUPT]
        40      1024     1  freebsd-boot  (512K)
      1064       984        - free -  (492K)
      2048   4194304     2  freebsd-swap  (2.0G)
   4196352  37744640     3  freebsd-zfs  (18G)
  41940992      2008        - free -  (1.0M)

freebsd13 # gpart recover nvd0
nvd0 recovered

freebsd13 # gpart show
=>      40  83886000  nvd0  GPT  (40G)
        40      1024     1  freebsd-boot  (512K)
      1064       984        - free -  (492K)
      2048   4194304     2  freebsd-swap  (2.0G)
   4196352  37744640     3  freebsd-zfs  (18G)
  41940992  41945048        - free -  (20G)

freebsd13 # gpart resize -i 3 -a 1m nvd0
nvd0p3 resized

freebsd13 # gpart show
=>      40  83886000  nvd0  GPT  (40G)
        40      1024     1  freebsd-boot  (512K)
      1064       984        - free -  (492K)
      2048   4194304     2  freebsd-swap  (2.0G)
   4196352  79687680     3  freebsd-zfs  (38G)
  83884032      2008        - free -  (1.0M)

freebsd13 # zpool status
  pool: zroot
 state: ONLINE
config:

        NAME        STATE     READ WRITE CKSUM
        zroot       ONLINE       0     0     0
          nvd0p3    ONLINE       0     0     0

freebsd13 # zpool list
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
zroot  17.5G  17.0G   544M        -         -    87%    96%  1.00x    ONLINE  -

freebsd13 # zpool set autoexpand=on zroot

freebsd13 # zpool online -e zroot nvd0p3

freebsd13 # zpool list
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
zroot  37.5G  17.0G  20.5G        -         -    41%    45%  1.00x    ONLINE  -

Summary

I hope I was able to provide all needed information.

Let me know in comments if I missed something.

UPDATE 1 – The sysutils/edk2 Issue

Recently a lot of people started to get problems with running UEFI machines on Bhyve. After a short investigation (details in the 273560 BUG report) the root of all cause was the new sysutils/edk2 version that caused the problem. The problem does not exists as long as You use the -A flag for bhyve(8) command. Unfortunately its not the default for vm-bhyve and -A options is needed in the bhyve_options parameter in each UEFI boot VM.

For example:

host % grep bhyve_options /vm/freebsd14uefi/freebsd14uefi.conf 
bhyve_options="-A"

Additional details for vm-bhyve available HERE.

Hope that helps.

EOF