Tag Archives: virtualization

FreeBSD Samba Share with FreeIPA/IDM Auth

I have covered FreeBSD with FreeIPA/IDM stuff many times before – and this time I did one step further. This guide will show you step by step how to setup FreeBSD based Samba server and serve shares with FreeIPA/IDM credentials. Many people do not reach reading to the end of the article so I will point that in the beginning that I am really grateful to Mariusz Zaborski (oshogbo) for his help – because without it – it just would not happen.

Last time I used Alma Linux 9.3 and many things seem to also change after I used the latest 9.4 version. First was different way of setting IP and gateway addresses and the second was different package names and other stuff related to yum(8) (or dnf(8) if you prefer – by some referred as Did Not Finished) package manager.

Older entries related to FreeBSD and FreeIPA/IDM are listed below:

This article will try to address and contain all steps needed – including setting up the FreeIPA/IDM server and Poudriere setup. All of these systems will be Bhyve virtual machines. Below You will find Table of Contents for this article.

  • List of Machines
  • FreeIPA/IDM Server – OS Install
  • FreeIPA/IDM Server – Setup
  • FreeIPA/IDM Server – Config
  • Poudriere Server – Setup
  • Poudriere Server – Build FreeIPA/IDM Client Packages
  • Poudriere Server – Update Repo/Packages
  • FreeBSD Samba Server – FreeIPA/IDM Connect
  • FreeBSD Samba Server – Configuration
  • Samba Linux Client
  • Samba Windows Client
  • Summary

Now to the content …

List of Machines

For the FreeIPA/IDM server and Linux test client I have used Alma Linux RHEL clone in most recent 9.4 version – but we know that Rocky Linux or Oracle Linux would also work well. I also tested some Windows 10 version to make sure it also works there.

If there is some reason I should use Rocky or Oracle instead – please let me know.

Linux FreeIPA/IDM server – idm.lab.org.

      OS: Alma Linux 9.4 amd64
      IP: 10.1.1.200/24
      GW: 10.1.1.1
  domain: lab.org
   realm: LAB.ORG
hostname: idm.lab.org

FreeBSD Poudriere Harvester – poudriere.lab.org.

      OS: FreeBSD 14.1-RELEASE amd64
      IP: 10.1.1.123/24
      GW: 10.1.1.1
     DNS: 1.1.1.1
  domain: -
   realm: -
hostname: poudriere.lab.org

FreeBSD Samba server – samba.lab.org.

      OS: FreeBSD 14.1-RELEASE amd64
      IP: 10.1.1.203/24
      GW: 10.1.1.1
     DNS: 10.1.1.200 (idm.lab.org)
  domain: lab.org
   realm: LAB.ORG
hostname: samba.lab.org

Linux Samba client – linux.lab.org.

      OS: Alma Linux 9.4 amd64
      IP: 10.1.1.202/24
      GW: 10.1.1.1
     DNS: 10.1.1.200 (idm.lab.org)
  domain: lab.org
   realm: LAB.ORG
hostname: linux.lab.org

Windows Samba client – windows.lab.org.

      OS: Windows 10 amd64
      IP: 10.1.1.201/24
      GW: 10.1.1.1
     DNS: 1.1.1.1
  domain: -
   realm: -
hostname: windows.lab.org

Here is how these machines are listed in vm(8) command.

host # vm list
NAME               DATASTORE  LOADER     CPU  MEMORY  VNC           AUTO  STATE
BASE-alma-9.4      default    uefi       2    2G      -             No    Stopped
BASE-freebsd-14.1  default    bhyveload  2    2G      -             No    Stopped
ghostbsd           default    uefi       2    8G      -             No    Stopped
idm                default    uefi       2    2G      0.0.0.0:5900  No    Running (82395)
samba              default    bhyveload  2    2G      -             No    Running (82431)
client             default    uefi       2    2G      0.0.0.0:5902  No    Running (27815)
poudriere          default    uefi       4    12G     0.0.0.0:5901  No    Running (14495)
windows10          default    uefi       2    4G      0.0.0.0:5903  No    Running (37341)
windows7           default    uefi       2    4G      -             No    Stopped

FreeBSD project provides ZFS based VM images … and we will use them – FreeBSD-14.1-RELEASE-amd64-zfs.raw.xz – this one exactly. As this guide will be too long anyway – please refer to previous articles about FreeBSD and FreeIPA/IDM on how to setup the VMs in Bhyve πŸ™‚

Now some legend and colors.

host # vm list                           // COMMANDS ON HOST SYSTEM
[root@idm ~]# kinit admin                // COMMANDS ON FreeIPA/IDM
root@poudriere:~ # service nginx enable  // COMMANDS ON FreeBSD Poudriere SERVER
root@samba:~ # net getdomainsid          // COMMANDS ON FreeBSD SAMBA SERVER
[root@client ~]# yum install cifs-utils  // COMMANDS ON Linux CLIENT

… and for the Windows machine we will point and click instead πŸ™‚

FreeIPA/IDM Server – OS Install

The installation of Alma Linux 9.4 is the default UEFI/LVM setup – I only added hostname – added IPv4 address and gateway – disabled IPv6 support.

In the earlier versions IP/GW information usually landed in the /etc/sysconfig/network-scripts/ifcfg-enp0s5 file for enp0s5 interface … not anymore. Whenever you like it or not – meet new nmcli(8) interface.

[root@idm ~]# nmcli
enp0s5: connected to enp0s5
        "Red Hat Virtio"
        ethernet (virtio_net), 58:9C:FC:0D:4C:DF, hw, mtu 1500
        ip4 default
        inet4 10.1.1.200/24
        route4 10.1.1.0/24 metric 100
        route4 default via 10.1.1.1 metric 100

lo: connected (externally) to lo
        "lo"
        loopback (unknown), 00:00:00:00:00:00, sw, mtu 65536
        inet4 127.0.0.1/8
        inet6 ::1/128

DNS configuration:
        servers: 127.0.0.1

        servers: 1.1.1.1
        domains: lab.org
        interface: enp0s5

Use "nmcli device show" to get complete information about known devices and
"nmcli connection show" to get an overview on active connection profiles.

Consult nmcli(1) and nmcli-examples(7) manual pages for complete usage details.

… and now for some example – to change IP address and gateway on a such Linux system you need to do these.

[root@idm ~]# nmcli con mod enp0s5 ipv4.address 10.1.1.202/24
[root@idm ~]# nmcli con mod enp0s5 ipv4.gateway 10.1.1.1
[root@idm ~]# nmcli networking off
[root@idm ~]# nmcli networking on

… and no – the old service network restart also does not work anymore.

[root@idm ~]# service network restart
Redirecting to /bin/systemctl restart network.service
Failed to restart network.service: Unit network.service not found.

… and keep in mind that You will be disconnected after the nmcli networking off so better run it inside GNU screen(1) for example … but Alma Linux repositories (same as RHEL ones) does not contain GNU screen(1) anymore.

[root@idm ~]# yum search screen
Last metadata expiration check: 2:30:04 ago on Thu Jun 20 11:03:55 2024.
====================================== Name & Summary Matched: screen ======================================
gnome-screenshot.x86_64 : A screenshot utility for GNOME
gnome-shell-extension-screenshot-window-sizer.noarch : Screenshot window sizer for GNOME Shell
=========================================== Name Matched: screen ===========================================
kacst-screen-fonts.noarch : Fonts for arabic from arabeyes project
========================================= Summary Matched: screen ==========================================
gnome-remote-desktop.x86_64 : GNOME Remote Desktop screen share service
gnome-shell-extension-heads-up-display.noarch : Display persistent on-screen message
gnome-shell-extension-window-list.noarch : Display a window list at the bottom of the screen in GNOME Shell
perl-Dumpvalue.noarch : Screen dump of Perl data
perl-Term-ANSIColor.noarch : Color screen output using ANSI escape sequences

Fortunately you can just download GNU screen(1) RPM from page like https://pkgs.org πŸ™‚

FreeIPA/IDM Server – Setup

Some more basic setup commands below.


[root@idm ~]# echo 10.1.1.200 idm.lab.org idm >> /etc/hosts

[root@idm ~]# hostnamectl set-hostname idm.lab.org

[root@idm ~]# timedatectl set-timezone Europe/Warsaw

[root@idm ~]# timedatectl set-local-rtc 0

[root@idm ~]# yum update -y

[root@idm ~]# reboot

This part is different then earlier as we do not do yum module enable idm:DL1 anymore … but its still needed on RHEL 8.x systems … consistency is not very well between RHEL and its clones these days.

[root@idm ~]# yum install -y bind-utils chrony nc ipa-server ipa-server-dns

[root@idm ~]# ipa-server-install                    \
                --domain lab.org                    \
                --realm LAB.ORG                     \
                --reverse-zone=1.1.10.in-addr.arpa. \
                --allow-zone-overlap                \
                --no-forwarders                     \
                --ntp-pool pool.ntp.org             \
                --setup-dns                         \
                --ds-password    password           \
                --admin-password password           \
                --unattended

(...)

This includes:
  * Configure a stand-alone CA (dogtag) for certificate management
  * Configure the NTP client (chronyd)
  * Create and configure an instance of Directory Server
  * Create and configure a Kerberos Key Distribution Center (KDC)
  * Configure Apache (httpd)
  * Configure DNS (bind)
  * Configure SID generation
  * Configure the KDC to enable PKINIT

(...)

==============================================================================
Setup complete

Next steps:
        1. You must make sure these network ports are open:
                TCP Ports:
                  * 80, 443: HTTP/HTTPS
                  * 389, 636: LDAP/LDAPS
                  * 88, 464: kerberos
                  * 53: bind
                UDP Ports:
                  * 88, 464: kerberos
                  * 53: bind
                  * 123: ntp

        2. You can now obtain a kerberos ticket using the command: 'kinit admin'
           This ticket will allow you to use the IPA tools (e.g., ipa user-add)
           and the web user interface.

Be sure to back up the CA certificates stored in /root/cacert.p12
These files are required to create replicas. The password for these
files is the Directory Manager password
The ipa-server-install command was successful

[root@idm ~]# ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
named Service: RUNNING
httpd Service: RUNNING
ipa-custodia Service: RUNNING
pki-tomcatd Service: RUNNING
ipa-otpd Service: RUNNING
ipa-dnskeysyncd Service: RUNNING
ipa: INFO: The ipactl command was successful

[root@idm ~]# systemctl enable httpd
[root@idm ~]# systemctl start  httpd

[root@idm ~]# systemctl disable firewalld
[root@idm ~]# systemctl stop    firewalld


You can now login to the FreeIPA/IDM web management at https://10.1.1.200/ page. Keep in mind that you will be redirected to https://idm.lab.org/ipa/ui/ immediately – so add 10.1.1.200 idm.lab.org idm to the hosts(5) file on the system from which you will be accessing it.

host # grep idm /etc/hosts
10.1.1.200  idm.lab.org  idm

After accessing it in the browser it will look like that.

You may login with admin username and the password you specified for the ipa-server-install(8) command or password if you just copy pasted that command πŸ™‚

We have our FreeIPA/IDM server installed.

FreeIPA/IDM Server – Config

… and after logging in I created a regular vermaden user as shown below.

Remember to reset your password by connecting to FreeIPA/IDM server with ssh(1) command.

host # ssh -l vermaden 10.1.1.200
(vermaden@10.1.1.200) Password:
(vermaden@10.1.1.200) Password expired. Change your password now.
Current Password:
(vermaden@10.1.1.200) New password:
(vermaden@10.1.1.200) Retype new password:

[vermaden@idm /]$ 

We need to now setup a HBAC Rule and Sudo Rule at FreeIPA/IDM.

Below HBAC Rule settings.

… and the Sudo Rule part.

Poudriere Server – Setup

One note for the FreeBSD commands below – please use /bin/sh shell (default for root since 14.0-RELEASE) for the commands … or zsh(1) for example … or other POSIX compatible shell. Some of these commands may not work properly on C based shells or in fish(1) shell.

If you want to know exact instructions to setup Bhyve VMs – please check my earlier FreeBSD with FreeIPA/IDM guides listed in the beginning of this article.

Configuration inside poudriere VM is defined below.

root@poudriere:~ # :> ~/.hushlogin

root@poudriere:~ # cat /etc/rc.conf
clear_tmp_enable="YES"
hostname="poudriere.lab.org"
ifconfig_vtnet0="inet 10.1.1.123/24"
defaultrouter="10.1.1.1"
sshd_enable="YES"
dumpdev="NO"
zfs_enable="YES"
nginx_enable="YES"


root@poudriere:~ # cat /etc/hosts
::1         localhost localhost.my.domain
127.0.0.1   localhost localhost.my.domain
10.1.1.123  poudriere.lab.org poudriere

root@poudriere:~ # service sshd start

root@poudriere:~ # mkdir -p /usr/local/etc/pkg/repos

root@poudriere:~ # sed -e 's|quarterly|latest|g' /etc/pkg/FreeBSD.conf > /usr/local/etc/pkg/repos/FreeBSD.conf

root@poudriere:~ # pkg install -y    \
                   beadm           \
                   lsblk           \
                   poudriere-devel \
                   nginx           \
                   git-lite        \
                   ccache4         \
                   tree

root@poudriere:~ # reboot

We will now setup actual Poudriere server.

root@poudriere:~ # export SSL=/usr/local/etc/ssl

root@poudriere:~ # mkdir -p \
                     /usr/ports/distfiles \
                     ${SSL}/keys \
                     ${SSL}/certs

root@poudriere:~ # chmod 0600 ${SSL}/keys

root@poudriere:~ # openssl genrsa -out ${SSL}/keys/poudriere.key 4096

root@poudriere:~ # openssl rsa \
                     -in  ${SSL}/keys/poudriere.key -pubout \
                     -out ${SSL}/certs/poudriere.cert

root@poudriere:~ # zfs create -p -o mountpoint=/var/ccache zroot/var/ccache

root@poudriere:~ # zfs list
NAME               USED  AVAIL  REFER  MOUNTPOINT
zroot              213K  96.4G    24K  none
zroot/var           48K  96.4G    24K  none
zroot/var/ccache    24K  96.4G    24K  /var/ccache

root@poudriere:~ # cat /usr/local/etc/poudriere.conf
ZPOOL=zroot
FREEBSD_HOST=ftp://ftp.freebsd.org
BASEFS=/usr/local/poudriere
ZROOTFS=/dev/null
POUDRIERE_DATA=/usr/local/poudriere/data
DISTFILES_CACHE=/usr/ports/distfiles
CCACHE_DIR=/var/ccache
CHECK_CHANGED_OPTIONS=verbose
CHECK_CHANGED_DEPS=yes
PKG_REPO_SIGNING_KEY=/usr/local/etc/ssl/keys/poudriere.key
URL_BASE=http://0.0.0.0/
USE_TMPFS=no
TMPFS_LIMIT=12
MAX_MEMORY=12
PARALLEL_JOBS=8
PREPARE_PARALLEL_JOBS=8
MAX_FILES=4096
KEEP_OLD_PACKAGES=yes
KEEP_OLD_PACKAGES_COUNT=3
CHECK_CHANGED_OPTIONS=verbose
CHECK_CHANGED_DEPS=yes
RESTRICT_NETWORKING=no

root@poudriere:~ # mkdir -p /usr/local/poudriere/data/logs/bulk

root@poudriere:~ # ln -s \
                     /usr/local/etc/ssl/certs/poudriere.cert \
                     /usr/local/poudriere/data/logs/bulk/poudriere.cert

root@poudriere:~ # service nginx enable

root@poudriere:~ # sed -i '' -E 's|text/plain[\t\ ]*txt|text/plain txt log|g' /usr/local/etc/nginx/mime.types

root@poudriere:~ # cat /usr/local/etc/nginx/nginx.conf

events {
  worker_connections 1024;
}

http {
  include      mime.types;
  default_type application/octet-stream;

  server {
    listen 80 default;
    server_name 0.0.0.0;
    root /usr/local/share/poudriere/html;

    location /data {
      alias /usr/local/poudriere/data/logs/bulk;
      autoindex on;
    }

    location /packages {
      root /usr/local/poudriere/data;
      autoindex on;
    }
  }
}

root@poudriere:~ # service nginx restart                              

root@poudriere:~ # ln -sf /var/ccache /root/.cache/ccache

root@poudriere:~ # cat /usr/local/etc/poudriere.d/make.conf
# general
ALLOW_UNSUPPORTED_SYSTEM=yes
DISABLE_LICENSES=yes

# ccache(1)
WITH_CCACHE_BUILD=yes

# ports options
FORCE_MAKE_JOBS=yes
MAKE_JOBS_UNSAFE=yes
MAKE_JOBS_NUMBER=8


root@poudriere:~ # cat /var/ccache/ccache.conf
max_size = 0
cache_dir = /var/ccache
base_dir = /var/ccache
hash_dir = false

root@poudriere:~ # poudriere jail -c -j 14-1-R-amd64 -v 14.1-RELEASE
(...)
[00:20:45] Jail 14-1-R-amd64 14.1-RELEASE amd64 is ready to be used

root@poudriere:~ # poudriere jail -l                               
JAILNAME     VERSION      ARCH  METHOD TIMESTAMP           PATH
14-1-R-amd64 14.1-RELEASE amd64 http   2024-06-21 09:44:27 /usr/local/poudriere/jails/14-1-R-amd64

Now … in the previous guides I just cloned the official FreeBSD Ports tree … not this time.

We will need Mariusz Zaborski (oshogbo) fork of the FreeBSD Ports tree.

If you would just want to clone it with git(1) command – then this is how to do this:

root@poudriere:~ # git clone - -depth=1 --branch oshogbo/freeipa https://github.com/oshogbo/freebsd-ports

… but we want to do this Poudriere way – so this is how to do it.

root@poudriere:~ # poudriere ports -c -p osho -B oshogbo/freeipa -U https://github.com/oshogbo/freebsd-ports
[00:00:00] Creating osho fs at /usr/local/poudriere/ports/osho... done
[00:00:00] Cloning the ports tree... done

root@poudriere:~ # poudriere ports -l
PORTSTREE METHOD    TIMESTAMP           PATH
osho      git+https 2024-06-21 11:42:53 /usr/local/poudriere/ports/osho

We now have needed FreeBSD Jail version and FreeBSD Ports tree that we can use to build needed packages.

Poudriere Server – Build FreeIPA/IDM Client Packages

Now we need to choose options for our FreeBSD Ports and then start the bulk process of fetching and building them.

root@poudriere:~ # poudriere options -c -n -p osho security/cyrus-sasl2-gssapi
//   SELECT: (*) GSSAPI_MIT

root@poudriere:~ # poudriere options -c -n -p osho net/openldap26-client
//   SELECT: [x] GSSAPI

root@poudriere:~ # poudriere options -c -n -p osho security/sudo
// DESELECT: [ ] PAM
//   SELECT: (*) GSSAPI_MIT
//   SELECT: (*) SSSD2

root@poudriere:~ # poudriere options -c -n -p osho net/samba416
//   SELECT: (*) GSSAPI_MIT

root@poudriere:~ # cat /usr/local/etc/poudriere.d/osho
security/krb5
security/sudo
security/sssd2
security/cyrus-sasl2
security/cyrus-sasl2-gssapi
security/pam_mkhomedir
net/openldap26-client
net/samba416
net/freeipa-client

root@poudriere:~ # poudriere bulk -j 14-1-R-amd64 -b latest -p osho -f /usr/local/etc/poudriere.d/osho

You can follow the status of the build process in the browser at https://10.1.1.123 page.

Poudriere Server – Update Repo/Packages

Everytime you will need to update the packages in that FreeIPA/IDM repo You will need to run these commands.

root@poudriere:~ # poudriere ports -u -p osho -B oshogbo/freeipa -U https://github.com/oshogbo/freebsd-ports

root@poudriere:~ # poudriere bulk -j 14-1-R-amd64 -b latest -p osho -f /usr/local/etc/poudriere.d/osho

You may as well update the FreeBSD Jail when needed.

root@poudriere:~ # poudriere jail -u -j 14-1-R-amd64

FreeBSD Samba Server – FreeIPA/IDM Connect

Same as with Poudriere server – you need to create FreeBSD Samba server as Bhyve VM.

Now – the needed configuration on FreeBSD 14.1-RELEASE system to first connect it to FreeIPA/IDM server.

root@samba:~ # mkdir -p               \
                 /usr/local/etc/ipa   \
                 /var/log/sssd        \
                 /var/run/sss/private \
                 /var/db/sss

root@samba:~ # echo 10.1.1.203 samba.lab.org samba >> /etc/hosts

root@samba:~ # echo 10.1.1.200 idm.lab.org   idm   >> /etc/hosts

root@samba:~ # hostname samba.lab.org

root@samba:~ # sysrc hostname=samba.lab.org

root@samba:~ # fetch -o /usr/local/etc/ipa/ca.crt http://idm.lab.org/ipa/config/ca.crt

Now we will need to add or FreeBSD Samba server to FreeIPA/IDM – instructions below.

[root@idm ~]# kinit admin
Password for admin@LAB.ORG: 
[root@idm ~]# ipa dnsrecord-add lab.org samba --a-rec=10.1.1.203 --a-create-reverse
  Record name: samba
  A record: 10.1.1.203

[root@idm ~]# ipa host-add samba.lab.org
-------------------------------
Added host "samba.lab.org"
-------------------------------
  Host name: samba.lab.org
  Principal name: host/samba.lab.org@LAB.ORG
  Principal alias: host/samba.lab.org@LAB.ORG
  Password: False
  Keytab: False
  Managed by: samba.lab.org

[root@idm ~]# ipa-getkeytab -s idm.lab.org -p host/samba.lab.org@LAB.ORG -k /root/samba.lab.org.keytab
Keytab successfully retrieved and stored in: /root/samba.lab.org.keytab

[root@idm ~]# cp /root/samba.lab.org.keytab /usr/share/ipa/html/

[root@idm ~]# chmod 644 /usr/share/ipa/html/samba.lab.org.keytab

Now lets get back to our FreeBSD Samba server.

root@samba:~ # fetch -o /usr/local/etc/ipa/krb5.keytab \
                 http://idm.lab.org/ipa/config/samba.lab.org.keytab

root@samba:~ # chmod 600 /usr/local/etc/ipa/krb5.keytab

root@samba:~ # mkdir -p /usr/local/etc/ssl/certs

root@samba:~ # mkdir -p /usr/local/etc/pkg/repos

root@samba:~ # sed 's|quarterly|latest|g' /etc/pkg/FreeBSD.conf > /usr/local/etc/pkg/repos/FreeBSD.conf

root@samba:~ # pkg install -y beadm

root@samba:~ # fetch -o /usr/local/etc/ssl/certs/poudriere.cert \
                 http://10.1.1.123/data/poudriere.cert

root@samba:~ # export IP=10.1.1.123

root@samba:~ # cat /usr/local/etc/pkg/repos/14-1-R-amd64-osho.conf
14-1-R-amd64-osho: {
  url: "http://${IP}/packages/14-1-R-amd64-osho/",
  mirror_type: "http",
  signature_type: "pubkey",
  pubkey: "/usr/local/etc/ssl/certs/poudriere.cert",
  enabled: yes,
  priority: 100
}

root@samba:~ # pkg update -f

root@samba:~ # pkg install -y         \
                 -r 14-1-R-amd64-osho \
                 krb5                 \
                 sudo                 \
                 sssd2                \
                 cyrus-sasl           \
                 cyrus-sasl-gssapi    \
                 openldap26-client    \
                 freeipa-client       \
                 pam_mkhomedir

root@samba:~ # cat << EOF >> /etc/ssh/ssh_config
GSSAPIAuthentication yes
EOF

root@samba:~ # cat << EOF >> /etc/ssh/sshd_config
GSSAPIAuthentication yes
UsePAM yes
EOF

root@samba:~ # cat << EOF > /usr/local/etc/sssd/sssd.conf
[sssd]
  config_file_version      = 2
  services                 = pam, ssh, sudo, ifp, pac, nss
  domains                  = lab.org
  timeout                  = 20

[domain/lab.org]
  ipa_server               = idm.lab.org
  ipa_domain               = lab.org
  pam_gssapi_services      = sudo, sudo-i
  enumerate                = True
  cache_credentials        = True
  override_homedir         = /home/%u
  default_shell            = /bin/sh
  ldap_group_nesting_level = 10
  default_ccache_template  = FILE:/tmp/krb5cc_:%U

  krb5_ccache_template     = FILE:/tmp/krb5cc_:%U
  krb5_server              = idm.lab.org:88
  krb5_realm               = LAB.ORG
  krb5_keytab              = /usr/local/etc/ipa/krb5.keytab
  krb5_auth_timeout        = 20

  id_provider              = ipa
  sudo_provider            = ipa
  access_provider          = ipa
  subdomains_provider      = ipa
  auth_provider            = ipa
  chpass_provider          = ipa
  selinux_provider         = none
EOF

root@samba:~ # chmod 600 /usr/local/etc/sssd/sssd.conf

root@samba:~ # cat << EOF > /etc/nsswitch.conf
group: files sss
group_compat: nis
hosts: files dns
networks: files
passwd: files sss
passwd_compat: nis
shells: files
services: compat
services_compat: nis
protocols: files
rpc: files
sudoers: sss files
netgroup: files
EOF

root@samba:~ # cat << EOF > /etc/rc.conf
hostname="samba.lab.org"
ifconfig_vtnet0="inet 10.1.1.203/24"
defaultrouter="10.1.1.1"
syslogd_flags="-ss"
clear_tmp_enable="YES"
sshd_enable="YES"
zfs_enable="YES"
sssd_enable="YES"
EOF

root@samba:~ # cat << EOF > /usr/local/etc/openldap/ldap.conf
BASE        dc=org,dc=lab
URI         ldap://idm.lab.org/
SASL_MECH   GSSAPI
SASL_REALM  LAB.ORG
ssl         start_tls
TLS_CACERT  /usr/local/etc/ipa/ca.crt
EOF

root@samba:~ # cat << EOF > /etc/krb5.conf
[libdefaults]
  default_realm        = LAB.ORG
  default_keytab_name  = FILE:/usr/local/etc/ipa/krb5.keytab
  default_tkt_enctypes = aes256-cts des-cbc-crc aes128-cts arcfour-hmac
  default_tgs_enctypes = aes256-cts des-cbc-crc aes128-cts arcfour-hmac
  dns_lookup_realm     = false
  dns_lookup_kdc       = false
  rdns                 = false
  ticket_lifetime      = 24h
  forwardable          = yes

[realms]
  LAB.ORG = {
    kdc            = idm.lab.org:88
    master_kdc     = idm.lab.org:88
    admin_server   = idm.lab.org:749
    default_domain = lab.org
    pkinit_anchors = FILE:/usr/local/etc/ipa/ca.crt
  }

[domain_realm]
  .lab.org = LAB.ORG
   lab.org = LAB.ORG

[logging]
  kdc          = FILE:/var/log/krb5/krb5kdc.log
  admin_server = FILE:/var/log/krb5/kadmin.log
  kadmin_local = FILE:/var/log/krb5/kadmin_local.log
  default      = FILE:/var/log/krb5/krb5lib.log
EOF

root@samba:~ # cat << EOF > /etc/pam.d/system
# AUTH
  auth      sufficient  pam_krb5.so                      no_warn try_first_pass
  auth      sufficient  /usr/local/lib/pam_sss.so        no_warn use_first_pass
  auth      required    pam_unix.so                      no_warn try_first_pass nullok

# ACCOUNT
  account   required    pam_login_access.so
  account   required    /usr/local/lib/pam_sss.so        ignore_unknown_user ignore_authinfo_unavail
  account   required    pam_unix.so

# SESSION
  session   required    pam_lastlog.so                   no_fail
  session   required    /usr/local/lib/pam_mkhomedir.so  mode=0700

# PASSWORD
  password  sufficient  /usr/local/lib/pam_sss.so        no_warn use_authtok
  password  required    pam_unix.so                      no_warn try_first_pass
EOF

root@samba:~ # cat << EOF > /etc/pam.d/sshd
# AUTH
  auth      sufficient  pam_krb5.so                      no_warn try_first_pass
  auth      sufficient  /usr/local/lib/pam_sss.so        no_warn use_first_pass
  auth      required    pam_unix.so                      no_warn try_first_pass

# ACCOUNT
  account   required    pam_nologin.so
  account   required    pam_login_access.so
  account   required    pam_unix.so
  account   required    /usr/local/lib/pam_sss.so        ignore_unknown_user ignore_authinfo_unavail

# SESSION
  session   required    pam_permit.so
  session   required    /usr/local/lib/pam_mkhomedir.so  mode=0700
  session   optional    /usr/local/lib/pam_sss.so

# PASSWORD
  password  sufficient  /usr/local/lib/pam_sss.so        no_warn use_authtok
  password  required    pam_unix.so                      no_warn try_first_pass
EOF

Our samba.lab.org in the FreeIPA/IDM below.

[root@idm ~]# ipa host-find samba
--------------
1 host matched
--------------
  Host name: samba.lab.org
  Principal name: host/samba.lab.org@LAB.ORG
  Principal alias: host/samba.lab.org@LAB.ORG
----------------------------
Number of entries returned 1
----------------------------

Now reboot(8) your samba.lab.org and You should be able to login to it with FreeIPA/IDM account.

host # ssh vermaden@10.1.1.203
(vermaden@10.1.1.203) Password:
Last login: Wed Mar  6 07:04:42 2024

vermaden@samba:~ $ grep vermaden /etc/group 
wheel:*:0:root,vermaden
operator:*:5:root,vermaden

vermaden@samba:~ $ klist
klist: No ticket file: /tmp/krb5cc_1000

vermaden@samba:~ $ kinit vermaden
vermaden@LAB.ORG's Password: 

vermaden@samba:~ $ klist
Credentials cache: FILE:/tmp/krb5cc_1000
        Principal: vermaden@LAB.ORG

  Issued                Expires               Principal
Jun 20 21:36:41 2024  Jun 21 20:38:26 2024  krbtgt/LAB.ORG@LAB.ORG

vermaden@samba:~ $ sudo -i
Password for vermaden@LAB.ORG: 

root@samba:~ #

Now we have FreeBSD 14.1-RELEASE connected to FreeIPA/IDM server.

FreeBSD Samba Server – Configuration

Now we will make configurations on both FreeBSD Samba server and FreeIPA/IDM.

First tasks on FreeIPA/IDM.

[root@idm ~]# yum install -y ipa-server-trust-ad

[root@idm ~]# ipa-adtrust-install  \
                --unattended       \
                --enable-compat    \
                --add-sids         \
                --admin-name=admin \
                --admin-password=password

Generally we want to add SIDs for existing users/groups and enable support for trusted domains for old clients.

Now more FreeIPA/IDM groups/permissions/roles settings.

[root@idm ~]# ipa group-add cifs-share
--------------------
Added group "cifs-share"
--------------------
  Group name: cifs-share
  GID: 1930000005

[root@idm ~]# ipa group-add-member cifs-share --users=vermaden
  Group name: cifs-share
  GID: 1930000005
  Member users: vermaden
-------------------------
Number of members added 1

[root@idm ~]# ipa permission-add "CIFS-Server-Read-Passwords" \
                 --attrs={ipaNTHash,ipaNTSecurityIdentifier} \
                 --type=user \
                 --right={read,search,compare} \
                 --bindtype=permission
------------------------------------------------------
Added permission "CIFS-Server-Read-Passwords"
------------------------------------------------------
  Permission name: CIFS-Server-Read-Passwords
  Granted rights: read, search, compare
  Effective attributes: ipaNTHash, ipaNTSecurityIdentifier
  Bind rule type: permission
  Subtree: cn=users,cn=accounts,dc=lab,dc=org
  Type: user
  Permission flags: SYSTEM, V2

[root@idm ~]# ipa privilege-add "CIFS-Server-Privilege"
---------------------------------------
Added privilege "CIFS-Server-Privilege"
---------------------------------------
  Privilege name: CIFS-Server-Privilege

[root@idm ~]# ipa privilege-add-permission "CIFS-Server-Privilege" --permission="CIFS-Server-Read-Passwords"
  Privilege name: CIFS-Server-Privilege
  Permissions: CIFS-Server-Read-Passwords
-----------------------------
Number of permissions added 1
-----------------------------

[root@idm ~]# ipa role-add "CIFS-Server"
------------------------
Added role "CIFS-Server"
------------------------
  Role name: CIFS-Server

[root@idm ~]# ipa role-add-privilege "CIFS-Server" --privilege="CIFS-Server-Privilege"
  Role name: CIFS-Server
  Privileges: CIFS-Server-Privilege
----------------------------
Number of privileges added 1
----------------------------

[root@idm ~]# ipa service-add cifs/samba.lab.org
-------------------------------------------------------------
Added service "cifs/idm-freebsd-samba.lab.org@LAB.ORG"
-------------------------------------------------------------
  Principal name: cifs/samba.lab.org@LAB.ORG
  Principal alias: cifs/samba.lab.org@LAB.ORG
  Managed by: samba.lab.org

[root@idm ~]# ipa role-add-member "CIFS-Server" --services=cifs/samba.lab.org
  Role name: CIFS-Server
  Privileges: CIFS-Server-Privilege
  Member services: cifs/samba.lab.org@LAB.ORG
-------------------------
Number of members added 1
-------------------------
              
[root@idm ~]# ipa-getkeytab -s idm.lab.org -p cifs/samba.lab.org -k cifs-samba.keytab
Keytab successfully retrieved and stored in: samba.keytab

[root@idm ~]# mv cifs-samba.keytab /usr/share/ipa/html/

[root@idm ~]# chmod 644 /usr/share/ipa/html/cifs-samba.keytab


Now we need to set some things on the FreeBSD Samba server.

root@samba:~ # fetch -o /usr/local/etc/smb4.keytab \
                 http://idm.lab.org/ipa/config/cifs-samba.keytab

root@samba:~ # cat << SAMBA > /usr/local/etc/smb4.conf
[global]
  workgroup                = LAB
  realm                    = LAB.ORG
  netbios name             = SAMBA
  security                 = user
  log file                 = /var/log/samba.log
  log level                = 1
  passdb backend           = ipasam:ldap://idm.lab.org
  ldapsam:trusted          = yes
  ldap suffix              = dc=lab,dc=org
  ldap user suffix         = cn=users,cn=accounts
  ldap machine suffix      = cn=computers,cn=accounts
  ldap group suffix        = cn=groups,cn=accounts
  ldap ssl                 = no
  idmap config * : backend = tdb  
  create krb5 conf         = no 
  dedicated keytab file    = FILE:/usr/local/etc/smb4.keytab
  kerberos method          = dedicated keytab
  state directory          = /var/lib/samba4
  cache directory          = /var/lib/samba4
  include                  = registry

[data]
  path       = /data
  writeable  = yes
  browsable  = yes
  public     = no
  write list = vermaden
SAMBA

root@samba:~ # mkdir /var/lib/samba4

root@samba:~ # mkdir /data

root@samba:~ # echo 1 > /data/1.txt

root@samba:~ # chown -R root:cifs-share /data

root@samba:~ # chmod 775 /data

root@samba:~ # pkg info -l net/freeipa-client | grep ipasam
        /usr/local/lib/ipasam.so

root@samba:~ # mkdir -p /usr/local/lib/samba4/modules/pdb

root@samba:~ # cp /usr/local/lib/ipasam.so /usr/local/lib/samba4/modules/pdb/ipasam.so

root@samba:~ # service samba_server enable

root@samba:~ # service samba_server start

There is now one small and tricky part w/o which it will still do not work. The SIDs needs to be the same. It is done by net(8) command installed by net/samba416 package.

root@samba:~ # pkg which -o $( which net )
/usr/local/bin/net was installed by package net/samba416

We will use two net(8) subcommands here – getdomainsid and sedlocalsid.

Lets start with the first one.

root@samba:~ # net getdomainsid
SID for local machine CIFS is: S-1-5-21-1629303435-4165009306-2230089581
SID for domain LAB is: S-1-5-21-547807538-239846452-1502288973

As you can see the local SID and domain SID are different – and this way it will not work.

The domain SID comes from FreeIPA/IDM and we need to copy it and set it also as local SID on FreeBSD Samba server.

root@samba:~ # net setlocalsid S-1-5-21-547807538-239846452-1502288973

root@samba:~ # net getdomainsid
SID for local machine CIFS is: S-1-5-21-547807538-239846452-1502288973
SID for domain LAB is: S-1-5-21-547807538-239846452-1502288973

Now we have Samba server on FreeBSD configured to serve shares by using credentials from FreeIPA/IDM.

The most important part of this setup is using ipasam.so Samba backend for FreeIPA/IDM users and passwords.

This ipasam.so file is installed by the net/freeipa-client package.

Samba Linux Client

The Samba client on Alma Linux 9.4 is installed in the same way as IDM so no need to repeat myself – only IP and hostname are different – and a set of installed packages of course.

First – lets make sure the local SID on the Linux Samba client is the same as domain SID from FreeIPA/IDM.

[root@client ~]# yum install -y samba-common-tools

[root@client ~]# net setlocalsid S-1-5-21-547807538-239846452-1502288973

Now we will try to use smbclient(1) to check if we can list contents of the Samba share from FreeBSD server.

[root@client ~]# yum install -y samba-client

[root@client ~]# smbclient //10.1.1.203/data -U LAB.ORG/vermaden
Password for [LAB.ORG\vermaden]:
Try "help" to get a list of possible commands.
smb: \> ls
  .                                   D        0  Wed Jun 19 22:37:05 2024
  ..                                  D        0  Sat Jun 22 19:05:30 2024
  1.txt                               N        2  Wed Jun 19 16:31:40 2024

                13345400 blocks of size 1024. 11608920 blocks available

smb: \> exit

So it works! … at least in read only mode. Now lets try to mount it to write something there.

[root@client ~]# yum install -y cifs-utils

[root@client ~]# mount.cifs -v //10.1.1.203/data /mnt -o username=vermaden,domain=LAB.ORG,password=password
mount.cifs kernel mount options: ip=10.1.1.203,unc=\\10.1.1.203\data,user=vermaden,domain=LAB.ORG,pass=********

[root@client ~]# ls -l /mnt
total 2
-rwxr-xr-x. 1 root root 2 Jun 19 16:31 1.txt

[root@client ~]# cat /mnt/1.txt 
1

[root@client ~]# echo 2 > /mnt/2.txt

[root@client ~]# cat /mnt/2.txt
2

The read/write mode also works.

Lets verify that the 2.txt is also present on the FreeBSD Samba server.

root@samba:~ # find /data -ls
 48606        1 drwxrwxr-x    2 root        cifs-share   6 Jun 19 22:37 /data
 68704        1 -rwxr--r--    1 vermaden    wheel        2 Jun 23 17:25 /data/2.txt
 68788        1 -rw-r--r--    1 root        cifs-share   2 Jun 19 16:31 /data/1.txt

Lets create another 3.txt file from the FreeBSD Samba server for another verification.

root@samba:~ # echo 3 > /data/3.txt

… and back to Linux client.

[root@client ~]# ls -l /mnt
total 2
-rwxr-xr-x. 1 root root 2 Jun 19 16:31 1.txt
-rwxr-xr-x. 1 root root 2 Jun 23 17:25 2.txt
-rwxr-xr-x. 1 root root 2 Jun 19 16:35 3.txt

So we have Linux client covered. It all works as desired.

One important thing here – the Linux client DID NOT joined the FreeIPA/IDM domain – its just a regular Linux system.

The FreeBSD Samba server displayed as SAMBA below.

[root@client ~]# net getdomainsid
SID for local machine CLIENT is: S-1-5-21-547807538-239846452-1502288973
SID for domain SAMBA is: S-1-5-21-547807538-239846452-1502288973

Now we will move to Windows platform.

Samba Windows Client

As Windows is GUI focused system we will present that information in Board of Directors friendly way – in graphical form πŸ™‚

Below you will see the process of typical Map Network Drive job.

I was able to read existing content and also to create new 4.win.txt file with some garbage content – so read/write access from Windows also works.

Summary

Let me know in comments how it went.

EOF

FreeBSD Bhyve Companion Tools

Some time ago I wrote about how FreeBSD Bhyve can be used in the FreeBSD Bhyve Virtualization article. It was quite short after I moved from VirtualBox – and to be honest – I am really glad that I moved to Bhyve.

When I was working with VirtualBox – I was not fully happy – it just felt that FreeBSD is not its native environment. While most things worked the networking stuff sometimes required all VMs reboot and sometimes even entire FreeBSD host hard reset was needed. It was not often – but still.

Now – after almost a year with Bhyve hypervisor – and using vm-bhyve toolkit – I would like to share some tools that make everyday life easier and more productive.

VNC Connection

In the VirtualBox world You either started a VM in SDL mode – with full graphical window and interface … or you started headless w/o any graphical output at all. With Bhyve its quite different. For the UEFI boot machines you start a VM process in the text mode w/o any additional graphics – but You are allowed/able to connect to that machine graphical framebuffer using – for example – vncviewer(1) command for VNC connection.

In the past I just first listed the VMs and then copied and pasted the VNC address to a vncviewer(1) tool.

… and it made me less happy with time – especially as I got more and more VMs – so for that purpose I wrote a simple vnc.sh script that takes Bhyve VM name as argument and then using vm(8) command listing finds the needed VNC address.

I even shortened it with vnc alias.

% alias vnc                                       
vnc=vnc.sh

% vnc ghostbsd

Pause/Resume Bhyve VM

In the past I implemented the desktop-pause.sh script to pause/resume UNIX applications – usually focused on the GUI ones – Pause Any Application – described here in details.

As Bhyve is a type 2 hypervisor – just a process in a FreeBSD OS – we can also use kill(1) to pause/resume the state of each Bhyve VM.

This is where bp.sh tool comes handy (BP stands for Bhyve Pause).

I even shortened it with bp alias.

% alias bp                                       
bp=bp.sh

% bp -p poudriere

You specify a VM name and you have an option to pause/resume the VM process or to toggle the pause/resume state.

The main use case I find for that is – for example – some VM takes too much CPU/RAM resources while You are doing something else on the host desktop/laptop/server system and you want to pause it to save resources for other things – let it be interactive work or even other Poudriere based machine that needs the same resources.

It works like a charm but I would like to point one warning here – either configure some time sync inside all of Your VMs – or just remember to update time in a VM after you paused it for some time. There are no other drawbacks.

Summary

While simple and not epic – these two additions made my day to day Bhyve life a lot easier and allowed me to focus on the work and not on the internals.

I hope they will do the same to Your workloads.

UPDATE 1 – More Listing Options

After some playing with these tools I often started another terminal to check something … not anymore after these updates.

Colored vm list Output

If you have small amount of Bhyve VMs then this feature will not help you much – but the more VMs you have – the more it is useful.

Now only 3 colors are used. Bold white for the headers. Green for the Running machines. Red for the Locked machines.

The thing I did is I wrapped a ‘generic’ vm list output inside vmc.sh script and added colors to it – check for yourself how it looks.

Now compare that to the default vm(8) listing.

Night and day I would say πŸ™‚

List Paused Machines

After I paused any machine I needed to use my other desktop-pause.sh script with -l option to list paused Bhyve VMs.

Now all this is integrated into the bp.sh tool.

Much better.

List VNC Enabled VMs

This last update is the smallest one – still sometimes usable.

While the default vm list output provides that information one may want to just have VNC information – and as we already have a dedicated vnc.sh tool for that – why not implement it here.

Summary

One can confuse vmc.sh with vnc.sh so I will try to make that logic.

The vmc.sh states for VM Listing Colored … and I really did not wanted to use vlc.sh name as someone could take it as some VLC media player wrapper πŸ™‚

The vnc.sh is for VNC connection.

Hope that helps.

UPDATE 2 – Small Useful Addons

While using mine vmc.sh | vnc.sh | bp.sh scripts I found some things that could be improved in them – and this update details these improvements to them.

This is the GitHub commit changelog of these changes.

Different Color for Bootloader State

The vmc.sh – the colored VM list primary task – is to color everything different then Stopped state – so I also added a different color (blue) to color VMs that are in the Bootloader state.

Not sure if useful for everyone – but I found it useful at least several times.

Resume All VMs

While its possible to pause/resume any Bhyve VM – and I sometimes ‘pause’ several VMs that together combine some environment – its PITA to resume them one by one – so I added a -R flag to resume ALL paused VMs – with single command.

Remove All Locks

Useful especially after unexpected reboot or crash. You had several or more VMs perfectly running – but after reboot all of them are locked because the run.lock file was not removed in the process.

For this – I have added -l (for lock) flag – to find and delete run.lock files in the Bhyve VM directory.

For now – the so called VM_DIR is hardlocked in the scripts – set it before using – in the future I plan to make this ‘dynamic’ and being gathered from /etc/rc.conf file or using ‘default’ value.

Check Duplicated MAC Addresses

If you move/migrate VMs between hosts and environments – or clone/copy them – its possible that you will end up with one or more VMs with the same MAC address.

… and while its perfectly possible to just type small UNIX pipe train to make that check – its a lot faster to have that option in vmc.sh command.

Check Duplicated UUID for VMs

While less problematic then duplicated MAC address – I also added a check for duplicated UUID of the Bhyve VMs.

UPDATE 3 – Show VMs Sizes

Another small pack of updates to the vmc.sh script to show size of each virtual machine.

Also to destroy virtual machine status – useful if you needed to kill bhyve(8) process and the machine was still treated as running or present in the FreeBSD system.

Other updates:

  • Add -s option to show VMs size.
  • Read VM_DIR from /etc/rc.conf file.
  • Upgrade removing lock files.
  • Add -d option to reset/destroy VM state.
  • Add check for doas(1) and sudo(8) existence.

Let me know if any other features would be useful.

Summary

Feel free to share your thoughts about these recent changes.

Regards.

EOF

FreeBSD Bhyve Virtualization

The Bhyve FreeBSD hypervisor (called/spelled ‘beehive’ usually) was created almost 10 years ago. Right now it offers speed and features that other similar solutions provide – such as KVM/VMware/XEN. You can check all the details in the FreeBSD Handbook for details. One of the last things Bhyve lacks is so called live migration between physical hosts but save state and resume from saved state are in the works currently so not long before that live migration. Up until recently I used mostly VirtualBox for my small virtualization needs. Recently I started to evaluate Bhyve and this time I am very pleased – the FreeBSD VirtualBox integration is not perfect anyway – for example – the USB passthru does not work since several years – and even when it worked – it was limited to USB 1.x speeds only. Also because of FreeBSD policy of pkg(8) packages building process – the VirtualBox packages remain broken for 3 months after each *.1 or upper release (*.2/*.3/…). The other impulse that forced me to switch from VirtualBox to Bhyve was the VirtualBox (in)stability. I often needed to restart crashed VirtualBox VMs because they failed for some unspecified reason.

bhyve-logo

One of the Bhyve features that I especially liked was that by default Bhyve only uses memory that guest system wanted to allocate. For example a FreeBSD virtual machine with 2 GB RAM set will use after boot about 70 MB RAM πŸ™‚

Another great feature I really liked about Bhyve was that I could suspend the host machine with all the VMs started – both on my ThinkPad W520 laptop and AMD Based FreeBSD Desktop and then it all successfully resumed. With VirtualBox you would have to poweroff all VMs because if you suspend with running VirtualBox VMs – it will just crash – its not possible to do suspend/resume cycle with VirtualBox.

The Table of Contents for this article is as follows:

  • FreeBSD Bhyve Virtualization
  • Bhyve Managers
  • Bhyve versus KVM and VMware
  • Bhyve libvirt/virt-manager GUI
  • vm-bhyve
    • Install/Setup
    • Networking
      • Server/Desktop LAN Bridge
      • Laptop WiFi NAT
      • Networking Restart
    • Datastores
    • Templates
    • NVMe
    • ISO Images
    • Guest OS Install
      • FreeBSD
        • X11 on FreeBSD
      • Linux
      • Windows 7
      • Windows 10
        • Force Windows 10 Offline Account
        • Windows 10 Bloat Removers
      • Windows 11
    • Dealing with Locked VMs
    • Disk Resize
  • Summary

Bhyve Managers

While VirtualBox has quite usable QT based GUI – the Bhyve does not have anything like that. I once seen some GUI QT prototype for Bhyve but it was very basic – so forget about that currently. There are however several web interfaces such as TrueNAS CORE or CBSD/CloneOS. There are also several terminal managers such as vm-bhyve. The older one iohyve has not been maintained for at least 6 long years. There is also libvirt Bhyve driver but more on that later.

Bhyve versus KVM and VMware

Klara Systems compared Bhyve to KVM and Benjamin Bryan compared it against VMware hypervisor. While Bhyve remains competitive against both of them there are two important aspects from Klara Systems that stand out and are worth repeating here.

First – using nvme driver is a lot faster then using more traditional virtio-blk or ahci-hd backends.

bhyve-nvme

Second – and this one seems strange – using a raw file is faster then using ZFS zvol device.

bhyve-raw-zvol

To summarize these thoughts – just just file on a disk like disk0.img and use nvme as storage backend everytime the guest operating system supports it.

Bhyve libvirt/virt-manager GUI

Theoretically the libvirt virtualization API supports Bhyve as one of its backends and the details about Bhyve driver – https://libvirt.org/drvbhyve.html – are available here. I have tried it with virt-manager and after some basic configuration I was able to start FreeBSD 13.2 installation … but it got frozen at the kernel messages and nothing more happened.

virt-manager-boot-menu

… and the moment it hanged below. I have tried multiple times with the same effect.

virt-manager-hang

I really liked the virtual machine settings window of virt-manager.

virt-manager-machine-settings

vm-bhyve

While You can use Bhyve directly with bhyve(8) and bhyvectl(8) commands – which I was doing in the past – after trying the vm-bhyve both on the desktop and server space – I really liked it and this is what I currently use. I just moved from vm-bhyve package to the newer vm-bhyve-devel one.

The vm(8) command is simple and provides all needed use cases.

host # vm help
vm-bhyve: Bhyve virtual machine management v1.6-devel (rev. 106001)
Usage: vm ...
    version
    init
    set [setting=value] [...]
    get [all|setting] [...]
    switch list
    switch info [name] [...]
    switch create [-t type] [-i interface] [-n vlan-id] [-m mtu] [-a address/prefix-len] [-b bridge] [-p] 
    switch vlan  
    switch nat  
    switch private  
    switch add  
    switch remove  
    switch destroy 
    datastore list
    datastore add  
    datastore remove 
    datastore add  
    list
    info [name] [...]
    create [-d datastore] [-t template] [-s size] [-m memory] [-c vCPUs] 
    install [-fi]  
    start [-fi]  [...]
    stop  [...]
    restart 
    console  [com1|com2]
    configure 
    rename  
    add [-d device] [-t type] [-s size|switch] 
    startall
    stopall
    reset  [-f] 
    poweroff [-f] 
    destroy [-f] 
    passthru
    clone  
    snapshot [-f] 
    rollback [-r] 
    iso [url]
    img [url]
    image list
    image create [-d description] [-u] 
    image destroy 
    image provision [-d datastore]  

Install/Setup

We need only several packages to add.

host # pkg install -y \
         vm-bhyve-devel \
         bhyve-firmware \
         edk2-bhyve \
         dnsmasq \
         grub2-bhyve \
         tigervnc-viewer \
         rdesktop

The setup is pretty easy also.

First we need to add several vm_* settings into the main FreeBSD /etc/rc.conf file.

  vm_enable=YES
  vm_dir="zfs:zroot/vm"
  vm_list=""
  vm_delay=3

Keep in mind that you will later use the vm_list="" for the list of VMs that you would like to be started at boot. Like vm_list="freebsd13 freebsd14uefi" for example. Then the vm list command would place [1] in at the freebsd13 name (as its first) and [2] in the freebsd14uefi name as this one is second on the list. See below.

host # vm list
NAME           DATASTORE  LOADER     CPU  MEMORY  VNC           AUTO     STATE
almalinux8     default    uefi       2    2G      0.0.0.0:5908  No       Running (11819)
freebsd13      default    bhyveload  1    256M    -             Yes [1]  Running (2342)
freebsd14      default    bhyveload  1    256M    -             No       Stopped
freebsd14uefi  default    uefi       2    8G      -             Yes [2]  Running (35394)
windows10      default    uefi       2    2G      -             No       Stopped
windows7       default    uefi       2    2G      -             No       Stopped

We need to create a dedicated ZFS dataset for our VMs. You can also use directory on UFS – check vm-bhyve documentation.

host # zfs create -o mountpoint=/vm zroot/vm

We will also copy the available templates to our new /vm dir.

host # cp -a /usr/local/share/examples/vm-bhyve /vm/.templates

Remember to check /vm/.templates/config.sample as it has the documentation for all available options.

host # head -12 /vm/.templates/config.sample
# This is a sample configuration file containing all supported options
# Please do not try and use this file itself for a guest
# For any option that contains a number in the name, such as "network0_type",
# you can add additional devices of that type by creating a new set of
# variables using the next number in sequence, e.g "network1_type"
#
# Please make sure all option names are specified in lowercase and
# at the beginning of the line. If there is any whitespace before
# the option name, the line will be ignored.
# The '#' character signifies the start of a comment, even within
# double-quotes, and so cannot be used inside any values.

We can now start initialize the vm-bhyve.

host # service vm start

Networking

There as many network setups as many FreeBSD has network capabilities – a lot! I this guide I will cover two most typical network setups for Bhyve. One would be the most server (or desktop) oriented – as it requires a LAN card to be used. The other one I would call a laptop one – that one would provide network connectivity using wlan0 WiFi interface.

No matter which one we will choose – we need to enable port forwarding on our FreeBSD host. Do that with these two commands.

host # sysrc gateway_enable=YES

host # sysctl net.inet.ip.forwarding=1

host # echo net.link.tap.up_on_open=1 >> /etc/sysctl.conf

host # sysctl net.link.tap.up_on_open=1

I assume that our FreeBSD host system would use 10.0.0.10/24 IP address and that 10.0.0.1 would be its default gateway.

Your host system main /etc/rc.conf file can looks as follows then.

host # cat /etc/rc.conf
# NETWORK
  hostname=host
  ifconfig_re0="inet 10.0.0.10/24 up"
  defaultrouter="10.0.0.1"
  gateway_enable=YES

# DAEMONS
  sshd_enable=YES
  zfs_enable=YES

# BHYVE
  vm_enable="YES"
  vm_dir="zfs:zroot/vm"
  vm_list=""
  vm_delay="3"

Server/Desktop LAN Bridge

We will use 10.0.0.0/24 network – the same that our host system uses. We will need one bridge/switch named vm-public with 10.0.0.100/24 address on it. Without that address later the dnsmasq will complain unknown interface vm-public about it. Information about the switches is kept in the /vm/.config/system.conf file. We will also need to add out LAN interface to the public switch. It will beΒ re0 interface in my case.

host # vm switch create -a 10.0.0.100/24 public

host # vm switch add public re0

host # vm switch list
NAME    TYPE      IFACE      ADDRESS        PRIVATE  MTU  VLAN  PORTS
public  standard  vm-public  10.0.0.100/24  no       -    -     re0

host # cat /vm/.config/system.conf
switch_list="public"
type_public="standard"
addr_public="10.0.0.100/24"
ports_public="re0"

To be honest the networking part setup is complete.

When you will be setting up your Bhyve VMs you will either use static 10.0.0.0/24 IP address space or just use DHCP and the one that is already on your network will take care of the rest (assuming you have one).

If you do not have one you may use dnsmasq service to do that easily.

host # cat /usr/local/etc/dnsmasq.conf
domain-needed
no-resolv
server=1.1.1.1
except-interface=lo0
bind-interfaces
local-service
dhcp-authoritative
interface=vm-public
dhcp-range=10.0.0.69,10.0.0.96

host # service dnsmasq enable

host # service dnsmasq start

That should do.

Now – to access the VMs in this bridged networking mode you can just ssh(1) to their IP address directly.

Laptop WiFi NAT

This is one of the cases where VirtualBox has one more feature over Bhyve. With VirtualBox its possible to use bridge mode over WiFi interface. Its not possible with Bhyve currently. I have submitted a proposal to FreeBSD Foundation to implement such configuration – especially as open source VirtualBox code already exists. Time will tell if it will be implemented or if there would be more important tasks to take care of.

We will use 10.1.1.0/24 network for our VM needs. We will also need only one vm-bhyve switch that we will use – and it will be the vm-public one with 10.1.1.1/24 address – we will be using it as a gateway for our VMs in NAT. Information about the switches is kept in the /vm/.config/system.conf file.

host # vm switch create -a 10.1.1.1/24 public

host # vm switch list
NAME    TYPE      IFACE      ADDRESS      PRIVATE  MTU  VLAN  PORTS
public  standard  vm-public  10.1.1.1/24  no       -    -     -

host # cat /vm/.config/system.conf 
switch_list="public"
type_public="standard"
addr_public="10.1.1.1/24"

Now the NAT part – we will do that with very simple pf(4) config.

host # cat /etc/pf.conf
# SKIP LOOPBACK
  set skip on lo0

# bhyve(8) VMs NAT WiFi
  nat on wlan0 from {10.1.1.1/24} to any -> (wlan0)

# bhyve(8) VMs NAT LAN
  nat on em0 from {10.1.1.1/24} to any -> (em0)

# bhyve(8) VMs NAT USB Tethering
  nat on ue0 from {10.1.1.1/24} to any -> (ue0)

# bhyve(8) VMs NAT USB Tethering
  nat on ue1 from {10.1.1.1/24} to any -> (ue1)

# PASS IN/OUT ALL
  pass in all
  pass out all

host # service pf enable

host # service pf start

You can check the stats of that pf(4) rules like that.

host # pfctl -Psn -vv
No ALTQ support in kernel
ALTQ related functions disabled
@0 nat on wlan0 inet from 10.1.1.0/24 to any -> (wlan0) round-robin
  [ Evaluations: 18774     Packets: 362277    Bytes: 352847937   States: 0     ]
  [ Inserted: uid 0 pid 69837 State Creations: 38    ]
@1 nat on ue0 inet from 10.1.1.0/24 to any -> (ue0) round-robin
  [ Evaluations: 5534      Packets: 1060      Bytes: 176979      States: 0     ]
  [ Inserted: uid 0 pid 8981 State Creations: 266    ]
@2 nat on em0 inet from 10.1.1.0/24 to any -> (em0) round-robin
  [ Evaluations: 13976     Packets: 0         Bytes: 0           States: 0     ]
  [ Inserted: uid 0 pid 48452 State Creations: 0     ]

Feel free to add all your pf(4) rules into the /etc/pf.conf file.

Now the DHCP server. For simplicity of the setup we will use dnsmasq daemon – but nothing prevents you from setting up a Highly Available DHCP Server instead using isc-dhcp44-server package.

host # cat /usr/local/etc/dnsmasq.conf
domain-needed
no-resolv
server=1.1.1.1
except-interface=lo0
bind-interfaces
local-service
dhcp-authoritative
interface=vm-public
dhcp-range=10.1.1.11,10.1.1.99

host # service dnsmasq enable

host # service dnsmasq start

Now you should be ready to setup Bhyve VMs on your laptop.

When I was using VirtualBox – it allowed me to use its Port Forwarding feature where I could add as many mappings in the NAT network type as shown below.

vbox-port-forwarding

… but with Bhyve its even better as no port forwarding in the NAT mode is needed at all! πŸ™‚

You can access the Bhyve VMs in NAT networking mode the same as you can in the bridged mode – with just ssh(1) to their IP address from directly from the 10.1.1.0/24 range.

So to put it bluntly – if a Bhyve VM in NAT mode got 10.1.1.33/24 IP address – then just ssh(1) to that IP address directly from the host system.

Networking Restart

Sometimes – when for example you laptop will boot without network connectivity – the tap(4) interfaces sometimes do not went UP.

bhyve-networking-restart

There is simple fix tor that problem – bhyve-network-restart.sh script.

Its shown below.

# ADD IP ADDRESS TO EACH vm-bhyve SWITCH
vm switch list \
  | sed 1d \
  | while read NAME TYPE IFACE ADDRESS PRIVATE MTU VLAN PORTS a0 a1 a2 a3 a4 a5 a6 a7 a8 a9
    do
      if [ "${ADDRESS}" != "-" ]
      then
             vm switch address ${NAME} ${ADDRESS}
        echo vm switch address ${NAME} ${ADDRESS}
      fi
    done        

# SET TO 'up' ALL vm-bhyve SWITCH MEMBERS
vm switch list \
  | sed 1d \
  | awk '{print $1}' \
  | while read SWITCH
    do
      ifconfig vm-${SWITCH} \
        | awk '/member:/ {print $2}' \
        | while read INTERFACE
          do
                 ifconfig ${INTERFACE} up
            echo ifconfig ${INTERFACE} up
          done
    done


Execute it everytime you lost connectivity with your VMs and you are done.

Datastores

While vm-bhyve supports multiple datastores – you will only need one – the default one.

host # vm datastore list
NAME            TYPE        PATH                      ZFS DATASET
default         zfs         /vm                       zroot/vm

Snapshots and Clones

The vm-bhyve also supports snapshots and clones of the VMs disks. Generally they are just ZFS snapshots and clones.

Templates

While vm-bhyve comes with several handy templates – they are incomplete – and small several changes makes the game more playable.

NVMe

First – we will implement the things that we know work faster – the nvme type for disk images instead of virt-blk or ahci-hd ones. Of course not all operating systems have support for such devices – for them we will use the latter options.

A fast way to change it to nvme is below.

host # sed -i '' s.virtio-blk.nvme.g /vm/.templates/freebsd.conf

ISO Images

Each VM needs an ISO image from which it will be installed. Of course you can also just create new VM and copy the disk contents from other server or use one of the FreeBSD images.

There are two ways to feed vm-bhyve with ISO images.

One is to fetch them from some URL.

host # vm iso http://ftp.freebsd.org/pub/FreeBSD/releases/ISO-IMAGES/13.2/FreeBSD-13.2-RELEASE-amd64-disc1.iso

host # vm iso
DATASTORE           FILENAME
default             FreeBSD-13.2-RELEASE-amd64-disc1.iso

The other way is to just simple copy ISO file to the /vm/.iso directory.

host # cp /home/vermaden/download/ubuntu-mate-23.04-desktop-amd64.iso /vm/.iso/

host # vm iso
DATASTORE           FILENAME
default             FreeBSD-13.2-RELEASE-amd64-disc1.iso
default             ubuntu-mate-23.04-desktop-amd64.iso

Guest OS Install

Generally each VM install is very similar as shown below.

host # vm create -t TEMPLATE NAME

host # vm install NAME ISO

host # vm console NAME

Example for FreeBSD is below.

host # vm create -t freebsd freebsd13

host # vm install freebsd13 FreeBSD-13.2-RELEASE-amd64-disc1.iso
Starting freebsd13
  * found guest in /vm/freebsd13
  * booting...

host # vm console freebsd13

You will probably see something like that below.

freebsd-loader-menu

Then you do the installation in the text mode and after reboot you have your running FreeBSD VM.

host # vm list
NAME          DATASTORE  LOADER     CPU  MEMORY  VNC  AUTO     STATE
freebsd13     default    bhyveload  1    256M    -    Yes [1]  Running (85315)

Some more info to display can be shown with info argument.

host # vm info freebsd13
------------------------
Virtual Machine: freebsd13
------------------------
  state: stopped
  datastore: default
  loader: bhyveload
  uuid: a91287a1-39d3-11ee-b73d-f0def1d6aea1
  cpu: 1
  memory: 256M

  network-interface
    number: 0
    emulation: virtio-net
    virtual-switch: public
    fixed-mac-address: 58:9c:fc:0b:98:30
    fixed-device: -

  virtual-disk
    number: 0
    device-type: file
    emulation: nvme
    options: -
    system-path: /vm/freebsd13/disk0.img
    bytes-size: 21474836480 (20.000G)
    bytes-used: 885089280 (844.086M)

  snapshots
    zroot/vm/freebsd13@fresh    85.2M   Mon Aug 14 11:18 2023

host # env EDITOR=cat vm configure freebsd13
loader="bhyveload"
cpu=1
memory=256M
network0_type="virtio-net"
network0_switch="public"
disk0_type="nvme"
disk0_name="disk0.img"
uuid="a91287a1-39d3-11ee-b73d-f0def1d6aea1"
network0_mac="58:9c:fc:0b:98:30"

If you want to edit and not only display the VM config use this.

host # vm configure freebsd13

FreeBSD

FreeBSD can be boot in two ways. One is with bhyveload which may be translated to legacy BIOS boot. You can also of course boot FreeBSD in UEFI mode.

host # cat /vm/.templates/freebsd.conf
loader="bhyveload"
cpu=1
memory=256M
network0_type="virtio-net"
network0_switch="public"
disk0_type="nvme"
disk0_name="disk0.img"

The above will use bhyveload and it mostly works … but sometimes if you want to install a lot newer version under Bhyve the loader may not have all the needed features. I was hit by this problem recently where I used FreeBSD 13.2-RELEASE for the FreeBSD host system and wanted to try 14.0-ALPHA1 version.

I described the details of this problem here – FreeBSD Bug 273099 – in a BUG report.

This is how such error looks like:

| FreeBSD/amd64 User boot lua, Revision 1.2
| ZFS: unsupported feature: com.klarasystems:vdev_zaps_v2
| ERROR: cannot open /boot/lua/loader.lua: no such file or directory.
| 
| Type '?' for a list of commands, 'help' for more detailed help.
| OK 

To overcome that you will need latest (more up to date then 14.0-ALPHA1 version) FreeBSD sources and below commands.

host # pkg install gitup

host # cp /usr/local/etc/gitup.conf.sample /usr/local/etc/gitup.conf

host # gitup current

host # cd /usr/src/stand

host # make

host # find /usr/obj -type f -name userboot_lua.so
/usr/obj/usr/src/amd64.amd64/stand/userboot/userboot_lua/userboot_lua.so

host # cp /usr/obj/usr/src/amd64.amd64/stand/userboot/userboot_lua/userboot_lua.so /vm/userboot_lua.so

Now – we need to add bhyveload_loader="/vm/userboot_lua.so" option to out FreeBSD 14.0-ALPHA1 machine config.

host # cat /vm/freebsd14/freebsd14.conf
loader="bhyveload"
bhyveload_loader="/vm/userboot_lua.so"
cpu=1
memory=256M
network0_type="virtio-net"
network0_switch="public"
disk0_type="nvme"
disk0_name="disk0.img"
uuid="975bca2a-39c4-11ee-b73d-f0def1d6aea1"
network0_mac="58:9c:fc:03:67:47"

Now it will boot properly.

Of course it was very easy to overcome that using UEFI boot instead.

host # cat /vm/freebsd14uefi/freebsd14uefi.conf
loader="uefi"
cpu=1
memory=256M
network0_type="virtio-net"
network0_switch="public"
disk0_type="nvme"
disk0_name="disk0.img"
uuid="35ca42b7-7f28-43eb-afd9-2488c5ec83cf"
network0_mac="58:9c:fc:0a:16:4b"

X11 on FreeBSD

When it comes to X11 graphical output of FreeBSD systems – there are three possibilities.

  • bhyveload – will use BIOS boot and vesa X11 driver with resolution up to 1024×768.
  • uefi – will use UEFI boot and scfb X11 driver with resolution up to 1920×1080.
  • passthru – you can passthru physical GPU and use it as on physical system.

Linux

By default for Linux the grub way is the proposed way. I do not use it as at it only allows console access – and even many so called enterprise grade Linux distributions such as AlmaLinux or Rocky have graphical installer that needs/wants graphical display … and that is only available in uefi mode.

Maybe for Alpine or Void Linux such approach may be usable … but uefi will also work very well – thus I do not see ANY advantages of using grub way here.

I will show you the next example based on AlmaLinux 8.x install but the same worked properly with Ubuntu Mate for example.

First the default template.

host # cat /vm/.templates/linux.conf
loader="uefi"
cpu=2
memory=4G
network0_type="virtio-net"
network0_switch="public"
disk0_type="nvme"
disk0_name="disk0.img"
xhci_mouse="yes"
graphics="yes"

The above added xhci_mouse="yes" uses more precise xhci(4) USB 3.x mouse driver and graphics="yes" forces the exposure of VNC connection.

With such template the installation looks like that.

host # cp AlmaLinux-8.8-x86_64-minimal.iso /vm/.iso/

host # vm create -t linux almalinux8

host # vm install almalinux8 AlmaLinux-8.8-x86_64-minimal.iso
Starting almalinux8
  * found guest in /vm/almalinux8
  * booting...

host # vm list
NAME           DATASTORE  LOADER     CPU  MEMORY  VNC           AUTO  STATE
almalinux8     default    uefi       2    4G      0.0.0.0:5900  No    Running (11819)

host % vncviewer -SendClipboard -AcceptClipboard -LowColorLevel -QualityLevel 6 :5900 &

The last vncviewer(1) command is executed as regular user. It comes from net/tigervnc-viewer package.

If you will be connecting to some external server then use IP address in the command.

host % vncviewer -SendClipboard -AcceptClipboard -LowColorLevel -QualityLevel 6 10.0.0.66::5900 &

After the Linux system is installed you may specify the exact VNC port or IP address. Also the screen resolution or enable/disable waiting for the VNC connection.

graphics_port="5900"
graphics_listen="0.0.0.0"
graphics_res="1400x900"
graphics_wait="no"

Windows 7

A lot of people will criticize me for this one – as Windows 7 is not an officially supported version anymore. I do not care about that when I want to use some localhost software … or older software that works better on older version. Not to mention that its one of the last Windows versions that does not force online Microsoft account down your throat. It also uses less resources and is more responsive.

First – the template – similar to the Linux one.

host # cat /vm/.templates/windows7.conf           
loader="uefi"
graphics="yes"
cpu=2
memory=2G
ahci_device_limit="8"
network0_type="e1000"
network0_switch="public"
disk0_type="ahci-hd"
disk0_name="disk0.img"
disk0_opts="sectorsize=512"
utctime="no"
bhyve_options="-s 8,hda,play=/dev/dsp,rec=/dev/dsp"

If you set the xhci_mouse="yes" option with Windows 7 – you will end up without a working mouse in VNC connection and you will have to make all the install and configuration by keyboard only.

One may think about adding xhci_mouse="yes" after installation when you will already have working RDP connection – but that would also require additional drivers. In theory – the device VEN_8086&DEV_1E31 name is recognized as Intel USB 3.0 eXtensible Host Controller … but for some reason anytime I wanted to install it – the Windows 7 system crashed and instantly rebooted.

The other even more important thing is having the disk0_opts="sectorsize=512" option. Without it the Windows 7 installer will fail with the following error.

win-7-install-error-NO-512-BLOCKS

The last option bhyve_options="-s 8,hda,play=/dev/dsp,rec=/dev/dsp" enables audio.

The install procedure is also similar to Linux.

host # cp win_7_amd64_sp1_en.iso /vm/.iso/

host # vm iso
DATASTORE           FILENAME
default             win_7_amd64_sp1_en.iso

host # vm create -t windows7 -s 40G windows7

host # vm install windows7 win_7_amd64_sp1_en.iso
Starting windows7
  * found guest in /vm/windows7
  * booting...

host # vm list
NAME           DATASTORE  LOADER     CPU  MEMORY  VNC           AUTO  STATE
windows7       default    uefi       2    2G      0.0.0.0:5900  No    Running (11819)

host % vncviewer -SendClipboard -AcceptClipboard -LowColorLevel -QualityLevel 6 :5900 &

After the install we should enable RDP connections for more features. Remember to select any version option.

win-7-computer-properties-advanced-remote

You can add one or more CD-ROM drives with following options in the configure argument.

disk1_type="ahci-cd"
disk1_dev="custom"
disk1_name="/vm/.iso/virtio-drivers.iso"

It would be easier for RDP connections to have static IP instead of a DHCP one.

win-7-network-adapter-IPv4-static

Now as we have the static 10.1.1.7 IP address we can use RDP connection with rdesktop(1) command.

host % rdesktop -u buser -p bpass -P -N -z -g 1800x1000 -a 24 -r sound:local -r disk:HOME=/home/vermaden 10.1.1.7
Autoselecting keyboard map 'en-us' from locale

ATTENTION! The server uses and invalid security certificate which can not be trusted for
the following identified reasons(s);

 1. Certificate issuer is not trusted by this system.

     Issuer: CN=vbox


Review the following certificate info before you trust it to be added as an exception.
If you do not trust the certificate the connection atempt will be aborted:

    Subject: CN=vbox
     Issuer: CN=vbox
 Valid From: Mon Aug 14 00:58:25 2023
         To: Mon Feb 12 23:58:25 2024

  Certificate fingerprints:

       sha1: 4ad853c40a8aa0560af315b691038202506e07ce
     sha256: 44ec8f7650486aef6261aea42da99caba4e84d7bc58341c0ca1bb8e28b81d222


Do you trust this certificate (yes/no)? yes
Connection established using SSL.

There are several useful options here.

The -u buser and -p bpass will take care of credentials.

The -P option enables caching of bitmaps to disk (persistent bitmap caching). This improves performance (especially on low bandwidth connections) and reduces network traffic.

The -N option enables numlock synchronization between the X11 server and remote RDP session.

The -z enables compression of the RDP datastream.

The -g 1800x1000 and -a 24 specifies resolution and color depth rate.

The -r disk:HOME=/home/vermaden enables transparent sharing of your home directory and additional share is shown in My Computer in the Windows 7 machine – very handy for sharing files between the host and guest VM as chown below.

win-7-sharing

The last one option -r sound:local specifies that the audio will be realized on the guest VM – this will only work if you added the bhyve_options="-s 8,hda,play=/dev/dsp,rec=/dev/dsp" to the Windows 7 Bhyve config. Alternatively without that hda(4) emulation you can use -r sound:remote option – this would use RDP protocol to transfer audio events from the guest machine to your host machine and then audio will be played then locally on your host machine.

Windows 10

Finally a supported version.

Template is similar to the Windows 7 one.

host # cat /vm/.templates/windows10.conf
loader="uefi"
graphics="yes"
xhci_mouse="yes"
cpu=2
memory=2G
ahci_device_limit="8"
network0_type="e1000"
network0_switch="public"
disk0_type="nvme"
disk0_name="disk0.img"
utctime="no"
bhyve_options="-s 8,hda,play=/dev/dsp,rec=/dev/dsp"

The Windows 10 supports the xhci_mouse="yes" so we enable and keep it all the time.

The Windows 10 does not need the disk0_opts="sectorsize=512" option.

As Windows 10 is newer – I recommended to use nvme – but I sometimes got these Windows machines frozen/locked up after some time and switching to ahci-hd seems to solve that problem. So to summarize – use ahci-hd on all Windows versions.

The last option bhyve_options="-s 8,hda,play=/dev/dsp,rec=/dev/dsp" enables audio.

The install procedure is also similar to Windows 7.

host # cp win_10_amd64_en_LTSC.iso /vm/.iso/

host # vm iso
DATASTORE           FILENAME
default             win_10_amd64_en_LTSC.iso

host # vm create -t windows10 -s 40G windows10

host # vm install windows10 win_10_amd64_en_LTSC.iso
Starting windows10
  * found guest in /vm/windows10
  * booting...

host # vm list
NAME           DATASTORE  LOADER     CPU  MEMORY  VNC           AUTO  STATE
windows10      default    uefi       2    2G      0.0.0.0:5900  No    Running (11819)

host % vncviewer -SendClipboard -AcceptClipboard -LowColorLevel -QualityLevel 6 :5900 &

After the install we should enable RDP connections for more features. Remember to select any version option.

win-10-advanced-settings-remote

You can add one or more CD-ROM drives with following options in the configure argument.

disk1_type="ahci-cd"
disk1_dev="custom"
disk1_name="/vm/.iso/virtio-drivers.iso"

It would be easier for RDP connections to have static IP instead of a DHCP one.

win-10-network-settings-adapter-properties-IPv4-static

Now as we have the static 10.1.1.8 IP address we can use RDP connection with rdesktop(1) command.

host % rdesktop -u buser -p bpass -P -N -z -g 1600x900 -a 24 -r sound:local -r disk:HOME=/home/vermaden 10.1.1.8
Autoselecting keyboard map 'en-us' from locale

ATTENTION! The server uses and invalid security certificate which can not be trusted for
the following identified reasons(s);

 1. Certificate issuer is not trusted by this system.

     Issuer: CN=DESKTOP-HKJ3H6T


Review the following certificate info before you trust it to be added as an exception.
If you do not trust the certificate the connection attempt will be aborted:

    Subject: CN=DESKTOP-HKJ3H6T
     Issuer: CN=DESKTOP-HKJ3H6T
 Valid From: Mon Aug 14 10:33:41 2023
         To: Tue Feb 13 09:33:41 2024

  Certificate fingerprints:

       sha1: 967d5cdb164e53f7eb4c5c17b0343f2f279fb709
     sha256: c08b732122a39c44d91fac2a9093724da12d2f3e6ea51613245d13cf762f4cd2


Do you trust this certificate (yes/no)? yes

Options are the same as with Windows 7 and they are described in the Windows 7 section.

Force Windows 10 Offline Account

To force creation of local account instead of forced online account you need to boot the Windows 10 without network.

Do the following steps to do that.

host # yes | vm poweroff windows10

host # vm configure windows10
- network0_type="e1000"
- network0_switch="public"

host # vm start windows10

Now create the offline account.

After creating it poweroff the Windows 10 VM.

host # vm configure windows10
+ network0_type="e1000"
+ network0_switch="public"

host # vm start windows10

Now you have local account on Windows 10 system.

Windows 10 Bloat Removers

You may consider using on of the known Windows 10 bloat removers available here:

Windows 11

The setup/install of Windows 11 is the same as Windows 10.

Dealing with Locked VMs

Lets assume that our host system crashed.

The vm-bhyve will left run.lock files in the machines dirs.

host # ls -l /vm/freebsd14uefi
total 1389223K
-rw-r--r-- 1 root wheel          32 2023-08-16 23:36 console
-rw------- 1 root wheel 21474836480 2023-08-16 23:46 disk0.img
-rw-r--r-- 1 root wheel         200 2023-08-16 23:35 freebsd14uefi.conf
-rw-r--r-- 1 root wheel          11 2023-08-16 23:36 run.lock
-rw-r--r-- 1 root wheel        5583 2023-08-16 23:36 vm-bhyve.log

host # vm list
NAME           DATASTORE  LOADER     CPU  MEMORY  VNC  AUTO     STATE
almalinux8     default    uefi       2    2G      -    No       Stopped
freebsd13      default    bhyveload  1    256M    -    Yes [1]  Running (19258)
freebsd13alt   default    bhyveload  1    256M    -    No       Stopped
freebsd14      default    bhyveload  1    256M    -    No       Stopped
freebsd14uefi  default    uefi       2    8G      -    No       Locked (w520.local)
windows10ltsc  default    uefi       2    2G      -    No       Stopped
windows7       default    uefi       2    2G      -    No       Stopped

host # rm /vm/freebsd14uefi/run.lock

host # vm list
NAME           DATASTORE  LOADER     CPU  MEMORY  VNC  AUTO     STATE
almalinux8     default    uefi       2    2G      -    No       Stopped
freebsd13      default    bhyveload  1    256M    -    Yes [1]  Running (19258)
freebsd13alt   default    bhyveload  1    256M    -    No       Stopped
freebsd14      default    bhyveload  1    256M    -    No       Stopped
freebsd14uefi  default    uefi       2    8G      -    No       Stopped
windows10ltsc  default    uefi       2    2G      -    No       Stopped
windows7       default    uefi       2    2G      -    No       Stopped

Now you may want to start the locked machine properly.

Disk Resize

By default vm-bhyve will create disks with 20 GB in size.

To resize the Bhyve virtual machine disk we would use truncate(1) command.

host # vm stop freebsd13

host # cd /vm/freebsd13

host # truncate -s 40G disk0.img

host # vm start freebsd13

If you are not sure about that – you may work on a copy instead.

host # vm stop freebsd13

host # truncate -s 40G disk0.img.NEW

host # dd bs=1m if=disk0.img of=disk0.img.NEW conv=notrunc status=progress
  20865613824 bytes (21 GB, 19 GiB) transferred 43.002s, 485 MB/s
20480+0 records in
20480+0 records out
21474836480 bytes transferred in 43.454036 secs (494196586 bytes/sec)

host # mv disk0.img disk0.img.BACKUP

host # mv disk0.img.NEW disk0.img

host # vm start freebsd13

Now we need to resize the filesystem inside the VM.

freebsd13 # lsblk
DEVICE         MAJ:MIN SIZE TYPE                                          LABEL MOUNT
nvd0             0:90   40G GPT                                               - -
  nvd0p1         0:91  512K freebsd-boot                           gpt/gptboot0 -
           -:-   492K -                                                 - -
  nvd0p2         0:92  2.0G freebsd-swap                              gpt/swap0 SWAP
  nvd0p3         0:93   18G freebsd-zfs                                gpt/zfs0 
           -:-   1.0M -                                                 - -

freebsd13 # geom disk list
Geom name: nvd0
Providers:
1. Name: nvd0
   Mediasize: 42949672960 (40G)
   Sectorsize: 512
   Mode: r2w2e3
   descr: bhyve-NVMe
   lunid: 589cfc2081410001
   ident: NVME-4-0
   rotationrate: 0
   fwsectors: 0
   fwheads: 0

freebsd13 # gpart show
=>      40  41942960  nvd0  GPT  (40G) [CORRUPT]
        40      1024     1  freebsd-boot  (512K)
      1064       984        - free -  (492K)
      2048   4194304     2  freebsd-swap  (2.0G)
   4196352  37744640     3  freebsd-zfs  (18G)
  41940992      2008        - free -  (1.0M)

freebsd13 # gpart recover nvd0
nvd0 recovered

freebsd13 # gpart show
=>      40  83886000  nvd0  GPT  (40G)
        40      1024     1  freebsd-boot  (512K)
      1064       984        - free -  (492K)
      2048   4194304     2  freebsd-swap  (2.0G)
   4196352  37744640     3  freebsd-zfs  (18G)
  41940992  41945048        - free -  (20G)

freebsd13 # gpart resize -i 3 -a 1m nvd0
nvd0p3 resized

freebsd13 # gpart show
=>      40  83886000  nvd0  GPT  (40G)
        40      1024     1  freebsd-boot  (512K)
      1064       984        - free -  (492K)
      2048   4194304     2  freebsd-swap  (2.0G)
   4196352  79687680     3  freebsd-zfs  (38G)
  83884032      2008        - free -  (1.0M)

freebsd13 # zpool status
  pool: zroot
 state: ONLINE
config:

        NAME        STATE     READ WRITE CKSUM
        zroot       ONLINE       0     0     0
          nvd0p3    ONLINE       0     0     0

freebsd13 # zpool list
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
zroot  17.5G  17.0G   544M        -         -    87%    96%  1.00x    ONLINE  -

freebsd13 # zpool set autoexpand=on zroot

freebsd13 # zpool online -e zroot nvd0p3

freebsd13 # zpool list
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
zroot  37.5G  17.0G  20.5G        -         -    41%    45%  1.00x    ONLINE  -

Summary

I hope I was able to provide all needed information.

Let me know in comments if I missed something.

UPDATE 1 – The sysutils/edk2 Issue

Recently a lot of people started to get problems with running UEFI machines on Bhyve. After a short investigation (details in the 273560 BUG report) the root of all cause was the new sysutils/edk2 version that caused the problem. The problem does not exists as long as You use the -A flag for bhyve(8) command. Unfortunately its not the default for vm-bhyve and -A options is needed in the bhyve_options parameter in each UEFI boot VM.

For example:

host % grep bhyve_options /vm/freebsd14uefi/freebsd14uefi.conf 
bhyve_options="-A"

Additional details for vm-bhyve available HERE.

Hope that helps.

UPDATE 2 – RHEL 8.10 Network Issue

About a day ago I needed to install Red Hat Linux 8.10 in a very specific way. The Bhyve virtual machine had two network interfaces. First and primary one – the one that will be used with default gateway and another interface – in which this machine also needs to be present. The second interface was in a network where DHCP server was running – I did not wanted to use it – so I specified a static (reserved) IP address. All of this was ‘clicked’ and entered in the GUI Anaconda installer. There also was need for specific DNS to use.

Below you will find screenshots from the install process.

rhel-8.10-install-0-minimal

rhel-8.10-install-1-interface-a

rhel-8.10-install-1-interface-b

After RHEL 8.10 reboot I checked – over VNC connection because there was no network connectivity to that machine at all – that something strange happened. This is how network interfaces looked like.

rhel-8.10-fix-000

No network configuration on first network interface at all and IP address (and other settings) from DHCP server … including wrong DNS server.

rhel-8.10-fix-001

While network interface enp0s5f0 and enp0s5f1 were available at install time – they have become gone – and after the reboot there only enp0s6f0 and enp0s6f1 ones without any configuration at all – known for nmcli(8) as Wired connection 1 and Wired connection 2 … WTF?

After a lot of manual work – removing all available connections using nmcli(8) and creating new ones – I finally got my configuration as I entered it into Anaconda installer several hours earlier.

UPDATE-2-rhel-8.10-fix-002

It generally required to copy/paste configs of network interfaces created during install to the newly created ones with new UUID values.

One thing that also was very strange to me was that nothing worked until I executed the nmcli con reload command – same shit as with systemd(1) when you – for example – edit /etc/fstab file and it will not work until you run systemctl daemon-reload command.

To be honest – I am considering making them ‘run’ at every [ENTER] press at the shell – to just not waste time on this bullshit.

Currently they are:

  • nmcli con reload
  • systemctl daemon-reload

I am curious if I should add another …

I do not even know how to summarize this UPDATE to be honest, Red Hat Linux is broken and you need to waste hours of debugging at every install or …

Lets hope this will save time for some people at least.

Sorry for screenshots from VNC instead of commands – but that is all I had back then.

UPDATE 3 – X11 on FreeBSD

Some people asked me to provide that information – so I added a subsection in the FreeBSD section about that.

UPDATE 4 – PCI Passthru

While FreeBSD Bhyve is not (yet) able to passthru single USB port – it is able to passthru entire PCI USB controller into Bhyve VM. I just tested that on Windows 7 VM – but works for any OS really.

First – list USB controllers you have in your system. Below is list of mine on ThinkPad W520 laptop:

FreeBSD # pciconf -lv | grep -E -e '(u|e|x)hci'
ehci0@pci0:0:26:0:      class=0x0c0320 rev=0x04 hdr=0x00 vendor=0x8086 device=0x1c2d subvendor=0x17aa subdevice=0x21cf
ehci1@pci0:0:29:0:      class=0x0c0320 rev=0x04 hdr=0x00 vendor=0x8086 device=0x1c26 subvendor=0x17aa subdevice=0x21cf
xhci0@pci0:14:0:0:      class=0x0c0330 rev=0x04 hdr=0x00 vendor=0x1033 device=0x0194 subvendor=0x17aa subdevice=0x21cf

So ThinkPad W520 has 3 such controllers … but now we need to check how many of them support MSI/MSI-X interrupts.

FreeBSD # pciconf -lvc

// DOES NOT HAVE MSI
ehci0@pci0:0:29:0:      class=0x0c0320 rev=0x04 hdr=0x00 vendor=0x8086 device=0x1c26 subvendor=0x17aa subdevice=0x21cf
    vendor     = 'Intel Corporation'
    device     = '6 Series/C200 Series Chipset Family USB Enhanced Host Controller'
    class      = serial bus
    subclass   = USB
    bar   [10] = type Memory, range 32, base 0xf2629000, size 1024, enabled
    cap 01[50] = powerspec 2  supports D0 D3  current D0
    cap 0a[58] = EHCI Debug Port at offset 0xa0 in map 0x14
    cap 13[98] = PCI Advanced Features: FLR TP

// DOES NOT HAVE MSI
ehci0@pci0:0:26:0:       class=0x0c0320 rev=0x04 hdr=0x00 vendor=0x8086 device=0x1c2d subvendor=0x17aa subdevice=0x21cf
    vendor     = 'Intel Corporation'
    device     = '6 Series/C200 Series Chipset Family USB Enhanced Host Controller'
    class      = serial bus
    subclass   = USB
    bar   [10] = type Memory, range 32, base 0xf262a000, size 1024, enabled
    cap 01[50] = powerspec 2  supports D0 D3  current D0
    cap 0a[58] = EHCI Debug Port at offset 0xa0 in map 0x14
    cap 13[98] = PCI Advanced Features: FLR TP

// HAS MSI
xhci0@pci0:14:0:0:      class=0x0c0330 rev=0x04 hdr=0x00 vendor=0x1033 device=0x0194 subvendor=0x17aa subdevice=0x21cf
    vendor     = 'NEC Corporation'
    device     = 'uPD720200 USB 3.0 Host Controller'
    class      = serial bus
    subclass   = USB
    bar   [10] = type Memory, range 64, base 0xf1400000, size 8192, enabled
    cap 01[50] = powerspec 3  supports D0 D3  current D0
    cap 05[70] = MSI supports 8 messages, 64 bit 
    cap 11[90] = MSI-X supports 8 messages, enabled
                 Table in map 0x10[0x1000], PBA in map 0x10[0x1080]
    cap 10[a0] = PCI-Express 2 endpoint max data 128(128) NS
                 max read 512
                 link x1(x1) speed 5.0(5.0) ASPM L1(L0s/L1) ClockPM enabled
    ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
    ecap 0003[140] = Serial 1 ffffffffffffffff
    ecap 0018[150] = LTR 1

So we can use only one of those USB controllers with Bhyve.

To do so – add below configuration to /boot/loader.conf file:

# BHYVE PCI PASSTHRU
  vmm_load=YES
  pptdevs="14/0/0"

The 14/0/0 represents the PCI BUS 14:0:0 from the pciconf -lvc command output.

Now – after reboot – modify the VM machine config to which you want to passthru that device.

FreeBSD # vi /vm/windows7/windows7.conf
(...)
bhyve_options="-s 7:0,hda,play=/dev/dsp,rec=/dev/dsp -s 11:0,passthru,14/0/0"
wired_memory="yes"
:wq

Now start that VM as usual.

FreeBSD # vm start windows7

I was able to successfully use my Jabra headphones in a testing Webex meeting:

The single USB port/device passthru for Bhyve is still in the works:

– https://github.com/FreeBSD-UPB/freebsd-src/wiki/USB-device-passthrough-for-bhyve

Hope that helps.

EOF