FreeBSD UNIX Wallpapers

About 5 years ago I have shared some ‘oldschool’ computers/machines wallpaper pack – Wallpapers from Tech Pron – available here. I believe this time the wallpapers I will share with You will be similar or even more interesting – the FreeBSD UNIX wallpapers pack 🙂

There are 91 of them – as gathered thru all of my FreeBSD years. Some are kinda special – at least for me – as for example the ones from the (now defunct) Polish BSD User Group ones.

Here are their miniatures.

Use the following link to download them all wallpapers-freebsd.tar.gz here.

If you are interested how these ‘sets’ images were created – then its these commands below:

% which montage 
/usr/local/bin/montage

% pkg which -o $( which montage )
/usr/local/bin/montage was installed by package graphics/ImageMagick6

% montage ${LIST_OF_30_IMAGES} -tile 6x5 -geometry '130x75!+0+0' -background "#666666" -borderwidth 1 OUT-1-3.png

UPDATE 1 – Wallpapers Based on Michael W. Lucas Absolute XxxxBSD Books

The Absolute FreeBSD and Absolute OpenBSD book covers – while being great books on the topic – have also really great covers – so I made wallpapers out of them 🙂

I added them to the original wallpapers-freebsd.tar.gz file.

Here they are:

EOF

FreeBSD Bhyve Virtualization

The Bhyve FreeBSD hypervisor (called/spelled ‘beehive’ usually) was created almost 10 years ago. Right now it offers speed and features that other similar solutions provide – such as KVM/VMware/XEN. You can check all the details in the FreeBSD Handbook for details. One of the last things Bhyve lacks is so called live migration between physical hosts but save state and resume from saved state are in the works currently so not long before that live migration. Up until recently I used mostly VirtualBox for my small virtualization needs. Recently I started to evaluate Bhyve and this time I am very pleased – the FreeBSD VirtualBox integration is not perfect anyway – for example – the USB passthru does not work since several years – and even when it worked – it was limited to USB 1.x speeds only. Also because of FreeBSD policy of pkg(8) packages building process – the VirtualBox packages remain broken for 3 months after each *.1 or upper release (*.2/*.3/…). The other impulse that forced me to switch from VirtualBox to Bhyve was the VirtualBox (in)stability. I often needed to restart crashed VirtualBox VMs because they failed for some unspecified reason.

One of the Bhyve features that I especially liked was that by default Bhyve only uses memory that guest system wanted to allocate. For example a FreeBSD virtual machine with 2 GB RAM set will use after boot about 70 MB RAM 🙂

Another great feature I really liked about Bhyve was that I could suspend the host machine with all the VMs started – both on my ThinkPad W520 laptop and AMD Based FreeBSD Desktop and then it all successfully resumed. With VirtualBox you would have to poweroff all VMs because if you suspend with running VirtualBox VMs – it will just crash – its not possible to do suspend/resume cycle with VirtualBox.

The Table of Contents for this article is as follows:

• FreeBSD Bhyve Virtualization
• Bhyve Managers
• Bhyve versus KVM and VMware
• Bhyve libvirt/virt-manager GUI
• vm-bhyve
  • Install/Setup
  • Networking
    • Server/Desktop LAN Bridge
    • Laptop WiFi NAT
    • Networking Restart
  • Datastores
  • Templates
  • NVMe
  • ISO Images
  • Guest OS Install
    • FreeBSD
    • Linux
    • Windows 7
    • Windows 10
      • Force Windows 10 Offline Account
      • Windows 10 Bloat Removers
    • Windows 11
  • Dealing with Locked VMs
  • Disk Resize
• Summary

Bhyve Managers

While VirtualBox has quite usable QT based GUI – the Bhyve does not have anything like that. I once seen some GUI QT prototype for Bhyve but it was very basic – so forget about that currently. There are however several web interfaces such as TrueNAS CORE or CBSD/CloneOS. There are also several terminal managers such as vm-bhyve. The older one iohyve has not been maintained for at least 6 long years. There is also libvirt Bhyve driver but more on that later.

Bhyve versus KVM and VMware

Klara Systems compared Bhyve to KVM and Benjamin Bryan compared it against VMware hypervisor. While Bhyve remains competitive against both of them there are two important aspects from Klara Systems that stand out and are worth repeating here.

First – using nvme driver is a lot faster then using more traditional virtio-blk or ahci-hd backends.

Second – and this one seems strange – using a raw file is faster then using ZFS zvol device.

To summarize these thoughts – just just file on a disk like disk0.img and use nvme as storage backend everytime the guest operating system supports it.

Bhyve libvirt/virt-manager GUI

Theoretically the libvirt virtualization API supports Bhyve as one of its backends and the details about Bhyve driver – https://libvirt.org/drvbhyve.html – are available here. I have tried it with virt-manager and after some basic configuration I was able to start FreeBSD 13.2 installation … but it got frozen at the kernel messages and nothing more happened.

… and the moment it hanged below. I have tried multiple times with the same effect.

I really liked the virtual machine settings window of virt-manager.

vm-bhyve

While You can use Bhyve directly with bhyve(8) and bhyvectl(8) commands – which I was doing in the past – after trying the vm-bhyve both on the desktop and server space – I really liked it and this is what I currently use. I just moved from vm-bhyve package to the newer vm-bhyve-devel one.

The vm(8) command is simple and provides all needed use cases.

host # vm help
vm-bhyve: Bhyve virtual machine management v1.6-devel (rev. 106001)
Usage: vm ...
    version
    init
    set [setting=value] [...]
    get [all|setting] [...]
    switch list
    switch info [name] [...]
    switch create [-t type] [-i interface] [-n vlan-id] [-m mtu] [-a address/prefix-len] [-b bridge] [-p] 
    switch vlan  
    switch nat  
    switch private  
    switch add  
    switch remove  
    switch destroy 
    datastore list
    datastore add  
    datastore remove 
    datastore add  
    list
    info [name] [...]
    create [-d datastore] [-t template] [-s size] [-m memory] [-c vCPUs] 
    install [-fi]  
    start [-fi]  [...]
    stop  [...]
    restart 
    console  [com1|com2]
    configure 
    rename  
    add [-d device] [-t type] [-s size|switch] 
    startall
    stopall
    reset  [-f] 
    poweroff [-f] 
    destroy [-f] 
    passthru
    clone  
    snapshot [-f] 
    rollback [-r] 
    iso [url]
    img [url]
    image list
    image create [-d description] [-u] 
    image destroy 
    image provision [-d datastore]  

Install/Setup

We need only several packages to add.

host # pkg install -y \
         vm-bhyve-devel \
         uefi-edk2-bhyve-csm \
         bhyve-firmware \
         edk2-bhyve \
         dnsmasq \
         grub2-bhyve \
         tigervnc-viewer \
         rdesktop

The setup is pretty easy also.

First we need to add several vm_* settings into the main FreeBSD /etc/rc.conf file.

  vm_enable=YES
  vm_dir="zfs:zroot/vm"
  vm_list=""
  vm_delay=3

Keep in mind that you will later use the vm_list="" for the list of VMs that you would like to be started at boot. Like vm_list="freebsd13 freebsd14uefi" for example. Then the vm list command would place [1] in at the freebsd13 name (as its first) and [2] in the freebsd14uefi name as this one is second on the list. See below.

host # vm list
NAME           DATASTORE  LOADER     CPU  MEMORY  VNC           AUTO     STATE
almalinux8     default    uefi       2    2G      0.0.0.0:5908  No       Running (11819)
freebsd13      default    bhyveload  1    256M    -             Yes [1]  Running (2342)
freebsd14      default    bhyveload  1    256M    -             No       Stopped
freebsd14uefi  default    uefi       2    8G      -             Yes [2]  Running (35394)
windows10      default    uefi       2    2G      -             No       Stopped
windows7       default    uefi       2    2G      -             No       Stopped

We need to create a dedicated ZFS dataset for our VMs. You can also use directory on UFS – check vm-bhyve documentation.

host # zfs create -o mountpoint=/vm zroot/vm

We will also copy the available templates to our new /vm dir.

host # cp -a /usr/local/share/examples/vm-bhyve /vm/.templates

Remember to check /vm/.templates/config.sample as it has the documentation for all available options.

host # head -12 /vm/.templates/config.sample
# This is a sample configuration file containing all supported options
# Please do not try and use this file itself for a guest
# For any option that contains a number in the name, such as "network0_type",
# you can add additional devices of that type by creating a new set of
# variables using the next number in sequence, e.g "network1_type"
#
# Please make sure all option names are specified in lowercase and
# at the beginning of the line. If there is any whitespace before
# the option name, the line will be ignored.
# The '#' character signifies the start of a comment, even within
# double-quotes, and so cannot be used inside any values.

We can now start initialize the vm-bhyve.

host # service vm start

Networking

There as many network setups as many FreeBSD has network capabilities – a lot! I this guide I will cover two most typical network setups for Bhyve. One would be the most server (or desktop) oriented – as it requires a LAN card to be used. The other one I would call a laptop one – that one would provide network connectivity using wlan0 WiFi interface.

No matter which one we will choose – we need to enable port forwarding on our FreeBSD host. Do that with these two commands.

host # sysrc gateway_enable=YES

host # sysctl net.inet.ip.forwarding=1

host # echo net.link.tap.up_on_open=1 >> /etc/sysctl.conf

host # sysctl net.link.tap.up_on_open=1

I assume that our FreeBSD host system would use 10.0.0.10/24 IP address and that 10.0.0.1 would be its default gateway.

Your host system main /etc/rc.conf file can looks as follows then.

host # cat /etc/rc.conf
# NETWORK
  hostname=host
  ifconfig_re0="inet 10.0.0.10/24 up"
  defaultrouter="10.0.0.1"
  gateway_enable=YES

# DAEMONS
  sshd_enable=YES
  zfs_enable=YES

# BHYVE
  vm_enable="YES"
  vm_dir="zfs:zroot/vm"
  vm_list=""
  vm_delay="3"

Server/Desktop LAN Bridge

We will use 10.0.0.0/24 network – the same that our host system uses. We will need one bridge/switch named vm-public with 10.0.0.100/24 address on it. Without that address later the dnsmasq will complain unknown interface vm-public about it. Information about the switches is kept in the /vm/.config/system.conf file. We will also need to add out LAN interface to the public switch. It will be re0 interface in my case.

host # vm switch create -a 10.0.0.100/24 public

host # vm switch add public re0

host # vm switch list
NAME    TYPE      IFACE      ADDRESS        PRIVATE  MTU  VLAN  PORTS
public  standard  vm-public  10.0.0.100/24  no       -    -     re0

host # cat /vm/.config/system.conf
switch_list="public"
type_public="standard"
addr_public="10.0.0.100/24"
ports_public="re0"

To be honest the networking part setup is complete.

When you will be setting up your Bhyve VMs you will either use static 10.0.0.0/24 IP address space or just use DHCP and the one that is already on your network will take care of the rest (assuming you have one).

If you do not have one you may use dnsmasq service to do that easily.

host # cat /usr/local/etc/dnsmasq.conf
port=0
no-resolv
server=1.1.1.1
except-interface=lo0
bind-interfaces
local-service
dhcp-authoritative
interface=vm-public
dhcp-range=10.0.0.69,10.0.0.96

host # service dnsmasq enable

host # service dnsmasq start

That should do.

Now – to access the VMs in this bridged networking mode you can just ssh(1) to their IP address directly.

Laptop WiFi NAT

This is one of the cases where VirtualBox has one more feature over Bhyve. With VirtualBox its possible to use bridge mode over WiFi interface. Its not possible with Bhyve currently. I have submitted a proposal to FreeBSD Foundation to implement such configuration – especially as open source VirtualBox code already exists. Time will tell if it will be implemented or if there would be more important tasks to take care of.

We will use 10.1.1.0/24 network for our VM needs. We will also need only one vm-bhyve switch that we will use – and it will be the vm-public one with 10.1.1.1/24 address – we will be using it as a gateway for our VMs in NAT. Information about the switches is kept in the /vm/.config/system.conf file.

host # vm switch create -a 10.1.1.1/24 public

host # vm switch list
NAME    TYPE      IFACE      ADDRESS      PRIVATE  MTU  VLAN  PORTS
public  standard  vm-public  10.1.1.1/24  no       -    -     -

host # cat /vm/.config/system.conf 
switch_list="public"
type_public="standard"
addr_public="10.1.1.1/24"

Now the NAT part – we will do that with very simple pf(4) config.

host # cat /etc/pf.conf
# SKIP LOOPBACK
  set skip on lo0

# bhyve(8) VMS NAT 
  nat on wlan0 from {10.1.1.1/24} to any -> (wlan0)

# PASS IN/OUT ALL
  pass in all
  pass out all

host # service pf enable

host # service pf start

You can check the stats of that pf(4) rules like that.

host # pfctl -Psn -vv
No ALTQ support in kernel
ALTQ related functions disabled
@0 nat on wlan0 inet from 10.1.1.0/24 to any -> (wlan0) round-robin
  [ Evaluations: 18774     Packets: 362277    Bytes: 352847937   States: 0     ]
  [ Inserted: uid 0 pid 69837 State Creations: 38    ]

Feel free to add all your pf(4) rules into the /etc/pf.conf file.

Now the DHCP server. For simplicity of the setup we will use dnsmasq daemon – but nothing prevents you from setting up a Highly Available DHCP Server instead using isc-dhcp44-server package.

host # cat /usr/local/etc/dnsmasq.conf
port=0
no-resolv
server=1.1.1.1
except-interface=lo0
bind-interfaces
local-service
dhcp-authoritative
interface=vm-public
dhcp-range=10.1.1.11,10.1.1.99

host # service dnsmasq enable

host # service dnsmasq start

Now you should be ready to setup Bhyve VMs on your laptop.

When I was using VirtualBox – it allowed me to use its Port Forwarding feature where I could add as many mappings in the NAT network type as shown below.

… but with Bhyve its even better as no port forwarding in the NAT mode is needed at all! 🙂

You can access the Bhyve VMs in NAT networking mode the same as you can in the bridged mode – with just ssh(1) to their IP address from directly from the 10.1.1.0/24 range.

So to put it bluntly – if a Bhyve VM in NAT mode got 10.1.1.33/24 IP address – then just ssh(1) to that IP address directly from the host system.

Networking Restart

Sometimes – when for example you laptop will boot without network connectivity – the tap(4) interfaces sometimes do not went UP.

There is simple fix tor that problem – bhyve-network-restart.sh script.

Its shown below.

# ADD IP ADDRESS TO EACH vm-bhyve SWITCH
vm switch list \
  | sed 1d \
  | while read NAME TYPE IFACE ADDRESS PRIVATE MTU VLAN PORTS a0 a1 a2 a3 a4 a5 a6 a7 a8 a9
    do
      if [ "${ADDRESS}" != "-" ]
      then
             vm switch address ${NAME} ${ADDRESS}
        echo vm switch address ${NAME} ${ADDRESS}
      fi
    done        

# SET TO 'up' ALL vm-bhyve SWITCH MEMBERS
vm switch list \
  | sed 1d \
  | awk '{print $1}' \
  | while read SWITCH
    do
      ifconfig vm-${SWITCH} \
        | awk '/member:/ {print $2}' \
        | while read INTERFACE
          do
                 ifconfig ${INTERFACE} up
            echo ifconfig ${INTERFACE} up
          done
    done

Execute it everytime you lost connectivity with your VMs and you are done.

Datastores

While vm-bhyve supports multiple datastores – you will only need one – the default one.

host # vm datastore list
NAME            TYPE        PATH                      ZFS DATASET
default         zfs         /vm                       zroot/vm

Snapshots and Clones

The vm-bhyve also supports snapshots and clones of the VMs disks. Generally they are just ZFS snapshots and clones.

Templates

While vm-bhyve comes with several handy templates – they are incomplete – and small several changes makes the game more playable.

NVMe

First – we will implement the things that we know work faster – the nvme type for disk images instead of virt-blk or ahci-hd ones. Of course not all operating systems have support for such devices – for them we will use the latter options.

A fast way to change it to nvme is below.

host # sed -i '' s.virtio-blk.nvme.g /vm/.templates/freebsd.conf

ISO Images

Each VM needs an ISO image from which it will be installed. Of course you can also just create new VM and copy the disk contents from other server or use one of the FreeBSD images.

There are two ways to feed vm-bhyve with ISO images.

One is to fetch them from some URL.

host # vm iso http://ftp.freebsd.org/pub/FreeBSD/releases/ISO-IMAGES/13.2/FreeBSD-13.2-RELEASE-amd64-disc1.iso

host # vm iso
DATASTORE           FILENAME
default             FreeBSD-13.2-RELEASE-amd64-disc1.iso

The other way is to just simple copy ISO file to the /vm/.iso directory.

host # cp /home/vermaden/download/ubuntu-mate-23.04-desktop-amd64.iso /vm/.iso/

host # vm iso
DATASTORE           FILENAME
default             FreeBSD-13.2-RELEASE-amd64-disc1.iso
default             ubuntu-mate-23.04-desktop-amd64.iso

Guest OS Install

Generally each VM install is very similar as shown below.

host # vm create -t TEMPLATE NAME

host # vm install NAME ISO

host # vm console NAME

Example for FreeBSD is below.

host # vm create -t freebsd freebsd13

host # vm install freebsd13 FreeBSD-13.2-RELEASE-amd64-disc1.iso
Starting freebsd13
  * found guest in /vm/freebsd13
  * booting...

host # vm console freebsd13

You will probably see something like that below.

Then you do the installation in the text mode and after reboot you have your running FreeBSD VM.

host # vm list
NAME          DATASTORE  LOADER     CPU  MEMORY  VNC  AUTO     STATE
freebsd13     default    bhyveload  1    256M    -    Yes [1]  Running (85315)

Some more info to display can be shown with info argument.

host # vm info freebsd13
------------------------
Virtual Machine: freebsd13
------------------------
  state: stopped
  datastore: default
  loader: bhyveload
  uuid: a91287a1-39d3-11ee-b73d-f0def1d6aea1
  cpu: 1
  memory: 256M

  network-interface
    number: 0
    emulation: virtio-net
    virtual-switch: public
    fixed-mac-address: 58:9c:fc:0b:98:30
    fixed-device: -

  virtual-disk
    number: 0
    device-type: file
    emulation: nvme
    options: -
    system-path: /vm/freebsd13/disk0.img
    bytes-size: 21474836480 (20.000G)
    bytes-used: 885089280 (844.086M)

  snapshots
    zroot/vm/freebsd13@fresh    85.2M   Mon Aug 14 11:18 2023

host # env EDITOR=cat vm configure freebsd13
loader="bhyveload"
cpu=1
memory=256M
network0_type="virtio-net"
network0_switch="public"
disk0_type="nvme"
disk0_name="disk0.img"
uuid="a91287a1-39d3-11ee-b73d-f0def1d6aea1"
network0_mac="58:9c:fc:0b:98:30"

If you want to edit and not only display the VM config use this.

host # vm configure freebsd13

FreeBSD

FreeBSD can be boot in two ways. One is with bhyveload which may be translated to legacy BIOS boot. You can also of course boot FreeBSD un UEFI mode.

host # cat /vm/.templates/freebsd.conf
loader="bhyveload"
cpu=1
memory=256M
network0_type="virtio-net"
network0_switch="public"
disk0_type="nvme"
disk0_name="disk0.img"

The above will use bhyveload and it mostly works … but sometimes if you want to install a lot newer version under Bhyve the loader may not have all the needed features. I was hit by this problem recently where I used FreeBSD 13.2-RELEASE for the FreeBSD host system and wanted to try 14.0-ALPHA1 version.

I described the details of this problem here – FreeBSD Bug 273099 – in a BUG report.

This is how such error looks like:

| FreeBSD/amd64 User boot lua, Revision 1.2
| ZFS: unsupported feature: com.klarasystems:vdev_zaps_v2
| ERROR: cannot open /boot/lua/loader.lua: no such file or directory.
| 
| Type '?' for a list of commands, 'help' for more detailed help.
| OK 

To overcome that you will need latest (more up to date then 14.0-ALPHA1 version) FreeBSD sources and below commands.

host # pkg install gitup

host # cp /usr/local/etc/gitup.conf.sample /usr/local/etc/gitup.conf

host # gitup current

host # cd /usr/src/stand

host # make

host # find /usr/obj -type f -name userboot_lua.so
/usr/obj/usr/src/amd64.amd64/stand/userboot/userboot_lua/userboot_lua.so

host # cp /usr/obj/usr/src/amd64.amd64/stand/userboot/userboot_lua/userboot_lua.so /vm/userboot_lua.so

Now – we need to add bhyveload_loader="/vm/userboot_lua.so" option to out FreeBSD 14.0-ALPHA1 machine config.

host # cat /vm/freebsd14/freebsd14.conf
loader="bhyveload"
bhyveload_loader="/vm/userboot_lua.so"
cpu=1
memory=256M
network0_type="virtio-net"
network0_switch="public"
disk0_type="nvme"
disk0_name="disk0.img"
uuid="975bca2a-39c4-11ee-b73d-f0def1d6aea1"
network0_mac="58:9c:fc:03:67:47"

Now it will boot properly.

Of course it was very easy to overcome that using UEFI boot instead.

host # cat /vm/freebsd14uefi/freebsd14uefi.conf
loader="uefi"
cpu=1
memory=256M
network0_type="virtio-net"
network0_switch="public"
disk0_type="nvme"
disk0_name="disk0.img"
uuid="35ca42b7-7f28-43eb-afd9-2488c5ec83cf"
network0_mac="58:9c:fc:0a:16:4b"

Linux

By default for Linux the grub way is the proposed way. I do not use it as at it olny allows console access – and even many so called enterprice grade Linux distributions such as AlmaLinux or Rocky have graphical installer that needs/wants graphical display … and that is only available in uefi mode.

Maybe for Alpine or Void Linux such approach may be usable … but uefi will also work very well – thus I do not see ANY advantages of using grub way here.

I will show you the next example based on AlmaLinux 8.x install but the same worked properly with Ubuntu Mate for example.

First the default template.

host # cat /vm/.templates/linux.conf
loader="uefi"
cpu=2
memory=4G
network0_type="virtio-net"
network0_switch="public"
disk0_type="nvme"
disk0_name="disk0.img"
xhci_mouse="yes"
graphics="yes"

The above added xhci_mouse="yes" uses more precise xhci(4) USB 3.x mouse driver and graphics="yes" forces the exposure of VNC connection.

With such template the installation looks like that.

host # cp AlmaLinux-8.8-x86_64-minimal.iso /vm/.iso/

host # vm create -t linux almalinux8

host # vm install almalinux8 AlmaLinux-8.8-x86_64-minimal.iso
Starting almalinux8
  * found guest in /vm/almalinux8
  * booting...

host # vm list
NAME           DATASTORE  LOADER     CPU  MEMORY  VNC           AUTO  STATE
almalinux8     default    uefi       2    4G      0.0.0.0:5900  No    Running (11819)

host % vncviewer -SendClipboard -AcceptClipboard -LowColorLevel -QualityLevel 6 :5900 &

The last vncviewer(1) command is executed as regular user. It comes from net/tigervnc-viewer package.

If you will be connecting to some external server then use IP address in the command.

host % vncviewer -SendClipboard -AcceptClipboard -LowColorLevel -QualityLevel 6 10.0.0.66::5900 &

After the Linux system is installed you may specify the exact VNC port or IP address. Also the screen resolution or enable/disable waiting for the VNC connection.

graphics_port="5900"
graphics_listen="0.0.0.0"
graphics_res="1400x900"
graphics_wait="no"

Windows 7

A lot of people will criticize me for this one – as Windows 7 is not an officially supported version anymore. I do not care about that when I want to use some localhost software … or older software that works better on older version. Not to mention that its one of the last Windows versions that does not force online Microsoft account down your throat. It also uses less resources and is more responsive.

First – the template – similar to the Linux one.

host # cat /vm/.templates/windows7.conf           
loader="uefi"
graphics="yes"
cpu=2
memory=2G
ahci_device_limit="8"
network0_type="e1000"
network0_switch="public"
disk0_type="ahci-hd"
disk0_name="disk0.img"
disk0_opts="sectorsize=512"
utctime="no"
bhyve_options="-s 8,hda,play=/dev/dsp,rec=/dev/dsp"

If you set the xhci_mouse="yes" option with Windows 7 – you will end up without a working mouse in VNC connection and you will have to make all the install and configuration by keyboard only.

One may think about adding xhci_mouse="yes" after installation when you will already have working RDP connection – but that would also reqiure additional drivers. In theory – the device VEN_8086&DEV_1E31 name is recognized as Intel USB 3.0 eXtensible Host Controller … but for some reason anytime I wanted to install it – the Windows 7 system crashed and instantly rebooted.

The other even more imporant thing is having the disk0_opts="sectorsize=512" option. Without it the Windows 7 instaler will fail with the following error.

The last option bhyve_options="-s 8,hda,play=/dev/dsp,rec=/dev/dsp" enables audio.

The install procedure is also similar to Linux.

host # cp win_7_amd64_sp1_en.iso /vm/.iso/

host # vm iso
DATASTORE           FILENAME
default             win_7_amd64_sp1_en.iso

host # vm create -t windows7 -s 40G windows7

host # vm install windows7 win_7_amd64_sp1_en.iso
Starting windows7
  * found guest in /vm/windows7
  * booting...

host # vm list
NAME           DATASTORE  LOADER     CPU  MEMORY  VNC           AUTO  STATE
windows7       default    uefi       2    2G      0.0.0.0:5900  No    Running (11819)

host % vncviewer -SendClipboard -AcceptClipboard -LowColorLevel -QualityLevel 6 :5900 &

After the install we should enable RDP connections for more features. Rememeber to select any version option.

You can add one or more CD-ROM drives with following options in the configure argument.

disk1_type="ahci-cd"
disk1_dev="custom"
disk1_name="/vm/.iso/virtio-drivers.iso"

It would be easier for RDP connections to have static IP instead of a DHCP one.

Now as we have the static 10.1.1.7 IP address we can use RDP connection with rdesktop(1) command.

host % rdesktop -u buser -p bpass -P -N -z -g 1800x1000 -a 24 -r sound:local -r disk:HOME=/home/vermaden 10.1.1.7
Autoselecting keyboard map 'en-us' from locale

ATTENTION! The server uses and invalid security certificate which can not be trusted for
the following identified reasons(s);

 1. Certificate issuer is not trusted by this system.

     Issuer: CN=vbox


Review the following certificate info before you trust it to be added as an exception.
If you do not trust the certificate the connection atempt will be aborted:

    Subject: CN=vbox
     Issuer: CN=vbox
 Valid From: Mon Aug 14 00:58:25 2023
         To: Mon Feb 12 23:58:25 2024

  Certificate fingerprints:

       sha1: 4ad853c40a8aa0560af315b691038202506e07ce
     sha256: 44ec8f7650486aef6261aea42da99caba4e84d7bc58341c0ca1bb8e28b81d222


Do you trust this certificate (yes/no)? yes
Connection established using SSL.

There are several useful options here.

The -u buser and -p bpass will take care of credentials.

The -P option enables caching of bitmaps to disk (persistent bitmap caching). This improves performance (especially on low bandwidth connections) and reduces network traffic.

The -N option enables numlock synchronization between the X11 server and remote RDP session.

The -z enables compression of the RDP datastream.

The -g 1800x1000 and -a 24 specifies resolution and color depth rate.

The -r disk:HOME=/home/vermaden enables transparent sharing of your home directory and additional share is shown in My Computer in the Windows 7 machine – very handy for sharing files between the host and guest VM as chown below.

The last one option -r sound:local specifies that the audio will be realized on the guest VM – this will only work if you added the bhyve_options="-s 8,hda,play=/dev/dsp,rec=/dev/dsp" to the Windows 7 Bhyve config. Alternatively without that hda(4) emulation you can use -r sound:remote option – this would use RDP protocol to transfer audio events from the guest machine to your host machine and then audio will be played then locally on your host machine.

Windows 10

Finally a supported version.

Template is similar to the Windows 7 one.

host # cat /vm/.templates/windows10.conf
loader="uefi"
graphics="yes"
xhci_mouse="yes"
cpu=2
memory=2G
ahci_device_limit="8"
network0_type="e1000"
network0_switch="public"
disk0_type="nvme"
disk0_name="disk0.img"
utctime="no"
bhyve_options="-s 8,hda,play=/dev/dsp,rec=/dev/dsp"

The Windows 10 supports the xhci_mouse="yes" so we enable and keep it all the time.

The Windows 10 does not need the disk0_opts="sectorsize=512" option.

As Windows 10 is newer – the nvme can (and should) be used for performance reasons.

The last option bhyve_options="-s 8,hda,play=/dev/dsp,rec=/dev/dsp" enables audio.

The install procedure is also similar to Windows 7.

host # cp win_10_amd64_en_LTSC.iso /vm/.iso/

host # vm iso
DATASTORE           FILENAME
default             win_10_amd64_en_LTSC.iso

host # vm create -t windows10 -s 40G windows10

host # vm install windows10 win_10_amd64_en_LTSC.iso
Starting windows10
  * found guest in /vm/windows10
  * booting...

host # vm list
NAME           DATASTORE  LOADER     CPU  MEMORY  VNC           AUTO  STATE
windows10      default    uefi       2    2G      0.0.0.0:5900  No    Running (11819)

host % vncviewer -SendClipboard -AcceptClipboard -LowColorLevel -QualityLevel 6 :5900 &

After the install we should enable RDP connections for more features. Remember to select any version option.

You can add one or more CD-ROM drives with following options in the configure argument.

disk1_type="ahci-cd"
disk1_dev="custom"
disk1_name="/vm/.iso/virtio-drivers.iso"

It would be easier for RDP connections to have static IP instead of a DHCP one.

Now as we have the static 10.1.1.8 IP address we can use RDP connection with rdesktop(1) command.

host % rdesktop -u buser -p bpass -P -N -z -g 1600x900 -a 24 -r sound:local -r disk:HOME=/home/vermaden 10.1.1.8
Autoselecting keyboard map 'en-us' from locale

ATTENTION! The server uses and invalid security certificate which can not be trusted for
the following identified reasons(s);

 1. Certificate issuer is not trusted by this system.

     Issuer: CN=DESKTOP-HKJ3H6T


Review the following certificate info before you trust it to be added as an exception.
If you do not trust the certificate the connection atempt will be aborted:

    Subject: CN=DESKTOP-HKJ3H6T
     Issuer: CN=DESKTOP-HKJ3H6T
 Valid From: Mon Aug 14 10:33:41 2023
         To: Tue Feb 13 09:33:41 2024

  Certificate fingerprints:

       sha1: 967d5cdb164e53f7eb4c5c17b0343f2f279fb709
     sha256: c08b732122a39c44d91fac2a9093724da12d2f3e6ea51613245d13cf762f4cd2


Do you trust this certificate (yes/no)? yes

Options are the same as with Windows 7 and they are described in the Windows 7 section.

Force Windows 10 Offline Account

To force creation of local account instead of forced online account you need to boot the Windows 10 without network.

Do the following steps to do that.

host # yes | vm poweroff windows10

host # vm configure windows10
- network0_type="e1000"
- network0_switch="public"

host # vm start windows10

Now create the offline account.

After creating it poweroff the Windows 10 VM.

host # vm configure windows10
+ network0_type="e1000"
+ network0_switch="public"

host # vm start windows10

Now you have local account on Windows 10 system.

Windows 10 Bloat Removers

You may consider using on of the known Windows 10 bloat removers available here:

• https://github.com/Sycnex/Windows10Debloater
• https://github.com/W4RH4WK/Debloat-Windows-10
• https://github.com/LeDragoX/Win-Debloat-Tools
• https://github.com/ChrisTitusTech/winutil

Windows 11

The setup/install of Windows 11 is the same as Windows 10.

Dealing with Locked VMs

Lets assume that our host system crashed.

The vm-bhyve will left run.lock files in the machines dirs.

host # ls -l /vm/freebsd14uefi
total 1389223K
-rw-r--r-- 1 root wheel          32 2023-08-16 23:36 console
-rw------- 1 root wheel 21474836480 2023-08-16 23:46 disk0.img
-rw-r--r-- 1 root wheel         200 2023-08-16 23:35 freebsd14uefi.conf
-rw-r--r-- 1 root wheel          11 2023-08-16 23:36 run.lock
-rw-r--r-- 1 root wheel        5583 2023-08-16 23:36 vm-bhyve.log

host # vm list
NAME           DATASTORE  LOADER     CPU  MEMORY  VNC  AUTO     STATE
almalinux8     default    uefi       2    2G      -    No       Stopped
freebsd13      default    bhyveload  1    256M    -    Yes [1]  Running (19258)
freebsd13alt   default    bhyveload  1    256M    -    No       Stopped
freebsd14      default    bhyveload  1    256M    -    No       Stopped
freebsd14uefi  default    uefi       2    8G      -    No       Locked (w520.local)
windows10ltsc  default    uefi       2    2G      -    No       Stopped
windows7       default    uefi       2    2G      -    No       Stopped

host # rm /vm/freebsd14uefi/run.lock

host # vm list
NAME           DATASTORE  LOADER     CPU  MEMORY  VNC  AUTO     STATE
almalinux8     default    uefi       2    2G      -    No       Stopped
freebsd13      default    bhyveload  1    256M    -    Yes [1]  Running (19258)
freebsd13alt   default    bhyveload  1    256M    -    No       Stopped
freebsd14      default    bhyveload  1    256M    -    No       Stopped
freebsd14uefi  default    uefi       2    8G      -    No       Stopped
windows10ltsc  default    uefi       2    2G      -    No       Stopped
windows7       default    uefi       2    2G      -    No       Stopped

Now you may want to start the locked machine properly.

Disk Resize

By default vm-bhyve will create disks with 20 GB in size.

To resize the Bhyve virtual machine disk we would use truncate(1) command.

host # vm stop freebsd13

host # cd /vm/freebsd13

host # truncate -s 40G disk0.img

host # vm start freebsd13

If you are not sure about that – you may work on a copy instead.

host # vm stop freebsd13

host # truncate -s 40G disk0.img.NEW

host # dd bs=1m if=disk0.img of=disk0.img.NEW conv=notrunc status=progress
  20865613824 bytes (21 GB, 19 GiB) transferred 43.002s, 485 MB/s
20480+0 records in
20480+0 records out
21474836480 bytes transferred in 43.454036 secs (494196586 bytes/sec)

host # mv disk0.img disk0.img.BACKUP

host # mv disk0.img.NEW disk0.img

host # vm start freebsd13

Now we need to resize the filesystem inside the VM.

freebsd13 # lsblk
DEVICE         MAJ:MIN SIZE TYPE                                          LABEL MOUNT
nvd0             0:90   40G GPT                                               - -
  nvd0p1         0:91  512K freebsd-boot                           gpt/gptboot0 -
           -:-   492K -                                                 - -
  nvd0p2         0:92  2.0G freebsd-swap                              gpt/swap0 SWAP
  nvd0p3         0:93   18G freebsd-zfs                                gpt/zfs0 
           -:-   1.0M -                                                 - -

freebsd13 # geom disk list
Geom name: nvd0
Providers:
1. Name: nvd0
   Mediasize: 42949672960 (40G)
   Sectorsize: 512
   Mode: r2w2e3
   descr: bhyve-NVMe
   lunid: 589cfc2081410001
   ident: NVME-4-0
   rotationrate: 0
   fwsectors: 0
   fwheads: 0

freebsd13 # gpart show
=>      40  41942960  nvd0  GPT  (40G) [CORRUPT]
        40      1024     1  freebsd-boot  (512K)
      1064       984        - free -  (492K)
      2048   4194304     2  freebsd-swap  (2.0G)
   4196352  37744640     3  freebsd-zfs  (18G)
  41940992      2008        - free -  (1.0M)

freebsd13 # gpart recover nvd0
nvd0 recovered

freebsd13 # gpart show
=>      40  83886000  nvd0  GPT  (40G)
        40      1024     1  freebsd-boot  (512K)
      1064       984        - free -  (492K)
      2048   4194304     2  freebsd-swap  (2.0G)
   4196352  37744640     3  freebsd-zfs  (18G)
  41940992  41945048        - free -  (20G)

freebsd13 # gpart resize -i 3 -a 1m nvd0
nvd0p3 resized

freebsd13 # gpart show
=>      40  83886000  nvd0  GPT  (40G)
        40      1024     1  freebsd-boot  (512K)
      1064       984        - free -  (492K)
      2048   4194304     2  freebsd-swap  (2.0G)
   4196352  79687680     3  freebsd-zfs  (38G)
  83884032      2008        - free -  (1.0M)

freebsd13 # zpool status
  pool: zroot
 state: ONLINE
config:

        NAME        STATE     READ WRITE CKSUM
        zroot       ONLINE       0     0     0
          nvd0p3    ONLINE       0     0     0

freebsd13 # zpool list
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
zroot  17.5G  17.0G   544M        -         -    87%    96%  1.00x    ONLINE  -

freebsd13 # zpool set autoexpand=on zroot

freebsd13 # zpool online -e zroot nvd0p3

freebsd13 # zpool list
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
zroot  37.5G  17.0G  20.5G        -         -    41%    45%  1.00x    ONLINE  -

Summary

I hope I was able to provide all needed information.

Let me know in comments if I missed something.

UPDATE 1 – The sysutils/edk2 Issue

Recently a lot of people started to get problems with running UEFI machines on Bhyve. After a short investigation (details in the 273560 BUG report) the root of all cause was the new sysutils/edk2 version that caused the problem. The problem does not exists as long as You use the -A flag for bhyve(8) command. Unfortunately its not the default for vm-bhyve and -A options is needed in the bhyve_options parameter in each UEFI boot VM.

For example:

host % grep bhyve_options /vm/freebsd14uefi/freebsd14uefi.conf 
bhyve_options="-A"

Additional details for vm-bhyve available HERE.

Hope that helps.

EOF

Is FreeBSD a Real UNIX?

That question has been asked and answered many times … but the answer was not always obvious to everyone.

Also today is the FreeBSD day – FreeBSD was created exactly 29 years ago – also on 19th on June.

UNIX System

The first UNIX was the AT&T UNIX whose development started in 1969 at Bell Labs. Then AT&T licensed UNIX to outside parties in late 1970s which have life many UNIX variants like:

• AIX from IBM
  
• BSD (Berkeley Software Distribution) at University of California
• SunOS/Solaris from Sun Microsystems
• HP-UX from HP
• Xenix from Microsoft

Of course AT&T also worked on UNIX and in the late 1980s jointly with Sun Microsystems they released the UNIX System V Release 4 (SVR4). Then in early 1990s the AT&T company sold its UNIX rights to Novell. Then Novell sold UNIX trademark to Open Group. This UNIX journey ends here as its Open Group ownership and they allow the use of the UNIX trademark for certified operating systems that comply with the Single UNIX Specification (SUS) (and pay $100 000 in the certification process).

BSD UNIX

The BSD UNIX had quite unusual version numbering with 1BSD, 2BSD, 3BSD for subsequent releases.

Below you will find most of them listed with some of their features.

1BSD

• run PDP-11
• ex(1) editor

2BSD

• run PDP-11
• vi(1) editor
• C shell

3BSD

• run VAX-11
• DEC VT100 terminal

4BSD

• run VAX-11
• DARPA funding for CSRG
• job control in C shell
• delivermail(8) – antecedent of sendmail(8)
• improved signals
• Curses programming library

4.1BSD

• performance improvements
• on par with VMS on many benchmarks

4.2BSD

• BBN preliminary TCP/IP implementation
• Berkeley Fast File System
• disk quotas
• job control
• run Motorola 68000
• introduction of Daemon mascot

4.3BSD

• improved performance
• more diverged TCP/IP implementation from BBN

4.3BSD-Reno

• move towards POSIX compliance
• NFS implementation
• status key (CTRL-T)
• run HP 9000
• named as “Greatest Software Ever Written”

4.3BSD-Net/1

• most standard UNIX utilities reimplemented without AT&T code
• nvi(1) – new vi(1)
• only few AT&T files remained in kernel

4.3BSD-Net/2

• all AT&T standard UNIX utilities reimplemented without AT&T code
• nearly complete operating system that was freely distributable

386BSD (sometimes also called 386/BSD)

• run Intel 80386
• basically 4.3BSD-Net/2 ported to x86
• base for NetBSD
• base for FreeBSD on 1.0-2.2.8 releases

BSD/386 (later renamed BSD/OS) by BSDi

• legal trouble with AT&T
• AT&T USL versus BSDi lawsuit filed against 4.3BSD-Net/2
• from 18,000 files only 3 had to be removed and 70 modified to show AT&T USL copyright.

4.4BSD-Encumbered

• released only to AT&T USL licensees

4.4BSD-Lite

• no longer requires AT&T USL license
• contained other changes over 4.4BSD-Encumbered

4.4BSD-Lite2

• last BSD UNIX release
• after this release CSRG was dissolved
• FreeBSD since 3.0 also used that code

Intel/x86 Port

The 386BSD UNIX (sometimes also called 386/BSD) was port of the 4.3BSD-Net/2 UNIX to the Intel/x86 architecture. Like recently FreeBSD added support for RISC-V or ARM64. When 386BSD UNIX existed – a group of people took that code and decided that they will develop it under FreeBSD name with their our own way of doing things like serving packages etc.

As you can see the FreeBSD UNIX is – with large simplification – nothing more then 4.3BSD-Net/2 ported to the x86 architecture. But can you call it UNIX? That depends.

FreeBSD UNIX

Let me give you an example from other industry. Can you call Tesla a ‘car’? Yes. Why? Because no one has the copyright for the ‘car’ term. Now imagine this – some company owns the copyright to the ‘car’ term. Now you would not be allowed to call Tesla a ‘car’ without first asking that company for permission. Without such permission you could name Tesla only a car-like vehicle … or automobile – but not simply a ‘car’.

Lets get back to the computing world. Can you officially name FreeBSD a UNIX? No. That is because Open Group company owns copyright to the ‘UNIX’ term. You can call it only a UNIX-like system … but that does not mean its not UNIX. You just can not call it like that because of the lawyers.

NetBSD team also gave good example with their ‘duck’ explanation – https://mollari.netbsd.org/about/call-it-a-duck.html – available here.

Apple paid $100 000 to call their Mac OS X (now macOS) system a UNIX officially.

Linux UNIX

Can you call Linux a UNIX system? Depends which one 🙂 Currently RHEL (Red Hat Enterprise Linux) is not a certified UNIX – Red Hat did not made the $100 000 worth official certification process with Open Group – you can not name RHEL a UNIX system … but RHEL fork can be called officially UNIX. The Inspur-UX is official UNIX because they did that $100 000 certification with Open Group. Probably because they thought that calling their RHEL fork ‘UNIX’ will make their business better.

Does it makes sense? No. But that is just business. Do not seek logic here. Money only.

Summary

Depends if you want the truth or just what the law officially allows 🙂 If you want the truth then yes – FreeBSD is a UNIX system – same as NetBSD or OpenBSD for example. But if you want to obey the law then you can not name these systems UNIX.

EOF

Books About FreeBSD

There are many books in which FreeBSD is covered or it is the one of the main objectives of such book. Today I will guide you through these books. I will try to focus on more up to date ones because it would be pointless (beyond historical purposes) to read them now.

The Hateful Eight

I will start with official FreeBSD documentation – as it offers 8 different books for you to get to know that UNIX operating system better. The two most known are FreeBSD Handbook and FreeBSD FAQ. The FreeBSD Handbook covers all/most general topics about FreeBSD operating system setup and administration while FreeBSD FAQ tries to answer most popular questions about it – and does it quite well. There are also other books that are more developer oriented. Below you will find the list of all available eight books from the FreeBSD project.

–	FreeBSD Handbook	–	https://freebsd.org/handbook
–	FreeBSD FAQ	–	https://freebsd.org/faq
–	FreeBSD Architecture Handbook	–	https://docs.freebsd.org/en/books/arch-handbook/
–	FreeBSD Developers Handbook	–	https://docs.freebsd.org/en/books/developers-handbook/
–	FreeBSD Porters Handbook	–	https://docs.freebsd.org/en/books/porters-handbook/
–	Design and Implementation of 4.4BSD Operating System	–	https://docs.freebsd.org/en/books/design-44bsd/
–	Project Model for FreeBSD	–	https://docs.freebsd.org/en/books/dev-model/
–	FreeBSD Documentation Project Primer for New Contributor	–	https://docs.freebsd.org/en/books/fdp-primer/

Example FreeBSD Handbook page from PDF file below.

… and while you can download and read PDF files (there are also EPUB/HTML/TXT formats available for download) you can also read it online – this is how the FreeBSD FAQ looks online.

I sometimes miss that FreeBSD project does not deliver dedicated book in the same PDF/EPUB/HTML/TXT manner for the FreeBSD man pages as for example Solaris or AIX does. At least you can read them online on the https://man.freebsd.org/command page where you will put the needed man page in the place of ‘command’ word – for example the manual page for gstat(8) command is available at https://man.freebsd.org/gstat URL.

If there are two man pages with the same name like crontab(1) and crontab(5) for example then add the man page section number after the slash (/) at the end of URL like that https://man.freebsd.org/crontab/5 – this will solve that problem.

Below you can see the Solaris 10 – Man Pages Section 1M – System Administration Commands book with all Solaris administrative (1M) commands. For FreeBSD the administrative commands are at (8) section.

The Usual Suspects

The most known books that cover FreeBSD operating system aspects are the ones that are authored by Michael W. Lucas (in more recent titles sometimes accompanied by Allan Jude).

The most famous (and useful) one is the Absolute FreeBSD – Complete Guide to FreeBSD in its most recent 3nd Edition. It is quite recent as its from 2019 year. You can get it both in digital (PDF/EPUB) and traditional printed way. If you already read the FreeBSD Handbook and FreeBSD FAQ are wondering what you should get next to continue your FreeBSD journey then this book is the answer. It fill all the gaps and bring a lot of additional information that you will find very useful in your day to day life with FreeBSD system.

Several books later (not only technical) Michael W. Lucas started the FreeBSD Mastery series with many interesting books about FreeBSD. For the two that cover the ZFS filesystem Allan Jude is also a coauthor.

Here they are:

• FreeBSD Mastery: Storage Essentials (2014)
• FreeBSD Mastery: Specialty Filesystems (2015)
• FreeBSD Mastery: ZFS (2015)
• FreeBSD Mastery: Advanced ZFS (2016)
• FreeBSD Mastery: Jails (2019)

If you are gonna read them – then also do it in that order sa they are listed above. You will need all that ‘introduction’ to get the most of FreeBSD Jails. Even Michael W. Lucas mentioned that you need to ‘do’ several other of his books to truly take advantage of all things written in the FreeBSD Mastery: Jails book. While the FreeBSD Mastery: Storage Essentials and FreeBSD Mastery: Specialty Filesystems are very close related to FreeBSD operating system the other two FreeBSD Mastery: ZFS and FreeBSD Mastery: Advanced ZFS have also a lot of general ZFS knowledge not only limited to FreeBSD operating system.

One additional word about the FreeBSD Mastery: Jails book as its quite ‘special’ in approach. For most parts of the FreeBSD operating system the FreeBSD Handbook covers some or most of the information and tasks needed to do convening particular topic. When it comes to FreeBSD Jails its not that simple anymore. There are two types of Jails. The ‘traditional’ Jails that use the host system network stack and the new ‘VNET’ Jails that bring their own – separate from the host – network stack. It gives you a lot more possibilities and features but it comes with only one downside. The official FreeBSD Handbook does not cover the ‘VNET’ Jails at all. Zero. Nada. Zip. None. You can get ‘some’ grasp about them from the FreeBSD man pages but that is definitely not enough. The ‘VNET’ Jails are of course production ready from many years but for some reason the missing chapter in the FreeBSD Handbook is still missing. That is the most important reason why you should get the FreeBSD Mastery: Jails book.

As we are in the FreeBSD Jails scope … I should also mention the FreeBSD Jails using VNETs book by Derik Ramirez from 2020.

Its not as ‘big’ as the FreeBSD Mastery: Jails but keep in mind that half of Michael W. Lucas content is about the iocage(8) framework … which is quite dead now unfortunately. The biggest upside of FreeBSD Jails using VNETs book is that it covers – as described in the book title – the VNET Jails. Also being released in 2020 its very up to date.

The Taste of Others

Written in 2018 by Manish Jain the book Beginning Modern Unix covers both FreeBSD and Linux operating systems at the same time.

It also beginner friendly and as author writes in Preparing for Part I – “The structure is intended to make things simple for Windows users planning to migrate to FreeBSD/Linux.” I also really liked the author explanation in the Preface about why he choose BSD/FreeBSD and GNU/Linux (like that syntax by the way). You can find part of it below.

To be honest I like that ‘dual’ approach with sections showing how to achieve the same on two different (yet somewhat similar) operating systems. It may be useful especially when writing playbooks for various configuration management software like Ansible or Salt. I know that most of these configuration management systems provide their own ‘general’ integrations like ‘install packages’ on a system and then it uses pkg(8) on FreeBSD and yum(8) on CentOS but sometimes not all integrations are available or they are also sometimes broken or buggy. I have heard several times this already that its safer to rely on your own scripts and ‘raw’ commands instead of the ‘integration modules’ in many cases. Of course your millage may vary.

The Beginning Modern Unix book covers many desktop related topics but it also covers POSIX shell scripting and basic C programming which is very nice.

Another interesting ‘dual’ book is UNIX The Textbook from 2017 by Syed Mansoor Sarwar and Robert M. Koretsky authors.

In its 3rd Edition it covers both Solaris and FreeBSD (in a form of PC-BSD tho) systems. As Solaris by default uses GNOME and PC-BSD (while it existed) used KDE the book also covers a lot about these two desktop environments.

It is illustrated with many useful diagrams of how things work on UNIX system. Below you can check the pipe explanation.

Another good part of the book is that it also covers a lot of POSIX shell scripting and C programming techniques. Even covering things like writing simple servers, inter process communication, threads and a lot more. The book is really huge with almost 1400 pages of useful content. The book leave you at the end of each chapter with QUESTIONS AND PROBLEMS. This approach reminds me of the legendary The C Programming Language written by Brian Kernighan and Dennis Ritchie UNIX fathers where it was also present.

Next one is very well known UNIX and Linux System Administration Handbook by Evi Nemeth and Garth Snyder and Trent R. Hein and Ben Whaley … but it depends which version you get 🙂

The 4th Edition is more conservative and covers Linux/Solaris/HP-UX/AIX systems. This is the description of which systems they have chosen in 2011 and why.

If you try to find FreeBSD there you will fail. They only mentioned it once and added that along with OpenBSD and NetBSD and that BSD systems “(…) enjoy somewhat less support from third-party software vendors.” See for yourself.

Lets now move 7 years forward to 2017 in which the 5th edition of the same UNIX and Linux System Administration Handbook book was released. Authors also changed a little with Dan Mackin being added to current lineup of authors of this book.

Besides modified cover I would risk a stance that its quite entirely different book right now. Its because the 5th Edition covers only Linux and … FreeBSD. No AIX. No Solaris. No HP-UX. Authors also added quite long justification on why they have chosen these operating systems and not the other ones. The time of oldschool UNIX dinosaurs that ruled for decades seem to slowly vanish. Its probably not without reason.

The last HP-UX version 11.31 (also known as 11i v3 variant) was released in 2007. Still uses ‘manual’ packages like in 1995. I like its ‘ecosystem’ tho. What I mean by that is that you can install and setup several HP-UX machines. Setup HP Serviceguard HA cluster on these machines and then make HP Virtual Machines or HP SRP Containers highly available between these hosts as Serviceguard services. I was fortunate enough to be able to see such 6 node cluster in action and it worked really well.

The last AIX version 7.2 was released in 2015 but I do not recall any ground braking features. Also AIX still does not have any modern package management and the most that IBM AIX developers could do was to adopt RPM database to add RPM packages along with the native ones. In 2021 the newer AIX 7.3 release also saw the light of day … but also without any groundbreaking features. In other words the last two AIX system releases fell only like a maintenance releases. While the HP-UX ‘ecosystem’ is ‘connected’ between their products and features its not that easy and simple in the AIX ‘POWER World’. For example its not possible to create similar setup with LPARs or WPARs that their high availability would be controlled by PowerHA cluster software. Even if you would create LPARs with purely virtual devices and storage from the SAN network. The POWER ecosystem offers a feature called Remote Restart for LPARs on HMC but its far from being close to what HP-UX ecosystem offers here.

As for Solaris … I think that Oracle taking over SUN is probably one of the worst things that could happen to Solaris. Oracle could extend and continue the OpenSolaris road started by SUN. It could move and expand Solaris to 21st century. Instead it also went the ‘maintenance’ road along with maximizing the cash outcome of Solaris ‘asset’ with 11.3 in 2015 and 11.4 in 2018 releases. Besides adding PF firewall from OpenBSD and Live Migration feature for Kernel Zones I do not recall any groundbreaking features to be added. Maybe some ZFS development but looking at what OpenZFS is achieving with each release these Oracle developments do not look ‘big’ at all. From what is known Oracle also fired most of SPARC and Solaris developers leaving only small teams to make it running in ‘maintenance mode’ up to 2034 to which Oracle promised to keep Solaris alive instead of moving forward with Solaris 12.0 which was even in alpha or beta state. I agree with authors statement in which they say that “The popularity of UNIX has been waning for some time, and most of the stalwart UNIX distributions (e.g., Solaris, HP-UX, and AIX) are no longer in common use.”

Below is the authors description why they have chosen the FreeBSD and Linux systems for the 5th Edition of their well respected and acknowledged book.

Back to FreeBSD world … and the authors quote why FreeBSD was included in this most recent version of their book – “The open source descendants of BSD are exceptions to this trend and continue to enjoy a cult following, particularly among operating system experts, free software evangelists, and security-minded administrators. In other words, some of the world’s foremost operating system authorities rely on the various BSD distributions. Apple’s macOS has a BSD heritage.” This most recent version is also little shorter with ‘only’ about 1200 pages while earlier edition topped at little over 1300. Keep in mind that newer edition covers Linux and FreeBSD while the older one had to describe and document to systems more.

Last but not least I should also mention the Book of PF – No Nonsense Guide to OpenBSD Firewall book by Peter N.M. Hansteen from 2015 in its most recent 3rd Edition.

While originally targeted at OpenBSD users the FreeBSD users will also be able to get a lot of useful knowledge about PF firewall that FreeBSD uses. Keep in mind that there are some syntax differences between OpenBSD and FreeBSD PF firewalls.

Source Code

Here you will find the books that are little less useful for sysadmins and more useful for developers and programmers. We will start with updated Design and Implementation of FreeBSD 11 Operating System from 2015 in 2nd Edition form. Written by one of the original BSD UNIX and FreeBSD developers Marshall Kirk McKusick along with other two FreeBSD developers George V. Neville-Neil and Robert N.M. Watson.

Its generally more up to date version of the official FreeBSD documentation available as Design and Implementation of 4.4BSD Operating System title from the FreeBSD project documentation page – which one of the authors is also Marshall Kirk McKusick. Is it worth to get it then? Absolutely. A lot have changed and many new technologies have been imported into FreeBSD source tree such as ZFS or DTrace or for example the ULE scheduler.

The final book that I would like to mention here is the FreeBSD Device Drivers book from 2012 written by Joseph Kong.

The book tries to achieve what it title says – to help you first understand and then modify or write your own device drivers. While it covers little older FreeBSD 8 version it is not a problem because the FreeBSD API and ABI change very slow and only when no other way is possible.

Another great tool for programmers and developers in the process of making FreeBSD better is book about DTrace – the dynamic tracing framework. The DTrace – Dynamic Tracing in Oracle Solaris, Mac OS X, and FreeBSD book.

Written by Brendan Gregg and Jim Mauro in 2011 greatly helps to jump into that topic in simple and straightforward way. Besides covering FreeBSD it also does cover Mac OS X, Solaris and even OpenSolaris. That means that it should also be useful for Illumos developers. The book contains a lot of DTrace scripts and examples on how to use that fantastic tool.

Another book that you may find useful in your FreeBSD programmer career is the Designing BSD Rootkits: An Introduction to Kernel Hacking book by Joseph Kong.

Written in 2007 still contains lots of up to date information for the FreeBSD hackers. While the term ‘rootkit’ may be taken ‘negatively’ the author itself describes the book by himself the best way – “Though rootkits have a fairly negative image, they can be used for both good and evil. Designing BSD Rootkits arms you with the knowledge you need to write offensive rootkits, to defend against malicious ones, and to explore the FreeBSD kernel and operating system in the process.” I agree with the author here. To be well protected against something you first need to know how that thing works. The book contains many code examples that you may compile and use and also build upon with your own ideas. Would recommend.

Gone with the Wind

If you know other FreeBSD related books then please let me know.

For those few that did not noticed – the titles of the headers are really great movies 🙂

UPDATE 1 – Practical Internet Server Configuration

Thanks to suggestion of fraxamo user from FreeBSD Forums another book will be added to this article.

I overlooked the Practical Internet Server Configuration from 2021 which was written by Robert La Lau and with Kenneth “Ken” Hess mentioned as technical reviewer.

The book takes FreeBSD 12.0-RELEASE, Debian 9 and CentOS 7 operating systems into account.

Author shows examples of server related tasks on each of these systems separately.

UPDATE 2 – Some Additions to Source Code Section

I forgot one of the most known classics here – the Advanced Programming in UNIX Environment (especially in its most recent 3rd Edition) from 2013 by W. Richard Stevens and Stephen A. Rago.

Nearly all examples have been tested on four modern platforms: Solaris 10, Mac OS X 10.6.8, FreeBSD 8.0 and Linux 3.2.

Another one worth mention is the Beautiful Code: Leading Programmers Explain How They Think from 2007 by Andy Oram and Greg Wilson.

While not the whole book is about FreeBSD – it has dedicated entire chapter called Another Level of Indirection by Diomidis Spinellis.

Aim of the chapter is to show how the flexibility and maintainability of the FreeBSD kernel is promoted by abstracting operations done in common by many drivers and filesystem modules.

UPDATE 3 – Video Courses and List of All Books

After some time I thought that it would be good idea to have just a simple list at the top of that article of all the books mentioned here.

The other thing that I totally forgotten is not about books – but about a videos/courses.

One is named Introduction to the FreeBSD Open Source Operating System from 2015 by Marshall Kirk McKusick itself.

It is available in at least two places:

• archive.org
• oreilly.com

The other one is FreeBSD 13.x – Mastering Jails one.

… and is available here:

• udemy.com

There is also Essentials of UNIX Administration with FreeBSD 12.1 course available here:

• udemy.com

But I did not had a chance to check it as I did not bought it.

… and here is the list of all FreeBSD books in one place.

List of All FreeBSD Books

Official Books

• FreeBSD Handbook
• FreeBSD FAQ
• FreeBSD Architecture Handbook
• FreeBSD Developers Handbook
• FreeBSD Porters Handbook
• Design and Implementation of 4.4BSD Operating System
• Project Model for FreeBSD
• FreeBSD Documentation Project Primer for New Contributor

Third Party Books

• Absolute FreeBSD – Complete Guide to FreeBSD – 3nd Edition (2019)
• FreeBSD Mastery: Storage Essentials (2014)
• FreeBSD Mastery: Specialty Filesystems (2015)
• FreeBSD Mastery: ZFS (2015)
• FreeBSD Mastery: Advanced ZFS (2016)
• FreeBSD Mastery: Jails (2019)
• FreeBSD Jails using VNETs (2020)
• Beginning Modern Unix (2018)
• UNIX the Textbook (2017)
• UNIX and Linux System Administration Handbook – 5th Edition (2017)
• Book of PF – No Nonsense Guide to OpenBSD Firewall – 3rd Edition (2015)
• Design and Implementation of FreeBSD 11 Operating System – 2nd Edition (2015)
• FreeBSD Device Drivers (2012)
• DTrace – Dynamic Tracing in Oracle Solaris/Mac OS X/FreeBSD (2011)
• Designing BSD Rootkits: Introduction to Kernel Hacking (2007)
• Practical Internet Server Configuration (2021)
• Advanced Programming in UNIX Environment – 3rd Edition (2013)
• Beautiful Code: Leading Programmers Explain How They Think (2007)

Feel free to pick the one that suits You best.

EOF

Quare FreeBSD?

I really wanted to make this article short … but I failed miserably. At least I tried to organize it well so one may get back to it after ‘some’ reading because its not a short lecture. I wanted to title it Why FreeBSD? but when you type that into your favorite duck.com search engine there are so many similar articles. I wanted it to have distinguished and unique name so I used Latin word for ‘why‘ which is ‘quare‘.

What FreeBSD can offer you that other operating systems does not? From all of the operating systems I used I find FreeBSD to suck the least. This post is not here to convince you to use or try FreeBSD – this you will have to do by yourself. This article will show you why FreeBSD is valuable or better alternative to other operating systems and is definitely not dying.

This is the Table of Contents for this article.

• Base System
• ZFS Boot Environments
• Rescue
• Audio
• Jails
• Bhyve
• FreeBSD Ports Infrastructure
• Updating/Building from Source
• Storage
• Init System
• Linux Binary Compatibility
• Simplicity
• Evolution Instead Rewriting
• Documentation
• Community
• Closing Thoughts
• External Discussions

Base System

When you install a Linux system its just a bunch of RPM or DEB packages. For example of you install CentOS 7.8 Minimal variant you end up with several hundred RPM packages installed. After a week or month many of these packages will get updates sometimes making this CentOS system unusable or even unbootable (recent GRUB Boothole problem for example). On the contrary FreeBSD comes with a Base System concept. This means that when you install FreeBSD you install a minimal system as a whole. No packages or subsystems to be separately updated. Just whole Base System. That means that /boot /bin /sbin /usr /etc /lib /libexec /rescue directories are untouchable by any packages. When you decide to install packages (or build them using FreeBSD Ports) they will all fall into the /usr/local prefix. That means /usr/local/etc for configuration. The /usr/local/bin and /usr/local/sbin directories for binaries. The /usr/local/lib and /usr/local/libexec for libraries and so on. The FreeBSD Base System kernel modules are kept in the same dir along with the kernel in the /boot/kernel directory. To make things tidy all kernel modules that are provided by packages go into the /boot/modules dir. Everything has its place and its separated.

That is separation between Base System binaries (at /bin /sbin /usr/bin /usr/sbin dirs) and Third Party Packages maintained by pkg(8) and are located at /usr/local/bin and /usr/local/sbin dirs. We all know differences between bin (user) and sbin (root) binaries but in FreeBSD there is also another more UFS related separation. When there was only UFS filesystem in the FreeBSD world the /bin and /sbin binaries were available at boot after the root (/) filesystem was mounted and yet before /usr  filesystem was mounted – this is historical (and still useful in the UFS setups) distinction dating to old UNIX days. In ZFS setups it does not matter as all files are on ZFS pool anyway.

Illustration of the Base System / dir and optional for third party applications /usr/local dir.

The FreeBSD Base System separation also helps with another thing – if any package gets the ‘great’ idea to install new compiler named cc and override the default system compiler … or to add libraries/includes in such a way that makes it super hard to get back into a working system. If some random FreeBSD package would add libc.so to /usr/local/lib dir then you are covered and not prevented from running programs as usual because FreeBSD system binaries are linked to stuff in /usr/lib dir. This is why there is PATH variable on UNIX systems (and FreeBSD as well) to set which directories should be searched for binaries first. On FreeBSD by default its set search Base System binaries dirs first and then Third Party Packages later.

You can update (or not) the Base System separately from the installed packages with freebsd-update(8) command when using RELEASE or by recompiling with make buildworld and make installworld commands when using STABLE/CURRENT systems. When it comes to packages you can update them using the pkg(8) tool or portmaster when building from FreeBSD Ports tree under /usr/ports dir. That means that any packages updates will not touch your FreeBSD Base System at all. For example when you mess up (and I have done that in the beginning of my FreeBSD journey) the compiled ports and packages and you want to start over the only thing you have to do is remove /usr/local and /boot/modules and /var/db/pkg directories. That’s it. You are just reverted to your Base System and can start over. This is just not possible when using Linux system. Even with Gentoo that many concepts are based on FreeBSD ideas does not have Base System feature. This Base System also have additional feature. Because its separated from packages version no one stops you from running oldshool FreeBSD 9.0 from 2012 and install there latest Firefox 80 or LibreOffice 7.0. You can not install latest Firefox on Ubuntu from 2012 …

One may be ‘afraid’ that such Base System independent from installed packages would take more space but nothing far more from the truth. The fresh installed FreeBSD 12.1 system uses less then 1 GB of disk space and takes less then 75 MB of RAM with sshd(8) running. For the comparison fresh CentOS 7.8 install with ‘Minimal’ set chosen takes 1.1 GB of disk space and uses more then 100 MB RAM with sshd(8) running. Such CentOS system is really naked and really needs more packages to be usable while FreeBSD with its Base System is far more capable and powerful and comes along with builtin latest version of LLVM/CLANG compiler suite for example.

More on the Base System topic:

• https://www.over-yonder.net/~fullermd/rants/bsd4linux/03
• https://freebsd.org/handbook/dirstructure.html
• https://man.freebsd.org/hier
• https://man.freebsd.org/freebsd-version
• https://man.freebsd.org/portmaster

ZFS Boot Environments

I have talked about this many times and probably one time too less because Linux world still ignores this bless. Having ZFS Boot Environments its such a game changer that once you realize how powerful it is you will never want to use a system that does not support it. The idea is that you can snapshot a running system at any moment of time and then reboot into that moment (or snapshot) if something happened. Its perfect solution for upgrade or changes to the system. The FreeBSD systems are already well ‘protected’ from problems arising after updating the packages but ZFS Boot Environments takes this to a whole new level.

Like in the movie Groundhog Day (1993) with ZFS Boot Environments you will have limitless chances to get your shit together. Even the Base System updates and changes are protected by it. You can even transport that Boot Environment by using zfs send and zfs recv commands to other system … or propagate it on many systems. You can create Jails containers from it … or install new version of FreeBSD in the new Boot Environment and reboot into it while still having your older ‘production’ system untouched.

More on the ZFS Boot Environments topic:

• https://is.gd/BECTL
• UFS Boot Environments
• UFS Boot Environments for ARM
• Upgrade FreeBSD with ZFS Boot Environments
• https://man.freebsd.org/beadm
• https://man.freebsd.org/bectl

Rescue

When you really mess up to the point that even Base System concept or ZFS Boot Environments feature did not stopped you from killing your FreeBSD installation then there is one more level of rescue … the Rescue subsystem.

You have about 150 statically linked binaries available at your disposal for the rescue mission of that FreeBSD installation. You probably think now that if its so many binaries then it probably takes a lot of space … nothing far more from the truth. Its actually one static binary with hardlinks … and it takes whooping 11 MB of disk space.

FreeBSD # ls -lh /rescue | head -5
total 1118446
-r-xr-xr-x  146 root  wheel    11M 2020.02.19 21:10 [
-r-xr-xr-x  146 root  wheel    11M 2020.02.19 21:10 bectl
-r-xr-xr-x  146 root  wheel    11M 2020.02.19 21:10 bsdlabel
-r-xr-xr-x  146 root  wheel    11M 2020.02.19 21:10 bunzip2

They Rescue subsystem even contains such binaries as bectl(8) for ZFS Boot Environments management or zfs(8) and zpool(8) commands for the ZFS filesystem. Here is complete list of these binaries.

FreeBSD # ls /rescue
[           dd               fsck_ffs      init       mdmfs          ping      rtsol        unlink
bectl       devfs            fsck_msdosfs  ipf        mkdir          ping6     savecore     unlzma
bsdlabel    df               fsck_ufs      iscsictl   mknod          pkill     sed          unxz
bunzip2     dhclient         fsdb          iscsid     more           poweroff  setfacl      unzstd
bzcat       dhclient-script  fsirand       kenv       mount          ps        sh           vi
bzip2       disklabel        gbde          kill       mount_cd9660   pwd       shutdown     whoami
camcontrol  dmesg            geom          kldconfig  mount_msdosfs  rcorder   sleep        xz
cat         dump             getfacl       kldload    mount_nfs      rdump     spppcontrol  xzcat
ccdconfig   dumpfs           glabel        kldstat    mount_nullfs   realpath  stty         zcat
chflags     dumpon           gpart         kldunload  mount_udf      reboot    swapon       zdb
chgrp       echo             groups        ldconfig   mount_unionfs  red       sync         zfs
chio        ed               gunzip        less       mt             rescue    sysctl       zpool
chmod       ex               gzcat         link       mv             restore   tail         zstd
chown       expr             gzip          ln         nc             rm        tar          zstdcat
chroot      fastboot         halt          ls         newfs          rmdir     tcsh         zstdmt
clri        fasthalt         head          lzcat      newfs_msdos    route     tee          
cp          fdisk            hostname      lzma       nextboot       routed    test         
csh         fsck             id            md5        nos-tun        rrestore  tunefs       
date        fsck_4.2bsd      ifconfig      mdconfig   pgrep          rtquery   umount   

More on the Rescue topic:

• https://man.freebsd.org/rescue
• https://docs.hetzner.com/robot/dedicated-server/operating-systems/freebsd-rescue-system/
• http://wiki.euserv.com/index.php/Manual_FreeBSD_Rescue_System/en

Audio

Not many people expect from FreeBSD to shine in that department but it shines a lot here and not from yesterday but from decades. Remember when Linux got rid of the old OSS subsystem with one channel and came up with ‘great’ idea to write ALSA? I remember because I used Linux back then. Disaster is very polite word to describe Linux audio stack back then … and then PulseAudio came and whole Linux audio system got much worse. Back then because of that one OSS channel and many ALSA channels meant that ONLY ONE application with OSS backend could do the sound (for example WINE). But if another application would want to ‘make’ sound using OSS and you already have WINE started then it will be soundless because that one and only OSS channel was already taken. And remember that ALSA was so bad back then that KDE or GNOME made their own sound daemons mixing audio in userspace that were incompatible with each other. That means if you used KDE and GNOME apps back then you could have sound from GNOME apps but not from KDE apps or vice versa. One big fucking audio hell on Linux.

Lets get back to FreeBSD audio then. What FreeBSD offered? A whooping 256 OSS channels mixed live in kernel for low latency. Everything audio related just worked out of the box – and still works today. You could have WINE or KDE/GNOME sound backends attached to their OSS channels and also ALSA apps getting their sound device without a problem. Even when you plugged a 5.1 surround system into FreeBSD it worked out of the box without any configuration and applications were able to use it immediately. That FreeBSD audio supremacy remains today as PulseAudio sound mixing in userspace while generally working incorporates large latency on Linux compared to in kernel FreeBSD mixing with low latency.

Comrade meka suggested that FreeBSD is also the only OS which has virtual_oss that allows mixing/resampling/compressing in user space and allows one to have Bluetooth headphones and USB microphone represented as single sound card.

More on the Audio topic:

• https://freebsd.org/doc/en/books/arch-handbook/oss.html
• https://freebsd.org/handbook/sound-setup.html
• https://man.freebsd.org/sound
• https://papers.freebsd.org/2019/fosdem/mekic-audio_studio/
• https://wiki.freebsd.org/Sound

Jails

The FreeBSD Jails are one of the oldest OS Level Virtualization implementations dating back to 1999. Even the Solaris Zones/Containers came five years later in 2004.

After Docker was introduced in Linux the term OS Level Virtualization became less used to the Containers term and now the FreeBSD Jails along with Solaris Zones/Containers are named 1st generation containers. But that naming nomenclature change does not make FreeBSD Jails less powerful. They are also really brain dead simple to use. You just need a directory – for example /jail/nextcloud – where you will extract the FreeBSD Base System for desired release version – for example base.txz from 12.1-RELEASE and create the Jail config in the /etc/jail.conf file as shown below.

FreeBSD # mkdir -p /jail/nextcloud
FreeBSD # fetch -o - http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/12.1-RELEASE/base.txz | tar --unlink -xpJf - -C /jail/nextcloud
FreeBSD # cat /etc/jail.conf
nextcloud {
  host.hostname = nextcloud.local;
  ip4.addr = 10.0.0.100;
  path = /jail/nextcloud;
}

Now you can start you Jail right away.

FreeBSD # service jail onestart nextcloud
Starting jails: nextcloud.

Voila! Your FreeBSD Jail is already running.

FreeBSD # jls
   JID  IP Address      Hostname                      Path
     1  10.0.0.100      nextcloud.local               /jail/nextcloud

You can of course have a trimmed down version of FreeBSD Base System in the Jail if that is needed. The ZFS filesystem also helps here greatly because with zfs clone only your ‘base’ Jail will take space and only the changes you make to Jails created from it. Thanks to other FreeBSD subsystem – the Linux Binary Compatibility – you can also create a Linux Jail – for example running Devuan or Ubuntu Jail.

The FreeBSD Jails are also very lightweight. You can boot and use about 1000 FreeBSD Jails on a single FreeBSD system with 4 GB RAM.

They are also very easy to debug and troubleshoot comparing even to plain Docker – not to even mention Kubernetes which requires whole team of highly skilled people to maintain.

The FreeBSD Jails may be configured/managed only by the Base System utilities such as jls(8)/jexec(8) but you can also select from many third party Jail management frameworks. From all available ones I would choose BastilleBSD because of their modern approach and many ready to use templates for all needed use cases.

If you want to check all the Jails features feel free to check the FreeBSD Jails Containers article.

More on the Jails topic:

• https://freebsd.org/handbook/jails.html
• https://man.freebsd.org/jail
• https://man.freebsd.org/jail.conf
• https://man.freebsd.org/jls
• https://man.freebsd.org/jexec
• https://bastillebsd.org
• https://web.archive.org/web/20170126214625/http://ivoras.sharanet.org/blog/tree/2009-10-20.the-night-of-1000-jails.html
• https://forums.freebsd.org/threads/setting-up-a-debian-linux-jail-on-freebsd.68434/

Bhyve

FreeBSD – same as Linux or any other respected operating system – comes with its own written virtualization hypervisor. On the FreeBSD land this native solution is called Bhyve (and spelled bee hive).

In the terms of features its similar to KVM/XEN/VMware/VirtualBox.

The only thing that its lacking is the Live Migration feature. Its in the works as the Save/Resume feature is in the testing and will be merged soon.

Besides that – its performance is on par or similar to the mentioned competition. In the end – they all use the same AMD/Intel CPU features that allow virtualization on x86 land.

If You want to check Bhyve hypervisor in all its glory – feel free to check the FreeBSD Bhyve Virtualization article.

Alternatively FreeBSD also supports XEN (as dom0) and VirtualBox hypervisors if Bhyve does not suit your needs.

More on the Bhyve topic:

• https://freebsd.org/handbook/virtualization-host-bhyve.html
• https://wiki.freebsd.org/bhyve
• https://man.freebsd.org/bhyve/8

FreeBSD Ports Infrastructure

This is one of another examples why FreeBSD rocks that much. When you install Ubuntu or CentOS in some version there is chance that you will end up with not latest versions of packages but with versions that were quite up-to-date when this distribution version was released. Its especially visible in the CentOS world (and its upstream enterprise source system from Red Hat) where packages are quite up-to-date when .0 (dot zero) release is published but are VERY outdated when .8 or .9 incarnation of that release is available. Not to even mention that Firefox for example is released every month …

As I said before when describing the FreeBSD Base System the FreeBSD Ports (and packages built from it available through pkg(8) command) are independent. That means that third party software from FreeBSD Ports is almost always up-to-date (or very close to it). You can even check it on the repology.org site for the details. Below you will find a ‘snapshot’ of the repology.org stats from time of writing this article. The ‘online’ table is very long so I copy/pasted just the systems relevant to the article.

One of the other advantages of FreeBSD Ports is that it offers really MASSIVE amount of software counting 40354 ports when writing this article and still rising. Amount of ready to be installed packages are little smaller with more then 32000 available.

I once migrated for a while to OpenSolaris in 2009 on my Dell Latitude D630 laptop because I really liked all the Solaris features (including ZFS and ZFS Boot Environments that were not available on FreeBSD back then) and the OpenSolaris GNOME based desktop was pretty nice back then even with Time Slider feature for ZFS snapshots in the Nautilus file manager. I got working WiFi connection, sound was working, generally everything on my laptop was supported and working with OpenSolaris … but there was no software. Of course ‘large’ projects like GIMP or OpenOffice was available even in the default pkg(8) repository but not much else. There was less then 4000 packages back then on OpenSolaris while about 25000 packages on FreeBSD if I recall correctly.

You can also easily browse available FreeBSD Ports (and its options) on the web by using the https://freshports.org/ page.

The count of FreeBSD Ports is one thing, the features is another. No matter which Linux distribution you are using you will find a software that was compiled and shipped without that needed flag that you desperately need. If you find such software on FreeBSD it ‘hurts’ only for a moment because you can VERY EASILY recompile that software with needed options and replace that ‘default’ package with yours. For example the FreeBSD project is afraid to provide packages of Lame because of existing MP3 patents, so multimedia/ffmpeg package is built without MP3 support (with --disable-libmp3lame flag). That is why I have my own audio/lame and multimedia/ffmpeg packages built with my configure options and that is very easy to achieve. You need to go to the /usr/ports/multimedia/ffmpeg dir type make config and select [x] LAME at the ncurses dialog. Your chosen options will be saved as plain /var/db/ports/multimedia_ffmpeg/options file. If you remove that file (or type make rmconfig) then these custom options will reset to defaults. Then you type make build deinstall install clean and your port with new options is ready and installed as package. Nothing more is needed. You can even lock that package from the pkg(8) upgrades with pkg lock -y ffmpeg command so it will not be modified later but its better to rebuild such packages everytime you do a pkg upgrade procedure because of libraries versions bump and changes. While its very easy and fast to create a script with these commands to make it more automated you can also use other parts of the FreeBSD Ports infrastructure – enter Poudriere (or Synth) – more on that in the next part.

You also do not have to configure each port that way (which could be PITA for large amount of ports) but you may specify your needed (OPTIONS_SET) or unwanted (OPTIONS_UNSET) parameters only once globally using the /etc/make.conf file. You can also specify which default versions of software you want to use, for example Apache 2.2 instead of 2.4 and PHP 7.0 instead of 7.2. You can find all default versions in the /usr/ports/Mk/bsd.default-versions.mk file. Once you setup these options you can build/rebuild or update your packages from FreeBSD Ports by portmaster(8) tool. Like on Gentoo Linux with USE flags. But this is the original. Gentoo took all/most of its ideas from FreeBSD system and its Ports infrastructure.

The Poudriere is a build framework that uses FreeBSD Ports and FreeBSD Jails to build requested packages in clean reproducible way. You can create whole new binary package repository for pkg(8) command to use with it. I mentioned Synth because while Poudriere is often used to produce whole package repository the Synth is usually used just to rebuild several packages that does not fit your needs.

There is one important things about FreeBSD Ports that is often misunderstood by newcomers. What is the difference between the Ports and packages that are fetched and installed by pkg(8) tool? Its quite simple. A package is just a build and installed port. Nothing more or less. When you use the binary packages using pkg(8) command you are using packages that someone (the FreeBSD project in that case) built for you from the FreeBSD Ports in some point in time. While FreeBSD strives to maintain as up-to-date built packages as possible its the nature of FreeBSD Ports that they are always more up-to-date then the built packages. That is why you may build and install a new version of needed packages by yourself using FreeBSD Ports. One may think of such usage when it comes to security holes. When some locally executed commands (like file(1) for example) has a security hole then its not critical for you to update it as fast as possible because that security hole can be harmless for you, but when new version of Firefox fixes very important security hole then its better to update from FreeBSD Ports version faster because waiting 2 days for the package to be built (along with other packages) can be too long.

More on the FreeBSD Ports topic:

• https://freebsd.org/handbook/ports-using.html
• https://freebsd.org/handbook/ports-poudriere.html
• https://man.freebsd.org/ports
• https://man.freebsd.org/build
• https://freshports.org/
• https://repology.org/repositories/statistics/newest

Updating/Building from Source

While the FreeBSD Ports infrastructure is for third party software the FreeBSD Base System (or its parts) also can be easily and convenient build from source. The FreeBSD kernel config is also very small and simple. While Linux kernel config contains thousands of options – 4432 for example in the default CentOS 8.2 install the FreeBSD GENERIC config has about 20 times options less – only 260 options. But that does not saturate the topic. You can start with MINIMAL FreeBSD kernel config which has only 75 options specified.

Linux # grep -c '^CONFIG' /boot/config-$( uname -r )
4432

FreeBSD # grep -c -E '^(device|options)' /usr/src/sys/amd64/conf/GENERIC
260

FreeBSD # grep -c -E '^(device|options)' /usr/src/sys/amd64/conf/MINIMAL
75

… and its not only about smaller amount of options. Can you tell my how many steps (and which ones are required) to rebuild CentOS or Ubuntu for example without Bluetooth support?

On the contrary its very simple (and fast) on the FreeBSD side. While /etc/make.conf file is used to enable/disable Ports options the /etc/src.conf file is used to enable/disable FreeBSD Base System options while building it from source. To build FreeBSD without Bluetooth support just add WITHOUT_BLUETOOTH=yes to the /etc/src.conf file and type these to build it:

FreeBSD # beadm create safe
FreeBSD # cd /usr/src
FreeBSD # make buildworld kernel
FreeBSD # reboot
FreeBSD # cd /usr/src
FreeBSD # etcupdate -p      # // IN THE PAST: mergemaster -p
FreeBSD # make installworld
FreeBSD # etcupdate -B      # // IN THE PAST: mergemaster -iU
FreeBSD # reboot

Voila! You now have FreeBSD without Bluetooth support … and if any of the steps failed or because of your lack of experience/expertise your FreeBSD system does not boot or is broken you can use tools from /rescue to try to fix it (or at least figure out what is broken) and when you do not want to cope with this jest select safe ZFS Boot Environment at the FreeBSD loader(8) to boot to the system before you started building modified version of FreeBSD. Yes, You are bulletproof here. While having 294 WITHOUT_X options and 125 WITH_X options you can really tune FreeBSD Base System to your needs.

FreeBSD # zgrep -c WITHOUT_ /usr/share/man/man5/src.conf.5.gz
294

FreeBSD # zgrep -c WITH_ /usr/share/man/man5/src.conf.5.gz
125

The big downside of updating FreeBSD by source is that you can not use the freebsd-update tools to do it … but nothing stops you from creating your own FreeBSD Update Server so you will be able to use freebsd-update by adding updates using a CURRENT or STABLE system instead of RELEASE. That process is described in the Build Your Own FreeBSD Update Server article of official FreeBSD documentation.

More on the FreeBSD Source Updates/Builds topic:

• https://freebsd.org//handbook/kernelconfig.html
• https://freebsd.org/handbook/makeworld.html
• https://freebsd.org/doc/en/articles/freebsd-update-server/
• https://man.freebsd.org/build
• https://man.freebsd.org/src.conf

Storage

Storage is one of the parts where FreeBSD really shines. Lots of people adore FreeBSD for well integrated ZFS filesystem and its really true. ZFS in FreeBSD has always been first class citizen. Lately OpenZFS 2.0 has been also integrated from the upstream joint FreeBSD and Linux repository. More and more FreeBSD features and solutions are using ZFS features.

Most of these people that like integrated ZFS in FreeBSD do not know about the FreeBSD GEOM modular disk transformation framework which provides various storage related features and utilities like software RAID0/RAID1/RAID10/RAID3/RAID5 configurations or transparent encryption of underlying devices with GELI/GDBE (like LUKS on Linux). It also allows transparent filesystem journaling for ANY filesystem with GJOURNAL (yes also for FAT32 or exFAT) or allows one to export block devices over network with GEOM GATE devices (like NFS for block devices).

FreeBSD also has its own FUSE implementation which allows all these FUSE based filesystems to work natively on FreeBSD. While lots of Linux folks know DRBD probably very few of them knew that FreeBSD comes with its own DRBD like solution called HAST – which does exactly the same thing. While ZFS has a lot features and possibilities FreeBSD still maintains and develops fast and small memory footprint UFS filesystem which today is used either with Soft Updates (SU) or Journaled Soft Updates (SUJ) depending on the use case. For example 10 TB data on UFS filesystem with Journaled Soft Updates (SUJ) takes about 1 minute under fsck(8). These storage solutions are available from FreeBSD Base System alone. The FreeBSD Ports offers much more with distributed filesystems solutions such as CEPH, LeoFS, LizardFS or Minio for Amazon S3 compatible storage.

More on the Storage topic:

• https://is.gd/bsdstg
• https://freebsd.org/handbook/disks.html
• https://freebsd.org/handbook/geom.html
• https://freebsd.org/handbook/zfs.html
• https://freebsd.org/handbook/filesystems.html
• https://man.freebsd.org/geom
• https://man.freebsd.org/geli
• https://man.freebsd.org/fusefs
• https://man.freebsd.org/newfs
• https://man.freebsd.org/hast.conf

Init System

FreeBSD offers really simple yet very powerful init system. It has system wide config under /etc/rc.conf file when you can enable/disable needed services with service_enable=YES and service_enable=NO stanzas. You do not even need to launch vi(1) to add them – just type sysrc service_enable=YES and they are added to the /etc/rc.conf file. There are also default values and services that are enabled and you will find them – along with many comments – in the /etc/defaults/rc.conf file. Each FreeBSD service file has PROVIDE/REQUIRE stanzas which are then used to automatically order the services to start. Services that can be run in parallel are started in parallel to save time. For example its pointless to start sshd(8) daemon without network. To start or stop the service you need to type service sshd start or service sshd stop command. But when a service is not enabled in the /etc/rc.conf file then you need to used add onestart and onestop instead. The Base System separation remains here as FreeBSD Base System services are located at /etc/rc.d directory and third party applications from ports/packages are kept under /usr/local prefix which means /usr/local/etc/rc.d dir.

When using systemd(1) you never know how the services gonna start because it will be different each time. Zero determinism. On FreeBSD you know exactly which services will start when because they are always ordered in the same state according to the PROVIDE/REQUIRE stanzas. FreeBSD also offers tools that will tell you the exact order – rcorder(8) – which can be used for all services, Base System services or third party services separately. There is also service -r command that will show you what was the order at the boot time.

FreeBSD # rcorder /etc/rc.d/* | head
/etc/rc.d/growfs
/etc/rc.d/sysctl
/etc/rc.d/hostid
/etc/rc.d/zvol
/etc/rc.d/dumpon
/etc/rc.d/ddb
/etc/rc.d/geli
/etc/rc.d/gbde
/etc/rc.d/ccd
/etc/rc.d/swap

FreeBSD # rcorder /usr/local/etc/rc.d/* | tail
/usr/local/etc/rc.d/hald
/usr/local/etc/rc.d/git_daemon
/usr/local/etc/rc.d/fscd
/usr/local/etc/rc.d/cupsd
/usr/local/etc/rc.d/cups_browsed
/usr/local/etc/rc.d/clamav-clamd
/usr/local/etc/rc.d/clamav-milter
/usr/local/etc/rc.d/clamav-freshclam
/usr/local/etc/rc.d/avahi-dnsconfd
/usr/local/etc/rc.d/aria2

FreeBSD # rcorder /etc/rc.d/* /usr/local/etc/rc.d/* 2> | grep -C 3 sshd
/etc/rc.d/ubthidhci
/etc/rc.d/syscons
/etc/rc.d/swaplate
/etc/rc.d/sshd
/etc/rc.d/cron
/etc/rc.d/jail
/etc/rc.d/localpkg

Adding new service to FreeBSD is also very easy as template for new service is very small and simple.

#!/bin/sh

. /etc/rc.subr

name=dummy
rcvar=dummy_enable

start_cmd="${name}_start"
stop_cmd=":"

load_rc_config $name
: ${dummy_enable:=no}
: ${dummy_msg="Nothing started."}

dummy_start()
{
	echo "$dummy_msg"
}

run_rc_command "$1"

If its not simple enough for you there is dedicated FreeBSD article about writing them – Practical rc.d Scripting in BSD – available here.

More on the Init System topic:

• https://freebsd.org/handbook/configtuning-rcd.html
• https://freebsd.org/doc/en/articles/rc-scripting/
• https://man.freebsd.org/rc
• https://man.freebsd.org/rc.conf
• https://man.freebsd.org/rc.local

Linux Binary Compatibility

While Linux can not be FreeBSD – the FreeBSD can be Linux – and its not some slow emulation – its implementation of Linux system calls. There was time when enterprises used to work with Linux only applications (not available on FreeBSD by then) using the Linux Binary Compatibility on FreeBSD because it was faster then running them natively on Linux – FreeBSD Used to Generate Spectacular Special Effects – an official FreeBSD Press Release about FreeBSD being used to generate spacial effects to the one of the best movies of all time – The Matrix (1999).

Today the LINUX_COMPAT is also natively fast and allows one to run Linux applications – even Linux games in X11 with hardware acceleration for graphics. Think of it as WINE but for Linux applications. It lives under /compat/linux directory. It even implements Linux /proc virtual filesystem which can be mounted at the /compat/linux/proc dir but its not mandatory. For any software that does not come with source code and works on Linux the Linux Binary Compatibility saves the day. For example the f.lux project. Before I got to know Redshift I used f.lux Linux binary using LINUX_COMPAT to suppress blue spectrum light from my FreeBSD screen. The Linux Binary Compatibility subsystem can also be used to run Linux bases FreeBSD Jails – with Devuan for example.

More on the Linux Binary Compatibility topic:

• https://freebsd.org/handbook/linuxemu.html
• https://freebsd.org/handbook/linuxemu-lbc-install.html
• https://freebsd.org/doc/en/articles/linux-emulation/

Simplicity

FreeBSD is simple but not coarse/ornery. For example as Linux the FreeBSD system also supports the /proc virtual filesystem but on FreeBSD its optional and not used by default while Linux could not live without it. But while Linux has mandatory /proc it also has another virtual filesystem residing under /sys … but why Linux people need two different virtual filesystems with similar purposes? Why they could not create everything under /proc as it already existed. That is big enigma for my sanity.

But /sys is not the end of that madness. Its just a beginning.

What about these?

• securityfs
• devpts
• cgroup
• pstore
• bpf
• configfs
• selinuxfs
• systemd-1
• mqueue
• debugfs
• hugetlbfs

Take a look at the FreeBSD mount(8) output after the default install on ZFS.

FreeBSD # mount
zroot/ROOT/12.1 on / (zfs, local, noatime, nfsv4acls)
devfs on /dev (devfs, local, multilabel)
zroot/tmp on /tmp (zfs, local, noatime, nosuid, nfsv4acls)
zroot/var/mail on /var/mail (zfs, local, nfsv4acls)
zroot/usr/home on /usr/home (zfs, local, noatime, nfsv4acls)
zroot/var/crash on /var/crash (zfs, local, noatime, noexec, nosuid, nfsv4acls)
zroot/var/log on /var/log (zfs, local, noatime, noexec, nosuid, nfsv4acls)
zroot/var/audit on /var/audit (zfs, local, noatime, noexec, nosuid, nfsv4acls)
zroot/var/tmp on /var/tmp (zfs, local, noatime, nosuid, nfsv4acls)
zroot/usr/src on /usr/src (zfs, local, noatime, nfsv4acls)
zroot/usr/ports on /usr/ports (zfs, local, noatime, nosuid, nfsv4acls)

Several ZFS datasets and one virtual devfs filesystem for /dev directory. With install on UFS it would be similar with several UFS partitions mounted instead of ZFS datasets.

Take a look at the CentOS 8.2 installation with just one physical root (/) XFS filesystem.

[root@centos8 ~]# mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,seclabel,size=919388k,nr_inodes=229847,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,seclabel)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,seclabel,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime,seclabel)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,cpuset)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,memory)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,blkio)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,hugetlb)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,net_cls,net_prio)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,cpu,cpuacct)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,freezer)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,perf_event)
cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,rdma)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,pids)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,devices)
configfs on /sys/kernel/config type configfs (rw,relatime)
/dev/sda1 on / type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
selinuxfs on /sys/fs/selinux type selinuxfs (rw,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17309)
mqueue on /dev/mqueue type mqueue (rw,relatime,seclabel)
debugfs on /sys/kernel/debug type debugfs (rw,relatime,seclabel)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,seclabel,pagesize=2M)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=187088k,mode=700)

Fuck me. Its even really hard to just find any REAL filesystem there … fortunately we can ask for only XFS filesystems to display.

[root@centos7 ~]# mount -t xfs
/dev/sda1 on / type xfs (rw,relatime,seclabel,attr2,inode64,noquota)

Lets get on the networking now. Lets assume that you want to make standard enterprise networking setup on a physical server with two interfaces aggregated together into highly available interface bond0 (lagg0 on FreeBSD) and then you want to put VLAN tag and IP address on that VLAN. The CentOS 7.x/8.x installer (Anaconda) will welcome you with this mess.

[root@centos7 ~]# ls -1 /etc/sysconfig/network-scripts/ifcfg-*
ifcfg-Bond_connection_1
ifcfg-eno49
ifcfg-eno49-1
ifcfg-eno50
ifcfg-eno50-1
ifcfg-VLAN_connection_1

[root@centos7 ~]# cat /etc/sysconfig/network-scripts/ifcfg-Bond_connection_1
DEVICE=bond0
BONDING_OPTS="miimon=1 updelay=0 downdelay=0 mode=active-backup"
TYPE=Bond
BONDING_MASTER=yes
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=dhcp
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_PRIVACY=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME="Bond connection 1"
UUID=ca85417f-8852-43bf-96ee-5bd3f0f83648
ONBOOT=yes

[root@centos7 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eno49
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=dhcp
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=eno49
UUID=2f60f50b-38ad-492a-b90a-ba736acf6792
DEVICE=eno49
ONBOOT=no

[root@centos7 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eno49-1
HWADDR=xx:xx:xx:xx:xx:xx
TYPE=Ethernet
NAME=eno49
UUID=342b8494-126d-4f3a-b749-694c8c922aa1
DEVICE=eno49
ONBOOT=yes
MASTER=bond0
SLAVE=yes

[root@centos7 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eno50
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=dhcp
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=eno50
UUID=4fd36e24-1c6d-4a65-a316-7a14e9a92965
DEVICE=eno50
ONBOOT=no

[root@centos7 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eno50-1
HWADDR=xx:xx:xx:xx:xx:xx
TYPE=Ethernet
NAME=eno50
UUID=a429b697-73c2-404d-9379-472cb3c35e06
DEVICE=eno50
ONBOOT=yes
MASTER=bond0
SLAVE=yes

[root@centos7 ~]# cat/etc/sysconfig/network-scripts/ifcfg-VLAN_connection_1
VLAN=yes
TYPE=Vlan
PHYSDEV=ca85417f-8852-43bf-96ee-5bd3f0f83648
VLAN_ID=601
REORDER_HDR=yes
GVRP=no
MVRP=no
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=none
IPADDR=10.20.30.40
PREFIX=24
GATEWAY=10.20.30.1
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_PRIVACY=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME="VLAN connection 1"
UUID=90f7a9bb-1443-4adf-a3eb-86a03b23ecfb
ONBOOT=yes

For the record – I have chosen ‘STATIC’ IPv4 address but installer made these interfaces to use DHCP and that STATIC address. That could be a bug but lets get to the point.

After manual fixing with vi(1) (and hour later) this is how it supposed to look.

[root@centos7 ~]# cat /etc/sysconfig/network
GATEWAY=10.20.30.1
NOZEROCONF=yes

[root@centos7 ~]# ls -1 /etc/sysconfig/network-scripts/ifcfg-*
ifcfg-bond0
ifcfg-bond0.601
ifcfg-eno49
ifcfg-eno50

[root@centos7 ~]# cat /etc/sysconfig/network-scripts/ifcfg-bond0
DEVICE=bond0
BONDING_OPTS="miimon=1 updelay=0 downdelay=0 mode=active-backup"
TYPE=Bond
BONDING_MASTER=yes
BOOTPROTO=none
IPV4_FAILURE_FATAL=no
IPV6INIT=no
ONBOOT=yes

[root@centos7 ~]# cat /etc/sysconfig/network-scripts/ifcfg-bond0.601
VLAN=yes
TYPE=Vlan
VLAN_ID=601
DEVICE=bond0.601
REORDER_HDR=yes
GVRP=no
MVRP=no
BOOTPROTO=none
IPADDR=10.20.30.40
PREFIX=24
IPV4_FAILURE_FATAL=no
IPV6INIT=no
ONBOOT=yes

[root@centos7 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eno49
BOOTPROTO=none
IPV4_FAILURE_FATAL=no
IPV6INIT=no
TYPE=Ethernet
NAME=eno49
DEVICE=eno49
ONBOOT=yes
MASTER=bond0
SLAVE=yes

[root@centos7 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eno50
BOOTPROTO=none
IPV4_FAILURE_FATAL=no
IPV6INIT=no
TYPE=Ethernet
NAME=eno50
DEVICE=eno50
ONBOOT=yes
MASTER=bond0
SLAVE=yes

Better … but still takes A LOT OF SPACE and several files to cover that quite simple setup. Not to mention its level of complication and making that very error prone way. The same configuration on FreeBSD would take just 7 lines within single /etc/rc.conf file as shown below.

FreeBSD # cat /etc/rc.conf
ifconfig_fxp0="up"
ifconfig_fxp1="up"
cloned_interfaces="lagg0"
ifconfig_lagg0="laggproto failover laggport fxp0 laggport fxp1"
vlans_lagg0="601"
ifconfig_lagg0_601="inet 10.20.30.40/24"
defaultrouter="10.20.30.1"

But that are not all the complications. In the past you could restart the network configuration on RHEL/CentOS/Rocky/… with simple service network restart command … not anymore. Now it will not work and you need to type two commands. They are nmcli networking off followed by nmcli networking on command … and to make sure that the second one will be executed and you will not loose the networking connection before executing it You need to type them one after another inside screen(1) or tmux(1) terminal multiplexer. Convenient as fuck 🙂

What about the boot process? FreeBSD boots from root on ZFS partition with just small 512 KB not mountable partition. No separate /boot device is needed. On the other side Linux always needs that separate /boot partition filled with GRUB modules. No matter if its ZFS or LVM. That is why implementation of ZFS Boot Environments is quite complicated on Linux because even if you have root on ZFS on a Linux system there is still unprotected /boot filesystem that can not be snapshoted with ZFS and has to be protected in old classic way which kill the idea of ZFS Boot Environments or Linux.

FreeBSD is really simple and well thought operating system. But also a very underestimated one.

Evolution Instead Rewriting

How many Linux tools or subsystems are abandoned or superseded by new ones? Why the ifconfig(8) command was not updated with new options and instead a new ip(8) command was introduced? Same with netstat(8) being replaced by ss(8). Same with arp(8)/iwconfig/route(8) and many more. What about whole init system? The Linux world has been taken over by systemd(1) whenever you like it or not. Even distributions that have grown their mature init systems like Ubuntu with its Upstart has moved to systemd(1) altogether. The distributions that do not use it are very few and considered a niche today.

In the FreeBSD land on the contrary such things happen only if there is no other way to implement new things. Its the last thing wanted in the FreeBSD. FreeBSD evolves and is developed with stability and backward compatibility in mind. Userland tools are grown and updated with new options instead of rewriting them over and over again. Not to mention how many new bugs are introduced by changing one tool to another.

More on the Evolution Instead Rewriting topic:

• https://duck.com/?q=deprecated+linux+commands
• https://duck.com/?q=deprecated+linux+subsystems+-windows

Documentation

Having system that can do almost anything but not knowing how to do that makes that system pretty useless (or at least pretty PITA to use). FreeBSD offers second to none documentation that is actively maintained and updated. Along with its legendary FreeBSD Handbook and FreeBSD FAQ the FreeBSD project also offers official FreeBSD Articles about various FreeBSD topics. The Man Pages are also very detailed and contain many examples. There is also FreeBSD Wiki page for work in progress documentation and ideas related to FreeBSD development and if you have any problems or questions related to FreeBSD there are official FreeBSD Forums and oldschool Mailing Lists available.

These were only the official project knowledge sources but there are also lots of FreeBSD books. You may also check my dedicated FreeBSD Books article for more in depth look about available FreeBSD books. Here are the best and up-to-date ones.

• Absolute FreeBSD – Complete Guide to FreeBSD – 3nd Edition (2019)
• Beginning Modern Unix (2018)
• Book of PF – 3rd Edition (2015)
• Design and Implementation of FreeBSD 11 Operating System – 2nd Edition (2015)
• FreeBSD Device Drivers (2012)
• FreeBSD Mastery – ZFS (2015)
• FreeBSD Mastery – Advanced ZFS (2016)
• FreeBSD Mastery – Storage Essentials (2014)
• FreeBSD Mastery – Specialty Filesystems (2015)
• FreeBSD Mastery – Jails (2019)

There are also two magazines that are dedicated to BSD and FreeBSD systems. Both are free and cover lots of interesting topics regarding FreeBSD.

• BSD Magazine
• FreeBSD Journal

With all this knowledge and support its really hard not to achieve what you need/want with FreeBSD system.

Community

Last but not least and I would say its even more important then good documentation (which FreeBSD has awesome). People that use FreeBSD do that consciously and are often experienced not only in FreeBSD land but also in topics related to other UNIX systems. Often they took long road of first using the Linux systems before finally setting on the FreeBSD land or they still do Linux administration for a living while resting using far more reasonable and sensible FreeBSD solution. I always find FreeBSD Community helpful and friendly. Always willingly helpful – especially towards newcomers. Even when you try to ‘force’ FreeBSD people to ‘fight’ in unjust/doubtful discussion they will reply with dignity and technical arguments instead of yelling at you.

The FreeBSD project even made several articles and Handbook chapters especially for Linux newcomers – or sometimes called systemd(1) refugees.

• For People New to Both FreeBSD and UNIX®
• FreeBSD Quickstart Guide for Linux® Users
• Explaining BSD

Closing Thoughts

I tried really hard to not make it a Linux rant but some may feel it that way – if so please remember that this was not my intention. FreeBSD like Linux and like any other operating system has its ups and downs. Hope that I showed you most interesting FreeBSD parts. I may add new sections here without a warning in the future 🙂

External Discussions

Discussions and comments from ‘external’ sources are available here:

• Lobsters
• Hacker News
• Reddit

EOF

FreeBSD Cluster with Pacemaker and Corosync

I always missed ‘proper’ cluster software for FreeBSD systems. Recently I got to run several Pacemaker/Corosync based clusters on Linux systems. I thought how to make similar high availability solutions on FreeBSD and I was really shocked when I figured out that both Pacemaker and Corosync tools are available in the FreeBSD Ports and packages as net/pacemaker2 and net/corosync2 respectively.

In this article I will check how well Pacemaker and Corosync cluster works on FreeBSD.

There are many definitions of a cluster. One that I like the most is that a cluster is a system that is still redundant after losing one of its nodes (is still a cluster). This means that 3 nodes is a minimum for a cluster by that definition. The two node clusters are quite problematic because of their biggest exposure to the split brain problem. That is why often in the two node clusters additional devices or systems are added to make sure that this split brain does not happen. For example one can add third node without any resources or services just as a ‘witness’ role. Other way is to add a shared disk resource that will serve the same purpose and often its a raw volume with SCSI-3 Persistent Reservation mechanism used.

Lab Setup

As usual it will be entirely VirtualBox based and it will consist of 3 hosts. To not create 3 same FreeBSD installations I used 12.1-RELEASE virtual machine image available from the FreeBSD Project directly:

• https://download.freebsd.org/ftp/releases/VM-IMAGES/12.1-RELEASE/

There are several formats available – qcow2/raw/vhd/vmdk – but as I will be using VirtualBox I used the VMDK one.

Here is the list of the machines for the GlusterFS cluster:

• 10.0.10.111 node1
• 10.0.10.112 node2
• 10.0.10.113 node3

Each VirtualBox virtual machine for FreeBSD is the default one (as suggested in the VirtualBox wizard) with 512 MB RAM and NAT Network as shown on the image below.

Here is the configuration of the NAT Network on VirtualBox.

Before we will try connect to our FreeBSD machines we need to make the minimal network configuration inside each VM. Each FreeBSD machine will have such minimal /etc/rc.conf file as shown example for node1 host.

root@node1:~ # cat /etc/rc.conf
hostname=node1
ifconfig_em0="inet 10.0.10.111/24 up"
defaultrouter=10.0.10.1
sshd_enable=YES

For the setup purposes we will need to allow root login on these FreeBSD machines with PermitRootLogin yes option in the /etc/ssh/sshd_config file. You will also need to restart the sshd(8) service after the changes.

root@node1:~ # grep PermitRootLogin /etc/ssh/sshd_config
PermitRootLogin yes

root@node1:~ # service sshd restart

By using NAT Network with Port Forwarding the FreeBSD machines will be accessible on the localhost ports. For example the node1 machine will be available on port 2211, the node2 machine will be available on port 2212 and so on. This is shown in the sockstat utility output below.

To connect to such machine from the VirtualBox host system you will need this command:

vboxhost % ssh -l root localhost -p 2211

Packages

As we now have ssh(1) connectivity we need to add needed packages. To make our VMs resolve DNS queries we need to add one last thing. We will also switch to ‘quarterly’ branch of the pkg(8) packages.

root@node1:~ # echo 'nameserver 1.1.1.1' > /etc/resolv.conf
root@node1:~ # sed -i '' s/quarterly/latest/g /etc/pkg/FreeBSD.conf

Remember to repeat these two upper commands on node2 and node3 systems.

Now we will add Pacemaker and Corosync packages.

root@node1:~ # pkg install pacemaker2 corosync2 crmsh

root@node2:~ # pkg install pacemaker2 corosync2 crmsh

root@node3:~ # pkg install pacemaker2 corosync2 crmsh

These are messages both from pacemaker2 and corosync2 that we need to address.

Message from pacemaker2-2.0.4:

--
For correct operation, maximum socket buffer size must be tuned
by performing the following command as root :

# sysctl kern.ipc.maxsockbuf=18874368

To preserve this setting across reboots, append the following
to /etc/sysctl.conf :

kern.ipc.maxsockbuf=18874368

======================================================================

Message from corosync2-2.4.5_1:

--
For correct operation, maximum socket buffer size must be tuned
by performing the following command as root :

# sysctl kern.ipc.maxsockbuf=18874368

To preserve this setting across reboots, append the following
to /etc/sysctl.conf :

kern.ipc.maxsockbuf=18874368

We need to change the kern.ipc.maxsockbuf parameter. Lets do it then.

root@node1:~ # echo 'kern.ipc.maxsockbuf=18874368' >> /etc/sysctl.conf
root@node1:~ # service sysctl restart

root@node2:~ # echo 'kern.ipc.maxsockbuf=18874368' >> /etc/sysctl.conf
root@node2:~ # service sysctl restart

root@node3:~ # echo 'kern.ipc.maxsockbuf=18874368' >> /etc/sysctl.conf
root@node3:~ # service sysctl restart

Lets check what binaries come with these packages.

root@node1:~ # pkg info -l pacemaker2 | grep bin
        /usr/local/sbin/attrd_updater
        /usr/local/sbin/cibadmin
        /usr/local/sbin/crm_attribute
        /usr/local/sbin/crm_diff
        /usr/local/sbin/crm_error
        /usr/local/sbin/crm_failcount
        /usr/local/sbin/crm_master
        /usr/local/sbin/crm_mon
        /usr/local/sbin/crm_node
        /usr/local/sbin/crm_report
        /usr/local/sbin/crm_resource
        /usr/local/sbin/crm_rule
        /usr/local/sbin/crm_shadow
        /usr/local/sbin/crm_simulate
        /usr/local/sbin/crm_standby
        /usr/local/sbin/crm_ticket
        /usr/local/sbin/crm_verify
        /usr/local/sbin/crmadmin
        /usr/local/sbin/fence_legacy
        /usr/local/sbin/iso8601
        /usr/local/sbin/pacemaker-remoted
        /usr/local/sbin/pacemaker_remoted
        /usr/local/sbin/pacemakerd
        /usr/local/sbin/stonith_admin

root@node1:~ # pkg info -l corosync2 | grep bin
        /usr/local/bin/corosync-blackbox
        /usr/local/sbin/corosync
        /usr/local/sbin/corosync-cfgtool
        /usr/local/sbin/corosync-cmapctl
        /usr/local/sbin/corosync-cpgtool
        /usr/local/sbin/corosync-keygen
        /usr/local/sbin/corosync-notifyd
        /usr/local/sbin/corosync-quorumtool

root@node1:~ # pkg info -l crmsh | grep bin
        /usr/local/bin/crm

Cluster Initialization

Now we will initialize our FreeBSD cluster.

First we need to make sure that names of the nodes are DNS resolvable.

root@node1:~ # tail -3 /etc/hosts

10.0.10.111 node1
10.0.10.112 node2
10.0.10.113 node3

root@node2:~ # tail -3 /etc/hosts

10.0.10.111 node1
10.0.10.112 node2
10.0.10.113 node3

root@node3:~ # tail -3 /etc/hosts

10.0.10.111 node1
10.0.10.112 node2
10.0.10.113 node3

Now we will generate the Corosync key.

root@node1:~ # corosync-keygen
Corosync Cluster Engine Authentication key generator.
Gathering 1024 bits for key from /dev/random.
Press keys on your keyboard to generate entropy.
Writing corosync key to /usr/local/etc/corosync/authkey.

root@node1:~ # echo $?
0

root@node1:~ # ls -l /usr/local/etc/corosync/authkey
-r--------  1 root  wheel  128 Sep  2 20:37 /usr/local/etc/corosync/authkey

Now the Corosync configuration file. For sure some examples were provided by the package maintainer.

root@node1:~ # pkg info -l corosync2 | grep example
        /usr/local/etc/corosync/corosync.conf.example
        /usr/local/etc/corosync/corosync.conf.example.udpu

We will take the second one as a base for our config.

root@node1:~ # cp /usr/local/etc/corosync/corosync.conf.example.udpu /usr/local/etc/corosync/corosync.conf

root@node1:~ # vi /usr/local/etc/corosync/corosync.conf
               /* LOTS OF EDITS HERE */

root@node1:~ # cat /usr/local/etc/corosync/corosync.conf

totem {
  version: 2
  crypto_cipher: aes256
  crypto_hash: sha256
  transport: udpu

  interface {
    ringnumber: 0
    bindnetaddr: 10.0.10.0
    mcastport: 5405
    ttl: 1
  }
}

logging {
  fileline: off
  to_logfile: yes
  to_syslog: no
  logfile: /var/log/cluster/corosync.log
  debug: off
  timestamp: on

  logger_subsys {
    subsys: QUORUM
    debug: off
  }
}

nodelist {

  node {
    ring0_addr: 10.0.10.111
    nodeid: 1
  }

  node {
    ring0_addr: 10.0.10.112
    nodeid: 2
  }

  node {
    ring0_addr: 10.0.10.113
    nodeid: 3
  }

}

quorum {
  provider: corosync_votequorum
  expected_votes: 2
}

Now we need to propagate both Corosync key and config across the nodes in the cluster.

We can use some simple tools created exactly for that like net/csync2 cluster synchronization tool for example but plain old net/rsync will serve as well.

root@node1:~ # pkg install -y rsync

root@node1:~ # rsync -av /usr/local/etc/corosync/ node2:/usr/local/etc/corosync/
The authenticity of host 'node2 (10.0.10.112)' can't be established.
ECDSA key fingerprint is SHA256:/ZDmln7GKi6n0kbad73TIrajPjGfQqJJX+ReSf3NMvc.
No matching host key fingerprint found in DNS.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'node2' (ECDSA) to the list of known hosts.
Password for root@node2:
sending incremental file list
./
authkey
corosync.conf
service.d/
uidgid.d/

sent 1,100 bytes  received 69 bytes  259.78 bytes/sec
total size is 4,398  speedup is 3.76

root@node1:~ # rsync -av /usr/local/etc/corosync/ node3:/usr/local/etc/corosync/
The authenticity of host 'node2 (10.0.10.112)' can't be established.
ECDSA key fingerprint is SHA256:/ZDmln7GKi6n0kbad73TIrajPjGfQqJJX+ReSf3NMvc.
No matching host key fingerprint found in DNS.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'node3' (ECDSA) to the list of known hosts.
Password for root@node3:
sending incremental file list
./
authkey
corosync.conf
service.d/
uidgid.d/

sent 1,100 bytes  received 69 bytes  259.78 bytes/sec
total size is 4,398  speedup is 3.76

Now lets check that they are the same.

root@node1:~ # cksum /usr/local/etc/corosync/{authkey,corosync.conf}
2277171666 128 /usr/local/etc/corosync/authkey
1728717329 622 /usr/local/etc/corosync/corosync.conf

root@node2:~ # cksum /usr/local/etc/corosync/{authkey,corosync.conf}
2277171666 128 /usr/local/etc/corosync/authkey
1728717329 622 /usr/local/etc/corosync/corosync.conf

root@node3:~ # cksum /usr/local/etc/corosync/{authkey,corosync.conf}
2277171666 128 /usr/local/etc/corosync/authkey
1728717329 622 /usr/local/etc/corosync/corosync.conf

Same.

We can now add corosync_enable=YES and pacemaker_enable=YES to the /etc/rc.conf file.

root@node1:~ # sysrc corosync_enable=YES
corosync_enable:  -> YES

root@node1:~ # sysrc pacemaker_enable=YES
pacemaker_enable:  -> YES

root@node2:~ # sysrc corosync_enable=YES
corosync_enable:  -> YES

root@node2:~ # sysrc pacemaker_enable=YES
pacemaker_enable:  -> YES

root@node3:~ # sysrc corosync_enable=YES
corosync_enable:  -> YES

root@node3:~ # sysrc pacemaker_enable=YES
pacemaker_enable:  -> YES

Lets start these services then.

root@node1:~ # service corosync start
Starting corosync.
Sep 02 20:55:35 notice  [MAIN  ] Corosync Cluster Engine ('2.4.5'): started and ready to provide service.
Sep 02 20:55:35 info    [MAIN  ] Corosync built-in features:
Sep 02 20:55:35 warning [MAIN  ] interface section bindnetaddr is used together with nodelist. Nodelist one is going to be used.
Sep 02 20:55:35 warning [MAIN  ] Please migrate config file to nodelist.

root@node1:~ # ps aux | grep corosync
root  1695   0.0  7.9 38340 38516  -  S    20:55    0:00.40 /usr/local/sbin/corosync
root  1699   0.0  0.1   524   336  0  R+   20:57    0:00.00 grep corosync

Do the same on the node2 and node3 systems.

The Pacemaker is not yet running so that will fail.

root@node1:~ # crm status
Could not connect to the CIB: Socket is not connected
crm_mon: Error: cluster is not available on this node
ERROR: status: crm_mon (rc=102): 

We will start it now.

root@node1:~ # service pacemaker start
Starting pacemaker.

root@node2:~ # service pacemaker start
Starting pacemaker.

root@node3:~ # service pacemaker start
Starting pacemaker.

You need to give it little time to start because if you will execute crm status command right away you will get 0 nodes configured message as shown below.

root@node1:~ # crm status
Cluster Summary:
  * Stack: unknown
  * Current DC: NONE
  * Last updated: Wed Sep  2 20:58:51 2020
  * Last change:  
  * 0 nodes configured
  * 0 resource instances configured


Full List of Resources:
  * No resources

… but after a while everything is detected and works as desired.

root@node1:~ # crm status
Cluster Summary:
  * Stack: corosync
  * Current DC: node2 (version 2.0.4-2deceaa3ae) - partition with quorum
  * Last updated: Wed Sep  2 21:02:49 2020
  * Last change:  Wed Sep  2 20:59:00 2020 by hacluster via crmd on node2
  * 3 nodes configured
  * 0 resource instances configured

Node List:
  * Online: [ node1 node2 node3 ]

Full List of Resources:
  * No resources

The Pacemaker runs properly.

root@node1:~ # ps aux | grep pacemaker
root      1716   0.0  0.5 10844   2396  -  Is   20:58     0:00.00 daemon: /usr/local/sbin/pacemakerd[1717] (daemon)
root      1717   0.0  5.2 49264  25284  -  S    20:58     0:00.27 /usr/local/sbin/pacemakerd
hacluster 1718   0.0  6.1 48736  29708  -  Ss   20:58     0:00.75 /usr/local/libexec/pacemaker/pacemaker-based
root      1719   0.0  4.5 40628  21984  -  Ss   20:58     0:00.28 /usr/local/libexec/pacemaker/pacemaker-fenced
root      1720   0.0  2.8 25204  13688  -  Ss   20:58     0:00.20 /usr/local/libexec/pacemaker/pacemaker-execd
hacluster 1721   0.0  3.9 38148  19100  -  Ss   20:58     0:00.25 /usr/local/libexec/pacemaker/pacemaker-attrd
hacluster 1722   0.0  2.9 25460  13864  -  Ss   20:58     0:00.17 /usr/local/libexec/pacemaker/pacemaker-schedulerd
hacluster 1723   0.0  5.4 49304  26300  -  Ss   20:58     0:00.41 /usr/local/libexec/pacemaker/pacemaker-controld
root      1889   0.0  0.6 11348   2728  0  S+   21:56     0:00.00 grep pacemaker

We can check how Corosync sees its members.

root@node1:~ # corosync-cmapctl | grep members
runtime.totem.pg.mrp.srp.members.1.config_version (u64) = 0
runtime.totem.pg.mrp.srp.members.1.ip (str) = r(0) ip(10.0.10.111) 
runtime.totem.pg.mrp.srp.members.1.join_count (u32) = 1
runtime.totem.pg.mrp.srp.members.1.status (str) = joined
runtime.totem.pg.mrp.srp.members.2.config_version (u64) = 0
runtime.totem.pg.mrp.srp.members.2.ip (str) = r(0) ip(10.0.10.112) 
runtime.totem.pg.mrp.srp.members.2.join_count (u32) = 1
runtime.totem.pg.mrp.srp.members.2.status (str) = joined
runtime.totem.pg.mrp.srp.members.3.config_version (u64) = 0
runtime.totem.pg.mrp.srp.members.3.ip (str) = r(0) ip(10.0.10.113) 
runtime.totem.pg.mrp.srp.members.3.join_count (u32) = 1
runtime.totem.pg.mrp.srp.members.3.status (str) = joined

… or the quorum information.

root@node1:~ # corosync-quorumtool
Quorum information
------------------
Date:             Wed Sep  2 21:00:38 2020
Quorum provider:  corosync_votequorum
Nodes:            3
Node ID:          1
Ring ID:          1/12
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   3
Highest expected: 3
Total votes:      3
Quorum:           2  
Flags:            Quorate 

Membership information
----------------------
    Nodeid      Votes Name
         1          1 10.0.10.111 (local)
         2          1 10.0.10.112
         3          1 10.0.10.113

The Corosync log file is filled with the following information.

root@node1:~ # cat /var/log/cluster/corosync.log
Sep 02 20:55:35 [1694] node1 corosync notice  [MAIN  ] Corosync Cluster Engine ('2.4.5'): started and ready to provide service.
Sep 02 20:55:35 [1694] node1 corosync info    [MAIN  ] Corosync built-in features:
Sep 02 20:55:35 [1694] node1 corosync warning [MAIN  ] interface section bindnetaddr is used together with nodelist. Nodelist one is going to be used.
Sep 02 20:55:35 [1694] node1 corosync warning [MAIN  ] Please migrate config file to nodelist.
Sep 02 20:55:35 [1694] node1 corosync notice  [TOTEM ] Initializing transport (UDP/IP Unicast).
Sep 02 20:55:35 [1694] node1 corosync notice  [TOTEM ] Initializing transmit/receive security (NSS) crypto: aes256 hash: sha256
Sep 02 20:55:35 [1694] node1 corosync notice  [TOTEM ] The network interface [10.0.10.111] is now up.
Sep 02 20:55:35 [1694] node1 corosync notice  [SERV  ] Service engine loaded: corosync configuration map access [0]
Sep 02 20:55:35 [1694] node1 corosync info    [QB    ] server name: cmap
Sep 02 20:55:35 [1694] node1 corosync notice  [SERV  ] Service engine loaded: corosync configuration service [1]
Sep 02 20:55:35 [1694] node1 corosync info    [QB    ] server name: cfg
Sep 02 20:55:35 [1694] node1 corosync notice  [SERV  ] Service engine loaded: corosync cluster closed process group service v1.01 [2]
Sep 02 20:55:35 [1694] node1 corosync info    [QB    ] server name: cpg
Sep 02 20:55:35 [1694] node1 corosync notice  [SERV  ] Service engine loaded: corosync profile loading service [4]
Sep 02 20:55:35 [1694] node1 corosync notice  [QUORUM] Using quorum provider corosync_votequorum
Sep 02 20:55:35 [1694] node1 corosync notice  [SERV  ] Service engine loaded: corosync vote quorum service v1.0 [5]
Sep 02 20:55:35 [1694] node1 corosync info    [QB    ] server name: votequorum
Sep 02 20:55:35 [1694] node1 corosync notice  [SERV  ] Service engine loaded: corosync cluster quorum service v0.1 [3]
Sep 02 20:55:35 [1694] node1 corosync info    [QB    ] server name: quorum
Sep 02 20:55:35 [1694] node1 corosync notice  [TOTEM ] adding new UDPU member {10.0.10.111}
Sep 02 20:55:35 [1694] node1 corosync notice  [TOTEM ] adding new UDPU member {10.0.10.112}
Sep 02 20:55:35 [1694] node1 corosync notice  [TOTEM ] adding new UDPU member {10.0.10.113}
Sep 02 20:55:35 [1694] node1 corosync notice  [TOTEM ] A new membership (10.0.10.111:4) was formed. Members joined: 1
Sep 02 20:55:35 [1694] node1 corosync warning [CPG   ] downlist left_list: 0 received
Sep 02 20:55:35 [1694] node1 corosync notice  [QUORUM] Members[1]: 1
Sep 02 20:55:35 [1694] node1 corosync notice  [MAIN  ] Completed service synchronization, ready to provide service.
Sep 02 20:58:14 [1694] node1 corosync notice  [TOTEM ] A new membership (10.0.10.111:8) was formed. Members joined: 2
Sep 02 20:58:14 [1694] node1 corosync warning [CPG   ] downlist left_list: 0 received
Sep 02 20:58:14 [1694] node1 corosync warning [CPG   ] downlist left_list: 0 received
Sep 02 20:58:14 [1694] node1 corosync notice  [QUORUM] This node is within the primary component and will provide service.
Sep 02 20:58:14 [1694] node1 corosync notice  [QUORUM] Members[2]: 1 2
Sep 02 20:58:14 [1694] node1 corosync notice  [MAIN  ] Completed service synchronization, ready to provide service.
Sep 02 20:58:19 [1694] node1 corosync notice  [TOTEM ] A new membership (10.0.10.111:12) was formed. Members joined: 3
Sep 02 20:58:19 [1694] node1 corosync warning [CPG   ] downlist left_list: 0 received
Sep 02 20:58:19 [1694] node1 corosync warning [CPG   ] downlist left_list: 0 received
Sep 02 20:58:19 [1694] node1 corosync warning [CPG   ] downlist left_list: 0 received
Sep 02 20:58:19 [1694] node1 corosync notice  [QUORUM] Members[3]: 1 2 3
Sep 02 20:58:19 [1694] node1 corosync notice  [MAIN  ] Completed service synchronization, ready to provide service.

Here is the configuration.

root@node1:~ # crm configure show
node 1: node1
node 2: node2
node 3: node3
property cib-bootstrap-options: \
        have-watchdog=false \
        dc-version=2.0.4-2deceaa3ae \
        cluster-infrastructure=corosync

As we will not be configuring the STONITH mechanism we will disable it.

root@node1:~ # crm configure property stonith-enabled=false

New configuraion with STONITH disabled.

root@node1:~ # crm configure show
node 1: node1
node 2: node2
node 3: node3
property cib-bootstrap-options: \
        have-watchdog=false \
        dc-version=2.0.4-2deceaa3ae \
        cluster-infrastructure=corosync \
        stonith-enabled=false

The STONITH configuration is out of scope of this article but properly configured STONITH looks like that.

First Service

We will now configure our first highly available service – a classic – a floating IP address 🙂

root@node1:~ # crm configure primitive IP ocf:heartbeat:IPaddr2 params ip=10.0.10.200 cidr_netmask="24" op monitor interval="30s"

Lets check how it behaves.

root@node1:~ # crm configure show
node 1: node1
node 2: node2
node 3: node3
primitive IP IPaddr2 \
        params ip=10.0.10.200 cidr_netmask=24 \
        op monitor interval=30s
property cib-bootstrap-options: \
        have-watchdog=false \
        dc-version=2.0.4-2deceaa3ae \
        cluster-infrastructure=corosync \
        stonith-enabled=false

Looks good – lets check the cluster status.

root@node1:~ # crm status
Cluster Summary:
  * Stack: corosync
  * Current DC: node2 (version 2.0.4-2deceaa3ae) - partition with quorum
  * Last updated: Wed Sep  2 22:03:35 2020
  * Last change:  Wed Sep  2 22:02:53 2020 by root via cibadmin on node1
  * 3 nodes configured
  * 1 resource instance configured

Node List:
  * Online: [ node1 node2 node3 ]

Full List of Resources:
  * IP  (ocf::heartbeat:IPaddr2):        Stopped

Failed Resource Actions:
  * IP_monitor_0 on node3 'not installed' (5): call=5, status='complete', exitreason='Setup problem: couldn't find command: ip', last-rc-change='2020-09-02 22:02:53Z', queued=0ms, exec=132ms
  * IP_monitor_0 on node2 'not installed' (5): call=5, status='complete', exitreason='Setup problem: couldn't find command: ip', last-rc-change='2020-09-02 22:02:54Z', queued=0ms, exec=120ms
  * IP_monitor_0 on node1 'not installed' (5): call=5, status='complete', exitreason='Setup problem: couldn't find command: ip', last-rc-change='2020-09-02 22:02:53Z', queued=0ms, exec=110ms

Crap. Linuxism. The ip(8) command is expected to be present in the system. This is FreeBSD and as any UNIX system it comes with ifconfig(8) command instead.

We will have to figure something else. For now we will delete our useless IP service.

root@node1:~ # crm configure delete IP

Status after deletion.

root@node1:~ # crm status
Cluster Summary:
  * Stack: corosync
  * Current DC: node2 (version 2.0.4-2deceaa3ae) - partition with quorum
  * Last updated: Wed Sep  2 22:04:34 2020
  * Last change:  Wed Sep  2 22:04:31 2020 by root via cibadmin on node1
  * 3 nodes configured
  * 0 resource instances configured

Node List:
  * Online: [ node1 node2 node3 ]

Full List of Resources:
  * No resources

Custom Resource

Lets check what resources are available by stock Pacemaker installation.

root@node1:~ # ls -l /usr/local/lib/ocf/resource.d/pacemaker
total 144
-r-xr-xr-x  1 root  wheel   7484 Aug 29 01:22 ClusterMon
-r-xr-xr-x  1 root  wheel   9432 Aug 29 01:22 Dummy
-r-xr-xr-x  1 root  wheel   5256 Aug 29 01:22 HealthCPU
-r-xr-xr-x  1 root  wheel   5342 Aug 29 01:22 HealthIOWait
-r-xr-xr-x  1 root  wheel   9450 Aug 29 01:22 HealthSMART
-r-xr-xr-x  1 root  wheel   6186 Aug 29 01:22 Stateful
-r-xr-xr-x  1 root  wheel  11370 Aug 29 01:22 SysInfo
-r-xr-xr-x  1 root  wheel   5856 Aug 29 01:22 SystemHealth
-r-xr-xr-x  1 root  wheel   7382 Aug 29 01:22 attribute
-r-xr-xr-x  1 root  wheel   7854 Aug 29 01:22 controld
-r-xr-xr-x  1 root  wheel  16134 Aug 29 01:22 ifspeed
-r-xr-xr-x  1 root  wheel  11040 Aug 29 01:22 o2cb
-r-xr-xr-x  1 root  wheel  11696 Aug 29 01:22 ping
-r-xr-xr-x  1 root  wheel   6356 Aug 29 01:22 pingd
-r-xr-xr-x  1 root  wheel   3702 Aug 29 01:22 remote

Not many … we will try to modify the Dummy service into an IP changer on FreeBSD.

root@node1:~ # cp /usr/local/lib/ocf/resource.d/pacemaker/Dummy /usr/local/lib/ocf/resource.d/pacemaker/ifconfig

root@node1:~ # vi /usr/local/lib/ocf/resource.d/pacemaker/ifconfig
               /* LOTS OF TYPING */

Because of the WordPress blogging system limitations I am forced to post this ifconfig resource as an image … but fear not – the text version is also available here – ifconfig.odt – for download.

Also the first version did not went that well …

root@node1:~ # setenv OCF_ROOT /usr/local/lib/ocf
root@node1:~ # ocf-tester -n resourcename /usr/local/lib/ocf/resource.d/pacemaker/ifconfig
Beginning tests for /usr/local/lib/ocf/resource.d/pacemaker/ifconfig...
* rc=3: Your agent has too restrictive permissions: should be 755
-:1: parser error : Start tag expected, '<' not found
usage: /usr/local/lib/ocf/resource.d/pacemaker/ifconfig {start|stop|monitor}
^
* rc=1: Your agent produces meta-data which does not conform to ra-api-1.dtd
* rc=3: Your agent does not support the meta-data action
* rc=3: Your agent does not support the validate-all action
* rc=0: Monitoring a stopped resource should return 7
* rc=0: The initial probe for a stopped resource should return 7 or 5 even if all binaries are missing
* Your agent does not support the notify action (optional)
* Your agent does not support the demote action (optional)
* Your agent does not support the promote action (optional)
* Your agent does not support master/slave (optional)
* rc=0: Monitoring a stopped resource should return 7
* rc=0: Monitoring a stopped resource should return 7
* rc=0: Monitoring a stopped resource should return 7
* Your agent does not support the reload action (optional)
Tests failed: /usr/local/lib/ocf/resource.d/pacemaker/ifconfig failed 9 tests

But after adding 755 mode to it and making several (hundred) changes it become usable.

root@node1:~ # vi /usr/local/lib/ocf/resource.d/pacemaker/ifconfig
             /* LOTS OF NERVOUS TYPING */
root@node1:~ # chmod 755 /usr/local/lib/ocf/resource.d/pacemaker/ifconfig
root@node1:~ # setenv OCF_ROOT /usr/local/lib/ocf
root@node1:~ # ocf-tester -n resourcename /usr/local/lib/ocf/resource.d/pacemaker/ifconfig
Beginning tests for /usr/local/lib/ocf/resource.d/pacemaker/ifconfig...
* Your agent does not support the notify action (optional)
* Your agent does not support the demote action (optional)
* Your agent does not support the promote action (optional)
* Your agent does not support master/slave (optional)
* Your agent does not support the reload action (optional)
/usr/local/lib/ocf/resource.d/pacemaker/ifconfig passed all tests

Looks usable.

The ifconfig resource. Its pretty limited and with hardcoded IP address as for now.

Lets try to add new IP resource to our FreeBSD cluster.

Tests

root@node1:~ # crm configure primitive IP ocf:pacemaker:ifconfig op monitor interval="30"

Added.

Lets see what status command now shows.

root@node1:~ # crm status
Cluster Summary:
  * Stack: corosync
  * Current DC: node2 (version 2.0.4-2deceaa3ae) - partition with quorum
  * Last updated: Wed Sep  2 22:44:52 2020
  * Last change:  Wed Sep  2 22:44:44 2020 by root via cibadmin on node1
  * 3 nodes configured
  * 1 resource instance configured

Node List:
  * Online: [ node1 node2 node3 ]

Full List of Resources:
  * IP  (ocf::pacemaker:ifconfig):       Started node1

Failed Resource Actions:
  * IP_monitor_0 on node3 'not installed' (5): call=24, status='Not installed', exitreason='', last-rc-change='2020-09-02 22:42:52Z', queued=0ms, exec=5ms
  * IP_monitor_0 on node2 'not installed' (5): call=24, status='Not installed', exitreason='', last-rc-change='2020-09-02 22:42:53Z', queued=0ms, exec=2ms

Crap. I forgot to copy this new ifconfig resource to the other nodes. Lets fix that now.

root@node1:~ # rsync -av /usr/local/lib/ocf/resource.d/pacemaker/ node2:/usr/local/lib/ocf/resource.d/pacemaker/
Password for root@node2:
sending incremental file list
./
ifconfig

sent 3,798 bytes  received 38 bytes  1,534.40 bytes/sec
total size is 128,003  speedup is 33.37

root@node1:~ # rsync -av /usr/local/lib/ocf/resource.d/pacemaker/ node3:/usr/local/lib/ocf/resource.d/pacemaker/
Password for root@node3:
sending incremental file list
./
ifconfig

sent 3,798 bytes  received 38 bytes  1,534.40 bytes/sec
total size is 128,003  speedup is 33.37

Lets stop, delete and re-add our precious resource now.

root@node1:~ # crm resource stop IP
root@node1:~ # crm configure delete IP
root@node1:~ # crm configure primitive IP ocf:pacemaker:ifconfig op monitor interval="30"

Fingers crossed.

root@node1:~ # crm status
Cluster Summary:
  * Stack: corosync
  * Current DC: node2 (version 2.0.4-2deceaa3ae) - partition with quorum
  * Last updated: Wed Sep  2 22:45:46 2020
  * Last change:  Wed Sep  2 22:45:43 2020 by root via cibadmin on node1
  * 3 nodes configured
  * 1 resource instance configured

Node List:
  * Online: [ node1 node2 node3 ]

Full List of Resources:
  * IP  (ocf::pacemaker:ifconfig):       Started node1

Looks like running properly.

Lets verify that its really up where it should be.

root@node1:~ # ifconfig em0
em0: flags=8843 metric 0 mtu 1500
        options=81009b
        ether 08:00:27:2a:78:60
        inet 10.0.10.111 netmask 0xffffff00 broadcast 10.0.10.255
        inet 10.0.10.200 netmask 0xffffff00 broadcast 10.0.10.255
        media: Ethernet autoselect (1000baseT )
        status: active
        nd6 options=29

root@node2:~ # ifconfig em0
em0: flags=8843 metric 0 mtu 1500
        options=81009b
        ether 08:00:27:80:50:05
        inet 10.0.10.112 netmask 0xffffff00 broadcast 10.0.10.255
        media: Ethernet autoselect (1000baseT )
        status: active
        nd6 options=29

root@node3:~ # ifconfig em0
em0: flags=8843 metric 0 mtu 1500
        options=81009b
        ether 08:00:27:74:5e:b9
        inet 10.0.10.113 netmask 0xffffff00 broadcast 10.0.10.255
        media: Ethernet autoselect (1000baseT )
        status: active
        nd6 options=29

Seems to be working.

Now lets try to move it to the other node in the cluster.

root@node1:~ # crm resource move IP node3
INFO: Move constraint created for IP to node3

root@node1:~ # crm status
Cluster Summary:
  * Stack: corosync
  * Current DC: node2 (version 2.0.4-2deceaa3ae) - partition with quorum
  * Last updated: Wed Sep  2 22:47:31 2020
  * Last change:  Wed Sep  2 22:47:28 2020 by root via crm_resource on node1
  * 3 nodes configured
  * 1 resource instance configured

Node List:
  * Online: [ node1 node2 node3 ]

Full List of Resources:
  * IP  (ocf::pacemaker:ifconfig):       Started node3

Switched properly to node3 system.

root@node3:~ # ifconfig em0
em0: flags=8843 metric 0 mtu 1500
        options=81009b
        ether 08:00:27:74:5e:b9
        inet 10.0.10.113 netmask 0xffffff00 broadcast 10.0.10.255
        inet 10.0.10.200 netmask 0xffffff00 broadcast 10.0.10.255
        media: Ethernet autoselect (1000baseT )
        status: active
        nd6 options=29

root@node1:~ # ifconfig em0
em0: flags=8843 metric 0 mtu 1500
        options=81009b
        ether 08:00:27:2a:78:60
        inet 10.0.10.111 netmask 0xffffff00 broadcast 10.0.10.255
        media: Ethernet autoselect (1000baseT )
        status: active
        nd6 options=29

Now we will poweroff the node3 system to check it that IP is really highly available.

root@node2:~ # crm status
Cluster Summary:
  * Stack: corosync
  * Current DC: node2 (version 2.0.4-2deceaa3ae) - partition with quorum
  * Last updated: Wed Sep  2 22:49:57 2020
  * Last change:  Wed Sep  2 22:47:29 2020 by root via crm_resource on node1
  * 3 nodes configured
  * 1 resource instance configured

Node List:
  * Online: [ node1 node2 node3 ]

Full List of Resources:
  * IP  (ocf::pacemaker:ifconfig):       Started node3

root@node3:~ # poweroff

root@node2:~ # crm status
Cluster Summary:
  * Stack: corosync
  * Current DC: node2 (version 2.0.4-2deceaa3ae) - partition with quorum
  * Last updated: Wed Sep  2 22:50:16 2020
  * Last change:  Wed Sep  2 22:47:29 2020 by root via crm_resource on node1
  * 3 nodes configured
  * 1 resource instance configured

Node List:
  * Online: [ node1 node2 ]
  * OFFLINE: [ node3 ]

Full List of Resources:
  * IP  (ocf::pacemaker:ifconfig):       Started node1

Seems that failover went well.

The crm command also colors various sections of its output.

Good to know that Pacemaker and Corosync cluster runs well on FreeBSD.

Some work is needed to write the needed resource files but one with some time and determination can surely put FreeBSD into a very capable highly available cluster.

EOF

FreeBSD Enterprise Storage at PBUG

Yesterday I was honored to give a talk about FreeBSD Enterprise Storage at the Polish BSD User Group meeting.

You are invited to download the PDF Slides – https://is.gd/bsdstg – available here.

The PBUG (Polish BSD User Group) meetings are very special. In “The Matrix” movie (which has been rendered on FreeBSD system by the way) – FreeBSD Used to Generate Spectacular Special Effects – details available here – its not possible to describe what the Matrix really is, one has to feel it. Enter it. The same I can tell you about the PBUG meetings. Its kinda like with the “Hangover” movie. What happens in Vegas PBUG meeting stays in Vegas PBUG meeting 🙂

If you will have the possibility and time then join the next Polish BSD User Group meeting. You will not regret it :>

UPDATE 1 – Shorter Unified Version

The original – https://is.gd/bsdstg – presentation is 187 pages long and is suited for live presentation while not the best for later ‘offline’ view.

I have created a unified version – https://is.gd/bsdstguni – with only 42 pages.

EOF

List Block Devices on FreeBSD lsblk(8) Style

When I have to work on Linux systems I usually miss many nice FreeBSD tools such as these for example to name the few:

• sockstat
• gstat
• top -b -o res
• top -m io -o total
• usbconfig
• rcorder
• beadm/bectl
• idprio/rtprio

… but sometimes – which rarely happens – Linux has some very useful tool that is not available on FreeBSD. An example of such tool is lsblk(8) that does one thing and does it quite well – lists block devices and their contents. It has some problems like listing a disk that is entirely used under ZFS pool on which lsblk(8) displays two partitions instead of information about ZFS just being there – but we all know how much in some circles the CDDL licensed ZFS is unloved in that GPL world.

Example lsblk(8) output from Linux system:

$ lsblk
NAME                         MAJ:MIN RM   SIZE RO TYPE   MOUNTPOINT
sr0                           11:0    1  1024M  0 rom
sda                            8:0    0 931.5G  0 disk
|-sda1                         8:1    0   500M  0 part   /boot
`-sda2                         8:2    0   931G  0 part
  |-vg_local-lv_root (dm-0)  253:0    0    50G  0 lvm    /
  |-vg_local-lv_swap (dm-1)  253:1    0  17.7G  0 lvm    [SWAP]
  `-vg_local-lv_home (dm-2)  253:2    0   1.8T  0 lvm    /home
sdc                            8:32   0 232.9G  0 disk
`-sdc1                         8:33   0 232.9G  0 part
  `-md1                        9:1    0 232.9G  0 raid10 /data
sdd                            8:48   0 232.9G  0 disk
`-sdd1                         8:49   0 232.9G  0 part
  `-md1                        9:1    0 232.9G  0 raid10 /data

What FreeBSD offers in this department? The camcontrol(8) and geom(8) commands are available. You can also use gpart(8) command to list partitions. Below you will find output of these commands from my single disk laptop. Please note that because of WordPress limitations I need to change all > < characters to ] [ ones in the commands outputs.

# camcontrol devlist
[Samsung SSD 860 EVO mSATA 1TB RVT41B6Q]  at scbus1 target 0 lun 0 (ada0,pass0)

% geom disk list
Geom name: ada0
Providers:
1. Name: ada0
   Mediasize: 1000204886016 (932G)
   Sectorsize: 512
   Mode: r1w1e2
   descr: Samsung SSD 860 EVO mSATA 1TB
   lunid: 5002538e402b4ddd
   ident: S41PNB0K303632D
   rotationrate: 0
   fwsectors: 63
   fwheads: 1

# gpart show
=>        40  1953525088  ada0  GPT  (932G)
          40      409600     1  efi  (200M)
      409640        1024     2  freebsd-boot  (512K)
      410664         984        - free -  (492K)
      411648  1953112064     3  freebsd-zfs  (931G)
  1953523712        1416        - free -  (708K)

They provide needed information in acceptable manner but only on systems with small amount of disks. What if you would like to display a summary of all system drives contents? This is where lsblk.sh comes handy. While lsblk(8) has many interesting features like --perms/--scsi/--inverse modes I focused to provide only the basic feature – to list the system block devices and their contents. As I have long and pleasing experience with writing shell scripts such as sysutils/beadm or sysutils/automount I though that writing lsblk.sh may be a good idea. I actually ‘open-sourced’ or should I say shared that project/idea in 2016 in this thread lsblk(8) Command for FreeBSD on FreeBSD Forums but lack of time really slowed that ‘side project’ development pace. I finally got back to it to finish it.

The lsblk.sh is generally small and simple shell script which tales less then 400 SLOC.

Here is example output of lsblk.sh command from my single disk laptop.

% lsblk.sh
DEVICE         MAJ:MIN  SIZE TYPE                      LABEL MOUNT
ada0             0:5b  932G GPT                           - -
  ada0p1         0:64  200M efi                    efiboot0 [UNMOUNTED]
  ada0p2         0:65  512K freebsd-boot           gptboot0 -
  [FREE]         -:-   492K -                             - -
  ada0p3         0:66  931G freebsd-zfs                zfs0 [ZFS]
  [FREE]         -:-   708K -                             - -

Same output in graphical window.

Below you will find an example lsblk.sh output from server with two system SSD drives (da0/da1) and two HDD data drives (da2/da3).

# lsblk.sh
DEVICE         MAJ:MIN SIZE TYPE                      LABEL MOUNT
da0              0:be  224G GPT                           - -
  da0p1          0:15a 200M efi                    efiboot0 [UNMOUNTED]
  da0p2          0:15b 512K freebsd-boot           gptboot0 -
  [FREE]         -:-   492K -                             - -
  da0p3          0:15c 2.0G freebsd-swap              swap0 [UNMOUNTED]
  da0p4          0:15d 221G freebsd-zfs                zfs0 [ZFS]
  [FREE]         -:-   580K -                             - -
da1              0:bf  224G GPT                           - -
  da1p1          0:16a 200M efi                    efiboot1 [UNMOUNTED]
  da1p2          0:16b 512K freebsd-boot           gptboot1 -
  [FREE]         -:-   492K -                             - -
  da1p3          0:16c 2.0G freebsd-swap              swap1 [UNMOUNTED]
  da1p4          0:16d 221G freebsd-zfs                zfs1 [ZFS]
  [FREE]         -:-   580K -                             - -
da2              0:c0   11T GPT                           - -
  da2p1          0:16e  11T freebsd-zfs                   - [ZFS]
  [FREE]         -:-   1.0G -                             - -
da3              0:c1   11T GPT                           - -
  da3p1          0:16f  11T freebsd-zfs                   - [ZFS]
  [FREE]         -:-   1.0G -                             - -

Below you will find other examples from other systems I have tested lsblk.sh on.

While lsblk.sh is not the fastest script on Earth (because of all the needed parsing) it does its job quite well. If you would like to install it in your system just type the command below:

# fetch -o /usr/local/bin/lsblk https://raw.githubusercontent.com/vermaden/scripts/master/lsblk.sh
# chmod +x /usr/local/bin/lsblk
# hash -r || rehash
# lsblk

If I got time which other original Linux lsblk(8) subcommand/option/argument is worth adding to the lsblk.sh script? 🙂

Regards.

UPDATE 1 – Added USAGE/HELP Information

Just added some usage information that can be displayed by specifying one of these as argument:

• h
• -h
• --h
• help
• -help
• --help

IMHO writing man page for such simple utility is needless. I think I will create dedicated man page when lsblk.sh tool will grow in size and options to comparable with the Linux lsblk(8) equivalent. Here is how it looks.

# lsblk.sh --help
usage:

  BASIC USAGE INFORMATION
  =======================
  # lsblk.sh [DISK]

example(s):

  LIST ALL BLOCK DEVICES IN SYSTEM
  --------------------------------
  # lsblk.sh
  DEVICE         MAJ:MIN SIZE TYPE                      LABEL MOUNT
  ada0             0:5b  932G GPT                           - -
    ada0p1         0:64  200M efi                    efiboot0 [UNMOUNTED]
    ada0p2         0:65  512K freebsd-boot           gptboot0 -
    [FREE]         -:-   492K -                             - -
    ada0p3         0:66  931G freebsd-zfs                zfs0 [ZFS]

  LIST ONLY da1 BLOCK DEVICE
  --------------------------
  # lsblk.sh da1
  DEVICE         MAJ:MIN SIZE TYPE                      LABEL MOUNT
  da1              0:80  2.0G MBR                           - -
    da1s1          0:80  2.0G freebsd                       - -
      da1s1a       0:81  1.0G freebsd-ufs                root /
      da1s1b       0:82  1.0G freebsd-swap               swap SWAP

hint(s):

  DISPLAY ALL DISKS IN SYSTEM
  ---------------------------
  # sysctl kern.disks
  kern.disks: ada0 da0 da1

Regards.

UPDATE 2 – Code Reorganization and 75% Rewrite

… at least this is what git(1) tries to tell me after commit message.

% git commit (...)
[master 12fd4aa] Rework entire flow. Split code into functions. Add many useful comments. In other words its 2.0 version.
 1 file changed, 494 insertions(+), 505 deletions(-)
 rewrite lsblk.sh (75%)

After several productive hours new incarnation of lsblk.sh is now available.

It has similar SLOC but its now smaller by a quarter … while doing more and with better accuracy. Great example why “less is more.”

% wc scripts/lsblk.sh.OLD
     491    2201   19721 scripts/lsblk.sh.OLD

% wc scripts/lsblk.sh
     494    1871   15472 scripts/lsblk.sh

Things that does not have simple solution are described below.

One of them is ‘double’ label for FAT filesystems. We have both /dev/gpt/efiboot0 label and FAT label is named EFISYS. We have to choose something here. As not all FAT filesystems have label I have chosen the GPT label.

% glabel status | grep ada0p1
  gpt/efiboot0     N/A  ada0p1
msdosfs/EFISYS     N/A  ada0p1

I was also not able to cover FUSE mounts. When you mount – for example – the /dev/da0 device as NTFS (with ntfs-3g) or exFAT (with mount.exfat) there is no visible difference in mount(8) output.

% mount -t fusefs
/dev/fuse on /mnt/ntfs (fusefs)
/dev/fuse on /mnt/exfat (fusefs)

When I mount such filesystem by my daemon (like sysutils/automount) I keep track of what device have been mounted to which directory in the /var/run/automount.state file. Then when I get the detach event for /dev/da0 device I know what to u(n)mount … but when I only have /dev/fuse device its just not possible.

… or maybe YOU know any way of extracting information from /dev/fuse (or generally from FUSE) what device is mounted where?

Now little presentation after update.

Here are various non ZFS filesystems mounted.

% mount -t nozfs
devfs on /dev (devfs, local, multilabel)
linprocfs on /compat/linux/proc (linprocfs, local)
tmpfs on /compat/linux/dev/shm (tmpfs, local)
/dev/label/ASD on /mnt/tmp (msdosfs, local)
/dev/fuse on /mnt/ntfs (fusefs)
/dev/md0s1f on /mnt/ufs.other (ufs, local)
/dev/gpt/OTHER on /mnt/fat.other (msdosfs, local)
/dev/md0s1a on /mnt/ufs (ufs, local)

… and here is how now lsblk.sh displays them.

% lsblk.sh
DEVICE         MAJ:MIN SIZE TYPE                      LABEL MOUNT
ada0             0:56  932G GPT                           - -
  ada0p1         0:64  200M efi                gpt/efiboot0 -
  ada0p2         0:65  512K freebsd-boot       gpt/gptboot0 -
  [FREE]         -:-   492K -                             - -
  ada0p3         0:66  931G freebsd-zfs                   - [ZFS]
  [FREE]         -:-   708K -                             - -
md0              0:28f 1.0G MBR                           - -
  md0s1          0:294 512M freebsd                       - -
    md0s1a       0:29a 100M freebsd-ufs                root /mnt/ufs
    md0s1b       0:29b  32M freebsd-swap         label/swap SWAP
    md0s1e       0:29c  64M freebsd-ufs                   - -
    md0s1f       0:29d 316M freebsd-ufs                   - /mnt/ufs.other
  md0s2          0:296 256M ntfs                          - -
  md0s3          0:297 256M fat32               msdosfs/ONE -
md1              0:2a4 1.0G msdosfs                   LARGE 
md2              0:298 2.0G GPT                           - -
  md2p1          0:29f 2.0G ms-basic-data         gpt/OTHER /mnt/fat.other

I used some file based memory devices for this. Now by default lsblk.sh also displays memory disks contents.

% mdconfig.sh -l
md0     vnode    1024M  /home/vermaden/FILE     
md2     vnode    2048M  /home/vermaden/FILE.GPT 
md1     vnode    1024M  /home/vermaden/FILER    

Here is how it looks in the xterm(1) terminal.

Regards.

UPDATE 3 – Added geli(8) Support

I thought that adding geli(8) support may be useful. The latest lsblk.sh now avoids code duplication for MOUNT and LABEL detection (moved into single unified function). Also added more comments for code readability and some minor fixes … and its again smaller 🙂

% wc lsblk.sh.1.0
     491    2201   19721 lsblk.sh.1.0

% wc lsblk.sh.2.0
     493    1861   15415 lsblk.sh.2.0

% wc lsblk.sh
     488    1820   15332 lsblk.sh

About 40% (according to git commit was changed this time (191 insertions and 196 deletions).

# git commit (...)
[master ec9985a] Add geli(8) support. Avoid code duplication and move MOUNT/LABEL detection into function. More comments. Minor fixes.
 1 file changed, 191 insertions(+), 196 deletions(-)

Also forgot to mention that now lsblk.sh thanks to smart optimizations (like not doing things twice and aggregating grep(1) | awk(1) pipes into single awk(1) queries) runs 3 times faster then the initial version 🙂

New output with geli(8) support below.

Regards.

UPDATE 4 – Added fuse(8) Support

As I wrote in the UPDATE 2 keeping track of what is mounted and where under fuse(8) is very hard as all mounted devices magically become /dev/fuse after mount is done.

After little research I found that this information (what really is mounted where by using fuse(8) interface under FreeBSD) is available after mounting procfs filesystem under /proc. You just need to cat cmdline entry for all PIDs of ntfs-3g. Its not perfect but the information at least is available.

# mount -t procfs proc /proc

# ps ax | grep ntfs-3g
45995  -  Is      0:00.00 ntfs-3g /dev/md1s2 /mnt/ntfs
59607  -  Is      0:00.00 ntfs-3g /dev/md3 /mnt/ntfs.another
83323  -  Is      0:00.00 ntfs-3g /dev/md3 /mnt/ntfs.another

# pgrep ntfs-3g
59607
83323
45995

% pgrep ntfs-3g | while read I; do cat /proc/$I/cmdline; echo; done
ntfs-3g/dev/md3/mnt/ntfs.another
ntfs-3g/dev/md3/mnt/ntfs.another
ntfs-3g/dev/md1s2/mnt/ntfs

This was the code prototype that worked for fuse(8) mountpoints detection.

    if [ -e /proc/0/status ]
    then
      FUSE_MOUNTS=$(
        while read PID
        do
          cat /proc/${PID}/cmdline
          echo
        done << ________EOF
          $( pgrep ntfs-3g )
________EOF
)
      FUSE_MOUNTS=$( echo "${FUSE_MOUNTS}" | sort -u )
      FUSE_MOUNTS=$( echo "${FUSE_MOUNTS}" | sed 's|ntfs-3g||g' )
      FUSE_CHECKS=$( echo "${FUSE_MOUNTS}" | grep /dev/${TARGET}/ )
      if [ "${FUSE_CHECKS}" != "" ]
      then
        MOUNT=$( echo "${FUSE_CHECKS}" | sed "s|/dev/${TARGET}||g" )
      fi
    fi
  fi

… and I have just realized that I found new (better) way of getting that information without mounting /proc filesystem – all you need to do is to display the ntfs-3g processes with their command line arguments, for example like that:

% ps -p $( pgrep ntfs-3g | tr '\n' ',' | sed '$s/.$//' ) -o command | sed 1d
ntfs-3g /dev/md1s2 /mnt/ntfs
ntfs-3g /dev/md3 /mnt/ntfs.another
ntfs-3g /dev/md3 /mnt/ntfs.another

So after I also thought that its only for NTFS (ntfs-3g(8) process) I also added exFAT support by searching for mount.exfat PIDs as well. The fuse(8) mount point detection works now for both NTFS and exFAT filesystems … and code to support it is even shorter.

  # TRY fuse(8) MOUNTS FROM PROCESSES
  if [ "${MOUNT_FOUND}" != "1" ]
  then
    FUSE_PIDS=$( pgrep mount.exfat ntfs-3g | tr '\n' ',' | sed '$s/.$//' )
    FUSE_MOUNTS=$( ps -p "${FUSE_PIDS}" -o command | sed 1d | sort -u )
    MOUNT=$( echo "${FUSE_MOUNTS}" |  grep "/dev/${TARGET} " | awk '{print $3}' )
  fi

I also changed how MAJOR and MINOR numbers are displayed – from HEX to DEC – as it is on Linux. The FreeBSD’s ls(1) from Base System displays these as HEX – for example you will get 0x2af value:

% ls -l /dev/md4
crw-rw----  1 root  operator  0x2af 2019.09.29 05:18 /dev/md4

But do the same with GNU equivalent by using gls(1) from FreeBSD Ports (from sysutils/coreutils package) and it shows MAJOR and MINOR in DEC values. The gls(1) is just ls(1) from the Linux world but as ls(1) name is already ‘taken’ by FreeBSD’s Base System tool the FreeBSD developers/maintainers add ‘g’ letter (for GNU) to distinguish them.

% gls -l /dev/md4
crw-rw---- 1 root 2, 175 2019-09-29 05:18 /dev/md4

… and they are also easier/faster to get with stat(1) tool.

  MAJ=$( stat -f "%Hr" /dev/${DEV} )
  MIN=$( stat -f "%Lr" /dev/${DEV} )

Latest lsblk.sh looks like that now.

… that is why I did not (yet) added lsblk.sh to the FreeBSD Ports. Several new versions with important features span across just two days 🙂

Regards.

UPDATE 5 – Another 69% Rewrite

After messing with gpart(8) more I found that using its -p flag which is a game changer. The difference is that with -p flag it displays names along partitions – its no longer needed to find the PREFIX and ‘create’ partition names.

Default gpart(8) output.

# gpart show md0
=>     63  2097089  md0  MBR  (1.0G)
       63  1048576    1  freebsd  (512M)
  1048639   524288    2  ntfs  (256M)
  1572927   524225    3  fat32  (256M)

Output of gpart(8) with -p flag.

# gpart show -p md0
=>     63  2097089    md0  MBR  (1.0G)
       63  1048576  md0s1  freebsd  (512M)
  1048639   524288  md0s2  ntfs  (256M)
  1572927   524225  md0s3  fat32  (256M)

That discovery implicated a quite large rewrite of lsblk.sh. The git commit estimates this as 69% code rewrite.

# git commit (...)
(...)
 1 file changed, 487 insertions(+), 501 deletions(-)
 rewrite lsblk.sh (69%)

The latest lsblk.sh has now these features:

• Previous bugs fixed.
• Detects exFAT labels.
• Is now 20% faster.
• Has less 10% SLOC.
• Has less 15% of code.
• Handles bsdlabel(8) on entire device properly.
• Handles exFAT on entire device properly.

The difference in code is shown below.

# wc lsblk.sh
     487    1791   13705 lsblk.sh

# wc lsblk.sh.OLD
     544    1931   16170 lsblk.sh.OLD

Latest lsblk.sh looks as usual but I now use ‘-‘ instead of ‘[UNMOUNTED]‘ one.

UPDATE 6 – New Updated and Fixed Version

The lsblk.sh has been updated to 3.4 version – also already updated in the FreeBSD Ports tree – in the sysutils/lsblk port.

The Changelog for this version is below:

• Add sysctl -n kern.disks to disk listing.
• Reset LABEL in __gpart_present function.
• Fix behavior with [bootme] and [bootonce] flags from gpart(8).
• Disable GPTID for label display.
• Add -d|–disks option to list entire disks only.

Keep in mind that lsblk.sh uses diskinfo(8) which – to function properly – need you to be in the operator group. You can add yourself to that group like that:

# pw groupmod operator -m yourself

… or by editing the /etc/group file.

Here is example output.

UPDATE 7 – More Fixes

The lsblk.sh has been updated to 3.5 version – also already updated in the FreeBSD Ports tree – in the sysutils/lsblk port.

The Changelog for this version is below:

• Remove control sequences and colors from output when listing disks.
• Use diskinfo(8) only for md(4) disks as geom(4) does not support them.
• Add new comments and rework some of the older ones.
• Add additional checks for SIZE gathering and printing.
• Properly print exFAT filesystem label when on entire device without partitions.
• Fix display of NTFS-3G mountpoints.
• Check automount(8) /var/run/automount.state for fusefs(5) filesystems.

Presence in the operator group is only needed for size for md(4)disks. All other disks size is gathered using geom(8)command now.

EOF