Tag Archives: zfs

Silent Fanless Dell Wyse 3030 LT FreeBSD Server

Why use quite outdated (released in 2016) Dell Wyse 3030 LT terminal under FreeBSD? There are probably many answers to that question. I recently started to use these for my backup purposes … and to be honest I am really pleased with them.

dell-wyse-3030

In the past I used to make my own boxes based on Mini ITX motherboards such as these:

… and to be honest – I was pretty pleased with their service. The reason that I started to look for alternative and settled on Dell Wyse 3030 LT terminals were Pico ITX power supplies. Both of them failed somewhere between now and 2018. I got replacements from China but then these boxes (without any active cooling) started to overheat. I also tried to replace motherboards with other ones – but still had the same problem. I really wanted to avoid active cooling because it creates a lot of pointless dust. Both inside and outside the case.

My current point of view is that these used Dell Wyse 3030 LT terminals are both cheaper at buy and later at usage costs as they draw a lot less power then the Mini ITX motherboard based ones.

Dell Wyse 3030 LT

The Dell Wyse 3030 LT terminals are not that much powerful – they should be treated as Raspberry Pi boards more then like a casual mini PC. Both because of their power draw and compute power. Here is how such Dell Wyse 3030 LT terminal looks like. I put a 2.5 USB attached hard disk on top of it – its WD Elements with 5 TB capacity. The USB pendrive that is plugged into one of the front USB ports is Lexar S47 32GB USB drive. This is where FreeBSD 13.2-RC6 is installed. I will of course update it when 13.2-RELEASE is ready to rumble.

dell-3030-wd

You may ask why WD Elements and its a good question. The answer is I do not care. I just sort the drives for ‘best price’ order and buy the cheapest one. The other Dell Wyse 3030 LT terminal I got has Seagate Basic Portable 2.5 USB drive instead – also 5 TB in capacity. Works the same well. Cost almost the same. I do not remember which one was better deal but they both were very close.

dell-3030-seagate

Ports

The Dell Wyse 3030 LT terminal has quite a few useful ports that most of them I find useful.

dell-wyse-3030-lt-ports

These are:

  • (1) — 1 x USB 3.0 (front)
  • (2) — 2 x USB 2.0 (front)
  • (3) — 1 x Mini Jack (front)
  • (4) — 1 x LAN 10/100/1000 Realtek RTL8111/8168/8411 (back)
  • (5) — 2 x DisplayPort (back)
  • (6) — 1 x USB 2.0 (back)
  • (*) — 1 x WLAN Intel AC-7265 (internal)

The hardware specs are not that great for today’s standards.

MODEL: Dell Wyse 3030 LT
  CPU: Intel Celeron N2807 2 x 1.58 GHz (2.16 GHz Turbo)
  RAM: 2 GB DDR3L (soldered on motherboard)
 eMMC: 4 GB Flash (soldered on motherboard)
 mPCI: Intel Dual Band Wireless-AC 7265

I use ZFS filesystem for my data and one may say that 2 GB RAM is not enough for ZFS needs … but OpenZFS page recommends 2 GB RAM as a minimum so I do not have to worry here.

Have enough memory: A minimum of 2GB of memory is recommended for ZFS. Additional memory is strongly recommended when the compression and deduplication features are enabled.

The source of that information comes from OpenZFS FAQ page.

In the past I have successfully run a 512 MB box with 2 x 2 TB ZFS mirror for several years. I have had zero crashes/panics/problems with it. Everything just worked like it suppose to. I also run several simple services on the same 512 MB box – like NFS and SAMBA services to export the available data to machines on the local network. The only thing I did back then was to limit the ARC usage up to 128 MB of RAM. Nothing more.

With current 2 GB RAM that I have configured ARC to be 128-256 MB with following settings in the /etc/sysctl.conf file.

# ZFS ARC
# ZFS TUNING
  vfs.zfs.arc.min=134217728
  vfs.zfs.arc.max=268435456

The details about that Intel Celeron N2807 CPU can be found on Intel ARK database.

Here are its specs. The most interesting ones are TDP(W).

N2807

The lscpu(1) shows following information about it.

wyse3030 % lscpu
Architecture:            amd64
Byte Order:              Little Endian
Total CPU(s):            2
Thread(s) per core:      1
Core(s) per socket:      2
Socket(s):               1
Vendor:                  GenuineIntel
CPU family:              6
Model:                   55
Model name:              Intel(R) Celeron(R) CPU  N2807  @ 1.58GHz
Stepping:                8
L1d cache:               24K
L1i cache:               32K
L2 cache:                1024K
Flags:                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 cflsh ds acpi mmx fxsr sse sse2 ss htt tm pbe sse3 pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm sse4_1 sse4_2 movbe popcnt tsc_deadline rdrnd tsc_adjust smep erms fpcsds syscall nx rdtscp lm lahf_lm

The FreeBSD information about it from /var/run/dmesg.boot looks as follows.

CPU: Intel(R) Celeron(R) CPU  N2807  @ 1.58GHz (1583.40-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x30678  Family=0x6  Model=0x37  Stepping=8
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x41d8e3bf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,MOVBE,POPCNT,TSCDLT,RDRAND>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x101<LAHF,Prefetch>
  Structured Extended Features=0x2282<TSCADJ,SMEP,ERMS,NFPUSG>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics

Several C-states and P-states are properly supported on FreeBSD as shown below.

wyse3030 % sysctl dev.cpu.0
dev.cpu.0.temperature: 44.0C
dev.cpu.0.coretemp.throttle_log: 0
dev.cpu.0.coretemp.tjmax: 100.0C
dev.cpu.0.coretemp.resolution: 1
dev.cpu.0.coretemp.delta: 56
dev.cpu.0.cx_method: C1/hlt C2/io
dev.cpu.0.cx_usage_counters: 21488971 0
dev.cpu.0.cx_usage: 100.00% 0.00% last 409us
dev.cpu.0.cx_lowest: C1
dev.cpu.0.cx_supported: C1/1/1 C2/3/104
dev.cpu.0.freq_levels: 2501/45000 2500/45000 2000/33977 1800/29872 1600/25926 1400/22148 1200/18525 1000/15060 800/11752
dev.cpu.0.freq: 1400
dev.cpu.0.%parent: acpi0
dev.cpu.0.%pnpinfo: _HID=none _UID=0 _CID=none
dev.cpu.0.%location: handle=\_PR_.CPU0
dev.cpu.0.%driver: cpu
dev.cpu.0.%desc: ACPI CPU

Keep in mind to load coretemp(4) module at system startup to get all needed information.

The output from sensors.sh is available below.

wyse3030 # sensors.sh

            BATTERY/AC/TIME/FAN/SPEED
------------------------------------
               dev.cpu.0.cx_supported: C1/1/1 C2/2/500 C3/3/1500
                   dev.cpu.0.cx_usage: 26.04% 12.27% 61.68% last 125us
                       dev.cpu.0.freq: 498
                hw.acpi.cpu.cx_lowest: C8
                            powerd(8): running

                  SYSTEM/TEMPERATURES
------------------------------------
                dev.cpu.0.temperature: 37.0C (max: 105.0C)
                dev.cpu.1.temperature: 40.0C (max: 105.0C)
      hw.acpi.thermal.tz0.temperature: 26.9C (max: 90.1C)

                   DISKS/TEMPERATURES
------------------------------------
        smart.da1.temperature_celsius: 37.0C

The biggest drawback is the lack of support for aesni(4) acceleration. Test I had made – with powerd(8) enabled – shows that its able to reach 15-16 MB/s of speed with calculating that encryption on these two cores. Its more then enough for my needs as I rarely go above 10-11 MB/s on the WiFi transfers.

If you are interested how the internals of Dell Wyse 3030 LT motherboard looks like – take a journey to David Parkinson page at Dell Wyse 3030 LT location. You will also find a lot information about other Thin Client software – often also called Terminals.

I did not bothered to unscrew mine knowing what David Parkinson shows on his site.

WiFi

As the Dell Wyse 3030 LT has the Intel Dual Band Wireless-AC 7265 WiFi card from 2014 – almost a decade old WiFi card – there is a big chance that FreeBSD will flawlessly support it and at least allow 802.11n protocol and speed – as its 9 years old.

Here are some specs from Intel on that WiFi card.

AC7265

… and here is the neat part 😀 – it does not work on FreeBSD – its not able to connect to WiFi access point – what were you thinking 🙂

This is what I tried at both 2.4GHz and 5.0GHz WiFi networks that are WPA2 protected.

wyse3030 # grep iwm /var/run/dmesg.boot 
iwm0: <Intel(R) Dual Band Wireless AC 7265> mem 0x88500000-0x88501fff at device 0.0 on pci2
iwm0: hw rev 0x210, fw ver 22.361476.0, address d0:57:7b:13:41:4c

wyse3030 # pciconf -lv iwm0
iwm0@pci0:2:0:0: class=0x028000 rev=0x59 hdr=0x00 vendor=0x8086 device=0x095a subvendor=0x8086 subdevice=0x5410
vendor = 'Intel Corporation'
device = 'Wireless 7265'
class = network

wyse3030 #
sysctl -n net.wlan.devices iwm0
wyse3030 #
ifconfig wlan0 create wlandev $( !! )
wyse3030 #
ifconfig wlan0 up
wyse3030 #
ifconfig wlan0 scan SSID/MESH ID BSSID CHAN RATE S:N INT CAPS wifinet d8:07:b6:aa:bb:00 7 54M -63:-96 100 EP APCHANREP WPA RSN BSSLOAD HTCAP VHTCAP VHTOPMODE
wifinet d8:07:b6:aa:bb:02 40 54M -80:-96 100 EP VHTPWRENV APCHANREP WPA RSN HTCAP VHTCAP

These are my WiFi networks that I will try to use:

  • WiFi 2.4GHz – wifinetd8:07:b6:aa:bb:00
  • WiFi 5.0GHz – wifinetd8:07:b6:aa:bb:02

Results below.

WiFi 2.4GHz

wyse3030 # wpa_supplicant -i wlan0 -c /etc/wpa_supplicant.conf
Successfully initialized wpa_supplicant
ioctl[SIOCS80211, op=20, val=0, arg_len=7]: Invalid argument
ioctl[SIOCS80211, op=20, val=0, arg_len=7]: Invalid argument
ioctl[SIOCS80211, op=103, val=0, arg_len=128]: Operation now in progress
wlan0: CTRL-EVENT-SCAN-FAILED ret=-1 retry=1
wlan0: Trying to associate with d8:07:b6:aa:bb:00 (SSID='wifinet' freq=2442 MHz)
wlan0: Authentication with d8:07:b6:aa:bb:00 timed out.
wlan0: CTRL-EVENT-DISCONNECTED bssid=d8:07:b6:aa:bb:00 reason=3 locally_generated=1
BSSID d8:07:b6:aa:bb:00 ignore list count incremented to 2, ignoring for 10 seconds
wlan0: CTRL-EVENT-DSCP-POLICY clear_all
wlan0: Trying to associate with d8:07:b6:aa:bb:00 (SSID='wifinet' freq=2442 MHz)
wlan0: Authentication with d8:07:b6:aa:bb:00 timed out.
wlan0: CTRL-EVENT-DISCONNECTED bssid=d8:07:b6:aa:bb:00 reason=3 locally_generated=1
BSSID d8:07:b6:aa:bb:00 ignore list count incremented to 2, ignoring for 10 seconds
wlan0: CTRL-EVENT-DSCP-POLICY clear_all
wlan0: Trying to associate with d8:07:b6:aa:bb:00 (SSID='wifinet' freq=2442 MHz)
wlan0: Authentication with d8:07:b6:aa:bb:00 timed out.
wlan0: CTRL-EVENT-DISCONNECTED bssid=d8:07:b6:aa:bb:00 reason=3 locally_generated=1
BSSID d8:07:b6:aa:bb:00 ignore list count incremented to 2, ignoring for 10 seconds
wlan0: CTRL-EVENT-DSCP-POLICY clear_all
wlan0: Trying to associate with d8:07:b6:aa:bb:00 (SSID='wifinet' freq=2442 MHz)
^C

WiFi 5.0GHz

wyse3030 # wpa_supplicant -i wlan0 -c /etc/wpa_supplicant.conf
Successfully initialized wpa_supplicant
ioctl[SIOCS80211, op=20, val=0, arg_len=7]: Invalid argument
ioctl[SIOCS80211, op=20, val=0, arg_len=7]: Invalid argument
wlan0: Trying to associate with d8:07:b6:aa:bb:02 (SSID='wifinet' freq=5200 MHz)
wlan0: Authentication with d8:07:b6:aa:bb:02 timed out.
wlan0: CTRL-EVENT-DISCONNECTED bssid=d8:07:b6:aa:bb:02 reason=3 locally_generated=1
BSSID d8:07:b6:aa:bb:02 ignore list count incremented to 2, ignoring for 10 seconds
wlan0: CTRL-EVENT-DSCP-POLICY clear_all
wlan0: Trying to associate with d8:07:b6:aa:bb:02 (SSID='wifinet' freq=5200 MHz)
wlan0: Authentication with d8:07:b6:aa:bb:02 timed out.
wlan0: CTRL-EVENT-DISCONNECTED bssid=d8:07:b6:aa:bb:02 reason=3 locally_generated=1
BSSID d8:07:b6:aa:bb:02 ignore list count incremented to 2, ignoring for 10 seconds
wlan0: CTRL-EVENT-DSCP-POLICY clear_all
wlan0: Trying to associate with d8:07:b6:aa:bb:02 (SSID='wifinet' freq=5200 MHz)
wlan0: Authentication with d8:07:b6:aa:bb:02 timed out.
wlan0: CTRL-EVENT-DISCONNECTED bssid=d8:07:b6:aa:bb:02 reason=3 locally_generated=1
BSSID d8:07:b6:aa:bb:02 ignore list count incremented to 2, ignoring for 10 seconds
wlan0: CTRL-EVENT-DSCP-POLICY clear_all
wlan0: Trying to associate with d8:07:b6:aa:bb:02 (SSID='wifinet' freq=5200 MHz)
wlan0: Authentication with d8:07:b6:aa:bb:02 timed out.
wlan0: CTRL-EVENT-DISCONNECTED bssid=d8:07:b6:aa:bb:02 reason=3 locally_generated=1
BSSID d8:07:b6:aa:bb:02 ignore list count incremented to 2, ignoring for 10 seconds
wlan0: CTRL-EVENT-SSID-TEMP-DISABLED id=0 ssid="wifinet" auth_failures=1 duration=10 reason=CONN_FAILED
wlan0: CTRL-EVENT-DSCP-POLICY clear_all
wlan0: CTRL-EVENT-SSID-REENABLED id=0 ssid="wifinet"
wlan0: Trying to associate with d8:07:b6:aa:bb:02 (SSID='wifinet' freq=5200 MHz)
wlan0: CTRL-EVENT-DISCONNECTED bssid=d8:07:b6:aa:bb:02 reason=3 locally_generated=1
wlan0: CTRL-EVENT-DSCP-POLICY clear_all
wlan0: CTRL-EVENT-DSCP-POLICY clear_all
wlan0: CTRL-EVENT-TERMINATING 
^C

… and to be fair – I have used latest and greatest FreeBSD 13.2-RC6 (the 13.2-RELEASE is probably available now). I even installed the comms/iwmbt-firmware package hoping it would solve anything – it does not change anything.

For the record – this is how properly working WiFi connection looks like on FreeBSD.

# ifconfig wlan0 up 
# wpa_supplicant -i wlan0 -c /etc/wpa_supplicant.conf
Successfully initialized wpa_supplicant
ioctl[SIOCS80211, op=20, val=0, arg_len=7]: Invalid argument
ioctl[SIOCS80211, op=20, val=0, arg_len=7]: Invalid argument
wlan0: Trying to associate with d8:07:b6:aa:bb:00 (SSID='wifinet' freq=2442 MHz)
wlan0: Associated with d8:07:b6:aa:bb:00
wlan0: WPA: Key negotiation completed with d8:07:b6:aa:bb:00 [PTK=CCMP GTK=CCMP]
wlan0: CTRL-EVENT-CONNECTED - Connection to d8:07:b6:aa:bb:00 completed [id=43 id_str=]

At this point (after you get the CTRL-EVENT-CONNECTED message) you hit [CTRL]+[Z] then type bg at the prompt to keep wpa_supplicant(8) process in the background – and then dhclient wlan0 to get the IP address, default gateway and DNS servers.

The best FreeBSD can get on that box when it comes to WiFi side is Realtek RTL8188CUS tiny USB dongle … but while its 802.11n capable – the FreeBSD is only able to support 802.11g speeds on it. Details here – Realtek RTL8188CUS – USB 802.11n WiFi Review – in one of my older articles.

You now probably understand why I have these two WiFi antennas detached 🙂 – but for the record – I have tested the WiFi connection with both antennas attached to that Intel AC 7265 card.

LAN

While WiFi on FreeBSD sucks greatly the builtin LAN – Realtek RTL8111/8168/8411 – works quite well with re(4) driver … but it needs one spacial quirk to work 🙂

All network connections need to be restarted at the end of the boot process to work – if you do not do that – you just do not have network connectivity at all – I was not able to track why its fucked up like that – so I went with the idea to use /etc/rc.local scripts as its already there and its really late in the boot process. Here are mine /etc/rc.local script contents that seem to workaround that fuckup well.

wyse3030 % cat /etc/rc.local
#! /bin/sh

# RESTART NETWORKING
  (
  service netif   restart 1> /dev/null 2> /dev/null
  service routing restart 1> /dev/null 2> /dev/null
  ) &

To not waste time I put both netif and routing services in the background for startup.

DisplayPort

The DisplayPort sockets just work – when attached to the 1080p monitor they just deliver. I tried only in the vt(4) console mode. I did not tried X11 mode – but I suspect that the CPU is so old that it will work flawlessly with i915kms.ko driver from graphics/drm-fbsd13-kmod package.

When it comes to early booting – set your native screen resolution in the /boot/loader.conf file as specified below.

# CONSOLE RESOLUTION # --------------------------------------------------------
  kern.vt.fb.default.mode=1920x1080    # FORCE FHD
  efi_max_resolution=1920x1080         # FORCE FHD

Also – if you (like me) like to have small/smaller font size in the terminal – this will also be helpful in the /boot/loader.conf file.

  screen.font=6x12          # USE SMALLEST FONT NO MATTER THE RESOLUTION

USB

The USB ports work as desired. Both USB 2.0 and USB 3.0 ports work without any issues.

I currently use Lexar S47 32 GB USB tiny dongle foe FreeBSD system. I did not yet tested the builtin eMMC 4 GB flash storage space – will probably add some UPDATE about it in the future.

Data

Lets get back to the main responsibility of that box – backups. In the past my main NAS system used to have two 5 TB disks in ZFS mirror – but as I have another offsite copy and another offline copy and I also have my main copy always with me – I now find it pointless to have this single NAS mirrored. Besides – its more then 3-2-1 backup rule anyway …

As this box has 4 USB ports one can of course even create raidz ZFS pool with 3 drives for data and 1 for parity. Nothing stops you from doing that. You will get 15 TB of usable storage that way … not sure how log would it take to rebuild through. As these are SMR drives you may consider raidz2 or mirror instead. I also have some tips for ZFS on SMR Drives here.

Costs

This is the neat part … and I am not joking here. My earlier setups cost about $110 to $120 only for the case/motherboard/PSUs/RAM/etc. You needed to add disk costs to that on top. But not with this one. I got several of these Dell Wyse 3030 LT terminals for less then $18 each … and it was not a one time offer – they just cost that much now.

The disks (5 TB 2.5 USB drives) that I use also got cheaper in time and its relatively easy to get a used one for about $80. Does not matter if its Seagate or Western Digital or whatever. They are very similar cheap 5400 rpm SMR drives.

This make my new backup solution cost about $100 for a 5 TB space of storage.

Configuration

This is the main FreeBSD config /etc/rc.conf file.

wyse3030 % cat /etc/rc.conf
# BOOT/SILENCE # -------------------------------------------------------------
  rc_startmsgs=NO
  rc_info=NO

# NETWORK # ------------------------------------------------------------------
  hostname=wyse3030
  ifconfig_re0="inet 10.0.0.2/24"
  defaultrouter="10.0.0.1"
  defaultroute_delay=3
  defaultroute_carrier_delay=3
  gateway_enable=YES

# MODULES/COMMON/BASE # ------------------------------------------------------
  kld_list="${kld_list} fusefs coretemp sem cpuctl ichsmb"

# POWER # --------------------------------------------------------------------
  performance_cx_lowest=Cmax
  economy_cx_lowest=Cmax
  powerd_enable=YES
  powerd_flags="-n adaptive -a hiadaptive -b adaptive -m 100 -M 2000"

# DAEMONS # ------------------------------------------------------------------
  syslogd_flags="-ss"
  sshd_enable=YES
  zfs_enable=YES
  sendmail_enable=NO
  sendmail_submit_enable=YES
  sendmail_outbound_enable=NO
  sendmail_msp_queue_enable=NO

# OTHER # --------------------------------------------------------------------
  keyrate=fast
  virecover_enable=NO
  update_motd=NO
  hostid_enable=NO
  entropy_file=NO
  savecore_enable=NO
  clear_tmp_enable=YES
  dumpdev=AUTO

Options that can be changed anytime at /etc/sysctl.conf file below.

wyse3030 % cat /etc/sysctl.conf
# SECURITY # ------------------------------------------------------------------
  security.bsd.see_jail_proc=0
  security.bsd.unprivileged_proc_debug=0

# SECURITY/RANDOM PID # -------------------------------------------------------
  kern.randompid=1

# ANNOYING THINGS # -----------------------------------------------------------
  vfs.usermount=1
  kern.coredump=0
  hw.syscons.bell=0
  kern.vt.enable_bell=0

# ZFS ASHIFT 4k BLOCK SIZE # --------------------------------------------------
  vfs.zfs.min_auto_ashift=12

# ZFS DELETE FUCKUP TRIM # ----------------------------------------------------
  vfs.zfs.vdev.trim_max_active=1

# ZFS ARC 128MB FORCE # -------------------------------------------------------
  vfs.zfs.arc.min=134217728
  vfs.zfs.arc.max=268435456

# JAILS/ALLOW UPGRADES IN JAILS # ---------------------------------------------
  security.jail.chflags_allowed=1

# JAILS/ALLOW RAW SOCKETS # ---------------------------------------------------
  security.jail.allow_raw_sockets=1

# ALLOW idprio(8) USE BY REGULAR USER # ---------------------------------------
  security.bsd.unprivileged_idprio=1

… and last but not least (actually first :> to be read) the /boot/loader.conf file.

wyse3030 % cat /boot/loader.conf
# CONSOLE COMMON # ------------------------------------------------------------
  autoboot_delay=2          # USE '-1' => NO WAIT | USE 'NO' => INFINITE WAIT
  loader_logo=none          # LOGO POSIBILITES: fbsdbw beastiebw beastie none
  loader_menu_frame=none    # REMOVE FRAMES FROM FreeBSD BOOT MENU
  screen.font=6x12          # USE SMALLEST FONT NO MATTER THE RESOLUTION

# CONSOLE RESOLUTION # --------------------------------------------------------
  kern.vt.fb.default.mode=1920x1080    # FORCE FHD
  efi_max_resolution=1920x1080         # FORCE FHD

# MODULES # -------------------------------------------------------------------
  geom_eli_load=YES
  cryptodev_load=YES
  zfs_load=YES

# DISABLE /dev/diskid/* AND /dev/gptid/* ENTRIES FOR DISKS # ------------------
  kern.geom.label.disk_ident.enable=0
  kern.geom.label.gptid.enable=0

# COLORS # --------------------------------------------------------------------
  kern.vt.color.0.rgb="#000000"
  kern.vt.color.1.rgb="#dc322f"
  kern.vt.color.2.rgb="#859900"
  kern.vt.color.3.rgb="#b58900"
  kern.vt.color.4.rgb="#268bd2"
  kern.vt.color.5.rgb="#ec0048"
  kern.vt.color.6.rgb="#2aa198"
  kern.vt.color.7.rgb="#94a3a5"
  kern.vt.color.8.rgb="#586e75"
  kern.vt.color.9.rgb="#cb4b16"
  kern.vt.color.10.rgb="#859900"
  kern.vt.color.11.rgb="#b58900"
  kern.vt.color.12.rgb="#268bd2"
  kern.vt.color.13.rgb="#d33682"
  kern.vt.color.14.rgb="#2aa198"
  kern.vt.color.15.rgb="#6c71c4"

# RACCT/RCTL RESOURCE LIMITS # ------------------------------------------------
  kern.racct.enable=1

# ZFS TUNING # ----------------------------------------------------------------
  vfs.zfs.prefetch_disable=1

# POWER MANAGEMENT POWER OFF DEVICES WITHOUT ATTACHED DRIVER # ----------------
  hw.pci.do_power_nodriver=3

# POWER MANAGEMENT FOR EVERY USED AHCI CHANNEL (ahcich 0-7) # -----------------
  hint.ahcich.0.pm_level=5
  hint.ahcich.1.pm_level=5
  hint.ahcich.2.pm_level=5
  hint.ahcich.3.pm_level=5
  hint.ahcich.4.pm_level=5
  hint.ahcich.5.pm_level=5
  hint.ahcich.6.pm_level=5
  hint.ahcich.7.pm_level=5

# DISABLE DESTRUCTIVE DTrace # ------------------------------------------------
  security.bsd.allow_destructive_dtrace=0

There is also this 5 TB disk drive configuration to be mentioned.

wyse3030 # gpart destroy -F da0
wyse3030 # dd if=/dev/zero of=/dev/da0 bs=1m status=progress count=10
wyse3030 # gpart create -s GPT da0
wyse3030 # gpart add -t freebsd-zfs -a 4k da0
wyse3030 # geli init -s 4096 da0p1
wyse3030 # geli attach da0p1

Why I use password based encryption and not based on key? If someone would steal that device then with attached key on – for example – USB drive – all the data would be accessible – and all encryption and effort would be lost. With password based encryption whoever steals the devices gets some nice /dev/random input from my disk because its encrypted.

In the end these boxes do not hang by themselves and when there is power outage – its not that common like twice a week. Its more like once a quarter – I do not mind typing my password once in a quarter to make my data safe from people that should not be allowed to access it.

ZFS Pool Configuration

I did not used anything spectacular for ZFS configuration – just disabled atime and enabled compression that would save some space. I also increased the recordsize to 1M so the compressratio would be a little better … but nothing more to be honest.

wyse3030 # zpool history wd
History for 'wd':
2023-04-05.14:46:01 zpool create wd /dev/da0p1.eli
2023-04-05.14:46:07 zfs set compression=zstd wd
2023-04-05.14:46:12 zfs set atime=off wd
2023-04-05.14:46:15 zfs set recordsize=1m wd
2023-04-05.14:51:46 zfs create wd/data
2023-04-05.14:51:50 zfs set mountpoint=none wd
2023-04-05.14:52:00 zfs set mountpoint=/data wd/data

As you probably guessed the wd name is for the Western Digital 5 TB drive. If you are curious (and you probably are) the second ZFS pool is called seagate 🙂

Last time I used the LZ4 compression for the ZFS pool. Not I use the ZSTD compression and the compressratio improved from 1.04 to 1.07 – almost twice more efficient.

wyse3030 % zfs get compressratio
NAME                                      PROPERTY       VALUE  SOURCE
wd                                        compressratio  1.07x  -
wd/data                                   compressratio  1.07x  -

One may think that 7% of more data space is almost ‘non-existent’ – think again. When it comes to terabytes it makes a big difference. In my case when my real data takes 4.2 TB and compressed one takes 3.9 TB its … 245 GB of space. For free. Thank you ZFS.

Quirks

The ‘quirks’ are things that need to be done differently then usual to make the device ‘happy’ – to make it work the way we want.

One of the Dell Wyse 3030 LT quirks is its bot process … it does not need any additional configuration – I just want to mention it – so you will not be stressed about it. After the FreeBSD loader(8) menu there is slight pause before continuing into the ‘rest’ of the boot process. Its several seconds long and its harmless. Here is how it looks on attached screen.

freebsd-bootloader

Power Usage Comparison

So its dirt cheap … and takes HALF of the power then my previews Mini ITX solution. The Mini ITX based systems used about 10.4 W of power when at idle state.

After measuring the Dell Wyse 3030 LT terminal power consumption with a kill-a-watt physical device – its only about 4.5 W only … and this is with % TB of 2.5 disk attached at idle state.

Another interesting fact is that I used the same ‘external’ power supply that I used for these Mini ITX boxes.

power-supply

Its 2.5A at 12V (which means 30 W)- exactly what a Dell Wyse 3030 LT terminal needs – according to its specification. As you can see from the photo above – the official power supply – while quite similar in size – needs a lot more space because of the additional 3-pin power cord – which is quite thick and takes a lot more space.

As in the past – I used python(1) [1] processes to load the CPU and dd(8) commands to load the drives I/O subsystem.

[1] # echo '999999999999999999 ** 999999999999999999' | python
[2] # dd if=/dev/da0  of=/dev/null  bs=1M
[3] # dd if=/dev/zero of=/data/FILE bs=1M

I will also include older Mini ITX based system measurements.

BOX        POWER  CPU          I/O
Wyse3030   3.8 W  IDLE         IDLE
MiniITX   10.5 W  IDLE         IDLE


BOX        POWER  CPU          I/O
Wyse3030  10.3 W  8 Thread(s)  3 DISK READ Thread(s) + 3 DISK WRITE Thread(s)
MiniITX   17.8 W  8 Thread(s)  3 DISK READ Thread(s) + 3 DISK WRITE Thread(s)

So after cutting CAPEX costs from about $110 to $15 I also lowered the OPEX price in half.

What can I say … I really appreciate and like the price/performance ratio 🙂

Cloud Storage Costs Comparison

I am not a fan of cloud based storage – its just damn overpriced to say the least.

In the past I have made multiple calculations about cloud storage costs and it will be no different this time. We need to assume some cost of 1kWh of power. In Poland its about $0.25 for 1kWh which means running a device that draws 1000 Watts for entire hour.

Assuming that our Dell Wyse 3030 LT uses about 4.5W all the time it would make the following costs.

  COST  TIME 
 $0.03  1 DAY(s)
 $9.96  1 YEAR(s)
$29.88  3 YEAR(s)
$49.80  5 YEAR(s)

Combining that with server cost ($100) we get TCO for our self hosted 5TB storage service.

COST  TIME
$110  1 YEAR(s)
$130  3 YEAR(s)
$150  5 YEAR(s)

This time after searching for cheapest cloud based storage I found these services.

  • Amazon Drive
  • Amazon S3 Glacier Storage
  • Backblaze B2 Cloud Storage
  • Google One
  • Dropbox Basic

Here is its cost summarized for 1 year period for 5TB of data.

PRICE  TIME       SERVICE
 $300  1 YEAR(s)  Amazon Drive
 $310  1 YEAR(s)  Google One
 $240  1 YEAR(s)  Amazon S3 Glacier Storage
 $450  1 YEAR(s)  Backblaze B2 Cloud Storage
 $360  1 YEAR(s)  Dropbox Basic

For the Backblaze B2 Cloud Storage I assumed average between upload/download price because upload is two times cheaper then download.

Here is its cost summarized for 5 year period for 5 TB of data.

PRICE  TIME       SERVICE
 $150  5 YEAR(s)  Dell Wyse 3030 LT + 5 TB USB 2.5 Drive
 $230  5 YEAR(s)  Dell Wyse 3030 LT + 5 TB USB 2.5 Drive (assuming drive failed)
$1500  5 YEAR(s)  Amazon Drive
$1550  5 YEAR(s)  Google One
$1200  5 YEAR(s)  Amazon S3 Glacier Storage
$2250  5 YEAR(s)  Backblaze B2 Cloud Storage
$1800  5 YEAR(s)  Dropbox Basic

… but there is also a catch here. Recently I was able to get a lifetime offer for 1 TB of cloud storage. It cost me about $170 for that 1 TB.

This is where it gets interesting – for 5 TB we would have to pay $850 for lifetime cloud storage. Lets calculate where the ‘personal’ NAS is comparable to ‘lifetime’ cloud storage.


PRICE   TIME       SERVICE
 $230   5 YEAR(s)  Dell Wyse 3030 LT + 5 TB USB 2.5 Drive => assuming drive failed 1 time(s)
 $460  10 YEAR(s)  Dell Wyse 3030 LT + 5 TB USB 2.5 Drive => assuming drive failed 2 time(s)
 $690  15 YEAR(s)  Dell Wyse 3030 LT + 5 TB USB 2.5 Drive => assuming drive failed 3 time(s)
 $920  20 YEAR(s)  Dell Wyse 3030 LT + 5 TB USB 2.5 Drive => assuming drive failed 4 time(s)
 $850  UNLIMITED   {5TB unlimited time cloud vendor}

So … it will take you about 20 years of your life (and an assumption that a drive will fail every 5 years with 4 drive fails total) to make it less expensive then the unlimited cloud storage. To be honest – I would get such unlimited cloud storage for most important files and documents for example in 1 TB or less size. Keep an encrypted backup copy the most important files and you are doing better then the classic 3-2-1 backup rule … assuming that you already have one online and one offline backup and also one onsite backup and one offsite backup … and at least two different medium types used 🙂

Drawbacks

WiFi is not supported and network requires restart after boot. Besides these I did not find any other issues.

ZFS on SMR Drives

The ZFS filesystem (more often called OpenZFS lately – as the project name) is a great filesystem for many purposes. From home or desktop/laptop solutions to enterprise offerings. Traditional disk drives have non overlapping magnetic tracks parallel to each other. These are PMR disks (Perpendicular Magnetic Recording). Hard disk drive manufacturers – to pack even more data into the same size platters – also offer SMR disks. In SMR disks data tracks are written to overlap part of previously written track – this results in narrower tracks and higher density. I will try to visualize this difference below using my favorite Enterprise Architect ASCII Edition software.

 PMR                    SMR

[xxx][___][___][___]   [xx[__[__[___]
[___][xxx][___][___]   [__[xx[__[___]
[___][___][xxx][___]   [__[__[xx[___]
[___][___][___][xxx]   [__[__[__[xxx]
[___][xxx][___][xxx]   [__[xx[__[xxx]
[xxx][___][___][xxx]   [xx[__[__[xxx]

12345678901234567890   12345678901234

I marked the filled blocks on both disks with xxx marks. As you can compare the below ‘size’ of the taken place the same data on SMR disk takes less physical space then on traditional PMR drives. This comes at a price through. Writes are little ‘crippled’ comparing to PMR drives. Especially heavy and random I/O writes are ‘problematic’ and slower on SMR drives … but it does not mean they are useless.

disk

For the backup or clone purposes they are more then enough. I personally use SMR drives for my backup solutions. Its just about price/performance ratio.

Here are mine backup solutions based on the SMR drives:

Speed

How ZFS behaves on SMR drives? Very well I would say. ZFS tries to pack as much random I/O into sequential with its ZFS features – described in detail in the zpool-features(7) man page for example.

I recently tried ZFS on top of GELI encrypted partition on a 5 TB external USB SMR drive. I needed to copy little more then 3 TB of data there. I used rsync(1) for that purpose. These are the arguments I use for my rsync(1) jobs.

% rsync --modify-window=1 -l -t -r -D -v -S -H --force    \
        --progress --no-whole-file --numeric-ids --delete \
        /files/ /media/external/files/

Of course I do not write all these options by hand – I just a script wrapper for that – rsync-delete.sh – available on my scripts page.

As I started to copy files on the drive I watched the write speeds using iostat(8) and zpool-iostat(8) tools. I expected quite slow operation but even with the enabled zstd compression and AES-XTS 256bit GELI encryption I got pretty decent results.

Here are the iostat(8) results. Each line means average of 10 minutes (600 seconds). Check the speeds for da0 drive below.

% iostat 600
       tty            ada0             ada1              da0             cpu
 tin  tout KB/t  tps  MB/s  KB/t  tps  MB/s  KB/t  tps  MB/s  us ni sy in id
   1     1  513  120  59.9  29.5   39   1.1   742   65  46.8   4  8 17  2 69
   0     2  615   94  56.6  19.1   22   0.4   751   68  49.8   1  3 14  1 82
   0     0  561  106  57.9  17.9   20   0.4   760   70  52.0   1  2 14  1 82
   0     0 1015   57  56.8  18.4   16   0.3   769   68  50.9   1  3 15  1 81
   0     0 1017   57  56.3  18.5   16   0.3   757   68  50.6   1  3 14  1 81
   0     1  752   72  53.0  16.6   23   0.4   765   67  50.1   1  1 13  0 85
   0     0 1014   51  50.1  16.5   21   0.3   723   68  48.3   1  1 13  0 86
   0     0 1012   51  50.2  19.8   18   0.3   743   68  49.2   1  1 12  0 86

And here are the zpool-iostat(8) results.

% zpool iostat POOL 600
              capacity     operations     bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
POOL        3.18T  1.37T      7     56  53.5K  40.7M
POOL        3.20T  1.34T      0     57  9.01K  41.4M
POOL        3.22T  1.33T      0     47  3.29K  32.3M
POOL        3.24T  1.31T      0     47  5.59K  33.9M
POOL        3.25T  1.29T      0     43  3.39K  24.3M
POOL        3.27T  1.28T      0     42  3.01K  25.5M
POOL        3.28T  1.27T      0     44  3.14K  26.8M
POOL        3.29T  1.26T      0     42  3.49K  23.9M

The drive was attached over USB 3.0 port so there was not 35 MB/s limitation from USB 2.0 port. I would say that the results are very decent and consistent.

Tuning

There are several settings that can help you squeeze maximum from these SMR drives on ZFS filesystem.

First are ZFS pool settings. You want the latest zstd compression to save some space. Also better compression means less physical bytes need to be written to the drive so less I/O operations. You should also turn atime into off state as it will not be needed. You should also increase recordsize to something really big like 1m (1 megabyte) so you will get higher compressratio and also will need to have less metadata for more ZFS blocks. Keep in mind that ZFS will still use variable block size and not only the 1m maximum. If something is smaller (like 100k) then it would take for example 80k (after applied zstd compression). You will not waste 920k here 🙂

Keep in mind that most newer and larger drives use 4k blocks (instead of 512b). Sometimes its 512e method which means that drive firmware will ‘present’ device with 512b blocks while underneath these eight 512k blocks just lay down on a single 4k block. For these reasons its important to keep in mind several things.

When adding new partitions with gpart(8) remember to align them to 4k with -a 4k argument.

# gpart add -t freebsd-zfs -a 4k da0

Next – when initializing the geli(8) encryption layer – make sure you add -s 4096 argument.

# geli init -s 4096 /dev/da0p1

The last thing is ZFS pool creation with proper ashift property – it can not be changed later. On FreeBSD UNIX its done that way:

# sysctl vfs.zfs.min_auto_ashift=12
# zpool create POOL da0
# zdb -C POOL | grep ashift
                ashift: 12

If you are curious what 12 means then below table will help you:

ASHIFT  BLOCKSIZE
     9  512b
    10  1k
    11  2k
    12  4k
    13  8k

Last but not least is the redundant_metadata option. By default its at all setting but its desired to set it into the most state. Do you need redundant metadata? I think not. When your single drive will fail the redundant metadata would not help and if your ZFS pool have some redundancy level like raidz or mirror then redundant metadata is also not needed because its just ‘normally’ redundant being spread across several disks.

Keep in mind that ZFS resilver process on some of these SMR drives can take forever. Some people from Reddit reported that they successfully resilvered their ZFS pools with SMR drives but that does not have to be the case for all SMR drives out there. You can also check Ars Technica tests of resilver on SMR disks.

Here is the summary of ZFS tunables suggested – you will find in depth description of all of them in the zfsprops(7) man page.

# zfs set redundant_metadata=most POOL
# zfs set compression=zstd        POOL
# zfs set atime=off               POOL
# zfs set recordsize=1m           POOL

In theory the TRIM operations upon deletion would create additional unwanted ‘stress’ for SMR drives which would mean that TRIM operations should be disabled for on non-SSD drives and you can disable them entirely on the ZFS pool level … but.

TRIM commands issued by the operating system allows SMR HDD internal controller to get the information that certain areas/blocks on that SMR HDD plates are no longer in use. It means that writes to such areas could be performed without slow read-modify-write pattern.

This means we are leaving the autotrim option as on (enabled) for SMR drives.

# zpool autotrim=on POOL

Also – if needed – you can manually trigger the TRIM operations with this command.

# zpool trim POOL
# zpool status POOL
  pool: POOL
 state: ONLINE
  scan: scrub repaired 0B in 02:17:22 with 0 errors on Sun May  8 05:18:22 2022
config:

        NAME          STATE     READ WRITE CKSUM
        POOL          ONLINE       0     0     0
          da0p1.eli   ONLINE       0     0     0  (trimming)

errors: No known data errors


By default the TRIM commands are executed at 64 rate on FreeBSD. You can limit them to 1 and still have them enabled with following sysctl(8) tunable.

# sysctl vfs.zfs.vdev.trim_max_active=1

If you want to make it survive across reboots then put it into the /etc/sysctl.conf file.

Logic could suggest that simpler/older filesystems such as FreeBSD UFS for example could be more suitable solution for SMR drives … but the reality shows that not so much. Check this Reddit thread for example – Appalling Performance on External USB SMR Drive – to name just one.

Hope this article will help you get most of your SMR drives.

Regards.

EOF

ZFS Compatibility

The best free filesystem on Earth – ZFS – also often named OpenZFS recently – has also become very portable in recent years of its development. The OpenZFS Distributions page lists 6 (six) operating systems already.

They are:

  • FreeBSD
  • Illumos
  • Linux
  • MacOS
  • NetBSD
  • Windows

… but if you would like to create a ZFS pool compatible with all of them … which options and ZFS features should you choose? There is OpenZFS Feature Flags page dedicated exactly to that topic.

zfs-feature-flags

These are the ones that have yes value in all operating systems.

  • async_destroy
  • bookmarks
  • empty_bpobj
  • enabled_txg
  • filesystem_limits
  • lz4_compress
  • hole_birth
  • multi_vdev_crash_dump
  • spacemap_histogram

I would also include these as only older NetBSD 4.0.5 version does not support them – but they are supported in newer NetBSD 5.3 version.

  • embedded_data
  • large_blocks
  • sha512
  • skein

There is also a dedicated zpool-features(7) man page for that information on the OpenZFS page and also zpool-features(7) man page on the FreeBSD page.

On the FreeBSD system the /usr/share/zfs/compatibility.d directory has files with supported ZFS Feature Flags for many major operating systems and ZFS versions.

% ls /usr/share/zfs/compatibility.d
2018
2019
2020
2021
compat-2018
compat-2019
compat-2020
compat-2021
freebsd-11.0
freebsd-11.1
freebsd-11.2
freebsd-11.3
freebsd-11.4
freebsd-12.0
freebsd-12.1
freebsd-12.2
freenas-11.0
freenas-11.1
freenas-11.2
freenas-11.3
freenas-9.10.2
grub2
openzfs-2.0-freebsd
openzfs-2.0-linux
openzfs-2.1-freebsd
openzfs-2.1-linux
openzfsonosx-1.7.0
openzfsonosx-1.8.1
openzfsonosx-1.9.3
openzfsonosx-1.9.4
truenas-12.0
ubuntu-18.04
ubuntu-20.04
zol-0.6.1
zol-0.6.4
zol-0.6.5
zol-0.7
zol-0.8

Unfortunately it misses NetBSD and Illumos systems for example … but having information from the OpenZFS Feature Flags page we can find Feature Flags set that will be supported everywhere.

Here are the stats for supported ZFS Feature Flags. The higher the number the more operating systems and ZFS version it covers.

% grep -h '^[^#]' /usr/share/zfs/compatibility.d/* \
    | sort -n \
    | uniq -c \
    | sort -n
   2 draid
   5 bookmark_written
   5 device_rebuild
   5 livelist
   5 log_spacemap
   5 redacted_datasets
   5 redaction_bookmarks
   5 zstd_compress
   9 allocation_classes
   9 bookmark_v2
   9 project_quota
   9 resilver_defer
  10 edonr
  10 encryption
  11 large_dnode
  11 userobj_accounting
  18 spacemap_v2
  20 device_removal
  20 obsolete_counts
  20 zpool_checkpoint
  31 sha512
  31 skein
  32 multi_vdev_crash_dump
  36 filesystem_limits
  36 large_blocks
  37 bookmarks
  37 embedded_data
  37 enabled_txg
  37 extensible_dataset
  37 hole_birth
  37 spacemap_histogram
  38 async_destroy
  38 empty_bpobj
  38 lz4_compress

As the GNU GRUB is very outdated when it comes to ZFS support it should be pretty bulletproof to use it as a starting point of limited ZFS Feature Flags support.

% cat /usr/share/zfs/compatibility.d/grub2
# Features which are supported by GRUB2
async_destroy
bookmarks
embedded_data
empty_bpobj
enabled_txg
extensible_dataset
filesystem_limits
hole_birth
large_blocks
lz4_compress
spacemap_histogram

To make sure we are compatible we will now cross-link the GNU GRUB data.

First we will ‘generate’ the grep(1) command arguments that we will use in the next command.

% grep '^[^#]' /usr/share/zfs/compatibility.d/grub2 \
  | while read I
    do
      echo "-e ' ${I}' \\"
    done
-e ' async_destroy' \
-e ' bookmarks' \
-e ' embedded_data' \
-e ' empty_bpobj' \
-e ' enabled_txg' \
-e ' extensible_dataset' \
-e ' filesystem_limits' \
-e ' hole_birth' \
-e ' large_blocks' \
-e ' lz4_compress' \
-e ' spacemap_histogram' \

Lets now use these arguments to filter the ZFS features.

% grep -h '^[^#]' /usr/share/zfs/compatibility.d/grub2/* \
    | sort -n \
    | uniq -c \
    | sort -n \
    | grep -e ' async_destroy' \
           -e ' bookmarks' \
           -e ' embedded_data' \
           -e ' empty_bpobj' \
           -e ' enabled_txg' \
           -e ' extensible_dataset' \
           -e ' filesystem_limits' \
           -e ' hole_birth' \
           -e ' large_blocks' \
           -e ' lz4_compress' \
           -e ' spacemap_histogram' \
    | wc -l
      11

% grep -h '^[^#]' /usr/share/zfs/compatibility.d/grub2 | wc -l
      11

So if seems that GRUB list of ZFS Feature Flags seems pretty compatible.

Lets now cross-reference that GRUB data with the data from the OpenZFS Feature Flags page.

I will create new /usr/share/zfs/compatibility.d/OZFF file that has these ZFS Feature Flags as content.

% cat /usr/share/zfs/compatibility.d/OZFF
async_destroy
bookmarks
empty_bpobj
enabled_txg
filesystem_limits
lz4_compress
hole_birth
multi_vdev_crash_dump
spacemap_histogram
embedded_data
large_blocks
sha512
skein

% wc -l /usr/share/zfs/compatibility.d/OZFF
      13

So there are 11 GRUB ZFS Feature Flags and 13 OpenZFS ‘Compatible’ Feature Flags.

Lets see how it they compare.

% cat /usr/share/zfs/compatibility.d/OZFF \
    | grep -e async_destroy \
           -e bookmarks \
           -e embedded_data \
           -e empty_bpobj \
           -e enabled_txg \
           -e extensible_dataset \
           -e filesystem_limits \
           -e hole_birth \
           -e large_blocks \
           -e lz4_compress \
           -e spacemap_histogram \
    | wc -l
      10

I expected 11 here instead of 10 … we will not have to compare GRUB results to the OpenZFS Feature Flags section.

% grep -h '^[^#]' /usr/share/zfs/compatibility.d/{grub2,OZFF} \
    | sort -n \
    | uniq -c \
    | sort -n
   1 extensible_dataset
   1 multi_vdev_crash_dump
   1 sha512
   1 skein
   2 async_destroy
   2 bookmarks
   2 embedded_data
   2 empty_bpobj
   2 enabled_txg
   2 filesystem_limits
   2 hole_birth
   2 large_blocks
   2 lz4_compress
   2 spacemap_histogram

Seems that we finally got our 10 most compatible OpenZFS Feature Flags set.

Its this set:

  • async_destroy
  • bookmarks
  • embedded_data
  • empty_bpobj
  • enabled_txg
  • filesystem_limits
  • hole_birth
  • large_blocks
  • lz4_compress
  • spacemap_histogram

To make it more comfortable to use we will put them it into the separate /usr/share/zfs/compatibility.d/COMPATIBLE file.

# cat /usr/share/zfs/compatibility.d/COMPATIBLE
async_destroy
bookmarks
embedded_data
empty_bpobj
enabled_txg
filesystem_limits
hole_birth
large_blocks
lz4_compress
spacemap_histogram

Lets now try to make that best effort most-compatible ZFS pool.

I will use one of my scripts – mdconfig.sh – to easy manipulate md(4) memory disks on FreeBSD.

# truncate -s 1g FILE
# mdconfig.sh -c FILE
IN: created vnode at /dev/md0
# zpool create -o compatibility=COMPATIBLE compatible /dev/md0
# zpool list
NAME         SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
compatible   960M   116K   960M        -         -     0%     0%  1.00x    ONLINE  -
zroot        118G  48.6G  69.4G        -         -    36%    41%  1.00x    ONLINE  -


Now lets see what is zpool upgrade command showing us.

# zpool upgrade
This system supports ZFS pool feature flags.

All pools are formatted using feature flags.


Some supported features are not enabled on the following pools. Once a
feature is enabled the pool may become incompatible with software
that does not support the feature. See zpool-features(7) for details.

Note that the pool 'compatibility' feature can be used to inhibit
feature upgrades.

POOL  FEATURE
---------------
compatible
      multi_vdev_crash_dump
      large_dnode
      sha512
      skein
      userobj_accounting
      encryption
      project_quota
      device_removal
      obsolete_counts
      zpool_checkpoint
      spacemap_v2
      allocation_classes
      resilver_defer
      bookmark_v2
      redaction_bookmarks
      redacted_datasets
      bookmark_written
      log_spacemap
      livelist
      device_rebuild
      zstd_compress
      draid
zroot
      userobj_accounting
      encryption
      project_quota
      allocation_classes
      resilver_defer
      bookmark_v2
      redaction_bookmarks
      redacted_datasets
      bookmark_written
      log_spacemap
      livelist
      device_rebuild
      zstd_compress
      draid

There are LOTS of ZFS Feature Flags to be activated but if we want to have our ZFS pool keep compatible – we will have to stay away from it 🙂

zfs-terminal

You may get impression that you miss a lot … but you do not miss that much. Here are the ZFS Feature Flags you can fully utilize.

# zpool get all compatible | grep -v disabled
NAME        PROPERTY                       VALUE                          SOURCE
compatible  size                           960M                           -
compatible  capacity                       0%                             -
compatible  altroot                        -                              default
compatible  health                         ONLINE                         -
compatible  guid                           5395735446052695775            -
compatible  version                        -                              default
compatible  bootfs                         -                              default
compatible  delegation                     on                             default
compatible  autoreplace                    off                            default
compatible  cachefile                      -                              default
compatible  failmode                       wait                           default
compatible  listsnapshots                  off                            default
compatible  autoexpand                     off                            default
compatible  dedupratio                     1.00x                          -
compatible  free                           960M                           -
compatible  allocated                      116K                           -
compatible  readonly                       off                            -
compatible  ashift                         0                              default
compatible  comment                        -                              default
compatible  expandsize                     -                              -
compatible  freeing                        0                              -
compatible  fragmentation                  0%                             -
compatible  leaked                         0                              -
compatible  multihost                      off                            default
compatible  checkpoint                     -                              -
compatible  load_guid                      17463015630652190527           -
compatible  autotrim                       off                            default
compatible  compatibility                  COMPATIBLE                     local
compatible  feature@async_destroy          enabled                        local
compatible  feature@empty_bpobj            enabled                        local
compatible  feature@lz4_compress           active                         local
compatible  feature@spacemap_histogram     active                         local
compatible  feature@enabled_txg            active                         local
compatible  feature@hole_birth             active                         local
compatible  feature@extensible_dataset     enabled                        local
compatible  feature@embedded_data          active                         local
compatible  feature@bookmarks              enabled                        local
compatible  feature@filesystem_limits      enabled                        local
compatible  feature@large_blocks           enabled                        local

You get the very decent LZ4 compression and also ZFS Bookmarks feature which are very useful feature for sync|recv mechanism.

Keep in mind that the -o compatibility= switch for zpool(8) is available on OpenZFS 2.1 or newer. The 2.1 version is already available on the FreeBSD 13.1-BETA* releases and will be part of the FreeBSD 13.1-RELEASE systems. To make use of it on older FreeBSD releases you will have to use openzfs and openzfs-kmod packages and also use the following settings in the /boot/loader.conf file.

From that one:

zfs_load=YES

Into that one:

zfs_load=NO
openzfs_load=YES

With these openzfs and openzfs-kmod packages and above settings in the /boot/loader.conf file you can use this OpenZFS 2.1 on FreeBSD 12.2 – on FreeBSD 12.3 – and on FreeBSD 13.0 … and of course on upcoming FreeBSD 13.1 release.

Not sure if I should have add anything more here. Feel free to remind me in the commends 🙂

EOF

ZFS Boot Environments Revolutions

I do not have to remind you that I am a big fan of ZFS Boot Environments feature. From the time when I first used it on OpenSolaris and Solaris systems I was really fascinated by it. Bulletproof upgrades and changes to entire system … and it was possible more then decade ago. Like a Dream. Today with beadm(8) and bectl(8) tools and also the FreeBSD loader(8) the ZFS Boot Environments are first class and one of the main features of the FreeBSD operating system.

Back in the more ‘normal’ times (before C19) I was able to talk two times about ZFS Boot Environments. I hope I explained them well.

  • 1st in Poland at PBUG meeting – with presentation available HERE.
  • 2nd in Holland at NLUUG conference – with presentation available HERE.

I do not know any downsides of ZFS Boot Environments but if you would stick a gun into my head and make me find one – I would say that you still have to reboot(8) to change to the other BE. This is about to change …

Reroot Instead Reboot

What is reroot? Its the ability to switch to other root filesystem without the need for full system reboot. The loaded and running kernel stays the same of course – but this is the only downside. This feature is implemented in the reboot(8) command with -r argument.

As we can read in the FreeBSD 10.3-RELEASE Release Notes page:

The initial implementation of “reroot” support has been added to the reboot(8) utility, allowing the root filesystem to be mounted from a temporary source filesystem without requiring a full system reboot. (r293744) (Sponsored by The FreeBSD Foundation)

How can reroot be useful here? It will save you a lot of time when you did not updated the kernel. There are two types of update strategies when using the ZFS Boot Environments. You can create new BE (as a backup world that you can get back to) and update the running system. Then you can use checkrestart(1) to verify which processes should be restarted because either binaries or libraries has been updated.

checkrestart.1

The other way was to create new separate BE (while not touching the running one) and then mount it and update that new BE and reboot into it later. This created a need to reboot(8) but not anymore. Especially when you just update the packages with pkg(8) command.

beadm(8)

With the new reroot option of beadm(8) you will tell FreeBSD to reroot your running kernel into specified BE. It will definitely have less impact in virtual machines as they reboot quite fast but imagine saved time on a server class physical machine with about 10 minutes lost for BIOS POST messages and initialization … or personal desktop/laptop GELI encrypted system without the need to type in again the GELI password to decrypt it after reboot.

beadm.reroot

On the screenshot above I use the latest FreeBSD 13.1-BETA1 but it works the same on other production FreeBSD releases such as 12.3-RELEASE or 13.0-RELEASE. The new upgraded beadm(8) is available from its home at GitHub page here:

I will add that updated version to the FreeBSD Ports tree later along with updated man page later.

Usage

Usage of this new feature is quite simple. You type beadm reroot BENAME in the terminal and FreeBSD reroots into that BE without reboot. Takes about 9-10 seconds on my 11 years old ThinkPad W520 so it may be even faster on your more up to date system.

# beadm list
BE        Active Mountpoint  Space Created
12.3      -      -            9.5G 2021-10-18 13:14
13.0.p6   -      -           13.9G 2022-01-27 11:07
13.0      -      -           12.9G 2022-03-05 15:02
13.0.safe -      -            2.8M 2022-03-08 14:54
13.1      NR     /            9.5G 2022-03-12 00:18
13.1.safe -      -          544.0M 2022-03-13 23:18

# beadm activate 13.1.safe
Activated successfully

# beadm reroot 13.1.safe

… and you are going the route similar to typing shutdown now on a running system. All services are stopped. Then root is changed to new one. Then system continues to boot along with starting all its services as usual. Just without the BIOS POST and the bootloader and kernel parts.

The reroot feature is especially useful in one of these scenarios:

From what I know the bectl(8) does not has that reroot feature but maybe it will be added to it somewhere in the future.

Summary

Not sure that ZFS Boot Environments Revolutions is the best title for this blog post, but as I used Reloaded on my 2nd ZFS Boot Environments presentation I though that sticking to The Matrix (1999) schema. I could of course do 3rd and updated presentation … but I am afraid that it will not happen … or at least not soon.

I did not thought that the FreeBSD Enterprise Storage presentation that I gave at 2020/02 PBUG would be my last – it was more then 2 years ago.

UPDATE 1 – Faster Upgrade with New beadm(8) Version

Today (2022/05/06) I introduced new beadm(8) version 1.3.5 that comes with new chroot(8) feature. It has already been committed to the FreeBSD Ports tree under 263805 PR so expect packages being available soon.

You can also update beadm(8) directly like that:

# fetch -o /usr/local/sbin/beadm https://raw.githubusercontent.com/vermaden/beadm

Now for the faster update process – here are the instructions depending on the shell you use.

  • ZSH / CSH
# beadm create 13.1-RC6
# beadm chroot 13.1-RC6
BE # zsh || csh
BE # yes | freebsd-update upgrade -r 13.1-RC6
BE # repeat 3 freebsd-update install
BE # exit
# beadm activate 13.1-RC6
# reboot
  • SH / BASH / FISH / KSH
# beadm create 13.1-RC6
# beadm chroot 13.1-RC6
BE # sh || bash || fish || ksh
BE # yes | freebsd-update upgrade -r 13.1-RC6
BE # seq 3 | xargs -I- freebsd-update install
BE # exit
# beadm activate 13.1-RC6
# reboot

Happy upgrading 🙂

EOF

Books About FreeBSD

There are many books in which FreeBSD is covered or it is the one of the main objectives of such book. Today I will guide you through these books. I will try to focus on more up to date ones because it would be pointless (beyond historical purposes) to read them now.

The Hateful Eight

I will start with official FreeBSD documentation – as it offers 8 different books for you to get to know that UNIX operating system better. The two most known are FreeBSD Handbook and FreeBSD FAQ. The FreeBSD Handbook covers all/most general topics about FreeBSD operating system setup and administration while FreeBSD FAQ tries to answer most popular questions about it – and does it quite well. There are also other books that are more developer oriented. Below you will find the list of all available eight books from the FreeBSD project.

FreeBSD Handbook https://freebsd.org/handbook
FreeBSD FAQ https://freebsd.org/faq
FreeBSD Architecture Handbook https://docs.freebsd.org/en/books/arch-handbook/
FreeBSD Developers Handbook https://docs.freebsd.org/en/books/developers-handbook/
FreeBSD Porters Handbook https://docs.freebsd.org/en/books/porters-handbook/
Design and Implementation of 4.4BSD Operating System https://docs.freebsd.org/en/books/design-44bsd/
Project Model for FreeBSD https://docs.freebsd.org/en/books/dev-model/
FreeBSD Documentation Project Primer for New Contributor https://docs.freebsd.org/en/books/fdp-primer/

Example FreeBSD Handbook page from PDF file below.

freebsd-handbook

… and while you can download and read PDF files (there are also EPUB/HTML/TXT formats available for download) you can also read it online – this is how the FreeBSD FAQ looks online.

freebsd-faq-www

I sometimes miss that FreeBSD project does not deliver dedicated book in the same PDF/EPUB/HTML/TXT manner for the FreeBSD man pages as for example Solaris or AIX does. At least you can read them online on the https://man.freebsd.org/command page where you will put the needed man page in the place of ‘command’ word – for example the manual page for gstat(8) command is available at https://man.freebsd.org/gstat URL.

If there are two man pages with the same name like crontab(1) and crontab(5) for example then add the man page section number after the slash (/) at the end of URL like that https://man.freebsd.org/crontab/5 – this will solve that problem.

Below you can see the Solaris 10 – Man Pages Section 1M – System Administration Commands book with all Solaris administrative (1M) commands. For FreeBSD the administrative commands are at (8) section.

sol10man1M

The Usual Suspects

The most known books that cover FreeBSD operating system aspects are the ones that are authored by Michael W. Lucas (in more recent titles sometimes accompanied by Allan Jude).

The most famous (and useful) one is the Absolute FreeBSD – Complete Guide to FreeBSD in its most recent 3nd Edition. It is quite recent as its from 2019 year. You can get it both in digital (PDF/EPUB) and traditional printed way. If you already read the FreeBSD Handbook and FreeBSD FAQ are wondering what you should get next to continue your FreeBSD journey then this book is the answer. It fill all the gaps and bring a lot of additional information that you will find very useful in your day to day life with FreeBSD system.

absolute-fbsd

Several books later (not only technical) Michael W. Lucas started the FreeBSD Mastery series with many interesting books about FreeBSD. For the two that cover the ZFS filesystem Allan Jude is also a coauthor.

mastery

Here they are:

  • FreeBSD Mastery: Storage Essentials (2014)
  • FreeBSD Mastery: Specialty Filesystems (2015)
  • FreeBSD Mastery: ZFS (2015)
  • FreeBSD Mastery: Advanced ZFS (2016)
  • FreeBSD Mastery: Jails (2019)

If you are gonna read them – then also do it in that order sa they are listed above. You will need all that ‘introduction’ to get the most of FreeBSD Jails. Even Michael W. Lucas mentioned that you need to ‘do’ several other of his books to truly take advantage of all things written in the FreeBSD Mastery: Jails book. While the FreeBSD Mastery: Storage Essentials and FreeBSD Mastery: Specialty Filesystems are very close related to FreeBSD operating system the other two FreeBSD Mastery: ZFS and FreeBSD Mastery: Advanced ZFS have also a lot of general ZFS knowledge not only limited to FreeBSD operating system.

One additional word about the FreeBSD Mastery: Jails book as its quite ‘special’ in approach. For most parts of the FreeBSD operating system the FreeBSD Handbook covers some or most of the information and tasks needed to do convening particular topic. When it comes to FreeBSD Jails its not that simple anymore. There are two types of Jails. The ‘traditional’ Jails that use the host system network stack and the new ‘VNET’ Jails that bring their own – separate from the host – network stack. It gives you a lot more possibilities and features but it comes with only one downside. The official FreeBSD Handbook does not cover the ‘VNET’ Jails at all. Zero. Nada. Zip. None. You can get ‘some’ grasp about them from the FreeBSD man pages but that is definitely not enough. The ‘VNET’ Jails are of course production ready from many years but for some reason the missing chapter in the FreeBSD Handbook is still missing. That is the most important reason why you should get the FreeBSD Mastery: Jails book.

As we are in the FreeBSD Jails scope … I should also mention the FreeBSD Jails using VNETs book by Derik Ramirez from 2020.

added-JAILS

Its not as ‘big’ as the FreeBSD Mastery: Jails but keep in mind that half of Michael W. Lucas content is about the iocage(8) framework … which is quite dead now unfortunately. The biggest upside of FreeBSD Jails using VNETs book is that it covers – as described in the book title – the VNET Jails. Also being released in 2020 its very up to date.

The Taste of Others

Written in 2018 by Manish Jain the book Beginning Modern Unix covers both FreeBSD and Linux operating systems at the same time.

beginning-modern-unix

It also beginner friendly and as author writes in Preparing for Part I“The structure is intended to make things simple for Windows users planning to migrate to FreeBSD/Linux.” I also really liked the author explanation in the Preface about why he choose BSD/FreeBSD and GNU/Linux (like that syntax by the way). You can find part of it below.

beginning-modern-unix-preface

To be honest I like that ‘dual’ approach with sections showing how to achieve the same on two different (yet somewhat similar) operating systems. It may be useful especially when writing playbooks for various configuration management software like Ansible or Salt. I know that most of these configuration management systems provide their own ‘general’ integrations like ‘install packages’ on a system and then it uses pkg(8) on FreeBSD and yum(8) on CentOS but sometimes not all integrations are available or they are also sometimes broken or buggy. I have heard several times this already that its safer to rely on your own scripts and ‘raw’ commands instead of the ‘integration modules’ in many cases. Of course your millage may vary.

The Beginning Modern Unix book covers many desktop related topics but it also covers POSIX shell scripting and basic C programming which is very nice.

Another interesting ‘dual’ book is UNIX The Textbook from 2017 by Syed Mansoor Sarwar and Robert M. Koretsky authors.

unix-textbook

In its 3rd Edition it covers both Solaris and FreeBSD (in a form of PC-BSD tho) systems. As Solaris by default uses GNOME and PC-BSD (while it existed) used KDE the book also covers a lot about these two desktop environments.

unix-textbook-preface

It is illustrated with many useful diagrams of how things work on UNIX system. Below you can check the pipe explanation.

unix-textbook-pipe

Another good part of the book is that it also covers a lot of POSIX shell scripting and C programming techniques. Even covering things like writing simple servers, inter process communication, threads and a lot more. The book is really huge with almost 1400 pages of useful content. The book leave you at the end of each chapter with QUESTIONS AND PROBLEMS. This approach reminds me of the legendary The C Programming Language written by Brian Kernighan and Dennis Ritchie UNIX fathers where it was also present.

Next one is very well known UNIX and Linux System Administration Handbook by Evi Nemeth and Garth Snyder and Trent R. Hein and Ben Whaley … but it depends which version you get 🙂

unix-linux-admin-4th

The 4th Edition is more conservative and covers Linux/Solaris/HP-UX/AIX systems. This is the description of which systems they have chosen in 2011 and why.

unix-linux-admin-4th-systems

If you try to find FreeBSD there you will fail. They only mentioned it once and added that along with OpenBSD and NetBSD and that BSD systems “(…) enjoy somewhat less support from third-party software vendors.” See for yourself.

unix-linux-admin-4th-freebsd

Lets now move 7 years forward to 2017 in which the 5th edition of the same UNIX and Linux System Administration Handbook book was released. Authors also changed a little with Dan Mackin being added to current lineup of authors of this book.

unix-linux-admin-5th

Besides modified cover I would risk a stance that its quite entirely different book right now. Its because the 5th Edition covers only Linux and … FreeBSD. No AIX. No Solaris. No HP-UX. Authors also added quite long justification on why they have chosen these operating systems and not the other ones. The time of oldschool UNIX dinosaurs that ruled for decades seem to slowly vanish. Its probably not without reason.

The last HP-UX version 11.31 (also known as 11i v3 variant) was released in 2007. Still uses ‘manual’ packages like in 1995. I like its ‘ecosystem’ tho. What I mean by that is that you can install and setup several HP-UX machines. Setup HP Serviceguard HA cluster on these machines and then make HP Virtual Machines or HP SRP Containers highly available between these hosts as Serviceguard services. I was fortunate enough to be able to see such 6 node cluster in action and it worked really well.

The last AIX version 7.2 was released in 2015 but I do not recall any ground braking features. Also AIX still does not have any modern package management and the most that IBM AIX developers could do was to adopt RPM database to add RPM packages along with the native ones. In 2021 the newer AIX 7.3 release also saw the light of day … but also without any groundbreaking features. In other words the last two AIX system releases fell only like a maintenance releases. While the HP-UX ‘ecosystem’ is ‘connected’ between their products and features its not that easy and simple in the AIX ‘POWER World’. For example its not possible to create similar setup with LPARs or WPARs that their high availability would be controlled by PowerHA cluster software. Even if you would create LPARs with purely virtual devices and storage from the SAN network. The POWER ecosystem offers a feature called Remote Restart for LPARs on HMC but its far from being close to what HP-UX ecosystem offers here.

As for Solaris … I think that Oracle taking over SUN is probably one of the worst things that could happen to Solaris. Oracle could extend and continue the OpenSolaris road started by SUN. It could move and expand Solaris to 21st century. Instead it also went the ‘maintenance’ road along with maximizing the cash outcome of Solaris ‘asset’ with 11.3 in 2015 and 11.4 in 2018 releases. Besides adding PF firewall from OpenBSD and Live Migration feature for Kernel Zones I do not recall any groundbreaking features to be added. Maybe some ZFS development but looking at what OpenZFS is achieving with each release these Oracle developments do not look ‘big’ at all. From what is known Oracle also fired most of SPARC and Solaris developers leaving only small teams to make it running in ‘maintenance mode’ up to 2034 to which Oracle promised to keep Solaris alive instead of moving forward with Solaris 12.0 which was even in alpha or beta state. I agree with authors statement in which they say that “The popularity of UNIX has been waning for some time, and most of the stalwart UNIX distributions (e.g., Solaris, HP-UX, and AIX) are no longer in common use.”

Below is the authors description why they have chosen the FreeBSD and Linux systems for the 5th Edition of their well respected and acknowledged book.

unix-linux-admin-5th-systems

Back to FreeBSD world … and the authors quote why FreeBSD was included in this most recent version of their book – “The open source descendants of BSD are exceptions to this trend and continue to enjoy a cult following, particularly among operating system experts, free software evangelists, and security-minded administrators. In other words, some of the world’s foremost operating system authorities rely on the various BSD distributions. Apple’s macOS has a BSD heritage.” This most recent version is also little shorter with ‘only’ about 1200 pages while earlier edition topped at little over 1300. Keep in mind that newer edition covers Linux and FreeBSD while the older one had to describe and document to systems more.

Last but not least I should also mention the Book of PF – No Nonsense Guide to OpenBSD Firewall book by Peter N.M. Hansteen from 2015 in its most recent 3rd Edition.

pf-book

While originally targeted at OpenBSD users the FreeBSD users will also be able to get a lot of useful knowledge about PF firewall that FreeBSD uses. Keep in mind that there are some syntax differences between OpenBSD and FreeBSD PF firewalls.

Source Code

Here you will find the books that are little less useful for sysadmins and more useful for developers and programmers. We will start with updated Design and Implementation of FreeBSD 11 Operating System from 2015 in 2nd Edition form. Written by one of the original BSD UNIX and FreeBSD developers Marshall Kirk McKusick along with other two FreeBSD developers George V. Neville-Neil and Robert N.M. Watson.

design-implementation-bsd

Its generally more up to date version of the official FreeBSD documentation available as Design and Implementation of 4.4BSD Operating System title from the FreeBSD project documentation page – which one of the authors is also Marshall Kirk McKusick. Is it worth to get it then? Absolutely. A lot have changed and many new technologies have been imported into FreeBSD source tree such as ZFS or DTrace or for example the ULE scheduler.

The final book that I would like to mention here is the FreeBSD Device Drivers book from 2012 written by Joseph Kong.

bsd-device-drivers

The book tries to achieve what it title says – to help you first understand and then modify or write your own device drivers. While it covers little older FreeBSD 8 version it is not a problem because the FreeBSD API and ABI change very slow and only when no other way is possible.

Another great tool for programmers and developers in the process of making FreeBSD better is book about DTrace – the dynamic tracing framework. The DTrace – Dynamic Tracing in Oracle Solaris, Mac OS X, and FreeBSD book.

added-DTRACE

Written by Brendan Gregg and Jim Mauro in 2011 greatly helps to jump into that topic in simple and straightforward way. Besides covering FreeBSD it also does cover Mac OS X, Solaris and even OpenSolaris. That means that it should also be useful for Illumos developers. The book contains a lot of DTrace scripts and examples on how to use that fantastic tool.

Another book that you may find useful in your FreeBSD programmer career is the Designing BSD Rootkits: An Introduction to Kernel Hacking book by Joseph Kong.

added-ROOTKITS

Written in 2007 still contains lots of up to date information for the FreeBSD hackers. While the term ‘rootkit’ may be taken ‘negatively’ the author itself describes the book by himself the best way – “Though rootkits have a fairly negative image, they can be used for both good and evil. Designing BSD Rootkits arms you with the knowledge you need to write offensive rootkits, to defend against malicious ones, and to explore the FreeBSD kernel and operating system in the process.” I agree with the author here. To be well protected against something you first need to know how that thing works. The book contains many code examples that you may compile and use and also build upon with your own ideas. Would recommend.

Gone with the Wind

If you know other FreeBSD related books then please let me know.

For those few that did not noticed – the titles of the headers are really great movies 🙂

Other FreeBSD Version in ZFS Boot Environment

The first FreeBSD 12.3-PRERELEASE snapshots are finally available. This means we can try them in a new ZFS Boot Environment without touching out currently running 13.0-RELEASE system. We can not take the usual path with creating new BE from our current one and upgrade it to newer version because 12.3 has older major version then the 13.0 one.

This is kinda a paradox in the FreeBSD release process that when released the 12.3-RELEASE will have some newer commits and features then older 13.0-RELEASE which was released earlier this year. Of course not all things that have been committed to HEAD goes into 12-STABLE or 13-STABLE automatically – but most of them do. Only the biggest changes will be limited only to 14.0-RELEASE – of course probably somewhere in the middle of 2022 when it will be having its release process.

One note about ZFS filesystem on FreeBSD. People often confuse ‘real’ ZFS Boot Environments with its trying-to-be substitutes like BTRFS snapshots or snapshots used by Ubuntu with zsysctl(8) command. Unfortunately they are only snapshots and are not full writable clones (or entire separate ZFS datasets). They can freeze your system in time so you will be able to get back to working configuration after updating packages for example – but You will not be able to install other separate version of a system as other ZFS dataset making it another independent ZFS Boot Environment.

Create New ZFS Dataset

host # beadm list
BE             Active Mountpoint  Space Created
13.0.w520      NR     /           12.8G 2021-09-14 17:27
13.0.w520.safe -      -            1.2G 2021-10-18 10:01

host # zfs list -r zroot/ROOT
NAME                        USED  AVAIL     REFER  MOUNTPOINT
zroot/ROOT                 12.8G  96.8G       88K  none
zroot/ROOT/13.0.w520       12.8G  96.8G     11.6G  /
zroot/ROOT/13.0.w520.safe     8K  96.8G     11.1G  /

host # zfs create -o mountpoint=/ -o canmount=off zroot/ROOT/12.3

host # beadm list
BE             Active Mountpoint  Space Created
13.0.w520      NR     /           12.8G 2021-09-14 17:27
13.0.w520.safe -      -            1.2G 2021-10-18 10:01
12.3           -      -           96.0K 2021-10-18 13:14

Install FreeBSD 12.3-PRERELEASE

host # beadm mount 12.3 /var/tmp/12.3
Mounted successfully on '/var/tmp/12.3'

host # beadm list
BE             Active Mountpoint     Space Created
13.0.w520      NR     /              12.8G 2021-09-14 17:27
13.0.w520.safe -      -               1.2G 2021-10-18 10:01
12.3           -      /var/tmp/12.3  96.0K 2021-10-18 13:14

host # curl -o - https://download.freebsd.org/ftp/snapshots/amd64/12.3-PRERELEASE/base.txz \
         | tar --unlink -xpf - -C /var/tmp/12.3
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  173M  100  173M    0     0  1889k      0  0:01:33  0:01:33 --:--:-- 2228k

host # exa -1 /var/tmp/12.3
bin
boot
dev
etc
lib
libexec
media
mnt
net
proc
rescue
root
sbin
tmp
usr
var
COPYRIGHT
sys

host # curl -o - https://download.freebsd.org/ftp/snapshots/amd64/12.3-PRERELEASE/kernel.txz \
         | tar --unlink -xpf - -C /var/tmp/12.3
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 43.3M  100 43.3M    0     0  1733k      0  0:00:25  0:00:25 --:--:-- 1663k

host # exa -lh /var/tmp/12.3/boot/kernel/kernel
Permissions Size User Date Modified    Name
.r-xr-xr-x   37M root 2021-10-14 06:31 /var/tmp/12.3/boot/kernel/kernel

host # curl -o - https://download.freebsd.org/ftp/snapshots/amd64/12.3-PRERELEASE/lib32.txz \
         | tar --unlink -xpf - -C /var/tmp/12.3

host # exa -ld /var/tmp/12.3/usr/lib32
drwxr-xr-x - root 2021-10-18 13:45 /var/tmp/12.3/usr/lib32

Install Same Packages as on Host

With the pkg prime-list we will get all installed by hand pkg(8)packages from our currently running system. You may omit this section or just install packages that you need instead all of them.

host # pkg prime-list > /var/tmp/12.3/pkg.prime-list

host # chroot /var/tmp/12.3 /bin/sh

(BE) # export PS1="BE # "

BE # mount -t devfs devfs /dev

BE # sed -i '' s/quarterly/latest/g /etc/pkg/FreeBSD.conf

BE # pkg install -y $( cat pkg.prime-list )
Bootstrapping pkg from pkg+http://pkg.FreeBSD.org/FreeBSD:12:amd64/latest, please wait...
Verifying signature with trusted certificate pkg.freebsd.org.2013102301... done
Installing pkg-1.17.2...
Extracting pkg-1.17.2: 100%
Updating FreeBSD repository catalogue...
Fetching meta.conf: 100%    163 B   0.2kB/s    00:01
Fetching packagesite.pkg: 100%    6 MiB   1.3MB/s    00:05
Processing entries: 100%
FreeBSD repository update completed. 31294 packages processed.
All repositories are up to date.
Updating database digests format: 100%
pkg: No packages available to install matching 'chromium' have been found in the repositories
pkg: No packages available to install matching 'drm-fbsd13-kmod' have been found in the repositories
pkg: No packages available to install matching 'geany-gtk2' have been found in the repositories
pkg: No packages available to install matching 'ramspeed' have been found in the repositories
pkg: No packages available to install matching 'vim-console' have been found in the repositories

As we can see some of the packages that we have installed in the FreeBSD 13.0-RELEASE system are not currently available in the ‘latestpkg(8) branch for the FreeBSD 12.3-PRERELEASE system. This sometimes happens when the build of such package will fail – but you may assume that such package will be available in a week or so as that is the period in which pkg(8) packages are (re)build in the ‘latest‘ branch.

We will now remove the missed packages and also rename some packages that may have different names for 12.x version of FreeBSD.

BE # sed -i '' \
         -e s/drm-fbsd13-kmod/drm-kmod/g \
         -e s/geany-gtk2/geany/g \
         -e s/vim-console/vim-tiny/g \
         pkg.prime-list

BE # pkg install -y $( cat pkg.prime-list | grep -v -e chromium -e ramspeed )
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
The following 1072 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        (...)

Number of packages to be installed: 1072

The process will require 11 GiB more space.
2 GiB to be downloaded.
(...)

BE # rm pkg.prime-list

After hour or so later our packages have been installed.

BE # pkg stats
Local package database:
        Installed packages: 1073
        Disk space occupied: 11 GiB

Remote package database(s):
        Number of repositories: 1
        Packages available: 31294
        Unique packages: 31294
        Total size of packages: 96 GiB

Copy Configuration Files

You can now reboot to plain and unconfigured FreeBSD system but you may as well copy your configuration files from your current working installation. These are the files I have copied.

First files from the Base System /etc and /boot places.

host # for I in /boot/loader.conf \
                /etc/hosts \
                /etc/fstab \
                /etc/rc.conf \
                /etc/sysctl.conf \
                /etc/wpa_supplicant.conf \
                /etc/jail.conf \
                /etc/devfs.rules \
                /etc/resolv.conf
       do
         cp "${I}" /var/tmp/12.3/"${I}"
         echo "${I}"
       done
/boot/loader.conf
/etc/hosts
/etc/fstab
/etc/rc.conf
/etc/sysctl.conf
/etc/wpa_supplicant.conf
/etc/jail.conf
/etc/devfs.rules
/etc/resolv.conf

Now the files for installed packages under /usr/local/etc dir.

host # for I in /usr/local/etc/X11/xorg.conf.d/* \
                /usr/local/etc/X11/xdm/{Xresources,Xsetup_0} \
                /usr/local/etc/automount.conf \
                /usr/local/etc/sudoers \
                /usr/local/etc/doas.conf \
                /usr/local/etc/zshrc
       do
         cp "${I}" /var/tmp/12.3/"${I}"
         echo "${I}"
       done
/usr/local/etc/X11/xorg.conf.d/card.conf
/usr/local/etc/X11/xorg.conf.d/flags.conf
/usr/local/etc/X11/xorg.conf.d/keyboard.conf
/usr/local/etc/X11/xorg.conf.d/touchpad.conf
/usr/local/etc/X11/xdm/Xresources
/usr/local/etc/X11/xdm/Xsetup_0
/usr/local/etc/automount.conf
/usr/local/etc/sudoers
/usr/local/etc/doas.conf
/usr/local/etc/zshrc

Add Users and Set Passwords

You should now add your regular user and set passwords for both your user and root account.

BE # pw useradd vermaden -u 1000 -d /home/vermaden -G wheel,operator,video,network,webcamd,vboxusers

BE # passwd root

BE # passwd vermaden

Reboot Into New ZFS Boot Environment

You may now exit the chroot(8) of that ZFS Boot Environment and reboot. In the FreeBSD loader(8) menu select the 12.3 boot environment.

BE # exit

host # umount /var/tmp/12.3/dev

host # beadm unmount 12.3
Unmounted successfully

host # beadm list -D
BE             Active Mountpoint  Space Created
13.0.w520      NR     /           11.3G 2021-09-14 17:27
13.0.w520.safe -      -           11.1G 2021-10-18 10:01
12.3        -      -            9.5G 2021-10-18 13:14

host # shutdown -r now

Testing New System

The 12.3-PRERELEASE system started fine for me. I was able to login and use system as usual. One important thing to note … the ZFS pools. I have another newer ZFS pool with zstd compression enabled … and I was not able to import that ZFS pool as FreeBSD 12.3-PREELEASE does not use OpenZFS 2.0 but an older FreeBSD in-house ZFS version.

# zpool import data
This pool uses the following feature(s) not supported by this system:
        org.freebsd:zstd_compress (zstd compression algorithm support.)
        com.delphix:log_spacemap (Log metaslab changes on a single spacemap and flush them periodically.)
        org.zfsonlinux:project_quota (space/object accounting based on project ID.)
        org.zfsonlinux:userobj_accounting (User/Group object accounting.)
cannot import 'data': unsupported version or feature

Keep that in mind … but you can also install newer OpenZFS from the FreeBSD Ports and this is what we will now do.

# pkg install -y openzfs openzfs-kmod
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
The following 2 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        openzfs: 2021090800
        openzfs-kmod: 2021090800

Number of packages to be installed: 2

The process will require 22 MiB more space.
4 MiB to be downloaded.
[1/2] Fetching openzfs-2021090800.pkg: 100%    3 MiB 975.3kB/s    00:03
[2/2] Fetching openzfs-kmod-2021090800.pkg: 100%    1 MiB 591.2kB/s    00:02
Checking integrity... done (0 conflicting)
[1/2] Installing openzfs-kmod-2021090800...
[1/2] Extracting openzfs-kmod-2021090800: 100%
pkg: Cannot open /dev/null:No such file or directory
[2/2] Installing openzfs-2021090800...
[2/2] Extracting openzfs-2021090800: 100%
=====
Message from openzfs-kmod-2021090800:

--
Amend /boot/loader.conf as follows to use this module:

- change zfs_load="YES" to NO
- change opensolaris_load="YES" to NO
- add openzfs_load="YES"
- (for ARM64) add cryptodev_load="YES"
=====
Message from openzfs-2021090800:

--
Ensure that any zfs-related commands, such as zpool, zfs, as used in scripts
and in your terminal sessions, use the correct path of /usr/local/sbin/ and
not the /sbin/ commands provided by the FreeBSD base system.

Consider setting this in your shell profile defaults!

We will now have to modify our /boot/loader.conf file.

host # beadm mount 12.3 /var/tmp/12.3
Mounted successfully on '/var/tmp/12.3'

host # chroot /var/tmp/12.3

BE # cp /boot/loader.conf /boot/loader.conf.ZFS

BE # vi /boot/loader.conf

BE # diff -u /boot/loader.conf.ZFS /boot/loader.conf
--- /boot/loader.conf.ZFS       2021-10-19 10:57:04.180732000 +0000
+++ /boot/loader.conf   2021-10-19 10:57:23.992145000 +0000
@@ -12,7 +12,8 @@

 # MODULES - BOOT
   geom_eli_load=YES
-  zfs_load=YES
+  zfs_load=NO
+  openzfs_load=YES

 # DISABLE /dev/diskid/* ENTRIES FOR DISKS
   kern.geom.label.disk_ident.enable=0

BE # shutdown -r now

After reboot and trying again I was able to import that newer ZFS pool.

Hope that you will find that guide useful.

Feel free to add your suggestions.

UPDATE 1 – Notes When Installing Newer Version

This guide was written when I tried FreeBSD 12.3 on a previously used by FreeBSD 13.0 system so bootcode was not needed to be updated. I just tried 13.1 on the same 13.0 system and these two steps are needed to updated the bootcode.

UEFI

For UEFI partition you will need to copy /boot/loader.efi file from the 13.1 installation which means /var/tmp/13.1 dir. Here is the command to be used.

host # gpart show -p ada1
=>       40  250069600    ada1  GPT  (119G)
         40     409600  ada1p1  efi  (200M)          <== UEFI BOOT PARTITION
     409640       1024  ada1p2  freebsd-boot  (512K) <== BIOS BOOT PARTITION
     410664        984          - free -  (492K)
     411648    2097152  ada1p3  freebsd-swap  (1.0G)
    2508800  247560192  ada1p4  freebsd-zfs  (118G)
  250068992        648          - free -  (324K)

host # mount_msdosfs /dev/ada1p1 /mnt

host # cp /var/tmp/13.1/boot/loader.efi /mnt/efi/boot/bootx64.efi

BIOS

For the systems that boot in legacy/BIOS mode you will use this gpart(8) command instead.

host # cd /var/tmp/13.1/boot
host # pwd
/var/tmp/13.1/boot
host # gpart bootcode -b ./pmbr -p ./gptzfsboot -i 2 ada1
partcode written to ada1p2
bootcode written to ada1

As FreeBSD often is installed as BIOS+UEFI boot capable both of these steps would be needed.

EOF

Upgrade FreeBSD with ZFS Boot Environments

I am known as a strong ZFS Boot Environment supporter … and not without a reason. I have stated the reasons ‘why’ many times but most (or all) of them are condensed here – https://is.gd/BECTL – in my presentation about it.

The upcoming FreeBSD 13.0-RELEASE looks very promising. In many tests it is almost TWICE as fast as the 12.2-RELEASE. Ouch!

The detailed tests are available on the phoronix.com site.

Having 12.2-RELEASE installed I wanted to check 13.0-BETA* to check if things that are important for me – like working suspend/resume for example – work as advertised on the newer version. It is the perfect task that can be achieved by using ZFS Boot Environments.

In the example below we will create entire new ZFS Boot Environment with clone of our current 12.2-RELEASE system and upgrade it there (in BE) to the 13.0-BETA3 version … and there will only be required on reboot – not three as in typical freebsd-update(8) upgrade procedure.

I assume that you have FreeBSD 12.2-RELEASE installed with ZFS (default ZFS FreeBSD install) and its installed in UEFI or UEFI+BIOS mode.

Here are the steps that will be needed.

(host) # beadm create 13                        # create new '13' ZFS Boot Environment
       Created successfully
(host) # beadm mount 13 /var/tmp/BE-13          # mount new '13' BE somewhere
       Mounted successfully on '/var/tmp/BE-13'
(host) # chroot /var/tmp/BE-13                  # make chroot(8) into that place
  (BE) # mount -t devfs devfs /dev              # mount the devfs(8) in that BE
  (BE) # rm -rf /var/db/freebsd-update          # remove any old patches
  (BE) # mkdir /var/db/freebsd-update           # create fresh dir for patches
  (BE) # freebsd-update upgrade -r 13.0-BETA3   # fetch the patches needed for upgrade
  (BE) # freebsd-update install                 # install kernel and kernel modules
  (BE) # freebsd-update install                 # install userspace/binaries/libraries
  (BE) # pkg upgrade                            # upgrade all packages with pkg(8)
  (BE) # freebsd-update install                 # remove old libraries and files
  (BE) # exit                                   # leave chroot(8) environment
(host) # umount /var/tmp/BE-13/dev              # umount the devfs(8) in that BE
(host) # beadm activate 13                      # activate new '13' BE
       Activated successfully

I am using mine sysutils/beadm for the process but you as well may use the bectl(8) from FreeBSD base system.

We will also need new FreeBSD loader(8) which will be updated this way – thanks to @JeffSipek for pointing that out.

On my system FreeBSD is installed on ada1 device.

(host) # gpart show -p ada1 | grep efi                # find UEFI msdosfs(5) partition
               40     409600  ada1p1  efi  (200M)     # <-- this one
(host) # mount_msdosfs /dev/ada1p1 /mnt               # mount it under /mnt
(host) # find /mnt                                    # display its contents
       /mnt
       /mnt/efi
       /mnt/efi/boot
       /mnt/efi/boot/bootx64.efi                      # update bootx64.efi file
(host) # cp /boot/boot1.efi /mnt/efi/boot/bootx64.efi # copy from /boot/boot1.efi file
(host) # umount /mnt                                  # unmount /mnt filesystem

There is small chance that you will not be able to mount the efi partition. Even fsck(8) is not able to help here.

Typical errors that some users faced look like that:

(host) # mount_msdosfs /dev/ada1p1 /mnt # error when trying to mount efi partition
       mount_msdosfs: /dev/ada1p1: Invalid argument

(host) # fsck_msdosfs -y /dev/ada1p1    # error when trying to fsck(8) that partition
       ** /dev/ada1p1
       Invalid signature in boot block: 0b6a

If you hit that problem then first backup your current efi partition to for example /BACKUP.ada1p1 file.

(host) # dd < /dev/ada1p1 > /BACKUP.ada1p1 bs=1m

Now we will create fresh efi partition from scratch.

(host) # newfs_msdos -F 32 -c 1 /dev/ada0p1            # create new FAT32 partition
(host) # mount_msdosfs /dev/ada0p1 /mnt                # mount it under /mnt
(host) # mkdir -p /mnt/efi/boot                        # create needed directories
(host) # cp /boot/loader.efi /mnt/efi/boot/bootx64.efi # copy from /boot/loader.efi file
(host) # umount /mnt                                   # unmount /mnt filesystem

Now you should have new ‘working’ efi partition.

The last step is to reboot(8) into the new 13.0-BETA3 system.

(host) # reboot

If you find any problems with new bootloader not being able to load your new FreeBSD then you may alternatively copy the /boot/boot1.efi instead of /boot/loader.efi into the /mnt/efi/boot/bootx64.efi place.

Keep in mind that if you boot from geli(8)encrypted system then /boot/loader.efi is mandatory and you will not be able to boot if you would use /boot/boot1.efi file instead.

Done.

You should now see the new FreeBSD loader(8) in all its glory 🙂

You may now enjoy latest FreeBSD 13.0-BETA3 installation.

Same steps will be required to update to soon to be available FreeBSD 13.0-RC* (RC1/RC2/RC3) version and finally FreeBSD 13.0-RELEASE hopefully somewhere in March 2021.

UPDATE 1 – What if Everything Went Fine

You now have most up to date FreeBSD system that should work faster then 12.2-RELEASE and you still has your older 12.2-RELEASE Boot Environment that you can go back to if you find any problems with 13.0 version.

On my system it looks like that:

(host) # beadm list
       BE   Active Mountpoint Space Created
       12.2 -      -           6.5G 2021-02-12 10:15
       13   NR     /          18.8G 2021-02-13 11:32

The Space column is little misleading as it takes into account snapshots space used for example. To get exact information each Boot Environment takes use -D option. This way you will get information about each Boot Environment space separately.

(host) # beadm list -D
       BE   Active Mountpoint  Space Created
       12.2 -      -            9.8G 2021-02-12 10:15
       13   NR     /            9.6G 2021-02-13 11:32

I will be keeping the 12.2-RELEASE Boot Environment for a while – maybe I will delete it a month or so after 13.0-RELEASE is available but if you tested all your needs and feel that 13.0 fulfills all your needs the same way or better then 12.2-RELEASE then you may delete that older Boot Environment with below command.

(host) # beadm destroy 12.2

UPDATE 2 – What if Something Goes Wrong

Generally if the new BE named ‘13‘ does not boot (or hangs at boot) then just select your earlier Boot Environment that you used before the upgrade – the one that has 12.2-RELEASE inside it.

You now have the system that worked for you before we proceed to the upgrade process.

If that fails (or bootloader is broken) then grab the FreeBSD-13.0-BETA3-amd64-memstick.img image and write it on some pendrive with dd(8) command.

# dd if=FreeBSD-13.0-BETA3-amd64-memstick.img of=/dev/da0 bs=1M status=progress

As you now have the pendrive with FreeBSD 13.0-BETA3 then you may boot from it and fix your installation. Pick LiveCD after its loaded. Then type root at login: prompt and hit [ENTER] for empty password.

The list of tasks that can be done now depends on what is broken and I can not guess every possible error and fix scenario so if you hit any problems during that upgrade process then just contact me with your preferred way and we will figure something out.

UPDATE 3 – Faster Upgrade with New beadm(8) Version

Today (2022/05/06) I introduced new beadm(8) version 1.3.5 that comes with new chroot(8) feature. It has already been committed to the FreeBSD Ports tree under 263805 PR so expect packages being available soon.

You can also update beadm(8) directly like that:

# fetch -o /usr/local/sbin/beadm https://raw.githubusercontent.com/vermaden/beadm

Now for the faster update process – here are the instructions depending on the shell you use.

  • ZSH / CSH
# beadm create 13.1-RC6
# beadm chroot 13.1-RC6
BE # zsh || csh
BE # yes | freebsd-update upgrade -r 13.1-RC6
BE # repeat 3 freebsd-update install
BE # exit
# beadm activate 13.1-RC6
# reboot
  • SH / BASH / FISH / KSH
# beadm create 13.1-RC6
# beadm chroot 13.1-RC6
BE # sh || bash || fish || ksh
BE # yes | freebsd-update upgrade -r 13.1-RC6
BE # seq 3 | xargs -I- freebsd-update install
BE # exit
# beadm activate 13.1-RC6
# reboot

Happy upgrading 🙂

EOF

Quare FreeBSD?

I really wanted to make this article short … but I failed miserably. At least I tried to organize it well so one may get back to it after ‘some’ reading because its not a short lecture. I wanted to title it Why FreeBSD? but when you type that into your favorite duck.com search engine there are so many similar articles. I wanted it to have distinguished and unique name so I used Latin word for ‘why‘ which is ‘quare‘.

logo-freebsd

What FreeBSD can offer you that other operating systems does not? From all of the operating systems I used I find FreeBSD to suck the least. This post is not here to convince you to use or try FreeBSD – this you will have to do by yourself. This article will show you why FreeBSD is valuable or better alternative to other operating systems and is definitely not dying.

This is the Table of Contents for this article.

  • Base System
  • ZFS Boot Environments
  • Rescue
  • Audio
  • Jails
  • FreeBSD Ports Infrastructure
  • Updating/Building from Source
  • Storage
  • Init System
  • Linux Binary Compatibility
  • Simplicity
  • Evolution Instead Rewriting
  • Documentation
  • Community
  • Closing Thoughts
  • External Discussions

Base System

When you install a Linux system its just a bunch of RPM or DEB packages. For example of you install CentOS 7.8 Minimal variant you end up with several hundred RPM packages installed. After a week or month many of these packages will get updates sometimes making this CentOS system unusable or even unbootable (recent GRUB Boothole problem for example). On the contrary FreeBSD comes with a Base System concept. This means that when you install FreeBSD you install a minimal system as a whole. No packages or subsystems to be separately updated. Just whole Base System. That means that /boot /bin /sbin /usr /etc /lib /libexec /rescue directories are untouchable by any packages. When you decide to install packages (or build them using FreeBSD Ports) they will all fall into the /usr/local prefix. That means /usr/local/etc for configuration. The /usr/local/bin and /usr/local/sbin directories for binaries. The /usr/local/lib and /usr/local/libexec for libraries and so on. The FreeBSD Base System kernel modules are kept in the same dir along with the kernel in the /boot/kernel directory. To make things tidy all kernel modules that are provided by packages go into the /boot/modules dir. Everything has its place and its separated.

That is separation between Base System binaries (at /bin /sbin /usr/bin /usr/sbin dirs) and Third Party Packages maintained by pkg(8) and are located at /usr/local/bin and /usr/local/sbin dirs. We all know differences between bin (user) and sbin (root) binaries but in FreeBSD there is also another more UFS related separation. When there was only UFS filesystem in the FreeBSD world the /bin and /sbin binaries were available at boot after the root (/) filesystem was mounted and yet before /usr  filesystem was mounted – this is historical (and still useful in the UFS setups) distinction dating to old UNIX days. In ZFS setups it does not matter as all files are on ZFS pool anyway.

Illustration of the Base System / dir and optional for third party applications /usr/local dir.

base-system-dirs

The FreeBSD Base System separation also helps with another thing – if any package gets the ‘great’ idea to install new compiler named cc and override the default system compiler … or to add libraries/includes in such a way that makes it super hard to get back into a working system. If some random FreeBSD package would add libc.so to /usr/local/lib dir then you are covered and not prevented from running programs as usual because FreeBSD system binaries are linked to stuff in /usr/lib dir. This is why there is PATH variable on UNIX systems (and FreeBSD as well) to set which directories should be searched for binaries first. On FreeBSD by default its set search Base System binaries dirs first and then Third Party Packages later.

You can update (or not) the Base System separately from the installed packages with freebsd-update(8) command when using RELEASE or by recompiling with make buildworld and make installworld commands when using STABLE/CURRENT systems. When it comes to packages you can update them using the pkg(8) tool or portmaster when building from FreeBSD Ports tree under /usr/ports dir. That means that any packages updates will not touch your FreeBSD Base System at all. For example when you mess up (and I have done that in the beginning of my FreeBSD journey) the compiled ports and packages and you want to start over the only thing you have to do is remove /usr/local and /boot/modules and /var/db/pkg directories. That’s it. You are just reverted to your Base System and can start over. This is just not possible when using Linux system. Even with Gentoo that many concepts are based on FreeBSD ideas does not have Base System feature. This Base System also have additional feature. Because its separated from packages version no one stops you from running oldshool FreeBSD 9.0 from 2012 and install there latest Firefox 80 or LibreOffice 7.0. You can not install latest Firefox on Ubuntu from 2012 …

One may be ‘afraid’ that such Base System independent from installed packages would take more space but nothing far more from the truth. The fresh installed FreeBSD 12.1 system uses less then 1 GB of disk space and takes less then 75 MB of RAM with sshd(8) running. For the comparison fresh CentOS 7.8 install with ‘Minimal’ set chosen takes 1.1 GB of disk space and uses more then 100 MB RAM with sshd(8) running. Such CentOS system is really naked and really needs more packages to be usable while FreeBSD with its Base System is far more capable and powerful and comes along with builtin latest version of LLVM/CLANG compiler suite for example.

More on the Base System topic:

ZFS Boot Environments

I have talked about this many times and probably one time too less because Linux world still ignores this bless. Having ZFS Boot Environments its such a game changer that once you realize how powerful it is you will never want to use a system that does not support it. The idea is that you can snapshot a running system at any moment of time and then reboot into that moment (or snapshot) if something happened. Its perfect solution for upgrade or changes to the system. The FreeBSD systems are already well ‘protected’ from problems arising after updating the packages but ZFS Boot Environments takes this to a whole new level.

groundhog

Like in the movie Groundhog Day (1993) with ZFS Boot Environments you will have limitless chances to get your shit together. Even the Base System updates and changes are protected by it. You can even transport that Boot Environment by using zfs send and zfs recv commands to other system … or propagate it on many systems. You can create Jails containers from it … or install new version of FreeBSD in the new Boot Environment and reboot into it while still having your older ‘production’ system untouched.

More on the ZFS Boot Environments topic:

Rescue

When you really mess up to the point that even Base System concept or ZFS Boot Environments feature did not stopped you from killing your FreeBSD installation then there is one more level of rescue … the Rescue subsystem.

rescue

You have about 150 statically linked binaries available at your disposal for the rescue mission of that FreeBSD installation. You probably think now that if its so many binaries then it probably takes a lot of space … nothing far more from the truth. Its actually one static binary with hardlinks … and it takes whooping 11 MB of disk space.

FreeBSD # ls -lh /rescue | head -5
total 1118446
-r-xr-xr-x  146 root  wheel    11M 2020.02.19 21:10 [
-r-xr-xr-x  146 root  wheel    11M 2020.02.19 21:10 bectl
-r-xr-xr-x  146 root  wheel    11M 2020.02.19 21:10 bsdlabel
-r-xr-xr-x  146 root  wheel    11M 2020.02.19 21:10 bunzip2

They Rescue subsystem even contains such binaries as bectl(8) for ZFS Boot Environments management or zfs(8) and zpool(8) commands for the ZFS filesystem. Here is complete list of these binaries.

FreeBSD # ls /rescue
[           dd               fsck_ffs      init       mdmfs          ping      rtsol        unlink
bectl       devfs            fsck_msdosfs  ipf        mkdir          ping6     savecore     unlzma
bsdlabel    df               fsck_ufs      iscsictl   mknod          pkill     sed          unxz
bunzip2     dhclient         fsdb          iscsid     more           poweroff  setfacl      unzstd
bzcat       dhclient-script  fsirand       kenv       mount          ps        sh           vi
bzip2       disklabel        gbde          kill       mount_cd9660   pwd       shutdown     whoami
camcontrol  dmesg            geom          kldconfig  mount_msdosfs  rcorder   sleep        xz
cat         dump             getfacl       kldload    mount_nfs      rdump     spppcontrol  xzcat
ccdconfig   dumpfs           glabel        kldstat    mount_nullfs   realpath  stty         zcat
chflags     dumpon           gpart         kldunload  mount_udf      reboot    swapon       zdb
chgrp       echo             groups        ldconfig   mount_unionfs  red       sync         zfs
chio        ed               gunzip        less       mt             rescue    sysctl       zpool
chmod       ex               gzcat         link       mv             restore   tail         zstd
chown       expr             gzip          ln         nc             rm        tar          zstdcat
chroot      fastboot         halt          ls         newfs          rmdir     tcsh         zstdmt
clri        fasthalt         head          lzcat      newfs_msdos    route     tee          
cp          fdisk            hostname      lzma       nextboot       routed    test         
csh         fsck             id            md5        nos-tun        rrestore  tunefs       
date        fsck_4.2bsd      ifconfig      mdconfig   pgrep          rtquery   umount   

More on the Rescue topic:

Audio

Not many people expect from FreeBSD to shine in that department but it shines a lot here and not from yesterday but from decades. Remember when Linux got rid of the old OSS subsystem with one channel and came up with ‘great’ idea to write ALSA? I remember because I used Linux back then. Disaster is very polite word to describe Linux audio stack back then … and then PulseAudio came and whole Linux audio system got much worse. Back then because of that one OSS channel and many ALSA channels meant that ONLY ONE application with OSS backend could do the sound (for example WINE). But if another application would want to ‘make’ sound using OSS and you already have WINE started then it will be soundless because that one and only OSS channel was already taken. And remember that ALSA was so bad back then that KDE or GNOME made their own sound daemons mixing audio in userspace that were incompatible with each other. That means if you used KDE and GNOME apps back then you could have sound from GNOME apps but not from KDE apps or vice versa. One big fucking audio hell on Linux.

audio

Lets get back to FreeBSD audio then. What FreeBSD offered? A whooping 256 OSS channels mixed live in kernel for low latency. Everything audio related just worked out of the box – and still works today. You could have WINE or KDE/GNOME sound backends attached to their OSS channels and also ALSA apps getting their sound device without a problem. Even when you plugged a 5.1 surround system into FreeBSD it worked out of the box without any configuration and applications were able to use it immediately. That FreeBSD audio supremacy remains today as PulseAudio sound mixing in userspace while generally working incorporates large latency on Linux compared to in kernel FreeBSD mixing with low latency.

Comrade meka suggested that FreeBSD is also the only OS which has virtual_oss that allows mixing/resampling/compressing in user space and allows one to have Bluetooth headphones and USB microphone represented as single sound card.

More on the Audio topic:

Jails

The FreeBSD Jails are one of the oldest OS Level Virtualization implementations dating back to 1999. Even the Solaris Zones/Containers came five years later in 2004.

containers

After Docker was introduced in Linux the term OS Level Virtualization became less used to the Containers term and now the FreeBSD Jails along with Solaris Zones/Containers are named 1st generation containers. But that naming nomenclature change does not make FreeBSD Jails less powerful. They are also really brain dead simple to use. You just need a directory – for example /jail/nextcloud – where you will extract the FreeBSD Base System for desired release version – for example base.txz from 12.1-RELEASE and create the Jail config in the /etc/jail.conf file as shown below.

FreeBSD # mkdir -p /jail/nextcloud
FreeBSD # fetch -o - http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/12.1-RELEASE/base.txz | tar --unlink -xpJf - -C /jail/nextcloud
FreeBSD # cat /etc/jail.conf
nextcloud {
  host.hostname = nextcloud.local;
  ip4.addr = 10.0.0.100;
  path = /jail/nextcloud;
}

Now you can start you Jail right away.

FreeBSD # service jail onestart nextcloud
Starting jails: nextcloud.

Voila! Your FreeBSD Jail is already running.

FreeBSD # jls
   JID  IP Address      Hostname                      Path
     1  10.0.0.100      nextcloud.local               /jail/nextcloud

You can of course have a trimmed down version of FreeBSD Base System in the Jail if that is needed. The ZFS filesystem also helps here greatly because with zfs clone only your ‘base’ Jail will take space and only the changes you make to Jails created from it. Thanks to other FreeBSD subsystem – the Linux Binary Compatibility – you can also create a Linux Jail – for example running Devuan or Ubuntu Jail.

The FreeBSD Jails are also very lightweight. You can boot and use about 1000 FreeBSD Jails on a single FreeBSD system with 4 GB RAM.

They are also very easy to debug and troubleshoot comparing even to plain Docker – not to even mention Kubernetes which requires whole team of highly skilled people to maintain.

The FreeBSD Jails may be configured/managed only by the Base System utilities such as jls(8)/jexec(8) but you can also select from many third party Jail management frameworks. From all available ones I would choose BastilleBSD because of their modern approach and many ready to use templates for all needed use cases.

More on the Jails topic:

FreeBSD Ports Infrastructure

This is one of another examples why FreeBSD rocks that much. When you install Ubuntu or CentOS in some version there is chance that you will end up with not latest versions of packages but with versions that were quite up-to-date when this distribution version was released. Its especially visible in the CentOS world (and its upstream enterprise source system from Red Hat) where packages are quite up-to-date when .0 (dot zero) release is published but are VERY outdated when .8 or .9 incarnation of that release is available. Not to even mention that Firefox for example is released every month …

packages

As I said before when describing the FreeBSD Base System the FreeBSD Ports (and packages built from it available through pkg(8) command) are independent. That means that third party software from FreeBSD Ports is almost always up-to-date (or very close to it). You can even check it on the repology.org site for the details. Below you will find a ‘snapshot’ of the repology.org stats from time of writing this article. The ‘online’ table is very long so I copy/pasted just the systems relevant to the article.

repology

One of the other advantages of FreeBSD Ports is that it offers really MASSIVE amount of software counting 40354 ports when writing this article and still rising. Amount of ready to be installed packages are little smaller with more then 32000 available.

I once migrated for a while to OpenSolaris in 2009 on my Dell Latitude D630 laptop because I really liked all the Solaris features (including ZFS and ZFS Boot Environments that were not available on FreeBSD back then) and the OpenSolaris GNOME based desktop was pretty nice back then even with Time Slider feature for ZFS snapshots in the Nautilus file manager. I got working WiFi connection, sound was working, generally everything on my laptop was supported and working with OpenSolaris … but there was no software. Of course ‘large’ projects like GIMP or OpenOffice was available even in the default pkg(8) repository but not much else. There was less then 4000 packages back then on OpenSolaris while about 25000 packages on FreeBSD if I recall correctly.

You can also easily browse available FreeBSD Ports (and its options) on the web by using the https://freshports.org/ page.

ports-later

The count of FreeBSD Ports is one thing, the features is another. No matter which Linux distribution you are using you will find a software that was compiled and shipped without that needed flag that you desperately need. If you find such software on FreeBSD it ‘hurts’ only for a moment because you can VERY EASILY recompile that software with needed options and replace that ‘default’ package with yours. For example the FreeBSD project is afraid to provide packages of Lame because of existing MP3 patents, so multimedia/ffmpeg package is built without MP3 support (with --disable-libmp3lame flag). That is why I have my own audio/lame and multimedia/ffmpeg packages built with my configure options and that is very easy to achieve. You need to go to the /usr/ports/multimedia/ffmpeg dir type make config and select [x] LAME at the ncurses dialog. Your chosen options will be saved as plain /var/db/ports/multimedia_ffmpeg/options file. If you remove that file (or type make rmconfig) then these custom options will reset to defaults. Then you type make build deinstall install clean and your port with new options is ready and installed as package. Nothing more is needed. You can even lock that package from the pkg(8) upgrades with pkg lock -y ffmpeg command so it will not be modified later but its better to rebuild such packages everytime you do a pkg upgrade procedure because of libraries versions bump and changes. While its very easy and fast to create a script with these commands to make it more automated you can also use other parts of the FreeBSD Ports infrastructure – enter Poudriere (or Synth) – more on that in the next part.

You also do not have to configure each port that way (which could be PITA for large amount of ports) but you may specify your needed (OPTIONS_SET) or unwanted (OPTIONS_UNSET) parameters only once globally using the /etc/make.conf file. You can also specify which default versions of software you want to use, for example Apache 2.2 instead of 2.4 and PHP 7.0 instead of 7.2. You can find all default versions in the /usr/ports/Mk/bsd.default-versions.mk file. Once you setup these options you can build/rebuild or update your packages from FreeBSD Ports by portmaster(8) tool. Like on Gentoo Linux with USE flags. But this is the original. Gentoo took all/most of its ideas from FreeBSD system and its Ports infrastructure.

The Poudriere is a build framework that uses FreeBSD Ports and FreeBSD Jails to build requested packages in clean reproducible way. You can create whole new binary package repository for pkg(8) command to use with it. I mentioned Synth because while Poudriere is often used to produce whole package repository the Synth is usually used just to rebuild several packages that does not fit your needs.

There is one important things about FreeBSD Ports that is often misunderstood by newcomers. What is the difference between the Ports and packages that are fetched and installed by pkg(8) tool? Its quite simple. A package is just a build and installed port. Nothing more or less. When you use the binary packages using pkg(8) command you are using packages that someone (the FreeBSD project in that case) built for you from the FreeBSD Ports in some point in time. While FreeBSD strives to maintain as up-to-date built packages as possible its the nature of FreeBSD Ports that they are always more up-to-date then the built packages. That is why you may build and install a new version of needed packages by yourself using FreeBSD Ports. One may think of such usage when it comes to security holes. When some locally executed commands (like file(1) for example) has a security hole then its not critical for you to update it as fast as possible because that security hole can be harmless for you, but when new version of Firefox fixes very important security hole then its better to update from FreeBSD Ports version faster because waiting 2 days for the package to be built (along with other packages) can be too long.

More on the FreeBSD Ports topic:

Updating/Building from Source

While the FreeBSD Ports infrastructure is for third party software the FreeBSD Base System (or its parts) also can be easily and convenient build from source. The FreeBSD kernel config is also very small and simple. While Linux kernel config contains thousands of options – 4432 for example in the default CentOS 8.2 install the FreeBSD GENERIC config has about 20 times options less – only 260 options. But that does not saturate the topic. You can start with MINIMAL FreeBSD kernel config which has only 75 options specified.

Linux # grep -c '^CONFIG' /boot/config-$( uname -r )
4432

FreeBSD # grep -c -E '^(device|options)' /usr/src/sys/amd64/conf/GENERIC
260

FreeBSD # grep -c -E '^(device|options)' /usr/src/sys/amd64/conf/MINIMAL
75

… and its not only about smaller amount of options. Can you tell my how many steps (and which ones are required) to rebuild CentOS or Ubuntu for example without Bluetooth support?

code

On the contrary its very simple (and fast) on the FreeBSD side. While /etc/make.conf file is used to enable/disable Ports options the /etc/src.conf file is used to enable/disable FreeBSD Base System options while building it from source. To build FreeBSD without Bluetooth support just add WITHOUT_BLUETOOTH=yes to the /etc/src.conf file and type these to build it:

FreeBSD # beadm create safe
FreeBSD # cd /usr/src
FreeBSD # make buildworld kernel
FreeBSD # reboot
FreeBSD # cd /usr/src
FreeBSD # etcupdate -p      # // IN THE PAST: mergemaster -p
FreeBSD # make installworld
FreeBSD # etcupdate -B      # // IN THE PAST: mergemaster -iU
FreeBSD # reboot

Voila! You now have FreeBSD without Bluetooth support … and if any of the steps failed or because of your lack of experience/expertise your FreeBSD system does not boot or is broken you can use tools from /rescue to try to fix it (or at least figure out what is broken) and when you do not want to cope with this jest select safe ZFS Boot Environment at the FreeBSD loader(8) to boot to the system before you started building modified version of FreeBSD. Yes, You are bulletproof here. While having 294 WITHOUT_X options and 125 WITH_X options you can really tune FreeBSD Base System to your needs.

FreeBSD # zgrep -c WITHOUT_ /usr/share/man/man5/src.conf.5.gz
294

FreeBSD # zgrep -c WITH_ /usr/share/man/man5/src.conf.5.gz
125

The big downside of updating FreeBSD by source is that you can not use the freebsd-update tools to do it … but nothing stops you from creating your own FreeBSD Update Server so you will be able to use freebsd-update by adding updates using a CURRENT or STABLE system instead of RELEASE. That process is described in the Build Your Own FreeBSD Update Server article of official FreeBSD documentation.

More on the FreeBSD Source Updates/Builds topic:

Storage

Storage is one of the parts where FreeBSD really shines. Lots of people adore FreeBSD for well integrated ZFS filesystem and its really true. ZFS in FreeBSD has always been first class citizen. Lately OpenZFS 2.0 has been also integrated from the upstream joint FreeBSD and Linux repository. More and more FreeBSD features and solutions are using ZFS features.

openzfs

Most of these people that like integrated ZFS in FreeBSD do not know about the FreeBSD GEOM modular disk transformation framework which provides various storage related features and utilities like software RAID0/RAID1/RAID10/RAID3/RAID5 configurations or transparent encryption of underlying devices with GELI/GDBE (like LUKS on Linux). It also allows transparent filesystem journaling for ANY filesystem with GJOURNAL (yes also for FAT32 or exFAT) or allows one to export block devices over network with GEOM GATE devices (like NFS for block devices).

storage

FreeBSD also has its own FUSE implementation which allows all these FUSE based filesystems to work natively on FreeBSD. While lots of Linux folks know DRBD probably very few of them knew that FreeBSD comes with its own DRBD like solution called HAST – which does exactly the same thing. While ZFS has a lot features and possibilities FreeBSD still maintains and develops fast and small memory footprint UFS filesystem which today is used either with Soft Updates (SU) or Journaled Soft Updates (SUJ) depending on the use case. For example 10 TB data on UFS filesystem with Journaled Soft Updates (SUJ) takes about 1 minute under fsck(8). These storage solutions are available from FreeBSD Base System alone. The FreeBSD Ports offers much more with distributed filesystems solutions such as CEPH, LeoFS, LizardFS or Minio for Amazon S3 compatible storage.

More on the Storage topic:

Init System

FreeBSD offers really simple yet very powerful init system. It has system wide config under /etc/rc.conf file when you can enable/disable needed services with service_enable=YES and service_enable=NO stanzas. You do not even need to launch vi(1) to add them – just type sysrc service_enable=YES and they are added to the /etc/rc.conf file. There are also default values and services that are enabled and you will find them – along with many comments – in the /etc/defaults/rc.conf file. Each FreeBSD service file has PROVIDE/REQUIRE stanzas which are then used to automatically order the services to start. Services that can be run in parallel are started in parallel to save time. For example its pointless to start sshd(8) daemon without network. To start or stop the service you need to type service sshd start or service sshd stop command. But when a service is not enabled in the /etc/rc.conf file then you need to used add onestart and onestop instead. The Base System separation remains here as FreeBSD Base System services are located at /etc/rc.d directory and third party applications from ports/packages are kept under /usr/local prefix which means /usr/local/etc/rc.d dir.

When using systemd(1) you never know how the services gonna start because it will be different each time. Zero determinism. On FreeBSD you know exactly which services will start when because they are always ordered in the same state according to the PROVIDE/REQUIRE stanzas. FreeBSD also offers tools that will tell you the exact order – rcorder(8) – which can be used for all services, Base System services or third party services separately. There is also service -r command that will show you what was the order at the boot time.

FreeBSD # rcorder /etc/rc.d/* | head
/etc/rc.d/growfs
/etc/rc.d/sysctl
/etc/rc.d/hostid
/etc/rc.d/zvol
/etc/rc.d/dumpon
/etc/rc.d/ddb
/etc/rc.d/geli
/etc/rc.d/gbde
/etc/rc.d/ccd
/etc/rc.d/swap

FreeBSD # rcorder /usr/local/etc/rc.d/* | tail
/usr/local/etc/rc.d/hald
/usr/local/etc/rc.d/git_daemon
/usr/local/etc/rc.d/fscd
/usr/local/etc/rc.d/cupsd
/usr/local/etc/rc.d/cups_browsed
/usr/local/etc/rc.d/clamav-clamd
/usr/local/etc/rc.d/clamav-milter
/usr/local/etc/rc.d/clamav-freshclam
/usr/local/etc/rc.d/avahi-dnsconfd
/usr/local/etc/rc.d/aria2

FreeBSD # rcorder /etc/rc.d/* /usr/local/etc/rc.d/* 2> | grep -C 3 sshd
/etc/rc.d/ubthidhci
/etc/rc.d/syscons
/etc/rc.d/swaplate
/etc/rc.d/sshd
/etc/rc.d/cron
/etc/rc.d/jail
/etc/rc.d/localpkg

Adding new service to FreeBSD is also very easy as template for new service is very small and simple.

#!/bin/sh

. /etc/rc.subr

name=dummy
rcvar=dummy_enable

start_cmd="${name}_start"
stop_cmd=":"

load_rc_config $name
: ${dummy_enable:=no}
: ${dummy_msg="Nothing started."}

dummy_start()
{
	echo "$dummy_msg"
}

run_rc_command "$1"

If its not simple enough for you there is dedicated FreeBSD article about writing them – Practical rc.d Scripting in BSD – available here.

More on the Init System topic:

Linux Binary Compatibility

While Linux can not be FreeBSD – the FreeBSD can be Linux – and its not some slow emulation – its implementation of Linux system calls. There was time when enterprises used to work with Linux only applications (not available on FreeBSD by then) using the Linux Binary Compatibility on FreeBSD because it was faster then running them natively on Linux – FreeBSD Used to Generate Spectacular Special Effects – an official FreeBSD Press Release about FreeBSD being used to generate spacial effects to the one of the best movies of all time – The Matrix (1999).

matrix

Today the LINUX_COMPAT is also natively fast and allows one to run Linux applications – even Linux games in X11 with hardware acceleration for graphics. Think of it as WINE but for Linux applications. It lives under /compat/linux directory. It even implements Linux /proc virtual filesystem which can be mounted at the /compat/linux/proc dir but its not mandatory. For any software that does not come with source code and works on Linux the Linux Binary Compatibility saves the day. For example the f.lux project. Before I got to know Redshift I used f.lux Linux binary using LINUX_COMPAT to suppress blue spectrum light from my FreeBSD screen. The Linux Binary Compatibility subsystem can also be used to run Linux bases FreeBSD Jails – with Devuan for example.

More on the Linux Binary Compatibility topic:

Simplicity

FreeBSD is simple but not coarse/ornery. For example as Linux the FreeBSD system also supports the /proc virtual filesystem but on FreeBSD its optional and not used by default while Linux could not live without it. But while Linux has mandatory /proc it also has another virtual filesystem residing under /sys … but why Linux people need two different virtual filesystems with similar purposes? Why they could not create everything under /proc as it already existed. That is big enigma for my sanity.

But /sys is not the end of that madness. Its just a beginning.

What about these?

  • securityfs
  • devpts
  • cgroup
  • pstore
  • bpf
  • configfs
  • selinuxfs
  • systemd-1
  • mqueue
  • debugfs
  • hugetlbfs

Take a look at the FreeBSD mount(8) output after the default install on ZFS.

FreeBSD # mount
zroot/ROOT/12.1 on / (zfs, local, noatime, nfsv4acls)
devfs on /dev (devfs, local, multilabel)
zroot/tmp on /tmp (zfs, local, noatime, nosuid, nfsv4acls)
zroot/var/mail on /var/mail (zfs, local, nfsv4acls)
zroot/usr/home on /usr/home (zfs, local, noatime, nfsv4acls)
zroot/var/crash on /var/crash (zfs, local, noatime, noexec, nosuid, nfsv4acls)
zroot/var/log on /var/log (zfs, local, noatime, noexec, nosuid, nfsv4acls)
zroot/var/audit on /var/audit (zfs, local, noatime, noexec, nosuid, nfsv4acls)
zroot/var/tmp on /var/tmp (zfs, local, noatime, nosuid, nfsv4acls)
zroot/usr/src on /usr/src (zfs, local, noatime, nfsv4acls)
zroot/usr/ports on /usr/ports (zfs, local, noatime, nosuid, nfsv4acls)

Several ZFS datasets and one virtual devfs filesystem for /dev directory. With install on UFS it would be similar with several UFS partitions mounted instead of ZFS datasets.

Take a look at the CentOS 8.2 installation with just one physical root (/) XFS filesystem.

[root@centos8 ~]# mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,seclabel,size=919388k,nr_inodes=229847,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,seclabel)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,seclabel,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime,seclabel)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,cpuset)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,memory)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,blkio)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,hugetlb)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,net_cls,net_prio)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,cpu,cpuacct)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,freezer)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,perf_event)
cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,rdma)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,pids)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,devices)
configfs on /sys/kernel/config type configfs (rw,relatime)
/dev/sda1 on / type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
selinuxfs on /sys/fs/selinux type selinuxfs (rw,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17309)
mqueue on /dev/mqueue type mqueue (rw,relatime,seclabel)
debugfs on /sys/kernel/debug type debugfs (rw,relatime,seclabel)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,seclabel,pagesize=2M)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=187088k,mode=700)

Fuck me. Its even really hard to just find any REAL filesystem there … fortunately we can ask for only XFS filesystems to display.

[root@centos7 ~]# mount -t xfs
/dev/sda1 on / type xfs (rw,relatime,seclabel,attr2,inode64,noquota)

Lets get on the networking now. Lets assume that you want to make standard enterprise networking setup on a physical server with two interfaces aggregated together into highly available interface bond0 (lagg0 on FreeBSD) and then you want to put VLAN tag and IP address on that VLAN. The CentOS 7.x/8.x installer (Anaconda) will welcome you with this mess.

[root@centos7 ~]# ls -1 /etc/sysconfig/network-scripts/ifcfg-*
ifcfg-Bond_connection_1
ifcfg-eno49
ifcfg-eno49-1
ifcfg-eno50
ifcfg-eno50-1
ifcfg-VLAN_connection_1

[root@centos7 ~]# cat /etc/sysconfig/network-scripts/ifcfg-Bond_connection_1
DEVICE=bond0
BONDING_OPTS="miimon=1 updelay=0 downdelay=0 mode=active-backup"
TYPE=Bond
BONDING_MASTER=yes
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=dhcp
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_PRIVACY=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME="Bond connection 1"
UUID=ca85417f-8852-43bf-96ee-5bd3f0f83648
ONBOOT=yes

[root@centos7 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eno49
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=dhcp
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=eno49
UUID=2f60f50b-38ad-492a-b90a-ba736acf6792
DEVICE=eno49
ONBOOT=no

[root@centos7 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eno49-1
HWADDR=xx:xx:xx:xx:xx:xx
TYPE=Ethernet
NAME=eno49
UUID=342b8494-126d-4f3a-b749-694c8c922aa1
DEVICE=eno49
ONBOOT=yes
MASTER=bond0
SLAVE=yes

[root@centos7 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eno50
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=dhcp
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=eno50
UUID=4fd36e24-1c6d-4a65-a316-7a14e9a92965
DEVICE=eno50
ONBOOT=no

[root@centos7 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eno50-1
HWADDR=xx:xx:xx:xx:xx:xx
TYPE=Ethernet
NAME=eno50
UUID=a429b697-73c2-404d-9379-472cb3c35e06
DEVICE=eno50
ONBOOT=yes
MASTER=bond0
SLAVE=yes

[root@centos7 ~]# cat/etc/sysconfig/network-scripts/ifcfg-VLAN_connection_1
VLAN=yes
TYPE=Vlan
PHYSDEV=ca85417f-8852-43bf-96ee-5bd3f0f83648
VLAN_ID=601
REORDER_HDR=yes
GVRP=no
MVRP=no
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=none
IPADDR=10.20.30.40
PREFIX=24
GATEWAY=10.20.30.1
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_PRIVACY=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME="VLAN connection 1"
UUID=90f7a9bb-1443-4adf-a3eb-86a03b23ecfb
ONBOOT=yes

For the record – I have chosen ‘STATIC’ IPv4 address but installer made these interfaces to use DHCP and that STATIC address. That could be a bug but lets get to the point.

After manual fixing with vi(1) (and hour later) this is how it supposed to look.

[root@centos7 ~]# cat /etc/sysconfig/network
GATEWAY=10.20.30.1
NOZEROCONF=yes

[root@centos7 ~]# ls -1 /etc/sysconfig/network-scripts/ifcfg-*
ifcfg-bond0
ifcfg-bond0.601
ifcfg-eno49
ifcfg-eno50

[root@centos7 ~]# cat /etc/sysconfig/network-scripts/ifcfg-bond0
DEVICE=bond0
BONDING_OPTS="miimon=1 updelay=0 downdelay=0 mode=active-backup"
TYPE=Bond
BONDING_MASTER=yes
BOOTPROTO=none
IPV4_FAILURE_FATAL=no
IPV6INIT=no
ONBOOT=yes

[root@centos7 ~]# cat /etc/sysconfig/network-scripts/ifcfg-bond0.601
VLAN=yes
TYPE=Vlan
VLAN_ID=601
DEVICE=bond0.601
REORDER_HDR=yes
GVRP=no
MVRP=no
BOOTPROTO=none
IPADDR=10.20.30.40
PREFIX=24
IPV4_FAILURE_FATAL=no
IPV6INIT=no
ONBOOT=yes

[root@centos7 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eno49
BOOTPROTO=none
IPV4_FAILURE_FATAL=no
IPV6INIT=no
TYPE=Ethernet
NAME=eno49
DEVICE=eno49
ONBOOT=yes
MASTER=bond0
SLAVE=yes

[root@centos7 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eno50
BOOTPROTO=none
IPV4_FAILURE_FATAL=no
IPV6INIT=no
TYPE=Ethernet
NAME=eno50
DEVICE=eno50
ONBOOT=yes
MASTER=bond0
SLAVE=yes

Better … but still takes A LOT OF SPACE and several files to cover that quite simple setup. Not to mention its level of complication and making that very error prone way. The same configuration on FreeBSD would take just 7 lines within single /etc/rc.conf file as shown below.

FreeBSD # cat /etc/rc.conf
ifconfig_fxp0="up"
ifconfig_fxp1="up"
cloned_interfaces="lagg0"
ifconfig_lagg0="laggproto failover laggport fxp0 laggport fxp1"
vlans_lagg0="601"
ifconfig_lagg0_601="inet 10.20.30.40/24"
defaultrouter="10.20.30.1"

What about the boot process? FreeBSD boots from root on ZFS partition with just small 512 KB not mountable partition. No separate /boot device is needed. On the other side Linux always needs that separate /boot partition filled with GRUB modules. No matter if its ZFS or LVM. That is why implementation of ZFS Boot Environments is quite complicated on Linux because even if you have root on ZFS on a Linux system there is still unprotected /boot filesystem that can not be snapshoted with ZFS and has to be protected in old classic way which kill the idea of ZFS Boot Environments or Linux.

FreeBSD is really simple and well thought operating system. But also a very underestimated one.

Evolution Instead Rewriting

How many Linux tools or subsystems are abandoned or superseded by new ones? Why the ifconfig(8) command was not updated with new options and instead a new ip(8) command was introduced? Same with netstat(8) being replaced by ss(8). Same with arp(8)/iwconfig/route(8) and many more. What about whole init system? The Linux world has been taken over by systemd(1) whenever you like it or not. Even distributions that have grown their mature init systems like Ubuntu with its Upstart has moved to systemd(1) altogether. The distributions that do not use it are very few and considered a niche today.

evolution

In the FreeBSD land on the contrary such things happen only if there is no other way to implement new things. Its the last thing wanted in the FreeBSD. FreeBSD evolves and is developed with stability and backward compatibility in mind. Userland tools are grown and updated with new options instead of rewriting them over and over again. Not to mention how many new bugs are introduced by changing one tool to another.

More on the Evolution Instead Rewriting topic:

Documentation

Having system that can do almost anything but not knowing how to do that makes that system pretty useless (or at least pretty PITA to use). FreeBSD offers second to none documentation that is actively maintained and updated. Along with its legendary FreeBSD Handbook and FreeBSD FAQ the FreeBSD project also offers official FreeBSD Articles about various FreeBSD topics. The Man Pages are also very detailed and contain many examples. There is also FreeBSD Wiki page for work in progress documentation and ideas related to FreeBSD development and if you have any problems or questions related to FreeBSD there are official FreeBSD Forums and oldschool Mailing Lists available.

documentation

These were only the official project knowledge sources but there are also lots of FreeBSD books. You may also check my dedicated FreeBSD Books article for more in depth look about available FreeBSD books. Here are the best and up-to-date ones.

  • Absolute FreeBSD – Complete Guide to FreeBSD – 3nd Edition (2019)
  • Beginning Modern Unix (2018)
  • Book of PF – 3rd Edition (2015)
  • Design and Implementation of FreeBSD 11 Operating System – 2nd Edition (2015)
  • FreeBSD Device Drivers (2012)
  • FreeBSD Mastery – ZFS (2015)
  • FreeBSD Mastery – Advanced ZFS (2016)
  • FreeBSD Mastery – Storage Essentials (2014)
  • FreeBSD Mastery – Specialty Filesystems (2015)
  • FreeBSD Mastery – Jails (2019)

There are also two magazines that are dedicated to BSD and FreeBSD systems. Both are free and cover lots of interesting topics regarding FreeBSD.

With all this knowledge and support its really hard not to achieve what you need/want with FreeBSD system.

Community

Last but not least and I would say its even more important then good documentation (which FreeBSD has awesome). People that use FreeBSD do that consciously and are often experienced not only in FreeBSD land but also in topics related to other UNIX systems. Often they took long road of first using the Linux systems before finally setting on the FreeBSD land or they still do Linux administration for a living while resting using far more reasonable and sensible FreeBSD solution. I always find FreeBSD Community helpful and friendly. Always willingly helpful – especially towards newcomers. Even when you try to ‘force’ FreeBSD people to ‘fight’ in unjust/doubtful discussion they will reply with dignity and technical arguments instead of yelling at you.

The FreeBSD project even made several articles and Handbook chapters especially for Linux newcomers – or sometimes called systemd(1) refugees.

Closing Thoughts

I tried really hard to not make it a Linux rant but some may feel it that way – if so please remember that this was not my intention. FreeBSD like Linux and like any other operating system has its ups and downs. Hope that I showed you most interesting FreeBSD parts. I may add new sections here without a warning in the future 🙂

External Discussions

Discussions and comments from ‘external’ sources are available here:

EOF

FreeBSD Enterprise Storage at PBUG

Yesterday I was honored to give a talk about FreeBSD Enterprise Storage at the Polish BSD User Group meeting.

You are invited to download the PDF Slides – https://is.gd/bsdstg – available here.

bsdstg

The PBUG (Polish BSD User Group) meetings are very special. In “The Matrix” movie (which has been rendered on FreeBSD system by the way) – FreeBSD Used to Generate Spectacular Special Effects – details available here – its not possible to describe what the Matrix really is, one has to feel it. Enter it. The same I can tell you about the PBUG meetings. Its kinda like with the “Hangover” movie. What happens in Vegas PBUG meeting stays in Vegas PBUG meeting 🙂

If you will have the possibility and time then join the next Polish BSD User Group meeting. You will not regret it :>

UPDATE 1 – Shorter Unified Version

The original – https://is.gd/bsdstg – presentation is 187 pages long and is suited for live presentation while not the best for later ‘offline’ view.

I have created a unified version – https://is.gd/bsdstguni – with only 42 pages.

EOF

Nextcloud 17 on FreeBSD 12.1

Not so long ago – almost 2 years from now – I wrote about setting up Nextcloud 13 on FreeBSD.

Today Nextcloud is at 17 version and the configuration that worked two years ago requires some tweaks.

nextcloud-logo.png

This guide will not cover the same information that is available in earlier Nextcloud 13 on FreeBSD article like settings to run Nextcloud inside FreeBSD Jail. Please refer to that earpier article for these settings.

Today we will use these as backends for Nextcloud 17.

  • PostgreSQL 12
  • PHP 7.3
  • Nginx 1.14 (with php-fpm)
  • Memcached 1.5.19

As Nextcloud in FreeBSD packages comes with MySQL and without PostgreSQL support we will need to build it from source using FreeBSD Ports.

Settings

Let’s fetch the latest FreeBSD Ports tree.

# rm -r /var/db/portsnap
# mkdir /var/db/portsnap
# portsnap auto

Now we need to configure needed options in the /etc/make.conf file.

# cat /etc/make.conf
WRKDIRPREFIX=${PORTSDIR}/obj
DEFAULT_VERSIONS+= php=7.3
DEFAULT_VERSIONS+= pgsql=12
OPTIONS_UNSET+=    MYSQL
OPTIONS_SET+=      PGSQL
OPTIONS_SET+=      IMAGICK
OPTIONS_SET+=      PCNTL
OPTIONS_SET+=      SMB
OPTIONS_SET+=      REDIS


Packages and Ports

First we will add some basic tools and things like PostgreSQL still using FreeBSD packages to save tome time instead of compiling them.

# pkg install \
    sudo \
    portmaster \
    beadm \
    lsblk \
    postgresql12-client \
    postgresql12-server \
    nginx \
    memcached \
    php73-pecl-memcached


Now we will compile Nextcloud and its dependencies using FreeBSD Ports – but with portmaster.

# env BATCH=yes portmaster \
    databases/php73-pdo_pgsql \
    databases/php73-pgsql \
    www/nextcloud 

PostgreSQL

We will now configure the FreeBSD’s Login Class for PostgreSQL database in the /etc/login.conf file.

# cat  /etc/login.conf

postgres:\
        :lang=en_US.UTF-8:\
        :setenv=LC_COLLATE=C:\
        :tc=default:

EOF

# cap_mkdb /etc/login.conf

… and PostgreSQL settings in main FreeBSD’s configuration /etc/rc.conf file.

# grep postgresql /etc/rc.conf
postgresql_enable=YES
postgresql_class=postgres
postgresql_data=/var/db/postgres/data12

Let’s initialize the PostgreSQL database.

# /usr/local/etc/rc.d/postgresql initdb
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locales
  COLLATE:  C
  CTYPE:    en_US.UTF-8
  MESSAGES: en_US.UTF-8
  MONETARY: en_US.UTF-8
  NUMERIC:  en_US.UTF-8
  TIME:     en_US.UTF-8
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/db/postgres/data12 ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Europe/Warsaw
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.

Success. You can now start the database server using:

    /usr/local/bin/pg_ctl -D /var/db/postgres/data12 -l logfile start


As PostgreSQL database uses 8k blocks let’s set it in ZFS. We could of course create dedicated dataset for this purpose if needed.

# zfs set recordsize=8k zroot/ROOT/default

Now, let’s start the PostgreSQL database.

# /usr/local/etc/rc.d/postgresql start
2019-12-31 11:47:04.918 CET [36089] LOG:  starting PostgreSQL 12.1 on amd64-portbld-freebsd12.0, compiled by FreeBSD clang version 6.0.1 (tags/RELEASE_601/final 335540) (based on LLVM 6.0.1), 64-bit
2019-12-31 11:47:04.918 CET [36089] LOG:  listening on IPv6 address "::1", port 5432
2019-12-31 11:47:04.918 CET [36089] LOG:  listening on IPv4 address "127.0.0.1", port 5432
2019-12-31 11:47:04.919 CET [36089] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"
2019-12-31 11:47:04.928 CET [36089] LOG:  ending log output to stderr
2019-12-31 11:47:04.928 CET [36089] HINT:  Future log output will go to log destination "syslog".

We will now create PostgreSQL database for our Nextcloud instance.

# psql -hlocalhost -Upostgres
psql (12.1)
Type "help" for help.

postgres=# CREATE USER nextcloud WITH PASSWORD 'NEXTCLOUD_DB_PASSWORD';
CREATE ROLE
postgres=# CREATE DATABASE nextcloud TEMPLATE template0 ENCODING 'UNICODE';
CREATE DATABASE
postgres=# ALTER DATABASE nextcloud OWNER TO nextcloud;
ALTER DATABASE
postgres=# \q

Keep in mind to put something more sophisticated in the NEXTCLOUD_DB_PASSWORD place.

PostgreSQL Cleanup and Indexing Script

Lets automate some PostgreSQL housekeeping.

# mkdir -p /var/db/postgres/bin
# chown postgres /var/db/postgres/bin
# vi /var/db/postgres/bin/vacuum.sh

#! /bin/sh

/usr/local/bin/vacuumdb -az 1> /dev/null 2> /dev/null
/usr/local/bin/reindexdb -a 1> /dev/null 2> /dev/null
/usr/local/bin/reindexdb -s 1> /dev/null 2> /dev/null
:wq

# cat /var/db/postgres/bin/vacuum.sh
#! /bin/sh

/usr/local/bin/vacuumdb -az 1> /dev/null 2> /dev/null
/usr/local/bin/reindexdb -a 1> /dev/null 2> /dev/null
/usr/local/bin/reindexdb -s 1> /dev/null 2> /dev/null

# chown postgres /var/db/postgres/bin/vacuum.sh
# chmod +x /var/db/postgres/bin/vacuum.sh

# su - postgres -c 'crontab -e'
0 0 * * * /var/db/postgres/bin/vacuum.sh
:wq
/tmp/crontab.JMg5BfT5HV: 2 lines, 42 characters.
crontab: installing new crontab

# su - postgres -c 'crontab -l'
0 0 * * * /var/db/postgres/bin/vacuum.sh

# su - postgres -c '/var/db/postgres/bin/vacuum.sh'

Nginx

Now its time for Nginx webserver.

# chown -R www:www /var/log/nginx

# ls -l /var/log/nginx
total 3
-rw-r-----  1 www  www   64 2019.12.31 00:00 access.log
-rw-r-----  1 www  www  133 2019.12.31 00:00 access.log.0.bz2
-rw-r-----  1 www  www   64 2019.12.31 00:00 error.log
-rw-r-----  1 www  www  133 2019.12.31 00:00 error.log.0.bz2

… and its main nginx.conf configuration file.

# cat /usr/local/etc/nginx/nginx.conf
user www;
worker_processes 4;
worker_rlimit_nofile 51200;
error_log /var/log/nginx/error.log;

events {
  worker_connections 1024;
}

http {
  include mime.types;
  default_type application/octet-stream;
  log_format main '$remote_addr - $remote_user [$time_local] "$request" ';
  access_log /var/log/nginx/access.log main;
  sendfile on;
  keepalive_timeout 65;

  upstream php-handler {
    server 127.0.0.1:9000;
  }

  server {
    # ENFORCE HTTPS
    listen 80;
    server_name nextcloud.domain.com;
    return 301 https://$server_name$request_uri;
  }

  server {
    listen 443 ssl http2;
    server_name nextcloud.domain.com;
    ssl_certificate /usr/local/etc/nginx/ssl/ssl-bundle.crt;
    ssl_certificate_key /usr/local/etc/nginx/ssl/server.key;

    # HEADERS SECURITY RELATED
    add_header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload;";
    add_header Referrer-Policy "no-referrer";

    # HEADERS
    add_header X-Content-Type-Options nosniff;
    add_header X-XSS-Protection "1; mode=block";
    add_header X-Robots-Tag none;
    add_header X-Download-Options noopen;
    add_header X-Permitted-Cross-Domain-Policies none;

    # PATH TO THE ROOT OF YOUR INSTALLATION
    root /usr/local/www/nextcloud/;

    location = /robots.txt {
      allow all;
      log_not_found off;
      access_log off;
    }

    location = /.well-known/carddav {
      return 301 $scheme://$host/remote.php/dav;
    }

    location = /.well-known/caldav {
      return 301 $scheme://$host/remote.php/dav;
    }

    # BUFFERS TIMEOUTS UPLOAD SIZES
    client_max_body_size 16400M;
    client_body_buffer_size 1048576k;
    send_timeout 3000;

    # ENABLE GZIP BUT DO NOT REMOVE ETag HEADERS
    gzip on;
    gzip_vary on;
    gzip_comp_level 4;
    gzip_min_length 256;
    gzip_proxied expired no-cache no-store private no_last_modified no_etag auth;
    gzip_types application/atom+xml application/javascript application/json application/ld+json application/manifest+json application/rss+xml application/vnd.geo+json application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/bmp image/svg+xml image/x-icon text/cache-manifest text/css text/plain text/vcard text/vnd.rim.location.xloc text/vtt text/x-component text/x-cross-domain-policy;

    location / {
      rewrite ^ /index.php$request_uri;
    }

    location ~ ^/(?:build|tests|config|lib|3rdparty|templates|data)/ {
      deny all;
    }

    location ~ ^/(?:\.|autotest|occ|issue|indie|db_|console) {
      deny all;
    }

    location ~ ^\/(?:index|remote|public|cron|core\/ajax\/update|status|ocs\/v[12]|updater\/.+|oc[ms]-provider\/.+)\.php(?:$|\/) {
      fastcgi_split_path_info ^(.+\.php)(/.*)$;
      include fastcgi_params;
      fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
      fastcgi_param PATH_INFO $fastcgi_path_info;
      fastcgi_param HTTPS on;
      fastcgi_param modHeadersAvailable true;
      fastcgi_param front_controller_active true;
      fastcgi_pass php-handler;
      fastcgi_intercept_errors on;
      fastcgi_request_buffering off;
      fastcgi_keep_conn off;
      fastcgi_buffers 16 256K;
      fastcgi_buffer_size 256k;
      fastcgi_busy_buffers_size 256k;
      fastcgi_temp_file_write_size 256k;
      fastcgi_send_timeout 3000s;
      fastcgi_read_timeout 3000s;
      fastcgi_connect_timeout 3000s;
    }

    location ~ ^\/(?:updater|oc[ms]-provider)(?:$|\/) {
      try_files $uri/ =404;
      index index.php;
    }

    # ADDING THE CACHE CONTROL HEADER FOR JS AND CSS FILES
    # MAKE SURE IT IS BELOW PHP BLOCK
    location ~ \.(?:css|js|woff2?|svg|gif)$ {
      try_files $uri /index.php$uri$is_args$args;
      add_header Cache-Control "public, max-age=15778463";
      # HEADERS SECURITY RELATED
      # IT IS INTENDED TO HAVE THOSE DUPLICATED TO ONES ABOVE
      add_header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload;";
      # HEADERS
      add_header X-Content-Type-Options nosniff;
      add_header X-XSS-Protection "1; mode=block";
      add_header X-Robots-Tag none;
      add_header X-Download-Options noopen;
      add_header X-Permitted-Cross-Domain-Policies none;
      # OPTIONAL: DONT LOG ACCESS TO ASSETS
      access_log off;
    }

    location ~ \.(?:png|html|ttf|ico|jpg|jpeg)$ {
      try_files $uri /index.php$uri$is_args$args;
      # OPTIONAL: DONT LOG ACCESS TO OTHER ASSETS
      access_log off;
    }
  }
}

OpenSSL HTTPS Certificates

We will generate a certificates needed for HTTPS service for Nextcloud.

# mkdir -p /usr/local/etc/nginx/ssl

# cd /usr/local/etc/nginx/ssl

# openssl genrsa -des3 -out server.key 2048
Generating RSA private key, 2048 bit long modulus (2 primes)
............+++++
....+++++
e is 65537 (0x010001)
Enter pass phrase for server.key: SERVER_KEY_PASSWORD
Verifying - Enter pass phrase for server.key: SERVER_KEY_PASSWORD

As usual use something more sensible then SERVER_KEY_PASSWORD string here 🙂

# openssl req -new -key server.key -out server.csr
Enter pass phrase for server.key:
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:PL
State or Province Name (full name) [Some-State]:lodzkie
Locality Name (eg, city) []:Lodz
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Vermaden Enterprises Ltd.
Organizational Unit Name (eg, section) []:IT Department
Common Name (e.g. server FQDN or YOUR name) []:nextcloud.domain.com
Email Address []:vermaden@interia.pl

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:


# cp server.key server.key.orig

# openssl rsa -in server.key.orig -out server.key
Enter pass phrase for server.key.orig: SERVER_KEY_PASSWORD
writing RSA key

# ls -l /usr/local/etc/nginx/ssl
total 7
-rw-r--r--  1 root  wheel  1151 2019.12.31 12:39 server.csr
-rw-------  1 root  wheel  1679 2019.12.31 12:41 server.key
-rw-------  1 root  wheel  1751 2019.12.31 12:40 server.key.orig

# openssl x509 -req -days 7000 -in server.csr -signkey server.key -out server.crt
Signature ok
subject=C = PL, ST = lodzkie, L = Lodz, O = Vermaden Enterprises Ltd., OU = IT Department, CN = nextcloud.domain.com, emailAddress = vermaden@interia.pl
Getting Private key

# ln -s /usr/local/etc/nginx/ssl/server.crt /usr/local/etc/nginx/ssl/ssl-bundle.crt

PHP

Here is the used PHP configuration with up to 16GB files for Nextcloud.

# grep '^[^;]' /usr/local/etc/php.ini
[PHP]
max_input_time=3600
engine = On
short_open_tag = On
precision = 14
output_buffering = OFF
zlib.output_compression = Off
implicit_flush = Off
unserialize_callback_func =
serialize_precision = 17
disable_functions =
disable_classes =
zend.enable_gc = On
expose_php = On
max_execution_time = 3600
max_input_time = 30000
memory_limit = 1024M
error_reporting = E_ALL & ~E_DEPRECATED & ~E_STRICT
display_errors = Off
display_startup_errors = Off
log_errors = On
log_errors_max_len = 1024
ignore_repeated_errors = Off
ignore_repeated_source = Off
report_memleaks = On
track_errors = Off
html_errors = On
error_log = /var/log/php.log
variables_order = "GPCS"
request_order = "GP"
register_argc_argv = Off
auto_globals_jit = On
post_max_size = 16400M
auto_prepend_file =
auto_append_file =
default_mimetype = "text/html"
default_charset = "UTF-8"
doc_root =
user_dir =
enable_dl = Off
file_uploads = On
upload_max_filesize = 16400M
max_file_uploads = 64
allow_url_fopen = On
allow_url_include = Off
default_socket_timeout = 300
[CLI Server]
cli_server.color = On
[Date]
date.timezone = Europe/Warsaw
[filter]
[iconv]
[intl]
[sqlite3]
[Pcre]
[Pdo]
[Pdo_mysql]
pdo_mysql.cache_size = 2000
pdo_mysql.default_socket=
[Phar]
[mail function]
SMTP = localhost
smtp_port = 25
mail.add_x_header = On
[SQL]
sql.safe_mode = Off
[ODBC]
odbc.allow_persistent = On
odbc.check_persistent = On
odbc.max_persistent = -1
odbc.max_links = -1
odbc.defaultlrl = 4096
odbc.defaultbinmode = 1
[Interbase]
ibase.allow_persistent = 1
ibase.max_persistent = -1
ibase.max_links = -1
ibase.timestampformat = "%Y-%m-%d %H:%M:%S"
ibase.dateformat = "%Y-%m-%d"
ibase.timeformat = "%H:%M:%S"
[MySQLi]
mysqli.max_persistent = -1
mysqli.allow_persistent = On
mysqli.max_links = -1
mysqli.cache_size = 2000
mysqli.default_port = 3306
mysqli.default_socket =
mysqli.default_host =
mysqli.default_user =
mysqli.default_pw =
mysqli.reconnect = Off
[mysqlnd]
mysqlnd.collect_statistics = On
mysqlnd.collect_memory_statistics = Off
[OCI8]
[PostgreSQL]
pgsql.allow_persistent = On
pgsql.auto_reset_persistent = Off
pgsql.max_persistent = -1
pgsql.max_links = -1
pgsql.ignore_notice = 0
pgsql.log_notice = 0
[bcmath]
bcmath.scale = 0
[browscap]
[Session]
session.save_handler = files
session.save_path = "/tmp"
session.use_strict_mode = 0
session.use_cookies = 1
session.use_only_cookies = 1
session.name = PHPSESSID
session.auto_start = 0
session.cookie_lifetime = 0
session.cookie_path = /
session.cookie_domain =
session.cookie_httponly =
session.serialize_handler = php
session.gc_probability = 1
session.gc_divisor = 1000
session.gc_maxlifetime = 1440
session.referer_check =
session.cache_limiter = nocache
session.cache_expire = 180
session.use_trans_sid = 0
session.hash_function = 0
session.hash_bits_per_character = 5
url_rewriter.tags = "a=href,area=href,frame=src,input=src,form=fakeentry"
[Assertion]
zend.assertions = -1
[COM]
[mbstring]
[gd]
[exif]
[Tidy]
tidy.clean_output = Off
[soap]
soap.wsdl_cache_enabled=1
soap.wsdl_cache_dir="/tmp"
soap.wsdl_cache_ttl=86400
soap.wsdl_cache_limit = 5
[sysvshm]
[ldap]
ldap.max_links = -1
[mcrypt]
[dba]
[opcache]
opcache.enable=1
opcache.enable_cli=1
opcache.interned_strings_buffer=8
opcache.max_accelerated_files=10000
opcache.memory_consumption=128
opcache.save_comments=1
opcache.revalidate_freq=1
[curl]
[openssl] 

PHP PostgreSQL Database Settings

Below are needed to make PHP work with PostgreSQL database.

# cat /usr/local/etc/php/ext-20-pgsql.ini
extension=pgsql.so

# cat  /usr/local/etc/php/ext-20-pgsql.ini

[PostgresSQL]
pgsql.allow_persistent = On
pgsql.auto_reset_persistent = Off
pgsql.max_persistent = -1
pgsql.max_links = -1
pgsql.ignore_notice = 0
pgsql.log_notice = 0
EOF

# cat /usr/local/etc/php/ext-20-pgsql.ini
extension=pgsql.so

[PostgresSQL]
pgsql.allow_persistent = On
pgsql.auto_reset_persistent = Off
pgsql.max_persistent = -1
pgsql.max_links = -1
pgsql.ignore_notice = 0
pgsql.log_notice = 0


… and the second one.

# cat /usr/local/etc/php/ext-30-pdo_pgsql.ini
extension=pdo_pgsql.so

# cat  /usr/local/etc/php/ext-30-pdo_pgsql.ini

[PostgresSQL]
pgsql.allow_persistent = On
pgsql.auto_reset_persistent = Off
pgsql.max_persistent = -1
pgsql.max_links = -1
pgsql.ignore_notice = 0
pgsql.log_notice = 0
EOF

# cat /usr/local/etc/php/ext-30-pdo_pgsql.ini
extension=pdo_pgsql.so

[PostgresSQL]
pgsql.allow_persistent = On
pgsql.auto_reset_persistent = Off
pgsql.max_persistent = -1
pgsql.max_links = -1
pgsql.ignore_notice = 0
pgsql.log_notice = 0

PHP FPM

Now the PHP FPM daemon.

# grep '^[^;]' /usr/local/etc/php-fpm.conf
[global]
pid = run/php-fpm.pid
error_log = log/php-fpm.log
syslog.facility = daemon
include=/usr/local/etc/php-fpm.d/*.conf

# touch /var/log/php-fpm.log

# chown www:www /var/log/php-fpm.log

# grep '^[^;]' /usr/local/etc/php-fpm.d/www.conf
[www]
user = www
group = www
listen = 127.0.0.1:9000
listen.backlog = -1
listen.owner = www
listen.group = www
listen.mode = 0660
listen.allowed_clients = 127.0.0.1
pm = static
pm.max_children = 8
pm.start_servers = 4
pm.min_spare_servers = 4
pm.max_spare_servers = 32
pm.process_idle_timeout = 1000s;
pm.max_requests = 500
request_terminate_timeout = 0
rlimit_files = 51200
env[HOSTNAME] = $HOSTNAME
env[PATH] = /usr/local/bin:/usr/bin:/bin
env[TMP] = /tmp
env[TMPDIR] = /tmp
env[TEMP] = /tmp

Start Backend Services

We will now start all ‘backend’ services needed for Nextcloud.

# service postgresql start
2020-01-02 13:18:05.970 CET [52233] LOG:  starting PostgreSQL 12.1 on amd64-portbld-freebsd12.0, compiled by FreeBSD clang version 6.0.1 (tags/RELEASE_601/final 335540) (based on LLVM 6.0.1), 64-bit
2020-01-02 13:18:05.974 CET [52233] LOG:  listening on IPv6 address "::1", port 5432
2020-01-02 13:18:05.974 CET [52233] LOG:  listening on IPv4 address "127.0.0.1", port 5432
2020-01-02 13:18:05.975 CET [52233] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"
2020-01-02 13:18:06.024 CET [52233] LOG:  ending log output to stderr
2020-01-02 13:18:06.024 CET [52233] HINT:  Future log output will go to log destination "syslog".

# service postgresql status
pg_ctl: server is running (PID: 36089)
/usr/local/bin/postgres "-D" "/var/db/postgres/data12"

# service php-fpm start
Performing sanity check on php-fpm configuration:
[02-Jan-2020 13:16:50] NOTICE: configuration file /usr/local/etc/php-fpm.conf test is successful

Starting php_fpm.

# service php-fpm status
php_fpm is running as pid 52193.

# service memcached start
Starting memcached.

# service memcached status
memcached is running as pid 52273.

# service nginx start
Performing sanity check on nginx configuration:
nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok
nginx: configuration file /usr/local/etc/nginx/nginx.conf test is successful
Starting nginx.

Nextcloud Configuration

I created a link named /data to the Nextcloud data directory located at /usr/local/www/nextcloud/data place.

# ln -s /usr/local/www/nextcloud/data /data

The we use Firefox or other web browser to finish the Nextcloud configuration.

Type https://1.2.3.4 in the browser where 1.2.3.4 is your Nextcloud instance IP address.

I am sorry but the following image is in the Polish language – I forgot to change it to English … but I assume you will what to put in these fields by context.

nextcloud-setup.png

After we finish the setup we go straight to Nextcloud Overview page at https://1.2.3.4/settings/admin/serverinfoto page to see what else needs to be taken care of.

nextcloud-setup-overview.png

Two issues needs to be addressed. One is about Nginx configuration, the other is about PostgreSQL, let’s fix them.

We will add needed header to the Nginx configuration file.

# diff -u /usr/local/etc/nginx/nginx.conf.OLD /usr/local/etc/nginx/nginx.conf
--- /usr/local/etc/nginx/nginx.conf.OLD  2020-01-02 14:21:58.359398000 +0100
+++ /usr/local/etc/nginx/nginx.conf      2020-01-02 14:21:42.823426000 +0100
@@ -46,6 +46,7 @@
     add_header X-Robots-Tag none;
     add_header X-Download-Options noopen;
     add_header X-Permitted-Cross-Domain-Policies none;
+    add_header X-Frame-Options "SAMEORIGIN";

     # PATH TO THE ROOT OF YOUR INSTALLATION
     root /usr/local/www/nextcloud/;

# service nginx reload
Performing sanity check on nginx configuration:
nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok
nginx: configuration file /usr/local/etc/nginx/nginx.conf test is successful

… and update the PostgreSQL convertion.

# sudo -u www /usr/local/bin/php /usr/local/www/nextcloud/occ db:convert-filecache-bigint
Following columns will be updated:

* mounts.storage_id
* mounts.root_id
* mounts.mount_id

This can take up to hours, depending on the number of files in your instance!
Continue with the conversion (y/n)? [n] y

Viola! Both of our problems are gone now.

nextcloud-setup-overview-fixed.png

Trusted Domains

When you will enter the Nextcloud using different domain you will get a warning about that.

To add new Trusted Domain to the Nextcloud config do the following.

Here is how it looks before changes.

# grep -A 3 trusted /usr/local/www/nextcloud/config/config.php
  'trusted_domains' =>
  array (
    0 => '1.2.3.4',
  ),

We will now add nextcloud.domain.com domain.

# vi /usr/local/www/nextcloud/config/config.php

# grep -A 4 trusted /usr/local/www/nextcloud/config/config.php
  'trusted_domains' =>
  array (
    0 => '1.2.3.4',
    1 => 'nextcloud.domain.com',
  ),

You can of course add more with successive numbers.

# grep -A 5 trusted /usr/local/www/nextcloud/config/config.php
  'trusted_domains' =>
  array (
    0 => '1.2.3.4',
    1 => 'nextcloud.domain.com',
    2 => 'cloud.domain.com',
  ),

This is the end of this guide. Feel free to share your thougths 🙂

Log Rotation with Newsyslog

Newsyslog is part of FreeBSD’s base system. We will add Nextcloud and backend daemons log files to Newsyslog configuration so they will be rotated.

 
# cat  /etc/newsyslog.conf
/data/nextcloud.log                          www:www     640  7     *    @T00  JC
/usr/local/www/nextcloud/data/nextcloud.log  www:www     640  7     *    @T00  JC
/var/log/php-fpm.log                         www:www     640  7     *    @T00  JC
/var/log/nginx/error.log                     www:www     640  7     *    @T00  JC
/var/log/nginx/access.log                    www:www     640  7     *    @T00  JC
EOF

Now you will not run out of free space when logs will grow in time.

EOF