Tag Archives: zfs

ZFS on SMR Drives

The ZFS filesystem (more often called OpenZFS lately – as the project name) is a great filesystem for many purposes. From home or desktop/laptop solutions to enterprise offerings. Traditional disk drives have non overlapping magnetic tracks parallel to each other. These are PMR disks (Perpendicular Magnetic Recording). Hard disk drive manufacturers – to pack even more data into the same size platters – also offer SMR disks. In SMR disks data tracks are written to overlap part of previously written track – this results in narrower tracks and higher density. I will try to visualize this difference below using my favorite Enterprise Architect ASCII Edition software.

 PMR                    SMR

[xxx][___][___][___]   [xx[__[__[___]
[___][xxx][___][___]   [__[xx[__[___]
[___][___][xxx][___]   [__[__[xx[___]
[___][___][___][xxx]   [__[__[__[xxx]
[___][xxx][___][xxx]   [__[xx[__[xxx]
[xxx][___][___][xxx]   [xx[__[__[xxx]

12345678901234567890   12345678901234

I marked the filled blocks on both disks with xxx marks. As you can compare the below ‘size’ of the taken place the same data on SMR disk takes less physical space then on traditional PMR drives. This comes at a price through. Writes are little ‘crippled’ comparing to PMR drives. Especially heavy and random I/O writes are ‘problematic’ and slower on SMR drives … but it does not mean they are useless.


For the backup or clone purposes they are more then enough. I personally use SMR drives for my backup solutions. Its just about price/performance ratio.

Here are mine backup solutions based on the SMR drives:


How ZFS behaves on SMR drives? Very well I would say. ZFS tries to pack as much random I/O into sequential with its ZFS features – described in detail in the zpool-features(7) man page for example.

I recently tried ZFS on top of GELI encrypted partition on a 5 TB external USB SMR drive. I needed to copy little more then 3 TB of data there. I used rsync(1) for that purpose. These are the arguments I use for my rsync(1) jobs.

% rsync --modify-window=1 -l -t -r -D -v -S -H --force    \
        --progress --no-whole-file --numeric-ids --delete \
        /files/ /media/external/files/

Of course I do not write all these options by hand – I just a script wrapper for that – rsync-delete.sh – available on my scripts page.

As I started to copy files on the drive I watched the write speeds using iostat(8) and zpool-iostat(8) tools. I expected quite slow operation but even with the enabled zstd compression and AES-XTS 256bit GELI encryption I got pretty decent results.

Here are the iostat(8) results. Each line means average of 10 minutes (600 seconds). Check the speeds for da0 drive below.

% iostat 600
       tty            ada0             ada1              da0             cpu
 tin  tout KB/t  tps  MB/s  KB/t  tps  MB/s  KB/t  tps  MB/s  us ni sy in id
   1     1  513  120  59.9  29.5   39   1.1   742   65  46.8   4  8 17  2 69
   0     2  615   94  56.6  19.1   22   0.4   751   68  49.8   1  3 14  1 82
   0     0  561  106  57.9  17.9   20   0.4   760   70  52.0   1  2 14  1 82
   0     0 1015   57  56.8  18.4   16   0.3   769   68  50.9   1  3 15  1 81
   0     0 1017   57  56.3  18.5   16   0.3   757   68  50.6   1  3 14  1 81
   0     1  752   72  53.0  16.6   23   0.4   765   67  50.1   1  1 13  0 85
   0     0 1014   51  50.1  16.5   21   0.3   723   68  48.3   1  1 13  0 86
   0     0 1012   51  50.2  19.8   18   0.3   743   68  49.2   1  1 12  0 86

And here are the zpool-iostat(8) results.

% zpool iostat POOL 600
              capacity     operations     bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
POOL        3.18T  1.37T      7     56  53.5K  40.7M
POOL        3.20T  1.34T      0     57  9.01K  41.4M
POOL        3.22T  1.33T      0     47  3.29K  32.3M
POOL        3.24T  1.31T      0     47  5.59K  33.9M
POOL        3.25T  1.29T      0     43  3.39K  24.3M
POOL        3.27T  1.28T      0     42  3.01K  25.5M
POOL        3.28T  1.27T      0     44  3.14K  26.8M
POOL        3.29T  1.26T      0     42  3.49K  23.9M

The drive was attached over USB 3.0 port so there was not 35 MB/s limitation from USB 2.0 port. I would say that the results are very decent and consistent.


There are several settings that can help you squeeze maximum from these SMR drives on ZFS filesystem.

First are ZFS pool settings. You want the latest zstd compression to save some space. Also better compression means less physical bytes need to be written to the drive so less I/O operations. You should also turn atime into off state as it will not be needed. You should also increase recordsize to something really big like 1m (1 megabyte) so you will get higher compressratio and also will need to have less metadata for more ZFS blocks. Keep in mind that ZFS will still use variable block size and not only the 1m maximum. If something is smaller (like 100k) then it would take for example 80k (after applied zstd compression). You will not waste 920k here ๐Ÿ™‚

Keep in mind that most newer and larger drives use 4k blocks (instead of 512b). Sometimes its 512e method which means that drive firmware will ‘present’ device with 512b blocks while underneath these eight 512k blocks just lay down on a single 4k block. For these reasons its important to keep in mind several things.

When adding new partitions with gpart(8) remember to align them to 4k with -a 4k argument.

# gpart add -t freebsd-zfs -a 4k da0

Next – when initializing the geli(8) encryption layer – make sure you add -s 4096 argument.

# geli init -s 4096 /dev/da0p1

The last thing is ZFS pool creation with proper ashift property – it can not be changed later. On FreeBSD UNIX its done that way:

# sysctl vfs.zfs.min_auto_ashift=12
# zpool create POOL da0
# zdb -C POOL | grep ashift
                ashift: 12

If you are curious what 12 means then below table will help you:

     9  512b
    10  1k
    11  2k
    12  4k
    13  8k

Last but not least is the redundant_metadata option. By default its at all setting but its desired to set it into the most state. Do you need redundant metadata? I think not. When your single drive will fail the redundant metadata would not help and if your ZFS pool have some redundancy level like raidz or mirror then redundant metadata is also not needed because its just ‘normally’ redundant being spread across several disks.

Keep in mind that ZFS resilver process on some of these SMR drives can take forever. Some people from Reddit reported that they successfully resilvered their ZFS pools with SMR drives but that does not have to be the case for all SMR drives out there. You can also check Ars Technica tests of resilver on SMR disks.

Here is the summary of ZFS tunables suggested – you will find in depth description of all of them in the zfsprops(7) man page.

# zfs set redundant_metadata=most POOL
# zfs set compression=zstd        POOL
# zfs set atime=off               POOL
# zfs set recordsize=1m           POOL

The TRIM operations upon deletion create additional unwanted ‘stress’ for SMR drives and TRIM operations are not needed on non-SSD drives so you can disable them entirely on the ZFS pool level.

# zpool autotrim=off POOL

If needed you can manually trigger the TRIM operations with this command.

# zpool trim POOL
# zpool status POOL
  pool: POOL
 state: ONLINE
  scan: scrub repaired 0B in 02:17:22 with 0 errors on Sun May  8 05:18:22 2022

        NAME          STATE     READ WRITE CKSUM
        POOL          ONLINE       0     0     0
          da0p1.eli   ONLINE       0     0     0  (trimming)

errors: No known data errors

If you do not want to disable TRIM operations then you can limit their rate. By default the TRIM commands are executed at 64 rate. You can limit them to 1 and still have them enabled with following sysctl(8) tunable.

# sysctl vfs.zfs.vdev.trim_max_active=1

If you want to make it survive across reboots then put it into the /etc/sysctl.conf file.

Logic could suggest that simpler/older filesystems such as FreeBSD UFS for example could be more suitable solution for SMR drives … but the reality shows that not so much. Check this Reddit thread for example – Appalling Performance on External USB SMR Drive – to name just one.

Hope this article will help you get most of your SMR drives.



ZFS Compatibility

The best free filesystem on Earth – ZFS – also often named OpenZFS recently – has also become very portable in recent years of its development. The OpenZFS Distributions page lists 6 (six) operating systems already.

They are:

  • FreeBSD
  • Illumos
  • Linux
  • MacOS
  • NetBSD
  • Windows

… but if you would like to create a ZFS pool compatible with all of them … which options and ZFS features should you choose? There is OpenZFS Feature Flags page dedicated exactly to that topic.


These are the ones that have yes value in all operating systems.

  • async_destroy
  • bookmarks
  • empty_bpobj
  • enabled_txg
  • filesystem_limits
  • lz4_compress
  • hole_birth
  • multi_vdev_crash_dump
  • spacemap_histogram

I would also include these as only older NetBSD 4.0.5 version does not support them – but they are supported in newer NetBSD 5.3 version.

  • embedded_data
  • large_blocks
  • sha512
  • skein

There is also a dedicated zpool-features(7) man page for that information on the OpenZFS page and also zpool-features(7) man page on the FreeBSD page.

On the FreeBSD system the /usr/share/zfs/compatibility.d directory has files with supported ZFS Feature Flags for many major operating systems and ZFS versions.

% ls /usr/share/zfs/compatibility.d

Unfortunately it misses NetBSD and Illumos systems for example … but having information from the OpenZFS Feature Flags page we can find Feature Flags set that will be supported everywhere.

Here are the stats for supported ZFS Feature Flags. The higher the number the more operating systems and ZFS version it covers.

% grep -h '^[^#]' /usr/share/zfs/compatibility.d/* \
    | sort -n \
    | uniq -c \
    | sort -n
   2 draid
   5 bookmark_written
   5 device_rebuild
   5 livelist
   5 log_spacemap
   5 redacted_datasets
   5 redaction_bookmarks
   5 zstd_compress
   9 allocation_classes
   9 bookmark_v2
   9 project_quota
   9 resilver_defer
  10 edonr
  10 encryption
  11 large_dnode
  11 userobj_accounting
  18 spacemap_v2
  20 device_removal
  20 obsolete_counts
  20 zpool_checkpoint
  31 sha512
  31 skein
  32 multi_vdev_crash_dump
  36 filesystem_limits
  36 large_blocks
  37 embedded_data
  37 enabled_txg
  37 extensible_dataset
  37 hole_birth
  37 spacemap_histogram
  38 async_destroy
  38 empty_bpobj
  38 lz4_compress

As the GNU GRUB is very outdated when it comes to ZFS support it should be pretty bulletproof to use it as a starting point of limited ZFS Feature Flags support.

% cat /usr/share/zfs/compatibility.d/grub2
# Features which are supported by GRUB2

To make sure we are compatible we will now cross-link the GNU GRUB data.

First we will ‘generate’ the grep(1) command arguments that we will use in the next command.

% grep '^[^#]' /usr/share/zfs/compatibility.d/grub2 \
  | while read I
      echo "-e ' ${I}' \\"
-e ' async_destroy' \
-e ' bookmarks' \
-e ' embedded_data' \
-e ' empty_bpobj' \
-e ' enabled_txg' \
-e ' extensible_dataset' \
-e ' filesystem_limits' \
-e ' hole_birth' \
-e ' large_blocks' \
-e ' lz4_compress' \
-e ' spacemap_histogram' \

Lets now use these arguments to filter the ZFS features.

% grep -h '^[^#]' /usr/share/zfs/compatibility.d/grub2/* \
    | sort -n \
    | uniq -c \
    | sort -n \
    | grep -e ' async_destroy' \
           -e ' bookmarks' \
           -e ' embedded_data' \
           -e ' empty_bpobj' \
           -e ' enabled_txg' \
           -e ' extensible_dataset' \
           -e ' filesystem_limits' \
           -e ' hole_birth' \
           -e ' large_blocks' \
           -e ' lz4_compress' \
           -e ' spacemap_histogram' \
    | wc -l

% grep -h '^[^#]' /usr/share/zfs/compatibility.d/grub2 | wc -l

So if seems that GRUB list of ZFS Feature Flags seems pretty compatible.

Lets now cross-reference that GRUB data with the data from the OpenZFS Feature Flags page.

I will create new /usr/share/zfs/compatibility.d/OZFF file that has these ZFS Feature Flags as content.

% cat /usr/share/zfs/compatibility.d/OZFF

% wc -l /usr/share/zfs/compatibility.d/OZFF

So there are 11 GRUB ZFS Feature Flags and 13 OpenZFS ‘Compatible’ Feature Flags.

Lets see how it they compare.

% cat /usr/share/zfs/compatibility.d/OZFF \
    | grep -e async_destroy \
           -e bookmarks \
           -e embedded_data \
           -e empty_bpobj \
           -e enabled_txg \
           -e extensible_dataset \
           -e filesystem_limits \
           -e hole_birth \
           -e large_blocks \
           -e lz4_compress \
           -e spacemap_histogram \
    | wc -l

I expected 11 here instead of 10 … we will not have to compare GRUB results to the OpenZFS Feature Flags section.

% grep -h '^[^#]' /usr/share/zfs/compatibility.d/{grub2,OZFF} \
    | sort -n \
    | uniq -c \
    | sort -n
   1 extensible_dataset
   1 multi_vdev_crash_dump
   1 sha512
   1 skein
   2 async_destroy
   2 embedded_data
   2 empty_bpobj
   2 enabled_txg
   2 filesystem_limits
   2 hole_birth
   2 large_blocks
   2 lz4_compress
   2 spacemap_histogram

Seems that we finally got our 10 most compatible OpenZFS Feature Flags set.

Its this set:

  • async_destroy
  • bookmarks
  • embedded_data
  • empty_bpobj
  • enabled_txg
  • filesystem_limits
  • hole_birth
  • large_blocks
  • lz4_compress
  • spacemap_histogram

To make it more comfortable to use we will put them it into the separate /usr/share/zfs/compatibility.d/COMPATIBLE file.

# cat /usr/share/zfs/compatibility.d/COMPATIBLE

Lets now try to make that best effort most-compatible ZFS pool.

I will use one of my scripts – mdconfig.sh – to easy manipulate md(4) memory disks on FreeBSD.

# truncate -s 1g FILE
# mdconfig.sh -c FILE
IN: created vnode at /dev/md0
# zpool create -o compatibility=COMPATIBLE compatible /dev/md0
# zpool list
compatible   960M   116K   960M        -         -     0%     0%  1.00x    ONLINE  -
zroot        118G  48.6G  69.4G        -         -    36%    41%  1.00x    ONLINE  -

Now lets see what is zpool upgrade command showing us.

# zpool upgrade
This system supports ZFS pool feature flags.

All pools are formatted using feature flags.

Some supported features are not enabled on the following pools. Once a
feature is enabled the pool may become incompatible with software
that does not support the feature. See zpool-features(7) for details.

Note that the pool 'compatibility' feature can be used to inhibit
feature upgrades.


There are LOTS of ZFS Feature Flags to be activated but if we want to have our ZFS pool keep compatible – we will have to stay away from it ๐Ÿ™‚


You may get impression that you miss a lot … but you do not miss that much. Here are the ZFS Feature Flags you can fully utilize.

# zpool get all compatible | grep -v disabled
NAME        PROPERTY                       VALUE                          SOURCE
compatible  size                           960M                           -
compatible  capacity                       0%                             -
compatible  altroot                        -                              default
compatible  health                         ONLINE                         -
compatible  guid                           5395735446052695775            -
compatible  version                        -                              default
compatible  bootfs                         -                              default
compatible  delegation                     on                             default
compatible  autoreplace                    off                            default
compatible  cachefile                      -                              default
compatible  failmode                       wait                           default
compatible  listsnapshots                  off                            default
compatible  autoexpand                     off                            default
compatible  dedupratio                     1.00x                          -
compatible  free                           960M                           -
compatible  allocated                      116K                           -
compatible  readonly                       off                            -
compatible  ashift                         0                              default
compatible  comment                        -                              default
compatible  expandsize                     -                              -
compatible  freeing                        0                              -
compatible  fragmentation                  0%                             -
compatible  leaked                         0                              -
compatible  multihost                      off                            default
compatible  checkpoint                     -                              -
compatible  load_guid                      17463015630652190527           -
compatible  autotrim                       off                            default
compatible  compatibility                  COMPATIBLE                     local
compatible  feature@async_destroy          enabled                        local
compatible  feature@empty_bpobj            enabled                        local
compatible  feature@lz4_compress           active                         local
compatible  feature@spacemap_histogram     active                         local
compatible  feature@enabled_txg            active                         local
compatible  feature@hole_birth             active                         local
compatible  feature@extensible_dataset     enabled                        local
compatible  feature@embedded_data          active                         local
compatible  feature@bookmarks              enabled                        local
compatible  feature@filesystem_limits      enabled                        local
compatible  feature@large_blocks           enabled                        local

You get the very decent LZ4 compression and also ZFS Bookmarks feature which are very useful feature for sync|recv mechanism.

Keep in mind that the -o compatibility= switch for zpool(8) is available on OpenZFS 2.1 or newer. The 2.1 version is already available on the FreeBSD 13.1-BETA* releases and will be part of the FreeBSD 13.1-RELEASE systems. To make use of it on older FreeBSD releases you will have to use openzfs and openzfs-kmod packages and also use the following settings in the /boot/loader.conf file.

From that one:


Into that one:


With these openzfs and openzfs-kmod packages and above settings in the /boot/loader.conf file you can use this OpenZFS 2.1 on FreeBSD 12.2 – on FreeBSD 12.3 – and on FreeBSD 13.0 … and of course on upcoming FreeBSD 13.1 release.

Not sure if I should have add anything more here. Feel free to remind me in the commends ๐Ÿ™‚


ZFS Boot Environments Revolutions

I do not have to remind you that I am a big fan of ZFS Boot Environments feature. From the time when I first used it on OpenSolaris and Solaris systems I was really fascinated by it. Bulletproof upgrades and changes to entire system … and it was possible more then decade ago. Like a Dream. Today with beadm(8) and bectl(8) tools and also the FreeBSD loader(8) the ZFS Boot Environments are first class and one of the main features of the FreeBSD operating system.

Back in the more ‘normal’ times (before C19) I was able to talk two times about ZFS Boot Environments. I hope I explained them well.

  • 1st in Poland at PBUG meeting – with presentation available HERE.
  • 2nd in Holland at NLUUG conference – with presentation available HERE.

I do not know any downsides of ZFS Boot Environments but if you would stick a gun into my head and make me find one – I would say that you still have to reboot(8) to change to the other BE. This is about to change …

Reroot Instead Reboot

What is reroot? Its the ability to switch to other root filesystem without the need for full system reboot. The loaded and running kernel stays the same of course – but this is the only downside. This feature is implemented in the reboot(8) command with -r argument.

As we can read in the FreeBSD 10.3-RELEASE Release Notes page:

The initial implementation of “reroot” support has been added to the reboot(8) utility, allowing the root filesystem to be mounted from a temporary source filesystem without requiring a full system reboot. (r293744) (Sponsored by The FreeBSD Foundation)

How can reroot be useful here? It will save you a lot of time when you did not updated the kernel. There are two types of update strategies when using the ZFS Boot Environments. You can create new BE (as a backup world that you can get back to) and update the running system. Then you can use checkrestart(1) to verify which processes should be restarted because either binaries or libraries has been updated.


The other way was to create new separate BE (while not touching the running one) and then mount it and update that new BE and reboot into it later. This created a need to reboot(8) but not anymore. Especially when you just update the packages with pkg(8) command.


With the new reroot option of beadm(8) you will tell FreeBSD to reroot your running kernel into specified BE. It will definitely have less impact in virtual machines as they reboot quite fast but imagine saved time on a server class physical machine with about 10 minutes lost for BIOS POST messages and initialization … or personal desktop/laptop GELI encrypted system without the need to type in again the GELI password to decrypt it after reboot.


On the screenshot above I use the latest FreeBSD 13.1-BETA1 but it works the same on other production FreeBSD releases such as 12.3-RELEASE or 13.0-RELEASE. The new upgraded beadm(8) is available from its home at GitHub page here:

I will add that updated version to the FreeBSD Ports tree later along with updated man page later.


Usage of this new feature is quite simple. You type beadm reroot BENAME in the terminal and FreeBSD reroots into that BE without reboot. Takes about 9-10 seconds on my 11 years old ThinkPad W520 so it may be even faster on your more up to date system.

# beadm list
BE        Active Mountpoint  Space Created
12.3      -      -            9.5G 2021-10-18 13:14
13.0.p6   -      -           13.9G 2022-01-27 11:07
13.0      -      -           12.9G 2022-03-05 15:02
13.0.safe -      -            2.8M 2022-03-08 14:54
13.1      NR     /            9.5G 2022-03-12 00:18
13.1.safe -      -          544.0M 2022-03-13 23:18

# beadm activate 13.1.safe
Activated successfully

# beadm reroot 13.1.safe

… and you are going the route similar to typing shutdown now on a running system. All services are stopped. Then root is changed to new one. Then system continues to boot along with starting all its services as usual. Just without the BIOS POST and the bootloader and kernel parts.

The reroot feature is especially useful in one of these scenarios:

From what I know the bectl(8) does not has that reroot feature but maybe it will be added to it somewhere in the future.


Not sure that ZFS Boot Environments Revolutions is the best title for this blog post, but as I used Reloaded on my 2nd ZFS Boot Environments presentation I though that sticking to The Matrix (1999) schema. I could of course do 3rd and updated presentation … but I am afraid that it will not happen … or at least not soon.

I did not thought that the FreeBSD Enterprise Storage presentation that I gave at 2020/02 PBUG would be my last – it was more then 2 years ago.

UPDATE 1 – Faster Upgrade with New beadm(8) Version

Today (2022/05/06) I introduced new beadm(8) version 1.3.5 that comes with new chroot(8) feature. It has already been committed to the FreeBSD Ports tree under 263805 PR so expect packages being available soon.

You can also update beadm(8) directly like that:

# fetch -o /usr/local/sbin/beadm https://raw.githubusercontent.com/vermaden/beadm

Now for the faster update process – here are the instructions depending on the shell you use.

  • ZSH / CSH
# beadm create 13.1-RC6
# beadm chroot 13.1-RC6
BE # zsh || csh
BE # yes | freebsd-update upgrade -r 13.1-RC6
BE # repeat 3 freebsd-update install
BE # exit
# beadm activate 13.1-RC6
# reboot
  • SH / BASH / FISH / KSH
# beadm create 13.1-RC6
# beadm chroot 13.1-RC6
BE # sh || bash || fish || ksh
BE # yes | freebsd-update upgrade -r 13.1-RC6
BE # seq 3 | xargs -I- freebsd-update install
BE # exit
# beadm activate 13.1-RC6
# reboot

Happy upgrading ๐Ÿ™‚


Books About FreeBSD

There are many books in which FreeBSD is covered or it is the one of the main objectives of such book. Today I will guide you through these books. I will try to focus on more up to date ones because it would be pointless (beyond historical purposes) to read them now.

The Hateful Eight

I will start with official FreeBSD documentation – as it offers 8 different books for you to get to know that UNIX operating system better. The two most known are FreeBSD Handbook and FreeBSD FAQ. The FreeBSD Handbook covers all/most general topics about FreeBSD operating system setup and administration while FreeBSD FAQ tries to answer most popular questions about it – and does it quite well. There are also other books that are more developer oriented. Below you will find the list of all available eight books from the FreeBSD project.

FreeBSD Handbook https://freebsd.org/handbook
FreeBSD FAQ https://freebsd.org/faq
FreeBSD Architecture Handbook https://docs.freebsd.org/en/books/arch-handbook/
FreeBSD Developers Handbook https://docs.freebsd.org/en/books/developers-handbook/
FreeBSD Porters Handbook https://docs.freebsd.org/en/books/porters-handbook/
Design and Implementation of 4.4BSD Operating System https://docs.freebsd.org/en/books/design-44bsd/
Project Model for FreeBSD https://docs.freebsd.org/en/books/dev-model/
FreeBSD Documentation Project Primer for New Contributor https://docs.freebsd.org/en/books/fdp-primer/

Example FreeBSD Handbook page from PDF file below.


… and while you can download and read PDF files (there are also EPUB/HTML/TXT formats available for download) you can also read it online – this is how the FreeBSD FAQ looks online.


I sometimes miss that FreeBSD project does not deliver dedicated book in the same PDF/EPUB/HTML/TXT manner for the FreeBSD man pages as for example Solaris or AIX does. At least you can read them online on the https://man.freebsd.org/command page where you will put the needed man page in the place of ‘command’ word – for example the manual page for gstat(8) command is available at https://man.freebsd.org/gstat URL.

If there are two man pages with the same name like crontab(1) and crontab(5) for example then add the man page section number after the slash (/) at the end of URL like that https://man.freebsd.org/crontab/5 – this will solve that problem.

Below you can see the Solaris 10 – Man Pages Section 1M – System Administration Commands book with all Solaris administrative (1M) commands. For FreeBSD the administrative commands are at (8) section.


The Usual Suspects

The most known books that cover FreeBSD operating system aspects are the ones that are authored by Michael W. Lucas (in more recent titles sometimes accompanied by Allan Jude).

The most famous (and useful) one is the Absolute FreeBSD – Complete Guide to FreeBSD in its most recent 3nd Edition. It is quite recent as its from 2019 year. You can get it both in digital (PDF/EPUB) and traditional printed way. If you already read the FreeBSD Handbook and FreeBSD FAQ are wondering what you should get next to continue your FreeBSD journey then this book is the answer. It fill all the gaps and bring a lot of additional information that you will find very useful in your day to day life with FreeBSD system.


Several books later (not only technical) Michael W. Lucas started the FreeBSD Mastery series with many interesting books about FreeBSD. For the two that cover the ZFS filesystem Allan Jude is also a coauthor.


Here they are:

  • FreeBSD Mastery: Storage Essentials (2014)
  • FreeBSD Mastery: Specialty Filesystems (2015)
  • FreeBSD Mastery: ZFS (2015)
  • FreeBSD Mastery: Advanced ZFS (2016)
  • FreeBSD Mastery: Jails (2019)

If you are gonna read them – then also do it in that order sa they are listed above. You will need all that ‘introduction’ to get the most of FreeBSD Jails. Even Michael W. Lucas mentioned that you need to ‘do’ several other of his books to truly take advantage of all things written in the FreeBSD Mastery: Jails book. While the FreeBSD Mastery: Storage Essentials and FreeBSD Mastery: Specialty Filesystems are very close related to FreeBSD operating system the other two FreeBSD Mastery: ZFS and FreeBSD Mastery: Advanced ZFS have also a lot of general ZFS knowledge not only limited to FreeBSD operating system.

One additional word about the FreeBSD Mastery: Jails book as its quite ‘special’ in approach. For most parts of the FreeBSD operating system the FreeBSD Handbook covers some or most of the information and tasks needed to do convening particular topic. When it comes to FreeBSD Jails its not that simple anymore. There are two types of Jails. The ‘traditional’ Jails that use the host system network stack and the new ‘VNET’ Jails that bring their own – separate from the host – network stack. It gives you a lot more possibilities and features but it comes with only one downside. The official FreeBSD Handbook does not cover the ‘VNET’ Jails at all. Zero. Nada. Zip. None. You can get ‘some’ grasp about them from the FreeBSD man pages but that is definitely not enough. The ‘VNET’ Jails are of course production ready from many years but for some reason the missing chapter in the FreeBSD Handbook is still missing. That is the most important reason why you should get the FreeBSD Mastery: Jails book.

As we are in the FreeBSD Jails scope … I should also mention the FreeBSD Jails using VNETs book by Derik Ramirez from 2020.


Its not as ‘big’ as the FreeBSD Mastery: Jails but keep in mind that half of Michael W. Lucas content is about the iocage(8) framework … which is quite dead now unfortunately. The biggest upside of FreeBSD Jails using VNETs book is that it covers – as described in the book title – the VNET Jails. Also being released in 2020 its very up to date.

The Taste of Others

Written in 2018 by Manish Jain the book Beginning Modern Unix covers both FreeBSD and Linux operating systems at the same time.


It also beginner friendly and as author writes in Preparing for Part I“The structure is intended to make things simple for Windows users planning to migrate to FreeBSD/Linux.” I also really liked the author explanation in the Preface about why he choose BSD/FreeBSD and GNU/Linux (like that syntax by the way). You can find part of it below.


To be honest I like that ‘dual’ approach with sections showing how to achieve the same on two different (yet somewhat similar) operating systems. It may be useful especially when writing playbooks for various configuration management software like Ansible or Salt. I know that most of these configuration management systems provide their own ‘general’ integrations like ‘install packages’ on a system and then it uses pkg(8) on FreeBSD and yum(8) on CentOS but sometimes not all integrations are available or they are also sometimes broken or buggy. I have heard several times this already that its safer to rely on your own scripts and ‘raw’ commands instead of the ‘integration modules’ in many cases. Of course your millage may vary.

The Beginning Modern Unix book covers many desktop related topics but it also covers POSIX shell scripting and basic C programming which is very nice.

Another interesting ‘dual’ book is UNIX The Textbook from 2017 by Syed Mansoor Sarwar and Robert M. Koretsky authors.


In its 3rd Edition it covers both Solaris and FreeBSD (in a form of PC-BSD tho) systems. As Solaris by default uses GNOME and PC-BSD (while it existed) used KDE the book also covers a lot about these two desktop environments.


It is illustrated with many useful diagrams of how things work on UNIX system. Below you can check the pipe explanation.


Another good part of the book is that it also covers a lot of POSIX shell scripting and C programming techniques. Even covering things like writing simple servers, inter process communication, threads and a lot more. The book is really huge with almost 1400 pages of useful content. The book leave you at the end of each chapter with QUESTIONS AND PROBLEMS. This approach reminds me of the legendary The C Programming Language written by Brian Kernighan and Dennis Ritchie UNIX fathers where it was also present.

Next one is very well known UNIX and Linux System Administration Handbook by Evi Nemeth and Garth Snyder and Trent R. Hein and Ben Whaley … but it depends which version you get ๐Ÿ™‚


The 4th Edition is more conservative and covers Linux/Solaris/HP-UX/AIX systems. This is the description of which systems they have chosen in 2011 and why.


If you try to find FreeBSD there you will fail. They only mentioned it once and added that along with OpenBSD and NetBSD and that BSD systems “(…) enjoy somewhat less support from third-party software vendors.” See for yourself.


Lets now move 7 years forward to 2017 in which the 5th edition of the same UNIX and Linux System Administration Handbook book was released. Authors also changed a little with Dan Mackin being added to current lineup of authors of this book.


Besides modified cover I would risk a stance that its quite entirely different book right now. Its because the 5th Edition covers only Linux and … FreeBSD. No AIX. No Solaris. No HP-UX. Authors also added quite long justification on why they have chosen these operating systems and not the other ones. The time of oldschool UNIX dinosaurs that ruled for decades seem to slowly vanish. Its probably not without reason.

The last HP-UX version 11.31 (also known as 11i v3 variant) was released in 2007. Still uses ‘manual’ packages like in 1995. I like its ‘ecosystem’ tho. What I mean by that is that you can install and setup several HP-UX machines. Setup HP Serviceguard HA cluster on these machines and then make HP Virtual Machines or HP SRP Containers highly available between these hosts as Serviceguard services. I was fortunate enough to be able to see such 6 node cluster in action and it worked really well.

The last AIX version 7.2 was released in 2015 but I do not recall any ground braking features. Also AIX still does not have any modern package management and the most that IBM AIX developers could do was to adopt RPM database to add RPM packages along with the native ones. In 2021 the newer AIX 7.3 release also saw the light of day … but also without any groundbreaking features. In other words the last two AIX system releases fell only like a maintenance releases. While the HP-UX ‘ecosystem’ is ‘connected’ between their products and features its not that easy and simple in the AIX ‘POWER World’. For example its not possible to create similar setup with LPARs or WPARs that their high availability would be controlled by PowerHA cluster software. Even if you would create LPARs with purely virtual devices and storage from the SAN network. The POWER ecosystem offers a feature called Remote Restart for LPARs on HMC but its far from being close to what HP-UX ecosystem offers here.

As for Solaris … I think that Oracle taking over SUN is probably one of the worst things that could happen to Solaris. Oracle could extend and continue the OpenSolaris road started by SUN. It could move and expand Solaris to 21st century. Instead it also went the ‘maintenance’ road along with maximizing the cash outcome of Solaris ‘asset’ with 11.3 in 2015 and 11.4 in 2018 releases. Besides adding PF firewall from OpenBSD and Live Migration feature for Kernel Zones I do not recall any groundbreaking features to be added. Maybe some ZFS development but looking at what OpenZFS is achieving with each release these Oracle developments do not look ‘big’ at all. From what is known Oracle also fired most of SPARC and Solaris developers leaving only small teams to make it running in ‘maintenance mode’ up to 2034 to which Oracle promised to keep Solaris alive instead of moving forward with Solaris 12.0 which was even in alpha or beta state. I agree with authors statement in which they say that “The popularity of UNIX has been waning for some time, and most of the stalwart UNIX distributions (e.g., Solaris, HP-UX, and AIX) are no longer in common use.”

Below is the authors description why they have chosen the FreeBSD and Linux systems for the 5th Edition of their well respected and acknowledged book.


Back to FreeBSD world … and the authors quote why FreeBSD was included in this most recent version of their book – “The open source descendants of BSD are exceptions to this trend and continue to enjoy a cult following, particularly among operating system experts, free software evangelists, and security-minded administrators. In other words, some of the worldโ€™s foremost operating system authorities rely on the various BSD distributions. Appleโ€™s macOS has a BSD heritage.” This most recent version is also little shorter with ‘only’ about 1200 pages while earlier edition topped at little over 1300. Keep in mind that newer edition covers Linux and FreeBSD while the older one had to describe and document to systems more.

Last but not least I should also mention the Book of PF – No Nonsense Guide to OpenBSD Firewall book by Peter N.M. Hansteen from 2015 in its most recent 3rd Edition.


While originally targeted at OpenBSD users the FreeBSD users will also be able to get a lot of useful knowledge about PF firewall that FreeBSD uses. Keep in mind that there are some syntax differences between OpenBSD and FreeBSD PF firewalls.

Source Code

Here you will find the books that are little less useful for sysadmins and more useful for developers and programmers. We will start with updated Design and Implementation of FreeBSD 11 Operating System from 2015 in 2nd Edition form. Written by one of the original BSD UNIX and FreeBSD developers Marshall Kirk McKusick along with other two FreeBSD developers George V. Neville-Neil and Robert N.M. Watson.


Its generally more up to date version of the official FreeBSD documentation available as Design and Implementation of 4.4BSD Operating System title from the FreeBSD project documentation page – which one of the authors is also Marshall Kirk McKusick. Is it worth to get it then? Absolutely. A lot have changed and many new technologies have been imported into FreeBSD source tree such as ZFS or DTrace or for example the ULE scheduler.

The final book that I would like to mention here is the FreeBSD Device Drivers book from 2012 written by Joseph Kong.


The book tries to achieve what it title says – to help you first understand and then modify or write your own device drivers. While it covers little older FreeBSD 8 version it is not a problem because the FreeBSD API and ABI change very slow and only when no other way is possible.

Another great tool for programmers and developers in the process of making FreeBSD better is book about DTrace – the dynamic tracing framework. The DTrace – Dynamic Tracing in Oracle Solaris, Mac OS X, and FreeBSD book.


Written by Brendan Gregg and Jim Mauro in 2011 greatly helps to jump into that topic in simple and straightforward way. Besides covering FreeBSD it also does cover Mac OS X, Solaris and even OpenSolaris. That means that it should also be useful for Illumos developers. The book contains a lot of DTrace scripts and examples on how to use that fantastic tool.

Another book that you may find useful in your FreeBSD programmer career is the Designing BSD Rootkits: An Introduction to Kernel Hacking book by Joseph Kong.


Written in 2007 still contains lots of up to date information for the FreeBSD hackers. While the term ‘rootkit’ may be taken ‘negatively’ the author itself describes the book by himself the best way – “Though rootkits have a fairly negative image, they can be used for both good and evil. Designing BSD Rootkits arms you with the knowledge you need to write offensive rootkits, to defend against malicious ones, and to explore the FreeBSD kernel and operating system in the process.” I agree with the author here. To be well protected against something you first need to know how that thing works. The book contains many code examples that you may compile and use and also build upon with your own ideas. Would recommend.

Gone with the Wind

If you know other FreeBSD related books then please let me know.

For those few that did not noticed – the titles of the headers are really great movies ๐Ÿ™‚

Other FreeBSD Version in ZFS Boot Environment

The first FreeBSD 12.3-PRERELEASE snapshots are finally available. This means we can try them in a new ZFS Boot Environment without touching out currently running 13.0-RELEASE system. We can not take the usual path with creating new BE from our current one and upgrade it to newer version because 12.3 has older major version then the 13.0 one.

This is kinda a paradox in the FreeBSD release process that when released the 12.3-RELEASE will have some newer commits and features then older 13.0-RELEASE which was released earlier this year. Of course not all things that have been committed to HEAD goes into 12-STABLE or 13-STABLE automatically – but most of them do. Only the biggest changes will be limited only to 14.0-RELEASE – of course probably somewhere in the middle of 2022 when it will be having its release process.

One note about ZFS filesystem on FreeBSD. People often confuse ‘real’ ZFS Boot Environments with its trying-to-be substitutes like BTRFS snapshots or snapshots used by Ubuntu with zsysctl(8) command. Unfortunately they are only snapshots and are not full writable clones (or entire separate ZFS datasets). They can freeze your system in time so you will be able to get back to working configuration after updating packages for example – but You will not be able to install other separate version of a system as other ZFS dataset making it another independent ZFS Boot Environment.

Create New ZFS Dataset

host # beadm list
BE             Active Mountpoint  Space Created
13.0.w520      NR     /           12.8G 2021-09-14 17:27
13.0.w520.safe -      -            1.2G 2021-10-18 10:01

host # zfs list -r zroot/ROOT
NAME                        USED  AVAIL     REFER  MOUNTPOINT
zroot/ROOT                 12.8G  96.8G       88K  none
zroot/ROOT/13.0.w520       12.8G  96.8G     11.6G  /
zroot/ROOT/13.0.w520.safe     8K  96.8G     11.1G  /

host # zfs create -o mountpoint=/ -o canmount=off zroot/ROOT/12.3

host # beadm list
BE             Active Mountpoint  Space Created
13.0.w520      NR     /           12.8G 2021-09-14 17:27
13.0.w520.safe -      -            1.2G 2021-10-18 10:01
12.3           -      -           96.0K 2021-10-18 13:14

Install FreeBSD 12.3-PRERELEASE

host # beadm mount 12.3 /var/tmp/12.3
Mounted successfully on '/var/tmp/12.3'

host # beadm list
BE             Active Mountpoint     Space Created
13.0.w520      NR     /              12.8G 2021-09-14 17:27
13.0.w520.safe -      -               1.2G 2021-10-18 10:01
12.3           -      /var/tmp/12.3  96.0K 2021-10-18 13:14

host # curl -o - https://download.freebsd.org/ftp/snapshots/amd64/12.3-PRERELEASE/base.txz \
         | tar --unlink -xpf - -C /var/tmp/12.3
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  173M  100  173M    0     0  1889k      0  0:01:33  0:01:33 --:--:-- 2228k

host # exa -1 /var/tmp/12.3

host # curl -o - https://download.freebsd.org/ftp/snapshots/amd64/12.3-PRERELEASE/kernel.txz \
         | tar --unlink -xpf - -C /var/tmp/12.3
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 43.3M  100 43.3M    0     0  1733k      0  0:00:25  0:00:25 --:--:-- 1663k

host # exa -lh /var/tmp/12.3/boot/kernel/kernel
Permissions Size User Date Modified    Name
.r-xr-xr-x   37M root 2021-10-14 06:31 /var/tmp/12.3/boot/kernel/kernel

host # curl -o - https://download.freebsd.org/ftp/snapshots/amd64/12.3-PRERELEASE/lib32.txz \
         | tar --unlink -xpf - -C /var/tmp/12.3

host # exa -ld /var/tmp/12.3/usr/lib32
drwxr-xr-x - root 2021-10-18 13:45 /var/tmp/12.3/usr/lib32

Install Same Packages as on Host

With the pkg prime-list we will get all installed by hand pkg(8)packages from our currently running system. You may omit this section or just install packages that you need instead all of them.

host # pkg prime-list > /var/tmp/12.3/pkg.prime-list

host # chroot /var/tmp/12.3 /bin/sh

(BE) # export PS1="BE # "

BE # mount -t devfs devfs /dev

BE # sed -i '' s/quarterly/latest/g /etc/pkg/FreeBSD.conf

BE # pkg install -y $( cat pkg.prime-list )
Bootstrapping pkg from pkg+http://pkg.FreeBSD.org/FreeBSD:12:amd64/latest, please wait...
Verifying signature with trusted certificate pkg.freebsd.org.2013102301... done
Installing pkg-1.17.2...
Extracting pkg-1.17.2: 100%
Updating FreeBSD repository catalogue...
Fetching meta.conf: 100%    163 B   0.2kB/s    00:01
Fetching packagesite.pkg: 100%    6 MiB   1.3MB/s    00:05
Processing entries: 100%
FreeBSD repository update completed. 31294 packages processed.
All repositories are up to date.
Updating database digests format: 100%
pkg: No packages available to install matching 'chromium' have been found in the repositories
pkg: No packages available to install matching 'drm-fbsd13-kmod' have been found in the repositories
pkg: No packages available to install matching 'geany-gtk2' have been found in the repositories
pkg: No packages available to install matching 'ramspeed' have been found in the repositories
pkg: No packages available to install matching 'vim-console' have been found in the repositories

As we can see some of the packages that we have installed in the FreeBSD 13.0-RELEASE system are not currently available in the ‘latestpkg(8) branch for the FreeBSD 12.3-PRERELEASE system. This sometimes happens when the build of such package will fail – but you may assume that such package will be available in a week or so as that is the period in which pkg(8) packages are (re)build in the ‘latest‘ branch.

We will now remove the missed packages and also rename some packages that may have different names for 12.x version of FreeBSD.

BE # sed -i '' \
         -e s/drm-fbsd13-kmod/drm-kmod/g \
         -e s/geany-gtk2/geany/g \
         -e s/vim-console/vim-tiny/g \

BE # pkg install -y $( cat pkg.prime-list | grep -v -e chromium -e ramspeed )
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
The following 1072 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:

Number of packages to be installed: 1072

The process will require 11 GiB more space.
2 GiB to be downloaded.

BE # rm pkg.prime-list

After hour or so later our packages have been installed.

BE # pkg stats
Local package database:
        Installed packages: 1073
        Disk space occupied: 11 GiB

Remote package database(s):
        Number of repositories: 1
        Packages available: 31294
        Unique packages: 31294
        Total size of packages: 96 GiB

Copy Configuration Files

You can now reboot to plain and unconfigured FreeBSD system but you may as well copy your configuration files from your current working installation. These are the files I have copied.

First files from the Base System /etc and /boot places.

host # for I in /boot/loader.conf \
                /etc/hosts \
                /etc/fstab \
                /etc/rc.conf \
                /etc/sysctl.conf \
                /etc/wpa_supplicant.conf \
                /etc/jail.conf \
                /etc/devfs.rules \
         cp "${I}" /var/tmp/12.3/"${I}"
         echo "${I}"

Now the files for installed packages under /usr/local/etc dir.

host # for I in /usr/local/etc/X11/xorg.conf.d/* \
                /usr/local/etc/X11/xdm/{Xresources,Xsetup_0} \
                /usr/local/etc/automount.conf \
                /usr/local/etc/sudoers \
                /usr/local/etc/doas.conf \
         cp "${I}" /var/tmp/12.3/"${I}"
         echo "${I}"

Add Users and Set Passwords

You should now add your regular user and set passwords for both your user and root account.

BE # pw useradd vermaden -u 1000 -d /home/vermaden -G wheel,operator,video,network,webcamd,vboxusers

BE # passwd root

BE # passwd vermaden

Reboot Into New ZFS Boot Environment

You may now exit the chroot(8) of that ZFS Boot Environment and reboot. In the FreeBSD loader(8) menu select the 12.3 boot environment.

BE # exit

host # umount /var/tmp/12.3/dev

host # beadm unmount 12.3
Unmounted successfully

host # beadm list -D
BE             Active Mountpoint  Space Created
13.0.w520      NR     /           11.3G 2021-09-14 17:27
13.0.w520.safe -      -           11.1G 2021-10-18 10:01
12.3        -      -            9.5G 2021-10-18 13:14

host # shutdown -r now

Testing New System

The 12.3-PRERELEASE system started fine for me. I was able to login and use system as usual. One important thing to note … the ZFS pools. I have another newer ZFS pool with zstd compression enabled … and I was not able to import that ZFS pool as FreeBSD 12.3-PREELEASE does not use OpenZFS 2.0 but an older FreeBSD in-house ZFS version.

# zpool import data
This pool uses the following feature(s) not supported by this system:
        org.freebsd:zstd_compress (zstd compression algorithm support.)
        com.delphix:log_spacemap (Log metaslab changes on a single spacemap and flush them periodically.)
        org.zfsonlinux:project_quota (space/object accounting based on project ID.)
        org.zfsonlinux:userobj_accounting (User/Group object accounting.)
cannot import 'data': unsupported version or feature

Keep that in mind … but you can also install newer OpenZFS from the FreeBSD Ports and this is what we will now do.

# pkg install -y openzfs openzfs-kmod
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
The following 2 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        openzfs: 2021090800
        openzfs-kmod: 2021090800

Number of packages to be installed: 2

The process will require 22 MiB more space.
4 MiB to be downloaded.
[1/2] Fetching openzfs-2021090800.pkg: 100%    3 MiB 975.3kB/s    00:03
[2/2] Fetching openzfs-kmod-2021090800.pkg: 100%    1 MiB 591.2kB/s    00:02
Checking integrity... done (0 conflicting)
[1/2] Installing openzfs-kmod-2021090800...
[1/2] Extracting openzfs-kmod-2021090800: 100%
pkg: Cannot open /dev/null:No such file or directory
[2/2] Installing openzfs-2021090800...
[2/2] Extracting openzfs-2021090800: 100%
Message from openzfs-kmod-2021090800:

Amend /boot/loader.conf as follows to use this module:

- change zfs_load="YES" to NO
- change opensolaris_load="YES" to NO
- add openzfs_load="YES"
- (for ARM64) add cryptodev_load="YES"
Message from openzfs-2021090800:

Ensure that any zfs-related commands, such as zpool, zfs, as used in scripts
and in your terminal sessions, use the correct path of /usr/local/sbin/ and
not the /sbin/ commands provided by the FreeBSD base system.

Consider setting this in your shell profile defaults!

We will now have to modify our /boot/loader.conf file.

host # beadm mount 12.3 /var/tmp/12.3
Mounted successfully on '/var/tmp/12.3'

host # chroot /var/tmp/12.3

BE # cp /boot/loader.conf /boot/loader.conf.ZFS

BE # vi /boot/loader.conf

BE # diff -u /boot/loader.conf.ZFS /boot/loader.conf
--- /boot/loader.conf.ZFS       2021-10-19 10:57:04.180732000 +0000
+++ /boot/loader.conf   2021-10-19 10:57:23.992145000 +0000
@@ -12,7 +12,8 @@

-  zfs_load=YES
+  zfs_load=NO
+  openzfs_load=YES


BE # shutdown -r now

After reboot and trying again I was able to import that newer ZFS pool.

Hope that you will find that guide useful.

Feel free to add your suggestions.

UPDATE 1 – Notes When Installing Newer Version

This guide was written when I tried FreeBSD 12.3 on a previously used by FreeBSD 13.0 system so bootcode was not needed to be updated. I just tried 13.1 on the same 13.0 system and these two steps are needed to updated the bootcode.


For UEFI partition you will need to copy /boot/loader.efi file from the 13.1 installation which means /var/tmp/13.1 dir. Here is the command to be used.

host # gpart show -p ada1
=>       40  250069600    ada1  GPT  (119G)
         40     409600  ada1p1  efi  (200M)          <== UEFI BOOT PARTITION
     409640       1024  ada1p2  freebsd-boot  (512K) <== BIOS BOOT PARTITION
     410664        984          - free -  (492K)
     411648    2097152  ada1p3  freebsd-swap  (1.0G)
    2508800  247560192  ada1p4  freebsd-zfs  (118G)
  250068992        648          - free -  (324K)

host # mount_msdosfs /dev/ada1p1 /mnt

host # cp /var/tmp/13.1/boot/loader.efi /mnt/efi/boot/bootx64.efi


For the systems that boot in legacy/BIOS mode you will use this gpart(8) command instead.

host # cd /var/tmp/13.1/boot
host # pwd
host # gpart bootcode -b ./pmbr -p ./gptzfsboot -i 2 ada1
partcode written to ada1p2
bootcode written to ada1

As FreeBSD often is installed as BIOS+UEFI boot capable both of these steps would be needed.


Upgrade FreeBSD with ZFS Boot Environments

I am known as a strong ZFS Boot Environment supporter … and not without a reason. I have stated the reasons ‘why’ many times but most (or all) of them are condensed here – https://is.gd/BECTL – in my presentation about it.

The upcoming FreeBSD 13.0-RELEASE looks very promising. In many tests it is almost TWICE as fast as the 12.2-RELEASE. Ouch!

The detailed tests are available on the phoronix.com site.

Having 12.2-RELEASE installed I wanted to check 13.0-BETA* to check if things that are important for me – like working suspend/resume for example – work as advertised on the newer version. It is the perfect task that can be achieved by using ZFS Boot Environments.

In the example below we will create entire new ZFS Boot Environment with clone of our current 12.2-RELEASE system and upgrade it there (in BE) to the 13.0-BETA3 version … and there will only be required on reboot – not three as in typical freebsd-update(8) upgrade procedure.

I assume that you have FreeBSD 12.2-RELEASE installed with ZFS (default ZFS FreeBSD install) and its installed in UEFI or UEFI+BIOS mode.

Here are the steps that will be needed.

(host) # beadm create 13                        # create new '13' ZFS Boot Environment
       Created successfully
(host) # beadm mount 13 /var/tmp/BE-13          # mount new '13' BE somewhere
       Mounted successfully on '/var/tmp/BE-13'
(host) # chroot /var/tmp/BE-13                  # make chroot(8) into that place
  (BE) # mount -t devfs devfs /dev              # mount the devfs(8) in that BE
  (BE) # rm -rf /var/db/freebsd-update          # remove any old patches
  (BE) # mkdir /var/db/freebsd-update           # create fresh dir for patches
  (BE) # freebsd-update upgrade -r 13.0-BETA3   # fetch the patches needed for upgrade
  (BE) # freebsd-update install                 # install kernel and kernel modules
  (BE) # freebsd-update install                 # install userspace/binaries/libraries
  (BE) # pkg upgrade                            # upgrade all packages with pkg(8)
  (BE) # freebsd-update install                 # remove old libraries and files
ย  (BE) # exit                                   # leave chroot(8) environment
(host) # umount /var/tmp/BE-13/dev              # umount the devfs(8) in that BE
(host) # beadm activate 13                      # activate new '13' BE
       Activated successfully

I am using mine sysutils/beadm for the process but you as well may use the bectl(8)ย from FreeBSD base system.

We will also need new FreeBSD loader(8) which will be updated this way – thanks to @JeffSipek for pointing that out.

On my system FreeBSD is installed on ada1 device.

(host) # gpart show -p ada1 | grep efi                # find UEFI msdosfs(5) partition
               40     409600  ada1p1  efi  (200M)     # <-- this one
(host) # mount_msdosfs /dev/ada1p1 /mnt               # mount it under /mnt
(host) # find /mnt                                    # display its contents
       /mnt/efi/boot/bootx64.efi                      # update bootx64.efi file
(host) # cp /boot/boot1.efi /mnt/efi/boot/bootx64.efi # copy from /boot/boot1.efi file
(host) # umount /mnt                                  # unmount /mnt filesystem

There is small chance that you will not be able to mount the efi partition. Even fsck(8) is not able to help here.

Typical errors that some users faced look like that:

(host) # mount_msdosfs /dev/ada1p1 /mnt # error when trying to mount efi partition
       mount_msdosfs: /dev/ada1p1: Invalid argument

(host) # fsck_msdosfs -y /dev/ada1p1    # error when trying to fsck(8) that partition
       ** /dev/ada1p1
       Invalid signature in boot block: 0b6a

If you hit that problem then first backup your current efi partition to for example /BACKUP.ada1p1 file.

(host) # dd < /dev/ada1p1 > /BACKUP.ada1p1 bs=1m

Now we will create fresh efi partition from scratch.

(host) # newfs_msdos -F 32 -c 1 /dev/ada0p1            # create new FAT32 partition
(host) # mount_msdosfs /dev/ada0p1 /mnt                # mount it under /mnt
(host) # mkdir -p /mnt/efi/boot                        # create needed directories
(host) # cp /boot/loader.efi /mnt/efi/boot/bootx64.efi # copy from /boot/loader.efi file
(host) # umount /mnt                                   # unmount /mnt filesystem

Now you should have new ‘working’ efi partition.

The last step is to reboot(8) into the new 13.0-BETA3 system.

(host) # reboot

If you find any problems with new bootloader not being able to load your new FreeBSD then you may alternatively copy the /boot/boot1.efi instead of /boot/loader.efi into the /mnt/efi/boot/bootx64.efi place.

Keep in mind that if you boot from geli(8)encrypted system then /boot/loader.efi is mandatory and you will not be able to boot if you would use /boot/boot1.efi file instead.


You should now see the new FreeBSD loader(8) in all its glory ๐Ÿ™‚

You may now enjoy latest FreeBSD 13.0-BETA3 installation.

Same steps will be required to update to soon to be available FreeBSD 13.0-RC* (RC1/RC2/RC3) version and finally FreeBSD 13.0-RELEASE hopefully somewhere in March 2021.

UPDATE 1 – What if Everything Went Fine

You now have most up to date FreeBSD system that should work faster then 12.2-RELEASE and you still has your older 12.2-RELEASE Boot Environment that you can go back to if you find any problems with 13.0 version.

On my system it looks like that:

(host) # beadm list
       BE   Active Mountpoint Space Created
       12.2 -      -           6.5G 2021-02-12 10:15
       13   NR     /          18.8G 2021-02-13 11:32

The Space column is little misleading as it takes into account snapshots space used for example. To get exact information each Boot Environment takes use -D option. This way you will get information about each Boot Environment space separately.

(host) # beadm list -D
       BE   Active Mountpoint  Space Created
       12.2 -      -            9.8G 2021-02-12 10:15
       13   NR     /            9.6G 2021-02-13 11:32

I will be keeping the 12.2-RELEASE Boot Environment for a while – maybe I will delete it a month or so after 13.0-RELEASE is available but if you tested all your needs and feel that 13.0 fulfills all your needs the same way or better then 12.2-RELEASE then you may delete that older Boot Environment with below command.

(host) # beadm destroy 12.2

UPDATE 2 – What if Something Goes Wrong

Generally if the new BE named ‘13‘ does not boot (or hangs at boot) then just select your earlier Boot Environment that you used before the upgrade – the one that has 12.2-RELEASE inside it.

You now have the system that worked for you before we proceed to the upgrade process.

If that fails (or bootloader is broken) then grab the FreeBSD-13.0-BETA3-amd64-memstick.img image and write it on some pendrive with dd(8) command.

# dd if=FreeBSD-13.0-BETA3-amd64-memstick.img of=/dev/da0 bs=1M status=progress

As you now have the pendrive with FreeBSD 13.0-BETA3 then you may boot from it and fix your installation. Pick LiveCD after its loaded. Then type root at login: prompt and hit [ENTER] for empty password.

The list of tasks that can be done now depends on what is broken and I can not guess every possible error and fix scenario so if you hit any problems during that upgrade process then just contact me with your preferred way and we will figure something out.

UPDATE 3 – Faster Upgrade with New beadm(8) Version

Today (2022/05/06) I introduced new beadm(8) version 1.3.5 that comes with new chroot(8) feature. It has already been committed to the FreeBSD Ports tree under 263805 PR so expect packages being available soon.

You can also update beadm(8) directly like that:

# fetch -o /usr/local/sbin/beadm https://raw.githubusercontent.com/vermaden/beadm

Now for the faster update process – here are the instructions depending on the shell you use.

  • ZSH / CSH
# beadm create 13.1-RC6
# beadm chroot 13.1-RC6
BE # zsh || csh
BE # yes | freebsd-update upgrade -r 13.1-RC6
BE # repeat 3 freebsd-update install
BE # exit
# beadm activate 13.1-RC6
# reboot
  • SH / BASH / FISH / KSH
# beadm create 13.1-RC6
# beadm chroot 13.1-RC6
BE # sh || bash || fish || ksh
BE # yes | freebsd-update upgrade -r 13.1-RC6
BE # seq 3 | xargs -I- freebsd-update install
BE # exit
# beadm activate 13.1-RC6
# reboot

Happy upgrading ๐Ÿ™‚


Quare FreeBSD?

I really wanted to make this article short … but I failed miserably. At least I tried to organize it well so one may get back to it after ‘some’ reading because its not a short lecture. I wanted to title it Why FreeBSD? but when you type that into your favorite duck.com search engine there are so many similar articles. I wanted it to have distinguished and unique name so I used Latin word for ‘why‘ which is ‘quare‘.


What FreeBSD can offer you that other operating systems does not? From all of the operating systems I used I find FreeBSD to suck the least. This post is not here to convince you to use or try FreeBSD – this you will have to do by yourself. This article will show you why FreeBSD is valuable or better alternative to other operating systems and is definitely not dying.

This is the Table of Contents for this article.

  • Base System
  • ZFS Boot Environments
  • Rescue
  • Audio
  • Jails
  • FreeBSD Ports Infrastructure
  • Updating/Building from Source
  • Storage
  • Init System
  • Linux Binary Compatibility
  • Simplicity
  • Evolution Instead Rewriting
  • Documentation
  • Community
  • Closing Thoughts
  • External Discussions

Base System

When you install a Linux system its just a bunch of RPM or DEB packages. For example of you install CentOS 7.8 Minimal variant you end up with several hundred RPM packages installed. After a week or month many of these packages will get updates sometimes making this CentOS system unusable or even unbootable (recent GRUB Boothole problem for example). On the contrary FreeBSD comes with a Base System concept. This means that when you install FreeBSD you install a minimal system as a whole. No packages or subsystems to be separately updated. Just whole Base System. That means that /boot /bin /sbin /usr /etc /lib /libexec /rescue directories are untouchable by any packages. When you decide to install packages (or build them using FreeBSD Ports) they will all fall into the /usr/local prefix. That means /usr/local/etc for configuration. The /usr/local/bin and /usr/local/sbin directories for binaries. The /usr/local/lib and /usr/local/libexec for libraries and so on. The FreeBSD Base System kernel modules are kept in the same dir along with the kernel in the /boot/kernel directory. To make things tidy all kernel modules that are provided by packages go into the /boot/modules dir. Everything has its place and its separated.

That is separation between Base System binaries (at /bin /sbin /usr/bin /usr/sbin dirs) and Third Party Packages maintained by pkg(8) and are located at /usr/local/bin and /usr/local/sbin dirs. We all know differences between bin (user) and sbin (root) binaries but in FreeBSD there is also another more UFS related separation. When there was only UFS filesystem in the FreeBSD world the /bin and /sbin binaries were available at boot after the root (/) filesystem was mounted and yet before /usrย  filesystem was mounted – this is historical (and still useful in the UFS setups) distinction dating to old UNIX days. In ZFS setups it does not matter as all files are on ZFS pool anyway.

The FreeBSD Base System separation also helps with another thing – if any package gets the ‘great’ idea to install new compiler named cc and override the default system compiler … or to add libraries/includes in such a way that makes it super hard to get back into a working system. If some random FreeBSD package would add libc.so to /usr/local/lib dir then you are covered and not prevented from running programs as usual because FreeBSD system binaries are linked to stuff in /usr/lib dir. This is why there is PATH variable on UNIX systems (and FreeBSD as well) to set which directories should be searched for binaries first. On FreeBSD by default its set search Base System binaries dirs first and then Third Party Packages later.

You can update (or not) the Base System separately from the installed packages with freebsd-update(8) command when using RELEASE or by recompiling with make buildworld and make installworld commands when using STABLE/CURRENT systems. When it comes to packages you can update them using the pkg(8) tool or portmaster when building from FreeBSD Ports tree under /usr/ports dir. That means that any packages updates will not touch your FreeBSD Base System at all. For example when you mess up (and I have done that in the beginning of my FreeBSD journey) the compiled ports and packages and you want to start over the only thing you have to do is remove /usr/local and /boot/modules and /var/db/pkg directories. That’s it. You are just reverted to your Base System and can start over. This is just not possible when using Linux system. Even with Gentoo that many concepts are based on FreeBSD ideas does not have Base System feature. This Base System also have additional feature. Because its separated from packages version no one stops you from running oldshool FreeBSD 9.0 from 2012 and install there latest Firefox 80 or LibreOffice 7.0. You can not install latest Firefox on Ubuntu from 2012 …

One may be ‘afraid’ that such Base System independent from installed packages would take more space but nothing far more from the truth. The fresh installed FreeBSD 12.1 system uses less then 1 GB of disk space and takes less then 75 MB of RAM with sshd(8) running. For the comparison fresh CentOS 7.8 install with ‘Minimal’ set chosen takes 1.1 GB of disk space and uses more then 100 MB RAM with sshd(8) running. Such CentOS system is really naked and really needs more packages to be usable while FreeBSD with its Base System is far more capable and powerful and comes along with builtin latest version of LLVM/CLANG compiler suite for example.

More on the Base System topic:

ZFS Boot Environments

I have talked about this many times and probably one time too less because Linux world still ignores this bless. Having ZFS Boot Environments its such a game changer that once you realize how powerful it is you will never want to use a system that does not support it. The idea is that you can snapshot a running system at any moment of time and then reboot into that moment (or snapshot) if something happened. Its perfect solution for upgrade or changes to the system. The FreeBSD systems are already well ‘protected’ from problems arising after updating the packages but ZFS Boot Environments takes this to a whole new level.


Like in the movie Groundhog Day (1993) with ZFS Boot Environments you will have limitless chances to get your shit together. Even the Base System updates and changes are protected by it. You can even transport that Boot Environment by using zfs send and zfs recv commands to other system … or propagate it on many systems. You can create Jails containers from it … or install new version of FreeBSD in the new Boot Environment and reboot into it while still having your older ‘production’ system untouched.

More on the ZFS Boot Environments topic:


When you really mess up to the point that even Base System concept or ZFS Boot Environments feature did not stopped you from killing your FreeBSD installation then there is one more level of rescue … the Rescue subsystem.


You have about 150 statically linked binaries available at your disposal for the rescue mission of that FreeBSD installation. You probably think now that if its so many binaries then it probably takes a lot of space … nothing far more from the truth. Its actually one static binary with hardlinks … and it takes whooping 11 MB of disk space.

# ls -lh /rescue | head -5
total 1118446
-r-xr-xr-x  146 root  wheel    11M 2020.02.19 21:10 [
-r-xr-xr-x  146 root  wheel    11M 2020.02.19 21:10 bectl
-r-xr-xr-x  146 root  wheel    11M 2020.02.19 21:10 bsdlabel
-r-xr-xr-x  146 root  wheel    11M 2020.02.19 21:10 bunzip2

They Rescue subsystem even contains such binaries as bectl(8) for ZFS Boot Environments management or zfs(8) and zpool(8) commands for the ZFS filesystem. Here is complete list of these binaries.

# ls /rescue
[           dd               fsck_ffs      init       mdmfs          ping      rtsol        unlink
bectl       devfs            fsck_msdosfs  ipf        mkdir          ping6     savecore     unlzma
bsdlabel    df               fsck_ufs      iscsictl   mknod          pkill     sed          unxz
bunzip2     dhclient         fsdb          iscsid     more           poweroff  setfacl      unzstd
bzcat       dhclient-script  fsirand       kenv       mount          ps        sh           vi
bzip2       disklabel        gbde          kill       mount_cd9660   pwd       shutdown     whoami
camcontrol  dmesg            geom          kldconfig  mount_msdosfs  rcorder   sleep        xz
cat         dump             getfacl       kldload    mount_nfs      rdump     spppcontrol  xzcat
ccdconfig   dumpfs           glabel        kldstat    mount_nullfs   realpath  stty         zcat
chflags     dumpon           gpart         kldunload  mount_udf      reboot    swapon       zdb
chgrp       echo             groups        ldconfig   mount_unionfs  red       sync         zfs
chio        ed               gunzip        less       mt             rescue    sysctl       zpool
chmod       ex               gzcat         link       mv             restore   tail         zstd
chown       expr             gzip          ln         nc             rm        tar          zstdcat
chroot      fastboot         halt          ls         newfs          rmdir     tcsh         zstdmt
clri        fasthalt         head          lzcat      newfs_msdos    route     tee          
cp          fdisk            hostname      lzma       nextboot       routed    test         
csh         fsck             id            md5        nos-tun        rrestore  tunefs       
date        fsck_4.2bsd      ifconfig      mdconfig   pgrep          rtquery   umount   

More on the Rescue topic:


Not many people expect from FreeBSD to shine in that department but it shines a lot here and not from yesterday but from decades. Remember when Linux got rid of the old OSS subsystem with one channel and came up with ‘great’ idea to write ALSA? I remember because I used Linux back then. Disaster is very polite word to describe Linux audio stack back then … and then PulseAudio came and whole Linux audio system got much worse. Back then because of that one OSS channel and many ALSA channels meant that ONLY ONE application with OSS backend could do the sound (for example WINE). But if another application would want to ‘make’ sound using OSS and you already have WINE started then it will be soundless because that one and only OSS channel was already taken. And remember that ALSA was so bad back then that KDE or GNOME made their own sound daemons mixing audio in userspace that were incompatible with each other. That means if you used KDE and GNOME apps back then you could have sound from GNOME apps but not from KDE apps or vice versa. One big fucking audio hell on Linux.


Lets get back to FreeBSD audio then. What FreeBSD offered? A whooping 256 OSS channels mixed live in kernel for low latency. Everything audio related just worked out of the box – and still works today. You could have WINE or KDE/GNOME sound backends attached to their OSS channels and also ALSA apps getting their sound device without a problem. Even when you plugged a 5.1 surround system into FreeBSD it worked out of the box without any configuration and applications were able to use it immediately. That FreeBSD audio supremacy remains today as PulseAudio sound mixing in userspace while generally working incorporates large latency on Linux compared to in kernel FreeBSD mixing with low latency.

Comrade meka suggested that FreeBSD is also the only OS which has virtual_oss that allows mixing/resampling/compressing in user space and allows one to have Bluetooth headphones and USB microphone represented as single sound card.

More on the Audio topic:


The FreeBSD Jails are one of the oldest OS Level Virtualization implementations dating back to 1999. Even the Solaris Zones/Containers came five years later in 2004.


After Docker was introduced in Linux the term OS Level Virtualization became less used to the Containers term and now the FreeBSD Jails along with Solaris Zones/Containers are named 1st generation containers. But that naming nomenclature change does not make FreeBSD Jails less powerful. They are also really brain dead simple to use. You just need a directory – for example /jail/nextcloud – where you will extract the FreeBSD Base System for desired release version – for example base.txz from 12.1-RELEASE and create the Jail config in the /etc/jail.conf file as shown below.

# mkdir -p /jail/nextcloud
# fetch -o - http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/12.1-RELEASE/base.txz | tar --unlink -xpJf - -C /jail/nextcloud
# cat /etc/jail.conf
nextcloud {
  host.hostname = nextcloud.local;
  ip4.addr =;
  path = /jail/nextcloud;

Now you can start you Jail right away.

# service jail onestart nextcloud
Starting jails: nextcloud.

Voila! Your FreeBSD Jail is already running.

# jls
   JID  IP Address      Hostname                      Path
     1      nextcloud.local               /jail/nextcloud

You can of course have a trimmed down version of FreeBSD Base System in the Jail if that is needed. The ZFS filesystem also helps here greatly because with zfs clone only your ‘base’ Jail will take space and only the changes you make to Jails created from it. Thanks to other FreeBSD subsystem – the Linux Binary Compatibility – you can also create a Linux Jail – for example running Devuan or Ubuntu Jail.

The FreeBSD Jails are also very lightweight. You can boot and use about 1000 FreeBSD Jails on a single FreeBSD system with 4 GB RAM.

They are also very easy to debug and troubleshoot comparing even to plain Docker – not to even mention Kubernetes which requires whole team of highly skilled people to maintain.

The FreeBSD Jails may be configured/managed only by the Base System utilities such as jls(8)/jexec(8) but you can also select from many third party Jail management frameworks. From all available ones I would choose BastilleBSD because of their modern approach and many ready to use templates for all needed use cases.

More on the Jails topic:

FreeBSD Ports Infrastructure

This is one of another examples why FreeBSD rocks that much. When you install Ubuntu or CentOS in some version there is chance that you will end up with not latest versions of packages but with versions that were quite up-to-date when this distribution version was released. Its especially visible in the CentOS world (and its upstream enterprise source system from Red Hat) where packages are quite up-to-date when .0 (dot zero) release is published but are VERY outdated when .8 or .9 incarnation of that release is available. Not to even mention that Firefox for example is released every month …


As I said before when describing the FreeBSD Base System the FreeBSD Ports (and packages built from it available through pkg(8) command) are independent. That means that third party software from FreeBSD Ports is almost always up-to-date (or very close to it). You can even check it on the repology.org site for the details. Below you will find a ‘snapshot’ of the repology.org stats from time of writing this article. The ‘online’ table is very long so I copy/pasted just the systems relevant to the article.


One of the other advantages of FreeBSD Ports is that it offers really MASSIVE amount of software counting 40354 ports when writing this article and still rising. Amount of ready to be installed packages are little smaller with more then 32000 available.

I once migrated for a while to OpenSolaris in 2009 on my Dell Latitude D630 laptop because I really liked all the Solaris features (including ZFS and ZFS Boot Environments that were not available on FreeBSD back then) and the OpenSolaris GNOME based desktop was pretty nice back then even with Time Slider feature for ZFS snapshots in the Nautilus file manager. I got working WiFi connection, sound was working, generally everything on my laptop was supported and working with OpenSolaris … but there was no software. Of course ‘large’ projects like GIMP or OpenOffice was available even in the default pkg(8) repository but not much else. There was less then 4000 packages back then on OpenSolaris while about 25000 packages on FreeBSD if I recall correctly.

You can also easily browse available FreeBSD Ports (and its options) on the web by using the https://freshports.org/ page.


The count of FreeBSD Ports is one thing, the features is another. No matter which Linux distribution you are using you will find a software that was compiled and shipped without that needed flag that you desperately need. If you find such software on FreeBSD it ‘hurts’ only for a moment because you can VERY EASILY recompile that software with needed options and replace that ‘default’ package with yours. For example the FreeBSD project is afraid to provide packages of Lame because of existing MP3 patents, so multimedia/ffmpeg package is built without MP3 support (with --disable-libmp3lame flag). That is why I have my own audio/lame and multimedia/ffmpeg packages built with my configure options and that is very easy to achieve. You need to go to the /usr/ports/multimedia/ffmpeg dir type make config and select [x] LAME at the ncurses dialog. Your chosen options will be saved as plain /var/db/ports/multimedia_ffmpeg/options file. If you remove that file (or type make rmconfig) then these custom options will reset to defaults. Then you type make build deinstall install clean and your port with new options is ready and installed as package. Nothing more is needed. You can even lock that package from the pkg(8) upgrades with pkg lock -y ffmpeg command so it will not be modified later but its better to rebuild such packages everytime you do a pkg upgrade procedure because of libraries versions bump and changes. While its very easy and fast to create a script with these commands to make it more automated you can also use other parts of the FreeBSD Ports infrastructure – enter Poudriere (or Synth) – more on that in the next part.

You also do not have to configure each port that way (which could be PITA for large amount of ports) but you may specify your needed (OPTIONS_SET) or unwanted (OPTIONS_UNSET) parameters only once globally using the /etc/make.conf file. You can also specify which default versions of software you want to use, for example Apache 2.2 instead of 2.4 and PHP 7.0 instead of 7.2. You can find all default versions in the /usr/ports/Mk/bsd.default-versions.mk file. Once you setup these options you can build/rebuild or update your packages from FreeBSD Ports by portmaster(8) tool. Like on Gentoo Linux with USE flags. But this is the original. Gentoo took all/most of its ideas from FreeBSD system and its Ports infrastructure.

The Poudriere is a build framework that uses FreeBSD Ports and FreeBSD Jails to build requested packages in clean reproducible way. You can create whole new binary package repository for pkg(8) command to use with it. I mentioned Synth because while Poudriere is often used to produce whole package repository the Synth is usually used just to rebuild several packages that does not fit your needs.

There is one important things about FreeBSD Ports that is often misunderstood by newcomers. What is the difference between the Ports and packages that are fetched and installed by pkg(8) tool? Its quite simple. A package is just a build and installed port. Nothing more or less. When you use the binary packages using pkg(8) command you are using packages that someone (the FreeBSD project in that case) built for you from the FreeBSD Ports in some point in time. While FreeBSD strives to maintain as up-to-date built packages as possible its the nature of FreeBSD Ports that they are always more up-to-date then the built packages. That is why you may build and install a new version of needed packages by yourself using FreeBSD Ports. One may think of such usage when it comes to security holes. When some locally executed commands (like file(1) for example) has a security hole then its not critical for you to update it as fast as possible because that security hole can be harmless for you, but when new version of Firefox fixes very important security hole then its better to update from FreeBSD Ports version faster because waiting 2 days for the package to be built (along with other packages) can be too long.

More on the FreeBSD Ports topic:

Updating/Building from Source

While the FreeBSD Ports infrastructure is for third party software the FreeBSD Base System (or its parts) also can be easily and convenient build from source. The FreeBSD kernel config is also very small and simple. While Linux kernel config contains thousands of options – 4432 for example in the default CentOS 8.2 install the FreeBSD GENERIC config has about 20 times options less – only 260 options. But that does not saturate the topic. You can start with MINIMAL FreeBSD kernel config which has only 75 options specified.

Linux # grep -c '^CONFIG' /boot/config-$( uname -r )

FreeBSD # grep -c -E '^(device|options)' /usr/src/sys/amd64/conf/GENERIC

FreeBSD # grep -c -E '^(device|options)' /usr/src/sys/amd64/conf/MINIMAL

… and its not only about smaller amount of options. Can you tell my how many steps (and which ones are required) to rebuild CentOS or Ubuntu for example without Bluetooth support?


On the contrary its very simple (and fast) on the FreeBSD side. While /etc/make.conf file is used to enable/disable Ports options the /etc/src.conf file is used to enable/disable FreeBSD Base System options while building it from source. To build FreeBSD without Bluetooth support just add WITHOUT_BLUETOOTH=yes to the /etc/src.conf file and type these to build it:

# beadm create safe
# cd /usr/src
# make buildworld kernel
# reboot
# cd /usr/src
# make installworld
# mergemaster -iU
# reboot

Voila! You now have FreeBSD without Bluetooth support … and if any of the steps failed or because of your lack of experience/expertise your FreeBSD system does not boot or is broken you can use tools from /rescue to try to fix it (or at least figure out what is broken) and when you do not want to cope with this jest select safe ZFS Boot Environment at the FreeBSD loader(8) to boot to the system before you started building modified version of FreeBSD. Yes, You are bulletproof here. While having 294 WITHOUT_X options and 125 WITH_X options you can really tune FreeBSD Base System to your needs.

# zgrep -c WITHOUT_ /usr/share/man/man5/src.conf.5.gz

# zgrep -c WITH_ /usr/share/man/man5/src.conf.5.gz

The big downside of updating FreeBSD by source is that you can not use the freebsd-update tools to do it … but nothing stops you from creating your own FreeBSD Update Server so you will be able to use freebsd-update by adding updates using a CURRENT or STABLE system instead of RELEASE. That process is described in the Build Your Own FreeBSD Update Server article of official FreeBSD documentation.

More on the FreeBSD Source Updates/Builds topic:


Storage is one of the parts where FreeBSD really shines. Lots of people adore FreeBSD for well integrated ZFS filesystem and its really true. ZFS in FreeBSD has always been first class citizen. Lately OpenZFS 2.0 has been also integrated from the upstream joint FreeBSD and Linux repository. More and more FreeBSD features and solutions are using ZFS features.


Most of these people that like integrated ZFS in FreeBSD do not know about the FreeBSD GEOM modular disk transformation framework which provides various storage related features and utilities like software RAID0/RAID1/RAID10/RAID3/RAID5 configurations or transparent encryption of underlying devices with GELI/GDBE (like LUKS on Linux). It also allows transparent filesystem journaling for ANY filesystem with GJOURNAL (yes also for FAT32 or exFAT) or allows one to export block devices over network with GEOM GATE devices (like NFS for block devices).


FreeBSD also has its own FUSE implementation which allows all these FUSE based filesystems to work natively on FreeBSD. While lots of Linux folks know DRBD probably very few of them knew that FreeBSD comes with its own DRBD like solution called HAST – which does exactly the same thing. While ZFS has a lot features and possibilities FreeBSD still maintains and develops fast and small memory footprint UFS filesystem which today is used either with Soft Updates (SU) or Journaled Soft Updates (SUJ) depending on the use case. For example 10 TB data on UFS filesystem with Journaled Soft Updates (SUJ) takes about 1 minute under fsck(8). These storage solutions are available from FreeBSD Base System alone. The FreeBSD Ports offers much more with distributed filesystems solutions such as CEPH, LeoFS, LizardFS or Minio for Amazon S3 compatible storage.

More on the Storage topic:

Init System

FreeBSD offers really simple yet very powerful init system. It has system wide config under /etc/rc.conf file when you can enable/disable needed services with service_enable=YES and service_enable=NO stanzas. You do not even need to launch vi(1) to add them – just type sysrc service_enable=YES and they are added to the /etc/rc.conf file. There are also default values and services that are enabled and you will find them – along with many comments – in the /etc/defaults/rc.conf file. Each FreeBSD service file has PROVIDE/REQUIRE stanzas which are then used to automatically order the services to start. Services that can be run in parallel are started in parallel to save time. For example its pointless to start sshd(8) daemon without network. To start or stop the service you need to type service sshd start or service sshd stop command. But when a service is not enabled in the /etc/rc.conf file then you need to used add onestart and onestop instead. The Base System separation remains here as FreeBSD Base System services are located at /etc/rc.d directory and third party applications from ports/packages are kept under /usr/local prefix which means /usr/local/etc/rc.d dir.

When using systemd(1) you never know how the services gonna start because it will be different each time. Zero determinism. On FreeBSD you know exactly which services will start when because they are always ordered in the same state according to the PROVIDE/REQUIRE stanzas. FreeBSD also offers tools that will tell you the exact order – rcorder(8) – which can be used for all services, Base System services or third party services separately. There is also service -r command that will show you what was the order at the boot time.

# rcorder /etc/rc.d/* | head

# rcorder /usr/local/etc/rc.d/* | tail

# rcorder /etc/rc.d/* /usr/local/etc/rc.d/* 2> | grep -C 3 sshd

Adding new service to FreeBSD is also very easy as template for new service is very small and simple.


. /etc/rc.subr



load_rc_config $name
: ${dummy_enable:=no}
: ${dummy_msg="Nothing started."}

	echo "$dummy_msg"

run_rc_command "$1"

If its not simple enough for you there is dedicated FreeBSD article about writing them – Practical rc.d Scripting in BSD – available here.

More on the Init System topic:

Linux Binary Compatibility

While Linux can not be FreeBSD – the FreeBSD can be Linux – and its not some slow emulation – its implementation of Linux system calls. There was time when enterprises used to work with Linux only applications (not available on FreeBSD by then) using the Linux Binary Compatibility on FreeBSD because it was faster then running them natively on Linux – FreeBSD Used to Generate Spectacular Special Effects – an official FreeBSD Press Release about FreeBSD being used to generate spacial effects to the one of the best movies of all time – The Matrix (1999).


Today the LINUX_COMPAT is also natively fast and allows one to run Linux applications – even Linux games in X11 with hardware acceleration for graphics. Think of it as WINE but for Linux applications. It lives under /compat/linux directory. It even implements Linux /proc virtual filesystem which can be mounted at the /compat/linux/proc dir but its not mandatory. For any software that does not come with source code and works on Linux the Linux Binary Compatibility saves the day. For example the f.lux project. Before I got to know Redshift I used f.lux Linux binary using LINUX_COMPAT to suppress blue spectrum light from my FreeBSD screen. The Linux Binary Compatibility subsystem can also be used to run Linux bases FreeBSD Jails – with Devuan for example.

More on the Linux Binary Compatibility topic:


FreeBSD is simple but not coarse/ornery. For example as Linux the FreeBSD system also supports the /proc virtual filesystem but on FreeBSD its optional and not used by default while Linux could not live without it. But while Linux has mandatory /proc it also has another virtual filesystem residing under /sys … but why Linux people need two different virtual filesystems with similar purposes? Why they could not create everything under /proc as it already existed. That is big enigma for my sanity.

But /sys is not the end of that madness. Its just a beginning.

What about these?

  • securityfs
  • devpts
  • cgroup
  • pstore
  • bpf
  • configfs
  • selinuxfs
  • systemd-1
  • mqueue
  • debugfs
  • hugetlbfs

Take a look at the FreeBSD mount(8) output after the default install on ZFS.

FreeBSD # mount
zroot/ROOT/12.1 on / (zfs, local, noatime, nfsv4acls)
devfs on /dev (devfs, local, multilabel)
zroot/tmp on /tmp (zfs, local, noatime, nosuid, nfsv4acls)
zroot/var/mail on /var/mail (zfs, local, nfsv4acls)
zroot/usr/home on /usr/home (zfs, local, noatime, nfsv4acls)
zroot/var/crash on /var/crash (zfs, local, noatime, noexec, nosuid, nfsv4acls)
zroot/var/log on /var/log (zfs, local, noatime, noexec, nosuid, nfsv4acls)
zroot/var/audit on /var/audit (zfs, local, noatime, noexec, nosuid, nfsv4acls)
zroot/var/tmp on /var/tmp (zfs, local, noatime, nosuid, nfsv4acls)
zroot/usr/src on /usr/src (zfs, local, noatime, nfsv4acls)
zroot/usr/ports on /usr/ports (zfs, local, noatime, nosuid, nfsv4acls)

Several ZFS datasets and one virtual devfs filesystem for /dev directory. With install on UFS it would be similar with several UFS partitions mounted instead of ZFS datasets.

Take a look at the CentOS 8.2 installation with just one physical root (/) XFS filesystem.

[root@centos8 ~]# mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,seclabel,size=919388k,nr_inodes=229847,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,seclabel)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,seclabel,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime,seclabel)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,cpuset)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,memory)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,blkio)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,hugetlb)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,net_cls,net_prio)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,cpu,cpuacct)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,freezer)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,perf_event)
cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,rdma)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,pids)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,devices)
configfs on /sys/kernel/config type configfs (rw,relatime)
/dev/sda1 on / type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
selinuxfs on /sys/fs/selinux type selinuxfs (rw,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17309)
mqueue on /dev/mqueue type mqueue (rw,relatime,seclabel)
debugfs on /sys/kernel/debug type debugfs (rw,relatime,seclabel)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,seclabel,pagesize=2M)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=187088k,mode=700)

Fuck me. Its even really hard to just find any REAL filesystem there … fortunately we can ask for only XFS filesystems to display.

[root@centos8 ~]# mount -t xfs
/dev/sda1 on / type xfs (rw,relatime,seclabel,attr2,inode64,noquota)

Lets get on the networking now. Lets assume that you want to make standard enterprise networking setup on a physical server with two interfaces aggregated together into highly available interface bond0 (lagg0 on FreeBSD) and then you want to put VLAN tag and IP address on that VLAN. The CentOS 7.x/8.x installer (Anaconda) will welcome you with this mess.

[root@centos7 ~]# ls -1 /etc/sysconfig/network-scripts/ifcfg-*

[root@centos7 ~]# cat /etc/sysconfig/network-scripts/ifcfg-Bond_connection_1
BONDING_OPTS="miimon=1 updelay=0 downdelay=0 mode=active-backup"
NAME="Bond connection 1"

[root@centos7 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eno49

[root@centos7 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eno49-1

[root@centos7 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eno50

[root@centos7 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eno50-1

[root@centos7 ~]# cat/etc/sysconfig/network-scripts/ifcfg-VLAN_connection_1
NAME="VLAN connection 1"

For the record – I have chosen ‘STATIC’ IPv4 address but installer made these interfaces to use DHCP and that STATIC address. That could be a bug but lets get to the point.

After manual fixing with vi(1) (and hour later) this is how it supposed to look.

[root@centos7 ~]# cat /etc/sysconfig/network

[root@centos7 ~]# ls -1 /etc/sysconfig/network-scripts/ifcfg-*

[root@centos7 ~]# cat /etc/sysconfig/network-scripts/ifcfg-bond0
BONDING_OPTS="miimon=1 updelay=0 downdelay=0 mode=active-backup"

[root@centos7 ~]# cat /etc/sysconfig/network-scripts/ifcfg-bond0.601

[root@centos7 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eno49

[root@centos7 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eno50

Better … but still takes A LOT OF SPACE and several files to cover that quite simple setup. Not to mention its level of complication and making that very error prone way. The same configuration on FreeBSD would take just 7 lines within single /etc/rc.conf file as shown below.

root@freebsd # cat /etc/rc.conf
ifconfig_lagg0="laggproto failover laggport fxp0 laggport fxp1"

What about the boot process? FreeBSD boots from root on ZFS partition with just small 512 KB not mountable partition. No separate /boot device is needed. On the other side Linux always needs that separate /boot partition filled with GRUB modules. No matter if its ZFS or LVM. That is why implementation of ZFS Boot Environments is quite complicated on Linux because even if you have root on ZFS on a Linux system there is still unprotected /boot filesystem that can not be snapshoted with ZFS and has to be protected in old classic way which kill the idea of ZFS Boot Environments or Linux.

FreeBSD is really simple and well thought operating system. But also a very underestimated one.

Evolution Instead Rewriting

How many Linux tools or subsystems are abandoned or superseded by new ones? Why the ifconfig(8) command was not updated with new options and instead a new ip(8) command was introduced? Same with netstat(8) being replaced by ss(8). Same with arp(8)/iwconfig/route(8) and many more. What about whole init system? The Linux world has been taken over by systemd(1) whenever you like it or not. Even distributions that have grown their mature init systems like Ubuntu with its Upstart has moved to systemd(1) altogether. The distributions that do not use it are very few and considered a niche today.


In the FreeBSD land on the counrary such things happen only if there is no other way to implement new things. Its the last thing wanted in the FreeBSD. FreeBSD evolves and is developed with stability and backward compatibility in mind. Userland tools are grown and updated with new options instead of rewriting them over and over again. Not to mention how many new bugs are introduced by changing one tool to another.

More on the Evolution Instead Rewriting topic:


Having system that can do almost anything but not knowing how to do that makes that system pretty useless (or at least pretty PITA to use). FreeBSD offers second to none documentation that is actively maintained and updated. Along with its legendary FreeBSD Handbook and FreeBSD FAQ the FreeBSD project also offers official FreeBSD Articles about various FreeBSD topics. The Man Pages are also very detailed and contain many examples. There is also FreeBSD Wiki page for work in progress documentation and ideas related to FreeBSD development and if you have any problems or questions related to FreeBSD there are official FreeBSD Forums and oldschool Mailing Lists available.


These were only the official project knowledge sources but there are also lots of FreeBSD books. Here are the best and up-to-date ones.

  • Absolute FreeBSD – Complete Guide to FreeBSD – 3nd Edition (2019)
  • Beginning Modern Unix (2018)
  • Book of PF – 3rd Edition (2015)
  • Design and Implementation of FreeBSD 11 Operating System – 2nd Edition (2015)
  • FreeBSD Device Drivers (2012)
  • FreeBSD Mastery – ZFS (2015)
  • FreeBSD Mastery – Advanced ZFS (2016)
  • FreeBSD Mastery – Storage Essentials (2014)
  • FreeBSD Mastery – Specialty Filesystems (2015)
  • FreeBSD Mastery – Jails (2019)

There are also two magazines that are dedicated to BSD and FreeBSD systems. Both are free and cover lots of interesting topics regarding FreeBSD.

With all this knowledge and support its really hard not to achieve what you need/want with FreeBSD system.


Last but not least and I would say its even more important then good documentation (which FreeBSD has awesome). People that use FreeBSD do that conciously and are often experienced not only in FreeBSD land but also in topics related to other UNIX systems. Often they took long road of first using the Linux systems before finally setting on the FreeBSD land or they still do Linux adminitration for a living while resting using far more reasonable and sensible FreeBSD solution. I always find FreeBSD Community helpful and friendly. Always willingly helpful – especially towards newcommers. Even when you try to ‘force’ FreeBSD people to ‘fight’ in unjust/doubtful discussion they will reply with dignity and technical arguments instead of yelling at you.

The FreeBSD project even made several articles and Handbook chapters especially for Linux newcommers (or sometimes called systemd(1) refugees).

Closing Thoughts

I tried really hard to not make it a Linux rant but some may feel it that way – if so please remember that this was not my intention. FreeBSD like Linux and like any other operating system has its ups and downs. Hope that I showed you most interesting FreeBSD parts. I may add new sections here without a warning in the future ๐Ÿ™‚

External Discussions

Discussions and comments from ‘external’ sources are available here:


FreeBSD Enterprise Storage at PBUG

Yesterday I was honored to give a talk about FreeBSD Enterprise Storage at the Polish BSD User Group meeting.

You are invited to download the PDF Slides โ€“ https://is.gd/bsdstg โ€“ available here.


The PBUG (Polish BSD User Group) meetings are very special. In “The Matrix” movie (which has been rendered on FreeBSD system by the way) – FreeBSD Used to Generate Spectacular Special Effects – details available here – its not possible to describe what the Matrix really is, one has to feel it. Enter it. The same I can tell you about the PBUG meetings. Its kinda like with the “Hangover” movie. What happens in Vegas PBUG meeting stays in Vegas PBUG meeting ๐Ÿ™‚

If you will have the possibility and time then join the next Polish BSD User Group meeting. You will not regret it :>

UPDATE 1 – Shorter Unified Version

The original – https://is.gd/bsdstg – presentation is 187 pages long and is suited for live presentation while not the best for later ‘offline’ view.

I have created a unified version – https://is.gd/bsdstguni – with only 42 pages.


Nextcloud 17 on FreeBSD 12.1

Not so long ago – almost 2 years from now – I wrote about setting up Nextcloud 13 on FreeBSD.

Today Nextcloud is at 17 version and the configuration that worked two years ago requires some tweaks.


This guide will not cover the same information that is available in earlier Nextcloud 13 on FreeBSD article like settings to run Nextcloud inside FreeBSD Jail. Please refer to that earpier article for these settings.

Today we will use these as backends for Nextcloud 17.

  • PostgreSQL 12
  • PHP 7.3
  • Nginx 1.14 (with php-fpm)
  • Memcached 1.5.19

As Nextcloud in FreeBSD packages comes with MySQL and without PostgreSQL support we will need to build it from source using FreeBSD Ports.


Let’s fetch the latest FreeBSD Ports tree.

# rm -r /var/db/portsnap
# mkdir /var/db/portsnap
# portsnap auto

Now we need to configure needed options in the /etc/make.conf file.

# cat /etc/make.conf

Packages and Ports

First we will add some basic tools and things like PostgreSQL still using FreeBSD packages to save tome time instead of compiling them.

# pkg install \
    sudo \
    portmaster \
    beadm \
    lsblk \
    postgresql12-client \
    postgresql12-server \
    nginx \
    memcached \

Now we will compile Nextcloud and its dependencies using FreeBSD Ports – but with portmaster.

# env BATCH=yes portmaster \
    databases/php73-pdo_pgsql \
    databases/php73-pgsql \


We will now configure the FreeBSD’s Login Class for PostgreSQL database in the /etc/login.conf file.

# cat  /etc/login.conf



# cap_mkdb /etc/login.conf

… and PostgreSQL settings in main FreeBSD’s configuration /etc/rc.conf file.

# grep postgresql /etc/rc.conf

Let’s initialize the PostgreSQL database.

# /usr/local/etc/rc.d/postgresql initdb
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locales
  CTYPE:    en_US.UTF-8
  TIME:     en_US.UTF-8
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/db/postgres/data12 ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Europe/Warsaw
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.

Success. You can now start the database server using:

    /usr/local/bin/pg_ctl -D /var/db/postgres/data12 -l logfile start

As PostgreSQL database uses 8k blocks let’s set it in ZFS. We could of course create dedicated dataset for this purpose if needed.

# zfs set recordsize=8k zroot/ROOT/default

Now, let’s start the PostgreSQL database.

# /usr/local/etc/rc.d/postgresql start
2019-12-31 11:47:04.918 CET [36089] LOG:  starting PostgreSQL 12.1 on amd64-portbld-freebsd12.0, compiled by FreeBSD clang version 6.0.1 (tags/RELEASE_601/final 335540) (based on LLVM 6.0.1), 64-bit
2019-12-31 11:47:04.918 CET [36089] LOG:  listening on IPv6 address "::1", port 5432
2019-12-31 11:47:04.918 CET [36089] LOG:  listening on IPv4 address "", port 5432
2019-12-31 11:47:04.919 CET [36089] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"
2019-12-31 11:47:04.928 CET [36089] LOG:  ending log output to stderr
2019-12-31 11:47:04.928 CET [36089] HINT:  Future log output will go to log destination "syslog".

We will now create PostgreSQL database for our Nextcloud instance.

# psql -hlocalhost -Upostgres
psql (12.1)
Type "help" for help.

postgres=# CREATE DATABASE nextcloud TEMPLATE template0 ENCODING 'UNICODE';
postgres=# ALTER DATABASE nextcloud OWNER TO nextcloud;
postgres=# \q

Keep in mind to put something more sophisticated in the NEXTCLOUD_DB_PASSWORD place.

PostgreSQL Cleanup and Indexing Script

Lets automate some PostgreSQL housekeeping.

# mkdir -p /var/db/postgres/bin
# chown postgres /var/db/postgres/bin
# vi /var/db/postgres/bin/vacuum.sh

#! /bin/sh

/usr/local/bin/vacuumdb -az 1> /dev/null 2> /dev/null
/usr/local/bin/reindexdb -a 1> /dev/null 2> /dev/null
/usr/local/bin/reindexdb -s 1> /dev/null 2> /dev/null

# cat /var/db/postgres/bin/vacuum.sh
#! /bin/sh

/usr/local/bin/vacuumdb -az 1> /dev/null 2> /dev/null
/usr/local/bin/reindexdb -a 1> /dev/null 2> /dev/null
/usr/local/bin/reindexdb -s 1> /dev/null 2> /dev/null

# chown postgres /var/db/postgres/bin/vacuum.sh
# chmod +x /var/db/postgres/bin/vacuum.sh

# su - postgres -c 'crontab -e'
0 0 * * * /var/db/postgres/bin/vacuum.sh
/tmp/crontab.JMg5BfT5HV: 2 lines, 42 characters.
crontab: installing new crontab

# su - postgres -c 'crontab -l'
0 0 * * * /var/db/postgres/bin/vacuum.sh

# su - postgres -c '/var/db/postgres/bin/vacuum.sh'


Now its time for Nginx webserver.

# chown -R www:www /var/log/nginx

# ls -l /var/log/nginx
total 3
-rw-r-----  1 www  www   64 2019.12.31 00:00 access.log
-rw-r-----  1 www  www  133 2019.12.31 00:00 access.log.0.bz2
-rw-r-----  1 www  www   64 2019.12.31 00:00 error.log
-rw-r-----  1 www  www  133 2019.12.31 00:00 error.log.0.bz2

… and its main nginx.conf configuration file.

# cat /usr/local/etc/nginx/nginx.conf
user www;
worker_processes 4;
worker_rlimit_nofile 51200;
error_log /var/log/nginx/error.log;

events {
  worker_connections 1024;

http {
  include mime.types;
  default_type application/octet-stream;
  log_format main '$remote_addr - $remote_user [$time_local] "$request" ';
  access_log /var/log/nginx/access.log main;
  sendfile on;
  keepalive_timeout 65;

  upstream php-handler {

  server {
    listen 80;
    server_name nextcloud.domain.com;
    return 301 https://$server_name$request_uri;

  server {
    listen 443 ssl http2;
    server_name nextcloud.domain.com;
    ssl_certificate /usr/local/etc/nginx/ssl/ssl-bundle.crt;
    ssl_certificate_key /usr/local/etc/nginx/ssl/server.key;

    add_header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload;";
    add_header Referrer-Policy "no-referrer";

    add_header X-Content-Type-Options nosniff;
    add_header X-XSS-Protection "1; mode=block";
    add_header X-Robots-Tag none;
    add_header X-Download-Options noopen;
    add_header X-Permitted-Cross-Domain-Policies none;

    root /usr/local/www/nextcloud/;

    location = /robots.txt {
      allow all;
      log_not_found off;
      access_log off;

    location = /.well-known/carddav {
      return 301 $scheme://$host/remote.php/dav;

    location = /.well-known/caldav {
      return 301 $scheme://$host/remote.php/dav;

    client_max_body_size 16400M;
    client_body_buffer_size 1048576k;
    send_timeout 3000;

    gzip on;
    gzip_vary on;
    gzip_comp_level 4;
    gzip_min_length 256;
    gzip_proxied expired no-cache no-store private no_last_modified no_etag auth;
    gzip_types application/atom+xml application/javascript application/json application/ld+json application/manifest+json application/rss+xml application/vnd.geo+json application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/bmp image/svg+xml image/x-icon text/cache-manifest text/css text/plain text/vcard text/vnd.rim.location.xloc text/vtt text/x-component text/x-cross-domain-policy;

    location / {
      rewrite ^ /index.php$request_uri;

    location ~ ^/(?:build|tests|config|lib|3rdparty|templates|data)/ {
      deny all;

    location ~ ^/(?:\.|autotest|occ|issue|indie|db_|console) {
      deny all;

    location ~ ^\/(?:index|remote|public|cron|core\/ajax\/update|status|ocs\/v[12]|updater\/.+|oc[ms]-provider\/.+)\.php(?:$|\/) {
      fastcgi_split_path_info ^(.+\.php)(/.*)$;
      include fastcgi_params;
      fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
      fastcgi_param PATH_INFO $fastcgi_path_info;
      fastcgi_param HTTPS on;
      fastcgi_param modHeadersAvailable true;
      fastcgi_param front_controller_active true;
      fastcgi_pass php-handler;
      fastcgi_intercept_errors on;
      fastcgi_request_buffering off;
      fastcgi_keep_conn off;
      fastcgi_buffers 16 256K;
      fastcgi_buffer_size 256k;
      fastcgi_busy_buffers_size 256k;
      fastcgi_temp_file_write_size 256k;
      fastcgi_send_timeout 3000s;
      fastcgi_read_timeout 3000s;
      fastcgi_connect_timeout 3000s;

    location ~ ^\/(?:updater|oc[ms]-provider)(?:$|\/) {
      try_files $uri/ =404;
      index index.php;

    location ~ \.(?:css|js|woff2?|svg|gif)$ {
      try_files $uri /index.php$uri$is_args$args;
      add_header Cache-Control "public, max-age=15778463";
      add_header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload;";
      # HEADERS
      add_header X-Content-Type-Options nosniff;
      add_header X-XSS-Protection "1; mode=block";
      add_header X-Robots-Tag none;
      add_header X-Download-Options noopen;
      add_header X-Permitted-Cross-Domain-Policies none;
      access_log off;

    location ~ \.(?:png|html|ttf|ico|jpg|jpeg)$ {
      try_files $uri /index.php$uri$is_args$args;
      access_log off;

OpenSSL HTTPS Certificates

We will generate a certificates needed for HTTPS service for Nextcloud.

# mkdir -p /usr/local/etc/nginx/ssl

# cd /usr/local/etc/nginx/ssl

# openssl genrsa -des3 -out server.key 2048
Generating RSA private key, 2048 bit long modulus (2 primes)
e is 65537 (0x010001)
Enter pass phrase for server.key: SERVER_KEY_PASSWORD
Verifying - Enter pass phrase for server.key: SERVER_KEY_PASSWORD

As usual use something more sensible then SERVER_KEY_PASSWORD string here ๐Ÿ™‚

# openssl req -new -key server.key -out server.csr
Enter pass phrase for server.key:
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
Country Name (2 letter code) [AU]:PL
State or Province Name (full name) [Some-State]:lodzkie
Locality Name (eg, city) []:Lodz
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Vermaden Enterprises Ltd.
Organizational Unit Name (eg, section) []:IT Department
Common Name (e.g. server FQDN or YOUR name) []:nextcloud.domain.com
Email Address []:vermaden@interia.pl

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:

# cp server.key server.key.orig

# openssl rsa -in server.key.orig -out server.key
Enter pass phrase for server.key.orig: SERVER_KEY_PASSWORD
writing RSA key

# ls -l /usr/local/etc/nginx/ssl
total 7
-rw-r--r--  1 root  wheel  1151 2019.12.31 12:39 server.csr
-rw-------  1 root  wheel  1679 2019.12.31 12:41 server.key
-rw-------  1 root  wheel  1751 2019.12.31 12:40 server.key.orig

# openssl x509 -req -days 7000 -in server.csr -signkey server.key -out server.crt
Signature ok
subject=C = PL, ST = lodzkie, L = Lodz, O = Vermaden Enterprises Ltd., OU = IT Department, CN = nextcloud.domain.com, emailAddress = vermaden@interia.pl
Getting Private key

# ln -s /usr/local/etc/nginx/ssl/server.crt /usr/local/etc/nginx/ssl/ssl-bundle.crt


Here is the used PHP configuration with up to 16GB files for Nextcloud.

# grep '^[^;]' /usr/local/etc/php.ini
engine = On
short_open_tag = On
precision = 14
output_buffering = OFF
zlib.output_compression = Off
implicit_flush = Off
unserialize_callback_func =
serialize_precision = 17
disable_functions =
disable_classes =
zend.enable_gc = On
expose_php = On
max_execution_time = 3600
max_input_time = 30000
memory_limit = 1024M
error_reporting = E_ALL & ~E_DEPRECATED & ~E_STRICT
display_errors = Off
display_startup_errors = Off
log_errors = On
log_errors_max_len = 1024
ignore_repeated_errors = Off
ignore_repeated_source = Off
report_memleaks = On
track_errors = Off
html_errors = On
error_log = /var/log/php.log
variables_order = "GPCS"
request_order = "GP"
register_argc_argv = Off
auto_globals_jit = On
post_max_size = 16400M
auto_prepend_file =
auto_append_file =
default_mimetype = "text/html"
default_charset = "UTF-8"
doc_root =
user_dir =
enable_dl = Off
file_uploads = On
upload_max_filesize = 16400M
max_file_uploads = 64
allow_url_fopen = On
allow_url_include = Off
default_socket_timeout = 300
[CLI Server]
cli_server.color = On
date.timezone = Europe/Warsaw
pdo_mysql.cache_size = 2000
[mail function]
SMTP = localhost
smtp_port = 25
mail.add_x_header = On
sql.safe_mode = Off
odbc.allow_persistent = On
odbc.check_persistent = On
odbc.max_persistent = -1
odbc.max_links = -1
odbc.defaultlrl = 4096
odbc.defaultbinmode = 1
ibase.allow_persistent = 1
ibase.max_persistent = -1
ibase.max_links = -1
ibase.timestampformat = "%Y-%m-%d %H:%M:%S"
ibase.dateformat = "%Y-%m-%d"
ibase.timeformat = "%H:%M:%S"
mysqli.max_persistent = -1
mysqli.allow_persistent = On
mysqli.max_links = -1
mysqli.cache_size = 2000
mysqli.default_port = 3306
mysqli.default_socket =
mysqli.default_host =
mysqli.default_user =
mysqli.default_pw =
mysqli.reconnect = Off
mysqlnd.collect_statistics = On
mysqlnd.collect_memory_statistics = Off
pgsql.allow_persistent = On
pgsql.auto_reset_persistent = Off
pgsql.max_persistent = -1
pgsql.max_links = -1
pgsql.ignore_notice = 0
pgsql.log_notice = 0
bcmath.scale = 0
session.save_handler = files
session.save_path = "/tmp"
session.use_strict_mode = 0
session.use_cookies = 1
session.use_only_cookies = 1
session.name = PHPSESSID
session.auto_start = 0
session.cookie_lifetime = 0
session.cookie_path = /
session.cookie_domain =
session.cookie_httponly =
session.serialize_handler = php
session.gc_probability = 1
session.gc_divisor = 1000
session.gc_maxlifetime = 1440
session.referer_check =
session.cache_limiter = nocache
session.cache_expire = 180
session.use_trans_sid = 0
session.hash_function = 0
session.hash_bits_per_character = 5
url_rewriter.tags = "a=href,area=href,frame=src,input=src,form=fakeentry"
zend.assertions = -1
tidy.clean_output = Off
soap.wsdl_cache_limit = 5
ldap.max_links = -1

PHP PostgreSQL Database Settings

Below are needed to make PHP work with PostgreSQL database.

# cat /usr/local/etc/php/ext-20-pgsql.ini

# cat  /usr/local/etc/php/ext-20-pgsql.ini

pgsql.allow_persistent = On
pgsql.auto_reset_persistent = Off
pgsql.max_persistent = -1
pgsql.max_links = -1
pgsql.ignore_notice = 0
pgsql.log_notice = 0

# cat /usr/local/etc/php/ext-20-pgsql.ini

pgsql.allow_persistent = On
pgsql.auto_reset_persistent = Off
pgsql.max_persistent = -1
pgsql.max_links = -1
pgsql.ignore_notice = 0
pgsql.log_notice = 0

… and the second one.

# cat /usr/local/etc/php/ext-30-pdo_pgsql.ini

# cat  /usr/local/etc/php/ext-30-pdo_pgsql.ini

pgsql.allow_persistent = On
pgsql.auto_reset_persistent = Off
pgsql.max_persistent = -1
pgsql.max_links = -1
pgsql.ignore_notice = 0
pgsql.log_notice = 0

# cat /usr/local/etc/php/ext-30-pdo_pgsql.ini

pgsql.allow_persistent = On
pgsql.auto_reset_persistent = Off
pgsql.max_persistent = -1
pgsql.max_links = -1
pgsql.ignore_notice = 0
pgsql.log_notice = 0


Now the PHP FPM daemon.

# grep '^[^;]' /usr/local/etc/php-fpm.conf
pid = run/php-fpm.pid
error_log = log/php-fpm.log
syslog.facility = daemon

# touch /var/log/php-fpm.log

# chown www:www /var/log/php-fpm.log

# grep '^[^;]' /usr/local/etc/php-fpm.d/www.conf
user = www
group = www
listen =
listen.backlog = -1
listen.owner = www
listen.group = www
listen.mode = 0660
listen.allowed_clients =
pm = static
pm.max_children = 8
pm.start_servers = 4
pm.min_spare_servers = 4
pm.max_spare_servers = 32
pm.process_idle_timeout = 1000s;
pm.max_requests = 500
request_terminate_timeout = 0
rlimit_files = 51200
env[PATH] = /usr/local/bin:/usr/bin:/bin
env[TMP] = /tmp
env[TMPDIR] = /tmp
env[TEMP] = /tmp

Start Backend Services

We will now start all ‘backend’ services needed for Nextcloud.

# service postgresql start
2020-01-02 13:18:05.970 CET [52233] LOG:  starting PostgreSQL 12.1 on amd64-portbld-freebsd12.0, compiled by FreeBSD clang version 6.0.1 (tags/RELEASE_601/final 335540) (based on LLVM 6.0.1), 64-bit
2020-01-02 13:18:05.974 CET [52233] LOG:  listening on IPv6 address "::1", port 5432
2020-01-02 13:18:05.974 CET [52233] LOG:  listening on IPv4 address "", port 5432
2020-01-02 13:18:05.975 CET [52233] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"
2020-01-02 13:18:06.024 CET [52233] LOG:  ending log output to stderr
2020-01-02 13:18:06.024 CET [52233] HINT:  Future log output will go to log destination "syslog".

# service postgresql status
pg_ctl: server is running (PID: 36089)
/usr/local/bin/postgres "-D" "/var/db/postgres/data12"

# service php-fpm start
Performing sanity check on php-fpm configuration:
[02-Jan-2020 13:16:50] NOTICE: configuration file /usr/local/etc/php-fpm.conf test is successful

Starting php_fpm.

# service php-fpm status
php_fpm is running as pid 52193.

# service memcached start
Starting memcached.

# service memcached status
memcached is running as pid 52273.

# service nginx start
Performing sanity check on nginx configuration:
nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok
nginx: configuration file /usr/local/etc/nginx/nginx.conf test is successful
Starting nginx.

Nextcloud Configuration

I created a link named /data to the Nextcloud data directory located at /usr/local/www/nextcloud/data place.

# ln -s /usr/local/www/nextcloud/data /data

The we use Firefox or other web browser to finish the Nextcloud configuration.

Type in the browser where is your Nextcloud instance IP address.

I am sorry but the following image is in the Polish language – I forgot to change it to English … but I assume you will what to put in these fields by context.


After we finish the setup we go straight to Nextcloud Overview page at page to see what else needs to be taken care of.


Two issues needs to be addressed. One is about Nginx configuration, the other is about PostgreSQL, let’s fix them.

We will add needed header to the Nginx configuration file.

# diff -u /usr/local/etc/nginx/nginx.conf.OLD /usr/local/etc/nginx/nginx.conf
--- /usr/local/etc/nginx/nginx.conf.OLD  2020-01-02 14:21:58.359398000 +0100
+++ /usr/local/etc/nginx/nginx.conf      2020-01-02 14:21:42.823426000 +0100
@@ -46,6 +46,7 @@
     add_header X-Robots-Tag none;
     add_header X-Download-Options noopen;
     add_header X-Permitted-Cross-Domain-Policies none;
+    add_header X-Frame-Options "SAMEORIGIN";

     root /usr/local/www/nextcloud/;

# service nginx reload
Performing sanity check on nginx configuration:
nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok
nginx: configuration file /usr/local/etc/nginx/nginx.conf test is successful

… and update the PostgreSQL convertion.

# sudo -u www /usr/local/bin/php /usr/local/www/nextcloud/occ db:convert-filecache-bigint
Following columns will be updated:

* mounts.storage_id
* mounts.root_id
* mounts.mount_id

This can take up to hours, depending on the number of files in your instance!
Continue with the conversion (y/n)? [n] y

Viola! Both of our problems are gone now.


Trusted Domains

When you will enter the Nextcloud using different domain you will get a warning about that.

To add new Trusted Domain to the Nextcloud config do the following.

Here is how it looks before changes.

# grep -A 3 trusted /usr/local/www/nextcloud/config/config.php
  'trusted_domains' =>
  array (
    0 => '',

We will now add nextcloud.domain.com domain.

# vi /usr/local/www/nextcloud/config/config.php

# grep -A 4 trusted /usr/local/www/nextcloud/config/config.php
  'trusted_domains' =>
  array (
    0 => '',
    1 => 'nextcloud.domain.com',

You can of course add more with successive numbers.

# grep -A 5 trusted /usr/local/www/nextcloud/config/config.php
  'trusted_domains' =>
  array (
    0 => '',
    1 => 'nextcloud.domain.com',
    2 => 'cloud.domain.com',

This is the end of this guide. Feel free to share your thougths ๐Ÿ™‚

Log Rotation with Newsyslog

Newsyslog is part of FreeBSD’s base system. We will add Nextcloud and backend daemons log files to Newsyslog configuration so they will be rotated.

# cat  /etc/newsyslog.conf
/data/nextcloud.log                          www:www     640  7     *    @T00  JC
/usr/local/www/nextcloud/data/nextcloud.log  www:www     640  7     *    @T00  JC
/var/log/php-fpm.log                         www:www     640  7     *    @T00  JC
/var/log/nginx/error.log                     www:www     640  7     *    @T00  JC
/var/log/nginx/access.log                    www:www     640  7     *    @T00  JC

Now you will not run out of free space when logs will grow in time.



FreeBSD Enterprise 1 PB Storage

Today FreeBSD operating system turns 26 years old. 19 June is an International FreeBSD Day. This is why I got something special today :). How about using FreeBSD as an Enterprise Storage solution on real hardware? This where FreeBSD shines with all its storage features ZFS included.

Today I will show you how I have built so called Enterprise Storage based on FreeBSD system along with more then 1 PB (Petabyte) of raw capacity.

I have build various storage related systems based on FreeBSD:

This project is different. How much storage space can you squeeze from a single 4U system? It turns out a lot! Definitely more then 1 PB (1024 TB) of raw storage space.

Here is the (non clickable) Table of Contents.

  • Hardware
  • Management Interface
  • FreeBSD System
    • Disks Preparation
    • ZFS Pool Configuration
    • ZFS Settings
    • Network Configuration
    • FreeBSD Configuration
  • Purpose
  • Performance
    • Network Performance
    • Disk Subsystem Performance
  • FreeNAS
  • UPDATE 1 – BSD Now 305
  • UPDATE 2 โ€“ Real Life Pictures in Data Center


There are 4U servers with 90-100 3.5″ drive slots which will allow you to pack 1260-1400 Terabytes of data (with 14 TB drives). Examples of such systems are:

I would use the first one – the TYAN FA100 for short name.


While both GlusterFS and Minio clusters were cone on virtual hardware (or even FreeBSD Jails containers) this one uses real physical hardware.

The build has following specifications.

 2 x 10-Core Intel Xeon Silver 4114 CPU @ 2.20GHz
 4 x 32 GB RAM DDR4 (128 GB Total)
 2 x Intel SSD DC S3500 240 GB (System)
90 x Toshiba HDD MN07ACA12TE 12 TB (Data)
 2 x Broadcom SAS3008 Controller
 2 x Intel X710 DA-2 10GE Card
 2 x Power Supply

Price of the whole system is about $65 000 – drives included. Here is how it looks.


One thing that you will need is a rack cabinet that is 1200 mm long to fit that monster ๐Ÿ™‚

Management Interface

The so called Lights Out management interface is really nice. Its not bloated, well organized and works quite fast. you can create several separate user accounts or can connect to external user services like LDAP/AD/RADIUS for example.


After logging in a simple Dashboard welcomes us.


We have access to various Sensor information available with temperatures of system components.


We have System Inventory information with installed hardware.


There is separate Settings menu for various setup options.


I know its 2019 but HTML5 only Remote Control (remote console) without need for any third party plugins like Java/Silverlight/Flash/… is very welcomed. It works very well too.



One is of course allowed to power on/off/cycle the box remotely.


The Maintenance menu for BIOS updates.



After booting into the BIOS/UEFI setup its possible to select from which drives to boot from. On the screenshots the two SSD drives prepared for system.


The BIOS/UEFI interface shows two Enclosures but its two Broadcom SAS3008 controllers. Some drive are attached via first Broadcom SAS3008 controller, the rest is attached via the second one, and they call them Enclosures instead od of controllers for some reason.


FreeBSD System

I have chosen latest FreeBSD 12.0-RELEASE for the purpose of this installation. Its generally very ‘default’ installation with ZFS mirror on two SSD disks. Nothing special.


The installation of course supports the ZFS Boot Environments bulletproof upgrades/changes feature.

# zpool list zroot
zroot   220G  3.75G   216G        -         -     0%     1%  1.00x  ONLINE  -

# zpool status zroot
  pool: zroot
 state: ONLINE
  scan: none requested

        NAME        STATE     READ WRITE CKSUM
        zroot       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            da91p4  ONLINE       0     0     0
            da11p4  ONLINE       0     0     0

errors: No known data errors

# df -g
Filesystem              1G-blocks Used  Avail Capacity  Mounted on
zroot/ROOT/default            211    2    209     1%    /
devfs                           0    0      0   100%    /dev
zroot/tmp                     209    0    209     0%    /tmp
zroot/usr/home                209    0    209     0%    /usr/home
zroot/usr/ports               210    0    209     0%    /usr/ports
zroot/usr/src                 210    0    209     0%    /usr/src
zroot/var/audit               209    0    209     0%    /var/audit
zroot/var/crash               209    0    209     0%    /var/crash
zroot/var/log                 209    0    209     0%    /var/log
zroot/var/mail                209    0    209     0%    /var/mail
zroot/var/tmp                 209    0    209     0%    /var/tmp

# beadm list
BE      Active Mountpoint  Space Created
default NR     /            2.4G 2019-05-24 13:24

Disks Preparation

From all the possible setups with 90 disks of 12 TB capacity I have chosen to go the RAID60 way – its ZFS equivalent of course. With 12 disks in each RAID6 (raidz2) group – there will be 7 such groups – we will have 84 used for the ZFS pool with 6 drives left as SPARE disks – that plays well for me. The disks distribution will look more or less like that.

   12  raidz2-0
   12  raidz2-1
   12  raidz2-2
   12  raidz2-3
   12  raidz2-4
   12  raidz2-5
   12  raidz2-6
    6  spares
   90  TOTAL

Here is how FreeBSD system sees these drives by camcontrol(8) command. Sorted by attached SAS controller – scbus(4).

# camcontrol devlist | sort -k 6
(AHCI SGPIO Enclosure 1.00 0001)   at scbus2 target 0 lun 0 (pass0,ses0)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 50 lun 0 (pass1,da0)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 52 lun 0 (pass2,da1)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 54 lun 0 (pass3,da2)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 56 lun 0 (pass5,da4)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 57 lun 0 (pass6,da5)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 59 lun 0 (pass7,da6)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 60 lun 0 (pass8,da7)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 66 lun 0 (pass9,da8)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 67 lun 0 (pass10,da9)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 74 lun 0 (pass11,da10)
(ATA INTEL SSDSC2KB24 0100)        at scbus3 target 75 lun 0 (pass12,da11)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 76 lun 0 (pass13,da12)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 82 lun 0 (pass14,da13)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 83 lun 0 (pass15,da14)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 85 lun 0 (pass16,da15)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 87 lun 0 (pass17,da16)
(Tyan B7118 0500)                  at scbus3 target 88 lun 0 (pass18,ses1)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 89 lun 0 (pass19,da17)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 90 lun 0 (pass20,da18)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 91 lun 0 (pass21,da19)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 92 lun 0 (pass22,da20)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 93 lun 0 (pass23,da21)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 94 lun 0 (pass24,da22)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 95 lun 0 (pass25,da23)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 96 lun 0 (pass26,da24)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 97 lun 0 (pass27,da25)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 98 lun 0 (pass28,da26)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 99 lun 0 (pass29,da27)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 100 lun 0 (pass30,da28)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 101 lun 0 (pass31,da29)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 102 lun 0 (pass32,da30)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 103 lun 0 (pass33,da31)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 104 lun 0 (pass34,da32)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 105 lun 0 (pass35,da33)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 106 lun 0 (pass36,da34)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 107 lun 0 (pass37,da35)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 108 lun 0 (pass38,da36)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 109 lun 0 (pass39,da37)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 110 lun 0 (pass40,da38)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 48 lun 0 (pass41,da39)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 49 lun 0 (pass42,da40)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 51 lun 0 (pass43,da41)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 53 lun 0 (pass44,da42)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 55 lun 0 (da43,pass45)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 59 lun 0 (pass46,da44)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 64 lun 0 (pass47,da45)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 67 lun 0 (pass48,da46)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 68 lun 0 (pass49,da47)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 69 lun 0 (pass50,da48)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 73 lun 0 (pass51,da49)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 76 lun 0 (pass52,da50)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 77 lun 0 (pass53,da51)
(Tyan B7118 0500)                  at scbus4 target 80 lun 0 (pass54,ses2)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 81 lun 0 (pass55,da52)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 82 lun 0 (pass56,da53)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 83 lun 0 (pass57,da54)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 84 lun 0 (pass58,da55)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 85 lun 0 (pass59,da56)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 86 lun 0 (pass60,da57)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 87 lun 0 (pass61,da58)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 88 lun 0 (pass62,da59)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 89 lun 0 (da63,pass66)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 90 lun 0 (pass64,da61)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 91 lun 0 (pass65,da62)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 92 lun 0 (da60,pass63)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 94 lun 0 (pass67,da64)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 97 lun 0 (pass68,da65)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 98 lun 0 (pass69,da66)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 99 lun 0 (pass70,da67)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 100 lun 0 (pass71,da68)
(Tyan B7118 0500)                  at scbus4 target 101 lun 0 (pass72,ses3)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 102 lun 0 (pass73,da69)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 103 lun 0 (pass74,da70)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 104 lun 0 (pass75,da71)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 105 lun 0 (pass76,da72)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 106 lun 0 (pass77,da73)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 107 lun 0 (pass78,da74)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 108 lun 0 (pass79,da75)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 109 lun 0 (pass80,da76)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 110 lun 0 (pass81,da77)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 111 lun 0 (pass82,da78)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 112 lun 0 (pass83,da79)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 113 lun 0 (pass84,da80)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 114 lun 0 (pass85,da81)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 115 lun 0 (pass86,da82)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 116 lun 0 (pass87,da83)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 117 lun 0 (pass88,da84)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 118 lun 0 (pass89,da85)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 119 lun 0 (pass90,da86)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 120 lun 0 (pass91,da87)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 121 lun 0 (pass92,da88)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 122 lun 0 (pass93,da89)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 123 lun 0 (pass94,da90)
(ATA INTEL SSDSC2KB24 0100)        at scbus4 target 124 lun 0 (pass95,da91)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 125 lun 0 (da3,pass4)

One my ask how to identify which disk is which when the FAILURE will came … this is where FreeBSD’s sesutil(8) command comes handy.

# sesutil locate all off
# sesutil locate da64 on

The first sesutil(8) command disables all location lights in the enclosure. The second one turns on the identification for disk da64.

I will also make sure to NOT use the whole space of each drive. Such idea may be pointless but imagine the following situation. Five 12 TB disks failed after 3 years. You can not get the same model drives so you get other 12 TB drives, maybe even from other manufacturer.

# grep da64 /var/run/dmesg.boot
da64 at mpr1 bus 0 scbus4 target 93 lun 0
da64:  Fixed Direct Access SPC-4 SCSI device
da64: Serial Number 98G0A1EQF95G
da64: 1200.000MB/s transfers
da64: Command Queueing enabled
da64: 11444224MB (23437770752 512 byte sectors)

A single 12 TB drive has 23437770752 of 512 byte sectors which equals 12000138625024 bytes of raw capacity.

# expr 23437770752 \* 512

Now image that these other 12 TB drives from other manufacturer will come with 4 bytes smaller size … ZFS will not allow their usage because their size is smaller.

This is why I will use exactly 11175 GB size of each drive which is more or less 1 GB short of its total 11176 GB size.

Below is command that will do that for me for all 90 disks.

# camcontrol devlist \
    | grep TOSHIBA \
    | awk '{print $NF}' \
    | awk -F ',' '{print $2}' \
    | tr -d ')' \
    | while read DISK
        gpart destroy -F                   ${DISK} 1> /dev/null 2> /dev/null
        gpart create -s GPT                ${DISK}
        gpart add -t freebsd-zfs -s 11175G ${DISK}

# gpart show da64
=>         40  23437770672  da64  GPT  (11T)
           40  23435673600     1  freebsd-zfs  (11T)
  23435673640      2097072        - free -  (1.0G)

ZFS Pool Configuration

Next, we will have to create our ZFS pool, its probably the longest zpool command I have ever executed ๐Ÿ™‚

As the Toshiba 12 TB disks have 4k sectors we will need to set vfs.zfs.min_auto_ashift to 12 to force them.

# sysctl vfs.zfs.min_auto_ashift=12
vfs.zfs.min_auto_ashift: 12 -> 12

# zpool create nas02 \
    raidz2  da0p1  da1p1  da2p1  da3p1  da4p1  da5p1  da6p1  da7p1  da8p1  da9p1 da10p1 da12p1 \
    raidz2 da13p1 da14p1 da15p1 da16p1 da17p1 da18p1 da19p1 da20p1 da21p1 da22p1 da23p1 da24p1 \
    raidz2 da25p1 da26p1 da27p1 da28p1 da29p1 da30p1 da31p1 da32p1 da33p1 da34p1 da35p1 da36p1 \
    raidz2 da37p1 da38p1 da39p1 da40p1 da41p1 da42p1 da43p1 da44p1 da45p1 da46p1 da47p1 da48p1 \
    raidz2 da49p1 da50p1 da51p1 da52p1 da53p1 da54p1 da55p1 da56p1 da57p1 da58p1 da59p1 da60p1 \
    raidz2 da61p1 da62p1 da63p1 da64p1 da65p1 da66p1 da67p1 da68p1 da69p1 da70p1 da71p1 da72p1 \
    raidz2 da73p1 da74p1 da75p1 da76p1 da77p1 da78p1 da79p1 da80p1 da81p1 da82p1 da83p1 da84p1 \
    spare  da85p1 da86p1 da87p1 da88p1 da89p1 da90p1

# zpool status
  pool: nas02
 state: ONLINE
  scan: scrub repaired 0 in 0 days 00:00:05 with 0 errors on Fri May 31 10:26:29 2019

        NAME        STATE     READ WRITE CKSUM
        nas02       ONLINE       0     0     0
          raidz2-0  ONLINE       0     0     0
            da0p1   ONLINE       0     0     0
            da1p1   ONLINE       0     0     0
            da2p1   ONLINE       0     0     0
            da3p1   ONLINE       0     0     0
            da4p1   ONLINE       0     0     0
            da5p1   ONLINE       0     0     0
            da6p1   ONLINE       0     0     0
            da7p1   ONLINE       0     0     0
            da8p1   ONLINE       0     0     0
            da9p1   ONLINE       0     0     0
            da10p1  ONLINE       0     0     0
            da12p1  ONLINE       0     0     0
          raidz2-1  ONLINE       0     0     0
            da13p1  ONLINE       0     0     0
            da14p1  ONLINE       0     0     0
            da15p1  ONLINE       0     0     0
            da16p1  ONLINE       0     0     0
            da17p1  ONLINE       0     0     0
            da18p1  ONLINE       0     0     0
            da19p1  ONLINE       0     0     0
            da20p1  ONLINE       0     0     0
            da21p1  ONLINE       0     0     0
            da22p1  ONLINE       0     0     0
            da23p1  ONLINE       0     0     0
            da24p1  ONLINE       0     0     0
          raidz2-2  ONLINE       0     0     0
            da25p1  ONLINE       0     0     0
            da26p1  ONLINE       0     0     0
            da27p1  ONLINE       0     0     0
            da28p1  ONLINE       0     0     0
            da29p1  ONLINE       0     0     0
            da30p1  ONLINE       0     0     0
            da31p1  ONLINE       0     0     0
            da32p1  ONLINE       0     0     0
            da33p1  ONLINE       0     0     0
            da34p1  ONLINE       0     0     0
            da35p1  ONLINE       0     0     0
            da36p1  ONLINE       0     0     0
          raidz2-3  ONLINE       0     0     0
            da37p1  ONLINE       0     0     0
            da38p1  ONLINE       0     0     0
            da39p1  ONLINE       0     0     0
            da40p1  ONLINE       0     0     0
            da41p1  ONLINE       0     0     0
            da42p1  ONLINE       0     0     0
            da43p1  ONLINE       0     0     0
            da44p1  ONLINE       0     0     0
            da45p1  ONLINE       0     0     0
            da46p1  ONLINE       0     0     0
            da47p1  ONLINE       0     0     0
            da48p1  ONLINE       0     0     0
          raidz2-4  ONLINE       0     0     0
            da49p1  ONLINE       0     0     0
            da50p1  ONLINE       0     0     0
            da51p1  ONLINE       0     0     0
            da52p1  ONLINE       0     0     0
            da53p1  ONLINE       0     0     0
            da54p1  ONLINE       0     0     0
            da55p1  ONLINE       0     0     0
            da56p1  ONLINE       0     0     0
            da57p1  ONLINE       0     0     0
            da58p1  ONLINE       0     0     0
            da59p1  ONLINE       0     0     0
            da60p1  ONLINE       0     0     0
          raidz2-5  ONLINE       0     0     0
            da61p1  ONLINE       0     0     0
            da62p1  ONLINE       0     0     0
            da63p1  ONLINE       0     0     0
            da64p1  ONLINE       0     0     0
            da65p1  ONLINE       0     0     0
            da66p1  ONLINE       0     0     0
            da67p1  ONLINE       0     0     0
            da68p1  ONLINE       0     0     0
            da69p1  ONLINE       0     0     0
            da70p1  ONLINE       0     0     0
            da71p1  ONLINE       0     0     0
            da72p1  ONLINE       0     0     0
          raidz2-6  ONLINE       0     0     0
            da73p1  ONLINE       0     0     0
            da74p1  ONLINE       0     0     0
            da75p1  ONLINE       0     0     0
            da76p1  ONLINE       0     0     0
            da77p1  ONLINE       0     0     0
            da78p1  ONLINE       0     0     0
            da79p1  ONLINE       0     0     0
            da80p1  ONLINE       0     0     0
            da81p1  ONLINE       0     0     0
            da82p1  ONLINE       0     0     0
            da83p1  ONLINE       0     0     0
            da84p1  ONLINE       0     0     0
          da85p1    AVAIL
          da86p1    AVAIL
          da87p1    AVAIL
          da88p1    AVAIL
          da89p1    AVAIL
          da90p1    AVAIL

errors: No known data errors

# zpool list nas02
nas02   915T  1.42M   915T        -         -     0%     0%  1.00x  ONLINE  -

# zfs list nas02
nas02    88K   675T   201K  none

ZFS Settings

As the primary role of this storage would be keeping files I will use one of the largest values for recordsize – 1 MB – this helps getting better compression ratio.

… but it will also serve as iSCSI Target in which we will try to fit in the native 4k blocks – thus 4096 bytes setting for iSCSI.

# zfs set compression=lz4         nas02
# zfs set atime=off               nas02
# zfs set mountpoint=none         nas02
# zfs set recordsize=1m           nas02
# zfs set redundant_metadata=most nas02
# zfs create                      nas02/nfs
# zfs create                      nas02/smb
# zfs create                      nas02/iscsi
# zfs set recordsize=4k           nas02/iscsi

Also one word on redundant_metadata as its not that obvious parameter. To quote the zfs(8) man page.

# man zfs
redundant_metadata=all | most
  Controls what types of metadata are stored redundantly.  ZFS stores
  an extra copy of metadata, so that if a single block is corrupted,
  the amount of user data lost is limited.  This extra copy is in
  addition to any redundancy provided at the pool level (e.g. by
  mirroring or RAID-Z), and is in addition to an extra copy specified
  by the copies property (up to a total of 3 copies).  For example if
  the pool is mirrored, copies=2, and redundant_metadata=most, then ZFS
  stores 6 copies of most metadata, and 4 copies of data and some

  When set to all, ZFS stores an extra copy of all metadata.  If a
  single on-disk block is corrupt, at worst a single block of user data
  (which is recordsize bytes long can be lost.)

  When set to most, ZFS stores an extra copy of most types of metadata.
  This can improve performance of random writes, because less metadata
  must be written.  In practice, at worst about 100 blocks (of
  recordsize bytes each) of user data can be lost if a single on-disk
  block is corrupt.  The exact behavior of which metadata blocks are
  stored redundantly may change in future releases.

  The default value is all.

From the description above we can see that its mostly useful on single device pools because when we have redundancy based on RAIDZ2 (RAID6 equivalent) we do not need to keep additional redundant copies of metadata. This helps to increase write performance.

For the record – iSCSI ZFS zvols are create with command like that one below – as sparse files – also called Thin Provisioning mode.

# zfs create -s -V 16T nas02/iscsi/test

As we have SPARE disks we will also need to enable the zfsd(8) daemon by adding zfsd_enable=YES to the /etc/rc.conf file.

We also need to enable autoreplace property for our pool because by default its set to off.

# zpool get autoreplace nas02
nas02  autoreplace  off      default

# zpool set autoreplace=on nas02

# zpool get autoreplace nas02
nas02  autoreplace  on       local

Other ZFS settings are in the /boot/loader.conf file. As this system has 128 GB RAM we will let ZFS use 50 to 75% of that amount for ARC.

# grep vfs.zfs /boot/loader.conf

Network Configuration

This is what I really like about FreeBSD. To setup LACP link aggregation tou just need 5 lines in /etc/rc.conf file. On Red Hat Enterprise Linux you would need several files with many lines each.

# head -5 /etc/rc.conf
  ifconfig_lagg0="laggproto lacp laggport ixl0 laggport ixl1 up"

# ifconfig lagg0
lagg0: flags=8843 metric 0 mtu 1500
        ether a0:42:3f:a0:42:3f
        inet netmask 0xffffff00 broadcast
        laggproto lacp lagghash l2,l3,l4
        laggport: ixl0 flags=1c
        laggport: ixl1 flags=1c
        groups: lagg
        media: Ethernet autoselect
        status: active
        nd6 options=29

The Intel X710 DA-2 10GE network adapter is fully supported under FreeBSD by the ixl(4) driver.


Cisco Nexus Configuration

This is the Cisco Nexus configuration needed to enable LACP aggregation.

First the ports.

NEXUS-1  Eth1/32  NAS02_IXL0  connected 3  full  a-10G  SFP-H10GB-A
NEXUS-2  Eth1/32  NAS02_IXL1  connected 3  full  a-10G  SFP-H10GB-A

… and now aggregation.

interface Ethernet1/32
  description NAS02_IXL1
  switchport access vlan 3
  mtu 9216
  channel-group 128 mode active
  no shutdown
interface port-channel128
  description NAS02
  switchport access vlan 3
  mtu 9216
  vpc 128

… and the same/similar on the second Cisco Nexus NEXUS-2 switch.

FreeBSD Configuration

These are three most important configuration files on any FreeBSD system.

I will now post all settings I use on this storage system.

The /etc/rc.conf file.

# cat /etc/rc.conf
  ifconfig_lagg0="laggproto lacp laggport ixl0 laggport ixl1 up"

  kld_list="${kld_list} aesni"




The /boot/loader.conf file.

# cat /boot/loader.conf












The /etc/sysctl.conf file.

# cat /etc/sysctl.conf







Why one would built such appliance? Because its a lot cheaper then to get the ‘branded’ one. Think about Dell EMC Data Domain for example – and not just ‘any’ Data Domain but almost the highest one – the Data Domain DD9300 at least. It would cost about ten times more at least … with smaller capacity and taking not 4U but closer to 14U with three DS60 expanders.

But you can actually make this FreeBSD Enterprise Storage behave like Dell EMC Data Domain .. or like their Dell EMC Elastic Cloud Storage for example.

The Dell EMC CloudBoost can be deployed somewhere on your VMware stack to provide the DDBoost deduplication. Then you would need OpenStack Swift as its one of the supported backed devices.



The OpenStack Swift package in FreeBSD is about 4-5 years behind reality (2.2.2) so you will have to use Bhyve here.

# pkg search swift
py27-swift-2.2.2_1             Highly available, distributed, eventually consistent object/blob store

Create Bhyve virtual machine on this FreeBSD Enterprise Storage with CentOS 7.6 system for example, then setup Swift there, but it will work. With 20 physical cores to spare and 128 GB RAM you would not even noticed its there.

This way you can use Dell EMC Networker with more then ten times cheaper storage.

In the past I also wrote about IBM Spectrum Protect (TSM) which would also greatly benefit from FreeBSD Enterprise Storage. I actually also use this FreeBSD based storage as space for IBM Spectrum Protect (TSM) container pool directories. Exported via iSCSI works like a charm.

You can also compare that FreeBSD Enterprise Storage to other storage appliances like iXsystems TrueNAS or EXAGRID.


You for sure would want to know how fast this FreeBSD Enterprise Storage performs ๐Ÿ™‚

I will share all performance data that I gathered with a pleasure.

Network Performance

First the network performance.

I user iperf3 as the benchmark.

I started the server on the FreeBSD side.

# iperf3 -s

… and then I started client on the Windows Server 2016 machine.

C:\iperf-3.1.3-win64>iperf3.exe -c nas02 -P 8
[SUM]   0.00-10.00  sec  10.8 GBytes  9.26 Gbits/sec                  receiver

This is with MTU 1500 – no Jumbo frames unfortunatelly ๐Ÿ˜ฆ

Unfortunatelly this system has only one physical 10GE interface but I did other test also. Using two such boxes with single 10GE interface. That saturated the dual 10GE LACP on FreeBSD side nicely.

I also exported NFS and iSCSI to Red Hat Enterprise Linux system. The network performance was about 500-600 MB/s on single 10GE interface. That would be 1000-1200 MB/s on LACP aggregation.

Disk Subsystem Performance

Now the disk subsystem.

First some naive test using diskinfo(8) FreeBSD’s builtin tool.

# diskinfo -ctv /dev/da12
        512             # sectorsize
        12000138625024  # mediasize in bytes (11T)
        23437770752     # mediasize in sectors
        4096            # stripesize
        0               # stripeoffset
        1458933         # Cylinders according to firmware.
        255             # Heads according to firmware.
        63              # Sectors according to firmware.
        ATA TOSHIBA MG07ACA1    # Disk descr.
        98H0A11KF95G    # Disk ident.
        id1,enc@n500e081010445dbd/type@0/slot@c/elmdesc@ArrayDevice11   # Physical path
        No              # TRIM/UNMAP support
        7200            # Rotation rate in RPM
        Not_Zoned       # Zone Mode

I/O command overhead:
        time to read 10MB block      0.067031 sec       =    0.003 msec/sector
        time to read 20480 sectors   2.619989 sec       =    0.128 msec/sector
        calculated command overhead                     =    0.125 msec/sector

Seek times:
        Full stroke:      250 iter in   5.665880 sec =   22.664 msec
        Half stroke:      250 iter in   4.263047 sec =   17.052 msec
        Quarter stroke:   500 iter in   6.867914 sec =   13.736 msec
        Short forward:    400 iter in   3.057913 sec =    7.645 msec
        Short backward:   400 iter in   1.979287 sec =    4.948 msec
        Seq outer:       2048 iter in   0.169472 sec =    0.083 msec
        Seq inner:       2048 iter in   0.469630 sec =    0.229 msec

Transfer rates:
        outside:       102400 kbytes in   0.478251 sec =   214114 kbytes/sec
        middle:        102400 kbytes in   0.605701 sec =   169060 kbytes/sec
        inside:        102400 kbytes in   1.303909 sec =    78533 kbytes/sec

So now we know how fast a single disk is.

Let’s repeast the same test on the ZFS zvol device.

# diskinfo -ctv /dev/zvol/nas02/iscsi/test
        512             # sectorsize
        17592186044416  # mediasize in bytes (16T)
        34359738368     # mediasize in sectors
        65536           # stripesize
        0               # stripeoffset
        Yes             # TRIM/UNMAP support
        Unknown         # Rotation rate in RPM

I/O command overhead:
        time to read 10MB block      0.004512 sec       =    0.000 msec/sector
        time to read 20480 sectors   0.196824 sec       =    0.010 msec/sector
        calculated command overhead                     =    0.009 msec/sector

Seek times:
        Full stroke:      250 iter in   0.006151 sec =    0.025 msec
        Half stroke:      250 iter in   0.008228 sec =    0.033 msec
        Quarter stroke:   500 iter in   0.014062 sec =    0.028 msec
        Short forward:    400 iter in   0.010564 sec =    0.026 msec
        Short backward:   400 iter in   0.011725 sec =    0.029 msec
        Seq outer:       2048 iter in   0.028198 sec =    0.014 msec
        Seq inner:       2048 iter in   0.028416 sec =    0.014 msec

Transfer rates:
        outside:       102400 kbytes in   0.036938 sec =  2772213 kbytes/sec
        middle:        102400 kbytes in   0.043076 sec =  2377194 kbytes/sec
        inside:        102400 kbytes in   0.034260 sec =  2988908 kbytes/sec

Almost 3 GB/s – not bad.

Time for even more oldschool test – the immortal dd(8) command.

This is with compression=off setting.

One process.

# dd if=/dev/zero of=FILE bs=128m status=progress
26172456960 bytes (26 GB, 24 GiB) transferred 16.074s, 1628 MB/s
202+0 records in
201+0 records out
26977763328 bytes transferred in 16.660884 secs (1619227644 bytes/sec)

Four concurrent processes.

# dd if=/dev/zero of=FILE${X} bs=128m status=progress
80933289984 bytes (81 GB, 75 GiB) transferred 98.081s, 825 MB/s
608+0 records in
608+0 records out
81604378624 bytes transferred in 98.990579 secs (824365101 bytes/sec)

Eight concurrent processes.

# dd if=/dev/zero of=FILE${X} bs=128m status=progress
174214610944 bytes (174 GB, 162 GiB) transferred 385.042s, 452 MB/s
1302+0 records in
1301+0 records out
174617264128 bytes transferred in 385.379296 secs (453104943 bytes/sec)

Lets summarize that data.

1 STREAM(s) ~ 1600 MB/s ~ 1.5 GB/s
4 STREAM(s) ~ 3300 MB/s ~ 3.2 GB/s
8 STREAM(s) ~ 3600 MB/s ~ 3.5 GB/s

So the disk subsystem is able to squeeze 3.5 GB/s of sustained speed in sequential writes. That us that if we would want to saturate it we would need to add additional two 10GE interfaces.

The disks were stressed only to about 55% which you can see in other useful FreeBSD tool – gstat(8) command.


Time for more ‘intelligent’ tests. The blogbench test.

First with compression disabled.

# time blogbench -d .
Frequency = 10 secs
Scratch dir = [.]
Spawning 3 writers...
Spawning 1 rewriters...
Spawning 5 commenters...
Spawning 100 readers...
Benchmarking for 30 iterations.
The test will run during 5 minutes.
Final score for writes:          6476
Final score for reads :        660436

blogbench -d .  280.58s user 4974.41s system 1748% cpu 5:00.54 total

Second with compression set to LZ4.

# time blogbench -d .
Frequency = 10 secs
Scratch dir = [.]
Spawning 3 writers...
Spawning 1 rewriters...
Spawning 5 commenters...
Spawning 100 readers...
Benchmarking for 30 iterations.
The test will run during 5 minutes.
Final score for writes:          7087
Final score for reads :        733932

blogbench -d .  299.08s user 5415.04s system 1900% cpu 5:00.68 total

Compression did not helped much, but helped.

To have some comparision we will run the same test on the system ZFS pool – two Intel SSD DC S3500 240 GB drives in mirror which have following features.

The Intel SSD DC S3500 240 GB drives:

  • Sequential Read (up to) 500 MB/s
  • Sequential Write (up to) 260 MB/s
  • Random Read (100% Span) 75000 IOPS
  • Random Write (100% Span) 7500 IOPS
# time blogbench -d .
Frequency = 10 secs
Scratch dir = [.]
Spawning 3 writers...
Spawning 1 rewriters...
Spawning 5 commenters...
Spawning 100 readers...
Benchmarking for 30 iterations.
The test will run during 5 minutes.
Final score for writes:          6109
Final score for reads :        654099

blogbench -d .  278.73s user 5058.75s system 1777% cpu 5:00.30 total

Now the randomio test. Its multithreaded disk I/O microbenchmark.

The usage is as follows.

usage: randomio filename nr_threads write_fraction_of_io fsync_fraction_of_writes io_size nr_seconds_between_samples

filename                    Filename or device to read/write.
write_fraction_of_io        What fraction of I/O should be writes - for example 0.25 for 25% write.
fsync_fraction_of_writes    What fraction of writes should be fsync'd.
io_size                     How many bytes to read/write (multiple of 512 bytes).
nr_seconds_between_samples  How many seconds to average samples over.

The randomio with 4k block.

# zfs create -s -V 1T nas02/iscsi/test
# randomio /dev/zvol/nas02/iscsi/test 8 0.25 1 4096 10
  total |  read:         latency (ms)       |  write:        latency (ms)
   iops |   iops   min    avg    max   sdev |   iops   min    avg    max   sdev
54137.7 |40648.4   0.0    0.1  575.8    2.2 |13489.4   0.0    0.3  405.8    2.6
66248.4 |49641.5   0.0    0.1   19.6    0.3 |16606.9   0.0    0.2   26.4    0.7
66411.0 |49817.2   0.0    0.1   19.7    0.3 |16593.8   0.0    0.2   20.3    0.7
64158.9 |48142.8   0.0    0.1  254.7    0.7 |16016.1   0.0    0.2  130.4    1.0
48454.1 |36390.8   0.0    0.1  542.8    2.7 |12063.3   0.0    0.3  507.5    3.2
66796.1 |50067.4   0.0    0.1   24.1    0.3 |16728.7   0.0    0.2   23.4    0.7
58512.2 |43851.7   0.0    0.1  576.5    1.7 |14660.5   0.0    0.2  307.2    1.7
63195.8 |47341.8   0.0    0.1  261.6    0.9 |15854.1   0.0    0.2  361.1    1.9
67086.0 |50335.6   0.0    0.1   20.4    0.3 |16750.4   0.0    0.2   25.1    0.8
67429.8 |50549.6   0.0    0.1   21.8    0.3 |16880.3   0.0    0.2   20.6    0.7

… and with 512 sector.

# zfs create -s -V 1T nas02/iscsi/test
# randomio /dev/zvol/nas02/iscsi/TEST 8 0.25 1 512 10
  total |  read:         latency (ms)       |  write:        latency (ms)
   iops |   iops   min    avg    max   sdev |   iops   min    avg    max   sdev
58218.9 |43712.0   0.0    0.1  501.5    2.1 |14506.9   0.0    0.2  272.5    1.6
66325.3 |49703.8   0.0    0.1  352.0    0.9 |16621.4   0.0    0.2  352.0    1.5
68130.5 |51100.8   0.0    0.1   24.6    0.3 |17029.7   0.0    0.2   24.4    0.7
68465.3 |51352.3   0.0    0.1   19.9    0.3 |17112.9   0.0    0.2   23.8    0.7
54903.5 |41249.1   0.0    0.1  399.3    1.9 |13654.4   0.0    0.3  335.8    2.2
61259.8 |45898.7   0.0    0.1  574.6    1.7 |15361.0   0.0    0.2  371.5    1.7
68483.3 |51313.1   0.0    0.1   22.9    0.3 |17170.3   0.0    0.2   26.1    0.7
56713.7 |42524.7   0.0    0.1  373.5    1.8 |14189.1   0.0    0.2  438.5    2.7
68861.4 |51657.0   0.0    0.1   21.0    0.3 |17204.3   0.0    0.2   21.7    0.7
68602.0 |51438.4   0.0    0.1   19.5    0.3 |17163.7   0.0    0.2   23.7    0.7

Both randomio tests were run with compression set to LZ4.

Next is bonnie++ benchmark. It has been run with compression set to LZ4.

# bonnie++ -d . -u root
Using uid:0, gid:0.
Writing a byte at a time...done
Writing intelligently...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  1.97       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
nas02.local 261368M   139  99 775132  99 589190  99   383  99 1638929  99 12930 2046
Latency             60266us    7030us    7059us   21553us    3844us    5710us
Version  1.97       ------Sequential Create------ --------Random Create--------
nas02.local         -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ 12680  44 +++++ +++ +++++ +++ 30049  99
Latency              2619us      43us     714ms    2748us      28us      58us

… and last but not least the fio benchmark. Also with LZ4 compression enabled.

# fio --randrepeat=1 --direct=1 --gtod_reduce=1 --name=test --filename=random_read_write.fio --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75
test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=64
Starting 1 process
Jobs: 1 (f=1): [m(1)][98.0%][r=38.0MiB/s,w=12.2MiB/s][r=9735,w=3128 IOPS][eta 00m:05s]
test: (groupid=0, jobs=1): err= 0: pid=35368: Tue Jun 18 15:14:44 2019
  read: IOPS=3157, BW=12.3MiB/s (12.9MB/s)(3070MiB/248872msec)
   bw (  KiB/s): min= 9404, max=57732, per=98.72%, avg=12469.84, stdev=3082.99, samples=497
   iops        : min= 2351, max=14433, avg=3117.15, stdev=770.74, samples=497
  write: IOPS=1055, BW=4222KiB/s (4323kB/s)(1026MiB/248872msec)
   bw (  KiB/s): min= 3179, max=18914, per=98.71%, avg=4166.60, stdev=999.23, samples=497
   iops        : min=  794, max= 4728, avg=1041.25, stdev=249.76, samples=497
  cpu          : usr=1.11%, sys=88.64%, ctx=677981, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=785920,262656,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=12.3MiB/s (12.9MB/s), 12.3MiB/s-12.3MiB/s (12.9MB/s-12.9MB/s), io=3070MiB (3219MB), run=248872-248872msec
  WRITE: bw=4222KiB/s (4323kB/s), 4222KiB/s-4222KiB/s (4323kB/s-4323kB/s), io=1026MiB (1076MB), run=248872-248872msec

Dunno how about you but I am satisfied with performance ๐Ÿ™‚


Originally I really wanted to use FreeNAS on these boxes and I even installed FreeNAS on them. It run nicely but … the security part of FreeNAS was not best.

This is the output of pkg audit command. Quite scarry.

root@freenas[~]# pkg audit -F
Fetching vuln.xml.bz2: 100%  785 KiB 804.3kB/s    00:01
python27-2.7.15 is vulnerable:
Python -- NULL pointer dereference vulnerability
CVE: CVE-2019-5010
WWW: https://vuxml.FreeBSD.org/freebsd/d74371d2-4fee-11e9-a5cd-1df8a848de3d.html

curl-7.62.0 is vulnerable:
curl -- multiple vulnerabilities
CVE: CVE-2019-3823
CVE: CVE-2019-3822
CVE: CVE-2018-16890
WWW: https://vuxml.FreeBSD.org/freebsd/714b033a-2b09-11e9-8bc3-610fd6e6cd05.html

libgcrypt-1.8.2 is vulnerable:
libgcrypt -- side-channel attack vulnerability
CVE: CVE-2018-0495
WWW: https://vuxml.FreeBSD.org/freebsd/9b5162de-6f39-11e8-818e-e8e0b747a45a.html

python36-3.6.5_1 is vulnerable:
Python -- NULL pointer dereference vulnerability
CVE: CVE-2019-5010
WWW: https://vuxml.FreeBSD.org/freebsd/d74371d2-4fee-11e9-a5cd-1df8a848de3d.html

pango-1.42.0 is vulnerable:
pango -- remote DoS vulnerability
CVE: CVE-2018-15120
WWW: https://vuxml.FreeBSD.org/freebsd/5a757a31-f98e-4bd4-8a85-f1c0f3409769.html

py36-requests-2.18.4 is vulnerable:
www/py-requests -- Information disclosure vulnerability
WWW: https://vuxml.FreeBSD.org/freebsd/50ad9a9a-1e28-11e9-98d7-0050562a4d7b.html

libnghttp2-1.31.0 is vulnerable:
nghttp2 -- Denial of service due to NULL pointer dereference
CVE: CVE-2018-1000168
WWW: https://vuxml.FreeBSD.org/freebsd/1fccb25e-8451-438c-a2b9-6a021e4d7a31.html

gnupg-2.2.6 is vulnerable:
gnupg -- unsanitized output (CVE-2018-12020)
CVE: CVE-2017-7526
CVE: CVE-2018-12020
WWW: https://vuxml.FreeBSD.org/freebsd/7da0417f-6b24-11e8-84cc-002590acae31.html

py36-cryptography-2.1.4 is vulnerable:
py-cryptography -- tag forgery vulnerability
CVE: CVE-2018-10903
WWW: https://vuxml.FreeBSD.org/freebsd/9e2d0dcf-9926-11e8-a92d-0050562a4d7b.html

perl5-5.26.1 is vulnerable:
perl -- multiple vulnerabilities
CVE: CVE-2018-6913
CVE: CVE-2018-6798
CVE: CVE-2018-6797
WWW: https://vuxml.FreeBSD.org/freebsd/41c96ffd-29a6-4dcc-9a88-65f5038fa6eb.html

libssh2-1.8.0,3 is vulnerable:
libssh2 -- multiple issues
CVE: CVE-2019-3862
CVE: CVE-2019-3861
CVE: CVE-2019-3860
CVE: CVE-2019-3858
WWW: https://vuxml.FreeBSD.org/freebsd/6e58e1e9-2636-413e-9f84-4c0e21143628.html

git-lite-2.17.0 is vulnerable:
Git -- Fix memory out-of-bounds and remote code execution vulnerabilities (CVE-2018-11233 and CVE-2018-11235)
CVE: CVE-2018-11235
CVE: CVE-2018-11233
WWW: https://vuxml.FreeBSD.org/freebsd/c7a135f4-66a4-11e8-9e63-3085a9a47796.html

gnutls-3.5.18 is vulnerable:
GnuTLS -- double free, invalid pointer access
CVE: CVE-2019-3836
CVE: CVE-2019-3829
WWW: https://vuxml.FreeBSD.org/freebsd/fb30db8f-62af-11e9-b0de-001cc0382b2f.html

13 problem(s) in the installed packages found.

root@freenas[~]# uname -a
FreeBSD freenas.local 11.2-STABLE FreeBSD 11.2-STABLE #0 r325575+95cc58ca2a0(HEAD): Mon May  6 19:08:58 EDT 2019     root@mp20.tn.ixsystems.com:/freenas-releng/freenas/_BE/objs/freenas-releng/freenas/_BE/os/sys/FreeNAS.amd64  amd64

root@freenas[~]# freebsd-version -uk

root@freenas[~]# sockstat -l4
root     uwsgi-3.6  4006  3  tcp4        *:*
root     uwsgi-3.6  3188  3  tcp4        *:*
nobody   mdnsd      3144  4  udp4   *:31417               *:*
nobody   mdnsd      3144  6  udp4   *:5353                *:*
www      nginx      3132  6  tcp4   *:443                 *:*
www      nginx      3132  8  tcp4   *:80                  *:*
root     nginx      3131  6  tcp4   *:443                 *:*
root     nginx      3131  8  tcp4   *:80                  *:*
root     ntpd       2823  21 udp4   *:123                 *:*
root     ntpd       2823  22 udp4       *:*
root     ntpd       2823  25 udp4         *:*
root     sshd       2743  5  tcp4   *:22                  *:*
root     syslog-ng  2341  19 udp4   *:1031                *:*
nobody   mdnsd      2134  3  udp4   *:39020               *:*
nobody   mdnsd      2134  5  udp4   *:5353                *:*
root     python3.6  236   22 tcp4   *:6000                *:*

I even tried to get explanation why FreeNAS has such outdated and insecure packages in their latest version – FreeNAS 11.2-U3 Vulnerabilities – a thread I started on their forums.

Unfortunatelly its their policy which you can summarize as ‘do not touch/change versions if its working’ – at last I got this implression.

Because if these security holes I can not recommend the use of FreeNAS and I movedto original – the FreeBSD system.

One other interesting note. After I installed FreeBSD I wanted to import the ZFS pool created by FreeNAS. This is what I got after executing the zpool import command.

# zpool import
   pool: nas02_gr06
     id: 1275660523517109367
  state: ONLINE
 status: The pool was last accessed by another system.
 action: The pool can be imported using its name or numeric identifier and
        the '-f' flag.
   see: http://illumos.org/msg/ZFS-8000-EY

        nas02_gr06  ONLINE
          raidz2-0  ONLINE
            da58p2  ONLINE
            da59p2  ONLINE
            da60p2  ONLINE
            da61p2  ONLINE
            da62p2  ONLINE
            da63p2  ONLINE
            da64p2  ONLINE
            da26p2  ONLINE
            da65p2  ONLINE
            da23p2  ONLINE
            da29p2  ONLINE
            da66p2  ONLINE
            da67p2  ONLINE
            da68p2  ONLINE

   pool: nas02_gr05
     id: 5642709896812665361
  state: ONLINE
 status: The pool was last accessed by another system.
 action: The pool can be imported using its name or numeric identifier and
        the '-f' flag.
   see: http://illumos.org/msg/ZFS-8000-EY

        nas02_gr05  ONLINE
          raidz2-0  ONLINE
            da20p2  ONLINE
            da30p2  ONLINE
            da34p2  ONLINE
            da50p2  ONLINE
            da28p2  ONLINE
            da38p2  ONLINE
            da51p2  ONLINE
            da52p2  ONLINE
            da27p2  ONLINE
            da32p2  ONLINE
            da53p2  ONLINE
            da54p2  ONLINE
            da55p2  ONLINE
            da56p2  ONLINE

   pool: nas02_gr04
     id: 2460983830075205166
  state: ONLINE
 status: The pool was last accessed by another system.
 action: The pool can be imported using its name or numeric identifier and
        the '-f' flag.
   see: http://illumos.org/msg/ZFS-8000-EY

        nas02_gr04  ONLINE
          raidz2-0  ONLINE
            da44p2  ONLINE
            da37p2  ONLINE
            da18p2  ONLINE
            da36p2  ONLINE
            da45p2  ONLINE
            da19p2  ONLINE
            da22p2  ONLINE
            da33p2  ONLINE
            da35p2  ONLINE
            da21p2  ONLINE
            da31p2  ONLINE
            da47p2  ONLINE
            da48p2  ONLINE
            da49p2  ONLINE

   pool: nas02_gr03
     id: 4878868173820164207
  state: ONLINE
 status: The pool was last accessed by another system.
 action: The pool can be imported using its name or numeric identifier and
        the '-f' flag.
   see: http://illumos.org/msg/ZFS-8000-EY

        nas02_gr03  ONLINE
          raidz2-0  ONLINE
            da81p2  ONLINE
            da71p2  ONLINE
            da14p2  ONLINE
            da15p2  ONLINE
            da80p2  ONLINE
            da16p2  ONLINE
            da88p2  ONLINE
            da17p2  ONLINE
            da40p2  ONLINE
            da41p2  ONLINE
            da25p2  ONLINE
            da42p2  ONLINE
            da24p2  ONLINE
            da43p2  ONLINE

   pool: nas02_gr02
     id: 3299037437134217744
  state: ONLINE
 status: The pool was last accessed by another system.
 action: The pool can be imported using its name or numeric identifier and
        the '-f' flag.
   see: http://illumos.org/msg/ZFS-8000-EY

        nas02_gr02  ONLINE
          raidz2-0  ONLINE
            da84p2  ONLINE
            da76p2  ONLINE
            da85p2  ONLINE
            da8p2   ONLINE
            da9p2   ONLINE
            da78p2  ONLINE
            da73p2  ONLINE
            da74p2  ONLINE
            da70p2  ONLINE
            da77p2  ONLINE
            da11p2  ONLINE
            da13p2  ONLINE
            da79p2  ONLINE
            da89p2  ONLINE

   pool: nas02_gr01
     id: 1132383125952900182
  state: ONLINE
 status: The pool was last accessed by another system.
 action: The pool can be imported using its name or numeric identifier and
        the '-f' flag.
   see: http://illumos.org/msg/ZFS-8000-EY

        nas02_gr01  ONLINE
          raidz2-0  ONLINE
            da91p2  ONLINE
            da75p2  ONLINE
            da0p2   ONLINE
            da82p2  ONLINE
            da1p2   ONLINE
            da83p2  ONLINE
            da2p2   ONLINE
            da3p2   ONLINE
            da4p2   ONLINE
            da5p2   ONLINE
            da86p2  ONLINE
            da6p2   ONLINE
            da7p2   ONLINE
            da72p2  ONLINE

It seems that FreeNAS does ZFS little differently and they create a separate pool for every RAIDZ2 target with dedicated spares. Interesting …

UPDATE 1 – BSD Now 305

The FreeBSD Enterprise 1 PB Storage article was featured in the BSD Now 305 – Changing Face of Unix episode.

Thanks for mentioning!

UPDATE 2 – Real Life Pictures in Data Center

Some of you asked for a real life pictures of this monster. Below you will find several pics taken at the data center.

Front case with cabling.


Alternate front view.


Back of the case with cabling.


Top view with disks.


Alternate top view.


Disks slots zoom.


SSD and HDD disks.