Tag Archives: zfs

Quare FreeBSD?

I really wanted to make this article short … but I failed miserably. At least I tried to organize it well so one may get back to it after ‘some’ reading because its not a short lecture. I wanted to title it Why FreeBSD? but when you type that into your favorite duck.com search engine there are so many similar articles. I wanted it to have distinguished and unique name so I used Latin word for ‘why‘ which is ‘quare‘.

logo-freebsd

What FreeBSD can offer you that other operating systems does not? From all of the operating systems I used I find FreeBSD to suck the least. This post is not here to convince you to use or try FreeBSD – this you will have to do by yourself. This article will show you why FreeBSD is valuable or better alternative to other operating systems and is definitely not dying.

This is the Table of Contents for this article.

  • Base System
  • ZFS Boot Environments
  • Rescue
  • Audio
  • Jails
  • FreeBSD Ports Infrastructure
  • Updating/Building from Source
  • Storage
  • Init System
  • Linux Binary Compatibility
  • Simplicity
  • Evolution Instead Rewriting
  • Documentation
  • Community
  • Closing Thoughts

Base System

When you install a Linux system its just a bunch of RPM or DEB packages. For example of you install CentOS 7.8 Minimal variant you end up with several hundred RPM packages installed. After a week or month many of these packages will get updates sometimes making this CentOS system unusable or even unbootable (recent GRUB Boothole problem for example). On the contrary FreeBSD comes with a Base System concept. This means that when you install FreeBSD you install a minimal system as a whole. No packages or subsystems to be separately updated. Just whole Base System. That means that /boot /bin /sbin /usr /etc /lib /libexec /rescue directories are untouchable by any packages. When you decide to install packages (or build them using FreeBSD Ports) they will all fall into the /usr/local prefix. That means /usr/local/etc for configuration. The /usr/local/bin and /usr/local/sbin directories for binaries. The /usr/local/lib and /usr/local/libexec for libraries and so on. The FreeBSD Base System kernel modules are kept in the same dir along with the kernel in the /boot/kernel directory. To make things tidy all kernel modules that are provided by packages go into the /boot/modules dir. Everything has its place and its separated.

That is separation between Base System binaries (at /bin /sbin /usr/bin /usr/sbin dirs) and Third Party Packages maintained by pkg(8) and are located at /usr/local/bin and /usr/local/sbin dirs. We all know differences between bin (user) and sbin (root) binaries but in FreeBSD there is also another more UFS related separation. When there was only UFS filesystem in the FreeBSD world the /bin and /sbin binaries were available at boot after the root (/) filesystem was mounted and yet before /usrย  filesystem was mounted – this is historical (and still useful in the UFS setups) distinction dating to old UNIX days. In ZFS setups it does not matter as all files are on ZFS pool anyway.

The FreeBSD Base System separation also helps with another thing – if any package gets the ‘great’ idea to install new compiler named cc and override the default system compiler … or to add libraries/includes in such a way that makes it super hard to get back into a working system. If some random FreeBSD package would add libc.so to /usr/local/lib dir then you are covered and not prevented from running programs as usual because FreeBSD system binaries are linked to stuff in /usr/lib dir. This is why there is PATH variable on UNIX systems (and FreeBSD as well) to set which directories should be searched for binaries first. On FreeBSD by default its set search Base System binaries dirs first and then Third Party Packages later.

You can update (or not) the Base System separately from the installed packages with freebsd-update(8) command when using RELEASE or by recompiling with make buildworld and make installworld commands when using STABLE/CURRENT systems. When it comes to packages you can update them using the pkg(8) tool or portmaster when building from FreeBSD Ports tree under /usr/ports dir. That means that any packages updates will not touch your FreeBSD Base System at all. For example when you mess up (and I have done that in the beginning of my FreeBSD journey) the compiled ports and packages and you want to start over the only thing you have to do is remove /usr/local and /boot/modules and /var/db/pkg directories. That’s it. You are just reverted to your Base System and can start over. This is just not possible when using Linux system. Even with Gentoo that many concepts are based on FreeBSD ideas does not have Base System feature. This Base System also have additional feature. Because its separated from packages version no one stops you from running oldshool FreeBSD 9.0 from 2012 and install there latest Firefox 80 or LibreOffice 7.0. You can not install latest Firefox on Ubuntu from 2012 …

One may be ‘afraid’ that such Base System independent from installed packages would take more space but nothing far more from the truth. The fresh installed FreeBSD 12.1 system uses less then 1 GB of disk space and takes less then 75 MB of RAM with sshd(8) running. For the comparison fresh CentOS 7.8 install with ‘Minimal’ set chosen takes 1.1 GB of disk space and uses more then 100 MB RAM with sshd(8) running. Such CentOS system is really naked and really needs more packages to be usable while FreeBSD with its Base System is far more capable and powerful and comes along with builtin latest version of LLVM/CLANG compiler suite for example.

More on the Base System topic:

ZFS Boot Environments

I have talked about this many times and probably one time too less because Linux world still ignores this bless. Having ZFS Boot Environments its such a game changer that once you realize how powerful it is you will never want to use a system that does not support it. The idea is that you can snapshot a running system at any moment of time and then reboot into that moment (or snapshot) if something happened. Its perfect solution for upgrade or changes to the system. The FreeBSD systems are already well ‘protected’ from problems arising after updating the packages but ZFS Boot Environments takes this to a whole new level.

groundhog

Like in the movie Groundhog Day (1993) with ZFS Boot Environments you will have limitless chances to get your shit toghether. Even the Base System updates and changes are protected by it. You can even transport that Boot Environment by using zfs send and zfs recv commands to other system … or propagate it on many systems. You can create Jails containers from it … or install new version of FreeBSD in the new Boot Environment and reboot into it while still having your older ‘production’ system untouched.

More on the ZFS Boot Environments topic:

Rescue

When you really mess up to the point that even Base System concept or ZFS Boot Environments feature did not stopped you from killing your FreeBSD installation then there is one more level of rescue … the Rescue subsystem.

rescue

You have about 150 statically linked binaries available at your disposal for the rescue mission of that FreeBSD installation. You probably think now that if its so many binaries then it probably takes a lot of space … nothing far more from the truth. Its actually one static binary with hardlinks … and it takes whooping 11 MB of disk space.

# ls -lh /rescue | head -5
total 1118446
-r-xr-xr-x  146 root  wheel    11M 2020.02.19 21:10 [
-r-xr-xr-x  146 root  wheel    11M 2020.02.19 21:10 bectl
-r-xr-xr-x  146 root  wheel    11M 2020.02.19 21:10 bsdlabel
-r-xr-xr-x  146 root  wheel    11M 2020.02.19 21:10 bunzip2

They Rescue subsystem even contains such binaries as bectl(8) for ZFS Boot Environments management or zfs(8) and zpool(8) commands for the ZFS filesystem. Here is complete list of these binaries.

# ls /rescue
[           dd               fsck_ffs      init       mdmfs          ping      rtsol        unlink
bectl       devfs            fsck_msdosfs  ipf        mkdir          ping6     savecore     unlzma
bsdlabel    df               fsck_ufs      iscsictl   mknod          pkill     sed          unxz
bunzip2     dhclient         fsdb          iscsid     more           poweroff  setfacl      unzstd
bzcat       dhclient-script  fsirand       kenv       mount          ps        sh           vi
bzip2       disklabel        gbde          kill       mount_cd9660   pwd       shutdown     whoami
camcontrol  dmesg            geom          kldconfig  mount_msdosfs  rcorder   sleep        xz
cat         dump             getfacl       kldload    mount_nfs      rdump     spppcontrol  xzcat
ccdconfig   dumpfs           glabel        kldstat    mount_nullfs   realpath  stty         zcat
chflags     dumpon           gpart         kldunload  mount_udf      reboot    swapon       zdb
chgrp       echo             groups        ldconfig   mount_unionfs  red       sync         zfs
chio        ed               gunzip        less       mt             rescue    sysctl       zpool
chmod       ex               gzcat         link       mv             restore   tail         zstd
chown       expr             gzip          ln         nc             rm        tar          zstdcat
chroot      fastboot         halt          ls         newfs          rmdir     tcsh         zstdmt
clri        fasthalt         head          lzcat      newfs_msdos    route     tee          
cp          fdisk            hostname      lzma       nextboot       routed    test         
csh         fsck             id            md5        nos-tun        rrestore  tunefs       
date        fsck_4.2bsd      ifconfig      mdconfig   pgrep          rtquery   umount   

More on the Rescue topic:

Audio

Not many people expect from FreeBSD to shine in that department but it shines a lot here and not from yesterday but from decades. Remember when Linux got rid of the old OSS subsystem with one channel and came up with ‘great’ idea to write ALSA? I remember because I used Linux back then. Disaster is very polite word to describe Linux audio stack back then … and then PulseAudio came and whole Linux audio system got much worse. Back then because of that one OSS channel and many ALSA channels meant that ONLY ONE application with OSS backend could do the sound (for example WINE). But if another application would want to ‘make’ sound using OSS and you already have WINE started then it will be soundless because that one and only OSS channel was already taken. And remember that ALSA was so bad back then that KDE or GNOME made their own sound daemons mixing audio in userspace that were incompatible with each other. That means if you used KDE and GNOME apps back then you could have sound from GNOME apps but not from KDE apps or vice versa. One big fucking audio hell on Linux.

audio

Lets get back to FreeBSD audio then. What FreeBSD offered? A whooping 256 OSS channels mixed live in kernel for low latency. Everything audio related just worked out of the box – and still works today. You could have WINE or KDE/GNOME sound backends attached to their OSS channels and also ALSA apps getting their sound device without a problem. Even when you plugged a 5.1 surround system into FreeBSD it worked out of the box without any configuration and applications were able to use it immediately. That FreeBSD audio supremacy remains today as PulseAudio sound mixing in userspace while generally working incorporates large latency on Liunx compared to in kernel FreeBSD mixing with low latency.

Comrade meka suggested that FreeBSD is also the only OS which has virtual_oss that allows mixing/resampling/compressing in user space and allows one to have Bluetooth headphones and USB microphone represented as single sound card.

More on the Audio topic:

Jails

The FreeBSD Jails are one of the oldest OS Level Virtualization implementations dating back to 1999. Even the Solaris Zones/Containers came five years later in 2004.

containers

After Docker was introduced in Linux the term OS Level Virtualization became less used to the Containers term and now the FreeBSD Jails along with Solaris Zones/Containers are named 1st generation containers. But that naming nomenclature change does not make FreeBSD Jails less powerful. They are also really brain dead simple to use. You just need a directory – for example /jail/nextcloud – where you will extract the FreeBSD Base System for desired release version – for example base.txz from 12.1-RELEASE and create the Jail config in the /etc/jail.conf file as shown below.

# mkdir -p /jail/nextcloud
# fetch -o - http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/12.1-RELEASE/base.txz | tar --unlink -xpJf - -C /jail/nextcloud
# cat /etc/jail.conf
nextcloud {
  host.hostname = nextcloud.local;
  ip4.addr = 10.0.0.100;
  path = /jail/nextcloud;
}

Now you can start you Jail right away.

# service jail onestart nextcloud
Starting jails: nextcloud.

Voila! Your FreeBSD Jail is already running.

# jls
   JID  IP Address      Hostname                      Path
     1  10.0.0.100      nextcloud.local               /jail/nextcloud

You can of course have a trimmed down version of FreeBSD Base System in the Jail if that is needed. The ZFS filesystem also helps here greatly because with zfs clone only your ‘base’ Jail will take space and only the changes you make to Jails created from it. Thanks to other FreeBSD subsystem – the Linux Binary Compatibility – you can also create a Linux Jail – for example running Devuan Jail.

The FreeBSD Jails are also very lightweight. You can boot and use about 1000 FreeBSD Jails on a single FreeBSD system with 4 GB RAM.

They are also very easy to debug and troubleshoot comparing even to plain Docker – not to even mention Kubernetes which requires whole team of highly skilled people to maintain.

The FreeBSD Jails may be configured/managed only by the Base System utilities such as jls(8)/jexec(8) but you can also select from many third party Jail management frameworks. From all available ones I would choose BastilleBSD because of their modern approach and many ready to use templates for all needed use cases.

More on the Jails topic:

FreeBSD Ports Infrastructure

This is one of another examples why FreeBSD rocks that much. When you install Ubuntu or CentOS in some version there is chance that you will end up with not latest versions of packages but with versions that were quite up-to-date when this distribution version was released. Its especially visible in the CentOS world (and its upstream enterprise source system from Red Hat) where packages are quite up-to-date when .0 (dot zero) release is published but are VERY outdated when .8 or .9 incarnation of that release is available. Not to even mention that Firefox for example is released every month …

packages

As I said before when describing the FreeBSD Base System the FreeBSD Ports (and packages built from it available through pkg(8)) are independent. That means that third party software from FreeBSD Ports is almost always up-to-date (or very close to it). You can even check it on the repology.org site for the details. Below you will find a ‘snapshot’ of the repology.org stats from time of writing this article. The ‘online’ table is very long so I copy/pasted just the systems relevant to the article.

repology

One of the other advantages of FreeBSD Ports is that it offers really MASSIVE amount of software counting 40354 ports when writing this article and still rising. Amount of ready to be installed packages are little smaller with more then 32000 available.

I once migrated for a while to OpenSolaris in 2009 on my Dell Latitude D630 laptop because I really liked all the Solaris features (including ZFS and ZFS Boot Environments that were not available on FreeBSD back then) and the OpenSolaris GNOME based desktop was pretty nice back then even with Time Slider feature for ZFS snapshots in the Nautilus file manager. I got working WiFi connection, sound was working, generally everything on my laptop was supported and working with OpenSolaris … but there was no software. Of course ‘large’ projects like GIMP or OpenOffice was available even in the default pkg(8) repository but not much else. There was less then 4000 packages back then on OpenSolaris while about 25000 packages on FreeBSD if I recall correctly.

You can also easily browse available FreeBSD Ports (and its options) on the web by using the https://freshports.org/ page.

ports

The count of FreeBSD Ports is one thing, the features is another. No matter which Linux distribution you are using you will find a software that was compiled and shipped without that needed flag that you desperately need. If you find such software on FreeBSD it ‘hurts’ only for a moment because you can VERY EASILY recompile that software with needed options and replace that ‘default’ package with yours. For example the FreeBSD project is afraid to provide packages of Lame because of existing MP3 patents, so multimedia/ffmpeg package is built without MP3 support (with --disable-libmp3lame flag). That is why I have my own audio/lame and multimedia/ffmpeg packages built with my configure options and that is very easy to achieve. You need to go to the /usr/ports/multimedia/ffmpeg dir type make config and select [x] LAME at the ncurses dialog. Your chosen options will be saved as plain /var/db/ports/multimedia_ffmpeg/options file. If you remove that file (or type make rmconfig) then these custom options will reset to defaults. Then you type make build deinstall install clean and your port with new options is ready and installed as package. Nothing more is needed. You can even lock that package from the pkg(8) upgrades with pkg lock -y ffmpeg command so it will not be modified later but its better to rebuild such packages everytime you do a pkg upgrade procedure because of libraries versions bump and changes. While its very easy and fast to create a script with these commands to make it more automated you can also use other parts of the FreeBSD Ports infrastructure – enter Poudriere (or Synth) – more on that in the next part.

You also do not have to configure each port that way (which could be PITA for large amount of ports) but you may specify your needed (OPTIONS_SET) or unwanted (OPTIONS_UNSET) parameters only once globally using the /etc/make.conf file. You can also specify which default versions of software you want to use, for example Apache 2.2 instead of 2.4 and PHP 7.0 instead of 7.2. You can find all default versions in the /usr/ports/Mk/bsd.default-versions.mk file. Once you setup these options you can build/rebuild or update your packages from FreeBSD Ports by portmaster(8) tool. Like on Gentoo Linux with USE flags. But this is the original. Gentoo took all/most of its ideas from FreeBSD system and its Ports infrastructure.

The Poudriere is a build framework that uses FreeBSD Ports and FreeBSD Jails to build requested packages in clean reproducible way. You can create whole new binary package repository for pkg(8) command to use with it. I mentioned Synth because while Poudriere is often used to produce whole package repository the Synth is usually used just to rebuild several packages that does not fit your needs.

There is one important things about FreeBSD Ports that is often misunderstood by newcomers. What is the difference between the Ports and packages that are fetched and installed by pkg(8) tool? Its quite simple. A package is just a build and installed port. Nothing more or less. When you use the binary packages using pkg(8) command you are using packages that someone (the FreeBSD project in that case) built for you from the FreeBSD Ports in some point in time. While FreeBSD strives to maintain as up-to-date built packages as possible its the nature of FreeBSD Ports that they are always more up-to-date then the built packages. That is why you may build and install a new version of needed packages by yourself using FreeBSD Ports. One may think of such usage when it comes to security holes. When some locally executed commands (like file(1) for example) has a security hole then its not critical for you to update it as fast as possible because that security hole can be harmless for you, but when new version of Firefox fixes very important security hole then its better to update from FreeBSD Ports version faster because waiting 2 days for the package to be built (along with other packages) can be too long.

More on the FreeBSD Ports topic:

Updating/Building from Source

While the FreeBSD Ports infrastructure is for third party software the FreeBSD Base System (or its parts) also can be easily and convenient build from source. The FreeBSD kernel config is also very small and simple. While Linux kernel config contains thousands of options – 4432 for example in the default CentOS 8.2 install the FreeBSD GENERIC config has about 20 times options less – only 260 options. But that does not saturate the topic. You can start with MINIMAL FreeBSD kernel config which has only 75 options specified.

Linux # grep -c '^CONFIG' /boot/config-$( uname -r )
4432

FreeBSD # grep -c -E '^(device|options)' /usr/src/sys/amd64/conf/GENERIC
260

FreeBSD # grep -c -E '^(device|options)' /usr/src/sys/amd64/conf/MINIMAL
75

… and its not only about smaller amount of options. Can you tell my how many steps (and which ones are required) to rebuild CentOS or Ubuntu for example without Bluetooth support?

code

On the contrary its very simple (and fast) on the FreeBSD side. While /etc/make.conf file is used to enable/disable Ports options the /etc/src.conf file is used to enable/disable FreeBSD Base System options while building it from source. To build FreeBSD without Bluetooth support just add WITHOUT_BLUETOOTH=yes to the /etc/src.conf file and type these to build it:

# beadm create safe
# cd /usr/src
# make buildworld kernel
# reboot
# cd /usr/src
# make installworld
# mergemaster -iU
# reboot

Voila! You now have FreeBSD without Bluetooth support … and if any of the steps failed or because of your lack of experience/expertise your FreeBSD system does not boot or is broken you can use tools from /rescue to try to fix it (or at least figure out what is broken) and when you do not want to cope with this jest select safe ZFS Boot Environment at the FreeBSD loader(8) to boot to the system before you started building modified version of FreeBSD. Yes, You are bulletproof here. While having 294 WITHOUT_X options and 125 WITH_X options you can really tune FreeBSD Base System to your needs.

# zgrep -c WITHOUT_ /usr/share/man/man5/src.conf.5.gz
294

# zgrep -c WITH_ /usr/share/man/man5/src.conf.5.gz
125

The big downside of updating FreeBSD by source is that you can not use the freebsd-update tools to do it … but nothing stops you from creating your own FreeBSD Update Server so you will be able to use freebsd-update by adding updates using a CURRENT or STABLE system instead of RELEASE. That process is described in the Build Your Own FreeBSD Update Server article of official FreeBSD documentation.

More on the FreeBSD Source Updates/Builds topic:

Storage

Storage is one of the parts where FreeBSD really shines. Lots of people adore FreeBSD for well integrated ZFS filesystem and its really true. ZFS in FreeBSD has always been first class citizen. Lately OpenZFS 2.0 has been also integrated from the upstream joint FreeBSD and Linux repository. More and more FreeBSD features and solutions are using ZFS features.

openzfs

Most of these people that like integrated ZFS in FreeBSD do not know about the FreeBSD GEOM modular disk transformation framework which provides various storage related features and utilities like software RAID0/RAID1/RAID10/RAID3/RAID5 configurations or transparent encryption of underlying devices with GELI/GDBE (like LUKS on Linux). It also allows transparent filesystem journaling for ANY filesystem with GJOURNAL (yes also for FAT32 or exFAT) or allows one to export block devices over network with GEOM GATE devices (like NFS for block devices).

storage

FreeBSD also has its own FUSE implementation which allows all these FUSE based filesystems to work natively on FreeBSD. While lots of Linux folks know DRBD probably very few of them knew that FreeBSD comes with its own DRBD like solution called HAST – which does exactly the same thing. While ZFS has a lot features and possibilities FreeBSD still maintains and develops fast and small memory footprint UFS filesystem which today is used either with Soft Updates (SU) or Journaled Soft Updates (SUJ) depending on the use case. For example 10 TB data on UFS filesystem with Journaled Soft Updates (SUJ) takes about 1 minute under fsck(8). These storage solutions are available from FreeBSD Base System alone. The FreeBSD Ports offers much more with distributed filesystems solutions such as CEPH, LeoFS, LizardFS or Minio for Amazon S3 compatible storage.

More on the Storage topic:

Init System

FreeBSD offers really simple yet very powerful init system. It has system wide config under /etc/rc.conf file when you can enable/disable needed services with service_enable=YES and service_enable=NO stanzas. You do not even need to launch vi(1) to add them – just type sysrc service_enable=YES and they are added to the /etc/rc.conf file. There are also default values and services that are enabled and you will find them – along with many comments – in the /etc/defaults/rc.conf file. Each FreeBSD service file has PROVIDE/REQUIRE stanzas which are then used to automatically order the services to start. Services that can be run in parallel are started in parallel to save time. For example its pointless to start sshd(8) daemon without network. To start or stop the serivice you need to type service sshd start or service sshd stop command. But when a service is not enabled in the /etc/rc.conf file then you need to used add onestart and onestop instead. The Base System separation remains here as FreeBSD Base System services are located at /etc/rc.d directory and third party applications from ports/packages are kept under /usr/local prefix which means /usr/local/etc/rc.d dir.

When using systemd(1) you never know how the services gonna start because it will be different each time. Zero determinism. On FreeBSD you know exactly which services will start when because they are always ordered in the same state according to the PROVIDE/REQUIRE stanzas. FreeBSD also offers tools that will tell you the exact order – rcorder(8) – which can be used for all services, Base System services or third party services separately. There is also service -r command that will show you what was the orfer at the boot time.

# rcorder /etc/rc.d/* | head
/etc/rc.d/growfs
/etc/rc.d/sysctl
/etc/rc.d/hostid
/etc/rc.d/zvol
/etc/rc.d/dumpon
/etc/rc.d/ddb
/etc/rc.d/geli
/etc/rc.d/gbde
/etc/rc.d/ccd
/etc/rc.d/swap

# rcorder /usr/local/etc/rc.d/* | tail
/usr/local/etc/rc.d/hald
/usr/local/etc/rc.d/git_daemon
/usr/local/etc/rc.d/fscd
/usr/local/etc/rc.d/cupsd
/usr/local/etc/rc.d/cups_browsed
/usr/local/etc/rc.d/clamav-clamd
/usr/local/etc/rc.d/clamav-milter
/usr/local/etc/rc.d/clamav-freshclam
/usr/local/etc/rc.d/avahi-dnsconfd
/usr/local/etc/rc.d/aria2

# rcorder /etc/rc.d/* /usr/local/etc/rc.d/* 2> | grep -C 3 sshd
/etc/rc.d/ubthidhci
/etc/rc.d/syscons
/etc/rc.d/swaplate
/etc/rc.d/sshd
/etc/rc.d/cron
/etc/rc.d/jail
/etc/rc.d/localpkg

Adding new service to FreeBSD is also very easy as template for new service is very small and simple.

#!/bin/sh

. /etc/rc.subr

name=dummy
rcvar=dummy_enable

start_cmd="${name}_start"
stop_cmd=":"

load_rc_config $name
: ${dummy_enable:=no}
: ${dummy_msg="Nothing started."}

dummy_start()
{
	echo "$dummy_msg"
}

run_rc_command "$1"

If its not simple enought for you there is dedicated FreeBSD article about writing them – Practical rc.d Scripting in BSD – available here.

More on the Init System topic:

Linux Binary Compatibility

While Linux can not be FreeBSD – the FreeBSD can be Linux – and its not some slow emulation – its implementation of Linux system calls. There was time when enterprises used to work with Linux only applications (not available on FreeBSD by then) using the Linux Binary Compatibility on FreeBSD because it was faster then running them natively on Linux – FreeBSD Used to Generate Spectacular Special Effects – an official FreeBSD Press Release about FreeBSD being used to generate spacial effects to the one of the best movies of all time – The Matrix (1999).

matrix

Today the LINUX_COMPAT is also natively fast and allows one to run Linux applications – even Linux games in X11 with hardware acceleration for graphics. Think of it as WINE but for Linux applications. It lives under /compat/linux directory. It even implements Linux /proc virtual filesystem which can be mounted at the /compat/linux/proc dir but its not mandatory. For any software that does not come with source code and works on Linux the Linux Binary Compatibility saves the day. For example the f.lux project. Before I got to know Redshift I used f.lux Linux binary using LINUX_COMPAT to suppress blue spectrum light from my FreeBSD screen. The Linux Binary Compatibility subsystem can also be used to run Linux bases FreeBSD Jails – with Devuan for example.

More on the Linux Binary Compatibility topic:

Simplicity

FreeBSD is simple but not coarse/ornery. For example as Linux the FreeBSD system also supports the /proc virtual filesystem but on FreeBSD its optional and not used by default while Linux could not live without it. But while Linux has mandatory /proc it also has another virtual filesystem residing under /sys … but why Linux people need two different virtual filesystems with similar purposes? Why they could not create everything under /proc as it already existed. That is big enigma for my sanity.

But /sys is not the end of that madness. Its just a beginning.

What about these?

  • securityfs
  • devpts
  • cgroup
  • pstore
  • bpf
  • configfs
  • selinuxfs
  • systemd-1
  • mqueue
  • debugfs
  • hugetlbfs

Take a look at the FreeBSD mount(8) output after the default install on ZFS.

FreeBSD # mount
zroot/ROOT/12.1 on / (zfs, local, noatime, nfsv4acls)
devfs on /dev (devfs, local, multilabel)
zroot/tmp on /tmp (zfs, local, noatime, nosuid, nfsv4acls)
zroot/var/mail on /var/mail (zfs, local, nfsv4acls)
zroot/usr/home on /usr/home (zfs, local, noatime, nfsv4acls)
zroot/var/crash on /var/crash (zfs, local, noatime, noexec, nosuid, nfsv4acls)
zroot/var/log on /var/log (zfs, local, noatime, noexec, nosuid, nfsv4acls)
zroot/var/audit on /var/audit (zfs, local, noatime, noexec, nosuid, nfsv4acls)
zroot/var/tmp on /var/tmp (zfs, local, noatime, nosuid, nfsv4acls)
zroot/usr/src on /usr/src (zfs, local, noatime, nfsv4acls)
zroot/usr/ports on /usr/ports (zfs, local, noatime, nosuid, nfsv4acls)

Several ZFS datasets and one virtual devfs filesystem for /dev directory. With install on UFS it would be similar with several UFS partitions mounted instead of ZFS datasets.

Take a look at the CentOS 8.2 installation with just one physical root (/) XFS filesystem.

[root@centos8 ~]# mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,seclabel,size=919388k,nr_inodes=229847,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,seclabel)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,seclabel,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime,seclabel)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,cpuset)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,memory)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,blkio)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,hugetlb)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,net_cls,net_prio)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,cpu,cpuacct)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,freezer)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,perf_event)
cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,rdma)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,pids)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,devices)
configfs on /sys/kernel/config type configfs (rw,relatime)
/dev/sda1 on / type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
selinuxfs on /sys/fs/selinux type selinuxfs (rw,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17309)
mqueue on /dev/mqueue type mqueue (rw,relatime,seclabel)
debugfs on /sys/kernel/debug type debugfs (rw,relatime,seclabel)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,seclabel,pagesize=2M)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=187088k,mode=700)

Fuck me. Its even really hard to just find any REAL filesystem there … fortunately we can ask for only XFS filesystems to display.

[root@centos8 ~]# mount -t xfs
/dev/sda1 on / type xfs (rw,relatime,seclabel,attr2,inode64,noquota)

Lets get on the networking now. Lets assume that you want to make standard enterprise networking setup on a physical server with two interfaces aggregated together into highly available interface bond0 (lagg0 on FreeBSD) and then you want to put VLAN tag and IP address on that VLAN. The CentOS 7.x/8.x installer (Anaconda) will welcome you with this mess.

[root@centos7 ~]# ls -1 /etc/sysconfig/network-scripts/ifcfg-*
ifcfg-Bond_connection_1
ifcfg-eno49
ifcfg-eno49-1
ifcfg-eno50
ifcfg-eno50-1
ifcfg-VLAN_connection_1

[root@centos7 ~]# cat /etc/sysconfig/network-scripts/ifcfg-Bond_connection_1
DEVICE=bond0
BONDING_OPTS="miimon=1 updelay=0 downdelay=0 mode=active-backup"
TYPE=Bond
BONDING_MASTER=yes
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=dhcp
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_PRIVACY=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME="Bond connection 1"
UUID=ca85417f-8852-43bf-96ee-5bd3f0f83648
ONBOOT=yes

[root@centos7 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eno49
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=dhcp
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=eno49
UUID=2f60f50b-38ad-492a-b90a-ba736acf6792
DEVICE=eno49
ONBOOT=no

[root@centos7 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eno49-1
HWADDR=xx:xx:xx:xx:xx:xx
TYPE=Ethernet
NAME=eno49
UUID=342b8494-126d-4f3a-b749-694c8c922aa1
DEVICE=eno49
ONBOOT=yes
MASTER=bond0
SLAVE=yes

[root@centos7 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eno50
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=dhcp
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=eno50
UUID=4fd36e24-1c6d-4a65-a316-7a14e9a92965
DEVICE=eno50
ONBOOT=no

[root@centos7 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eno50-1
HWADDR=xx:xx:xx:xx:xx:xx
TYPE=Ethernet
NAME=eno50
UUID=a429b697-73c2-404d-9379-472cb3c35e06
DEVICE=eno50
ONBOOT=yes
MASTER=bond0
SLAVE=yes

[root@centos7 ~]# cat/etc/sysconfig/network-scripts/ifcfg-VLAN_connection_1
VLAN=yes
TYPE=Vlan
PHYSDEV=ca85417f-8852-43bf-96ee-5bd3f0f83648
VLAN_ID=601
REORDER_HDR=yes
GVRP=no
MVRP=no
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=none
IPADDR=10.20.30.40
PREFIX=24
GATEWAY=10.20.30.1
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_PRIVACY=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME="VLAN connection 1"
UUID=90f7a9bb-1443-4adf-a3eb-86a03b23ecfb
ONBOOT=yes

For the record – I have choosen ‘STATIC’ IPv4 address but installer made these interfaces to use DHCP and that STATIC address. That could be a bug but lets get to the point.

After manual fixing with vi(1) (and hour later) this is how it supposed to look.

[root@centos7 ~]# cat /etc/sysconfig/network
GATEWAY=10.20.30.1
NOZEROCONF=yes

[root@centos7 ~]# ls -1 /etc/sysconfig/network-scripts/ifcfg-*
ifcfg-bond0
ifcfg-bond0.601
ifcfg-eno49
ifcfg-eno50

[root@centos7 ~]# cat /etc/sysconfig/network-scripts/ifcfg-bond0
DEVICE=bond0
BONDING_OPTS="miimon=1 updelay=0 downdelay=0 mode=active-backup"
TYPE=Bond
BONDING_MASTER=yes
BOOTPROTO=none
IPV4_FAILURE_FATAL=no
IPV6INIT=no
ONBOOT=yes

[root@centos7 ~]# cat /etc/sysconfig/network-scripts/ifcfg-bond0.601
VLAN=yes
TYPE=Vlan
VLAN_ID=601
DEVICE=bond0.601
REORDER_HDR=yes
GVRP=no
MVRP=no
BOOTPROTO=none
IPADDR=10.20.30.40
PREFIX=24
IPV4_FAILURE_FATAL=no
IPV6INIT=no
ONBOOT=yes

[root@centos7 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eno49
BOOTPROTO=none
IPV4_FAILURE_FATAL=no
IPV6INIT=no
TYPE=Ethernet
NAME=eno49
DEVICE=eno49
ONBOOT=yes
MASTER=bond0
SLAVE=yes

[root@centos7 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eno50
BOOTPROTO=none
IPV4_FAILURE_FATAL=no
IPV6INIT=no
TYPE=Ethernet
NAME=eno50
DEVICE=eno50
ONBOOT=yes
MASTER=bond0
SLAVE=yes

Better … but still takes A LOT OF SPACE and several files to cover that quite simple setup. Not to mention its level of complication and making that very error prone way. The same configuration on FreeBSD would take just 7 lines within single /etc/rc.conf file as shown below.

ifconfig_fxp0="up"
ifconfig_fxp1="up"
cloned_interfaces="lagg0"
ifconfig_lagg0="laggproto failover laggport fxp0 laggport fxp1"
vlans_lagg0="601"
ifconfig_lagg0_601="inet 10.20.30.40/24"
defaultrouter="10.20.30.1"

What about the boot process? FreeBSD boots from root on ZFS partition with just small 512 KB not mountable partition. No separate /boot device is needed. On the other side Linux always needs that separate /boot partition filled with GRUB modules. No matter if its ZFS or LVM. That is why implementation of ZFS Boot Environments is quite complicated on Linux bacause even if you have root on ZFS on a Linux system there is still unprotected /boot filesystem that can not be snapshoted with ZFS and has to be protected in old classic way which kill the idea of ZFS Boot Environments or Linux.

FreeBSD is really simple and well thought operating system. But also a very underestimated one.

Evolution Instead Rewriting

How many Linux tools or subsystems are abandoned or superseeded by new ones? Why the ifconfig(8) command was not updated with new options and instead a new ip(8) command was introduced? Same with netstat(8) being replaced by ss(8). Same with arp(8)/iwconfig/route(8) and many more. What about whole init system? The Linux world has been taken over by systemd(1) whenever you like it or not. Even distributions that have grown their mature init systems like Ubuntu with its Upstart has moved to systemd(1) altogether. The distributions that do not use it are very few and considered a niche today.

evolution

In the FreeBSD land on the countary such things happen only if there is no other way to implement new things. Its the last thing wanted in the FreeBSD. FreeBSD evolves and is developed with stability and backward compatibility in mind. Userland tools are grown and updated with new options instead of rewriting them over and over again. Not to mention how many new bugs are introduced by changing one tool to another.

More on the Evolution Instead Rewriting topic:

Documentation

Having system that can do almost anything but not knowing how to do that makes that system pretty useless (or at least pretty PITA to use). FreeBSD offers second to none documentation that is actively maintained and updated. Along with its legendary FreeBSD Handbook and FreeBSD FAQ the FreeBSD project also offers official FreeBSD Articles about various FreeBSD topics. The Man Pages are also very detailed and contain many examples. There is also FreeBSD Wiki page for work in progress documentation and ideas related to FreeBSD development and if you have any problems or questions related to FreeBSD there are official FreeBSD Forums and oldschool Mailing Lists available.

documentation

These were only the official project knowledge sources but there are also lots of FreeBSD books. Here are the best and up-to-date ones.

  • Absolute FreeBSD – Complete Guide to FreeBSD – 3nd Edition (2019)
  • Beginning Modern Unix (2018)
  • Book of PF – 3rd Edition (2015)
  • Design and Implementation of FreeBSD 11 Operating System – 2nd Edition (2015)
  • FreeBSD Device Drivers (2012)
  • FreeBSD Mastery – ZFS (2015)
  • FreeBSD Mastery – Advanced ZFS (2016)
  • FreeBSD Mastery – Storage Essentials (2014)
  • FreeBSD Mastery – Specialty Filesystems (2015)
  • FreeBSD Mastery – Jails (2019)

There are also two magazines that are dedicated to BSD and FreeBSD systems. Both are free and cover lots of interesting topics regarding FreeBSD.

With all this knowledge and support its really hard not to achieve what you need/want with FreeBSD system.

Community

Last but not least and I would say its even more important then good documentation (which FreeBSD has awesome). People that use FreeBSD do that conciously and are often experienced not only in FreeBSD land but also in topics related to other UNIX systems. Often they took long road of first using the Linux systems before finally setting on the FreeBSD land or they still do Linux adminitration for a living while resting using far more reasonable and sensible FreeBSD solution. I always find FreeBSD Community helpful and friendly. Always willingly helpful – especially towards newcommers. Even when you try to ‘force’ FreeBSD people to ‘fight’ in unjust/doubtful discussion they will reply with dignity and technical arguments instead of yelling at you.

The FreeBSD project even made several articles and Handbook chapters especially for Linux newcommers (or sometimes called systemd(1) refugees).

Closing Thoughts

I tried really hard to not make it a Linux rant but some may feel it that way – if so please remember that this was not my intention. FreeBSD like Linux and like any other operating system has its ups and downs. Hope that I showed you most interesting FreeBSD parts. I may add new sections here without a warning in the future ๐Ÿ™‚

EOF

ย 

FreeBSD Enterprise Storage at PBUG

Yesterday I was honored to give a talk about FreeBSD Enterprise Storage at the Polish BSD User Group meeting.

You are invited to download the PDF Slides โ€“ https://is.gd/bsdstg โ€“ available here.

bsdstg

The PBUG (Polish BSD User Group) meetings are very special. In “The Matrix” movie (which has been rendered on FreeBSD system by the way) – FreeBSD Used to Generate Spectacular Special Effects – details available here – its not possible to describe what the Matrix really is, one has to feel it. Enter it. The same I can tell you about the PBUG meetings. Its kinda like with the “Hangover” movie. What happens in Vegas PBUG meeting stays in Vegas PBUG meeting ๐Ÿ™‚

If you will have the possibility and time then join the next Polish BSD User Group meeting. You will not regret it :>

UPDATE 1 – Shorter Unified Version

The original – https://is.gd/bsdstg – presentation is 187 pages long and is suited for live presentation while not the best for later ‘offline’ view.

I have created a unified version – https://is.gd/bsdstguni – with only 42 pages.

EOF

Nextcloud 17 on FreeBSD 12.1

Not so long ago – almost 2 years from now – I wrote about setting up Nextcloud 13 on FreeBSD.

Today Nextcloud is at 17 version and the configuration that worked two years ago requires some tweaks.

nextcloud-logo.png

This guide will not cover the same information that is available in earlier Nextcloud 13 on FreeBSD article like settings to run Nextcloud inside FreeBSD Jail. Please refer to that earpier article for these settings.

Today we will use these as backends for Nextcloud 17.

  • PostgreSQL 12
  • PHP 7.3
  • Nginx 1.14 (with php-fpm)
  • Memcached 1.5.19

As Nextcloud in FreeBSD packages comes with MySQL and without PostgreSQL support we will need to build it from source using FreeBSD Ports.

Settings

Let’s fetch the latest FreeBSD Ports tree.

# rm -r /var/db/portsnap
# mkdir /var/db/portsnap
# portsnap auto

Now we need to configure needed options in the /etc/make.conf file.

# cat /etc/make.conf
WRKDIRPREFIX=${PORTSDIR}/obj
DEFAULT_VERSIONS+= php=7.3
DEFAULT_VERSIONS+= pgsql=12
OPTIONS_UNSET+=    MYSQL
OPTIONS_SET+=      PGSQL
OPTIONS_SET+=      IMAGICK
OPTIONS_SET+=      PCNTL
OPTIONS_SET+=      SMB
OPTIONS_SET+=      REDIS


Packages and Ports

First we will add some basic tools and things like PostgreSQL still using FreeBSD packages to save tome time instead of compiling them.

# pkg install \
    sudo \
    portmaster \
    beadm \
    lsblk \
    postgresql12-client \
    postgresql12-server \
    nginx \
    memcached \
    php73-pecl-memcached


Now we will compile Nextcloud and its dependencies using FreeBSD Ports – but with portmaster.

# env BATCH=yes portmaster \
    databases/php73-pdo_pgsql \
    databases/php73-pgsql \
    www/nextcloud 

PostgreSQL

We will now configure the FreeBSD’s Login Class for PostgreSQL database in the /etc/login.conf file.

# cat  /etc/login.conf

postgres:\
        :lang=en_US.UTF-8:\
        :setenv=LC_COLLATE=C:\
        :tc=default:

EOF

# cap_mkdb /etc/login.conf

… and PostgreSQL settings in main FreeBSD’s configuration /etc/rc.conf file.

# grep postgresql /etc/rc.conf
postgresql_enable=YES
postgresql_class=postgres
postgresql_data=/var/db/postgres/data12

Let’s initialize the PostgreSQL database.

# /usr/local/etc/rc.d/postgresql initdb
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locales
  COLLATE:  C
  CTYPE:    en_US.UTF-8
  MESSAGES: en_US.UTF-8
  MONETARY: en_US.UTF-8
  NUMERIC:  en_US.UTF-8
  TIME:     en_US.UTF-8
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/db/postgres/data12 ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Europe/Warsaw
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.

Success. You can now start the database server using:

    /usr/local/bin/pg_ctl -D /var/db/postgres/data12 -l logfile start


As PostgreSQL database uses 8k blocks let’s set it in ZFS. We could of course create dedicated dataset for this purpose if needed.

# zfs set recordsize=8k zroot/ROOT/default

Now, let’s start the PostgreSQL database.

# /usr/local/etc/rc.d/postgresql start
2019-12-31 11:47:04.918 CET [36089] LOG:  starting PostgreSQL 12.1 on amd64-portbld-freebsd12.0, compiled by FreeBSD clang version 6.0.1 (tags/RELEASE_601/final 335540) (based on LLVM 6.0.1), 64-bit
2019-12-31 11:47:04.918 CET [36089] LOG:  listening on IPv6 address "::1", port 5432
2019-12-31 11:47:04.918 CET [36089] LOG:  listening on IPv4 address "127.0.0.1", port 5432
2019-12-31 11:47:04.919 CET [36089] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"
2019-12-31 11:47:04.928 CET [36089] LOG:  ending log output to stderr
2019-12-31 11:47:04.928 CET [36089] HINT:  Future log output will go to log destination "syslog".

We will now create PostgreSQL database for our Nextcloud instance.

# psql -hlocalhost -Upostgres
psql (12.1)
Type "help" for help.

postgres=# CREATE USER nextcloud WITH PASSWORD 'NEXTCLOUD_DB_PASSWORD';
CREATE ROLE
postgres=# CREATE DATABASE nextcloud TEMPLATE template0 ENCODING 'UNICODE';
CREATE DATABASE
postgres=# ALTER DATABASE nextcloud OWNER TO nextcloud;
ALTER DATABASE
postgres=# \q

Keep in mind to put something more sophisticated in the NEXTCLOUD_DB_PASSWORD place.

PostgreSQL Cleanup and Indexing Script

Lets automate some PostgreSQL housekeeping.

# mkdir -p /var/db/postgres/bin
# chown postgres /var/db/postgres/bin
# vi /var/db/postgres/bin/vacuum.sh

#! /bin/sh

/usr/local/bin/vacuumdb -az 1> /dev/null 2> /dev/null
/usr/local/bin/reindexdb -a 1> /dev/null 2> /dev/null
/usr/local/bin/reindexdb -s 1> /dev/null 2> /dev/null
:wq

# cat /var/db/postgres/bin/vacuum.sh
#! /bin/sh

/usr/local/bin/vacuumdb -az 1> /dev/null 2> /dev/null
/usr/local/bin/reindexdb -a 1> /dev/null 2> /dev/null
/usr/local/bin/reindexdb -s 1> /dev/null 2> /dev/null

# chown postgres /var/db/postgres/bin/vacuum.sh
# chmod +x /var/db/postgres/bin/vacuum.sh

# su - postgres -c 'crontab -e'
0 0 * * * /var/db/postgres/bin/vacuum.sh
:wq
/tmp/crontab.JMg5BfT5HV: 2 lines, 42 characters.
crontab: installing new crontab

# su - postgres -c 'crontab -l'
0 0 * * * /var/db/postgres/bin/vacuum.sh

# su - postgres -c '/var/db/postgres/bin/vacuum.sh'

Nginx

Now its time for Nginx webserver.

# chown -R www:www /var/log/nginx

# ls -l /var/log/nginx
total 3
-rw-r-----  1 www  www   64 2019.12.31 00:00 access.log
-rw-r-----  1 www  www  133 2019.12.31 00:00 access.log.0.bz2
-rw-r-----  1 www  www   64 2019.12.31 00:00 error.log
-rw-r-----  1 www  www  133 2019.12.31 00:00 error.log.0.bz2

… and its main nginx.conf configuration file.

# cat /usr/local/etc/nginx/nginx.conf
user www;
worker_processes 4;
worker_rlimit_nofile 51200;
error_log /var/log/nginx/error.log;

events {
  worker_connections 1024;
}

http {
  include mime.types;
  default_type application/octet-stream;
  log_format main '$remote_addr - $remote_user [$time_local] "$request" ';
  access_log /var/log/nginx/access.log main;
  sendfile on;
  keepalive_timeout 65;

  upstream php-handler {
    server 127.0.0.1:9000;
  }

  server {
    # ENFORCE HTTPS
    listen 80;
    server_name nextcloud.domain.com;
    return 301 https://$server_name$request_uri;
  }

  server {
    listen 443 ssl http2;
    server_name nextcloud.domain.com;
    ssl_certificate /usr/local/etc/nginx/ssl/ssl-bundle.crt;
    ssl_certificate_key /usr/local/etc/nginx/ssl/server.key;

    # HEADERS SECURITY RELATED
    add_header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload;";
    add_header Referrer-Policy "no-referrer";

    # HEADERS
    add_header X-Content-Type-Options nosniff;
    add_header X-XSS-Protection "1; mode=block";
    add_header X-Robots-Tag none;
    add_header X-Download-Options noopen;
    add_header X-Permitted-Cross-Domain-Policies none;

    # PATH TO THE ROOT OF YOUR INSTALLATION
    root /usr/local/www/nextcloud/;

    location = /robots.txt {
      allow all;
      log_not_found off;
      access_log off;
    }

    location = /.well-known/carddav {
      return 301 $scheme://$host/remote.php/dav;
    }

    location = /.well-known/caldav {
      return 301 $scheme://$host/remote.php/dav;
    }

    # BUFFERS TIMEOUTS UPLOAD SIZES
    client_max_body_size 16400M;
    client_body_buffer_size 1048576k;
    send_timeout 3000;

    # ENABLE GZIP BUT DO NOT REMOVE ETag HEADERS
    gzip on;
    gzip_vary on;
    gzip_comp_level 4;
    gzip_min_length 256;
    gzip_proxied expired no-cache no-store private no_last_modified no_etag auth;
    gzip_types application/atom+xml application/javascript application/json application/ld+json application/manifest+json application/rss+xml application/vnd.geo+json application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/bmp image/svg+xml image/x-icon text/cache-manifest text/css text/plain text/vcard text/vnd.rim.location.xloc text/vtt text/x-component text/x-cross-domain-policy;

    location / {
      rewrite ^ /index.php$request_uri;
    }

    location ~ ^/(?:build|tests|config|lib|3rdparty|templates|data)/ {
      deny all;
    }

    location ~ ^/(?:\.|autotest|occ|issue|indie|db_|console) {
      deny all;
    }

    location ~ ^\/(?:index|remote|public|cron|core\/ajax\/update|status|ocs\/v[12]|updater\/.+|oc[ms]-provider\/.+)\.php(?:$|\/) {
      fastcgi_split_path_info ^(.+\.php)(/.*)$;
      include fastcgi_params;
      fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
      fastcgi_param PATH_INFO $fastcgi_path_info;
      fastcgi_param HTTPS on;
      fastcgi_param modHeadersAvailable true;
      fastcgi_param front_controller_active true;
      fastcgi_pass php-handler;
      fastcgi_intercept_errors on;
      fastcgi_request_buffering off;
      fastcgi_keep_conn off;
      fastcgi_buffers 16 256K;
      fastcgi_buffer_size 256k;
      fastcgi_busy_buffers_size 256k;
      fastcgi_temp_file_write_size 256k;
      fastcgi_send_timeout 3000s;
      fastcgi_read_timeout 3000s;
      fastcgi_connect_timeout 3000s;
    }

    location ~ ^\/(?:updater|oc[ms]-provider)(?:$|\/) {
      try_files $uri/ =404;
      index index.php;
    }

    # ADDING THE CACHE CONTROL HEADER FOR JS AND CSS FILES
    # MAKE SURE IT IS BELOW PHP BLOCK
    location ~ \.(?:css|js|woff2?|svg|gif)$ {
      try_files $uri /index.php$uri$is_args$args;
      add_header Cache-Control "public, max-age=15778463";
      # HEADERS SECURITY RELATED
      # IT IS INTENDED TO HAVE THOSE DUPLICATED TO ONES ABOVE
      add_header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload;";
      # HEADERS
      add_header X-Content-Type-Options nosniff;
      add_header X-XSS-Protection "1; mode=block";
      add_header X-Robots-Tag none;
      add_header X-Download-Options noopen;
      add_header X-Permitted-Cross-Domain-Policies none;
      # OPTIONAL: DONT LOG ACCESS TO ASSETS
      access_log off;
    }

    location ~ \.(?:png|html|ttf|ico|jpg|jpeg)$ {
      try_files $uri /index.php$uri$is_args$args;
      # OPTIONAL: DONT LOG ACCESS TO OTHER ASSETS
      access_log off;
    }
  }
}

OpenSSL HTTPS Certificates

We will generate a certificates needed for HTTPS service for Nextcloud.

# mkdir -p /usr/local/etc/nginx/ssl

# cd /usr/local/etc/nginx/ssl

# openssl genrsa -des3 -out server.key 2048
Generating RSA private key, 2048 bit long modulus (2 primes)
............+++++
....+++++
e is 65537 (0x010001)
Enter pass phrase for server.key: SERVER_KEY_PASSWORD
Verifying - Enter pass phrase for server.key: SERVER_KEY_PASSWORD

As usual use something more sensible then SERVER_KEY_PASSWORD string here ๐Ÿ™‚

# openssl req -new -key server.key -out server.csr
Enter pass phrase for server.key:
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:PL
State or Province Name (full name) [Some-State]:lodzkie
Locality Name (eg, city) []:Lodz
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Vermaden Enterprises Ltd.
Organizational Unit Name (eg, section) []:IT Department
Common Name (e.g. server FQDN or YOUR name) []:nextcloud.domain.com
Email Address []:vermaden@interia.pl

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:


# cp server.key server.key.orig

# openssl rsa -in server.key.orig -out server.key
Enter pass phrase for server.key.orig: SERVER_KEY_PASSWORD
writing RSA key

# ls -l /usr/local/etc/nginx/ssl
total 7
-rw-r--r--  1 root  wheel  1151 2019.12.31 12:39 server.csr
-rw-------  1 root  wheel  1679 2019.12.31 12:41 server.key
-rw-------  1 root  wheel  1751 2019.12.31 12:40 server.key.orig

# openssl x509 -req -days 7000 -in server.csr -signkey server.key -out server.crt
Signature ok
subject=C = PL, ST = lodzkie, L = Lodz, O = Vermaden Enterprises Ltd., OU = IT Department, CN = nextcloud.domain.com, emailAddress = vermaden@interia.pl
Getting Private key

# ln -s /usr/local/etc/nginx/ssl/server.crt /usr/local/etc/nginx/ssl/ssl-bundle.crt

PHP

Here is the used PHP configuration with up to 16GB files for Nextcloud.

# grep '^[^;]' /usr/local/etc/php.ini
[PHP]
max_input_time=3600
engine = On
short_open_tag = On
precision = 14
output_buffering = OFF
zlib.output_compression = Off
implicit_flush = Off
unserialize_callback_func =
serialize_precision = 17
disable_functions =
disable_classes =
zend.enable_gc = On
expose_php = On
max_execution_time = 3600
max_input_time = 30000
memory_limit = 1024M
error_reporting = E_ALL & ~E_DEPRECATED & ~E_STRICT
display_errors = Off
display_startup_errors = Off
log_errors = On
log_errors_max_len = 1024
ignore_repeated_errors = Off
ignore_repeated_source = Off
report_memleaks = On
track_errors = Off
html_errors = On
error_log = /var/log/php.log
variables_order = "GPCS"
request_order = "GP"
register_argc_argv = Off
auto_globals_jit = On
post_max_size = 16400M
auto_prepend_file =
auto_append_file =
default_mimetype = "text/html"
default_charset = "UTF-8"
doc_root =
user_dir =
enable_dl = Off
file_uploads = On
upload_max_filesize = 16400M
max_file_uploads = 64
allow_url_fopen = On
allow_url_include = Off
default_socket_timeout = 300
[CLI Server]
cli_server.color = On
[Date]
date.timezone = Europe/Warsaw
[filter]
[iconv]
[intl]
[sqlite3]
[Pcre]
[Pdo]
[Pdo_mysql]
pdo_mysql.cache_size = 2000
pdo_mysql.default_socket=
[Phar]
[mail function]
SMTP = localhost
smtp_port = 25
mail.add_x_header = On
[SQL]
sql.safe_mode = Off
[ODBC]
odbc.allow_persistent = On
odbc.check_persistent = On
odbc.max_persistent = -1
odbc.max_links = -1
odbc.defaultlrl = 4096
odbc.defaultbinmode = 1
[Interbase]
ibase.allow_persistent = 1
ibase.max_persistent = -1
ibase.max_links = -1
ibase.timestampformat = "%Y-%m-%d %H:%M:%S"
ibase.dateformat = "%Y-%m-%d"
ibase.timeformat = "%H:%M:%S"
[MySQLi]
mysqli.max_persistent = -1
mysqli.allow_persistent = On
mysqli.max_links = -1
mysqli.cache_size = 2000
mysqli.default_port = 3306
mysqli.default_socket =
mysqli.default_host =
mysqli.default_user =
mysqli.default_pw =
mysqli.reconnect = Off
[mysqlnd]
mysqlnd.collect_statistics = On
mysqlnd.collect_memory_statistics = Off
[OCI8]
[PostgreSQL]
pgsql.allow_persistent = On
pgsql.auto_reset_persistent = Off
pgsql.max_persistent = -1
pgsql.max_links = -1
pgsql.ignore_notice = 0
pgsql.log_notice = 0
[bcmath]
bcmath.scale = 0
[browscap]
[Session]
session.save_handler = files
session.save_path = "/tmp"
session.use_strict_mode = 0
session.use_cookies = 1
session.use_only_cookies = 1
session.name = PHPSESSID
session.auto_start = 0
session.cookie_lifetime = 0
session.cookie_path = /
session.cookie_domain =
session.cookie_httponly =
session.serialize_handler = php
session.gc_probability = 1
session.gc_divisor = 1000
session.gc_maxlifetime = 1440
session.referer_check =
session.cache_limiter = nocache
session.cache_expire = 180
session.use_trans_sid = 0
session.hash_function = 0
session.hash_bits_per_character = 5
url_rewriter.tags = "a=href,area=href,frame=src,input=src,form=fakeentry"
[Assertion]
zend.assertions = -1
[COM]
[mbstring]
[gd]
[exif]
[Tidy]
tidy.clean_output = Off
[soap]
soap.wsdl_cache_enabled=1
soap.wsdl_cache_dir="/tmp"
soap.wsdl_cache_ttl=86400
soap.wsdl_cache_limit = 5
[sysvshm]
[ldap]
ldap.max_links = -1
[mcrypt]
[dba]
[opcache]
opcache.enable=1
opcache.enable_cli=1
opcache.interned_strings_buffer=8
opcache.max_accelerated_files=10000
opcache.memory_consumption=128
opcache.save_comments=1
opcache.revalidate_freq=1
[curl]
[openssl] 

PHP PostgreSQL Database Settings

Below are needed to make PHP work with PostgreSQL database.

# cat /usr/local/etc/php/ext-20-pgsql.ini
extension=pgsql.so

# cat  /usr/local/etc/php/ext-20-pgsql.ini

[PostgresSQL]
pgsql.allow_persistent = On
pgsql.auto_reset_persistent = Off
pgsql.max_persistent = -1
pgsql.max_links = -1
pgsql.ignore_notice = 0
pgsql.log_notice = 0
EOF

# cat /usr/local/etc/php/ext-20-pgsql.ini
extension=pgsql.so

[PostgresSQL]
pgsql.allow_persistent = On
pgsql.auto_reset_persistent = Off
pgsql.max_persistent = -1
pgsql.max_links = -1
pgsql.ignore_notice = 0
pgsql.log_notice = 0


… and the second one.

# cat /usr/local/etc/php/ext-30-pdo_pgsql.ini
extension=pdo_pgsql.so

# cat  /usr/local/etc/php/ext-30-pdo_pgsql.ini

[PostgresSQL]
pgsql.allow_persistent = On
pgsql.auto_reset_persistent = Off
pgsql.max_persistent = -1
pgsql.max_links = -1
pgsql.ignore_notice = 0
pgsql.log_notice = 0
EOF

# cat /usr/local/etc/php/ext-30-pdo_pgsql.ini
extension=pdo_pgsql.so

[PostgresSQL]
pgsql.allow_persistent = On
pgsql.auto_reset_persistent = Off
pgsql.max_persistent = -1
pgsql.max_links = -1
pgsql.ignore_notice = 0
pgsql.log_notice = 0

PHP FPM

Now the PHP FPM daemon.

# grep '^[^;]' /usr/local/etc/php-fpm.conf
[global]
pid = run/php-fpm.pid
error_log = log/php-fpm.log
syslog.facility = daemon
include=/usr/local/etc/php-fpm.d/*.conf

# touch /var/log/php-fpm.log

# chown www:www /var/log/php-fpm.log

# grep '^[^;]' /usr/local/etc/php-fpm.d/www.conf
[www]
user = www
group = www
listen = 127.0.0.1:9000
listen.backlog = -1
listen.owner = www
listen.group = www
listen.mode = 0660
listen.allowed_clients = 127.0.0.1
pm = static
pm.max_children = 8
pm.start_servers = 4
pm.min_spare_servers = 4
pm.max_spare_servers = 32
pm.process_idle_timeout = 1000s;
pm.max_requests = 500
request_terminate_timeout = 0
rlimit_files = 51200
env[HOSTNAME] = $HOSTNAME
env[PATH] = /usr/local/bin:/usr/bin:/bin
env[TMP] = /tmp
env[TMPDIR] = /tmp
env[TEMP] = /tmp

Start Backend Services

We will now start all ‘backend’ services needed for Nextcloud.

# service postgresql start
2020-01-02 13:18:05.970 CET [52233] LOG:  starting PostgreSQL 12.1 on amd64-portbld-freebsd12.0, compiled by FreeBSD clang version 6.0.1 (tags/RELEASE_601/final 335540) (based on LLVM 6.0.1), 64-bit
2020-01-02 13:18:05.974 CET [52233] LOG:  listening on IPv6 address "::1", port 5432
2020-01-02 13:18:05.974 CET [52233] LOG:  listening on IPv4 address "127.0.0.1", port 5432
2020-01-02 13:18:05.975 CET [52233] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"
2020-01-02 13:18:06.024 CET [52233] LOG:  ending log output to stderr
2020-01-02 13:18:06.024 CET [52233] HINT:  Future log output will go to log destination "syslog".

# service postgresql status
pg_ctl: server is running (PID: 36089)
/usr/local/bin/postgres "-D" "/var/db/postgres/data12"

# service php-fpm start
Performing sanity check on php-fpm configuration:
[02-Jan-2020 13:16:50] NOTICE: configuration file /usr/local/etc/php-fpm.conf test is successful

Starting php_fpm.

# service php-fpm status
php_fpm is running as pid 52193.

# service memcached start
Starting memcached.

# service memcached status
memcached is running as pid 52273.

# service nginx start
Performing sanity check on nginx configuration:
nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok
nginx: configuration file /usr/local/etc/nginx/nginx.conf test is successful
Starting nginx.

Nextcloud Configuration

I created a link named /data to the Nextcloud data directory located at /usr/local/www/nextcloud/data place.

# ln -s /usr/local/www/nextcloud/data /data

The we use Firefox or other web browser to finish the Nextcloud configuration.

Type https://1.2.3.4 in the browser where 1.2.3.4 is your Nextcloud instance IP address.

I am sorry but the following image is in the Polish language – I forgot to change it to English … but I assume you will what to put in these fields by context.

nextcloud-setup.png

After we finish the setup we go straight to Nextcloud Overview page at https://1.2.3.4/settings/admin/serverinfoto page to see what else needs to be taken care of.

nextcloud-setup-overview.png

Two issues needs to be addressed. One is about Nginx configuration, the other is about PostgreSQL, let’s fix them.

We will add needed header to the Nginx configuration file.

# diff -u /usr/local/etc/nginx/nginx.conf.OLD /usr/local/etc/nginx/nginx.conf
--- /usr/local/etc/nginx/nginx.conf.OLD  2020-01-02 14:21:58.359398000 +0100
+++ /usr/local/etc/nginx/nginx.conf      2020-01-02 14:21:42.823426000 +0100
@@ -46,6 +46,7 @@
     add_header X-Robots-Tag none;
     add_header X-Download-Options noopen;
     add_header X-Permitted-Cross-Domain-Policies none;
+    add_header X-Frame-Options "SAMEORIGIN";

     # PATH TO THE ROOT OF YOUR INSTALLATION
     root /usr/local/www/nextcloud/;

# service nginx reload
Performing sanity check on nginx configuration:
nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok
nginx: configuration file /usr/local/etc/nginx/nginx.conf test is successful

… and update the PostgreSQL convertion.

# sudo -u www /usr/local/bin/php /usr/local/www/nextcloud/occ db:convert-filecache-bigint
Following columns will be updated:

* mounts.storage_id
* mounts.root_id
* mounts.mount_id

This can take up to hours, depending on the number of files in your instance!
Continue with the conversion (y/n)? [n] y

Viola! Both of our problems are gone now.

nextcloud-setup-overview-fixed.png

Trusted Domains

When you will enter the Nextcloud using different domain you will get a warning about that.

To add new Trusted Domain to the Nextcloud config do the following.

Here is how it looks before changes.

# grep -A 3 trusted /usr/local/www/nextcloud/config/config.php
  'trusted_domains' =>
  array (
    0 => '1.2.3.4',
  ),

We will now add nextcloud.domain.com domain.

# vi /usr/local/www/nextcloud/config/config.php

# grep -A 4 trusted /usr/local/www/nextcloud/config/config.php
  'trusted_domains' =>
  array (
    0 => '1.2.3.4',
    1 => 'nextcloud.domain.com',
  ),

You can of course add more with successive numbers.

# grep -A 5 trusted /usr/local/www/nextcloud/config/config.php
  'trusted_domains' =>
  array (
    0 => '1.2.3.4',
    1 => 'nextcloud.domain.com',
    2 => 'cloud.domain.com',
  ),

This is the end of this guide. Feel free to share your thougths ๐Ÿ™‚

Log Rotation with Newsyslog

Newsyslog is part of FreeBSD’s base system. We will add Nextcloud and backend daemons log files to Newsyslog configuration so they will be rotated.

 
# cat  /etc/newsyslog.conf
/data/nextcloud.log                          www:www     640  7     *    @T00  JC
/usr/local/www/nextcloud/data/nextcloud.log  www:www     640  7     *    @T00  JC
/var/log/php-fpm.log                         www:www     640  7     *    @T00  JC
/var/log/nginx/error.log                     www:www     640  7     *    @T00  JC
/var/log/nginx/access.log                    www:www     640  7     *    @T00  JC
EOF

Now you will not run out of free space when logs will grow in time.

EOF

ย 

FreeBSD Enterprise 1 PB Storage

Today FreeBSD operating system turns 26 years old. 19 June is an International FreeBSD Day. This is why I got something special today :). How about using FreeBSD as an Enterprise Storage solution on real hardware? This where FreeBSD shines with all its storage features ZFS included.

Today I will show you how I have built so called Enterprise Storage based on FreeBSD system along with more then 1 PB (Petabyte) of raw capacity.

I have build various storage related systems based on FreeBSD:

This project is different. How much storage space can you squeeze from a single 4U system? It turns out a lot! Definitely more then 1 PB (1024 TB) of raw storage space.

Here is the (non clickable) Table of Contents.

  • Hardware
  • Management Interface
  • BIOS/UEFI
  • FreeBSD System
    • Disks Preparation
    • ZFS Pool Configuration
    • ZFS Settings
    • Network Configuration
    • FreeBSD Configuration
  • Purpose
  • Performance
    • Network Performance
    • Disk Subsystem Performance
  • FreeNAS
  • UPDATE 1 – BSD Now 305
  • UPDATE 2 โ€“ Real Life Pictures in Data Center

Hardware

There are 4U servers with 90-100 3.5″ drive slots which will allow you to pack 1260-1400 Terabytes of data (with 14 TB drives). Examples of such systems are:

I would use the first one – the TYAN FA100 for short name.

logo-tyan.png

While both GlusterFS and Minio clusters were cone on virtual hardware (or even FreeBSD Jails containers) this one uses real physical hardware.

The build has following specifications.

 2 x 10-Core Intel Xeon Silver 4114 CPU @ 2.20GHz
 4 x 32 GB RAM DDR4 (128 GB Total)
 2 x Intel SSD DC S3500 240 GB (System)
90 x Toshiba HDD MN07ACA12TE 12 TB (Data)
 2 x Broadcom SAS3008 Controller
 2 x Intel X710 DA-2 10GE Card
 2 x Power Supply

Price of the whole system is about $65 000 – drives included. Here is how it looks.

tyan-fa100-small.jpg

One thing that you will need is a rack cabinet that is 1200 mm long to fit that monster ๐Ÿ™‚

Management Interface

The so called Lights Out management interface is really nice. Its not bloated, well organized and works quite fast. you can create several separate user accounts or can connect to external user services like LDAP/AD/RADIUS for example.

n01.png

After logging in a simple Dashboard welcomes us.

n02.png

We have access to various Sensor information available with temperatures of system components.

n03

We have System Inventory information with installed hardware.

n04.png

There is separate Settings menu for various setup options.

n05.png

I know its 2019 but HTML5 only Remote Control (remote console) without need for any third party plugins like Java/Silverlight/Flash/… is very welcomed. It works very well too.

n06.png

n07.png

One is of course allowed to power on/off/cycle the box remotely.

n08.png

The Maintenance menu for BIOS updates.

n09.png

BIOS/UEFI

After booting into the BIOS/UEFI setup its possible to select from which drives to boot from. On the screenshots the two SSD drives prepared for system.

nas01.png

The BIOS/UEFI interface shows two Enclosures but its two Broadcom SAS3008 controllers. Some drive are attached via first Broadcom SAS3008 controller, the rest is attached via the second one, and they call them Enclosures instead od of controllers for some reason.

nas05.png

FreeBSD System

I have chosen latest FreeBSD 12.0-RELEASE for the purpose of this installation. Its generally very ‘default’ installation with ZFS mirror on two SSD disks. Nothing special.

logo-freebsd.jpg

The installation of course supports the ZFS Boot Environments bulletproof upgrades/changes feature.

# zpool list zroot
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
zroot   220G  3.75G   216G        -         -     0%     1%  1.00x  ONLINE  -

# zpool status zroot
  pool: zroot
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        zroot       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            da91p4  ONLINE       0     0     0
            da11p4  ONLINE       0     0     0

errors: No known data errors

# df -g
Filesystem              1G-blocks Used  Avail Capacity  Mounted on
zroot/ROOT/default            211    2    209     1%    /
devfs                           0    0      0   100%    /dev
zroot/tmp                     209    0    209     0%    /tmp
zroot/usr/home                209    0    209     0%    /usr/home
zroot/usr/ports               210    0    209     0%    /usr/ports
zroot/usr/src                 210    0    209     0%    /usr/src
zroot/var/audit               209    0    209     0%    /var/audit
zroot/var/crash               209    0    209     0%    /var/crash
zroot/var/log                 209    0    209     0%    /var/log
zroot/var/mail                209    0    209     0%    /var/mail
zroot/var/tmp                 209    0    209     0%    /var/tmp

# beadm list
BE      Active Mountpoint  Space Created
default NR     /            2.4G 2019-05-24 13:24

Disks Preparation

From all the possible setups with 90 disks of 12 TB capacity I have chosen to go the RAID60 way – its ZFS equivalent of course. With 12 disks in each RAID6 (raidz2) group – there will be 7 such groups – we will have 84 used for the ZFS pool with 6 drives left as SPARE disks – that plays well for me. The disks distribution will look more or less like that.

DISKS  CONTENT
   12  raidz2-0
   12  raidz2-1
   12  raidz2-2
   12  raidz2-3
   12  raidz2-4
   12  raidz2-5
   12  raidz2-6
    6  spares
   90  TOTAL

Here is how FreeBSD system sees these drives by camcontrol(8) command. Sorted by attached SAS controller – scbus(4).

# camcontrol devlist | sort -k 6
(AHCI SGPIO Enclosure 1.00 0001)   at scbus2 target 0 lun 0 (pass0,ses0)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 50 lun 0 (pass1,da0)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 52 lun 0 (pass2,da1)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 54 lun 0 (pass3,da2)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 56 lun 0 (pass5,da4)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 57 lun 0 (pass6,da5)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 59 lun 0 (pass7,da6)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 60 lun 0 (pass8,da7)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 66 lun 0 (pass9,da8)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 67 lun 0 (pass10,da9)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 74 lun 0 (pass11,da10)
(ATA INTEL SSDSC2KB24 0100)        at scbus3 target 75 lun 0 (pass12,da11)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 76 lun 0 (pass13,da12)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 82 lun 0 (pass14,da13)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 83 lun 0 (pass15,da14)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 85 lun 0 (pass16,da15)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 87 lun 0 (pass17,da16)
(Tyan B7118 0500)                  at scbus3 target 88 lun 0 (pass18,ses1)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 89 lun 0 (pass19,da17)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 90 lun 0 (pass20,da18)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 91 lun 0 (pass21,da19)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 92 lun 0 (pass22,da20)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 93 lun 0 (pass23,da21)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 94 lun 0 (pass24,da22)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 95 lun 0 (pass25,da23)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 96 lun 0 (pass26,da24)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 97 lun 0 (pass27,da25)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 98 lun 0 (pass28,da26)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 99 lun 0 (pass29,da27)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 100 lun 0 (pass30,da28)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 101 lun 0 (pass31,da29)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 102 lun 0 (pass32,da30)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 103 lun 0 (pass33,da31)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 104 lun 0 (pass34,da32)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 105 lun 0 (pass35,da33)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 106 lun 0 (pass36,da34)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 107 lun 0 (pass37,da35)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 108 lun 0 (pass38,da36)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 109 lun 0 (pass39,da37)
(ATA TOSHIBA MG07ACA1 0101)        at scbus3 target 110 lun 0 (pass40,da38)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 48 lun 0 (pass41,da39)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 49 lun 0 (pass42,da40)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 51 lun 0 (pass43,da41)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 53 lun 0 (pass44,da42)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 55 lun 0 (da43,pass45)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 59 lun 0 (pass46,da44)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 64 lun 0 (pass47,da45)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 67 lun 0 (pass48,da46)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 68 lun 0 (pass49,da47)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 69 lun 0 (pass50,da48)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 73 lun 0 (pass51,da49)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 76 lun 0 (pass52,da50)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 77 lun 0 (pass53,da51)
(Tyan B7118 0500)                  at scbus4 target 80 lun 0 (pass54,ses2)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 81 lun 0 (pass55,da52)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 82 lun 0 (pass56,da53)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 83 lun 0 (pass57,da54)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 84 lun 0 (pass58,da55)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 85 lun 0 (pass59,da56)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 86 lun 0 (pass60,da57)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 87 lun 0 (pass61,da58)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 88 lun 0 (pass62,da59)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 89 lun 0 (da63,pass66)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 90 lun 0 (pass64,da61)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 91 lun 0 (pass65,da62)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 92 lun 0 (da60,pass63)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 94 lun 0 (pass67,da64)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 97 lun 0 (pass68,da65)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 98 lun 0 (pass69,da66)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 99 lun 0 (pass70,da67)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 100 lun 0 (pass71,da68)
(Tyan B7118 0500)                  at scbus4 target 101 lun 0 (pass72,ses3)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 102 lun 0 (pass73,da69)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 103 lun 0 (pass74,da70)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 104 lun 0 (pass75,da71)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 105 lun 0 (pass76,da72)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 106 lun 0 (pass77,da73)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 107 lun 0 (pass78,da74)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 108 lun 0 (pass79,da75)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 109 lun 0 (pass80,da76)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 110 lun 0 (pass81,da77)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 111 lun 0 (pass82,da78)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 112 lun 0 (pass83,da79)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 113 lun 0 (pass84,da80)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 114 lun 0 (pass85,da81)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 115 lun 0 (pass86,da82)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 116 lun 0 (pass87,da83)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 117 lun 0 (pass88,da84)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 118 lun 0 (pass89,da85)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 119 lun 0 (pass90,da86)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 120 lun 0 (pass91,da87)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 121 lun 0 (pass92,da88)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 122 lun 0 (pass93,da89)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 123 lun 0 (pass94,da90)
(ATA INTEL SSDSC2KB24 0100)        at scbus4 target 124 lun 0 (pass95,da91)
(ATA TOSHIBA MG07ACA1 0101)        at scbus4 target 125 lun 0 (da3,pass4)

One my ask how to identify which disk is which when the FAILURE will came … this is where FreeBSD’s sesutil(8) command comes handy.

# sesutil locate all off
# sesutil locate da64 on

The first sesutil(8) command disables all location lights in the enclosure. The second one turns on the identification for disk da64.

I will also make sure to NOT use the whole space of each drive. Such idea may be pointless but imagine the following situation. Five 12 TB disks failed after 3 years. You can not get the same model drives so you get other 12 TB drives, maybe even from other manufacturer.

# grep da64 /var/run/dmesg.boot
da64 at mpr1 bus 0 scbus4 target 93 lun 0
da64:  Fixed Direct Access SPC-4 SCSI device
da64: Serial Number 98G0A1EQF95G
da64: 1200.000MB/s transfers
da64: Command Queueing enabled
da64: 11444224MB (23437770752 512 byte sectors)

A single 12 TB drive has 23437770752 of 512 byte sectors which equals 12000138625024 bytes of raw capacity.

# expr 23437770752 \* 512
12000138625024

Now image that these other 12 TB drives from other manufacturer will come with 4 bytes smaller size … ZFS will not allow their usage because their size is smaller.

This is why I will use exactly 11175 GB size of each drive which is more or less 1 GB short of its total 11176 GB size.

Below is command that will do that for me for all 90 disks.

# camcontrol devlist \
    | grep TOSHIBA \
    | awk '{print $NF}' \
    | awk -F ',' '{print $2}' \
    | tr -d ')' \
    | while read DISK
      do
        gpart destroy -F                   ${DISK} 1> /dev/null 2> /dev/null
        gpart create -s GPT                ${DISK}
        gpart add -t freebsd-zfs -s 11175G ${DISK}
      done

# gpart show da64
=>         40  23437770672  da64  GPT  (11T)
           40  23435673600     1  freebsd-zfs  (11T)
  23435673640      2097072        - free -  (1.0G)


ZFS Pool Configuration

Next, we will have to create our ZFS pool, its probably the longest zpool command I have ever executed ๐Ÿ™‚

As the Toshiba 12 TB disks have 4k sectors we will need to set vfs.zfs.min_auto_ashift to 12 to force them.

# sysctl vfs.zfs.min_auto_ashift=12
vfs.zfs.min_auto_ashift: 12 -> 12

# zpool create nas02 \
    raidz2  da0p1  da1p1  da2p1  da3p1  da4p1  da5p1  da6p1  da7p1  da8p1  da9p1 da10p1 da12p1 \
    raidz2 da13p1 da14p1 da15p1 da16p1 da17p1 da18p1 da19p1 da20p1 da21p1 da22p1 da23p1 da24p1 \
    raidz2 da25p1 da26p1 da27p1 da28p1 da29p1 da30p1 da31p1 da32p1 da33p1 da34p1 da35p1 da36p1 \
    raidz2 da37p1 da38p1 da39p1 da40p1 da41p1 da42p1 da43p1 da44p1 da45p1 da46p1 da47p1 da48p1 \
    raidz2 da49p1 da50p1 da51p1 da52p1 da53p1 da54p1 da55p1 da56p1 da57p1 da58p1 da59p1 da60p1 \
    raidz2 da61p1 da62p1 da63p1 da64p1 da65p1 da66p1 da67p1 da68p1 da69p1 da70p1 da71p1 da72p1 \
    raidz2 da73p1 da74p1 da75p1 da76p1 da77p1 da78p1 da79p1 da80p1 da81p1 da82p1 da83p1 da84p1 \
    spare  da85p1 da86p1 da87p1 da88p1 da89p1 da90p1

# zpool status
  pool: nas02
 state: ONLINE
  scan: scrub repaired 0 in 0 days 00:00:05 with 0 errors on Fri May 31 10:26:29 2019
config:

        NAME        STATE     READ WRITE CKSUM
        nas02       ONLINE       0     0     0
          raidz2-0  ONLINE       0     0     0
            da0p1   ONLINE       0     0     0
            da1p1   ONLINE       0     0     0
            da2p1   ONLINE       0     0     0
            da3p1   ONLINE       0     0     0
            da4p1   ONLINE       0     0     0
            da5p1   ONLINE       0     0     0
            da6p1   ONLINE       0     0     0
            da7p1   ONLINE       0     0     0
            da8p1   ONLINE       0     0     0
            da9p1   ONLINE       0     0     0
            da10p1  ONLINE       0     0     0
            da12p1  ONLINE       0     0     0
          raidz2-1  ONLINE       0     0     0
            da13p1  ONLINE       0     0     0
            da14p1  ONLINE       0     0     0
            da15p1  ONLINE       0     0     0
            da16p1  ONLINE       0     0     0
            da17p1  ONLINE       0     0     0
            da18p1  ONLINE       0     0     0
            da19p1  ONLINE       0     0     0
            da20p1  ONLINE       0     0     0
            da21p1  ONLINE       0     0     0
            da22p1  ONLINE       0     0     0
            da23p1  ONLINE       0     0     0
            da24p1  ONLINE       0     0     0
          raidz2-2  ONLINE       0     0     0
            da25p1  ONLINE       0     0     0
            da26p1  ONLINE       0     0     0
            da27p1  ONLINE       0     0     0
            da28p1  ONLINE       0     0     0
            da29p1  ONLINE       0     0     0
            da30p1  ONLINE       0     0     0
            da31p1  ONLINE       0     0     0
            da32p1  ONLINE       0     0     0
            da33p1  ONLINE       0     0     0
            da34p1  ONLINE       0     0     0
            da35p1  ONLINE       0     0     0
            da36p1  ONLINE       0     0     0
          raidz2-3  ONLINE       0     0     0
            da37p1  ONLINE       0     0     0
            da38p1  ONLINE       0     0     0
            da39p1  ONLINE       0     0     0
            da40p1  ONLINE       0     0     0
            da41p1  ONLINE       0     0     0
            da42p1  ONLINE       0     0     0
            da43p1  ONLINE       0     0     0
            da44p1  ONLINE       0     0     0
            da45p1  ONLINE       0     0     0
            da46p1  ONLINE       0     0     0
            da47p1  ONLINE       0     0     0
            da48p1  ONLINE       0     0     0
          raidz2-4  ONLINE       0     0     0
            da49p1  ONLINE       0     0     0
            da50p1  ONLINE       0     0     0
            da51p1  ONLINE       0     0     0
            da52p1  ONLINE       0     0     0
            da53p1  ONLINE       0     0     0
            da54p1  ONLINE       0     0     0
            da55p1  ONLINE       0     0     0
            da56p1  ONLINE       0     0     0
            da57p1  ONLINE       0     0     0
            da58p1  ONLINE       0     0     0
            da59p1  ONLINE       0     0     0
            da60p1  ONLINE       0     0     0
          raidz2-5  ONLINE       0     0     0
            da61p1  ONLINE       0     0     0
            da62p1  ONLINE       0     0     0
            da63p1  ONLINE       0     0     0
            da64p1  ONLINE       0     0     0
            da65p1  ONLINE       0     0     0
            da66p1  ONLINE       0     0     0
            da67p1  ONLINE       0     0     0
            da68p1  ONLINE       0     0     0
            da69p1  ONLINE       0     0     0
            da70p1  ONLINE       0     0     0
            da71p1  ONLINE       0     0     0
            da72p1  ONLINE       0     0     0
          raidz2-6  ONLINE       0     0     0
            da73p1  ONLINE       0     0     0
            da74p1  ONLINE       0     0     0
            da75p1  ONLINE       0     0     0
            da76p1  ONLINE       0     0     0
            da77p1  ONLINE       0     0     0
            da78p1  ONLINE       0     0     0
            da79p1  ONLINE       0     0     0
            da80p1  ONLINE       0     0     0
            da81p1  ONLINE       0     0     0
            da82p1  ONLINE       0     0     0
            da83p1  ONLINE       0     0     0
            da84p1  ONLINE       0     0     0
        spares
          da85p1    AVAIL
          da86p1    AVAIL
          da87p1    AVAIL
          da88p1    AVAIL
          da89p1    AVAIL
          da90p1    AVAIL

errors: No known data errors

# zpool list nas02
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
nas02   915T  1.42M   915T        -         -     0%     0%  1.00x  ONLINE  -

# zfs list nas02
NAME    USED  AVAIL  REFER  MOUNTPOINT
nas02    88K   675T   201K  none

ZFS Settings

As the primary role of this storage would be keeping files I will use one of the largest values for recordsize – 1 MB – this helps getting better compression ratio.

… but it will also serve as iSCSI Target in which we will try to fit in the native 4k blocks – thus 4096 bytes setting for iSCSI.

# zfs set compression=lz4         nas02
# zfs set atime=off               nas02
# zfs set mountpoint=none         nas02
# zfs set recordsize=1m           nas02
# zfs set redundant_metadata=most nas02
# zfs create                      nas02/nfs
# zfs create                      nas02/smb
# zfs create                      nas02/iscsi
# zfs set recordsize=4k           nas02/iscsi

Also one word on redundant_metadata as its not that obvious parameter. To quote the zfs(8) man page.

# man zfs
(...)
redundant_metadata=all | most
  Controls what types of metadata are stored redundantly.  ZFS stores
  an extra copy of metadata, so that if a single block is corrupted,
  the amount of user data lost is limited.  This extra copy is in
  addition to any redundancy provided at the pool level (e.g. by
  mirroring or RAID-Z), and is in addition to an extra copy specified
  by the copies property (up to a total of 3 copies).  For example if
  the pool is mirrored, copies=2, and redundant_metadata=most, then ZFS
  stores 6 copies of most metadata, and 4 copies of data and some
  metadata.

  When set to all, ZFS stores an extra copy of all metadata.  If a
  single on-disk block is corrupt, at worst a single block of user data
  (which is recordsize bytes long can be lost.)

  When set to most, ZFS stores an extra copy of most types of metadata.
  This can improve performance of random writes, because less metadata
  must be written.  In practice, at worst about 100 blocks (of
  recordsize bytes each) of user data can be lost if a single on-disk
  block is corrupt.  The exact behavior of which metadata blocks are
  stored redundantly may change in future releases.

  The default value is all.
(...)

From the description above we can see that its mostly useful on single device pools because when we have redundancy based on RAIDZ2 (RAID6 equivalent) we do not need to keep additional redundant copies of metadata. This helps to increase write performance.

For the record – iSCSI ZFS zvols are create with command like that one below – as sparse files – also called Thin Provisioning mode.

# zfs create -s -V 16T nas02/iscsi/test

As we have SPARE disks we will also need to enable the zfsd(8) daemon by adding zfsd_enable=YES to the /etc/rc.conf file.

We also need to enable autoreplace property for our pool because by default its set to off.

# zpool get autoreplace nas02
NAME   PROPERTY     VALUE    SOURCE
nas02  autoreplace  off      default

# zpool set autoreplace=on nas02

# zpool get autoreplace nas02
NAME   PROPERTY     VALUE    SOURCE
nas02  autoreplace  on       local

Other ZFS settings are in the /boot/loader.conf file. As this system has 128 GB RAM we will let ZFS use 50 to 75% of that amount for ARC.

# grep vfs.zfs /boot/loader.conf
  vfs.zfs.prefetch_disable=1
  vfs.zfs.cache_flush_disable=1
  vfs.zfs.vdev.cache.size=16M
  vfs.zfs.arc_min=64G
  vfs.zfs.arc_max=96G
  vfs.zfs.deadman_enabled=0

Network Configuration

This is what I really like about FreeBSD. To setup LACP link aggregation tou just need 5 lines in /etc/rc.conf file. On Red Hat Enterprise Linux you would need several files with many lines each.

# head -5 /etc/rc.conf
  defaultrouter="10.20.30.254"
  ifconfig_ixl0="up"
  ifconfig_ixl1="up"
  cloned_interfaces="lagg0"
  ifconfig_lagg0="laggproto lacp laggport ixl0 laggport ixl1 10.20.30.2/24 up"

# ifconfig lagg0
lagg0: flags=8843 metric 0 mtu 1500
        options=e507bb
        ether a0:42:3f:a0:42:3f
        inet 10.20.30.2 netmask 0xffffff00 broadcast 10.20.30.255
        laggproto lacp lagghash l2,l3,l4
        laggport: ixl0 flags=1c
        laggport: ixl1 flags=1c
        groups: lagg
        media: Ethernet autoselect
        status: active
        nd6 options=29

The Intel X710 DA-2 10GE network adapter is fully supported under FreeBSD by the ixl(4) driver.

intel-x710-da-2.jpg

Cisco Nexus Configuration

This is the Cisco Nexus configuration needed to enable LACP aggregation.

First the ports.

NEXUS-1  Eth1/32  NAS02_IXL0  connected 3  full  a-10G  SFP-H10GB-A
NEXUS-2  Eth1/32  NAS02_IXL1  connected 3  full  a-10G  SFP-H10GB-A

… and now aggregation.

interface Ethernet1/32
  description NAS02_IXL1
  switchport
  switchport access vlan 3
  mtu 9216
  channel-group 128 mode active
  no shutdown
!
interface port-channel128
  description NAS02
  switchport
  switchport access vlan 3
  mtu 9216
  vpc 128

… and the same/similar on the second Cisco Nexus NEXUS-2 switch.

FreeBSD Configuration

These are three most important configuration files on any FreeBSD system.

I will now post all settings I use on this storage system.

The /etc/rc.conf file.

# cat /etc/rc.conf
# NETWORK
  hostname="nas02.local"
  defaultrouter="10.20.30.254"
  ifconfig_ixl0="up"
  ifconfig_ixl1="up"
  cloned_interfaces="lagg0"
  ifconfig_lagg0="laggproto lacp laggport ixl0 laggport ixl1 10.20.30.2/24 up"

# KERNEL MODULES
  kld_list="${kld_list} aesni"

# DAEMON | YES
  zfs_enable=YES
  zfsd_enable=YES
  sshd_enable=YES
  ctld_enable=YES
  powerd_enable=YES

# DAEMON | NFS SERVER
  nfs_server_enable=YES
  nfs_client_enable=YES
  rpc_lockd_enable=YES
  rpc_statd_enable=YES
  rpcbind_enable=YES
  mountd_enable=YES
  mountd_flags="-r"

# OTHER
  dumpdev=NO

The /boot/loader.conf file.

# cat /boot/loader.conf
# BOOT OPTIONS
  autoboot_delay=3
  kern.geom.label.disk_ident.enable=0
  kern.geom.label.gptid.enable=0

# DISABLE INTEL HT
  machdep.hyperthreading_allowed=0

# UPDATE INTEL CPU MICROCODE AT BOOT BEFORE KERNEL IS LOADED
  cpu_microcode_load=YES
  cpu_microcode_name=/boot/firmware/intel-ucode.bin

# MODULES
  zfs_load=YES
  aio_load=YES

# RACCT/RCTL RESOURCE LIMITS
  kern.racct.enable=1

# DISABLE MEMORY TEST @ BOOT
  hw.memtest.tests=0

# PIPE KVA LIMIT | 320 MB
  kern.ipc.maxpipekva=335544320

# IPC
  kern.ipc.shmseg=1024
  kern.ipc.shmmni=1024
  kern.ipc.shmseg=1024
  kern.ipc.semmns=512
  kern.ipc.semmnu=256
  kern.ipc.semume=256
  kern.ipc.semopm=256
  kern.ipc.semmsl=512

# LARGE PAGE MAPPINGS
  vm.pmap.pg_ps_enabled=1

# ZFS TUNING
  vfs.zfs.prefetch_disable=1
  vfs.zfs.cache_flush_disable=1
  vfs.zfs.vdev.cache.size=16M
  vfs.zfs.arc_min=64G
  vfs.zfs.arc_max=96G

# ZFS DISABLE PANIC ON STALE I/O
  vfs.zfs.deadman_enabled=0

# NEWCONS SUSPEND
  kern.vt.suspendswitch=0

The /etc/sysctl.conf file.

# cat /etc/sysctl.conf
# ZFS ASHIFT
  vfs.zfs.min_auto_ashift=12

# SECURITY
  security.bsd.stack_guard_page=1

# SECURITY INTEL MDS (MICROARCHITECTURAL DATA SAMPLING) MITIGATION
  hw.mds_disable=3

# DISABLE ANNOYING THINGS
  kern.coredump=0
  hw.syscons.bell=0

# IPC
  kern.ipc.shmmax=4294967296
  kern.ipc.shmall=2097152
  kern.ipc.somaxconn=4096
  kern.ipc.maxsockbuf=5242880
  kern.ipc.shm_allow_removed=1

# NETWORK
  kern.ipc.maxsockbuf=16777216
  kern.ipc.soacceptqueue=1024
  net.inet.tcp.recvbuf_max=8388608
  net.inet.tcp.sendbuf_max=8388608
  net.inet.tcp.mssdflt=1460
  net.inet.tcp.minmss=1300
  net.inet.tcp.syncache.rexmtlimit=0
  net.inet.tcp.syncookies=0
  net.inet.tcp.tso=0
  net.inet.ip.process_options=0
  net.inet.ip.random_id=1
  net.inet.ip.redirect=0
  net.inet.icmp.drop_redirect=1
  net.inet.tcp.always_keepalive=0
  net.inet.tcp.drop_synfin=1
  net.inet.tcp.fast_finwait2_recycle=1
  net.inet.tcp.icmp_may_rst=0
  net.inet.tcp.msl=8192
  net.inet.tcp.path_mtu_discovery=0
  net.inet.udp.blackhole=1
  net.inet.tcp.blackhole=2
  net.inet.tcp.hostcache.expire=7200
  net.inet.tcp.delacktime=20

Purpose

Why one would built such appliance? Because its a lot cheaper then to get the ‘branded’ one. Think about Dell EMC Data Domain for example – and not just ‘any’ Data Domain but almost the highest one – the Data Domain DD9300 at least. It would cost about ten times more at least … with smaller capacity and taking not 4U but closer to 14U with three DS60 expanders.

But you can actually make this FreeBSD Enterprise Storage behave like Dell EMC Data Domain .. or like their Dell EMC Elastic Cloud Storage for example.

The Dell EMC CloudBoost can be deployed somewhere on your VMware stack to provide the DDBoost deduplication. Then you would need OpenStack Swift as its one of the supported backed devices.

emc-cloudboost-swift-cover.png

emc-cloudboost-swift-support.png

The OpenStack Swift package in FreeBSD is about 4-5 years behind reality (2.2.2) so you will have to use Bhyve here.

# pkg search swift
(...)
py27-swift-2.2.2_1             Highly available, distributed, eventually consistent object/blob store
(...)

Create Bhyve virtual machine on this FreeBSD Enterprise Storage with CentOS 7.6 system for example, then setup Swift there, but it will work. With 20 physical cores to spare and 128 GB RAM you would not even noticed its there.

This way you can use Dell EMC Networker with more then ten times cheaper storage.

In the past I also wrote about IBM Spectrum Protect (TSM) which would also greatly benefit from FreeBSD Enterprise Storage. I actually also use this FreeBSD based storage as space for IBM Spectrum Protect (TSM) container pool directories. Exported via iSCSI works like a charm.

You can also compare that FreeBSD Enterprise Storage to other storage appliances like iXsystems TrueNAS or EXAGRID.

Performance

You for sure would want to know how fast this FreeBSD Enterprise Storage performs ๐Ÿ™‚

I will share all performance data that I gathered with a pleasure.

Network Performance

First the network performance.

I user iperf3 as the benchmark.

I started the server on the FreeBSD side.

# iperf3 -s

… and then I started client on the Windows Server 2016 machine.

C:\iperf-3.1.3-win64>iperf3.exe -c nas02 -P 8
(...)
[SUM]   0.00-10.00  sec  10.8 GBytes  9.26 Gbits/sec                  receiver
(..)

This is with MTU 1500 – no Jumbo frames unfortunatelly ๐Ÿ˜ฆ

Unfortunatelly this system has only one physical 10GE interface but I did other test also. Using two such boxes with single 10GE interface. That saturated the dual 10GE LACP on FreeBSD side nicely.

I also exported NFS and iSCSI to Red Hat Enterprise Linux system. The network performance was about 500-600 MB/s on single 10GE interface. That would be 1000-1200 MB/s on LACP aggregation.

Disk Subsystem Performance

Now the disk subsystem.

First some naive test using diskinfo(8) FreeBSD’s builtin tool.

# diskinfo -ctv /dev/da12
/dev/da12
        512             # sectorsize
        12000138625024  # mediasize in bytes (11T)
        23437770752     # mediasize in sectors
        4096            # stripesize
        0               # stripeoffset
        1458933         # Cylinders according to firmware.
        255             # Heads according to firmware.
        63              # Sectors according to firmware.
        ATA TOSHIBA MG07ACA1    # Disk descr.
        98H0A11KF95G    # Disk ident.
        id1,enc@n500e081010445dbd/type@0/slot@c/elmdesc@ArrayDevice11   # Physical path
        No              # TRIM/UNMAP support
        7200            # Rotation rate in RPM
        Not_Zoned       # Zone Mode

I/O command overhead:
        time to read 10MB block      0.067031 sec       =    0.003 msec/sector
        time to read 20480 sectors   2.619989 sec       =    0.128 msec/sector
        calculated command overhead                     =    0.125 msec/sector

Seek times:
        Full stroke:      250 iter in   5.665880 sec =   22.664 msec
        Half stroke:      250 iter in   4.263047 sec =   17.052 msec
        Quarter stroke:   500 iter in   6.867914 sec =   13.736 msec
        Short forward:    400 iter in   3.057913 sec =    7.645 msec
        Short backward:   400 iter in   1.979287 sec =    4.948 msec
        Seq outer:       2048 iter in   0.169472 sec =    0.083 msec
        Seq inner:       2048 iter in   0.469630 sec =    0.229 msec

Transfer rates:
        outside:       102400 kbytes in   0.478251 sec =   214114 kbytes/sec
        middle:        102400 kbytes in   0.605701 sec =   169060 kbytes/sec
        inside:        102400 kbytes in   1.303909 sec =    78533 kbytes/sec

So now we know how fast a single disk is.

Let’s repeast the same test on the ZFS zvol device.

# diskinfo -ctv /dev/zvol/nas02/iscsi/test
/dev/zvol/nas02/iscsi/test
        512             # sectorsize
        17592186044416  # mediasize in bytes (16T)
        34359738368     # mediasize in sectors
        65536           # stripesize
        0               # stripeoffset
        Yes             # TRIM/UNMAP support
        Unknown         # Rotation rate in RPM

I/O command overhead:
        time to read 10MB block      0.004512 sec       =    0.000 msec/sector
        time to read 20480 sectors   0.196824 sec       =    0.010 msec/sector
        calculated command overhead                     =    0.009 msec/sector

Seek times:
        Full stroke:      250 iter in   0.006151 sec =    0.025 msec
        Half stroke:      250 iter in   0.008228 sec =    0.033 msec
        Quarter stroke:   500 iter in   0.014062 sec =    0.028 msec
        Short forward:    400 iter in   0.010564 sec =    0.026 msec
        Short backward:   400 iter in   0.011725 sec =    0.029 msec
        Seq outer:       2048 iter in   0.028198 sec =    0.014 msec
        Seq inner:       2048 iter in   0.028416 sec =    0.014 msec

Transfer rates:
        outside:       102400 kbytes in   0.036938 sec =  2772213 kbytes/sec
        middle:        102400 kbytes in   0.043076 sec =  2377194 kbytes/sec
        inside:        102400 kbytes in   0.034260 sec =  2988908 kbytes/sec

Almost 3 GB/s – not bad.

Time for even more oldschool test – the immortal dd(8) command.

This is with compression=off setting.

One process.

# dd if=/dev/zero of=FILE bs=128m status=progress
26172456960 bytes (26 GB, 24 GiB) transferred 16.074s, 1628 MB/s
202+0 records in
201+0 records out
26977763328 bytes transferred in 16.660884 secs (1619227644 bytes/sec)

Four concurrent processes.

# dd if=/dev/zero of=FILE${X} bs=128m status=progress
80933289984 bytes (81 GB, 75 GiB) transferred 98.081s, 825 MB/s
608+0 records in
608+0 records out
81604378624 bytes transferred in 98.990579 secs (824365101 bytes/sec)

Eight concurrent processes.

# dd if=/dev/zero of=FILE${X} bs=128m status=progress
174214610944 bytes (174 GB, 162 GiB) transferred 385.042s, 452 MB/s
1302+0 records in
1301+0 records out
174617264128 bytes transferred in 385.379296 secs (453104943 bytes/sec)

Lets summarize that data.

1 STREAM(s) ~ 1600 MB/s ~ 1.5 GB/s
4 STREAM(s) ~ 3300 MB/s ~ 3.2 GB/s
8 STREAM(s) ~ 3600 MB/s ~ 3.5 GB/s

So the disk subsystem is able to squeeze 3.5 GB/s of sustained speed in sequential writes. That us that if we would want to saturate it we would need to add additional two 10GE interfaces.

The disks were stressed only to about 55% which you can see in other useful FreeBSD tool – gstat(8) command.

n10.png

Time for more ‘intelligent’ tests. The blogbench test.

First with compression disabled.

# time blogbench -d .
Frequency = 10 secs
Scratch dir = [.]
Spawning 3 writers...
Spawning 1 rewriters...
Spawning 5 commenters...
Spawning 100 readers...
Benchmarking for 30 iterations.
The test will run during 5 minutes.
(...)
Final score for writes:          6476
Final score for reads :        660436

blogbench -d .  280.58s user 4974.41s system 1748% cpu 5:00.54 total

Second with compression set to LZ4.

# time blogbench -d .
Frequency = 10 secs
Scratch dir = [.]
Spawning 3 writers...
Spawning 1 rewriters...
Spawning 5 commenters...
Spawning 100 readers...
Benchmarking for 30 iterations.
The test will run during 5 minutes.
(...)
Final score for writes:          7087
Final score for reads :        733932

blogbench -d .  299.08s user 5415.04s system 1900% cpu 5:00.68 total

Compression did not helped much, but helped.

To have some comparision we will run the same test on the system ZFS pool – two Intel SSD DC S3500 240 GB drives in mirror which have following features.

The Intel SSD DC S3500 240 GB drives:

  • Sequential Read (up to) 500 MB/s
  • Sequential Write (up to) 260 MB/s
  • Random Read (100% Span) 75000 IOPS
  • Random Write (100% Span) 7500 IOPS
# time blogbench -d .
Frequency = 10 secs
Scratch dir = [.]
Spawning 3 writers...
Spawning 1 rewriters...
Spawning 5 commenters...
Spawning 100 readers...
Benchmarking for 30 iterations.
The test will run during 5 minutes.
(...)
Final score for writes:          6109
Final score for reads :        654099

blogbench -d .  278.73s user 5058.75s system 1777% cpu 5:00.30 total

Now the randomio test. Its multithreaded disk I/O microbenchmark.

The usage is as follows.

usage: randomio filename nr_threads write_fraction_of_io fsync_fraction_of_writes io_size nr_seconds_between_samples

filename                    Filename or device to read/write.
write_fraction_of_io        What fraction of I/O should be writes - for example 0.25 for 25% write.
fsync_fraction_of_writes    What fraction of writes should be fsync'd.
io_size                     How many bytes to read/write (multiple of 512 bytes).
nr_seconds_between_samples  How many seconds to average samples over.

The randomio with 4k block.

# zfs create -s -V 1T nas02/iscsi/test
# randomio /dev/zvol/nas02/iscsi/test 8 0.25 1 4096 10
  total |  read:         latency (ms)       |  write:        latency (ms)
   iops |   iops   min    avg    max   sdev |   iops   min    avg    max   sdev
--------+-----------------------------------+----------------------------------
54137.7 |40648.4   0.0    0.1  575.8    2.2 |13489.4   0.0    0.3  405.8    2.6
66248.4 |49641.5   0.0    0.1   19.6    0.3 |16606.9   0.0    0.2   26.4    0.7
66411.0 |49817.2   0.0    0.1   19.7    0.3 |16593.8   0.0    0.2   20.3    0.7
64158.9 |48142.8   0.0    0.1  254.7    0.7 |16016.1   0.0    0.2  130.4    1.0
48454.1 |36390.8   0.0    0.1  542.8    2.7 |12063.3   0.0    0.3  507.5    3.2
66796.1 |50067.4   0.0    0.1   24.1    0.3 |16728.7   0.0    0.2   23.4    0.7
58512.2 |43851.7   0.0    0.1  576.5    1.7 |14660.5   0.0    0.2  307.2    1.7
63195.8 |47341.8   0.0    0.1  261.6    0.9 |15854.1   0.0    0.2  361.1    1.9
67086.0 |50335.6   0.0    0.1   20.4    0.3 |16750.4   0.0    0.2   25.1    0.8
67429.8 |50549.6   0.0    0.1   21.8    0.3 |16880.3   0.0    0.2   20.6    0.7
^C

… and with 512 sector.

# zfs create -s -V 1T nas02/iscsi/test
# randomio /dev/zvol/nas02/iscsi/TEST 8 0.25 1 512 10
  total |  read:         latency (ms)       |  write:        latency (ms)
   iops |   iops   min    avg    max   sdev |   iops   min    avg    max   sdev
--------+-----------------------------------+----------------------------------
58218.9 |43712.0   0.0    0.1  501.5    2.1 |14506.9   0.0    0.2  272.5    1.6
66325.3 |49703.8   0.0    0.1  352.0    0.9 |16621.4   0.0    0.2  352.0    1.5
68130.5 |51100.8   0.0    0.1   24.6    0.3 |17029.7   0.0    0.2   24.4    0.7
68465.3 |51352.3   0.0    0.1   19.9    0.3 |17112.9   0.0    0.2   23.8    0.7
54903.5 |41249.1   0.0    0.1  399.3    1.9 |13654.4   0.0    0.3  335.8    2.2
61259.8 |45898.7   0.0    0.1  574.6    1.7 |15361.0   0.0    0.2  371.5    1.7
68483.3 |51313.1   0.0    0.1   22.9    0.3 |17170.3   0.0    0.2   26.1    0.7
56713.7 |42524.7   0.0    0.1  373.5    1.8 |14189.1   0.0    0.2  438.5    2.7
68861.4 |51657.0   0.0    0.1   21.0    0.3 |17204.3   0.0    0.2   21.7    0.7
68602.0 |51438.4   0.0    0.1   19.5    0.3 |17163.7   0.0    0.2   23.7    0.7
^C

Both randomio tests were run with compression set to LZ4.

Next is bonnie++ benchmark. It has been run with compression set to LZ4.

# bonnie++ -d . -u root
Using uid:0, gid:0.
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  1.97       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
nas02.local 261368M   139  99 775132  99 589190  99   383  99 1638929  99 12930 2046
Latency             60266us    7030us    7059us   21553us    3844us    5710us
Version  1.97       ------Sequential Create------ --------Random Create--------
nas02.local         -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ 12680  44 +++++ +++ +++++ +++ 30049  99
Latency              2619us      43us     714ms    2748us      28us      58us

… and last but not least the fio benchmark. Also with LZ4 compression enabled.

# fio --randrepeat=1 --direct=1 --gtod_reduce=1 --name=test --filename=random_read_write.fio --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75
test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=64
fio-3.13
Starting 1 process
Jobs: 1 (f=1): [m(1)][98.0%][r=38.0MiB/s,w=12.2MiB/s][r=9735,w=3128 IOPS][eta 00m:05s]
test: (groupid=0, jobs=1): err= 0: pid=35368: Tue Jun 18 15:14:44 2019
  read: IOPS=3157, BW=12.3MiB/s (12.9MB/s)(3070MiB/248872msec)
   bw (  KiB/s): min= 9404, max=57732, per=98.72%, avg=12469.84, stdev=3082.99, samples=497
   iops        : min= 2351, max=14433, avg=3117.15, stdev=770.74, samples=497
  write: IOPS=1055, BW=4222KiB/s (4323kB/s)(1026MiB/248872msec)
   bw (  KiB/s): min= 3179, max=18914, per=98.71%, avg=4166.60, stdev=999.23, samples=497
   iops        : min=  794, max= 4728, avg=1041.25, stdev=249.76, samples=497
  cpu          : usr=1.11%, sys=88.64%, ctx=677981, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=785920,262656,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=12.3MiB/s (12.9MB/s), 12.3MiB/s-12.3MiB/s (12.9MB/s-12.9MB/s), io=3070MiB (3219MB), run=248872-248872msec
  WRITE: bw=4222KiB/s (4323kB/s), 4222KiB/s-4222KiB/s (4323kB/s-4323kB/s), io=1026MiB (1076MB), run=248872-248872msec

Dunno how about you but I am satisfied with performance ๐Ÿ™‚

FreeNAS

Originally I really wanted to use FreeNAS on these boxes and I even installed FreeNAS on them. It run nicely but … the security part of FreeNAS was not best.

This is the output of pkg audit command. Quite scarry.

root@freenas[~]# pkg audit -F
Fetching vuln.xml.bz2: 100%  785 KiB 804.3kB/s    00:01
python27-2.7.15 is vulnerable:
Python -- NULL pointer dereference vulnerability
CVE: CVE-2019-5010
WWW: https://vuxml.FreeBSD.org/freebsd/d74371d2-4fee-11e9-a5cd-1df8a848de3d.html

curl-7.62.0 is vulnerable:
curl -- multiple vulnerabilities
CVE: CVE-2019-3823
CVE: CVE-2019-3822
CVE: CVE-2018-16890
WWW: https://vuxml.FreeBSD.org/freebsd/714b033a-2b09-11e9-8bc3-610fd6e6cd05.html

libgcrypt-1.8.2 is vulnerable:
libgcrypt -- side-channel attack vulnerability
CVE: CVE-2018-0495
WWW: https://vuxml.FreeBSD.org/freebsd/9b5162de-6f39-11e8-818e-e8e0b747a45a.html

python36-3.6.5_1 is vulnerable:
Python -- NULL pointer dereference vulnerability
CVE: CVE-2019-5010
WWW: https://vuxml.FreeBSD.org/freebsd/d74371d2-4fee-11e9-a5cd-1df8a848de3d.html

pango-1.42.0 is vulnerable:
pango -- remote DoS vulnerability
CVE: CVE-2018-15120
WWW: https://vuxml.FreeBSD.org/freebsd/5a757a31-f98e-4bd4-8a85-f1c0f3409769.html

py36-requests-2.18.4 is vulnerable:
www/py-requests -- Information disclosure vulnerability
WWW: https://vuxml.FreeBSD.org/freebsd/50ad9a9a-1e28-11e9-98d7-0050562a4d7b.html

libnghttp2-1.31.0 is vulnerable:
nghttp2 -- Denial of service due to NULL pointer dereference
CVE: CVE-2018-1000168
WWW: https://vuxml.FreeBSD.org/freebsd/1fccb25e-8451-438c-a2b9-6a021e4d7a31.html

gnupg-2.2.6 is vulnerable:
gnupg -- unsanitized output (CVE-2018-12020)
CVE: CVE-2017-7526
CVE: CVE-2018-12020
WWW: https://vuxml.FreeBSD.org/freebsd/7da0417f-6b24-11e8-84cc-002590acae31.html

py36-cryptography-2.1.4 is vulnerable:
py-cryptography -- tag forgery vulnerability
CVE: CVE-2018-10903
WWW: https://vuxml.FreeBSD.org/freebsd/9e2d0dcf-9926-11e8-a92d-0050562a4d7b.html

perl5-5.26.1 is vulnerable:
perl -- multiple vulnerabilities
CVE: CVE-2018-6913
CVE: CVE-2018-6798
CVE: CVE-2018-6797
WWW: https://vuxml.FreeBSD.org/freebsd/41c96ffd-29a6-4dcc-9a88-65f5038fa6eb.html

libssh2-1.8.0,3 is vulnerable:
libssh2 -- multiple issues
CVE: CVE-2019-3862
CVE: CVE-2019-3861
CVE: CVE-2019-3860
CVE: CVE-2019-3858
WWW: https://vuxml.FreeBSD.org/freebsd/6e58e1e9-2636-413e-9f84-4c0e21143628.html

git-lite-2.17.0 is vulnerable:
Git -- Fix memory out-of-bounds and remote code execution vulnerabilities (CVE-2018-11233 and CVE-2018-11235)
CVE: CVE-2018-11235
CVE: CVE-2018-11233
WWW: https://vuxml.FreeBSD.org/freebsd/c7a135f4-66a4-11e8-9e63-3085a9a47796.html

gnutls-3.5.18 is vulnerable:
GnuTLS -- double free, invalid pointer access
CVE: CVE-2019-3836
CVE: CVE-2019-3829
WWW: https://vuxml.FreeBSD.org/freebsd/fb30db8f-62af-11e9-b0de-001cc0382b2f.html

13 problem(s) in the installed packages found.

root@freenas[~]# uname -a
FreeBSD freenas.local 11.2-STABLE FreeBSD 11.2-STABLE #0 r325575+95cc58ca2a0(HEAD): Mon May  6 19:08:58 EDT 2019     root@mp20.tn.ixsystems.com:/freenas-releng/freenas/_BE/objs/freenas-releng/freenas/_BE/os/sys/FreeNAS.amd64  amd64

root@freenas[~]# freebsd-version -uk
11.2-STABLE
11.2-STABLE

root@freenas[~]# sockstat -l4
USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS
root     uwsgi-3.6  4006  3  tcp4   127.0.0.1:9042        *:*
root     uwsgi-3.6  3188  3  tcp4   127.0.0.1:9042        *:*
nobody   mdnsd      3144  4  udp4   *:31417               *:*
nobody   mdnsd      3144  6  udp4   *:5353                *:*
www      nginx      3132  6  tcp4   *:443                 *:*
www      nginx      3132  8  tcp4   *:80                  *:*
root     nginx      3131  6  tcp4   *:443                 *:*
root     nginx      3131  8  tcp4   *:80                  *:*
root     ntpd       2823  21 udp4   *:123                 *:*
root     ntpd       2823  22 udp4   10.49.13.99:123       *:*
root     ntpd       2823  25 udp4   127.0.0.1:123         *:*
root     sshd       2743  5  tcp4   *:22                  *:*
root     syslog-ng  2341  19 udp4   *:1031                *:*
nobody   mdnsd      2134  3  udp4   *:39020               *:*
nobody   mdnsd      2134  5  udp4   *:5353                *:*
root     python3.6  236   22 tcp4   *:6000                *:*


I even tried to get explanation why FreeNAS has such outdated and insecure packages in their latest version – FreeNAS 11.2-U3 Vulnerabilities – a thread I started on their forums.

Unfortunatelly its their policy which you can summarize as ‘do not touch/change versions if its working’ – at last I got this implression.

Because if these security holes I can not recommend the use of FreeNAS and I movedto original – the FreeBSD system.

One other interesting note. After I installed FreeBSD I wanted to import the ZFS pool created by FreeNAS. This is what I got after executing the zpool import command.

# zpool import
   pool: nas02_gr06
     id: 1275660523517109367
  state: ONLINE
 status: The pool was last accessed by another system.
 action: The pool can be imported using its name or numeric identifier and
        the '-f' flag.
   see: http://illumos.org/msg/ZFS-8000-EY
 config:

        nas02_gr06  ONLINE
          raidz2-0  ONLINE
            da58p2  ONLINE
            da59p2  ONLINE
            da60p2  ONLINE
            da61p2  ONLINE
            da62p2  ONLINE
            da63p2  ONLINE
            da64p2  ONLINE
            da26p2  ONLINE
            da65p2  ONLINE
            da23p2  ONLINE
            da29p2  ONLINE
            da66p2  ONLINE
            da67p2  ONLINE
            da68p2  ONLINE
        spares
          da69p2

   pool: nas02_gr05
     id: 5642709896812665361
  state: ONLINE
 status: The pool was last accessed by another system.
 action: The pool can be imported using its name or numeric identifier and
        the '-f' flag.
   see: http://illumos.org/msg/ZFS-8000-EY
 config:

        nas02_gr05  ONLINE
          raidz2-0  ONLINE
            da20p2  ONLINE
            da30p2  ONLINE
            da34p2  ONLINE
            da50p2  ONLINE
            da28p2  ONLINE
            da38p2  ONLINE
            da51p2  ONLINE
            da52p2  ONLINE
            da27p2  ONLINE
            da32p2  ONLINE
            da53p2  ONLINE
            da54p2  ONLINE
            da55p2  ONLINE
            da56p2  ONLINE
        spares
          da57p2

   pool: nas02_gr04
     id: 2460983830075205166
  state: ONLINE
 status: The pool was last accessed by another system.
 action: The pool can be imported using its name or numeric identifier and
        the '-f' flag.
   see: http://illumos.org/msg/ZFS-8000-EY
 config:

        nas02_gr04  ONLINE
          raidz2-0  ONLINE
            da44p2  ONLINE
            da37p2  ONLINE
            da18p2  ONLINE
            da36p2  ONLINE
            da45p2  ONLINE
            da19p2  ONLINE
            da22p2  ONLINE
            da33p2  ONLINE
            da35p2  ONLINE
            da21p2  ONLINE
            da31p2  ONLINE
            da47p2  ONLINE
            da48p2  ONLINE
            da49p2  ONLINE
        spares
          da46p2

   pool: nas02_gr03
     id: 4878868173820164207
  state: ONLINE
 status: The pool was last accessed by another system.
 action: The pool can be imported using its name or numeric identifier and
        the '-f' flag.
   see: http://illumos.org/msg/ZFS-8000-EY
 config:

        nas02_gr03  ONLINE
          raidz2-0  ONLINE
            da81p2  ONLINE
            da71p2  ONLINE
            da14p2  ONLINE
            da15p2  ONLINE
            da80p2  ONLINE
            da16p2  ONLINE
            da88p2  ONLINE
            da17p2  ONLINE
            da40p2  ONLINE
            da41p2  ONLINE
            da25p2  ONLINE
            da42p2  ONLINE
            da24p2  ONLINE
            da43p2  ONLINE
        spares
          da39p2

   pool: nas02_gr02
     id: 3299037437134217744
  state: ONLINE
 status: The pool was last accessed by another system.
 action: The pool can be imported using its name or numeric identifier and
        the '-f' flag.
   see: http://illumos.org/msg/ZFS-8000-EY
 config:

        nas02_gr02  ONLINE
          raidz2-0  ONLINE
            da84p2  ONLINE
            da76p2  ONLINE
            da85p2  ONLINE
            da8p2   ONLINE
            da9p2   ONLINE
            da78p2  ONLINE
            da73p2  ONLINE
            da74p2  ONLINE
            da70p2  ONLINE
            da77p2  ONLINE
            da11p2  ONLINE
            da13p2  ONLINE
            da79p2  ONLINE
            da89p2  ONLINE
        spares
          da90p2

   pool: nas02_gr01
     id: 1132383125952900182
  state: ONLINE
 status: The pool was last accessed by another system.
 action: The pool can be imported using its name or numeric identifier and
        the '-f' flag.
   see: http://illumos.org/msg/ZFS-8000-EY
 config:

        nas02_gr01  ONLINE
          raidz2-0  ONLINE
            da91p2  ONLINE
            da75p2  ONLINE
            da0p2   ONLINE
            da82p2  ONLINE
            da1p2   ONLINE
            da83p2  ONLINE
            da2p2   ONLINE
            da3p2   ONLINE
            da4p2   ONLINE
            da5p2   ONLINE
            da86p2  ONLINE
            da6p2   ONLINE
            da7p2   ONLINE
            da72p2  ONLINE
        spares
          da87p2



It seems that FreeNAS does ZFS little differently and they create a separate pool for every RAIDZ2 target with dedicated spares. Interesting …

UPDATE 1 – BSD Now 305

The FreeBSD Enterprise 1 PB Storage article was featured in the BSD Now 305 – Changing Face of Unix episode.

Thanks for mentioning!

UPDATE 2 – Real Life Pictures in Data Center

Some of you asked for a real life pictures of this monster. Below you will find several pics taken at the data center.

Front case with cabling.

tyan-real-01.jpg

Alternate front view.

tyan-real-09.jpg

Back of the case with cabling.

tyan-real-02.jpg

Top view with disks.

tyan-real-03

Alternate top view.

tyan-real-07.jpg

Disks slots zoom.

tyan-real-08.jpg

SSD and HDD disks.

tyan-real-06.jpg

EOF

Silent Fanless FreeBSD Server – Redundant Backup

I brought up this topic in the past. It was in the form of more theoretical Silent Fanless FreeBSD Desktop/Server post and more hands-on Silent Fanless FreeBSD Server – DIY Backup article.

One of the comments after the latter was that I compared non-redundant backup solution (single disk) to redundant backup in the cloud. Today – as this is my main backup system – I would like to show you redundant backup solution with two disks in ZFS mirror along with real power usage measurements. This time I got ASRock J3355B-ITX motherboard with only 10W TDP which includes 2-core Celeron J3355 2.0-2.5 GHz CPU and small shiny REALAN H80 Mini ITX case. It looks very nice and comes from AliExpress at very low $33 price for new unit along with free shipping.

Build

Here is how the REALAN H80 case looks like.

realan-H80-render

The ASRock J3355B-ITX motherboard.

asrock-J3355B-ITX.jpg

Same as with the earlier build the internal Seagate BarraCuda 5TB 2.5 SATA drives costs about $200. The same Seagate Backup Plus 5TB 2.5 disk in external case with USB 3.0 port costs nearly half of that price – only $120 – at least in the Europe/Poland location. I took the decision to buy external ones and rip off their cases. That saved me about $160.

Here is the simple performance benchmark of these 2.5 disks.

% which pv
pv: aliased to pv -t -r -a -b -W -B 1048576

% pv  /dev/null
1.35GiB 0:00:10 [ 137MiB/s] [ 137MiB/s]
^C

% dd  /dev/null bs=8M
127+0 records in
127+0 records out
1065353216 bytes transferred in 7.494081 secs (142159287 bytes/sec)
^C

About 135MB/s per disk.

The ripped of parts of Seagate Backup Plus USB cases.

external-case-parts.jpg

What made me laugh was that as I got different cases colors (silver and gray) the disks inside also had different colors (green and blue) :>

disks-bottom

… but their part number is the same, here they are mounted on a REALAN H80 disks holder.

disks-mounted

For the record – several REALAN H80 case real shots (not renders). First its front.

realan-H80-front

Back.

realan-H80-back.jpg

Side with USB port.

realan-H80-side-usb

Bottom.

realan-H80-bottom.jpg

Top.

realan-H80-top

Case parts.

realan-H80-parts.jpg

Generally the REALAN H80 looks really nice. Little lower REALAN H60 (without COM slots/holes in the back) looks even better but I wanted to make sure that I will have room and space for hot air in that case – as space was not a problem for me.

Cost

The complete price tops at $220 total. Here are the parts used.

PRICE  COMPONENT
  $49  CPU/Motherboard ASRock J3355B-ITX Mini-ITX
  $10  RAM 4GB DDR3
  $13  PSU 12V 7.5A 90W Pico (internal)
   $2  PSU 12V 2.5A 30W Leader Electronics (external)
  $33  Supermicro SC101i
   $3  SanDisk Fit 16GB USB 2.0 Drive (system)
 $120  Seagate 5TB 2.5 drive (ONE)
 $120  Seagate 5TB 2.5 drive (TWO)
 $350  TOTAL

That is $110 for the ‘system’ and additional $240 for ‘data’ drives.

Today I would probably get the ASRock N3150DC-ITX or Gigabyte GA-N3160TN motherboard instead because of builtin DC jack slot (compatible with 19V power adapter) on its back. This will eliminate the need for additional internal Pico PSU power supply …

The ASRock N3150DC-ITX with builtin DC jack.

asrock-N3150DC-ITX.jpg

The Gigabyte GA-N3160TN with builtin DC jack.

gigabyte-GA-N3160TN.jpg

The Gigabyte GA-N3160TN is also very low profile motherboard as you can see from the back.

gigabyte-GA-N3160TN-back-other.jpg

It may be good idea to use this one instead ASRock N3150DC-ITX to get more space above the motherboard.

ย 

PSU

As in the earlier Silent Fanless FreeBSD Server – DIY Backup article I used small 12V 2.5A 30W compact and cheap external PSU instead of the large 90W PSU from FSP Group. As these low power motherboard does not need a lot of power.

New Leader Electronics PSU label.

silent-backup-psu-ext-label.jpg

The internal power supply is Pico PSU which now tops as 12V 7.5A 90W power.

silent-backup-psu-pico-12V-90W.jpg

Power Consumption

I also measured the power consumption with power meter.

silent-backup-power-meter.jpg

The whole box with two Seagate BarraCuda 5TB 2.5 drives for data on ZFS mirror and SanDisk 16GB USB 2.0 system drive used about 10.4W in idle state.

I used all needed settings from my earlier The Power to Serve โ€“ FreeBSD Power Management article with CPU speed limited between 0.4GHz and 1.2GHz.

The powerd(8) settings in the /etc/rc.conf file are below.

powerd_flags="-n hiadaptive -a hiadaptive -b hiadaptive -m 400 -M 1200"

I used python(1) [1] to load the CPU and dd(8) to load the drives. I used dd(8) on the ZFS pool so 1 disk thread will read [2] and write [3] from/to both 2.5 disks. I temporary disabled LZ4 compression for the write tests.

[1] # echo '999999999999999999 ** 999999999999999999' | python
[2] # dd  /dev/null bs=1M
[3] # dd > /data/FILE < /dev/zero bs=1M
POWER   CPU LOAD         I/O LOAD
10.4 W  IDLE             IDLE
12.9 W  IDLE             1 DISK READ Thread(s)
14.3 W  IDLE             1 DISK READ Thread(s) + 1 DISK WRITE Thread(s)
17.2 W  IDLE             3 DISK READ Thread(s) + 3 DISK WRITE Thread(s)
11.0 W  8 CPU Thread(s)  IDLE
13.4 W  8 CPU Thread(s)  1 DISK READ Thread(s)
15.0 W  8 CPU Thread(s)  1 DISK READ Thread(s) + 1 DISK WRITE Thread(s)
17.8 W  8 CPU Thread(s)  3 DISK READ Thread(s) + 3 DISK WRITE Thread(s)

That’s not much remembering that 6W TDP power motherboard ASRock N3150B-ITX with just single Maxtor M3 4TB 2.5 USB 3.0 drive used 16.0W with CPU and I/O loaded. Only 1.8W more (on loaded system) with redundancy on two 2.5 disks.

Commands

The crypto FreeBSD kernel module was able to squeeze about 68MB/s of random data from /dev/random as this CPU has built in hardware AES-NI acceleration. Note to Linux users – the /dev/random and /dev/urandom are the same thing on FreeBSD. I used both dd(8) and pv(1) commands for this simple test. I made two tests with powerd(8) enabled and disabled to check the difference between CPU speed at 1.2GHz and at 2.5GHz with Turbo mode.

Full speed with Turbo enabled (note 2001 instead of 2000 for CPU frequency)..

# /etc/rc.d/powerd stop
Stopping powerd.
Waiting for PIDS: 1486.

% sysctl dev.cpu.0.freq
dev.cpu.0.freq: 2001

% which pv
pv: aliased to pv -t -r -a -b -W -B 1048576

% dd  /dev/null
1.91GiB 0:00:31 [68.7MiB/s] [68.1MiB/s]
265+0 records in
265+0 records out
2222981120 bytes transferred in 33.566154 secs (70226864 bytes/sec)
^C

CPU limited to 1.2GHz with powerd(8) daemon was able to squeeze about 24MB/s.

# service powerd start
Starting powerd.

% which pv
pv: aliased to pv -t -r -a -b -W -B 1048576

% dd  /dev/null
568MiB 0:00:23 [25.3MiB/s] [24.7MiB/s]
71+0 records in
71+0 records out
595591168 bytes transferred in 23.375588 secs (25479195 bytes/sec
^C

Below I will show you the data from dmesg(8) about the used USB and 2.5 drives.

The dmesg(8) information for the SanDisk Fit USB 2.0 16GB drive.

# grep da0 /var/run/dmesg.boot
da0 at umass-sim1 bus 1 scbus3 target 0 lun 0
da0:  Removable Direct Access SPC-4 SCSI device
da0: Serial Number 4C530002030502100093
da0: 400.000MB/s transfers
da0: 14663MB (30031250 512 byte sectors)
da0: quirks=0x2

… and two Seagate BarraCuda 5TB 2.5 drives.

# grep ada /var/run/dmesg.boot
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0:  ACS-3 ATA SATA 3.x device
ada0: Serial Number WCJ0DRJE
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 4769307MB (9767541168 512 byte sectors)
ada1 at ahcich1 bus 0 scbus1 target 0 lun 0
ada1:  ACS-3 ATA SATA 3.x device
ada1: Serial Number WCJ0213S
ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 4769307MB (9767541168 512 byte sectors)

The whole /var/run/dmesg.boot content (without disks) is shown below.

# cat /var/run/dmesg.boot
Copyright (c) 1992-2018 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.2-RELEASE-p7 #0: Tue Dec 18 08:29:33 UTC 2018
    root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
FreeBSD clang version 6.0.0 (tags/RELEASE_600/final 326565) (based on LLVM 6.0.0)
VT(vga): resolution 640x480
CPU: Intel(R) Celeron(R) CPU J3355 @ 2.00GHz (1996.88-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x506c9  Family=0x6  Model=0x5c  Stepping=9
  Features=0xbfebfbff
  Features2=0x4ff8ebbf
  AMD Features=0x2c100800
  AMD Features2=0x101
  Structured Extended Features=0x2294e283
  XSAVE Features=0xf
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
  TSC: P-state invariant, performance statistics
real memory  = 4294967296 (4096 MB)
avail memory = 3700518912 (3529 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: 
WARNING: L1 data cache covers less APIC IDs than a core
0 < 1
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s)
ioapic0  irqs 0-119 on motherboard
SMP: AP CPU #1 Launched!
Timecounter "TSC" frequency 1996877678 Hz quality 1000
random: entropy device external interface
kbd1 at kbdmux0
netmap: loaded module
module_register_init: MOD_LOAD (vesa, 0xffffffff80ff4580, 0) error 19
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
nexus0
vtvga0:  on motherboard
cryptosoft0:  on motherboard
acpi0:  on motherboard
unknown: I/O range not supported
cpu0:  on acpi0
cpu1:  on acpi0
attimer0:  port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
atrtc0:  port 0x70-0x77 on acpi0
atrtc0: Warning: Couldn't map I/O.
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
hpet0:  iomem 0xfed00000-0xfed003ff irq 8 on acpi0
Timecounter "HPET" frequency 19200000 Hz quality 950
Event timer "HPET" frequency 19200000 Hz quality 550
Event timer "HPET1" frequency 19200000 Hz quality 440
Event timer "HPET2" frequency 19200000 Hz quality 440
Event timer "HPET3" frequency 19200000 Hz quality 440
Event timer "HPET4" frequency 19200000 Hz quality 440
Event timer "HPET5" frequency 19200000 Hz quality 440
Event timer "HPET6" frequency 19200000 Hz quality 440
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0:  port 0x408-0x40b on acpi0
pcib0:  port 0xcf8-0xcff on acpi0
pci0:  on pcib0
vgapci0:  port 0xf000-0xf03f mem 0x90000000-0x90ffffff,0x80000000-0x8fffffff irq 19 at device 2.0 on pci0
vgapci0: Boot video device
hdac0:  mem 0x91210000-0x91213fff,0x91000000-0x910fffff irq 25 at device 14.0 on pci0
pci0:  at device 15.0 (no driver attached)
ahci0:  port 0xf090-0xf097,0xf080-0xf083,0xf060-0xf07f mem 0x91214000-0x91215fff,0x91218000-0x912180ff,0x91217000-0x912177ff irq 19 at device 18.0 on pci0
ahci0: AHCI v1.31 with 2 6Gbps ports, Port Multiplier supported
ahcich0:  at channel 0 on ahci0
ahcich1:  at channel 1 on ahci0
pcib1:  irq 22 at device 19.0 on pci0
pci1:  on pcib1
pcib2:  irq 20 at device 19.2 on pci0
pci2:  on pcib2
re0:  port 0xe000-0xe0ff mem 0x91104000-0x91104fff,0x91100000-0x91103fff irq 20 at device 0.0 on pci2
re0: Using 1 MSI-X message
re0: Chip rev. 0x4c000000
re0: MAC rev. 0x00000000
miibus0:  on re0
rgephy0:  PHY 1 on miibus0
rgephy0:  none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
re0: Using defaults for TSO: 65518/35/2048
re0: Ethernet address: 70:85:c2:3f:53:41
re0: netmap queues/slots: TX 1/256, RX 1/256
xhci0:  mem 0x91200000-0x9120ffff irq 17 at device 21.0 on pci0
xhci0: 32 bytes context size, 64-bit DMA
usbus0 on xhci0
usbus0: 5.0Gbps Super Speed USB v3.0
isab0:  at device 31.0 on pci0
isa0:  on isab0
acpi_button0:  on acpi0
acpi_tz0:  on acpi0
atkbdc0:  at port 0x60,0x64 on isa0
atkbd0:  irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
ppc0: cannot reserve I/O port range
est0:  on cpu0
est1:  on cpu1
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Timecounters tick every 1.000 msec
hdacc0:  at cad 0 on hdac0
hdaa0:  at nid 1 on hdacc0
ugen0.1:  at usbus0
uhub0:  on usbus0
pcm0:  at nid 21 and 24,26 on hdaa0
pcm1:  at nid 20 and 25 on hdaa0
pcm2:  at nid 27 on hdaa0
hdacc1:  at cad 2 on hdac0
hdaa1:  at nid 1 on hdacc1
pcm3:  at nid 3 on hdaa1
uhub0: 15 ports with 15 removable, self powered
ugen0.2:  at usbus0
uhub1 on uhub0
uhub1:  on usbus0
uhub1: 4 ports with 4 removable, self powered
Trying to mount root from zfs:zroot/ROOT/default []...
random: unblocking device.
re0: link state changed to DOWN

ZFS Pool Configuration

To get higher LZ4 compression ratio I use larger blocksize (1MB) on this ZFS mirror pool. Here is the ZFS pool status.

% zpool status data
  pool: data
 state: ONLINE
  scan: scrub repaired 0 in 44h14m with 0 errors on Mon Feb 11 07:13:42 2019
config:

        NAME                STATE     READ WRITE CKSUM
        data                ONLINE       0     0     0
          mirror-0          ONLINE       0     0     0
            label/WCJ0213S  ONLINE       0     0     0
            label/WCJ0DRJE  ONLINE       0     0     0

errors: No known data errors

I get 4% compression (1.04x) on that ZFS pool. Its about 80% filled with lots of movies and photos so while such compression ratio may not be great it gives a lot of space. For example 4% of 4TB of data is about 160GB of ‘free’ space.

% zfs get compressratio data
NAME                                    PROPERTY       VALUE  SOURCE
data                                    compressratio  1.04x  -

Here is the ZFS pool configuration.

# zpool history
History for 'data':
2018-11-12.01:18:33 zpool create data mirror /dev/label/WCJ0229Z /dev/label/WCJ0DPHF
2018-11-12.01:19:11 zfs set mountpoint=none data
2018-11-12.01:19:16 zfs set compression=lz4 data
2018-11-12.01:19:21 zfs set atime=off data
2018-11-12.01:19:34 zfs set primarycache=metadata data
2018-11-12.01:19:40 zfs set secondarycache=metadata data
2018-11-12.01:19:45 zfs set redundant_metadata=most data
2018-11-12.01:19:51 zfs set recordsize=1m data
(...)

We do not need redundant_metadata as we already have two disks, its useful only on single disks configurations.

Self Solution Cost

As in the earlier post I will again calculate how much energy this server would consume. Currently 1kWh of power costs about $0.20 in Europe/Poland (rounded up). This means that running computer with 1000W power usage for 1 hour would cost you $0.20 on electricity bill. This system uses 10.4W idle and 12.9W when single disk read occurs. For most of the time server will be idle so I assume 11.0W average for the pricing purposes.

That would cost us $0.0022 for 11.0W device running for 1 hour.

Below you will also find calculations for 1 day (24x multiplier), 1 year (another 365.25x multiplier) and 3 years (another 3x multiplier).

   COST  TIME
$0.0022  1 HOUR(S)
$0.0528  1 DAY(S)
$19.285  1 YEAR(S)
$57.856  3 YEAR(S)
$96.426  5 YEAR(S)

Combining that with server cost ($350) we get TCO for our self hosted 5TB storage service.

   COST  TIME
$369.29  1 YEAR(S)
$407.86  3 YEAR(S)
$446.43  5 YEAR(S)

Our total 3 years TCO is $407.86 and 5 years is $446.43. Its for running system non-stop. We can also implement features like Wake On LAN to limit that power usage even more.

Cloud Storage Prices

This time after searching for cheapest cloud based storage I found these services.

  • Amazon Drive
  • Amazon S3 Glacier Storage
  • Backblaze B2 Cloud Storage
  • Google One

Here is its cost summarized for 1 year period for 5TB of data.

PRICE  TIME       SERVICE
 $300  1 YEAR(S)  Amazon Drive
 $310  1 YEAR(S)  Google One
 $240  1 YEAR(S)  Amazon S3 Glacier Storage
 $450  1 YEAR(S)  Backblaze B2 Cloud Storage

For the Backblaze B2 Cloud Storage I assumed average between upload/download price because upload is two times cheaper then download.

Here is its cost summarized for 3 year period for 5TB of data.

PRICE  TIME       SERVICE
 $900  3 YEAR(S)  Amazon Drive
 $930  3 YEAR(S)  Google One
 $720  3 YEAR(S)  Amazon S3 Glacier Storage
$1350  3 YEAR(S)  Backblaze B2 Cloud Storage

Here is its cost summarized for 5 year period for 5TB of data.

PRICE  TIME       SERVICE
$1500  5 YEAR(S)  Amazon Drive
$1550  5 YEAR(S)  Google One
$1200  5 YEAR(S)  Amazon S3 Glacier Storage
$2250  5 YEAR(S)  Backblaze B2 Cloud Storage

Now lets compare costs of our own server to various cloud services.

If we would run our server for just 1 year the price will be similar.

PRICE  TIME       SERVICE
 $369  1 YEAR(S)  Self Build NAS
 $300  1 YEAR(S)  Amazon Drive
 $310  1 YEAR(S)  Google One
 $240  1 YEAR(S)  Amazon S3 Glacier Storage
 $450  1 YEAR(S)  Backblaze B2 Cloud Storage

It gets interesting when we compare 3 years costs. Its two times cheaper to self host our own server then use cloud services. One may argue that clouds are located in many places but even if we would buy two such boxes and put one – for example in our friends place at Jamaica – or other parts of the world.

PRICE  TIME       SERVICE
 $408  3 YEAR(S)  Self Build NAS
 $528  3 YEAR(S)  Self Build NAS (assuming one of the drives failed)
 $900  3 YEAR(S)  Amazon Drive
 $930  3 YEAR(S)  Google One
 $720  3 YEAR(S)  Amazon S3 Glacier Storage
$1350  3 YEAR(S)  Backblaze B2 Cloud Storage

… but with 5 years using cloud service instead of self hosted NAS solution is 3-5 times more expensive … and these were the cheapest cloud services I was able to find. I do not even want to know how much would it cos on Dropbox for example ๐Ÿ™‚

PRICE  TIME       SERVICE
 $447  5 YEAR(S)  Self Build NAS
 $567  5 YEAR(S)  Self Build NAS (assuming one of the drives failed)
$1500  5 YEAR(S)  Amazon Drive
$1550  5 YEAR(S)  Google One
$1200  5 YEAR(S)  Amazon S3 Glacier Storage
$2250  5 YEAR(S)  Backblaze B2 Cloud Storage

… and ‘anywhere’ access is not an argument for cloud services because you can get external IP address for you NAS or use Dynamic DNS – for free. You may also wonder why I compare such ‘full featured NAS’ with S3 storage … well with rclone (rsync for cloud storage) you are able to synchronize your files with almost anything ๐Ÿ™‚

Not to mention how much more privacy you have with keeping all your data to yourself … but that is priceless.

You can also setup a lot more services on such hardware – like FreeNAS with Bhyve/Jails virtualization … or Nextcloud instance … or Syncthing … while cloud storage is only that – a storage in the cloud.

Summary

Not sure what else could I include in this article. If you have an idea what else could I cover then let me know.

EOF

ย 

Less Known pkg(8) Features

I was asked many times to write an article about pkg(8) – the current FreeBSD modern package manager sometimes also called PKGng.

In this entry I will try to describe less known pkg(8) features.

About 8 years ago – when pkg(8) did not even existed – I wrote HOWTO: keeping FreeBSD’s base system and packages up-to-date post. It was even later published in the BSD Magazine 2012/01 episode (Issue 30).

Back in 2011 keeping packages up to date was little more tricky then it is now. You was forced to use the FreeBSD’s STABLE branch for them as packages in RELEASE were never updated – like currently it is in the OpenBSD world. The packages in FreeBSD’s STABLE branch were built every 2 weeks which was enough at that time.

You could of course compile everything from FreeBSD Ports using portmaster but you will waste lots of time for compiling your life. When pkg_add/pkg_delete/pkg_info were THE package tools on FreeBSD the pkg_upgrade script from the bsdadminscripts package was quite helpful with the upgrade process. It would fetch latest available packages from the STABLE branch FTP server and update installed packages. To check for the security issues in packages another external tools called portaudit was needed.

Today we have pkg(8) with all its features along with pkg upgrade to update the installed packages. Thanks to pkg audit the third party tool portaudit is not longer needed. We even have pkg autoremove to automatically remove unneeded dependencies.

I will try not to copy information available on the already great FreeBSD Handbook described in the 4.4. Using pkg for Binary Package Management chapter.

Older FreeBSD Versions

Before FreeBSD 10.x to use new pkg(8) tools instead of the old pkg_* ones there was need to have WITH_PKGNG=yes in the /etc/make.conf file.

Currently only the only supported releases of FreeBSD are recently released 12.0 and still more stable and polished 11.2 so there is no need to put anything in the /etc/make.conf file anymore to use pkg(8) framework.

Database

The pkg(8) database (SQLite database actually) is kept in the /var/db/pkg directory.

These are the contents of the /var/db/pkg dir just after pkg(8) bootstrap process.

# find /var/db/pkg
/var/db/pkg
/var/db/pkg/FreeBSD.meta
/var/db/pkg/vuln.xml
/var/db/pkg/local.sqlite
/var/db/pkg/repo-FreeBSD.sqlite

The most important file is the /var/db/pkg/local.sqlite file as this is the database of installed packages and its files. By typing pkg shell you can actually connect to this SQLite database with SQLite interpreter.

# pkg shell
-- Loading resources from /home/vermaden/.sqliterc
SQLite version 3.15.2 2016-11-28 19:13:37
Enter ".help" for usage hints.
> .q
#

If for some reason you will find that pkg(8) tools does not work or are broken you may connect to it with sqlite3 command from the sqlite3 package. Do not use the sqlite package as it holds the 2.x version of SQLite which is not forward compatible with the 3.x version used by pkg(8)

# file /var/db/pkg/*
/var/db/pkg/FreeBSD.meta:        ASCII text
/var/db/pkg/local.sqlite:        SQLite 3.x database, user version 34, last written using SQLite version 3015002
/var/db/pkg/repo-FreeBSD.sqlite: SQLite 3.x database, user version 2014, last written using SQLite version 3015002
/var/db/pkg/vuln.xml:            XML 1.0 document, UTF-8 Unicode text, with very long lines

# sqlite3 /var/db/pkg/local.sqlite
-- Loading resources from /home/vermaden/.sqliterc
SQLite version 3.26.0 2018-12-01 12:34:55
Enter ".help" for usage hints.
> .q
#

Lock/Unlock

With pkg(8) specified packages can now be locked with pkg lock command. This means that the pkg upgrade or even pkg delete operations (or pkg autoremove) would not touch them. You can list locked packages with -l options as shown below.

# pkg lock -l
Currently locked packages:
conky-1.9.0_6
exfat-utils-1.2.8
ffmpeg-4.1_1,1
fusefs-exfat-1.2.8
lame-3.100_2

# pkg delete exfat-utils
Checking integrity... done (0 conflicting)
The following package(s) are locked and may not be removed:

        exfat-utils

1 packages requested for removal: 1 locked, 0 missing
# 

As you can see its not possible to pkg delete the locked exfat-utils package. You will first have to unlock it with pkg unlock command. You can do that interactively or not with -y option as shown below.

# pkg unlock exfat-utils
exfat-utils-1.2.8: unlock this package? [y/N]: y
Unlocking exfat-utils-1.2.8

# pkg lock -y exfat-utils
Locking exfat-utils-1.2.8

Now, why would you lock any packages?

Based on my experience these are potential reasons to lock certain packages:

  • You combine packages with ports.
  • Package for the port does not exist.
  • Official package has different default options then yours.
  • You really want to use older version of package.

Actually I use lock/unlock mechanism because all of the above are true for me.

I combine ports and packages (practice often discouraged in the FreeBSD world) because some software I use is not available as packages – because of licensing issues. These are anything related to Microsoft exFAT filesystem (exfat-utils/fusefs-exfat) and MP3 (lame). What is more astonishing for me is that OpenBSD provides lame package since YEARS yet FreeBSD team is still scared of the patents. I also need to build custom version of ffmpeg package – just to include lame support but still custom. The last thing I keep locked is Conky. It was and still is working great in 1.9 version but its developers broke it badly in the 1.10 version (now even 1.11 is available). It was just not possible to right click with mouse on the desktop and have Openbox menu – or to name the issue – Conky did not pass mouse events to the Window Manager that ruled the desktop. So I used one of the other Ports tools, the portdowngrade to fetch last 1.9 files into my Ports tree, then compile the 1.9 conky package and lock it for good.

You probably already know that I prefer to run dzen2 for screen information but I use conky rarely for my ‘FreeBSD Dashboard’ with all needed information that I enable only when I need it – with [Scroll Lock] key.

For the record – here is how it looks.

vermaden_2019-01-16_21-42-52.png

Provides

If you also happen to be RHEL/Fedora (or just yum/rpm) user you probably missed the ‘provides’ feature on FreeBSD pkg(8) package manager. Why it is so useful? Because with ‘provides’ database you can install packages by specifying the exact binary or file name of the package. For example You can type yum install /sbin/ifconfig to install net-tools package because ‘provides’ database will have that needed information.

What if I tell you that You can achieve similar functionality with pkg(8) tool?

The pkg-provides plugin allows you to query which package provides a particular file directly with pkg(8) tool.

It is even available as pkg-provides package. Below I will show you how to install and configure it. First install the pkg-provides package.

# pkg search provides
pkg-provides-0.5.0             Pkg plugin for querying which package provides a particular file

# pkg install pkg-provides
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        pkg-provides: 0.5.0 [FreeBSD]

Number of packages to be installed: 1

10 KiB to be downloaded.

Proceed with this action? [y/N]: y
[1/1] Fetching pkg-provides-0.5.0.txz: 100%   10 KiB   9.8kB/s    00:01    
Checking integrity... done (0 conflicting)
[1/1] Installing pkg-provides-0.5.0...
[1/1] Extracting pkg-provides-0.5.0: 100%
Message from pkg-provides-0.5.0:

======================= pkg plugin activation ========================
  In order to use the pkg-provides plugin you need to enable plugins in pkg.
  To do this, uncomment the following lines in /usr/local/etc/pkg.conf file
  and add pkg-provides to the supported plugin list

  PKG_PLUGINS_DIR = "/usr/local/lib/pkg/";
  PKG_ENABLE_PLUGINS = true;
  PLUGINS [ provides ];

  After that run `pkg plugins' to see the plugins handled by pkg`.

  To update the provides database run `pkg provides -u`

  ====================================================================

Then configure the /usr/local/etc/pkg.conf file.

# cat << __EOF__ >> /usr/local/etc/pkg.conf
PKG_PLUGINS_DIR = "/usr/local/lib/pkg/";
PKG_ENABLE_PLUGINS = true;
PLUGINS [ provides ];
__EOF__
Now you have new command called pkg provides as shown below.
# pkg provides
usage: pkg provides [-uf] pattern

A plugin for querying which package provides a particular file

# pkg provides bin/pldd
Provides database not found, please update first.

You can update the ‘provides’ database with -u option.

# pkg provides -u
Fetching provides database: 100%   29 MiB 700.9kB/s    00:43    
Extracting database....success

Example usage of pkg provides plugin.

# pkg provides bin/pldd
Name    : ptools2-0.5
Desc    : Toolset based on Solaris ptools functionality
Repo    : FreeBSD
Filename: /usr/local/bin/pldd

Name    : linux_base-c7-7.4.1708_6
Desc    : Base set of packages needed in Linux mode (Linux CentOS 7.4.1708)
Repo    : FreeBSD
Filename: /compat/linux/usr/bin/pldd

# pkg install /compat/linux/usr/bin/pldd
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
pkg: No packages available to install matching '/compat/linux/usr/bin/pldd' have been found in the repositories

Althou its not possible to for example install linux_base-c7 package by typing pkg install /compat/linux/usr/bin/pldd command its possible to check which package contains that file.

Next time you will type the pkg upgrade command you would also see provides database updating

# pkg upgrade
Updating FreeBSD repository catalogue...
Fetching meta.txz: 100%    944 B   0.9kB/s    00:01    
Fetching packagesite.txz: 100%    6 MiB 376.5kB/s    00:18    
Processing entries: 100%
Fetching provides database: 100%   29 MiB 386.3kB/s    01:18    
Extracting database....success
FreeBSD repository update completed. 32542 packages processed.
All repositories are up to date.
Checking integrity... done (0 conflicting)
(...)

The pkg provides database takes some notable space in the /var/db/pkg directory.

# file /var/db/pkg/* /var/db/pkg/*/* | sort -n
/var/db/pkg/FreeBSD.meta: ASCII text
/var/db/pkg/local.sqlite: SQLite 3.x database, user version 34, last written using SQLite version 3015002
/var/db/pkg/provides: directory
/var/db/pkg/provides/provides.db: ASCII text
/var/db/pkg/repo-FreeBSD.sqlite: SQLite 3.x database, user version 2014, last written using SQLite version 3015002
/var/db/pkg/vuln.xml: XML 1.0 document, UTF-8 Unicode text, with very long lines

If you use ZFS compression like LZ4 then it will not take much as shown below.

# du -csm /var/db/pkg/*
1       /var/db/pkg/FreeBSD.meta
32      /var/db/pkg/local.sqlite
72      /var/db/pkg/provides
33      /var/db/pkg/repo-FreeBSD.sqlite
2       /var/db/pkg/vuln.xml
138     total

… but if You use UFS then that almost 600 MB database may scare you a little ๐Ÿ™‚

# du -csmA /var/db/pkg/*
1       /var/db/pkg/FreeBSD.meta
68      /var/db/pkg/local.sqlite
571     /var/db/pkg/provides
52      /var/db/pkg/repo-FreeBSD.sqlite
6       /var/db/pkg/vuln.xml
694     total

Which

While the pkg provides needed information for the files of packages that are not yet installed the pkg which command is the pkg(8) equivalent of the classic UNIX which command. It shows to which package a file belongs to (or not at all).

# pkg which /boot/modules/drm.ko
/boot/modules/drm.ko was installed by package drm-fbsd11.2-kmod-4.11g20181210

# pkg which /boot/kernel/drm.ko
/boot/kernel/drm.ko was not found in the database

Double Your Gun Double Your Fun

Sometimes its faster to use both ‘whiches’ at the same time to get the needed answer.

# which firefox
/usr/local/bin/firefox

# pkg which `which firefox`
/usr/local/bin/firefox was installed by package firefox-64.0.2,1

Periodic

It may happen that you will see something like that one below.

# pkg install parallel
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
pkg: Cannot get an advisory lock on a database, it is locked by another process

… but You did not launched any other pkg(8) instances, what is going on here? Lets check the ps(1) output.

# ps ax | grep pkg
 8540  -  S        0:00.00 /bin/sh - /usr/local/etc/periodic/daily/411.pkg-backup
 8551  -  S        0:00.00 /usr/local/sbin/pkg shell .dump
 8555  -  D        0:01.08 /usr/local/sbin/pkg shell .dump

The FreeBSD’s periodic scripts are doing their job.

To check which are they look here.

# find /etc/periodic /usr/local/etc/periodic -name \*pkg\*
/usr/local/etc/periodic/daily/490.status-pkg-changes
/usr/local/etc/periodic/daily/411.pkg-backup
/usr/local/etc/periodic/security/460.pkg-checksum
/usr/local/etc/periodic/security/410.pkg-audit
/usr/local/etc/periodic/weekly/400.status-pkg

If You think that any of those activities are not needed then you may disable them with these values in the /etc/periodic.conf file.

# find /etc/periodic /usr/local/etc/periodic -name \*pkg\* | xargs grep -m 1 -E -o "[a-z_]+_enable" 
/usr/local/etc/periodic/daily/490.status-pkg-changes:daily_status_pkgng_changes_enable
/usr/local/etc/periodic/daily/411.pkg-backup:daily_backup_pkgng_enable
/usr/local/etc/periodic/security/460.pkg-checksum:security_status_pkgchecksum_enable
/usr/local/etc/periodic/security/410.pkg-audit:security_status_pkgaudit_enable
/usr/local/etc/periodic/weekly/400.status-pkg:weekly_status_pkgng_enable

For example if you would like to disable the /usr/local/etc/periodic/daily/490.status-pkg-changes execution you will need to add daily_status_pkgng_changes_enable=yes into the /etc/periodic.conf file.

Lefs chack again for the ps(1) output then.

# ps ax | grep pkg
 8574  0  S+       0:00.00 grep --color pkg

The periodic job has already finished. You may now install your package as usual.

# pkg install parallel
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        parallel: 20171222

Number of packages to be installed: 1

The process will require 3 MiB more space.
1 MiB to be downloaded.

Proceed with this action? [y/N]: n
# 

Stats

While the pkg stats command provides some stats on the installed packages its not that useful to find which packages take most space.

# pkg stats
Local package database:
        Installed packages: 1081
        Disk space occupied: 9 GiB

Remote package database(s):
        Number of repositories: 1
        Packages available: 32518
        Unique packages: 32518
        Total size of packages: 78 GiB

There is also pkg size command that will only display space used by packages but without package name … not very useful.

# pkg size | head
10.5MiB
2.06MiB
27.4MiB
2.59MiB
5.17MiB
515KiB
23.2MiB
609KiB
587KiB
127KiB

Also the man page for pkg size does not exist.

# man pkg-size
No manual entry for pkg-size

You can use pkg info -as command but it will not only not sort its output in any way – it will also display the space usage in various units like KiB/MiB/GiB which does not help … fortunatelly -h option of sort comes with help.

Using following alias you can sort packages by its space usage. I limited the output to 20 largest packages but feel free to change it to your needs.

# alias pkg-size='pkg info -as | sort -k 2 -h | tail -20 | column -t'
# which pkg-size
pkg-size: aliased to pkg info -as | sort -k 2 -h | tail -20 | column -t
# pkg-size
python27-2.7.15          68.2MiB
gtk3-3.22.30_4           68.8MiB
opencollada-1.6.68_1     75.8MiB
py27-ansible-2.7.5       88.6MiB
argyllcms-1.9.2_4        92.4MiB
webkit2-gtk3-2.22.5      92.9MiB
gimp-app-2.10.8_1,1      95.4MiB
python36-3.6.8           104MiB
samba47-4.7.12           145MiB
openjdk8-8.192.26_3      162MiB
boost-libs-1.69.0        163MiB
thunderbird-60.4.0_1     167MiB
firefox-64.0.2,1         174MiB
binutils-2.30_7,1        195MiB
linux_base-c6-6.10       197MiB
gcc6-6.5.0_3             241MiB
chromium-71.0.3578.98_2  251MiB
libreoffice-6.0.7_4      353MiB
virtualbox-ose-5.2.22_2  375MiB
llvm60-6.0.1_5           818MiB

Short Names

The pkg(8) tools also support short names for the arguments. For example you do not have to type pkg autoremove. Only the pkg autor part is needed for the command to work.

Example short names blow.

# pkg autor
# pkg upg
# pkg inf

Metadata

vermaden_2019-01-16_21-32-07.png

Many problems with pkg(8) are triggered by old metadata database. In case you face any pkg(8) issue first update (forcefully) its database as shown below.

# pkg update -f
Updating FreeBSD repository catalogue...
Fetching meta.txz: 100%    944 B   0.9kB/s    00:01    
Fetching packagesite.txz: 100%    6 MiB 352.9kB/s    00:19    
Processing entries: 100%
Fetching provides database: 100%   28 MiB 658.3kB/s    00:44    
Extracting database....success
FreeBSD repository update completed. 31778 packages processed.
All repositories are up to date.

For the record – the ‘provides’ database is also updated in such process.

Fixing Broken Dependency

There was time when one missing dependency about vulnerable www/libxul19 package started to torture me for some time.

I was even despered to compile everything with portmaster already.

I started with portmaster --check-depends command, but said no ‘n‘ when asked for fix as it will downgrade a lot of packages needlessly.

# portmaster --check-depends
(...)
Checking dependencies: evince
graphics/evince has a missing dependency: www/libxul19
(...)

>>> Missing package dependencies were detected.
>>> Found 1 issue(s) in total with your package database.

The following packages will be installed:

        Downgrading perl: 5.14.2_3 -> 5.14.2_2
        Downgrading glib: 2.34.3 -> 2.28.8_5
        Downgrading gio-fam-backend: 2.34.3 -> 2.28.8_1
        Downgrading libffi: 3.0.12 -> 3.0.11
        Downgrading gobject-introspection: 1.34.2 -> 0.10.8_3
        Downgrading atk: 2.6.0 -> 2.0.1
        Downgrading gdk-pixbuf2: 2.26.5 -> 2.23.5_3
        Downgrading pango: 1.30.1 -> 1.28.4_1
        Downgrading gtk-update-icon-cache: 2.24.17 -> 2.24.6_1
        Downgrading dbus: 1.6.8 -> 1.4.14_4
        Downgrading gtk: 2.24.17 -> 2.24.6_2
        Downgrading dbus-glib: 0.100.1 -> 0.94
        Installing libxul: 1.9.2.28_1

The installation will require 66 MB more space

38 MB to be downloaded

>>> Try to fix the missing dependencies [y/N]: n
>>> Summary of actions performed:

www/libxul19 dependency failed to be fixed

>>> There are still missing dependencies.
>>> You are advised to try fixing them manually.

>>> Also make sure to check 'pkg updating' for known issues.

Lets see what pkg(8) shows we have installed.

# pkg info | grep libxul
libxul-10.0.12                 Mozilla runtime package that can be used to bootstrap XUL+XPCOM apps

# pkg info -qoa | grep libxul
www/libxul

So the problem is that we have installed www/libxul instead of www/libxul19 and that is why portmaster (and not only) complains about it.

Before pkg(8) was introduced it was easy just to grep -r the entire /var/db/pkg directory with its ‘file database’ but now its quite more complicated as the package database is kept in SQLite database. Using pkg shell command You can connect to that database. Lets check what we can find there.

# pkg shell
SQLite version 3.7.13 2012-06-11 02:05:22
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> .databases
seq  name             file
---  ---------------  ----------------------------------------------------------
0    main             /var/db/pkg/local.sqlite
sqlite> .tables
categories       licenses         pkg_directories  scripts
deps             mtree            pkg_groups       shlibs
directories      options          pkg_licenses     users
files            packages         pkg_shlibs
groups           pkg_categories   pkg_users
sqlite> .header on
sqlite> .mode column
sqlite> pragma table_info(deps);
cid         name        type        notnull     dflt_value  pk
----------  ----------  ----------  ----------  ----------  ----------
0           origin      TEXT        1                       1
1           name        TEXT        1                       0
2           version     TEXT        1                       0
3           package_id  INTEGER     0                       1
sqlite> .quit

So now we know that ‘deps‘ table is probably what we are looking for ;).

As pkg shell is quite limited for SQLite ‘browsing’ I will use the sqlite3 command itself. By limited I mean that You can not type pkg shell "select * from deps;" query, You first need to start pkg shell and then You can type your query.

# sqlite3 -column /var/db/pkg/local.sqlite "select * from deps;" | grep libxul
www/libxul19   libxul      1.9.2.28_1  104

The second column is name so lets try to use it.

sqlite3 -header -column /var/db/pkg/local.sqlite "select * from deps where name='libxul';"
origin        name        version     package_id
------------  ----------  ----------  ----------
www/libxul19  libxul      1.9.2.28_1  104

So now we have the ‘problematic’ dependency entry nailed, lets modify it a little to the real installed packages state.

# sqlite3 /var/db/pkg/local.sqlite "update deps set origin='www/libxul' where name='libxul';"
# sqlite3 /var/db/pkg/local.sqlite "update deps set version='10.0.12' where name='libxul';"

You can of course use the ‘official’ way by using the pkg shell command.

# pkg shell
SQLite version 3.7.13 2012-06-11 02:05:22
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> update deps set origin='www/libxul' where name='libxul';
sqlite> update deps set version='10.0.12' where name='libxul';
sqlite> .header on
sqlite> .mode column
sqlite> select * from deps where name='libxul';
origin      name        version     package_id
----------  ----------  ----------  ----------
www/libxul  libxul      10.0.12     104
sqlite> .quit

Now portmaster is happy and does not complain about any missing dependencies.

# portmaster --check-depends
(...)
Checking dependencies: zenity
Checking dependencies: zip
Checking dependencies: zsh
# 

Viola! Problem solved ๐Ÿ˜‰

… but pkg(8) has a tool for that already ๐Ÿ™‚

Its called pkg set and two most useful options from man pkg-set are.

  -n oldname:newname, --change-name oldname:newname
       Change the package name of a given dependency from oldname to newname.

(...)

  -o oldorigin:neworigin, --change-origin oldorigin:neworigin
       Change the port origin of a given dependency from oldorigin to neworigin.
       This corresponds to the port directory that the package originated from.
       Typically, this is only needed for upgrading a library or package that
       has MOVED or when the default version of a major port dependency changes.
       (DEPRECATED) Usually this will be explained in /usr/ports/UPDATING.
       Also see pkg-updating(8) and EXAMPLES.

In our case we would use pkg set -o www/libxul19:www/libxul command.

Not sure if it will solve that problem in the same way as I also updated the version in the database.

UPDATING

If you get into any trouble with the pkg upgrade command then you should also check latest version of the /usr/ports/UPDATING file – available after updating the Ports tree with portsnap fetch update command for example.

It describes what important has changed in Ports (and packages as packages are built from Ports).

# less /usr/ports/UPDATING

(...)
20180518:
  AFFECTS: users of sysutils/ansible*
  AUTHOR: lifanov@FreeBSD.org

  Ansible ports are now flavored. Package names for Ansible changed
  to include python version. Poudriere and package users don't need
  to do anything.

  To rename an installed package to match the new naming scheme,
  for example, for ansible24, run:

   # pkg set -n ansible24:py27-ansible24

(...)

20180214:
  AFFECTS: users of lang/ruby23
  AUTHOR: swills@FreeBSD.org

  The default ruby version has been updated from 2.3 to 2.4.

  If you compile your own ports you may keep 2.3 as the default version by
  adding the following lines to your /etc/make.conf file:

  #
  # Keep ruby 2.3 as default version
  #
  DEFAULT_VERSIONS+=ruby=2.3

  If you wish to update to the new default version, you need to first stop any
  software that uses ruby. Then, you will need to follow these steps, depending
  upon how you manage your system.

  If you use pkgng, simply upgrade:
  # pkg upgrade

  If you use portmaster, install new ruby, then rebuild all ports that depend
  on ruby:
  # portmaster -o lang/ruby24 lang/ruby23
  # portmaster -R -r ruby-2.4

  If you use portupgrade, install new ruby, then rebuild all ports that depend
  on ruby:

  # pkg delete -f ruby portupgrade
  # make -C /usr/ports/ports-mgmt/portupgrade install clean
  # pkg set -o lang/ruby23:lang/ruby24
  # portupgrade -x ruby-2.4.\* -fr lang/ruby24

(...)

The pkg(8) framework also has a tool for that with pkg updating command. Check man pkg-updating page for details. The most common use case would be using the -d argument with date as shown below.

# pkg updating -d 20190101
20190103:
  AFFECTS: users of multimedia/vlc*
  AUTHOR: riggs@FreeBSD.org

  The multimedia/vlc port has been upgraded to 3.0.5, the latest upstream
  release. Subsequently, multimedia/vlc-qt4 and multimedia/vlc3 have been
  retired and removed from the ports tree. Users who previously used
  multimedia/vlc3 might want to switch to multimedia/vlc with the following
  commands:

  # pkg install multimedia/vlc
    or
  # portmaster -o multimedia/vlc multimedia/vlc3
    or
  # portupgrade -o multimedia/vlc multimedia/vlc3

You may as well check the UPDATING file online at the https://www.freshports.org/UPDATING address.

Bulletproof Upgrades with ZFS Boot Environments

To be absolutely sure that you will have a working system no matter what will went wrong with the pkg upgrade command just use the ZFS Boot Environments. I have made talks in Poland at PBUG and in Netherlands at NLUUG about its features not so long ago. The latest PDF presentation is still available at the https://is.gd/BECTL link.

The procedure with beadm command looks like that.

# beadm create safepoint
Created successfully

# beadm list
BE           Active Mountpoint  Space Created
11.2-RELEASE NR     /            5.7G 2018-12-01 13:09
safepoint    -      -          316.0K 2019-01-16 23:03

# pkg upgrade

Now if anything wrong will not happen You still have fully working system under the safepoint boot environment name.

Just reboot into it (select it in the FreeBSD loader) and you are back with working system, like you would be back in time with time machine.

Query

You can also use pkg query command to seek for intormation you need.

For example to ’emulate’ the pkg info -r pkg-name argument which displays the list of packages which require pkg-name you can use pkg query command as shown below.

# pkg info -r sqlite3
sqlite3-3.26.0:
        colord-gtk-0.1.26
        py27-sqlite3-2.7.15_7
        freeciv-2.5.10
        colord-1.3.5
        libsoup-2.62.3
        libsoup-gnome-2.62.3
        subversion-1.11.0_1
        nss-3.41_1
        webkit-gtk2-2.4.11_19
        filezilla-3.36.0_1
        epiphany-3.28.5_1
        darktable-2.4.4_3
        aria2-1.34.0_1
        webkit2-gtk3-2.22.5
        qt5-webkit-5.212.0.a2_17
        qt5-sqldrivers-sqlite3-5.12.0
        hugin-2018.0.0_6
        pidgin-2.13.0
        thunderbird-60.4.0_1
        midori-0.7.0
        firefox-64.0.2,1

# pkg query -e '%n = sqlite3' %ro
graphics/colord-gtk
databases/py-sqlite3
games/freeciv
graphics/colord
devel/libsoup
devel/libsoup-gnome
devel/subversion
security/nss
www/webkit-gtk2
ftp/filezilla
www/epiphany
graphics/darktable
www/aria2
www/webkit2-gtk3
www/qt5-webkit
databases/qt5-sqldrivers-sqlite3
graphics/hugin
net-im/pidgin
mail/thunderbird
www/midori
www/firefox

If you would like to know when each package was installed for the first time then use this spell below.

# pkg query "%t %n-%v" \
    | sort -n \
    | while read timestamp pkgname
      do
        echo "$(date -r $timestamp) $pkgname"
      done | ( head; echo; tail )
Fri Jul  7 14:17:29 CEST 2017 libpciaccess-0.13.5
Fri Jul  7 14:17:35 CEST 2017 libedit-3.1.20170329_2,1
Fri Jul  7 14:18:09 CEST 2017 font-util-1.3.1
Fri Jul  7 14:18:10 CEST 2017 xcb-util-0.4.0_2,1
Fri Jul  7 15:26:56 CEST 2017 xcb-util-renderutil-0.3.9_1
Fri Jul  7 15:26:57 CEST 2017 dejavu-2.37
Fri Jul  7 15:27:00 CEST 2017 font-misc-meltho-1.0.3_3
Fri Jul  7 15:27:02 CEST 2017 font-misc-ethiopic-1.0.3_3
Fri Jul  7 15:27:06 CEST 2017 font-bh-ttf-1.0.3_3
Fri Jul  7 15:27:08 CEST 2017 tpm-emulator-0.7.4_2

Sun Jan 13 20:48:01 CET 2019 firefox-64.0.2,1
Sun Jan 13 20:48:01 CET 2019 htop-2.2.0_1
Wed Jan 16 23:08:21 CET 2019 vlc-3.0.6,4
Wed Jan 16 23:08:21 CET 2019 xdg-utils-1.1.3
Wed Jan 16 23:08:25 CET 2019 phonon-qt4-4.10.2
Wed Jan 16 23:08:25 CET 2019 physfs-3.0.1
Wed Jan 16 23:08:25 CET 2019 py27-pyasn1-0.4.5
Wed Jan 16 23:08:26 CET 2019 chromium-71.0.3578.98_2
Wed Jan 16 23:08:26 CET 2019 moreutils-0.63
Wed Jan 16 23:08:26 CET 2019 p5-URI-1.76

You can also display packages that will not be removed by pkg autoremove command because You installed them directly.

# pkg query -e "%a != 1" "%n" | tail
xmp
xorg
xprintidle
xterm
xxkb
youtube_dl
zenity
zfs-stats
zip
zsh

Rosetta Stone

The FreeBSD Wiki page also provides some table but the information is incomplete.

Thus I copied the table and filled the missing data.

Below you will find the updated Rosetta Stone between old pkg_* tools compared to current pkg(8) framework.

Function Old pkg_* Tools New pkg(8) Tools
List of installed packages. pkg_info pkg info
Basic info about package. pkg_info pkgname-pkgversion pkg info pkgname
pkg info category/name
pkg info pkgname-pkgversion
Detailed info about package. N/A pkg info -f pkgname
pkg info -f category/name
pkg info -f pkgname-pkgversion
List all files in installed package. pkg_info -L pkgname-pkgversion pkg info -l pkgname
pkg info -l category/name
pkg info -l pkgname-pkgversion
Find which package provides file. pkg_info -W /path/to/my/file pkg which /path/to/my/file
Install local package. pkg_add ./localpkg.tbz pkg add ./localpkg.txz
Install remote package. pkg_add -r mypackage pkg install mypackage
pkg install category/name
pkg install pkgname-pkgversion
Search for remote package. ls /usr/ports/* | grep mypackage pkg search mypackage
pkg search category/name
pkg search pkgname-pkgversion
Search for detailed info about remote package. make search name=mypackage
make search key=mypackage
pkg search -f mypackage
pkg search -f category/name
pkg search -f pkgname-pkgversion
Reverse deps of installed package. pkg_info -R pkgname-pkgversion pkg info -r mypackage
pkg info -r category/name
pkg info -r pkgname-pkgversion
Deps of installed package. pkg_info -r pkgname-pkgversion pkg info -d mypackage
pkg info -d category/name
pkg info -d pkgname-pkgversion
Remove unused packages install as dep. N/A pkg autoremove
Binary upgrade installed packages. pkg_upgrade (FreeBSD Ports) pkg upgrade
Create remote repository. N/A pkg repo /directory/with/packages
Manipulate packages in jail. N/A pkg -j
Manipulate packages in chroot. pkg_add -C pkg -c
Info about installed packages using RE. pkg_info -x pkg info -x
Info about installed packages using extended RE. pkg_info -X pkg info -X
Info about installed packages using globbing. pkg_info pkg info -g
Check for known vulnerabilities. portaudit (FreeBSD Ports) pkg audit
Out of date packages. pkg_version -l < pkg version -l <
Out of date packages. pkg_version -Il < pkg version -Il <
Out of date packages compared to remote repo. N/A pkg upgrade -n
Statistic about installed packages. N/A pkg stat
Checking for missing dependency (with fix). N/A pkg check -d
Port origin. pkg_info -o pkg info -o

If you know any other useful pkg(8) spells then let me know ๐Ÿ™‚

EOF

 

FreeBSD Desktop – Part 2.1 – Install FreeBSD 12

This article is an update/rewrite to the already published FreeBSD Desktop – Part 2 – Install. With the upcoming introduction of the FreeBSD 12.0-RELESE version new possibilities arise when it comes to installation. I already talked/showed that method in my ZFS Boot Environments Reloaded at NLUUG presentation but to make it more available and obvious part of my FreeBSD Desktop series I write about it again in dedicated article entry.

You may want to check other articles in the FreeBSD Desktop series on the FreeBSD Desktop – Global Page where you will find links to all episodes of the series along with table of contents for each episodeโ€™s contents.

Now (in FreeBSD 12.x) it is possible to install FreeBSD on GELI encrypted root on ZFS pool without any additional partitions or filesystems. No longer separate UFS or ZFS boot pool /boot filesystem is needed. And what is even more appealing such setup is supported both on UEFI and BIOS (also refereed as Legacy or CSM) systems. Such setup is also compatible with both new bectl(8) utility and the old proven beadm(8) tool. It is also nice that to make such setup you only need to choose the Auto ZFS option from the bsdinstall(8) so you will not have to do it by hand. I advice using GPT (BIOS+UEFI) as it will support both system types so when you are running BIOS system now and will move the disk to other system that boots with UEFI it will also just work out of the box.

The FreeBSD 12.0 is currently at the RC1 stage so we will use that one for below examples of such setup. The 12.0-RELEASE is expected to arise before Christmas if no significant problems or bugs will be found on the road to RC2 and RC3 editions.

For the record here is the FreeBSD 12.0-RC1 Availability information page and aggregated FreeBSD 12.0-RELEASE Release Notes for the upcoming new major FreeBSD version, but it is not yet complete/ready.

I will only show one install process that will work for both UEFI and BIOS systems – the crucial option here is GPT (BIOS+UEFI) to select (which is also the default one). The other option that You need to select is Yes for the Encryption part and also select the SWAP size. You may as well do not use swap and enter ‘0‘ here which means that SWAP partition will not be created. You may as well create ZFS ZVOL partition for SWAP on ZFS pool later or just create a file like /SWAP and enable it as SWAP. No matter which SWAP option you will choose if your system swaps then you are too low on memory and neither of these methods are better or worse then.

freebsd-install-01.png

freebsd-install-02.png

freebsd-install-03.png

One last thing about the default FreeBSD (no matter if 11.x or 12.x) ZFS dataset/filesystem layout. I showed it on my ZFS Boot Environments/ZFS Boot Environments Reloaded presentations but without any text comment as I talked it live.

By default both /var and /usr filesystems are part of the Boot Environment. They are protected and snapshoted during the beadm create newbe process (or by bectl(8) also). Its appears that /var and /usr are separate processes when you type zfs list commend as shown on the slide below.

zroot-layout-01.png

… but when you check the canmount parameter for all ZFS datasets, then it become obvious that /usr and /var are ’empty’ datasets (not mounted).

zroot-layout-02.png

… and also confirmation from theย df(1) tool.

zroot-layout-03.png

I asked FreeBSD Developers what is the reason for such construct and its for the mountpoint inheritance purposes. For example when zroot/usr has mountpoint set to /usr then when you create zroot/usr/local dataset, then it will automatically get the /usr/local for the mountpoint parameter by inheritance. At the first sight it may be misleading (I also got caught) but it makes sense when you think about it.

The only filesystems that are NOT included for the Boot Environment protection are these:

  • /usr/home
  • /usr/ports
  • /usr/src
  • /var/audit
  • /var/crash
  • /var/log
  • /var/mail
  • /var/tmp

While in most cases it is not needed to protect these in the Boot Environment protection if you want to also protect these type these two comments to move all the /usr/* and /var/* datasets/filesystems into the Boot Environment pool/ROOT/dataset. It will work on a running system without need for reboot, just make sure you use -u flag.

# zfs rename -u zroot/usr zroot/ROOT/default/usr
# zfs rename -u zroot/var zroot/ROOT/default/var

Now grab that FreeBSD ISO and install it the best possible way up to date ๐Ÿ™‚

You will probably want to get amd64 version which is suitable for both 64-bit AMD and Intel systems.

EOF

ย 

ZFS Boot Environments Reloaded at NLUUG Autumn Conference 2018

Today I was honored to give ZFS Boot Environments Reloaded talk at the NLUUG Najaarsconferentie 2018 (the NLUUG Autumn Conference 2018).

zfsbectl-shot-256

Dunno for how long the conference page will be available so below you will find both conference page and its program.

NLUUG 2018 NL Site

… and the program.

NLUUG 2018 NL Program

There were lots of interesting talks, I really liked the Salt workshop at the end of the day. I wanted to also attend Ansible presentation in the morning but having spend almost 17 hours in the car driving from central Poland to Netherlands I really needed some sleep ๐Ÿ™‚

You are invited to download the PDF Slides – https://is.gd/BECTL – available here.

As I just finished the talk the video from the talk is not yet available, but I will update the post when it will be.

Besides obvious things that ZFS Boot Environments can do I also talked about upcoming FreeBSD 12.0-RELEASE improvements in this area. Currently FreeBSD 12.0-RELEASE is planned to be released around Christmas later this year and is at the BETA4 stage currently.

Also here is the abstractย that I added at the conference page about the presentation.

Abstract

As long as sysadmins need to maintain, change and update operating systems there is always need to protect against problems that may occur during these operations. Various solutions were used starting from simple backup/restore procedures or copying the contents of system filesystems into spare disks to snapshots and clones recently. None of these solutions were transparent enough or bulletproof enough to provide complete and consistent protection again failures in the change or update process.

One of such holy grails is ZFS Boot Environments solution. It protects entire system (and even additional data when needed) against almost any change or update process. As ZFS Boot Environments matured in Solaris/Illumos systems and then on FreeBSD UNIX other systems started to copy its principles to provide similar solutions such as snapper with BTRFS in SUSE or Boot Environment Manager for DragonFly BSD with their HAMMER filesystem.

The presentation aims to walk through the history of these solutions with the focus on practical ZFS Boot Environments solutions and examples.

ย 
Hope you will enjoy it.

EOF

Silent Fanless FreeBSD Server – DIY Backup

I already once wrote about this topic at the Silent Fanless FreeBSD Desktop/Server article. To my pleasant surprise BSD NOW Episode 253: Silence of the Fans featured my article for which I am very grateful. Today I would like to show another practical example of such setup and with more hands on approach along with real power usage measurements with power meter. I also got more power efficient ASRock N3150B-ITX motherboard with only 6W TDP which includes 4-core Celeron N3150 CPU and also nice small Supermicro SC101i Mini ITX case. Keep in mind that ASRock also made very similar N3150-ITX motherboard (no ‘B’ in model name) with different ports/connectors that may better suit your needs better.

You may also check the follow up Silent Fanless FreeBSD Server โ€“ Redundant Backup article.

Build

Here is how the Supermicro SC101i case looks like with ASRock N3150B-ITX motherboard installed.

silent-backup-case-external.jpg

silent-backup-case-back.jpg

One thing that surprised me very much was the hard disk cost. The internal Seagate 4TB ST4000LM024 2.5 SATA drive costs about $180-190 but the same disk sold as Maxtor M3 4TB 2.5 disk in external case with Maxtor brand (which is owned by Seagate anyway) and USB 3.0 port costs half of that – about $90-100. At least in Europe/Poland location.

I think you do already know where I am going with my thoughts. I will use an external Maxtor M3 4TB 2.5 drive and connect it via the USB 3.0 port in this setup. While SATA III provides theoretical throughput of 6Gbps the USB 3.0 provides 5Gbps theoretical throughput. The difference can be important for low latency high throughput SSD drives that approach 580MB/s speed but not for traditional rotational disks moving gently at 5400RPM.

The maximum performance I was able to squeeze from this Maxtor M3 4TB 2.5 USB 3.0 drive was 90MB/s write speed and 120MB/s read speed using pv(1) tool, and that was at the beginning of the disk. These speeds will drop to about 70MB/s and 90MB/s at the end of the disk respectively for write and read operations. We are not even approaching SATA I standard here which tops at 1.5Gbps. Thus it will not make a difference or not a significant one for sure for such storage.

At first I wanted to make a hole on the motherboard end steel plate (somewhere beside the back ports) with drill to get outside with USB cable from the case and attach it to one of the USB 3.0 ports at the back of the motherboard but fortunately I got better idea. This motherboard has connector for internal USB 3.0 (so called front panel USB on the case) so I bought Akyga AK-CA-57 front panel cable with USB 3.0 port and connected everything inside the case.

This is the Akyga AK-CA-57 USB 3.0 cable.

silent-backup-usb-akyga-cable-AK-CA-57.jpg

If I was going to install two USB 3.0 disks using this method I would use one of these cables instead:

The only problem can be more physical one – will it blend will it fit? Fortunately I was able to find a way to fit it in the case and there is even space for the second disk. As this will be my offsite backup replacement which is only 3rd stage/offsite backup I do not need to create redundant mirror/RAID1 protection but it’s definitely possible with two Maxtor M3 4TB 2.5 USB 3.0 drives.

The opened Supermicro SC101i case with ASRock N3150B-ITX motherboard inside and attached Pico PSU looks like that.

silent-backup-mobo-case.jpg

With attached Akyga AK-CA-57 USB 3.0 cable things get little narrow, but with proper cable lay you will still be able to fit another internal 2.5 SATA disk or external 2.5 USB 3.0 disk.

silent-backup-mobo-case-blue.jpg

I attached Akyga AK-CA-57 cable to this USB 3.0 connector on the motherboard.

silent-backup-mobo-case-usb.jpg

Case with Maxtor M3 4TB disk. The disk placement required little modifications.

silent-backup-mobo-case-blue-disk.jpg

I created custom disk holders using steel plates I got from window mosquito net set for my home but you should be able to get something similar in any hardware shop. I modified them a little with pliers.

silent-backup-handles

I also ‘silenced’ the disk vibrations with felt stickers.

silent-backup-silence.jpg

The silenced disk in the Supermicro SC101i case.

silent-backup-mobo-case-blue-disk-silence.jpg

Ancestor

Before this setup I used Raspberry Pi 2B with external Western Digital 2TB 2.5 USB 3.0 disk but the storage space requirements become larger so I needed to increase that. It was of course with GELI encryption and ZFS with enabled LZ4 compression on top. The four humble ARM32 cores and soldered 1GB of RAM was able to squeeze whooping 5MB/s read/write experience from this ZFS/GELI setup but that was not hurting me as I used rsync(1) for differential backups and the Internet connection to that box was limited to about 1.5MB/s. I would still use that setup but it just won’t boot with that larger Maxtor M3 4TB disk because it requires more power and I already used stronger 5V 3.1A charger then 5V 2.0A suggested by vendor. Even the safe_mode_gpio=4 and max_usb_current=1 options at /boot/msdos/config.txt did not help.

Cost

The complete setup price tops at $220 total. Here are the parts used.

PRICE  COMPONENT
  $59  CPU/Motherboard ASRock N3150B-ITX Mini-ITX
  $14  RAM Crucial 4GB DDR3L 1.35V
  $13  PSU 12V 7.5A 90W Pico (internal)
   $2  PSU 12V 2.5A 30W Leader Electronics (external)
  $29  Supermicro SC101i (used)
   $3  Akyga AK-CA-57 USB 3.0 Cable
   $3  SanDisk Fit 16GB USB 2.0 Drive (system)
  $95  Maxtor M3 4TB 2.5 USB 3.0 Drive (data)
 $220  TOTAL

PSU

In earlier Silent Fanless FreeBSD Desktop/Server article I used quite large 90W PSU from FSP Group. From the PSUs that I owned only ThinkPad W520/W530 bricks can compete in size with this beast. As this motherboard will use very little power (details lower) it will require a lot smaller PSU. As the FSP Group PSU has IEC C14 slot it also requires additional IEC C13 power cable which makes it even bigger solution. The new 12V 2.5A 30W is very compact and also costs fraction of the 90W FSP Group gojira.

New Leader Electronics PSU label.

silent-backup-psu-ext-label.jpg

Below you can see the comparison for yourself.

silent-backup-psu-compare

I also got cheaper and less powerful Pico PSU which now tops as 12V 7.5A 90W power.

silent-backup-psu-pico-12V-90W.jpg

Power Consumption

This is where it gets really interesting. I measured the power consumption with power meter.

silent-backup-power-meter.jpg

Idle

When this box is booted without any media attached it uses only 7.5W of power idling. While the system was idle with SanDisk 16GB USB 2.0 drive (on which FreeBSD was installed) it used about 8.0W of power. When booted with Maxtor M3 4TB disk inside and SanDisk 16GB USB 2.0 drive attached it run idle at about 8.5W of power.

Load

As I do not need full CPU speed I limited the CPU speed in powerd(8) options to 1.2Ghz. With this limit set the fully loaded system with all 4 cores busy at 100% and two dd(8) processes for read both boot SanDisk 16GB drive and Maxtor M3 4TB disk and with GELI enabled ZFS pool doing scrub operation in progress and additional two find(1) processes for both disks it would not pass the 13.9W barrier. Without CPU limitation (that means Intel Turbo Boost enabled) the system used 16.0W of power at most.

Summary of power usage for this box.

 POWER  TYPE  CONFIGURATION
 7.5 W  IDLE  System
 8.0 W  IDLE  System + SanDisk 16GB drive
 8.5 W  IDLE  System + SanDisk 16GB drive + Maxtor M3 4TB drive + CPU 1.2 Ghz limit
 8.5 W  IDLE  System + SanDisk 16GB drive + Maxtor M3 4TB drive
13.9 W  LOAD  System + SanDisk 16GB drive + Maxtor M3 4TB drive + CPU 1.2 Ghz limit
16.0 W  LOAD  System + SanDisk 16GB drive + Maxtor M3 4TB drive

For comparision the Raspberry Pi 2B with 16GB MicroSD card attached used only 1.5W but we all know how slow it is. When used with Western Digital 2TB 2.5 USB 3.0 drive it used about 2.2W at idle state.

Configuration for Low Power Consumption

Below are FreeBSD configuration files used in this box to lower the power consumption.

The /etc/sysctl.conf file.

# ANNOYING THINGS
  vfs.usermount=1
  kern.coredump=0
  hw.syscons.bell=0
  kern.vt.enable_bell=0

# LIMIT ZFS ARC EFFICIENTLY
  kern.maxvnodes=32768

# ALLOW UPGRADES IN JAILS
  security.jail.chflags_allowed=1

# ALLOW RAW SOCKETS IN JAILS
  security.jail.param.allow.raw_sockets=1
  security.jail.allow_raw_sockets=1

# RANDOM PID
  kern.randompid=12345

# PERFORMANCE/ALL SHARED MEMORY SEGMENTS WILL BE MAPPED TO UNPAGEABLE RAM 
  kern.ipc.shm_use_phys=1

# MEMORY OVERCOMMIT SEE tuning(7)
  vm.overcommit=2

# NETWORK/DO NOT SEND RST ON SEGMENTS TO CLOSED PORTS
  net.inet.tcp.blackhole=2

# NETWORK/DO NOT SEND PORT UNREACHABLES FOR REFUSED CONNECTS
  net.inet.udp.blackhole=1

# NETWORK/ENABLE SCTP BLACKHOLING blackhole(4) FOR MORE DETAILS
  net.inet.sctp.blackhole=1

# NETWORK/MAX SIZE OF AUTOMATIC RECEIVE BUFFER (2097152) [4x]
  net.inet.tcp.recvbuf_max=8388608

# NETWORK/MAX SIZE OF AUTOMATIC SEND BUFFER (2097152) [4x]
  net.inet.tcp.sendbuf_max=8388608

# NETWORK/MAXIMUM SOCKET BUFFER SIZE (5242880) [3.2x]
  kern.ipc.maxsockbuf=16777216

# NETWORK/MAXIMUM LISTEN SOCKET PENDING CONNECTION ACCEPT QUEUE SIZE (128) [8x]
  kern.ipc.soacceptqueue=1024

# NETWORK/DEFAULT tcp MAXIMUM SEGMENT SIZE (536) [2.7x]
  net.inet.tcp.mssdflt=1460

# NETWORK/MINIMUM TCP MAXIMUM SEGMENT SIZE (216) [6x]
  net.inet.tcp.minmss=1300

# NETWORK/LIMIT ON SYN/ACK RETRANSMISSIONS (3)
  net.inet.tcp.syncache.rexmtlimit=0

# NETWORK/USE TCP SYN COOKIES IF THE SYNCACHE OVERFLOWS (1)
  net.inet.tcp.syncookies=0

# NETWORK/ENABLE TCP SEGMENTATION OFFLOAD (1)
  net.inet.tcp.tso=0

# NETWORK/ENABLE IP OPTIONS PROCESSING ([LS]SRR, RR, TS) (1)
  net.inet.ip.process_options=0

# NETWORK/ASSIGN RANDOM ip_id VALUES (0)
  net.inet.ip.random_id=1

# NETWORK/ENABLE SENDING IP REDIRECTS (1)
  net.inet.ip.redirect=0

# NETWORK/IGNORE ICMP REDIRECTS (0)
  net.inet.icmp.drop_redirect=1

# NETWORK/ASSUME SO_KEEPALIVE ON ALL TCP CONNECTIONS (1)
  net.inet.tcp.always_keepalive=0

# NETWORK/DROP TCP PACKETS WITH SYN+FIN SET (0)
  net.inet.tcp.drop_synfin=1

# NETWORK/RECYCLE CLOSED FIN_WAIT_2 CONNECTIONS FASTER (0)
  net.inet.tcp.fast_finwait2_recycle=1

# NETWORK/CERTAIN ICMP UNREACHABLE MESSAGES MAY ABORT CONNECTIONS IN SYN_SENT (1)
  net.inet.tcp.icmp_may_rst=0

# NETWORK/MAXIMUM SEGMENT LIFETIME (30000) [0.27x]
  net.inet.tcp.msl=8192

# NETWORK/ENABLE PATH MTU DISCOVERY (1)
  net.inet.tcp.path_mtu_discovery=0

# NETWORK/EXPIRE TIME OF TCP HOSTCACHE ENTRIES (3600) [2x]
  net.inet.tcp.hostcache.expire=7200

# NETWORK/TIME BEFORE DELAYED ACK IS SENT (100) [0.2x]
  net.inet.tcp.delacktime=20

The /boot/loader.conf file.

# BOOT OPTIONS
  autoboot_delay=1
  boot_mute=YES

# MODULES FOR BOOT
  zfs_load=YES

# DISABLE HYPER THREADING
  machdep.hyperthreading_allowed=0

# REDUCE NUMBER OF SOUND GENERATED INTERRUPTS
  hw.snd.latency=7

# RACCT/RCTL RESOURCE LIMITS
  kern.racct.enable=1

# PIPE KVA LIMIT | 320 MB
  kern.ipc.maxpipekva=335544320

# NUMBER OF SEGMENTS PER PROCESS
  kern.ipc.shmseg=1024

# LARGE PAGE MAPPINGS
  vm.pmap.pg_ps_enabled=1

# SHARED MEMORY
  kern.ipc.shmmni=1024
  kern.ipc.shmseg=1024

# ZFS TUNING
  vfs.zfs.prefetch_disable=1
  vfs.zfs.cache_flush_disable=1
  vfs.zfs.vdev.cache.size=16M
  vfs.zfs.arc_min=32M
  vfs.zfs.arc_max=128M
  vfs.zfs.txg.timeout=1

# NETWORK MAX SEND QUEUE SIZE
  net.link.ifqmaxlen=2048

# POWER OFF DEVICES WITHOUT ATTACHED DRIVER
  hw.pci.do_power_nodriver=3

# AHCI POWER MANAGEMENT FOR EVERY USED CHANNEL (ahcich 0-7)
  hint.ahcich.0.pm_level=5
  hint.ahcich.1.pm_level=5
  hint.ahcich.2.pm_level=5
  hint.ahcich.3.pm_level=5
  hint.ahcich.4.pm_level=5
  hint.ahcich.5.pm_level=5
  hint.ahcich.6.pm_level=5
  hint.ahcich.7.pm_level=5

# GELI THREADS
  kern.geom.eli.threads=2
  kern.geom.eli.batch=1

The /etc/rc.conf file.

# NETWORK
  hostname=offsite.local
  background_dhclient=YES
  extra_netfs_types=NFS
  defaultroute_delay=3
  defaultroute_carrier_delay=3

# MODULES/COMMON/BASE
  kld_list="${kld_list} aesni geom_eli"
  kld_list="${kld_list} fuse coretemp sem cpuctl ichsmb cc_htcp"
  kld_list="${kld_list} libiconv cd9660_iconv msdosfs_iconv udf_iconv"

# POWER
  performance_cx_lowest=C1
  economy_cx_lowest=Cmax
  powerd_enable=YES
  powerd_flags="-n adaptive -a hiadaptive -b adaptive -m 400 -M 1200"

# DAEMONS | yes
  zfs_enable=YES
  nfs_client_enable=YES
  syslogd_flags='-s -s'
  sshd_enable=YES

# DAEMONS | no
  sendmail_enable=NONE
  sendmail_submit_enable=NO
  sendmail_outbound_enable=NO
  sendmail_msp_queue_enable=NO

# FS
  fsck_y_enable=YES
  clear_tmp_enable=YES
  clear_tmp_X=YES
  growfs_enable=YES

# OTHER
  keyrate=fast
  font8x14=vgarom-8x14
  virecover_enable=NO
  update_motd=NO
  devfs_system_ruleset=desktop
  hostid_enable=NO

USB Boot Drive

I was not sure if I should use USB 2.0 drive or USB 3.0 drive for FreeBSD system so I got both versions from SanDisk and tested their performance with pv(1) and diskinfo(8) tools. The pv(1) utility had options enabled shown below and for diskinfo(8) the -c and -i parameters were used.

% which pv
pv: aliased to pv -t -r -a -b -W -B 1048576

The dmesg(8) information for the SanDisk Fit USB 2.0 16GB drive.

# dmesg | tail -6
da0 at umass-sim0 bus 0 scbus3 target 0 lun 0
da0:  Removable Direct Access SPC-4 SCSI device
da0: Serial Number 4C530001100609104091
da0: 40.000MB/s transfers
da0: 15060MB (30842880 512 byte sectors)
da0: quirks=0x2

The dmesg(8) information for the SanDisk Fit USB 3.0 16GB drive.

# dmesg | tail -6
da0 at umass-sim0 bus 0 scbus3 target 0 lun 0
da0:  Removable Direct Access SPC-4 SCSI device
da0: Serial Number 4C530 001070202100093
da0: 40.000MB/s transfers
da0: 14663MB (30031250 512 byte sectors)
da0: quirks=0x2

There is also noticeable size difference as the USB 2.0 version has additional 400 MB of space!

By the way … the SanDisk Fit USB 3.0 16GB came with this sticker inside the box – a serial number for the RescuePRO Deluxe software – which I will never use. Not because its bad or something but because I have no such needs. You may take it … of course unless someone else did not took it already ๐Ÿ™‚

silent-backup-license.jpg

Below are the results of the benchmarks, I tested them in both USB 2.0 and USB 3.0 ports.


                   DRIVE  USB  pv/READ  pv/WRITE  diskinfo/OVERHEAD  diskinfo/IOPS
SanDisk Fit USB 2.0 16GB  2.0   29MB/s     5MB/s   0.712msec/sector           2521
SanDisk Fit USB 2.0 16GB  3.0   33MB/s     5MB/s   0.799msec/sector           2441
SanDisk Fit USB 3.0 16GB  2.0   35MB/s     9MB/s   0.618msec/sector           1920
SanDisk Fit USB 3.0 16GB  3.0   91MB/s    11MB/s   0.567msec/sector           1588

What is also interesting is that while USB 2.0 version has lower throughput it has more IOPS then the newer USB 3.0 incarnation of the SanDisk Fit drive. I also did other more real life test. I checked how long would it take to boot FreeBSD system installed on each of them from the loader(8) screen to the login: prompt. The difference is 5 seconds. Details are shown below.

 TIME  DRIVE
  28s  SanDisk Fit USB 3.0 16GB
  33s  SanDisk Fit USB 2.0 16GB

With such small ~15% difference I will use SanDisk Fit USB 2.0 16GB as it sticks out little less outside from the slot as shown below.

silent-backup-usb-drives.jpg

Cloud Storage Prices Comparison

The Tarsnap“online backups for the truly paranoid” – costs $0.25/GB/month. The price in Tarsnap is for data transmitted after deduplication and compression but that does not change much here. For my data the compressratio property from ZFS dataset is at 3% (1.03). When I estimate deduplication savings with zdb -S pool command I get additional 1% of the savings (1.01). Lets assume that with both deduplication and compression it would take 5% (1.05) savings. That would lower the Tarsnap price to $0.2375/GB/month.

The Backblaze B2 Cloud Storage – storage costs $0.005/GB/month.

Our single 4TB disk solution costs $230 for lets say 3 years. You can expect disk failure after that period but it may serve you as well for another 3 years. Now as we know the cloud storage prices lets calculate price for 4TB data stored for 3 years in these cloud services.

Self Solution Electricity Cost

We also need to calculate how much energy our build solution would consume. Currently 1kWh of power costs about $0.20 in Europe/Poland (rounded up). This means that running computer with 1000W power usage for 1 hour would cost you $0.20 on electricity bill. Our solution idles at 8.5W and uses 13.9W when fully loaded. It will be idle for most of the time so I will assume that it will use 10W on average here. That would cost us $0.002 for 10W device running for 1 hour.

Below you will also find calculations for 1 day (24x multiplier), 1 year (another 365.25x multiplier) and 3 years (another 3x multiplier).

  COST  TIME
$0.002  1 HOUR
$0.048  1 DAY
$17.53  1 YEAR
$52.60  3 YEARS

Our total 3 years electricity cost is $282.60 for building and then running the system non-stop. We can also implement features like Wake On LAN to limit that power usage even more for example.

Here are these cloud storage service providers prices.


PROVIDER     PRICE  DATA  TIME
Tarsnap    $0.2375   1GB  1 Month
Backblaze  $0.0050   1GB  1 Month

The price for 1 month of keeping 4TB of data on these providers looks as follows.


PROVIDER   PRICE  DATA  TIME
Tarsnap     $973   4TB  1 Month
Backblaze    $20   4TB  1 Month

For just 1 month the Tarsnap is 4 TIMES more expensive the keeping the backup on your self computer with 4TB disk. The Backblaze service is at 1/10 cost which is still reasonable.

Lets compare prices for 3 years of 4TB storage.


PROVIDER    PRICE  DATA  TIME
Tarsnap    $35021   4TB  3 Years
Backblaze    $737   4TB  3 Years

After 3 years the Backblaze solutions is about 2.5 TIMES more expensive then our personal setup, but if you really do not want to create your solution the difference for 3 years is not that big. The Tarsnap is out of bounds here being more then 120 TIMES more expensive then self hosted solution. Remember that I also did not included costs for transferring the data into or from the cloud storage. That would make cloud storage costs even bigger depending how often you would want to pull/push your data.

EOF

Valuable News – 2018/08/25

UNIX

OpenBSD adds kcov(4) kernel code coverage tracing driver.
So far 8 distinct panics have been found and fixed.
https://marc.info/?l=openbsd-cvs&m=153467896308034&w=2

GCC 8.2 now packaged and available in Illumos/OpenIndiana.
https://bsd.network/@sehnsucht/100581557620270760
https://pkg.openindiana.org/hipster/info/0/developer%2Fgcc-8%408.2.0%2C5.11-2018.0.0.0%3A20180815T204704Z

FreeBSD arc4random is now based on ChaCha20 implementation from OpenBSD.
https://twitter.com/lattera/status/1031280553301925888
https://svnweb.freebsd.org/base?view=revision&revision=338059

Valve forked WINE into Proton as compatibility tool for Steam Play.
https://github.com/ValveSoftware/Proton/
https://steamcommunity.com/games/221410/announcements/detail/1696055855739350561

AMD Threadripper 2990WX 32-core/64-thread on DragonFly BSD.
http://apollo.backplane.com/DFlyMisc/threadripper.txt
http://lists.dragonflybsd.org/pipermail/users/2018-August/357858.html

Using 10GE Adapters with PowerVM SEA – Virtual Ethernet Considerations.
http://ibmsystemsmag.com/aix/administrator/virtualization/using-10gbit-ethernet-adapters/

Native ZFS Encryption on FreeBSD CFT on the road to 12.0-RELEASE.
https://lists.freebsd.org/pipermail/freebsd-current/2018-August/070832.html

Backup FreeNAS and TrueNAS to Backblaze B2 Cloud.
https://www.backblaze.com/blog/how-to-setup-freenas-cloud-storage/

Colin Percival heroic (I am not joking here) fight for removing unneeded sleeps during boot on FreeBSD.
https://twitter.com/cperciva/status/1031928231635677184
https://reviews.freebsd.org/D16723

Writing SYSTEMD service files.
https://twitter.com/mulander/status/1031908074733428736
https://obsd.pl/mfm/iptables/

Illumos/Tribblix packages of openjdk9 and openjdk10 available.
https://twitter.com/ptribble/status/1031650238266789893
https://twitter.com/ptribble/status/1031900360271491074
http://pkgs.tribblix.org/openjdk/

Difference between OpenBSD xenodm and regular xdm.
https://undeadly.org/cgi?action=article&sid=20160911231712

X.Org Security Advisory – 2018/08/21.
http://seclists.org/oss-sec/2018/q3/146

FreeBSD removes legacy DRM and DRM2 from its tree.
https://twitter.com/f0andrey/status/1032234624544583680
https://svnweb.freebsd.org/base?view=revision&revision=338172

OmniOS CE (Community Edition) r151026p/r151024ap/r151022bn with CVE-2018-15473 addressed.
https://omniosce.org/article/releases-026p-024ap-022bn.html

Running Mastodon on FreeBSD.
https://ftfl.ca/blog/2017-05-23-mastodon-freebsd.html

Upgrading Mastodon on FreeBSD.
https://ftfl.ca/blog/2017-05-27-mastodon-freebsd-upgrade.html

KDE Plasma 5.x on Pinebook Laptop.
https://twitter.com/SoftpediaLinux/status/1032262240437723137

FreeBSD – Raspberry Pi 3B+ – UART.
https://blackdot.be/2018/08/freebsd-uart-and-raspberry-pi-3-b/

FreeBSD – Raspberry Pi 3B+ – Remote Access Console.
https://blackdot.be/2018/08/remote-access-console-using-raspberry-pi-3b-and-freebsd/

FreeBSD 12.x has LUA loader enabled by default.
https://twitter.com/bsdimp/status/1031638933690441728

In Other BSDs for 2018/08/18.
https://www.dragonflydigest.com/2018/08/18/21609.html

Shared library load order randomization in HardenedBSD for use with Firefox/Chromium/Iridium.
https://twitter.com/lattera/status/1030823681843507202

Researchers Blame ‘Monolithic’ Linux Code Base for Critical Vulnerabilities.
https://threatpost.com/researchers-blame-monolithic-linux-code-base-for-critical-vulnerabilities/136785/

2018/08/23 is the End of Life for NetBSD 6.x tree.
https://www.netbsd.org/changes/#netbsd6eol

Carlos Neira ZCAGE is now able to create BHYVE Branded Zones on Illumos.
https://bsd.network/@sehnsucht/100599247272911030
https://www.npmjs.com/package/zcage
https://asciinema.org/a/QLnjO8J2NVVPQrs3jh0EKEGta

FreeNAS 11.1-U6 Available.
https://twitter.com/FreeBSD_News/status/1032666675194167297
https://www.ixsystems.com/blog/library/freenas-11-1-u6/

FreeBSD vs. DragonFly BSD vs. Linux on AMD Threadripper 2990WX.
https://www.phoronix.com/scan.php?page=article&item=bsd-threadripper-2990wx

Disable SMT/Hyperthreading in all Intel BIOSes – Theo de Raadt.
https://marc.info/?l=openbsd-tech&m=153504937925732&w=2

OpenSSH 7.8 Released.
https://www.openssh.com/releasenotes.html#7.8

TRIM Consolidation on UFS/FFS Filesystems on FreeBSD.
https://lists.freebsd.org/pipermail/freebsd-current/2018-August/070797.html

FreeBSD vt(4) will now cache most recently drawn text to not redraw it.
https://reviews.freebsd.org/D16723

What is New in Solaris 11.4?
https://www.oracle.com/a/ocom/docs/dc/sev100738019-ww-us-on-ce1-ie1a-ev.html

OpenBSD Foundation gets first 2018 Iridium ($100K+) donation.
https://undeadly.org/cgi?action=article;sid=20180824145543

How to Run a More Secure Browser.
https://www.dragonflybsd.org/docs/docs/handbook/RunSecureBrowser/

Hardware

IBM POWER9 E950 and E980 Servers Launched.
https://www.servethehome.com/ibm-power9-e950-and-e980-servers-launched/

Intel Microcode EULA Prohibits Benchmarking!
https://twitter.com/RaptorEng/status/1031919319909892096
https://pastebin.com/raw/J8MXpPdh

GIGABYTE Cavium ThunderX2 1U and 2U Systems.
https://www.anandtech.com/show/13234/gigabyte-starts-sales-of-cavium-thunderx2-to-general-customers

Fujitsu Presents Post-K arm64 A64FXโ„ข CPU CPU Specifications with 48 Computing Cores and 4 Assistant Cores.
http://www.fujitsu.com/global/about/resources/news/press-releases/2018/0822-02.html

A4000TX ATX Motherboard.
http://www.amibay.com/showthread.php?101477-A4000TX-ATX-Amiga-motherboard

IBM POWER9 Scale Up CPUs with Huge IO and Effective 32 Channel DDR4.
https://www.servethehome.com/ibm-power9-hc30/

Life

Why We Sleep by Matthew Walker review – how more sleep can save your life.
https://www.theguardian.com/books/2017/sep/21/why-we-sleep-by-matthew-walker-review
https://youtube.be/pwaWilO_Pig

Bullshit jobs and the yoke of managerial feudalism.
https://www.economist.com/open-future/2018/06/29/bullshit-jobs-and-the-yoke-of-managerial-feudalism

Why Garbagemen Should Earn More Than Bankers.
https://evonomics.com/why-garbage-men-should-earn-more-than-bankers/

Solitude.
https://www.pa-mar.net/Lifestyle/Solitude.html

Akrasia Effect – Why We Dont Follow Through on What We Set Out to Do and What to Do About It.
https://jamesclear.com/akrasia

Other

Move/migrate Oracle and MySQL databases to PostgreSQL.
http://www.ora2pg.com/start.html
https://github.com/darold/ora2pg/releases

LIDL Killed SAP Migration After Spending 500 Million Dollars.
https://it.toolbox.com/blogs/clintonjones/lidl-cans-sap-project-after-spending-half-a-billion-073118

All BlackHat 2018 Attendee Registration Data Hacked and Available via Unauthenticated API.
https://ninja.style/post/bcard/
https://twitter.com/binitamshah/status/1032084847345459204

GOG Launches FCKDRM to Promote DRM-Free Art and Media.
https://torrentfreak.com/gog-launches-fckdrm-to-promote-drm-free-art-and-media-180822/

EOF