Simple FreeBSD Poudriere Harvester Guide

The Poudriere is one of the topics I kinda omitted. I thought that official pkg(8) packages are more then enough and that occasional needed package recompilations are not the reasons to use a harvester such as Poudriere for that task. What was my needed recompilations? Usually the audio/lame because FreeBSD did not provided package for it for more then a decade while OpenBSD did … and multimedia/ffmpeg just to include that lame(1) support. Also at the beginning of exFAT format the sysutils/exfat-utils and sysutils/fusefs-exfat needed to be built by hand because of additional license agreement.

The Table of Contents for this article:

• Poudriere Name
• Poudriere Features
• FreeBSD Host Setup
• Poudriere Setup
• Poudriere Jails
• Poudriere Ports Tree
• Poudriere Packages to Build List
• Poudriere Options
• Nginx
• Memcached
• Ccache
• Poudriere Packages Build Process
• Our Repository Client Setup
• Next Builds
• Summary

There was as time when I had a dedicated script that would do just that – rebuild several ports as packages after the pkg upgrade cycle.

% cat pkg-recompile.sh
#! /bin/sh

# OPTIONS
  PORTS='audio/lame multimedia/ffmpeg sysutils/exfat-utils sysutils/fusefs-exfat'

# ONLY root CAN BUILD PACKAGES/PORTS
if [ "$( whoami )" != "root" ]
then
  echo "ER: only 'root' may use this script"
  exit 1
fi

# BUILD PACKAGES
case ${1} in
  # REBUILD PACKAGES
  (b|build)
    for PORT in ${PORTS}
    do
      pkg unlock -y ${PORT} 1> /dev/null 2> /dev/null
      env BATCH=yes DISABLE_VULNERABILITIES=yes \
        make -C /usr/ports/${PORT} build deinstall install clean &
      MAKE=${!}
      rctl -a process:${MAKE}:pcpu:deny=40
      wait ${MAKE}
      pkg lock -y ${PORT}
    done
    ;;

  # CLEAN
  (c|clean)
    for PORT in ${PORTS}
    do
      make -C /usr/ports/${PORT} clean
    done
    ;;

  # USAGE
  (*)
    echo "usage: ${0##*/} b|c|build|clean"
    exit 1
    ;;

esac

Now none of that is needed anymore … unless you want to Connect FreeBSD to FreeIPA/IDM server … this is the case where Poudriere comes pretty handy. You may configure/rebuild needed packages by hand or use a tool that will do that for you and you will then just use its custom build packages repository to install them on multiple systems. Scale often changes many things and this is not different with Poudriere tool.

Poudriere Name

… its quite unfortunate to say the least. I needed some time to actually learn to remember that name properly. Not sure I have any useful tips here – I just split it in half to make it easier to remember – as poud and riere parts. In the beginning I interpreted the pond part as British Pound … uncorrected of course. I really wish the author would name it simpler – like Rebuild or Bob the Builder for example 🙂

So what does Poudriere really means? Its a French translation of Powder Keg – which means a place where gunpowder was stored. There is some logic in that … as all the power (gun power) is in the packages … and Poudriere creates that place.

UPDATE: Dan Langille just reminded me that Poudriere replaced Powder Keg which was a similar tool – that explains the French translation for the name 🙂

Poudriere Features

The Poudriere is a bulk package builder and port tester. It uses a ‘clean’ FreeBSD Jails containers to build packages for defined FreeBSD version (supported or not) and a ‘snapshot’ of a FreeBSD Ports tree. Most of the time it will be the latest FreeBSD Ports tree ‘snapshot’ but nothing prevents you from using older ‘snapshots’ with older packages versions when needed.

Then the results (and logs) of these builds are available as HTML pages and you can (and probably should) host them as some WWW server.

All of this seems scary, complicated and pointless bloat to some … but it gets simple and obvious once you try it. You know me – I have an allergy for bullshit and bloat and Poudriere is really far from both of them.

Its kinda like with the famous UNIX co-creator Dennis Ritchie quote – “UNIX is basically a simple operating system, but you have to be a genius to understand the simplicity.” – just kidding – Poudriere actually is quite simple once you start using it – and that is probably the biggest obstacle to knowing it better. Just start using it (in a VM or in a Jail) to get to know it better and once you know it better – setup it properly and just use. Its far from over-complicated solutions such as SELinux, systemd(1) or Kubernetes.

FreeBSD Host Setup

First things first. The ‘three kings’ of and FreeBSD system configuration:

• /boot/loader.conf
• /etc/rc.conf
• /etc/sysctl.conf

Here they are:

The /boot/loader.conf file.

# cat /boot/loader.conf
security.bsd.allow_destructive_dtrace=0
kern.geom.label.disk_ident.enable=0
kern.geom.label.gptid.enable=0
cryptodev_load=YES
zfs_load=YES

The /etc/rc.conf file.

# cat /etc/rc.conf
  clear_tmp_enable=YES
  syslogd_flags="-ss"
  sendmail_enable=NONE
  hostname=fbsdpr
  ifconfig_em0="inet 10.0.10.123/24"
  defaultrouter="10.0.10.1"
  sshd_enable=YES
  dumpdev=AUTO
  zfs_enable=YES
  nginx_enable=YES
  memcached_enable=YES
  memcached_flags="-l localhost -m 8192"

The /etc/sysctl.conf file.

# cat /etc/sysctl.conf
vfs.zfs.min_auto_ashift=12

From the three above only the /etc/rc.conf is important as the other two only have settings from the bsdinstall(8) installer – as used with the Auto ZFS option.

We will also need to populate /etc/resolv.conf file to have DNS configured.

# echo nameserver 1.1.1.1 > /etc/resolv.conf

Pick your personal DNS server favorite here if the Cloudflare one does not suit your needs.

Poudriere Setup

First we need to install some packages – especially the Poudriere package – to make them more up-to-date we would prefer the latest branch of pkg(8) packages.

# sed -i '' -e 's|quarterly|latest|g' /etc/pkg/FreeBSD.conf

# grep latest /etc/pkg/FreeBSD.conf
  url: "pkg+http://pkg.FreeBSD.org/${ABI}/latest",

Now we can add needed packages.

# env ASSUME_ALWAYS_YES=yes \
    pkg install -y \
      poudriere \
      portmaster \
      screen \
      tmux \
      zsh \
      beadm \
      lsblk \
      nginx \
      git-lite \
      htop \
      tree \
      ccache-memcached \
      memcached \
      groff

Some of them are less needed but they definitely does not hurt for workflow. Absolute minimum are:

• poudriere
• nginx
• git-lite
• ccache-memcached
• groff

Now … lets prepare some dirs and certs to make our packages signed.

# SSL=/usr/local/etc/ssl
# mkdir -p   ${SSL}/{keys,certs} /usr/ports/distfiles
# chmod 0600 ${SSL}/keys
# openssl genrsa -out ${SSL}/keys/poudriere.key 4096
# openssl rsa -in ${SSL}/keys/poudriere.key -pubout -out ${SSL}/certs/poudriere.cert

As we will be using ZFS for Poudriere we will now create dedicated dataset for it.

# zfs create -o mountpoint=/usr/local/poudriere zroot/poudriere

Now we will create the Poudriere config file.

# cat << EOF > /usr/local/etc/poudriere.conf
ZPOOL=zroot
BASEFS=/usr/local/poudriere
ZROOTFS=/usr/local/poudriere
FREEBSD_HOST=ftp://ftp.freebsd.org
POUDRIERE_DATA=/usr/local/poudriere/data
CHECK_CHANGED_OPTIONS=verbose
CHECK_CHANGED_DEPS=yes
PKG_REPO_SIGNING_KEY=/usr/local/etc/ssl/keys/poudriere.key
URL_BASE=http://10.0.10.123/
USE_TMPFS=yes
TMPFS_LIMIT=8
MAX_MEMORY=8
MAX_FILES=2048
DISTFILES_CACHE=/usr/ports/distfiles
KEEP_OLD_PACKAGES=yes
KEEP_OLD_PACKAGES_COUNT=3
CHECK_CHANGED_OPTIONS=verbose
CHECK_CHANGED_DEPS=yes
CCACHE_DIR=/var/ccache
RESTRICT_NETWORKING=no
EOF

Its quite basic – yet it will do the job.

Poudriere Jails

Next we will create Poudriere Jails for each FreeBSD version we want to create pkg(8) repositories with packages.

I will create Jails for all FreeBSD 13.x versions – supported or not.

# poudriere jail -c -j 13-2-R-amd64 -v 13.2-RELEASE
# poudriere jail -c -j 13-1-R-amd64 -v 13.1-RELEASE
# poudriere jail -c -j 13-0-R-amd64 -v 13.0-RELEASE -m ftp-archive

Keep in mind that you need to specify additional -m ftp-archive argument for unsupported FreeBSD versions.

After some time you will end up with ready to use Poudriere FreeBSD Jails containers as shown below.

# poudriere jail -l
13-0-R-amd64 13.0-RELEASE-p13 amd64 ftp-archive 2023-04-28 03:22:05 /usr/local/poudriere/jails/13-0-R-amd64
13-1-R-amd64 13.1-RELEASE-p7  amd64 http        2023-04-27 23:17:13 /usr/local/poudriere/jails/13-1-R-amd64
13-2-R-amd64 13.2-RELEASE     amd64 http        2023-04-27 23:15:30 /usr/local/poudriere/jails/13-2-R-amd64

Poudriere Ports Tree

You may already have an up-to-date FreeBSD Ports tree on your disk at usual /usr/ports location – but we need Poudriere to get its own one too.

# poudriere ports -c

After fetching one you can list it like that.

# poudriere ports -l
default   git+https 2023-04-27 06:16:42 /usr/local/poudriere/ports/default

Poudriere Packages to Build List

Here is the best part – you do not need to build all 33000+ ports – you may specify just several of them. This is what we would do now.

# cat << EOF > /usr/local/etc/poudriere.d/list
sysutils/beadm
sysutils/lsblk
devel/m4
EOF

Accepting all possible licenses is also a good idea.

# echo DISABLE_LICENSES=yes >> /usr/local/etc/poudriere.d/make.conf

We will also specify which options should (and should not) be included in our built packages.

# cat << EOF > /usr/local/etc/poudriere.d/13-0-R-amd64-make.conf
OPTIONS_UNSET+= DOCS NLS X11 EXAMPLES
ALLOW_UNSUPPORTED_SYSTEM=yes
DISABLE_LICENSES=yes
EOF

# cat << EOF > /usr/local/etc/poudriere.d/13-1-R-amd64-make.conf
OPTIONS_UNSET+= DOCS NLS X11 EXAMPLES
ALLOW_UNSUPPORTED_SYSTEM=yes
DISABLE_LICENSES=yes
EOF

# cat << EOF > /usr/local/etc/poudriere.d/13-2-R-amd64-make.conf
OPTIONS_UNSET+= DOCS NLS X11 EXAMPLES
ALLOW_UNSUPPORTED_SYSTEM=yes
DISABLE_LICENSES=yes
EOF

Poudriere Options

You may choose the Poudriere packages options the same interactive way you do when you use the FreeBSD Ports tree.

To do that – here are the needed spells.

# poudriere options -c -j 13-0-R-amd64 -f /usr/local/etc/poudriere.d/list
# poudriere options -c -j 13-1-R-amd64 -f /usr/local/etc/poudriere.d/list
# poudriere options -c -j 13-2-R-amd64 -f /usr/local/etc/poudriere.d/list

Nginx

To make most of the Poudriere you will also need some web server. I have chosen Nginx for that task as its currently ‘the’ standard for the Internet.

Its setup is not complicated – just repeat steps below and you are done.

# service nginx enable

# sed -i '' -E 's|text/plain[\t\ ]*txt|text/plain txt log|g' /usr/local/etc/nginx/mime.types

# cat << EOF > /usr/local/etc/nginx/nginx.conf
events {
  worker_connections 1024;
}

http {
  include      mime.types;
  default_type application/octet-stream;

  server {
    listen 80 default;
    server_name ${IP};
    root /usr/local/share/poudriere/html;

    location /data {
      alias /usr/local/poudriere/data/logs/bulk;
      autoindex on;
    }

    location /packages {
      root /usr/local/poudriere/data;
      autoindex on;
    }
  }
}
EOF

# service nginx restart

The /usr/local/etc/nginx/mime.types part will allow you to display the *.log files in the browser instead of ‘forcing’ the browser to pointlessly download them.

Our web server seems to work properly.

# sockstat -l4
USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS
www      nginx      95263 7  tcp4   *:80                  *:*
root     nginx      95262 7  tcp4   *:80                  *:*
root     sshd       706   5  tcp4   *:22                  *:*

Memcached

As we will be using devel/ccache-memcached package to speed up builds – we would also need memcached to be running.

We already have needed configuration in /etc/rc.conf file so we only need to start it.

# grep memcached /etc/rc.conf
  memcached_enable="YES"
  memcached_flags="-l localhost -m 8192"

# service memcached restart

# sockstat -l4
USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS
nobody   memcached  37844 13 tcp4   127.0.0.1:11211       *:*
www      nginx      69727 7  tcp4   *:80                  *:*
root     nginx      69726 7  tcp4   *:80                  *:*
root     sshd       706   5  tcp4   *:22                  *:*

Ccache

We will now configure ccache(1) service.

Like success have many father the same ccache(1) have many configs – to be sure – we will propagate all of them.

We will use /var/ccache dir for the ccache(1) cache dir = but feel free to put it somewhere else or even with a dedicated ZFS dataset.

# mkdir -p /var/ccache

# cat << EOF > /usr/local/etc/ccache.conf
max_size = 0
cache_dir = /var/ccache
base_dir = /var/ccache
memcached_conf = --SERVER=localhost:11211
memcached_only = true
EOF

# cat << EOF > /root/.ccache/ccache.conf
max_size = 0
cache_dir = /var/ccache
base_dir = /var/ccache
memcached_conf = --SERVER=localhost:11211
memcached_only = true
EOF

# cat << EOF > /var/ccache/ccache.conf
max_size = 0
cache_dir = /var/ccache
base_dir = /var/ccache
memcached_conf = --SERVER=localhost:11211
memcached_only = true
EOF

The ccache(1) stats output after several builds below.

# ccache -s
cache directory                     /var/ccache
primary config                      /var/ccache/ccache.conf
secondary config      (readonly)    /usr/local/etc/ccache.conf
stats updated                       Fri Apr 28 04:10:17 2023
cache hit (direct)                  4510
cache hit (preprocessed)             713
cache miss                          2481
cache hit rate                     67.80 %
called for link                     5616
called for preprocessing            1476
multiple source files                 18
compile failed                      1143
preprocessor error                   351
bad compiler arguments                72
unsupported source language            9
autoconf compile/link               3147
no input file                        441
cleanups performed                     0
files in cache                      6303
cache size                          26.6 MB

Poudriere Packages Build Process

Now as you have everything configured and ready – you may build your custom packages.

These next commands will build repositories with your configured packages.

# poudriere bulk -j 13-0-R-amd64 -f /usr/local/etc/poudriere.d/list
# poudriere bulk -j 13-1-R-amd64 -f /usr/local/etc/poudriere.d/list
# poudriere bulk -j 13-2-R-amd64 -f /usr/local/etc/poudriere.d/list

The Poudriere console output is pretty colorful and nice – here is how it looks.

Lets check how it look on the browser side.

Lets ‘click’ the 13-2-R-amd64-default name for some details.

It shows that 14 packages has been built in the process and all of them succeed.

Lets now click the date_time build name.

You will now see the details about that build run – with logs and time needed for each package build.

You can ‘click’ on the package name to get the build log details.

Here is how our three repositories look after the build process.

# ls -l /usr/local/poudriere/data/packages
total 26K
drwxr-xr-x 4 root wheel 14 2023-04-28 04:10 13-0-R-amd64-default/
drwxr-xr-x 4 root wheel 14 2023-04-28 04:06 13-1-R-amd64-default/
drwxr-xr-x 4 root wheel 14 2023-04-28 04:07 13-2-R-amd64-default/

# tree /usr/local/poudriere/data/packages
/usr/local/poudriere/data/packages
├── 13-0-R-amd64-default
│   ├── All -> .latest/All
│   ├── Latest -> .latest/Latest
│   ├── meta.conf -> .latest/meta.conf
│   ├── meta.pkg -> .latest/meta.pkg
│   ├── meta.txz -> .latest/meta.txz
│   ├── packagesite.pkg -> .latest/packagesite.pkg
│   └── packagesite.txz -> .latest/packagesite.txz
├── 13-1-R-amd64-default
│   ├── All -> .latest/All
│   ├── Latest -> .latest/Latest
│   ├── meta.conf -> .latest/meta.conf
│   ├── meta.pkg -> .latest/meta.pkg
│   ├── meta.txz -> .latest/meta.txz
│   ├── packagesite.pkg -> .latest/packagesite.pkg
│   └── packagesite.txz -> .latest/packagesite.txz
└── 13-2-R-amd64-default
    ├── All -> .latest/All
    ├── Latest -> .latest/Latest
    ├── meta.conf -> .latest/meta.conf
    ├── meta.pkg -> .latest/meta.pkg
    ├── meta.txz -> .latest/meta.txz
    ├── packagesite.pkg -> .latest/packagesite.pkg
    └── packagesite.txz -> .latest/packagesite.txz

10 directories, 15 files

Our Repository Client Setup

Now we need to configure our FreeBSD clients to start using our Poudriere created repositories.

Here is what we need to do on such client.

# mkdir -p /usr/local/etc/pkg/repos

# cat << EOF > /usr/local/etc/pkg/repos/13-2-R-amd64.conf
13-2-R-amd64: {
    url: "http:/10.0.10.123/packages/13-2-R-amd64-default/",
    mirror_type: "http",
    signature_type: "pubkey",
    pubkey: "/usr/local/etc/ssl/certs/poudriere.cert",
    enabled: yes,
    priority: 100
}
EOF

# pkg update -f
Updating FreeBSD repository catalogue...
Fetching meta.conf: 100%    163 B   0.2kB/s    00:01
Fetching packagesite.pkg: 100%    7 MiB 626.5kB/s    00:11
Processing entries: 100%
FreeBSD repository update completed. 32980 packages processed.
Updating 13-2-R-amd64-HEAD repository catalogue...
Fetching meta.conf: 100%    163 B   0.2kB/s    00:01
Fetching packagesite.pkg: 100%    6 KiB   5.7kB/s    00:01
Processing entries: 100%
13-2-R-amd64-HEAD repository update completed. 14 packages processed.
All repositories are up to date.

We now have our first FreeBSD client system configured against our Poudriere created repository.

Lets install/update the m4 package for a test.

# pkg install m4
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
Updating 13-2-R-amd64-HEAD repository catalogue...
13-2-R-amd64-HEAD repository is up to date.
All repositories are up to date.
The following 2 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        m4: 1.4.19,1 [13-2-R-amd64-HEAD]

Installed packages to be REINSTALLED:
        pkg-1.19.1_1 [13-2-R-amd64-HEAD] (options changed)

Number of packages to be installed: 1
Number of packages to be reinstalled: 1

The process will require 2 MiB more space.
9 MiB to be downloaded.

Proceed with this action? [y/N]: y
[1/2] Fetching pkg-1.19.1_1.pkg: 100%    8 MiB   8.7MB/s    00:01
[2/2] Fetching m4-1.4.19,1.pkg: 100%  214 KiB 218.7kB/s    00:01
Checking integrity... done (0 conflicting)
[1/2] Reinstalling pkg-1.19.1_1...
[1/2] Extracting pkg-1.19.1_1: 100%
[2/2] Installing m4-1.4.19,1...
[2/2] Extracting m4-1.4.19,1: 100%

Now some FreeBSD client notes … if you are a Linux fan you probably know that – for example on Red Hat Linux (and its clones) – its relatively easy to just list the configured repositories with yum repolist command.

# yum repolist
repo id         repo name
appstream       CentOS Linux 8 - AppStream
baseos          CentOS Linux 8 - BaseOS
epel            Extra Packages for Enterprise Linux 8 - x86_64
epel-modular    Extra Packages for Enterprise Linux Modular 8 - x86_64
extras          CentOS Linux 8 - Extras

… unfortunately there is not 1:1 equivalent on FreeBSD side for pkg(8) repositories.

The closest one that is available out of the box are:

# pkg -vv | grep -A 999 '^Repositories:'
Repositories:
  FreeBSD: {
    url             : "pkg+http://pkg.FreeBSD.org/FreeBSD:13:amd64/latest",
    enabled         : yes,
    priority        : 0,
    mirror_type     : "SRV",
    signature_type  : "FINGERPRINTS",
    fingerprints    : "/usr/share/keys/pkg"
  }
  13-2-R-amd64: {
    url             : "http://10.0.10.123/packages/13-2-R-amd64-default/",
    enabled         : yes,
    priority        : 100,
    mirror_type     : "HTTP",
    signature_type  : "PUBKEY",
    pubkey          : "/usr/local/etc/ssl/certs/poudriere.cert"
  }

# grep -h '^[^#]' /etc/pkg/* /usr/local/etc/pkg/repos/*
FreeBSD: {
  url: "pkg+http://pkg.FreeBSD.org/${ABI}/latest",
  mirror_type: "srv",
  signature_type: "fingerprints",
  fingerprints: "/usr/share/keys/pkg",
  enabled: yes
}
13-2-R-amd64: {
    url: "http://10.0.10.123/packages/13-2-R-amd64-default/",
    mirror_type: "http",
    signature_type: "pubkey",
    pubkey: "/usr/local/etc/ssl/certs/poudriere.cert",
    enabled: yes,
    priority: 100
}

Not very pretty.

After asking at GitHub for potential solution or pkg(8)man page overlook I was informed that there is no such option … so I came with my own POSIX /bin/sh shell scripts/functions for the rescue 🙂

For example pkg-repo-var-en.sh that will list all pkg(8) repositories with their enabled/disabled status.

# ./pkg-repo-var-en.sh
REPO          ENABLED  PRIO  URL
FreeBSD       yes      0     pkg+http://pkg.FreeBSD.org/${ABI}/latest
13-2-R-amd64  yes      100   http://10.0.10.123/packages/13-2-R-amd64

# cat pkg-repo-var-en.sh
#! /bin/sh

( # HEADER
  echo -e "\nREPO ENABLED PRIO URL"
  for REPO in /etc/pkg/* /usr/local/etc/pkg/repos/*
  do
    REPOCUR=$( grep '^[^#]' "${REPO}" )

    # REPO
    echo "${REPOCUR}" | awk -F ':' '/\{[\ \t]*$/ {printf(" %s ",$1)}'

    # ENABLED
    echo "${REPOCUR}" | awk '/enabled:/ {printf(" %s ",$NF)}' | tr -cd '[a-zA-Z ]'

    # PRIO
    if echo "${REPOCUR}" | grep -q priority
    then
      echo "${REPOCUR}" | awk '/priority:/ {printf(" %s ",$NF)}' | tr -cd '[0-9 ]'
    else
      echo -n " 0 "
    fi

    # URL
    echo "${REPOCUR}" | grep '^[^#]' | awk -F'"' '/url:/ {print $2}' | tr -d ','

  done 2> /dev/null
) | column -t 2> /dev/null

Next Builds

So … you have successfully build your custom repository once … these are the steps to create new up-to-date package every next time.

// UPDATE JAILS
# poudriere jail -u -j 13-0-R-amd64
# poudriere jail -u -j 13-1-R-amd64
# poudriere jail -u -j 13-2-R-amd64

// UPDATE PORTS
# poudriere ports -u

// BUILD REPOSITORIES
# poudriere bulk -j 13-0-R-amd64 -f /usr/local/etc/poudriere.d/list
# poudriere bulk -j 13-1-R-amd64 -f /usr/local/etc/poudriere.d/list
# poudriere bulk -j 13-2-R-amd64 -f /usr/local/etc/poudriere.d/list

For example – if there are not new packages/changes – this is how the Poudriere output would look like.

Summary

Not sure what should I add here more … I definitely need to compare Poudriere against ports-mgmt/synth alternative one day … but it is not this day – as Aragorn once said 🙂

Pick your packages to customize and have fun.

Take care.

Silent Fanless Dell Wyse 3030 LT FreeBSD Server

Why use quite outdated (released in 2016) Dell Wyse 3030 LT terminal under FreeBSD? There are probably many answers to that question. I recently started to use these for my backup purposes … and to be honest I am really pleased with them.

In the past I used to make my own boxes based on Mini ITX motherboards such as these:

• Silent Fanless FreeBSD Server/Desktop
• Silent Fanless FreeBSD Server – DIY Backup
• Silent Fanless FreeBSD Server – Redundant Backup

… and to be honest – I was pretty pleased with their service. The reason that I started to look for alternative and settled on Dell Wyse 3030 LT terminals were Pico ITX power supplies. Both of them failed somewhere between now and 2018. I got replacements from China but then these boxes (without any active cooling) started to overheat. I also tried to replace motherboards with other ones – but still had the same problem. I really wanted to avoid active cooling because it creates a lot of pointless dust. Both inside and outside the case.

My current point of view is that these used Dell Wyse 3030 LT terminals are both cheaper at buy and later at usage costs as they draw a lot less power then the Mini ITX motherboard based ones.

Dell Wyse 3030 LT

The Dell Wyse 3030 LT terminals are not that much powerful – they should be treated as Raspberry Pi boards more then like a casual mini PC. Both because of their power draw and compute power. Here is how such Dell Wyse 3030 LT terminal looks like. I put a 2.5 USB attached hard disk on top of it – its WD Elements with 5 TB capacity. The USB pendrive that is plugged into one of the front USB ports is Lexar S47 32GB USB drive. This is where FreeBSD 13.2-RC6 is installed. I will of course update it when 13.2-RELEASE is ready to rumble.

You may ask why WD Elements and its a good question. The answer is I do not care. I just sort the drives for ‘best price’ order and buy the cheapest one. The other Dell Wyse 3030 LT terminal I got has Seagate Basic Portable 2.5 USB drive instead – also 5 TB in capacity. Works the same well. Cost almost the same. I do not remember which one was better deal but they both were very close.

Ports

The Dell Wyse 3030 LT terminal has quite a few useful ports that most of them I find useful.

These are:

• (1) — 1 x USB 3.0 (front)
• (2) — 2 x USB 2.0 (front)
• (3) — 1 x Mini Jack (front)
• (4) — 1 x LAN 10/100/1000 Realtek RTL8111/8168/8411 (back)
• (5) — 2 x DisplayPort (back)
• (6) — 1 x USB 2.0 (back)
• (*) — 1 x WLAN Intel AC-7265 (internal)

The hardware specs are not that great for today’s standards.

MODEL: Dell Wyse 3030 LT
  CPU: Intel Celeron N2807 2 x 1.58 GHz (2.16 GHz Turbo)
  RAM: 2 GB DDR3L (soldered on motherboard)
 eMMC: 4 GB Flash (soldered on motherboard)
 mPCI: Intel Dual Band Wireless-AC 7265

I use ZFS filesystem for my data and one may say that 2 GB RAM is not enough for ZFS needs … but OpenZFS page recommends 2 GB RAM as a minimum so I do not have to worry here.

Have enough memory: A minimum of 2GB of memory is recommended for ZFS. Additional memory is strongly recommended when the compression and deduplication features are enabled.

The source of that information comes from OpenZFS FAQ page.

In the past I have successfully run a 512 MB box with 2 x 2 TB ZFS mirror for several years. I have had zero crashes/panics/problems with it. Everything just worked like it suppose to. I also run several simple services on the same 512 MB box – like NFS and SAMBA services to export the available data to machines on the local network. The only thing I did back then was to limit the ARC usage up to 128 MB of RAM. Nothing more.

With current 2 GB RAM that I have configured ARC to be 128-256 MB with following settings in the /etc/sysctl.conf file.

# ZFS ARC
# ZFS TUNING
  vfs.zfs.arc.min=134217728
  vfs.zfs.arc.max=268435456

The details about that Intel Celeron N2807 CPU can be found on Intel ARK database.

Here are its specs. The most interesting ones are TDP(W).

The lscpu(1) shows following information about it.

wyse3030 % lscpu
Architecture:            amd64
Byte Order:              Little Endian
Total CPU(s):            2
Thread(s) per core:      1
Core(s) per socket:      2
Socket(s):               1
Vendor:                  GenuineIntel
CPU family:              6
Model:                   55
Model name:              Intel(R) Celeron(R) CPU  N2807  @ 1.58GHz
Stepping:                8
L1d cache:               24K
L1i cache:               32K
L2 cache:                1024K
Flags:                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 cflsh ds acpi mmx fxsr sse sse2 ss htt tm pbe sse3 pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm sse4_1 sse4_2 movbe popcnt tsc_deadline rdrnd tsc_adjust smep erms fpcsds syscall nx rdtscp lm lahf_lm

The FreeBSD information about it from /var/run/dmesg.boot looks as follows.

CPU: Intel(R) Celeron(R) CPU  N2807  @ 1.58GHz (1583.40-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x30678  Family=0x6  Model=0x37  Stepping=8
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x41d8e3bf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,MOVBE,POPCNT,TSCDLT,RDRAND>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x101<LAHF,Prefetch>
  Structured Extended Features=0x2282<TSCADJ,SMEP,ERMS,NFPUSG>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics

Several C-states and P-states are properly supported on FreeBSD as shown below.

wyse3030 % sysctl dev.cpu.0
dev.cpu.0.temperature: 44.0C
dev.cpu.0.coretemp.throttle_log: 0
dev.cpu.0.coretemp.tjmax: 100.0C
dev.cpu.0.coretemp.resolution: 1
dev.cpu.0.coretemp.delta: 56
dev.cpu.0.cx_method: C1/hlt C2/io
dev.cpu.0.cx_usage_counters: 21488971 0
dev.cpu.0.cx_usage: 100.00% 0.00% last 409us
dev.cpu.0.cx_lowest: C1
dev.cpu.0.cx_supported: C1/1/1 C2/3/104
dev.cpu.0.freq_levels: 2501/45000 2500/45000 2000/33977 1800/29872 1600/25926 1400/22148 1200/18525 1000/15060 800/11752
dev.cpu.0.freq: 1400
dev.cpu.0.%parent: acpi0
dev.cpu.0.%pnpinfo: _HID=none _UID=0 _CID=none
dev.cpu.0.%location: handle=\_PR_.CPU0
dev.cpu.0.%driver: cpu
dev.cpu.0.%desc: ACPI CPU

Keep in mind to load coretemp(4) module at system startup to get all needed information.

The output from sensors.sh is available below.

wyse3030 # sensors.sh

            BATTERY/AC/TIME/FAN/SPEED
------------------------------------
               dev.cpu.0.cx_supported: C1/1/1 C2/2/500 C3/3/1500
                   dev.cpu.0.cx_usage: 26.04% 12.27% 61.68% last 125us
                       dev.cpu.0.freq: 498
                hw.acpi.cpu.cx_lowest: C8
                            powerd(8): running

                  SYSTEM/TEMPERATURES
------------------------------------
                dev.cpu.0.temperature: 37.0C (max: 105.0C)
                dev.cpu.1.temperature: 40.0C (max: 105.0C)
      hw.acpi.thermal.tz0.temperature: 26.9C (max: 90.1C)

                   DISKS/TEMPERATURES
------------------------------------
        smart.da1.temperature_celsius: 37.0C

The biggest drawback is the lack of support for aesni(4) acceleration. Test I had made – with powerd(8) enabled – shows that its able to reach 15-16 MB/s of speed with calculating that encryption on these two cores. Its more then enough for my needs as I rarely go above 10-11 MB/s on the WiFi transfers.

If you are interested how the internals of Dell Wyse 3030 LT motherboard looks like – take a journey to David Parkinson page at Dell Wyse 3030 LT location. You will also find a lot information about other Thin Client software – often also called Terminals.

I did not bothered to unscrew mine knowing what David Parkinson shows on his site.

WiFi

As the Dell Wyse 3030 LT has the Intel Dual Band Wireless-AC 7265 WiFi card from 2014 – almost a decade old WiFi card – there is a big chance that FreeBSD will flawlessly support it and at least allow 802.11n protocol and speed – as its 9 years old.

Here are some specs from Intel on that WiFi card.

… and here is the neat part 😀 – it does not work on FreeBSD – its not able to connect to WiFi access point – what were you thinking 🙂

This is what I tried at both 2.4GHz and 5.0GHz WiFi networks that are WPA2 protected.

wyse3030 # grep iwm /var/run/dmesg.boot 
iwm0: <Intel(R) Dual Band Wireless AC 7265> mem 0x88500000-0x88501fff at device 0.0 on pci2
iwm0: hw rev 0x210, fw ver 22.361476.0, address d0:57:7b:13:41:4c

wyse3030 # pciconf -lv iwm0
iwm0@pci0:2:0:0: class=0x028000 rev=0x59 hdr=0x00 vendor=0x8086 device=0x095a subvendor=0x8086 subdevice=0x5410
    vendor = 'Intel Corporation'
    device = 'Wireless 7265'
    class = network

wyse3030 # sysctl -n net.wlan.devices
iwm0

wyse3030 # ifconfig wlan0 create wlandev $( !! )

wyse3030 # ifconfig wlan0 up

wyse3030 # ifconfig wlan0 scan
SSID/MESH ID            BSSID              CHAN RATE    S:N     INT CAPS
wifinet                 d8:07:b6:aa:bb:00    7   54M  -63:-96   100 EP   APCHANREP WPA RSN BSSLOAD HTCAP VHTCAP VHTOPMODE
wifinet                 d8:07:b6:aa:bb:02   40   54M  -80:-96   100 EP   VHTPWRENV APCHANREP WPA RSN HTCAP VHTCAP 

These are my WiFi networks that I will try to use:

• WiFi 2.4GHz – wifinet – d8:07:b6:aa:bb:00
• WiFi 5.0GHz – wifinet – d8:07:b6:aa:bb:02

Results below.

WiFi 2.4GHz

wyse3030 # wpa_supplicant -i wlan0 -c /etc/wpa_supplicant.conf
Successfully initialized wpa_supplicant
ioctl[SIOCS80211, op=20, val=0, arg_len=7]: Invalid argument
ioctl[SIOCS80211, op=20, val=0, arg_len=7]: Invalid argument
ioctl[SIOCS80211, op=103, val=0, arg_len=128]: Operation now in progress
wlan0: CTRL-EVENT-SCAN-FAILED ret=-1 retry=1
wlan0: Trying to associate with d8:07:b6:aa:bb:00 (SSID='wifinet' freq=2442 MHz)
wlan0: Authentication with d8:07:b6:aa:bb:00 timed out.
wlan0: CTRL-EVENT-DISCONNECTED bssid=d8:07:b6:aa:bb:00 reason=3 locally_generated=1
BSSID d8:07:b6:aa:bb:00 ignore list count incremented to 2, ignoring for 10 seconds
wlan0: CTRL-EVENT-DSCP-POLICY clear_all
wlan0: Trying to associate with d8:07:b6:aa:bb:00 (SSID='wifinet' freq=2442 MHz)
wlan0: Authentication with d8:07:b6:aa:bb:00 timed out.
wlan0: CTRL-EVENT-DISCONNECTED bssid=d8:07:b6:aa:bb:00 reason=3 locally_generated=1
BSSID d8:07:b6:aa:bb:00 ignore list count incremented to 2, ignoring for 10 seconds
wlan0: CTRL-EVENT-DSCP-POLICY clear_all
wlan0: Trying to associate with d8:07:b6:aa:bb:00 (SSID='wifinet' freq=2442 MHz)
wlan0: Authentication with d8:07:b6:aa:bb:00 timed out.
wlan0: CTRL-EVENT-DISCONNECTED bssid=d8:07:b6:aa:bb:00 reason=3 locally_generated=1
BSSID d8:07:b6:aa:bb:00 ignore list count incremented to 2, ignoring for 10 seconds
wlan0: CTRL-EVENT-DSCP-POLICY clear_all
wlan0: Trying to associate with d8:07:b6:aa:bb:00 (SSID='wifinet' freq=2442 MHz)
^C

WiFi 5.0GHz

wyse3030 # wpa_supplicant -i wlan0 -c /etc/wpa_supplicant.conf
Successfully initialized wpa_supplicant
ioctl[SIOCS80211, op=20, val=0, arg_len=7]: Invalid argument
ioctl[SIOCS80211, op=20, val=0, arg_len=7]: Invalid argument
wlan0: Trying to associate with d8:07:b6:aa:bb:02 (SSID='wifinet' freq=5200 MHz)
wlan0: Authentication with d8:07:b6:aa:bb:02 timed out.
wlan0: CTRL-EVENT-DISCONNECTED bssid=d8:07:b6:aa:bb:02 reason=3 locally_generated=1
BSSID d8:07:b6:aa:bb:02 ignore list count incremented to 2, ignoring for 10 seconds
wlan0: CTRL-EVENT-DSCP-POLICY clear_all
wlan0: Trying to associate with d8:07:b6:aa:bb:02 (SSID='wifinet' freq=5200 MHz)
wlan0: Authentication with d8:07:b6:aa:bb:02 timed out.
wlan0: CTRL-EVENT-DISCONNECTED bssid=d8:07:b6:aa:bb:02 reason=3 locally_generated=1
BSSID d8:07:b6:aa:bb:02 ignore list count incremented to 2, ignoring for 10 seconds
wlan0: CTRL-EVENT-DSCP-POLICY clear_all
wlan0: Trying to associate with d8:07:b6:aa:bb:02 (SSID='wifinet' freq=5200 MHz)
wlan0: Authentication with d8:07:b6:aa:bb:02 timed out.
wlan0: CTRL-EVENT-DISCONNECTED bssid=d8:07:b6:aa:bb:02 reason=3 locally_generated=1
BSSID d8:07:b6:aa:bb:02 ignore list count incremented to 2, ignoring for 10 seconds
wlan0: CTRL-EVENT-DSCP-POLICY clear_all
wlan0: Trying to associate with d8:07:b6:aa:bb:02 (SSID='wifinet' freq=5200 MHz)
wlan0: Authentication with d8:07:b6:aa:bb:02 timed out.
wlan0: CTRL-EVENT-DISCONNECTED bssid=d8:07:b6:aa:bb:02 reason=3 locally_generated=1
BSSID d8:07:b6:aa:bb:02 ignore list count incremented to 2, ignoring for 10 seconds
wlan0: CTRL-EVENT-SSID-TEMP-DISABLED id=0 ssid="wifinet" auth_failures=1 duration=10 reason=CONN_FAILED
wlan0: CTRL-EVENT-DSCP-POLICY clear_all
wlan0: CTRL-EVENT-SSID-REENABLED id=0 ssid="wifinet"
wlan0: Trying to associate with d8:07:b6:aa:bb:02 (SSID='wifinet' freq=5200 MHz)
wlan0: CTRL-EVENT-DISCONNECTED bssid=d8:07:b6:aa:bb:02 reason=3 locally_generated=1
wlan0: CTRL-EVENT-DSCP-POLICY clear_all
wlan0: CTRL-EVENT-DSCP-POLICY clear_all
wlan0: CTRL-EVENT-TERMINATING 
^C

… and to be fair – I have used latest and greatest FreeBSD 13.2-RC6 (the 13.2-RELEASE is probably available now). I even installed the comms/iwmbt-firmware package hoping it would solve anything – it does not change anything.

For the record – this is how properly working WiFi connection looks like on FreeBSD.

# ifconfig wlan0 up 
# wpa_supplicant -i wlan0 -c /etc/wpa_supplicant.conf
Successfully initialized wpa_supplicant
ioctl[SIOCS80211, op=20, val=0, arg_len=7]: Invalid argument
ioctl[SIOCS80211, op=20, val=0, arg_len=7]: Invalid argument
wlan0: Trying to associate with d8:07:b6:aa:bb:00 (SSID='wifinet' freq=2442 MHz)
wlan0: Associated with d8:07:b6:aa:bb:00
wlan0: WPA: Key negotiation completed with d8:07:b6:aa:bb:00 [PTK=CCMP GTK=CCMP]
wlan0: CTRL-EVENT-CONNECTED - Connection to d8:07:b6:aa:bb:00 completed [id=43 id_str=]

At this point (after you get the CTRL-EVENT-CONNECTED message) you hit [CTRL]+[Z] then type bg at the prompt to keep wpa_supplicant(8) process in the background – and then dhclient wlan0 to get the IP address, default gateway and DNS servers.

The best FreeBSD can get on that box when it comes to WiFi side is Realtek RTL8188CUS tiny USB dongle … but while its 802.11n capable – the FreeBSD is only able to support 802.11g speeds on it. Details here – Realtek RTL8188CUS – USB 802.11n WiFi Review – in one of my older articles.

You now probably understand why I have these two WiFi antennas detached 🙂 – but for the record – I have tested the WiFi connection with both antennas attached to that Intel AC 7265 card.

LAN

While WiFi on FreeBSD sucks greatly the builtin LAN – Realtek RTL8111/8168/8411 – works quite well with re(4) driver … but it needs one spacial quirk to work 🙂

All network connections need to be restarted at the end of the boot process to work – if you do not do that – you just do not have network connectivity at all – I was not able to track why its fucked up like that – so I went with the idea to use /etc/rc.local scripts as its already there and its really late in the boot process. Here are mine /etc/rc.local script contents that seem to workaround that fuckup well.

wyse3030 % cat /etc/rc.local
#! /bin/sh

# RESTART NETWORKING
  (
  service netif   restart 1> /dev/null 2> /dev/null
  service routing restart 1> /dev/null 2> /dev/null
  ) &

To not waste time I put both netif and routing services in the background for startup.

DisplayPort

The DisplayPort sockets just work – when attached to the 1080p monitor they just deliver. I tried only in the vt(4) console mode. I did not tried X11 mode – but I suspect that the CPU is so old that it will work flawlessly with i915kms.ko driver from graphics/drm-fbsd13-kmod package.

When it comes to early booting – set your native screen resolution in the /boot/loader.conf file as specified below.

# CONSOLE RESOLUTION # --------------------------------------------------------
  kern.vt.fb.default.mode=1920x1080    # FORCE FHD
  efi_max_resolution=1920x1080         # FORCE FHD

Also – if you (like me) like to have small/smaller font size in the terminal – this will also be helpful in the /boot/loader.conf file.

  screen.font=6x12          # USE SMALLEST FONT NO MATTER THE RESOLUTION

USB

The USB ports work as desired. Both USB 2.0 and USB 3.0 ports work without any issues.

I currently use Lexar S47 32 GB USB tiny dongle foe FreeBSD system. I did not yet tested the builtin eMMC 4 GB flash storage space – will probably add some UPDATE about it in the future.

Data

Lets get back to the main responsibility of that box – backups. In the past my main NAS system used to have two 5 TB disks in ZFS mirror – but as I have another offsite copy and another offline copy and I also have my main copy always with me – I now find it pointless to have this single NAS mirrored. Besides – its more then 3-2-1 backup rule anyway …

As this box has 4 USB ports one can of course even create raidz ZFS pool with 3 drives for data and 1 for parity. Nothing stops you from doing that. You will get 15 TB of usable storage that way … not sure how log would it take to rebuild through. As these are SMR drives you may consider raidz2 or mirror instead. I also have some tips for ZFS on SMR Drives here.

Costs

This is the neat part … and I am not joking here. My earlier setups cost about $110 to $120 only for the case/motherboard/PSUs/RAM/etc. You needed to add disk costs to that on top. But not with this one. I got several of these Dell Wyse 3030 LT terminals for less then $18 each … and it was not a one time offer – they just cost that much now.

The disks (5 TB 2.5 USB drives) that I use also got cheaper in time and its relatively easy to get a used one for about $80. Does not matter if its Seagate or Western Digital or whatever. They are very similar cheap 5400 rpm SMR drives.

This make my new backup solution cost about $100 for a 5 TB space of storage.

Configuration

This is the main FreeBSD config /etc/rc.conf file.

wyse3030 % cat /etc/rc.conf
# BOOT/SILENCE # -------------------------------------------------------------
  rc_startmsgs=NO
  rc_info=NO

# NETWORK # ------------------------------------------------------------------
  hostname=wyse3030
  ifconfig_re0="inet 10.0.0.2/24"
  defaultrouter="10.0.0.1"
  defaultroute_delay=3
  defaultroute_carrier_delay=3
  gateway_enable=YES

# MODULES/COMMON/BASE # ------------------------------------------------------
  kld_list="${kld_list} fusefs coretemp sem cpuctl ichsmb"

# POWER # --------------------------------------------------------------------
  performance_cx_lowest=Cmax
  economy_cx_lowest=Cmax
  powerd_enable=YES
  powerd_flags="-n adaptive -a hiadaptive -b adaptive -m 100 -M 2000"

# DAEMONS # ------------------------------------------------------------------
  syslogd_flags="-ss"
  sshd_enable=YES
  zfs_enable=YES
  sendmail_enable=NO
  sendmail_submit_enable=YES
  sendmail_outbound_enable=NO
  sendmail_msp_queue_enable=NO

# OTHER # --------------------------------------------------------------------
  keyrate=fast
  virecover_enable=NO
  update_motd=NO
  hostid_enable=NO
  entropy_file=NO
  savecore_enable=NO
  clear_tmp_enable=YES
  dumpdev=AUTO

Options that can be changed anytime at /etc/sysctl.conf file below.

wyse3030 % cat /etc/sysctl.conf
# SECURITY # ------------------------------------------------------------------
  security.bsd.see_jail_proc=0
  security.bsd.unprivileged_proc_debug=0

# SECURITY/RANDOM PID # -------------------------------------------------------
  kern.randompid=1

# ANNOYING THINGS # -----------------------------------------------------------
  vfs.usermount=1
  kern.coredump=0
  hw.syscons.bell=0
  kern.vt.enable_bell=0

# ZFS ASHIFT 4k BLOCK SIZE # --------------------------------------------------
  vfs.zfs.min_auto_ashift=12

# ZFS DELETE FUCKUP TRIM # ----------------------------------------------------
  vfs.zfs.vdev.trim_max_active=1

# ZFS ARC 128MB FORCE # -------------------------------------------------------
  vfs.zfs.arc.min=134217728
  vfs.zfs.arc.max=268435456

# JAILS/ALLOW UPGRADES IN JAILS # ---------------------------------------------
  security.jail.chflags_allowed=1

# JAILS/ALLOW RAW SOCKETS # ---------------------------------------------------
  security.jail.allow_raw_sockets=1

# ALLOW idprio(8) USE BY REGULAR USER # ---------------------------------------
  security.bsd.unprivileged_idprio=1

… and last but not least (actually first :> to be read) the /boot/loader.conf file.

wyse3030 % cat /boot/loader.conf
# CONSOLE COMMON # ------------------------------------------------------------
  autoboot_delay=2          # USE '-1' => NO WAIT | USE 'NO' => INFINITE WAIT
  loader_logo=none          # LOGO POSIBILITES: fbsdbw beastiebw beastie none
  loader_menu_frame=none    # REMOVE FRAMES FROM FreeBSD BOOT MENU
  screen.font=6x12          # USE SMALLEST FONT NO MATTER THE RESOLUTION

# CONSOLE RESOLUTION # --------------------------------------------------------
  kern.vt.fb.default.mode=1920x1080    # FORCE FHD
  efi_max_resolution=1920x1080         # FORCE FHD

# MODULES # -------------------------------------------------------------------
  geom_eli_load=YES
  cryptodev_load=YES
  zfs_load=YES

# DISABLE /dev/diskid/* AND /dev/gptid/* ENTRIES FOR DISKS # ------------------
  kern.geom.label.disk_ident.enable=0
  kern.geom.label.gptid.enable=0

# COLORS # --------------------------------------------------------------------
  kern.vt.color.0.rgb="#000000"
  kern.vt.color.1.rgb="#dc322f"
  kern.vt.color.2.rgb="#859900"
  kern.vt.color.3.rgb="#b58900"
  kern.vt.color.4.rgb="#268bd2"
  kern.vt.color.5.rgb="#ec0048"
  kern.vt.color.6.rgb="#2aa198"
  kern.vt.color.7.rgb="#94a3a5"
  kern.vt.color.8.rgb="#586e75"
  kern.vt.color.9.rgb="#cb4b16"
  kern.vt.color.10.rgb="#859900"
  kern.vt.color.11.rgb="#b58900"
  kern.vt.color.12.rgb="#268bd2"
  kern.vt.color.13.rgb="#d33682"
  kern.vt.color.14.rgb="#2aa198"
  kern.vt.color.15.rgb="#6c71c4"

# RACCT/RCTL RESOURCE LIMITS # ------------------------------------------------
  kern.racct.enable=1

# ZFS TUNING # ----------------------------------------------------------------
  vfs.zfs.prefetch_disable=1

# POWER MANAGEMENT POWER OFF DEVICES WITHOUT ATTACHED DRIVER # ----------------
  hw.pci.do_power_nodriver=3

# POWER MANAGEMENT FOR EVERY USED AHCI CHANNEL (ahcich 0-7) # -----------------
  hint.ahcich.0.pm_level=5
  hint.ahcich.1.pm_level=5
  hint.ahcich.2.pm_level=5
  hint.ahcich.3.pm_level=5
  hint.ahcich.4.pm_level=5
  hint.ahcich.5.pm_level=5
  hint.ahcich.6.pm_level=5
  hint.ahcich.7.pm_level=5

# DISABLE DESTRUCTIVE DTrace # ------------------------------------------------
  security.bsd.allow_destructive_dtrace=0

There is also this 5 TB disk drive configuration to be mentioned.

wyse3030 # gpart destroy -F da0
wyse3030 # dd if=/dev/zero of=/dev/da0 bs=1m status=progress count=10
wyse3030 # gpart create -s GPT da0
wyse3030 # gpart add -t freebsd-zfs -a 4k da0
wyse3030 # geli init -s 4096 da0p1
wyse3030 # geli attach da0p1

Why I use password based encryption and not based on key? If someone would steal that device then with attached key on – for example – USB drive – all the data would be accessible – and all encryption and effort would be lost. With password based encryption whoever steals the devices gets some nice /dev/random input from my disk because its encrypted.

In the end these boxes do not hang by themselves and when there is power outage – its not that common like twice a week. Its more like once a quarter – I do not mind typing my password once in a quarter to make my data safe from people that should not be allowed to access it.

ZFS Pool Configuration

I did not used anything spectacular for ZFS configuration – just disabled atime and enabled compression that would save some space. I also increased the recordsize to 1M so the compressratio would be a little better … but nothing more to be honest.

wyse3030 # zpool history wd
History for 'wd':
2023-04-05.14:46:01 zpool create wd /dev/da0p1.eli
2023-04-05.14:46:07 zfs set compression=zstd wd
2023-04-05.14:46:12 zfs set atime=off wd
2023-04-05.14:46:15 zfs set recordsize=1m wd
2023-04-05.14:51:46 zfs create wd/data
2023-04-05.14:51:50 zfs set mountpoint=none wd
2023-04-05.14:52:00 zfs set mountpoint=/data wd/data

As you probably guessed the wd name is for the Western Digital 5 TB drive. If you are curious (and you probably are) the second ZFS pool is called seagate 🙂

Last time I used the LZ4 compression for the ZFS pool. Not I use the ZSTD compression and the compressratio improved from 1.04 to 1.07 – almost twice more efficient.

wyse3030 % zfs get compressratio
NAME                                      PROPERTY       VALUE  SOURCE
wd                                        compressratio  1.07x  -
wd/data                                   compressratio  1.07x  -

One may think that 7% of more data space is almost ‘non-existent’ – think again. When it comes to terabytes it makes a big difference. In my case when my real data takes 4.2 TB and compressed one takes 3.9 TB its … 245 GB of space. For free. Thank you ZFS.

Quirks

The ‘quirks’ are things that need to be done differently then usual to make the device ‘happy’ – to make it work the way we want.

One of the Dell Wyse 3030 LT quirks is its bot process … it does not need any additional configuration – I just want to mention it – so you will not be stressed about it. After the FreeBSD loader(8) menu there is slight pause before continuing into the ‘rest’ of the boot process. Its several seconds long and its harmless. Here is how it looks on attached screen.

Power Usage Comparison

So its dirt cheap … and takes HALF of the power then my previews Mini ITX solution. The Mini ITX based systems used about 10.4 W of power when at idle state.

After measuring the Dell Wyse 3030 LT terminal power consumption with a kill-a-watt physical device – its only about 4.5 W only … and this is with % TB of 2.5 disk attached at idle state.

Another interesting fact is that I used the same ‘external’ power supply that I used for these Mini ITX boxes.

Its 2.5A at 12V (which means 30 W)- exactly what a Dell Wyse 3030 LT terminal needs – according to its specification. As you can see from the photo above – the official power supply – while quite similar in size – needs a lot more space because of the additional 3-pin power cord – which is quite thick and takes a lot more space.

As in the past – I used python(1) [1] processes to load the CPU and dd(8) commands to load the drives I/O subsystem.

[1] # echo '999999999999999999 ** 999999999999999999' | python
[2] # dd if=/dev/da0  of=/dev/null  bs=1M
[3] # dd if=/dev/zero of=/data/FILE bs=1M

I will also include older Mini ITX based system measurements.

BOX        POWER  CPU          I/O
Wyse3030   3.8 W  IDLE         IDLE
MiniITX   10.5 W  IDLE         IDLE


BOX        POWER  CPU          I/O
Wyse3030  10.3 W  8 Thread(s)  3 DISK READ Thread(s) + 3 DISK WRITE Thread(s)
MiniITX   17.8 W  8 Thread(s)  3 DISK READ Thread(s) + 3 DISK WRITE Thread(s)

So after cutting CAPEX costs from about $110 to $15 I also lowered the OPEX price in half.

What can I say … I really appreciate and like the price/performance ratio 🙂

Cloud Storage Costs Comparison

I am not a fan of cloud based storage – its just damn overpriced to say the least.

In the past I have made multiple calculations about cloud storage costs and it will be no different this time. We need to assume some cost of 1kWh of power. In Poland its about $0.25 for 1kWh which means running a device that draws 1000 Watts for entire hour.

Assuming that our Dell Wyse 3030 LT uses about 4.5W all the time it would make the following costs.

  COST  TIME 
 $0.03  1 DAY(s)
 $9.96  1 YEAR(s)
$29.88  3 YEAR(s)
$49.80  5 YEAR(s)

Combining that with server cost ($100) we get TCO for our self hosted 5TB storage service.

COST  TIME
$110  1 YEAR(s)
$130  3 YEAR(s)
$150  5 YEAR(s)

This time after searching for cheapest cloud based storage I found these services.

• Amazon Drive
• Amazon S3 Glacier Storage
• Backblaze B2 Cloud Storage
• Google One
• Dropbox Basic
  

Here is its cost summarized for 1 year period for 5TB of data.

PRICE  TIME       SERVICE
 $300  1 YEAR(s)  Amazon Drive
 $310  1 YEAR(s)  Google One
 $240  1 YEAR(s)  Amazon S3 Glacier Storage
 $450  1 YEAR(s)  Backblaze B2 Cloud Storage
 $360  1 YEAR(s)  Dropbox Basic

For the Backblaze B2 Cloud Storage I assumed average between upload/download price because upload is two times cheaper then download.

Here is its cost summarized for 5 year period for 5 TB of data.

PRICE  TIME       SERVICE
 $150  5 YEAR(s)  Dell Wyse 3030 LT + 5 TB USB 2.5 Drive
 $230  5 YEAR(s)  Dell Wyse 3030 LT + 5 TB USB 2.5 Drive (assuming drive failed)
$1500  5 YEAR(s)  Amazon Drive
$1550  5 YEAR(s)  Google One
$1200  5 YEAR(s)  Amazon S3 Glacier Storage
$2250  5 YEAR(s)  Backblaze B2 Cloud Storage
$1800  5 YEAR(s)  Dropbox Basic

… but there is also a catch here. Recently I was able to get a lifetime offer for 1 TB of cloud storage. It cost me about $170 for that 1 TB.

This is where it gets interesting – for 5 TB we would have to pay $850 for lifetime cloud storage. Lets calculate where the ‘personal’ NAS is comparable to ‘lifetime’ cloud storage.

PRICE   TIME       SERVICE
 $230   5 YEAR(s)  Dell Wyse 3030 LT + 5 TB USB 2.5 Drive => assuming drive failed 1 time(s)
 $460  10 YEAR(s)  Dell Wyse 3030 LT + 5 TB USB 2.5 Drive => assuming drive failed 2 time(s)
 $690  15 YEAR(s)  Dell Wyse 3030 LT + 5 TB USB 2.5 Drive => assuming drive failed 3 time(s)
 $920  20 YEAR(s)  Dell Wyse 3030 LT + 5 TB USB 2.5 Drive => assuming drive failed 4 time(s)
 $850  UNLIMITED   {5TB unlimited time cloud vendor}

So … it will take you about 20 years of your life (and an assumption that a drive will fail every 5 years with 4 drive fails total) to make it less expensive then the unlimited cloud storage. To be honest – I would get such unlimited cloud storage for most important files and documents for example in 1 TB or less size. Keep an encrypted backup copy the most important files and you are doing better then the classic 3-2-1 backup rule … assuming that you already have one online and one offline backup and also one onsite backup and one offsite backup … and at least two different medium types used 🙂

Drawbacks

WiFi is not supported and network requires restart after boot. Besides these I did not find any other issues.

Connect FreeBSD 13.2 to FreeIPA/IDM

About half a year ago I wrote about how to Connect FreeBSD to FreeIPA/Red Hat Identity Management with available then FreeBSD 13.1-RELEASE version. Time flies and we have newer FreeBSD release around the corner. FreeBSD 13.2 is currently at RC5 level and I believe we should see 13.2-RELEASE official announcement within hours – but even if it would take RC6 and RELEASE being available in weeks later – the instructions will not change for this setup.

This guide will focus only on FreeBSD part of configuring and building packages with needed options to make it connect (and work with) FreeIPA/IDM properly. To have a working FreeIPA/IDM server you need to do instructions from the FreeIPA section of the mentioned above article – up to the FreeBSD Client section. Then – when specified in the article – also the Finish Setup with Web Browser in FreeIPA/IDM Page section needs to be done from the earlier article.

In other words – this article updates the FreeBSD Client section with instructions to use latest upcoming FreeBSD 13.2-RELEASE and newer up-to-date packages versions.

One note about conventions (and colors).

% command                            // command executed on local system
[root@idm ~]# command                // command executed on FreeIPA/IDM server
# diff Makefile.OLD Makefile         // command executed on FreeBSD client system
< krb5>=1.10:security/krb5 \         // parts of configs/commands are colored for better visibility
> krb5-119>=1.10:security/krb5-119 \ // parts of configs/commands are colored for better visibility

FreeBSD Client

We will use these software versions as shown below.

# pkg info | grep -e krb -e sudo -e sssd -e openldap -e samba -e cyrus -e mkhomedir
cyrus-sasl-2.1.28           RFC 2222 SASL (Simple Authentication and Security Layer)
cyrus-sasl-gssapi-2.1.28    SASL GSSAPI authentication plugin
krb5-119-1.19.4             MIT implementation of RFC 4120 network authentication service
openldap26-client-2.6.4     Open source LDAP client implementation
pam_mkhomedir-0.2           Create HOME with a PAM module on demand
samba413-4.13.17_4          Free SMB/CIFS and AD/DC server and client for Unix
sssd-1.16.5_8               System Security Services Daemon
sudo-1.9.13p3               Allow others to run commands as root

This is the main /etc/rc.conf config file.

# cat /etc/rc.conf
  hostname="fbsd132.vercorp.org"
  ifconfig_em0="inet 10.0.0.50/24"
  defaultrouter="10.0.0.1"
  clear_tmp_enable=YES
  syslogd_flags="-ss"
  sshd_enable=YES
  dumpdev=AUTO
  zfs_enable=YES
  ntpd_enable=YES
  sssd_enable="YES"
  rc_info=NO
  rc_startmsgs=NO

FreeBSD Packages

We will fetch up-to-date FreeBSD Ports tree.

# portsnap auto

We will also configure make(1) to store all its compilation results under /usr/ports/obj dir.

# cat << EOF > /etc/make.conf
WRKDIRPREFIX=/usr/ports/obj
EOF

We will switch to latest branch of pkg(8) packages.

# sed -i '' s/quarterly/latest/g /etc/pkg/FreeBSD.conf

# grep latest /etc/pkg/FreeBSD.conf
  url: "pkg+http://pkg.FreeBSD.org/${ABI}/latest",

We will install packages that are needed for compilation and also essential pam_mkhomedir package.

# pkg install -y \
    autoconf \
    automake \
    bind-tools \
    c-ares \
    cmocka \
    dialog4ports \
    ding-libs \
    docbook-xsl \
    gettext-tools \
    gmake \
    ldb22 \
    libtool \
    libxslt \
    m4 \
    nss \
    pcre \
    pkgconf \
    pam_mkhomedir

Now we will install our ‘core’ packages – and delete them right after – to waste less time on compilation.

# pkg install   -y krb5-119 sudo cyrus-sasl cyrus-sasl-gssapi openldap26-client samba413

# pkg delete -f -y krb5-119 sudo cyrus-sasl cyrus-sasl-gssapi openldap26-client samba413

# pkg install   -y krb5-119

By default the security/krb5-119 package is built without LDAP option. This is what we need for this step. We will later rebuild it with LDAP option.

In case something changed between posting that article and LDAP option being included or not in the security/krb5-119 package – below are instructions to rebuild it without LDAP option.

# cd /usr/ports/security/krb5-119

# make config
[ ] LDAP

# make build deinstall install

Now we need to rebuild security/cyrus-sasl2-gssapi package with GSSAPI_MIT option enabled.

# cd /usr/ports/security/cyrus-sasl2-gssapi

# make config
(x) GSSAPI_MIT

# make build deinstall install

Next rebuild the net/openldap26-client package with GSSAPI option enabled.

# cd /usr/ports/net/openldap26-client

# make config
[x] GSSAPI

# make build deinstall install

Next we are back to security/krb5-119 package and now it needs to be rebuilt with LDAP option.

# cd /usr/ports/security/krb5-119

# make config
[x] LDAP

# make build deinstall install

Now with security/sssd we will need a small patch because without it security/sssd will try to use newer security/krb5-120 package – which does not work with FreeIPA/IDM for now.

# cd /usr/ports/security/sssd

# cp Makefile Makefile.OLD

# sed -i '' 's|krb5>=1.10:security/krb5|krb5-119>=1.10:security/krb5-119|g' Makefile

# diff Makefile.OLD Makefile
32c32
<               krb5>=1.10:security/krb5 \
---
>               krb5-119>=1.10:security/krb5-119 \

Now the rebuild instructions.

# cd /usr/ports/security/sssd

# make config
[x] SMB

# make build deinstall install

Now we need to delete security/sssd and security/sudo packages to rebuild them again … yes its stupid.

# pkg delete -f sudo sssd

We now need to build and install security/sudo without PAM option and with SSSD option and with GSSAPI_MIT option enabled.

# cd /usr/ports/security/sudo

# make config
[ ] PAM
[x] SSSD
--- Kerberos 5 Authentication
(*) GSSAPI_MIT

# make build deinstall install

FreeBSD Setup

We now move to the configuration process as we have our packages installed.

Create needed dirs.

# mkdir -p \
    /usr/local/etc/ipa \
    /var/log/sssd \
    /var/run/sss/private \
    /var/db/sss

Set system hostname.

# grep hostname /etc/rc.conf
  hostname="fbsd132.vercorp.org"

# hostname fbsd132.vercorp.org

# hostname
fbsd132.vercorp.org

# cat << EOF > /etc/hosts
::1             localhost localhost.my.domain
127.0.0.1       localhost localhost.my.domain
10.0.0.50       fbsd132.vercorp.org fbsd132
10.0.0.40       idm.vercorp.org idm
EOF

# cat /etc/hosts
::1             localhost localhost.my.domain
127.0.0.1       localhost localhost.my.domain
10.0.0.50       fbsd132.vercorp.org fbsd132
10.0.0.40       idm.vercorp.org idm

Fetch the FreeIPA/IDM certificate.

# fetch -o /usr/local/etc/ipa/ca.crt http://10.0.0.40/ipa/config/ca.crt
/usr/local/etc/ipa/ca.crt                             1635  B 8227 kBps    00s

FreeIPA/IDM Setup Part

We now need to execute instructions on /IDM to connect FreeBSD to it.

Add A and PTR DNS records for our 10.0.0.50 IP address with fbsd132.vercorp.org hostname.

We also need to generate key for fbsd132.vercorp.org system.

[root@idm ~]# kinit admin
Password for admin@VERCORP.ORG:

[root@idm ~]# ipa dnsrecord-add vercorp.org fbsd132 --a-rec=10.0.0.50 --a-create-reverse
  Record name: fbsd132
  A record: 10.0.0.50

[root@idm ~]# ipa host-add fbsd132.vercorp.org
--------------------------------
Added host "fbsd132.vercorp.org"
--------------------------------
  Host name: fbsd132.vercorp.org
  Principal name: host/fbsd132.vercorp.org@VERCORP.ORG
  Principal alias: host/fbsd132.vercorp.org@VERCORP.ORG
  Password: False
  Keytab: False
  Managed by: fbsd132.vercorp.org

[root@idm ~]# ipa-getkeytab -s idm.vercorp.org \
                            -p host/fbsd132.vercorp.org@VERCORP.ORG \
                            -k /root/fbsd132.vercorp.org.keytab
Keytab successfully retrieved and stored in: /root/fbsd132.vercorp.org.keytab

[root@idm ~]# cp \
                /root/fbsd132.vercorp.org.keytab \
                /usr/share/ipa/html/fbsd132.vercorp.org.keytab

[root@idm ~]# chmod 644 /usr/share/ipa/html/fbsd132.vercorp.org.keytab

Now we will get fbsd132.vercorp.org system key from FreeIPA/IDM server.

# fetch -o /usr/local/etc/ipa/krb5.keytab http://10.0.0.40/ipa/config/fbsd132.vercorp.org.keytab
/usr/local/etc/ipa/krb5.keytab                         176  B  975 kBps    00s

# chmod 600 /usr/local/etc/ipa/krb5.keytab

Now we will move to creating needed config files.

The /usr/local/etc/openldap/ldap.conf file.

# cat << EOF > /usr/local/etc/openldap/ldap.conf
BASE        dc=org,dc=vercorp
URI         ldap://idm.vercorp.org/
#SIZELIMIT  12
#TIMELIMIT  15
#DEREF      never
SASL_MECH   GSSAPI
SASL_REALM  VERCORP.ORG
ssl         start_tls
TLS_CACERT  /usr/local/etc/ipa/ca.crt
EOF

… and /etc/krb5.conf file.

# cat << EOF > /etc/krb5.conf
[libdefaults]
  default_realm = VERCORP.ORG
  default_keytab_name = FILE:/usr/local/etc/ipa/krb5.keytab
  default_tkt_enctypes = aes256-cts des-cbc-crc aes128-cts arcfour-hmac
  default_tgs_enctypes = aes256-cts des-cbc-crc aes128-cts arcfour-hmac
  dns_lookup_realm = false
  dns_lookup_kdc = false
  rdns = false
  ticket_lifetime = 24h
  forwardable = yes

[realms]
  VERCORP.ORG = {
    kdc = idm.vercorp.org:88
    master_kdc = idm.vercorp.org:88
    admin_server = idm.vercorp.org:749
    default_domain = vercorp.org
    pkinit_anchors = FILE:/usr/local/etc/ipa/ca.crt
}

[domain_realm]
  .vercorp.org = VERCORP.ORG
  vercorp.org = VERCORP.ORG

[logging]
  kdc = FILE:/var/log/krb5/krb5kdc.log
  admin_server = FILE:/var/log/krb5/kadmin.log
  kadmin_local = FILE:/var/log/krb5/kadmin_local.log
  default = FILE:/var/log/krb5/krb5lib.log
EOF

… and /usr/local/etc/sssd/sssd.conf file.

# cat << EOF > /usr/local/etc/sssd/sssd.conf

[domain/vercorp.org]
# debug_level = 9
cache_credentials = True
krb5_store_password_if_offline = True
krb5_realm = VERCORP.ORG
ipa_domain = vercorp.org
id_provider = ipa
auth_provider = ipa
access_provider = ipa
ipa_hostname = fbsd132.vercorp.org
chpass_provider = ipa
ipa_server = _srv_, idm.vercorp.org
ldap_tls_cacert = /usr/local/etc/ipa/ca.crt
krb5_keytab = /usr/local/etc/ipa/krb5.keytab

[sssd]
services = nss, pam, ssh, sudo
config_file_version = 2
domains = vercorp.org

[nss]
filter_users = root,toor
homedir_substring = /usr/home/%u

[pam]

[sudo]
# debug_level = 0x3ff0

[ssh]
EOF

# chmod 600 /usr/local/etc/sssd/sssd.conf

FreeBSD have user account under /usr/home to make sure /home also points there.

# ln -s /usr/home /home

Now we need modify several FreeBSD Base System files such as /etc/nsswitch.conf or PAM and SSH configuration.

# cat << EOF > /etc/nsswitch.conf
#
# nsswitch.conf(5) - name service switch configuration file
# $FreeBSD$
#
group: files sss
group_compat: nis
hosts: files dns
# netgroup: compat
networks: files
passwd: files sss
passwd_compat: nis
shells: files
services: compat
services_compat: nis
protocols: files
rpc: files
sudoers: sss files
netgroup: files
EOF

The /etc/pam.d/system file.

# cat << EOF > /etc/pam.d/system
#
# $FreeBSD$
#
# System-wide defaults
#

# auth
auth            sufficient      pam_opie.so             no_warn no_fake_prompts
auth            requisite       pam_opieaccess.so       no_warn allow_local
auth            sufficient      pam_krb5.so             no_warn try_first_pass
#auth           sufficient      pam_ssh.so              no_warn try_first_pass
auth            sufficient      /usr/local/lib/pam_sss.so       use_first_pass
auth            required        pam_unix.so             no_warn try_first_pass nullok

# account
#account        required        pam_krb5.so
account         required        pam_login_access.so
account         required        pam_unix.so
account         required        /usr/local/lib/pam_sss.so       ignore_unknown_user ignore_authinfo_unavail

# session
#session        optional        pam_ssh.so              want_agent
session         required        pam_lastlog.so          no_fail
session         required        /usr/local/lib/pam_mkhomedir.so mode=0700

# password
#password       sufficient      pam_krb5.so             no_warn try_first_pass
password        sufficient      /usr/local/lib/pam_sss.so       use_authtok
password        required        pam_unix.so             no_warn try_first_pass
EOF

The /etc/pam.d/sshd file.

# cat << EOF > /etc/pam.d/sshd
#
# $FreeBSD$
#
# PAM configuration for the "sshd" service
#

# auth
auth            sufficient      pam_opie.so             no_warn no_fake_prompts
auth            requisite       pam_opieaccess.so       no_warn allow_local
auth            sufficient      pam_krb5.so             no_warn try_first_pass
#auth           sufficient      pam_ssh.so              no_warn try_first_pass
auth            sufficient      /usr/local/lib/pam_sss.so       use_first_pass
auth            required        pam_unix.so             no_warn try_first_pass

# account
account         required        pam_nologin.so
#account        required        pam_krb5.so
account         required        pam_login_access.so
account         required        pam_unix.so
account         required        /usr/local/lib/pam_sss.so       ignore_unknown_user ignore_authinfo_unavail

# session
#session        optional        pam_ssh.so              want_agent
session         required        pam_permit.so
session         required        /usr/local/lib/pam_mkhomedir.so mode=0700

# password
#password       sufficient      pam_krb5.so             no_warn try_first_pass
password        sufficient      /usr/local/lib/pam_sss.so       use_authtok
password        required        pam_unix.so             no_warn try_first_pass
EOF

The /etc/ssh/ssh_config file.

# cat << EOF >> /etc/ssh/ssh_config
GSSAPIAuthentication yes
EOF

The /etc/ssh/sshd_config file.

# cat << EOF >> /etc/ssh/sshd_config
GSSAPIAuthentication yes
UsePAM yes
EOF

Automatic startup of sssd(8) daemon (not to confuse with sshd(8)) needs to be also configured.

# service sssd enable
# service sssd start

Now … go to the original Connect FreeBSD to FreeIPA/Red Hat Identity Management article and ‘do’ the Finish Setup with Web Browser in FreeIPA/IDM Page section instructions – all the way up to the FreeBSD FreeIPA Login Test section.

FreeBSD FreeIPA Login Test

Now we can try to login to our FreeBSD client … but first several other things.

If you just created your ‘regular’ FreeIPA/IDM user – you need to set a permanent password for it. Its not possible by first logging to the FreeBSD system. You will get following error on the FreeBSD client.

# grep sshd /var/log/messages
(...)
Mar 24 20:51:46 fbsd132 [sssd[krb5_child[779]]][779]: Password has expired
(...)

You need to change this password by logging into the FreeIPA/IDM Linux system. Here is how.

% ssh 10.0.0.40 -l vermaden
(vermaden@10.0.0.40) Password:
(vermaden@10.0.0.40) Password expired. Change your password now.
Current Password:
(vermaden@10.0.0.40) New password:
(vermaden@10.0.0.40) Retype new password:
Could not chdir to home directory /home/vermaden: No such file or directory
[vermaden@idm /]$ exit

Another thing – make sure that time is synchronized between the client and FreeIPA/IDM systems – you will get following error on the FreeBSD client if they are not in sync.

# grep sssd /var/log/messages
(...)
Mar 24 20:51:46 fbsd132 [sssd[krb5_child[779]]][779]: Clock skew too great
(...)

… or …

PAM: User account has expired for vermaden from 10.0.0.3

Here is fast way to sync their times against pool.ntp.org host.

Feel free to setup some permanent time sync solution on both of them. You can also enable NTP server on FreeIPA/IDM during its configuration and sync all FreeIPA/IDM clients against it.

# ntpdate pool.ntp.org

# date
Wed Mar 29 10:34:14 CEST 2023

[root@idm ~]# chronyd -q 'server pool.ntp.org iburst' || chronyc -a 'burst 4/4'

[root@idm ~]# date
Wed Mar 29 10:34:14 CEST 2023

Now we can finally login to our FreeBSD client.

% ssh 10.0.0.50 -l vermaden
(vermaden@10.0.0.50) Password:

vermaden@fbsd132:~ $ uname -prism
FreeBSD 13.2-RC3 amd64 amd64 GENERIC

vermaden@fbsd132:~ $ grep $( whoami ) /etc/passwd

vermaden@fbsd132:~ $ id
uid=1408200003(vermaden) gid=1408200000(admins) groups=1408200000(admins)

Done. Seems to work properly.

Latest Samba 4.16

You can also use latest Samba 4.16 version instead of the 4.13 default one.

The needed changes are below – to be executed before package building.

# cat << EOF >> /etc/make.conf
# net/samba416
DEFAULT_VERSIONS+=samba=4.16
EOF

Rebuilding process for net/samba416 package.

# cd /usr/ports/net/samba416

# make config
(*) GSSAPI_MIT

# make build deinstall install

Feel free to share your thoughts on this in the comments section.

EOF

Connect FreeBSD to FreeIPA/Red Hat Identity Management

Corporate needs are simple – one ring to rule them all place to get users from. On the open source path there are several ways to achieve that. Alongside ‘plain’ OpenLDAP there is also FreeIPA – which is open source and free version of the Red Hat Identity Management (IDM).

This guide will show you how to make basic FreeIPA install and connect a FreeBSD 13.1-RELEASE system to it.

The Table of Contents for this article looks as follows.

• Connect FreeBSD to FreeIPA/Red Hat Identity Management
• FreeIPA
  • FreeIPA Requirements
  • FreeIPA Setup
• FreeBSD Client
  • FreeBSD Packages
  • FreeBSD Setup
  • FreeIPA/IDM Setup Part
  • Finish Setup with Web Browser in FreeIPA/IDM Page
  • FreeBSD FreeIPA Login Test
• FreeBSD Jail as FreeIPA/IDM Client
  • Basic FreeBSD Jail Preparations
  • Configure FreeBSD Jail to Connect to FreeIPA/IDM Server
• Linux FreeIPA/IDM Client
  • FreeIPA/IDM Setup Part
  • Linux FreeIPA/IDM Client Setup

FreeIPA

As typical CentOS is dead (yes there is CentOS Stream available but its not a RHEL clone) I have two options here. Rocky Linux and Alma Linux. I will use the latter as it seems more popular and more up to date. In this guide the FreeIPA/IDM server will hosted on the Alma Linux 8.6 system.

To make things easier I installed that Alma Linux 8.6 with single 20GB / root on the /dev/sda1 partition. No separate /boot. No LVM. Just good old plain fucking simple single raw partition for everything. Seems no one does it these days 🙂

[root@idm ~]# lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda      8:0    0   20G  0 disk
└─sda1   8:1    0   20G  0 part /

FreeIPA Requirements

The minimum hardware requirements for installing FreeIPA server are as follows.

RAM: 4GB
CPU: 2
HDD: 10GB
DNS: fully qualified domain name for FreeIPA
     must be resolvable from DNS server configured in system

For this installation I have used the 10.0.0.0/24 network.

The FreeIPA/IDM server will get the 10.0.0.40 IP.

The FreeIPA/IDM Linux client rhlike.vercorp.org will get the 10.0.0.41 IP.

The FreeIPA/IDM FreeBSD client fbsd.vercorp.org will get the 10.0.0.42 IP.

The FreeIPA/IDM FreeBSD client fbsdjail.vercorp.org will get the 10.0.0.43 IP.

For lack of better ideas I have used vercorp.org/VERCORP.ORG as domain/realm names and idm.vercorp.org as the FreeIPA/IDM hostname.

Details of FreeIPA/IDM idm.vercorp.org system below.

      IP: 10.0.0.40/24
      GW: 10.0.0.1
     DNS: 1.1.1.1
hostname: idm.vercorp.org
  domain: vercorp.org
   realm: VERCORP.ORG

FreeBSD fbsd.vercorp.org system.

      IP: 10.0.0.42/24
      GW: 10.0.0.1
hostname: fbsd.vercorp.org

FreeBSD Jail fbsdjail.vercorp.org system.

      IP: 10.0.0.43/24
      GW: 10.0.0.1
hostname: fbsdjail.vercorp.org

If you are curious what is hidden in the 10.0.0.42 IP – then its a typical Alma Linux that I first used to try if the FreeIPA/IDM works at all 🙂

FreeIPA Setup

Because Anaconda is far from being a usable installer – this is how the only enp0s3 interface config looks like after manual intervention.

[root@idm ~]# cat /etc/sysconfig/network-scripts/ifcfg-enp0s3
NAME=enp0s3
DEVICE=enp0s3
TYPE=Ethernet
BOOTPROTO=none
UUID=b49cd1ab-d3eb-421d-b408-c052acc077da
ONBOOT=yes
IPADDR=10.0.0.40
NETMASK=255.255.255.0
GATEWAY=10.0.0.1
IPV6INIT=no
DNS1=1.1.1.1
PROXY_METHOD=none
BROWSER_ONLY=no
DEFROUTE=yes
IPV4_FAILURE_FATAL=no

Before we will go into setup procedures – we will update that Alma Linux system.

[root@idm ~]# yum update -y
[root@idm ~]# reboot

After the reboot we will disable the IPv6 stack for main interface (enp0s3) as FreeIPA installer has some problem with it. We do not need IPv6 here anyway so …

[root@idm ~]# cat << EOF >> /etc/sysctl.conf
# DISABLE IPv6 FOR MAIN enp0s3 INTERFACE
net.ipv6.conf.enp0s3.disable_ipv6=1
EOF

Lets set our FreeIPA/IDM hostname if we missed that at the Anaconda installer part. We will also setup the system time zone.

[root@idm ~]# hostnamectl set-hostname idm.vercorp.org

[root@idm ~]# hostnamectl
   Static hostname: idm.vercorp.org
         Icon name: computer-vm
           Chassis: vm
        Machine ID: b8bfb8bcc23147eb9cc7b62c72a09c32
           Boot ID: 06158ef430d9467d959076ab4396314e
    Virtualization: oracle
  Operating System: AlmaLinux 8.6 (Sky Tiger)
       CPE OS Name: cpe:/o:almalinux:almalinux:8::baseos
            Kernel: Linux 4.18.0-372.26.1.el8_6.x86_64
      Architecture: x86-64

[root@idm ~]# timedatectl set-timezone Europe/Warsaw

[root@idm ~]# timedatectl set-local-rtc 0

[root@idm ~]# timedatectl
               Local time: Mon 2022-10-17 15:08:28 CEST
           Universal time: Mon 2022-10-17 13:08:28 UTC
                 RTC time: Sat 2022-10-15 00:28:10
                Time zone: Europe/Warsaw (CEST, +0200)
System clock synchronized: yes
              NTP service: inactive
          RTC in local TZ: no

Lets add the system IP and name to the /etc/hosts file now.

[root@idm ~]# echo "$( hostname -i | awk '{print $NF}' )   $( hostname ) $( hostname -s )" >> /etc/hosts

[root@idm ~]# grep idm /etc/hosts
10.0.0.40   idm.vercorp.org idm

[root@idm ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
10.0.0.40   idm.vercorp.org idm

We need to enable the FreeIPA/IDM module in yum(8).

[root@idm ~]# yum module list idm
Last metadata expiration check: 0:00:31 ago on Tue 18 Oct 2022 01:02:51 PM CEST.
AlmaLinux 8 - AppStream
Name  Stream      Profiles                                  Summary
idm   DL1         adtrust, client, common [d], dns, server  The Red Hat Enterprise Linux Identity Management system module
idm   client [d]  common [d]                                RHEL IdM long term support client module

Hint: [d]efault, [e]nabled, [x]disabled, [i]nstalled

[root@idm ~]# yum module enable idm:DL1 -y
Last metadata expiration check: 0:01:06 ago on Tue 18 Oct 2022 01:02:51 PM CEST.
Dependencies resolved.
=====================================================================================================
 Package                 Architecture           Version                  Repository              Size
=====================================================================================================
Enabling module streams:
 389-ds                                         1.4
 httpd                                          2.4
 idm                                            DL1
 pki-core                                       10.6
 pki-deps                                       10.6

Transaction Summary
=====================================================================================================

Complete!

Some additional step needed.

[root@idm ~]# yum distro-sync
Last metadata expiration check: 0:10:07 ago on Mon 17 Oct 2022 03:14:32 PM CEST.
Dependencies resolved.
Nothing to do.
Complete!

We will now install FreeIPA/IDM with DNS as this setup is the most simplistic one. We focus on FreeBSD part here in that article.

[root@idm ~]# yum install -y bind-utils chrony nc
[root@idm ~]# yum module install idm:DL1/dns -y

One can use the ‘interactive’ installer and answer ‘by hand’ for all the asked questions – but to be honest I prefer to type my command once and make it ‘happen’ altogether without my time wasted.

As you probably guessed – we will use the unattended mode for the FreeIPA/IDM installer.

[root@idm ~]# ipa-server-install \
    --domain vercorp.org \
    --realm VERCORP.ORG \
    --reverse-zone=0.0.10.in-addr.arpa. \
    --no-forwarders \
    --no-ntp \
    --setup-dns \
    --ds-password    [password] \
    --admin-password [password] \
    --unattended

Checking DNS domain 0.0.10.in-addr.arpa., please wait ...

The log file for this installation can be found in /var/log/ipaserver-install.log
==============================================================================
This program will set up the IPA Server.
Version 4.9.8

This includes:
  * Configure a stand-alone CA (dogtag) for certificate management
  * Create and configure an instance of Directory Server
  * Create and configure a Kerberos Key Distribution Center (KDC)
  * Configure Apache (httpd)
  * Configure DNS (bind)
  * Configure SID generation
  * Configure the KDC to enable PKINIT

Excluded by options:
  * Configure the NTP client (chronyd)

Warning: skipping DNS resolution of host idm.vercorp.org
Checking DNS domain vercorp.org., please wait ...
Checking DNS domain 0.0.10.in-addr.arpa., please wait ...
Checking DNS domain 0.0.10.in-addr.arpa., please wait ...
Using reverse zone(s) 0.0.10.in-addr.arpa.
Trust is configured but no NetBIOS domain name found, setting it now.

The IPA Master Server will be configured with:
Hostname:       idm.vercorp.org
IP address(es): 10.0.0.40
Domain name:    vercorp.org
Realm name:     VERCORP.ORG

The CA will be configured with:
Subject DN:   CN=Certificate Authority,O=VERCORP.ORG
Subject base: O=VERCORP.ORG
Chaining:     self-signed

BIND DNS server will be configured to serve IPA domain with:
Forwarders:       No forwarders
Forward policy:   only
Reverse zone(s):  0.0.10.in-addr.arpa.

Disabled p11-kit-proxy
Configuring directory server (dirsrv). Estimated time: 30 seconds
  [1/41]: creating directory server instance
Validate installation settings ...
Create file system structures ...
Perform SELinux labeling ...
Create database backend: dc=vercorp,dc=org ...
Perform post-installation tasks ...
  [2/41]: tune ldbm plugin
  [3/41]: adding default schema
  [4/41]: enabling memberof plugin
  [5/41]: enabling winsync plugin
  [6/41]: configure password logging
  [7/41]: configuring replication version plugin
  [8/41]: enabling IPA enrollment plugin
  [9/41]: configuring uniqueness plugin
  [10/41]: configuring uuid plugin
  [11/41]: configuring modrdn plugin
  [12/41]: configuring DNS plugin
  [13/41]: enabling entryUSN plugin
  [14/41]: configuring lockout plugin
  [15/41]: configuring topology plugin
  [16/41]: creating indices
  [17/41]: enabling referential integrity plugin
  [18/41]: configuring certmap.conf
  [19/41]: configure new location for managed entries
  [20/41]: configure dirsrv ccache and keytab
  [21/41]: enabling SASL mapping fallback
  [22/41]: restarting directory server
  [23/41]: adding sasl mappings to the directory
  [24/41]: adding default layout
  [25/41]: adding delegation layout
  [26/41]: creating container for managed entries
  [27/41]: configuring user private groups
  [28/41]: configuring netgroups from hostgroups
  [29/41]: creating default Sudo bind user
  [30/41]: creating default Auto Member layout
  [31/41]: adding range check plugin
  [32/41]: creating default HBAC rule allow_all
  [33/41]: adding entries for topology management
  [34/41]: initializing group membership
  [35/41]: adding master entry
  [36/41]: initializing domain level
  [37/41]: configuring Posix uid/gid generation
  [38/41]: adding replication acis
  [39/41]: activating sidgen plugin
  [40/41]: activating extdom plugin
  [41/41]: configuring directory to start on boot
Done configuring directory server (dirsrv).
Configuring Kerberos KDC (krb5kdc)
  [1/10]: adding kerberos container to the directory
  [2/10]: configuring KDC
  [3/10]: initialize kerberos container
  [4/10]: adding default ACIs
  [5/10]: creating a keytab for the directory
  [6/10]: creating a keytab for the machine
  [7/10]: adding the password extension to the directory
  [8/10]: creating anonymous principal
  [9/10]: starting the KDC
  [10/10]: configuring KDC to start on boot
Done configuring Kerberos KDC (krb5kdc).
Configuring kadmin
  [1/2]: starting kadmin
  [2/2]: configuring kadmin to start on boot
Done configuring kadmin.
Configuring ipa-custodia
  [1/5]: Making sure custodia container exists
  [2/5]: Generating ipa-custodia config file
  [3/5]: Generating ipa-custodia keys
  [4/5]: starting ipa-custodia
  [5/5]: configuring ipa-custodia to start on boot
Done configuring ipa-custodia.
Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes
  [1/29]: configuring certificate server instance
  [2/29]: stopping certificate server instance to update CS.cfg
  [3/29]: backing up CS.cfg
  [4/29]: Add ipa-pki-wait-running
  [5/29]: secure AJP connector
  [6/29]: reindex attributes
  [7/29]: exporting Dogtag certificate store pin
  [8/29]: disabling nonces
  [9/29]: set up CRL publishing
  [10/29]: enable PKIX certificate path discovery and validation
  [11/29]: authorizing RA to modify profiles
  [12/29]: authorizing RA to manage lightweight CAs
  [13/29]: Ensure lightweight CAs container exists
  [14/29]: Ensuring backward compatibility
  [15/29]: starting certificate server instance
  [16/29]: configure certmonger for renewals
  [17/29]: requesting RA certificate from CA
  [18/29]: publishing the CA certificate
  [19/29]: adding RA agent as a trusted user
  [20/29]: configure certificate renewals
  [21/29]: Configure HTTP to proxy connections
  [22/29]: updating IPA configuration
  [23/29]: enabling CA instance
  [24/29]: importing IPA certificate profiles
  [25/29]: migrating certificate profiles to LDAP
  [26/29]: adding default CA ACL
  [27/29]: adding 'ipa' CA entry
  [28/29]: configuring certmonger renewal for lightweight CAs
  [29/29]: deploying ACME service
Done configuring certificate server (pki-tomcatd).
Configuring directory server (dirsrv)
  [1/3]: configuring TLS for DS instance
  [2/3]: adding CA certificate entry
  [3/3]: restarting directory server
Done configuring directory server (dirsrv).
Configuring ipa-otpd
  [1/2]: starting ipa-otpd
  [2/2]: configuring ipa-otpd to start on boot
Done configuring ipa-otpd.
Configuring the web interface (httpd)
  [1/22]: stopping httpd
  [2/22]: backing up ssl.conf
  [3/22]: disabling nss.conf
  [4/22]: configuring mod_ssl certificate paths
  [5/22]: setting mod_ssl protocol list
  [6/22]: configuring mod_ssl log directory
  [7/22]: disabling mod_ssl OCSP
  [8/22]: adding URL rewriting rules
  [9/22]: configuring httpd
Nothing to do for configure_httpd_wsgi_conf
  [10/22]: setting up httpd keytab
  [11/22]: configuring Gssproxy
  [12/22]: setting up ssl
  [13/22]: configure certmonger for renewals
  [14/22]: publish CA cert
  [15/22]: clean up any existing httpd ccaches
  [16/22]: enable ccache sweep
  [17/22]: configuring SELinux for httpd
  [18/22]: create KDC proxy config
  [19/22]: enable KDC proxy
  [20/22]: starting httpd
  [21/22]: configuring httpd to start on boot
  [22/22]: enabling oddjobd
Done configuring the web interface (httpd).
Configuring Kerberos KDC (krb5kdc)
  [1/1]: installing X509 Certificate for PKINIT
Done configuring Kerberos KDC (krb5kdc).
Applying LDAP updates
Upgrading IPA:. Estimated time: 1 minute 30 seconds
  [1/10]: stopping directory server
  [2/10]: saving configuration
  [3/10]: disabling listeners
  [4/10]: enabling DS global lock
  [5/10]: disabling Schema Compat
  [6/10]: starting directory server
  [7/10]: upgrading server
  [8/10]: stopping directory server
  [9/10]: restoring configuration
  [10/10]: starting directory server
Done.
Restarting the KDC
dnssec-validation yes
Configuring DNS (named)
  [1/12]: generating rndc key file
  [2/12]: adding DNS container
  [3/12]: setting up our zone
  [4/12]: setting up reverse zone
  [5/12]: setting up our own record
  [6/12]: setting up records for other masters
  [7/12]: adding NS record to the zones
  [8/12]: setting up kerberos principal
  [9/12]: setting up named.conf
created new /etc/named.conf
created named user config '/etc/named/ipa-ext.conf'
created named user config '/etc/named/ipa-options-ext.conf'
created named user config '/etc/named/ipa-logging-ext.conf'
  [10/12]: setting up server configuration
  [11/12]: configuring named to start on boot
  [12/12]: changing resolv.conf to point to ourselves
Done configuring DNS (named).
Restarting the web server to pick up resolv.conf changes
Configuring DNS key synchronization service (ipa-dnskeysyncd)
  [1/7]: checking status
  [2/7]: setting up bind-dyndb-ldap working directory
  [3/7]: setting up kerberos principal
  [4/7]: setting up SoftHSM
  [5/7]: adding DNSSEC containers
  [6/7]: creating replica keys
  [7/7]: configuring ipa-dnskeysyncd to start on boot
Done configuring DNS key synchronization service (ipa-dnskeysyncd).
Restarting ipa-dnskeysyncd
Restarting named
Updating DNS system records
Configuring SID generation
  [1/8]: creating samba domain object
  [2/8]: adding admin(group) SIDs
  [3/8]: adding RID bases
  [4/8]: updating Kerberos config
'dns_lookup_kdc' already set to 'true', nothing to do.
  [5/8]: activating sidgen task
  [6/8]: restarting Directory Server to take MS PAC and LDAP plugins changes into account
  [7/8]: adding fallback group
  [8/8]: adding SIDs to existing users and groups
This step may take considerable amount of time, please wait..
Done.
Configuring client side components
This program will set up IPA client.
Version 4.9.8

Using existing certificate '/etc/ipa/ca.crt'.
Client hostname: idm.vercorp.org
Realm: VERCORP.ORG
DNS Domain: vercorp.org
IPA Server: idm.vercorp.org
BaseDN: dc=vercorp,dc=org

Configured /etc/sssd/sssd.conf
Systemwide CA database updated.
Adding SSH public key from /etc/ssh/ssh_host_ed25519_key.pub
Adding SSH public key from /etc/ssh/ssh_host_ecdsa_key.pub
Adding SSH public key from /etc/ssh/ssh_host_rsa_key.pub
SSSD enabled
Configured /etc/openldap/ldap.conf
Configured /etc/ssh/ssh_config
Configured /etc/ssh/sshd_config
Configuring vercorp.org as NIS domain.
Client configuration complete.
The ipa-client-install command was successful

==============================================================================
Setup complete

Next steps:
  1. You must make sure these network ports are open:
    TCP Ports:
      * 80, 443: HTTP/HTTPS
      * 389, 636: LDAP/LDAPS
      * 88, 464: kerberos
      * 53: bind
    UDP Ports:
      * 88, 464: kerberos
      * 53: bind

  2. You can now obtain a kerberos ticket using the command: 'kinit admin'
     This ticket will allow you to use the IPA tools (e.g., ipa user-add)
     and the web user interface.
  3. Kerberos requires time synchronization between clients
     and servers for correct operation. You should consider enabling chronyd.

Be sure to back up the CA certificates stored in /root/cacert.p12
These files are required to create replicas. The password for these
files is the Directory Manager password
The ipa-server-install command was successful
[root@idm ~]#

Seems that all went well and now we have our FreeIPA/IDM installed.

Lets check several things.

[root@idm ~]# ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
named Service: RUNNING
httpd Service: RUNNING
ipa-custodia Service: RUNNING
pki-tomcatd Service: RUNNING
ipa-otpd Service: RUNNING
ipa-dnskeysyncd Service: RUNNING
ipa: INFO: The ipactl command was successful

[root@idm ~]# systemctl list-unit-files | grep ipa | grep service
ipa-ccache-sweep.service                   disabled
ipa-custodia.service                       disabled
ipa-dnskeysyncd.service                    disabled
ipa-healthcheck.service                    disabled
ipa-ods-exporter.service                   disabled
ipa-otpd@.service                          static
ipa.service                                enabled

Seems to be installed and working.

What about the /etc/sssd/sssd.conf config.

[root@idm ~]# cat /etc/sssd/sssd.conf
[domain/vercorp.org]

id_provider = ipa
ipa_server_mode = True
ipa_server = idm.vercorp.org
ipa_domain = vercorp.org
ipa_hostname = idm.vercorp.org
auth_provider = ipa
chpass_provider = ipa
access_provider = ipa
cache_credentials = True
ldap_tls_cacert = /etc/ipa/ca.crt
krb5_store_password_if_offline = True
[sssd]
services = nss, pam, ifp, ssh, sudo

domains = vercorp.org
[nss]
homedir_substring = /home
memcache_timeout = 600

[pam]

[sudo]

[autofs]

[ssh]

[pac]

[ifp]
allowed_uids = ipaapi, root

[session_recording]

There is also /etc/ssh/ssh_config.d/04-ipa.conf file …

[root@idm ~]# cat /etc/ssh/ssh_config.d/04-ipa.conf
# IPA-related configuration changes to ssh_config
#
PubkeyAuthentication yes
GlobalKnownHostsFile /var/lib/sss/pubconf/known_hosts
#VerifyHostKeyDNS yes

# assumes that if a user does not have shell (/sbin/nologin),
# this will return nonzero exit code and proxy command will be ignored
Match exec true
  ProxyCommand /usr/bin/sss_ssh_knownhostsproxy -p %p %h

For some reason we need to manually enable and start the Apache HTTP server.

[root@idm ~]# systemctl enable --now httpd

You can now access the FreeIPA/IDM at your browser on the http://idm.vercorp.org address. If you use the http://10.0.0.40 IP then you will be redirected to the http://idm.vercorp.org address so make sure you have added that host to your local /etc/hosts file.

On the second screen you will see that I have already created the vermaden user.

FreeBSD Client

UPDATE: Newer instructions for FreeBSD 13.2-RELEASE are available at the Connect FreeBSD 13.2 to FreeIPA/IDM article.

We will get straight into the point. After having FreeBSD 13.1-RELEASE installed in the most ‘default’ way in the bsdinstall(8) installer – the Auto ZFS road – we will now fetch the up to date FreeBSD Ports tree with portsnap(8) tool.

This is the main /etc/rc.conf config file.

# cat /etc/rc.conf
hostname="fbsd"
ifconfig_vtnet0="inet 10.0.0.42 netmask 255.255.255.0"
defaultrouter="10.0.0.1"
sshd_enable="YES"
moused_enable="YES"
dumpdev="AUTO"
zfs_enable="YES"
syslogd_flags="-ss"
sssd_enable="YES"

FreeBSD Packages

We need to have FreeBSD Ports tree.

# portsnap auto

Lets check what is the current default SAMBA version on FreeBSD.

# grep -i samba /usr/ports/Mk/*default*
        PYTHON2 PYTHON3 RUBY RUST SAMBA SSL TCLTK VARNISH
SAMBA_DEFAULT?=         4.12

To omit a lot of pointless compilation I will first install all precompiled packages – that would also install needed dependencies. Then we will rebuild just the several needed packages.

# pkg install krb5 sudo cyrus-sasl cyrus-sasl-gssapi pam_mkhomedir openldap26-client samba412

Settings in the /etc/make.conf file.

# cat /etc/make.conf
# USE /usr/ports/obj PLACE
WRKDIRPREFIX=${PORTSDIR}/obj

OPTIONS_UNSET=       DOCS EXAMPLES DEBUG X11
OPTIONS_UNSET+=      DBUS GSSAPI_BASE GSSAPI_HEIMDAL
OPTIONS_SET+=        GSSAPI_MIT
WITH_GSSAPI=         yes
WANT_OPENLDAP_SASL=  yes

Lets configure the needed ports.

# \
for I in /usr/ports/security/sudo \
         /usr/ports/security/sssd \
         /usr/ports/security/krb5 \
         /usr/ports/security/cyrus-sasl2-gssapi \
         /usr/ports/security/pam_mkhomedir \
         /usr/ports/net/openldap26-client
do
  make -C ${I} rmconfig
done

# \
for I in /usr/ports/security/sudo \
         /usr/ports/security/sssd \
         /usr/ports/security/krb5 \
         /usr/ports/security/cyrus-sasl2-gssapi \
         /usr/ports/security/pam_mkhomedir \
         /usr/ports/net/openldap26-client
do
  make -C ${I} config-recursive
done

The curses(3X) selection will look more or less like these below.

=> security/sudo | OPTIONS___________________________________________________________________________
[x] AUDIT              Enable BSM audit support
[ ] DISABLE_AUTH       Do not require authentication by default
[ ] DISABLE_ROOT_SUDO  Do not allow root to run sudo
[ ] DOCS               Build and/or install documentation
[ ] EXAMPLES           Build and/or install examples
[ ] INSULTS            Enable insults on failures
[x] LDAP               LDAP protocol support
[ ] NLS                Native Language Support
[ ] NOARGS_SHELL       Run a shell if no arguments are given
[ ] OPIE               Enable one-time passwords (no PAM support)
[ ] PAM                Pluggable authentication module support
[ ] PYTHON             Enable python plugin support
[x] SSSD               Enable SSSD backend support.
=> security/sudo | Kerberos 5 Authentication (no PAM support)
( ) GSSAPI_BASE        GSSAPI support via base system (needs Kerberos)
( ) GSSAPI_HEIMDAL     GSSAPI support via security/heimdal
(*) GSSAPI_MIT         GSSAPI support via security/krb5


=> security/sssd | OPTIONS___________________________________________________________________________
[ ] DOCS  Build and/or install documentation
[x] SMB   Install IPA and AD providers (requires Samba4)


=> security/cyrus-sasl2-gssapi | OPTIONS_____________________________________________________________
( ) GSSAPI_BASE     GSSAPI support via base system (needs Kerberos)
( ) GSSAPI_HEIMDAL  GSSAPI support via security/heimdal
(*) GSSAPI_MIT      GSSAPI support via security/krb5


=> net/openldap26-client | OPTIONS___________________________________________________________________
[ ] DEBUG   Build with debugging support
[ ] DOCS    Build and/or install documentation
[ ] FETCH   Enable fetch(3) support
[x] GSSAPI  With GSSAPI support


=> security/krb5 | OPTIONS___________________________________________________________________________
[ ] DNS_FOR_REALM  Enable DNS lookups for Kerberos realm names
[ ] EXAMPLES       Build and/or install examples
[x] KRB5_HTML      Install krb5 HTML documentation
[x] KRB5_PDF       Install krb5 PDF documentation
[x] LDAP           LDAP protocol support
[ ] LMDB           OpenLDAP Lightning Memory-Mapped Database support
[ ] NLS            Native Language Support
=> security/krb5 | Command Line Editing for kadmin and ktutil
(*) READLINE       Command line editing via libreadline
( ) LIBEDIT        Command line editing via libedit

We can now check what options have been saved.

# cat /var/db/ports/security_sudo-sssd/options
_OPTIONS_READ=sudo-sssd-1.9.11p3
_FILE_COMPLETE_OPTIONS_LIST=AUDIT DISABLE_AUTH DISABLE_ROOT_SUDO DOCS EXAMPLES INSULTS LDAP NLS NOARGS_SHELL OPIE PAM PYTHON SSSD GSSAPI_BASE GSSAPI_HEIMDAL GSSAPI_MIT
OPTIONS_FILE_SET+=AUDIT
OPTIONS_FILE_SET+=GSSAPI_MIT
OPTIONS_FILE_SET+=LDAP
OPTIONS_FILE_SET+=NLS
OPTIONS_FILE_SET+=SSSD
OPTIONS_FILE_UNSET+=DISABLE_AUTH
OPTIONS_FILE_UNSET+=DISABLE_ROOT_SUDO
OPTIONS_FILE_UNSET+=DOCS
OPTIONS_FILE_UNSET+=EXAMPLES
OPTIONS_FILE_UNSET+=GSSAPI_BASE
OPTIONS_FILE_UNSET+=GSSAPI_HEIMDAL
OPTIONS_FILE_UNSET+=INSULTS
OPTIONS_FILE_UNSET+=NOARGS_SHELL
OPTIONS_FILE_UNSET+=OPIE
OPTIONS_FILE_UNSET+=PAM
OPTIONS_FILE_UNSET+=PYTHON

# cat /var/db/ports/security_sssd/options
_OPTIONS_READ=sssd-1.16.5_6
_FILE_COMPLETE_OPTIONS_LIST=DOCS SMB
OPTIONS_FILE_UNSET+=DOCS
OPTIONS_FILE_SET+=SMB

# cat /var/db/ports/security_cyrus-sasl2/options
_OPTIONS_READ=cyrus-sasl-2.1.28
_FILE_COMPLETE_OPTIONS_LIST=ALWAYSTRUE AUTHDAEMOND DOCS KEEP_DB_OPEN  OBSOLETE_CRAM_ATTR OBSOLETE_DIGEST_ATTR  SASLDB_IN_VAR BDB1 BDB GDBM LMDB ANONYMOUS CRAM DIGEST LOGIN NTLM OTP PLAIN SCRAM
OPTIONS_FILE_SET+=ANONYMOUS
OPTIONS_FILE_SET+=AUTHDAEMOND
OPTIONS_FILE_SET+=BDB1
OPTIONS_FILE_SET+=CRAM
OPTIONS_FILE_SET+=DIGEST
OPTIONS_FILE_SET+=LOGIN
OPTIONS_FILE_SET+=NTLM
OPTIONS_FILE_SET+=OBSOLETE_CRAM_ATTR
OPTIONS_FILE_SET+=OBSOLETE_DIGEST_ATTR
OPTIONS_FILE_SET+=OTP
OPTIONS_FILE_SET+=PLAIN
OPTIONS_FILE_SET+=SCRAM
OPTIONS_FILE_UNSET+=ALWAYSTRUE
OPTIONS_FILE_UNSET+=BDB
OPTIONS_FILE_UNSET+=DOCS
OPTIONS_FILE_UNSET+=GDBM
OPTIONS_FILE_UNSET+=KEEP_DB_OPEN
OPTIONS_FILE_UNSET+=LMDB
OPTIONS_FILE_UNSET+=SASLDB_IN_VAR

# cat /var/db/ports/net_openldap26-client/options
_OPTIONS_READ=openldap26-client-2.6.3
_FILE_COMPLETE_OPTIONS_LIST=DEBUG DOCS FETCH GSSAPI
OPTIONS_FILE_SET+=GSSAPI
OPTIONS_FILE_UNSET+=DEBUG
OPTIONS_FILE_UNSET+=DOCS
OPTIONS_FILE_UNSET+=FETCH

# cat /var/db/ports/security_krb5/options
# This file is auto-generated by 'make config'.
# Options for krb5-1.20
_OPTIONS_READ=krb5-1.20
_FILE_COMPLETE_OPTIONS_LIST=DNS_FOR_REALM EXAMPLES KRB5_HTML KRB5_PDF LDAP LMDB NLS READLINE LIBEDIT
OPTIONS_FILE_UNSET+=DNS_FOR_REALM
OPTIONS_FILE_UNSET+=EXAMPLES
OPTIONS_FILE_SET+=KRB5_HTML
OPTIONS_FILE_SET+=KRB5_PDF
OPTIONS_FILE_SET+=LDAP
OPTIONS_FILE_UNSET+=LMDB
OPTIONS_FILE_SET+=NLS
OPTIONS_FILE_SET+=READLINE
OPTIONS_FILE_UNSET+=LIBEDIT

… and build them.

# \
for I in /usr/ports/security/sudo \
         /usr/ports/security/sssd \
         /usr/ports/security/krb5
         /usr/ports/security/cyrus-sasl2-gssapi \
         /usr/ports/security/pam_mkhomedir \
         /usr/ports/net/openldap26-client \
do
  make -C ${I} build
done

… unsinstall them and create their packages.

# \
for I in /usr/ports/security/sudo \
         /usr/ports/security/sssd \
         /usr/ports/security/krb5
         /usr/ports/security/cyrus-sasl2-gssapi \
         /usr/ports/security/pam_mkhomedir \
         /usr/ports/net/openldap26-client \
do
  make -C ${I} deinstall package
done

After some time we have our built packages.

# find /usr/ports/obj/ -name \*.pkg
/usr/ports/obj/usr/ports/lang/perl5.32/work/perl-5.32.1/symbian/ext/Moped/Msg/Msg.pkg
/usr/ports/obj/usr/ports/security/pam_mkhomedir/work/pkg/pam_mkhomedir-0.2.pkg
/usr/ports/obj/usr/ports/security/cyrus-sasl2-gssapi/work/pkg/cyrus-sasl-gssapi-2.1.28.pkg
/usr/ports/obj/usr/ports/security/krb5/work/pkg/krb5-1.20.pkg
/usr/ports/obj/usr/ports/security/sudo/work/pkg/sudo-1.9.11p3.pkg
/usr/ports/obj/usr/ports/security/sssd/work-default/pkg/sssd-1.16.5_6.pkg
/usr/ports/obj/usr/ports/net/openldap26-client/work/pkg/openldap26-client-2.6.3.pkg

In the future I need to add some short guide to build them regularly with Synth or Poudriere 🙂

Now we need to install these packages in quite nontypical way. One by one – in specified order. It will not be needed it we would have a separate additional pkg(8) repository with Poudriere build packages.

Our packages:

# ls -1 *.pkg
cyrus-sasl-gssapi-2.1.28.pkg
krb5-1.20.pkg
openldap26-client-2.6.3.pkg
pam_mkhomedir-0.2.pkg
sssd-smb4-1.16.5_6.pkg
sudo-sssd-1.9.11p3.pkg

… and their install process.

# pkg install -y c-ares

# pkg add sssd-smb4-1.16.5_6.pkg

# pkg install -y cyrus-sasl-gssapi-2.1.28.pkg
(...)
New packages to be INSTALLED:
        cyrus-sasl: 2.1.28
        cyrus-sasl-gssapi: 2.1.28
        krb5: 1.20
(...)

# pkg delete -f -y krb5
(...)
Installed packages to be REMOVED:
        krb5: 1.20
(...)

# pkg add krb5-1.20.pkg

# pkg install -y sssd

# pkg add sudo-sssd-1.9.11p3.pkg

# pkg delete -f -y sssd
(...)
Installed packages to be REMOVED:
        sssd: 1.16.5_6
(...)

# pkg install ding-libs ldb21 nspr nss pcre samba412

# pkg add sssd-smb4-1.16.5_6.pkg

FreeBSD Setup

Create needed dirs.

# mkdir -p \
    /usr/local/etc/ipa \
    /var/log/sssd \
    /var/run/sss/private \
    /var/db/sss

Set system hostname.

# hostname fbsd.vercorp.org

# hostname
fbsd.vercorp.org

Fetch the FreeIPA/IDM certificate.

# fetch -o /usr/local/etc/ipa/ca.crt http://10.0.0.40/ipa/config/ca.crt

FreeIPA/IDM Setup Part

We need to execute several instructions on the FreeIPA/IDM to connect FreeBSD client to it.

Its adding the A and PTR records in DNS for the 10.0.0.42 address and adding the fbsd.vercorp.org host.

We also need to generate the key for our fbsd.vercorp.org system.

[root@idm ~]# kinit admin
Password for admin@VERCORP.ORG:

[root@idm ~]# ipa dnsrecord-add vercorp.org fbsd --a-rec=10.0.0.42 --a-create-reverse
  Record name: fbsd
  A record: 10.0.0.42

[root@idm ~]# ipa host-add fbsd.vercorp.org
-----------------------------
Added host "fbsd.vercorp.org"
-----------------------------
  Host name: fbsd.vercorp.org
  Principal name: host/fbsd.vercorp.org@VERCORP.ORG
  Principal alias: host/fbsd.vercorp.org@VERCORP.ORG
  Password: False
  Keytab: False
  Managed by: fbsd.vercorp.org

[root@idm ~]# ipa-getkeytab -s idm.vercorp.org -p host/fbsd.vercorp.org@VERCORP.ORG -k /root/fbsd.vercorp.org.keytab
Keytab successfully retrieved and stored in: /root/fbsd.vercorp.org.keytab

Now we need to get ‘our’ key from FreeIPA/IDM server … along with proper /etc/hosts file.

# scp root@10.0.0.40:/root/fbsd.vercorp.org.keytab /usr/local/etc/ipa/krb5.keytab

# cat << EOF > /etc/hosts
::1              localhost localhost.my.domain
127.0.0.1        localhost localhost.my.domain
10.0.0.40        idm.vercorp.org  idm
10.0.0.42        fbsd.vercorp.org fbsd
EOF

Now the /usr/local/etc/openldap/ldap.conf file.

# cat << EOF > /usr/local/etc/openldap/ldap.conf
BASE        dc=org,dc=vercorp
URI         ldap://idm.vercorp.org/
#SIZELIMIT  12
#TIMELIMIT  15
#DEREF      never
SASL_MECH   GSSAPI
SASL_REALM  VERCORP.ORG
ssl         start_tls
TLS_CACERT  /usr/local/etc/ipa/ca.crt
EOF

… and /etc/krb5.conf file.

# cat << EOF > /etc/krb5.conf
[libdefaults]
  default_realm = VERCORP.ORG
  default_keytab_name = FILE:/usr/local/etc/ipa/krb5.keytab
  default_tkt_enctypes = aes256-cts des-cbc-crc aes128-cts arcfour-hmac
  default_tgs_enctypes = aes256-cts des-cbc-crc aes128-cts arcfour-hmac
  dns_lookup_realm = false
  dns_lookup_kdc = false
  rdns = false
  ticket_lifetime = 24h
  forwardable = yes

[realms]
  VERCORP.ORG = {
    kdc = idm.vercorp.org:88
    master_kdc = idm.vercorp.org:88
    admin_server = idm.vercorp.org:749
    default_domain = vercorp.org
    pkinit_anchors = FILE:/usr/local/etc/ipa/ca.crt
}

[domain_realm]
  .vercorp.org = VERCORP.ORG
  vercorp.org = VERCORP.ORG

[logging]
  kdc = FILE:/var/log/krb5/krb5kdc.log
  admin_server = FILE:/var/log/krb5/kadmin.log
  kadmin_local = FILE:/var/log/krb5/kadmin_local.log
  default = FILE:/var/log/krb5/krb5lib.log
EOF

… and /usr/local/etc/sssd/sssd.conf file.

# cat << EOF > /usr/local/etc/sssd/sssd.conf

[domain/vercorp.org]
# debug_level = 9
cache_credentials = True
krb5_store_password_if_offline = True
krb5_realm = VERCORP.ORG
ipa_domain = vercorp.org
id_provider = ipa
auth_provider = ipa
access_provider = ipa
ipa_hostname = fbsd.vercorp.org
chpass_provider = ipa
ipa_server = _srv_, idm.vercorp.org
ldap_tls_cacert = /usr/local/etc/ipa/ca.crt
krb5_keytab = /usr/local/etc/ipa/krb5.keytab

[sssd]
services = nss, pam, ssh, sudo
config_file_version = 2
domains = vercorp.org

[nss]
filter_users = root,toor
homedir_substring = /usr/home/%u

[pam]

[sudo]
# debug_level = 0x3ff0

[ssh]
EOF

# chmod 600 /usr/local/etc/sssd/sssd.conf

FreeBSD have user account under /usr/home to make sure /home also points there.

# ln -s /usr/home /home

The automatic startup of sssd(8) daemon (not to confuse with sshd(8)) needs to be also configured.

# sysrc sssd_enable=YES
# service sssd start

We also need to configure /etc/nsswitch.conf file.

# cp /etc/nsswitch.conf /etc/nsswitch.conf.BCK

# diff -u /etc/nsswitch.conf.BCK /etc/nsswitch.conf
--- /etc/nsswitch.conf.BCK      2022-10-24 20:10:09.163251000 +0200
+++ /etc/nsswitch.conf  2022-10-24 20:10:57.207406000 +0200
@@ -2,15 +2,17 @@
 # nsswitch.conf(5) - name service switch configuration file
 # $FreeBSD$
 #
-group: compat
+group: files sss
 group_compat: nis
 hosts: files dns
-netgroup: compat
+# netgroup: compat
 networks: files
-passwd: compat
+passwd: files sss
 passwd_compat: nis
 shells: files
 services: compat
 services_compat: nis
 protocols: files
 rpc: files
+sudoers: sss files
+netgroup: files

The final /etc/nsswitch.conf file looks as follows.

# cat /etc/nsswitch.conf
#
# nsswitch.conf(5) - name service switch configuration file
# $FreeBSD$
#
group: files sss
group_compat: nis
hosts: files dns
# netgroup: compat
networks: files
passwd: files sss
passwd_compat: nis
shells: files
services: compat
services_compat: nis
protocols: files
rpc: files
sudoers: sss files
netgroup: files

Now the /etc/pam.d/system file.

# cp /etc/pam.d/system /root/etc---pam.d---system.BCK

# diff -u  /root/etc---pam.d---system.BCK /etc/pam.d/system
--- /root/etc---pam.d---system.BCK      2022-10-24 20:13:05.546657000 +0200
+++ /etc/pam.d/system   2022-10-24 20:16:36.722666000 +0200
@@ -7,19 +7,23 @@
 # auth
 auth           sufficient      pam_opie.so             no_warn no_fake_prompts
 auth           requisite       pam_opieaccess.so       no_warn allow_local
-#auth          sufficient      pam_krb5.so             no_warn try_first_pass
+auth           sufficient      pam_krb5.so             no_warn try_first_pass
 #auth          sufficient      pam_ssh.so              no_warn try_first_pass
+auth           sufficient      /usr/local/lib/pam_sss.so       use_first_pass
 auth           required        pam_unix.so             no_warn try_first_pass nullok

 # account
 #account       required        pam_krb5.so
 account                required        pam_login_access.so
 account                required        pam_unix.so
+account                required        /usr/local/lib/pam_sss.so       ignore_unknown_user ignore_authinfo_unavail

 # session
 #session       optional        pam_ssh.so              want_agent
 session                required        pam_lastlog.so          no_fail
+session                required        /usr/local/lib/pam_mkhomedir.so mode=0700

 # password
 #password      sufficient      pam_krb5.so             no_warn try_first_pass
+password       sufficient      /usr/local/lib/pam_sss.so       use_authtok
 password       required        pam_unix.so             no_warn try_first_pass

The final /etc/pam.d/system file looks as follows.

# cat /etc/pam.d/system
#
# $FreeBSD$
#
# System-wide defaults
#

# auth
auth            sufficient      pam_opie.so             no_warn no_fake_prompts
auth            requisite       pam_opieaccess.so       no_warn allow_local
auth            sufficient      pam_krb5.so             no_warn try_first_pass
#auth           sufficient      pam_ssh.so              no_warn try_first_pass
auth            sufficient      /usr/local/lib/pam_sss.so       use_first_pass
auth            required        pam_unix.so             no_warn try_first_pass nullok

# account
#account        required        pam_krb5.so
account         required        pam_login_access.so
account         required        pam_unix.so
account         required        /usr/local/lib/pam_sss.so       ignore_unknown_user ignore_authinfo_unavail

# session
#session        optional        pam_ssh.so              want_agent
session         required        pam_lastlog.so          no_fail
session         required        /usr/local/lib/pam_mkhomedir.so mode=0700

# password
#password       sufficient      pam_krb5.so             no_warn try_first_pass
password        sufficient      /usr/local/lib/pam_sss.so       use_authtok
password        required        pam_unix.so             no_warn try_first_pass

Now its time for /etc/pam.d/sshd file.

# cp /etc/pam.d/sshd /root/etc---pam.d---sshd.BCK

# diff -u /root/etc---pam.d---sshd.BCK /etc/pam.d/sshd
--- /root/etc---pam.d---sshd.BCK        2022-10-24 20:17:34.063630000 +0200
+++ /etc/pam.d/sshd     2022-10-24 20:19:16.165810000 +0200
@@ -7,8 +7,9 @@
 # auth
 auth           sufficient      pam_opie.so             no_warn no_fake_prompts
 auth           requisite       pam_opieaccess.so       no_warn allow_local
-#auth          sufficient      pam_krb5.so             no_warn try_first_pass
+auth           sufficient      pam_krb5.so             no_warn try_first_pass
 #auth          sufficient      pam_ssh.so              no_warn try_first_pass
+auth           sufficient      /usr/local/lib/pam_sss.so       use_first_pass
 auth           required        pam_unix.so             no_warn try_first_pass

 # account
@@ -16,11 +17,14 @@
 #account       required        pam_krb5.so
 account                required        pam_login_access.so
 account                required        pam_unix.so
+account                required        /usr/local/lib/pam_sss.so       ignore_unknown_user ignore_authinfo_unavail

 # session
 #session       optional        pam_ssh.so              want_agent
 session                required        pam_permit.so
+session                required        /usr/local/lib/pam_mkhomedir.so mode=0700

 # password
 #password      sufficient      pam_krb5.so             no_warn try_first_pass
+password       sufficient      /usr/local/lib/pam_sss.so       use_authtok
 password       required        pam_unix.so             no_warn try_first_pass

Final /etc/pam.d/sshd file below.

# cat /etc/pam.d/sshd
#
# $FreeBSD$
#
# PAM configuration for the "sshd" service
#

# auth
auth            sufficient      pam_opie.so             no_warn no_fake_prompts
auth            requisite       pam_opieaccess.so       no_warn allow_local
auth            sufficient      pam_krb5.so             no_warn try_first_pass
#auth           sufficient      pam_ssh.so              no_warn try_first_pass
auth            sufficient      /usr/local/lib/pam_sss.so       use_first_pass
auth            required        pam_unix.so             no_warn try_first_pass

# account
account         required        pam_nologin.so
#account        required        pam_krb5.so
account         required        pam_login_access.so
account         required        pam_unix.so
account         required        /usr/local/lib/pam_sss.so       ignore_unknown_user ignore_authinfo_unavail

# session
#session        optional        pam_ssh.so              want_agent
session         required        pam_permit.so
session         required        /usr/local/lib/pam_mkhomedir.so mode=0700

# password
#password       sufficient      pam_krb5.so             no_warn try_first_pass
password        sufficient      /usr/local/lib/pam_sss.so       use_authtok
password        required        pam_unix.so             no_warn try_first_pass

Small modification in the /etc/ssh/ssh_config and /etc/ssh/sshd_config files.

# cat << EOF >> /etc/ssh/ssh_config
GSSAPIAuthentication yes
EOF

# cat << EOF >> /etc/ssh/sshd_config
GSSAPIAuthentication yes
UsePAM yes
EOF

Finish Setup with Web Browser in FreeIPA/IDM Page

Visit the https://idm.vercorp.org/ipa/ui/#/e/hbacrule/details/freebsd page.

Next create the HBAC Rule named freebsd as showed below.

… and the Sudo Rule named freebsd name.

FreeBSD FreeIPA Login Test

After all these time consuming and pointless instructions we can now finally try to login to our FreeBSD client.

% ssh -l vermaden 10.0.0.42
(vermaden@10.0.0.42) Password:
Last login: Mon Oct 24 21:06:36 2022 from 10.0.0.3
FreeBSD 13.1-RELEASE releng/13.1-n250148-fc952ac2212 GENERIC

Welcome to FreeBSD!

Release Notes, Errata: https://www.FreeBSD.org/releases/
Security Advisories:   https://www.FreeBSD.org/security/
FreeBSD Handbook:      https://www.FreeBSD.org/handbook/
FreeBSD FAQ:           https://www.FreeBSD.org/faq/
Questions List: https://lists.FreeBSD.org/mailman/listinfo/freebsd-questions/
FreeBSD Forums:        https://forums.FreeBSD.org/

Documents installed with the system are in the /usr/local/share/doc/freebsd/
directory, or can be installed later with:  pkg install en-freebsd-doc
For other languages, replace "en" with a language code like de or fr.

Show the version of FreeBSD installed:  freebsd-version ; uname -a
Please include that output and any error messages when posting questions.
Introduction to manual pages:  man man
FreeBSD directory layout:      man hier

To change this login announcement, see motd(5).
You can upload the dmesg of your system to help developers get an overview of commonly
used hardware and peripherals for FreeBSD. Use the curl package to upload it like this:
curl -v -d "nickname=$USER" -d "description=FreeBSD/$(uname -m) on \
$(kenv smbios.system.maker) $(kenv smbios.system.product)" -d "do=addd" \
--data-urlencode 'dmesg@/var/run/dmesg.boot' http://dmesgd.nycbug.org/index.cgi

vermaden@fbsd:~ $ :> ~/.hushlogin

vermaden@fbsd:~ $ id
uid=1408200003(vermaden) gid=1408200000(admins) groups=1408200000(admins)

vermaden@fbsd:~ $ pwd
/home/vermaden

vermaden@fbsd:~ $ grep vermaden /etc/passwd /etc/group
vermaden@fbsd:~ $

vermaden@fbsd:~ $ getent passwd vermaden
vermaden:*:1408200003:1408200000:vermaden vermaden:/home/vermaden:/bin/sh

vermaden@fbsd:~ $ sudo su -
Password for vermaden@VERCORP.ORG:

root@fbsd:~ # logout

vermaden@fbsd:~ $ sudo -i

root@fbsd:~ #

Strange … seems to work properly 🙂

FreeBSD Jail as FreeIPA/IDM Client

As ‘full’ FreeBSD system is able to connect to the FreeIPA/IDM server we will not configure FreeBSD Jail to do the same.

The FreeIPA/IDM FreeBSD client fbsdjail.vercorp.orgwill get the 10.0.0.43 IP.

Basic FreeBSD Jail Preparations

Lets setup the Jail for a start.

# mkdir -p /jail/fbsdjail /jail/BASE

# cd /jail/fbsdjail

# fetch -o /jail/BASE/13.1-RELEASE-base.txz https://download.freebsd.org/ftp/releases/amd64/13.1-RELEASE/base.txz

# tar --unlink -xvf ../BASE/13.1-RELEASE-base.txz

# cat << EOF > /etc/jail.conf
# GLOBAL
  exec.start = "/bin/sh /etc/rc";
  exec.stop = "/bin/sh /etc/rc.shutdown";
  exec.clean;
  exec.consolelog = "/var/log/jail_${name}_console.log";
  mount.devfs;
  host.hostname = ${name};
  path = /jail/${name};

# JAILS
  fbsdjail {
    ip4.addr = 10.0.0.43;
    host.hostname = fbsdjail.vercorp.org;
    interface = wlan0;
    allow.raw_sockets;
    allow.sysvipc;
  }
EOF

# cat /etc/resolv.conf | tee /jail/fbsdjail/etc/resolv.conf
nameserver 10.0.0.1

# echo 10.0.0.43 fbsdjail.vercorp.org fbsdjail | tee -a /etc/hosts | tee -a /jail/fbsdjail/etc/hosts

# cat /etc/hosts /jail/fbsdjail/etc/hosts
10.0.0.43 fbsdjail.vercorp.org fbsdjail
10.0.0.43 fbsdjail.vercorp.org fbsdjail

# cat << EOF > /jail/fbsdjail/etc/rc.conf
# DAEMONS | yes
  syslogd_flags="-ss"
  sshd_enable=YES

# OTHER
  clear_tmp_enable=YES
  clear_tmp_X=YES
  dumpdev=NO
  update_motd=NO
EOF

# sed -i '' s/quarterly/latest/g /jail/fbsdjail/etc/pkg/FreeBSD.conf

# grep latest /jail/fbsdjail/etc/pkg/FreeBSD.conf
  url: "pkg+http://pkg.FreeBSD.org/${ABI}/latest",

Now we can start our FreeBSD Jail.

# service jail onestart fbsdjail
Starting jails: fbsdjail.

# jls
   JID  IP Address      Hostname                      Path
     1  10.0.0.43       fbsdjail.vercorp.org          /jail/fbsdjail

# jls -v
   JID  Hostname                      Path
        Name                          State
        CPUSetID
        IP Address(es)
     1  fbsdjail.vercorp.org          /jail/fbsdjail
        fbsdjail                      ACTIVE
        3
        10.0.0.43

# jexec fbsdjail

root@fbsdjail:/ #

Our FreeBSD Jail works. Lets move to next steps.

Configure FreeBSD Jail to Connect to FreeIPA/IDM Server

I could not repaste all of the instructions above – the ones that we used for a ‘full’ FreeBSD system – but the same applies to a FreeBSD Jail. 🙂

This means that earlier Basic FreeBSD Jail Preparations section covers all that is needed in case of ‘full’ FreeBSD versus FreeBSD Jail when it comes to the FreeIPA/IDM connection.

Linux FreeIPA/IDM Client

This article is not about Linux client – which is pretty straight-forward to connect to the FreeIPA/IDM server – but for the completness of the topic – here are the instructions I used to attach Alma Linux to the FreeIPA/IDM server.

Linux rhlike.vercorp.org system.

      IP: 10.0.0.41/24
      GW: 10.0.0.1
hostname: rhlike.vercorp.org

First – install the @idm:client and sssd packages.

client # yum -y install @idm:client sssd

FreeIPA/IDM Setup Part

Now – as earlier with FreeBSD – the FreeIPA/IDM part comes to play.

[root@idm ~]# kinit admin
Password for admin@VERCORP.ORG:

[root@idm ~]# klist
Ticket cache: KCM:0
Default principal: admin@VERCORP.ORG

Valid starting       Expires              Service principal
10/19/2022 13:33:52  10/20/2022 13:11:28  krbtgt/VERCORP.ORG@VERCORP.ORG

[root@idm ~]# ipa user-find
---------------
2 users matched
---------------
  User login: admin
  Last name: Administrator
  Home directory: /home/admin
  Login shell: /bin/bash
  Principal alias: admin@VERCORP.ORG, root@VERCORP.ORG
  UID: 1896600000
  GID: 1896600000
  Account disabled: False

  User login: vermaden
  First name: vermaden
  Last name: vermaden
  Home directory: /home/vermaden
  Login shell: /bin/sh
  Principal name: vermaden@VERCORP.ORG
  Principal alias: vermaden@VERCORP.ORG
  Email address: vermaden@vercorp.org
  UID: 1896600003
  GID: 1000
  Account disabled: False
----------------------------
Number of entries returned 2
----------------------------

[root@idm ~]# id vermaden
uid=1408200003(vermaden) gid=1408200000(admins) groups=1408200000(admins)

[root@idm ~]# ipa dnsrecord-add vercorp.org rhlike --a-rec 10.0.0.41
  Record name: rhlike
  A record: 10.0.0.41

We are done on the FreeIPA/IDM side.

Linux FreeIPA/IDM Client Setup

We will now continue our work on the Linux client.

client # echo "10.0.0.40   idm.vercorp.org"    >> /etc/hosts

client # echo "10.0.0.41   rhlike.vercorp.org" >> /etc/hosts

client # hostnamectl set-hostname rhlike.vercorp.org

client # cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
10.0.0.40   idm.vercorp.org
10.0.0.41   rhlike.vercorp.org

client # ipa-client-install --uninstall

client # ipa-client-install \
           --hostname=rhlike.vercorp.org \
           --mkhomedir \
           --server=idm.vercorp.org \
           --domain vercorp.org \
           --realm VERCORP.ORG

This program will set up IPA client.
Version 4.9.8

Autodiscovery of servers for failover cannot work with this configuration.
If you proceed with the installation, services will be configured to always access the discovered server for all operations and will not fail over to other servers in case of failure.
Proceed with fixed values and no DNS discovery? [no]: yes
Do you want to configure chrony with NTP server or pool address? [no]: no
Client hostname: rhlike.vercorp.org
Realm: VERCORP.ORG
DNS Domain: vercorp.org
IPA Server: idm.vercorp.org
BaseDN: dc=vercorp,dc=org

Continue to configure the system with these values? [no]: yes
Synchronizing time
No SRV records of NTP servers found and no NTP server or pool address was provided.
Using default chrony configuration.
Attempting to sync time with chronyc.
Time synchronization was successful.
User authorized to enroll computers: admin
Password for admin@VERCORP.ORG:
Successfully retrieved CA cert
    Subject:     CN=Certificate Authority,O=VERCORP.ORG
    Issuer:      CN=Certificate Authority,O=VERCORP.ORG
    Valid From:  2022-10-18 14:52:50
    Valid Until: 2042-10-18 14:52:50

Enrolled in IPA realm VERCORP.ORG
Created /etc/ipa/default.conf
Configured /etc/sssd/sssd.conf
Configured /etc/krb5.conf for IPA realm VERCORP.ORG
Systemwide CA database updated.
Hostname (rhlike.vercorp.org) does not have A/AAAA record.
Failed to update DNS records.
Missing A/AAAA record(s) for host rhlike.vercorp.org: 10.0.0.41.
Missing reverse record(s) for address(es): 10.0.0.41.
Adding SSH public key from /etc/ssh/ssh_host_ed25519_key.pub
Adding SSH public key from /etc/ssh/ssh_host_ecdsa_key.pub
Adding SSH public key from /etc/ssh/ssh_host_rsa_key.pub
Could not update DNS SSHFP records.
SSSD enabled
Configured /etc/openldap/ldap.conf
Configured /etc/ssh/ssh_config
Configured /etc/ssh/sshd_config
Configuring vercorp.org as NIS domain.
Client configuration complete.
The ipa-client-install command was successful

client # reboot

Now we will test how it goes with login against the FreeIPA/IDM server.

laptop % ssh -l vermaden 10.0.0.41
(vermaden@10.0.0.41) Password:
(vermaden@10.0.0.41) Password expired. Change your password now.
Current Password:
(vermaden@10.0.0.41) New password:
(vermaden@10.0.0.41) Retype new password:
Last failed login: Wed Oct 19 00:47:57 CEST 2022 from 10.0.0.3 on ssh:notty
There was 1 failed login attempt since the last successful login.
/usr/bin/id: cannot find name for group ID 1000

[vermaden@rhlike ~]$ w
 00:48:16 up 29 min,  2 users,  load average: 0.22, 0.13, 0.16
USER     TTY      FROM             LOGIN@   IDLE   JCPU   PCPU WHAT
root     pts/0    10.0.0.3         00:40   58.00s  0.03s  0.03s -bash
vermaden pts/1    10.0.0.3         00:48    0.00s  0.02s  0.01s w

[vermaden@rhlike ~]$ sudo su -

[root@rhlike ~]# getent passwd admin
admin:*:1896600000:1896600000:Administrator:/home/admin:/bin/bash

[root@rhlike ~]# getent passwd vermaden
vermaden:*:1896600003:1000:vermaden vermaden:/home/vermaden:/bin/sh

Seems to work at least OK 🙂

I do not have anything more to add to this guide.

If you have – then please let me know in comments 🙂

Regards.

EOF

NFS Server Inside FreeBSD VNET Jail

FreeBSD Jails is a great piece of container technology pioneered several years before Solaris Zones. Not to mention 15 years before Docker was born. Today they still work great and offer some new features like entire network stack for each Jail called VNET. Unfortunately they also have downsides. For example anything related to NFS is broken inside FreeBSD Jails (especially when they are VNET based Jails) and the relevant 251347 Bug Report remains unsolved.

There is however a way to run NFS server inside VNET based FreeBSD Jail – we will use userspace NFS server implementation instead of using the FreeBSD base system kernel space NFS server. Its available as net/unfs3 package and this is exactly what we will gonna use for this guide.

Same in plain text below.

/ % cd /usr/ports/net/unfs3

/usr/ports/net/unfs3 % cat pkg-descr
UNFS3 is a user-space implementation of the NFSv3 server specification. It
provides a daemon for the MOUNT and NFS protocols, which are used by NFS
clients for accessing files on the server.
Since it runs in user-space, you can use it in a jail.

WWW: https://unfs3.github.io/

/usr/ports/net/unfs3 % pkg info -l unfs3           
unfs3-0.9.22_2:
        /usr/local/man/man7/tags.7.gz
        /usr/local/man/man8/unfsd.8.gz
        /usr/local/sbin/unfsd
        /usr/local/share/licenses/unfs3-0.9.22_2/BSD3CLAUSE
        /usr/local/share/licenses/unfs3-0.9.22_2/LICENSE
        /usr/local/share/licenses/unfs3-0.9.22_2/catalog.mk

Its also pity that VNET feature for FreeBSD Jails is not well documented. Search the FreeBSD Handbook or FreeBSD FAQ for the VNET or VIMAGE keywords. Not a single match. There are only man pages and some stuff left in the /usr/share/examples/jails dir. There is also FreeBSD Mastery: Jails book by Michael W. Lucas but its 3 years old already.

Setup

Below you will find the list of systems we will use in this guide.

10.0.10.250  host
10.0.10.251  nfs_server

The host is a common FreeBSD server installed on a physical or virtual machine. We will also use it as out NFS client and mount the NFS share there. The nfs_server is a FreeBSD Jail with VNET separate network stack enabled. We will run NFS server from this host nfs_server system. Both of them run latest FreeBSD 13.1-RELEASE but I suspect that it should also work the same on older versions.

FreeBSD Host and NFS Client (host)

First we will setup the host machine. Its typical default ZFS FreeBSD install – nothing special about that. To use the VNET enabled Jails we will use jib tool from the /usr/share/examples/jails directory as we will need it to automate epair(4) interfaces management.

root@host:/ # install -o root -g wheel -m 0555 /usr/share/examples/jails/jib /usr/sbin/jib

Our next step would be to fetch and setup the nfs_server FreeBSD Jail. We will not waste time in compilation – we will fetch the base.txz directly from FreeBSD page.

root@host:/ # mkdir -p /jail/BASE
root@host:/jail/BASE # cd /jail/BASE
root@host:/jail/BASE # fetch http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/13.1-RELEASE/base.txz
root@host:/jail/BASE # mv base.txz 13.1-base.txz

Now the nfs_server FreeBSD Jail.

root@host:/ # mkdir -p /jail/nfs_server
root@host:/jail/nfs_server # cd /jail/nfs_server
root@host:/jail/nfs_server # tar -xzf /jail/BASE/13.1-base.txz --unlink

The main FreeBSD /etc/rc.conf configuration file does not hold any special setting – pretty usual stuff.

root@host:/ # cat /etc/rc.conf
# NETWORK
  hostname="host"
  ifconfig_em0="inet 10.0.10.250/24 up"
  defaultrouter="10.0.10.1"
  gateway_enable="YES"

# DAEMONS
  dumpdev="AUTO"
  sshd_enable="YES"
  zfs_enable="YES"
  sendmail_enable="NO"
  sendmail_submit_enable="NO"
  sendmail_outbound_enable="NO"
  sendmail_msp_queue_enable="NO"
  update_motd="NO"

# JAILS
  jail_enable="YES"
  jail_parallel_start="YES"
  jail_list="nfs_server"

The nfs_server FreeBSD Jail as configured in the /etc/jail.conf config file.

root@host:/ # cat /etc/jail.conf
nfs_server {
  path = "/jail/${name}";
  host.hostname = "${name}";
  allow.raw_sockets = 1;
  allow.set_hostname = 1;
  allow.sysvipc = 1;
  mount.devfs;
  exec.clean;
  vnet;
  vnet.interface = "e0b_${name}";
  exec.prestart += "/usr/sbin/jib addm -b _bridge0 ${name} em0";
  exec.poststop += "/usr/sbin/jib destroy ${name}";
  exec.start += "/bin/sh /etc/rc";
  exec.stop = "/bin/sh /etc/rc.shutdown";
  exec.consolelog = "/var/log/jail_${name}_console.log";
}

… and last but not least lets make sure the following DNS haiku will not bother us 🙂

Setup the /etc/hosts on the host system.

root@host:/ # tail -3 /etc/hosts
10.0.10.250 host
10.0.10.251 nfs_server

FreeBSD NFS Server VNET Jail (nfs_server)

As our FreeBSD Jail is installed we will now start it and configure it.

root@host:/ # service jail onestart nfs_server

root@host:/ # jls
   JID  IP Address      Hostname                      Path
     1                  nfs_server                    /jail/nfs_server

root@host:/ # jexec 1

root@nfs_server:/ # 

First we will install latest net/unfs3 package – this userspace NFS server is also very minimal and does not have any dependencies.

root@nfs_server:/ # echo nameserver 1.1.1.1 > /etc/resolv.conf

root@nfs_server:/ # sed -i '' s/quarterly/latest/g /etc/pkg/FreeBSD.conf

root@nfs_server:/ # pkg install unfs3

root@nfs_server:/ # pkg info -qoa
ports-mgmt/pkg
net/unfs3

Now we will configure our NFS share under /share dir and start the unfsd(8) userspace NFS server.

root@nfs_server:/ # mkdir /share

root@nfs_server:/ # cat /etc/exports
/share  10.0.10.250(rw,no_root_squash,no_all_squash)

… last but not least – DNS 🙂

root@nfs_server:/ # tail -3 /etc/hosts
10.0.10.250 host
10.0.10.251 nfs_server

As we are using VNET network stack in a FreeBSD Jail we will have to address the network interface in the Jails /etc/rc.conf file. The unfsd(8) daemon does not start without rpcbind service so we will also enable it.

root@nfs_server:/ # cat /etc/rc.conf
# NETWORK
  hostname="nfs_server"
  ifconfig_e0b_nfs_server="10.0.10.251/24 up"
  defaultrouter="10.0.10.1"

# DAEMONS
  sshd_enable="YES"
  rpcbind_enable="YES"
  sendmail_enable="NO"
  sendmail_submit_enable="NO"
  sendmail_outbound_enable="NO"
  sendmail_msp_queue_enable="NO"

We will make unfsd(8) start automatically at Jails start with plain old /etc/rc.local file.

root@nfs_server:/ # cat /etc/rc.local 
/usr/local/sbin/unfsd &

We will not restart our FreeBSD Jail to make these changes take effect.

root@host:/ # service jail onerestart nfs_server

root@host:/ # jls
   JID  IP Address      Hostname                      Path
     2                  nfs_server                    /jail/nfs_server

root@host:/ # jexec 2

root@nfs_server:/ # 

After startup we can see that unfsd(8) is listening on a NFS 2049 port.

root@nfs_server:/ # sockstat -l4
USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS      
root     sshd       1261  4  tcp4   *:22                  *:*
root     sendmail   1241  5  tcp4   127.0.0.1:25          *:*
root     unfsd      1223  3  udp4   *:2049                *:*
root     unfsd      1223  4  tcp4   *:2049                *:*
root     rpcbind    1196  9  udp4   *:111                 *:*
root     rpcbind    1196  10 udp4   *:842                 *:*
root     rpcbind    1196  11 tcp4   *:111                 *:*
root     syslogd    1188  6  udp4   *:514                 *:*
  

We should have our epair(4) interface called e0b_nfs_server addressed properly.

root@nfs_server:/ # ifconfig 
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
e0b_nfs_server: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 0e:27:dd:b3:81:88
        hwaddr 02:30:0d:9f:57:0b
        inet 10.0.10.251 netmask 0xffffff00 broadcast 10.0.10.255
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

Mount /share on NFS Client

I added that NFS entry to the /etc/fstab file on the host machine.

root@host:~ # cat /etc/fstab 
#DEV         #MNT       #TYPE    #OPT  #DUMP/PASS
/dev/ada0p1  /boot/efi  msdosfs  rw    2 2
/dev/ada0p3  none       swap     sw    0 0

#DEV                #MNT  #TYPE  #OPT       #DUMP/PASS
10.0.10.251:/share  /mnt  nfs    rw,noauto  0 0

We will now attempt to mount the /share NFS export on the host machine.

root@host:/ # mount /mnt

root@host:/ # mount | grep share
10.0.10.251:/share on /mnt (nfs)

root@host:/ # cd /mnt

root@host:/mnt # :> FILE

root@host:/mnt # ls -l FILE
-rw-r--r-- 1 root root 0 2022-05-21 22:53 FILE

root@host:/mnt # rm FILE

Seems to work properly.

Here are also network interfaces on the host machine.

root@host:/ # ifconfig 
em0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4810099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER,NOMAP>
        ether 08:00:27:b3:81:88
        inet 10.0.10.250 netmask 0xffffff00 broadcast 10.0.10.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
em0_bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 58:9c:fc:10:ff:dd
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: e0a_nfs_server flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 4 priority 128 path cost 2000
        member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 20000
        groups: bridge
        nd6 options=9<PERFORMNUD,IFDISABLED>
e0a_nfs_server: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 02:27:dd:b3:81:88
        hwaddr 02:30:0d:9f:57:0a
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

Future of NFS Server in Jails

This setup – while allowing to run the NFS server inside FreeBSD Jail with even VNET enabled has its drawbacks unfortunately. First is that it is run in userspace instead of kernel space – which means its slower. Second is that the unfsd(8) only implements NFS version 3 – so no version 4 is not possible.

Where we can go from here? Like with WiFi stuff IMHO the FreeBSD Foundation could step in to sponsor the missing bits of NFS server and VNET to make these native tools work they should. Its up to you to put the pressure on the FreeBSD Foundation when they as what are you missing from the FreeBSD UNIX system that could be improved with one of their projects. You may also join the discussion at the 251347 Bug Report of course.

I think its a big loss that native kernel space NFS server is not currently possible with VNET FreeBSD Jails.

EOF

Unbound DNS Blacklist

Today I will show you how to configure unbound(8) to block spam/malicious/malware domains at DNS level.

I will use FreeBSD for that purpose but you can use any system that unbound(8) runs on.

Earlier I used generated /etc/hosts file but that was limited in several ways. The ZSH shell will autocomplete all these blocked domains to the ssh(1)/scp(1) commands (which takes needless time and shows useless completions). Subdomains are not handled. The malicious.com is blocked but ads.malicious.com is not. You need to duplicate all those domains in the /etc/hosts file.

TL;DR

Not all people have time for my long boring stories so this is meritum of this article.

# rm -rf /var/unbound
# mkdir -p /var/unbound/conf.d
# chown -R unbound:unbound /var/unbound
# service local_unbound setup
# service local_unbound enable
# service local_unbound start
# mkdir /root/bin
# cd 
# fetch -o /root/bin/unbound-blacklist-fetch.sh \
> https://raw.githubusercontent.com/vermaden/scripts/master/unbound-blacklist-fetch.sh
# chmod +x /root/bin/unbound-blacklist-fetch.sh
# /root/bin/unbound-blacklist-fetch.sh
# service local_unbound restart
# cat << BSD >> /var/cron/tabs/root
> # FETCH FRESH unbound(8) BLACKLIST
>   0 0 * * * /root/bin/unbound-blacklist-fetch.sh
> BSD

Whole Story

The unbound(8) caching DNS resolver has been added to FreeBSD base system in 2014 with 10.0-RELEASE version so being on FreeBSD you do not need to install anything. We will start with cleaning the any existing unbound(8) configuration which relies at /var/unbound. Keep in mind that /etc/unbound links to it.

# ls -l -d /etc/unbound /var/unbound
lrwxr-xr-x 1 root    wheel   14 2019.09.21 16:23 /etc/unbound -> ../var/unbound
drwxr-xr-x 3 unbound unbound  8 2020.11.17 16:48 /var/unbound

# rm -rf /var/unbound

# mkdir -p /var/unbound/conf.d

# chown -R unbound:unbound /var/unbound

The service local_unbound setup will create all needed configuration.

Just keep in mind that this process will setup all DNS servers that you have in the /etc/resolv.conf file.

You may want to put two of your favorite DNS servers before this process.

Configuration

# cat << BSD > /etc/resolv.conf
nameserver 9.9.9.9
nameserver 1.1.1.1
BSD

# service local_unbound setup
Performing initial setup.
destination: 
Extracting forwarders from /etc/resolv.conf.
/var/unbound/forward.conf created
/var/unbound/lan-zones.conf created
/var/unbound/control.conf created
/var/unbound/unbound.conf created
/etc/resolvconf.conf created
Original /etc/resolv.conf saved as /var/backups/resolv.conf.20201115.235254

# rm /var/backups/resolv.conf.20201115.235254

# find /var/unbound
/var/unbound
/var/unbound/lan-zones.conf
/var/unbound/control.conf
/var/unbound/unbound.conf
/var/unbound/forward.conf

% find /var/unbound -ls
 12685  17  drwxr-xr-x  3  unbound  unbound    8  Nov 17 16:48  /var/unbound
 13072   1  -rw-r--r--  1  root     unbound   98  Nov 17 05:00  /var/unbound/forward.conf
 12688   9  -rw-r--r--  1  root     unbound  354  Nov 15 23:56  /var/unbound/unbound.conf
 12686   1  drwxr-xr-x  2  unbound  unbound    3  Nov 16 00:23  /var/unbound/conf.d
 12158   9  -rw-r--r--  1  root     unbound  193  Nov 15 23:56  /var/unbound/control.conf
 11732   9  -rw-r--r--  1  root     unbound  189  Nov 15 23:56  /var/unbound/lan-zones.conf

# tail -n 999 /var/unbound/*
==> /var/unbound/conf.d <==
tail: /var/unbound/conf.d: Is a directory

==> /var/unbound/control.conf <==
# This file was generated by local-unbound-setup.
# Modifications will be overwritten.
remote-control:
	control-enable: yes
	control-interface: /var/run/local_unbound.ctl
	control-use-cert: no

==> /var/unbound/forward.conf <==
# Generated by resolvconf

forward-zone:
	name: "."
	forward-addr: 9.9.9.9
	forward-addr: 1.1.1.1

==> /var/unbound/lan-zones.conf <==
# This file was generated by local-unbound-setup.
# Modifications will be overwritten.
server:
	# Unblock reverse lookups for LAN addresses
	unblock-lan-zones: yes
	insecure-lan-zones: yes

==> /var/unbound/unbound.conf <==
# This file was generated by local-unbound-setup.
# Modifications will be overwritten.
server:
	username: unbound
	directory: /var/unbound
	chroot: /var/unbound
	pidfile: /var/run/local_unbound.pid
	auto-trust-anchor-file: /var/unbound/root.key

include: /var/unbound/lan-zones.conf
include: /var/unbound/control.conf
include: /var/unbound/conf.d/*.conf

We will now enable the local_unbound service and start it. At this point without any DNS blocking configuration.

# service local_unbound enable
local_unbound enabled in /etc/rc.conf

# service local_unbound start
Starting local_unbound.

The /etc/resolv.conf will now have hour favorite DNS servers hashed/disabled and 127.0.0.1 address will be specified. You can also use sockstat(8) to check that unbound(8) is indeed listening on port 53.

# cat /etc/resolv.conf
# nameserver 9.9.9.9
# nameserver 1.1.1.1
nameserver 127.0.0.1
options edns0

% sockstat -l -4
USER COMMAND PID FD PROTO LOCAL ADDRESS FOREIGN ADDRESS 
unbound local-unbo 7362 5 udp4 127.0.0.1:53 *:*
unbound local-unbo 7362 6 tcp4 127.0.0.1:53 *:*

Test

After unbound(8) has been enabled it should now be visible that first DNS request should be longer and the second one and following requests should be very fast.

% time host ftp.freebsd.org
ftp.freebsd.org is an alias for ftp.geo.freebsd.org.
ftp.geo.freebsd.org has address 139.178.72.202
ftp.geo.freebsd.org has address 213.138.116.78
ftp.geo.freebsd.org has address 139.178.72.202
ftp.geo.freebsd.org has IPv6 address 2604:1380:2000:9501::15:0
ftp.geo.freebsd.org has IPv6 address 2001:41c8:112:8300::15:0
ftp.geo.freebsd.org has IPv6 address 2604:1380:2000:9501::15:0
ftp.geo.freebsd.org mail is handled by 0 .
host ftp.freebsd.org  0.00s user 0.01s system 1% cpu 0.501 total

% time host ftp.freebsd.org
ftp.freebsd.org is an alias for ftp.geo.freebsd.org.
ftp.geo.freebsd.org has address 139.178.72.202
ftp.geo.freebsd.org has address 213.138.116.78
ftp.geo.freebsd.org has address 139.178.72.202
ftp.geo.freebsd.org has IPv6 address 2604:1380:2000:9501::15:0
ftp.geo.freebsd.org has IPv6 address 2001:41c8:112:8300::15:0
ftp.geo.freebsd.org has IPv6 address 2604:1380:2000:9501::15:0
ftp.geo.freebsd.org mail is handled by 0 .
host ftp.freebsd.org  0.01s user 0.00s system 88% cpu 0.007 total

Yep. Works.

Blacklist

I have written a simple and short unbound-blacklist-fetch.sh to automate the process of generating up to date DNS blocked domains config.

It uses one unbound(8) source and several hosts(5) sources, then combines them in unbound(8) compatible format while removing the duplicated entries.

We will now fetch it, put it under /root/bin directory (or use your favorite one), make it executable and start it.

# mkdir /root/bin

# fetch -o /root/bin/unbound-blacklist-fetch.sh \
> https://raw.githubusercontent.com/vermaden/scripts/master/unbound-blacklist-fetch.sh

# chmod +x /root/bin/unbound-blacklist-fetch.sh

# /root/bin/unbound-blacklist-fetch.sh

# ls -l /var/unbound/conf.d/blacklist.conf
-rw-r--r-- 1 root unbound 3003929 2020.11.16 00:23 /var/unbound/conf.d/blacklist.conf

# tail /var/unbound/conf.d/blacklist.conf
local-zone: "zyrtec.1.p2l.info" always_nxdomain
local-zone: "zyrtec.3.p2l.info" always_nxdomain
local-zone: "zyrtec.4.p2l.info" always_nxdomain
local-zone: "zyski-z-innowacji.pl" always_nxdomain
local-zone: "zytpirwai.net" always_nxdomain
local-zone: "zz.cqcounter.com" always_nxdomain
local-zone: "zzhc.vnet.cn" always_nxdomain
local-zone: "zzz.clickbank.net" always_nxdomain
local-zone: "zzz.onion.pet" always_nxdomain
local-zone: "zzzrtrcm2.com" always_nxdomain

The unbound(8) daemon already includes all /var/unbound/conf.d/*.conf files and we use that here.

You can change where the script generates blocked domains config under the # SETTINGS section directly in the script.

% grep -A 5 SETTINGS scripts/unbound-blacklist-fetch.sh 
# SETTINGS
FILE=/var/unbound/conf.d/blacklist.conf
TEMP=/tmp/unbound
TYPE=always_nxdomain
ECHO=0

After the /var/unbound/conf.d/blacklist.conf file is generated you can now restart the unbound(8) service.

# service local_unbound restart
Stopping local_unbound.
Waiting for PIDS: 87745.
Starting local_unbound.
Waiting for nameserver to start... good

We will also add that script to crontab(5) so it will fetch fresh information every day.

# cat << BSD >> /var/cron/tabs/root
> 
> # FETCH FRESH unbound(8) BLACKLIST
>   0 0 * * * /root/bin/unbound-blacklist-fetch.sh
> 
> BSD

# crontab -l | tail -4

# FETCH FRESH unbound(8) BLACKLIST
  0 0 * * * /root/bin/unbound-blacklist-fetch.sh

Test Blocked Domains

From 60000+ blocked domains I have chosen ad.track.us.org as target for verification.

% ping ad.track.us.org
ping: cannot resolve ad.track.us.org: Unknown host

% host ad.track.us.org
Host ad.track.us.org not found: 3(NXDOMAIN)

% dog ad.track.us.org
Status: NXDomain

% dog @1.1.1.1 ad.track.us.org
CNAME ad.track.us.org. 11m30s   "track.us.org."
    A track.us.org.     6m30s   185.59.208.177

As You can see the domain is successfully blocked.

The above blocking configuration does not mean that I will now disable the uBlock Origin plugin from Firefox but its a welcome addition to blocking unwanted information tools workshop.

UPDATE 1 – Reworked Script and Alternatives

After reading comments on Hacker News / Lobsters / Reddit I got a lot of good ideas how to improve my script even more.

Some people suggested that very similar functionality already exists in dns/void-zones-tools package on FreeBSD. One can also use get_unbound_adblock.sh script or lie-to-me solution.

There are also more sophisticated tools like Pi-hole which also include DHCP server and web interface for management and statistics. Unfortunately Pi-hole does not run on FreeBSD.

After reworking and adding additional sources to my unbound-blacklist-fetch.sh script its now twice the amount of blocked unwanted domains. In the first release about 60000 domains were blocked. Now its more then 120000.

Here is the distribution of data between various types of sources.

% wc -lc /tmp/unbound/lists-*
   54587 1059592 /tmp/unbound/lists-domains
  143553 4115745 /tmp/unbound/lists-hosts
   32867 1596409 /tmp/unbound/lists-unbound
  231007 6771746 total

Now the /var/unbound/conf.d/blacklist.conf before these changes.

% wc -l blacklist.conf
   60009 blacklist.conf

% ls -l /var/unbound/conf.d/blacklist.conf
-rw-r--r-- 1 root unbound 2907535 2020-11-20 00:00 /var/unbound/conf.d/blacklist.conf

… and after adding additional sources.

% wc -l blacklist.conf
  122190 blacklist.conf

% ls -l /var/unbound/conf.d/blacklist.conf
-rw-r--r-- 1 root unbound 6086623 2020-11-20 15:07 /var/unbound/conf.d/blacklist.conf

Here is also performance summary about which part takes what amount of time.

Combining various sources and generating the final config takes about 5 seconds.

Most of the time is spent in fetching the data from various sources.

The script is already uploaded to the GitHub repo.

Just fetch it and enjoy 🙂

UPDATE 2 – Huge Domains List Version

Thanks to Luca Castagnini from bsd.network who pointed me to https://oisd.nl/ site with HUGE list of domains that can/could/should be blocked I made another variant (or version) of the script unbound-blacklist-fetch-huge.sh with a total of 145 (!) various sources for domains to block.

It of course takes little longer to fetch and generate then the ‘casual’ version.

Its little less then 2 minutes to fetch and generate new config while the longest part is the fetching of those 145 sources. Generation takes about 15 seconds.

These 145 sources provide more then a million domains to block.

% wc -l /tmp/unbound/* 
 551704 lists-domains
 439505 lists-hosts
  60835 lists-unbound
1052044 total

The script after removing duplicated entries makes little more then 480000 domains of it.

% wc -l /var/unbound/conf.d/blacklist.conf 
 484829 /var/unbound/conf.d/blacklist.conf

Unfortunately it comes at a price. In this HUGE variant with domains from 145 sources the unbound(8) server now uses about 150 MB of RAM.

% top -b -o res|grep -E 'RES|unbound'
  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C    TIME    WCPU COMMAND
75849 unbound       1  20    0   158M   149M select   4    0:03   0.00% local-unbound

I leave up to you which version to use and which sources to choose for blocking, but as my Firefox with about 20 tabs opened takes little more then 4226 MB of RAM these additional 150 MB from unbound(8) does not hurt that much 🙂

% ./FIREFOX.RAM.sh
4226 MB

% cat FIREFOX.RAM.sh 
#! /bin/sh

SUM=0

top -b -o res \
  | sed 1,10d \
  | grep firefox \
  | awk '{print $7}' \
  | tr -cd '0-9\n' \
  | while read I
    do
      SUM=$(( ${SUM} + ${I} ))
      echo ${SUM}
    done | tail -1 | tr -d '\n'
echo " MB"

One more thing related to Firefox. After checking ‘free’ memory with Firefox running and after closing it the difference was about 2.6 GB which means that above script to calculate Firefox memory usage is not a lot accurate 🙂

EOF

FreeBSD Cluster with Pacemaker and Corosync

I always missed ‘proper’ cluster software for FreeBSD systems. Recently I got to run several Pacemaker/Corosync based clusters on Linux systems. I thought how to make similar high availability solutions on FreeBSD and I was really shocked when I figured out that both Pacemaker and Corosync tools are available in the FreeBSD Ports and packages as net/pacemaker2 and net/corosync2 respectively.

In this article I will check how well Pacemaker and Corosync cluster works on FreeBSD.

There are many definitions of a cluster. One that I like the most is that a cluster is a system that is still redundant after losing one of its nodes (is still a cluster). This means that 3 nodes is a minimum for a cluster by that definition. The two node clusters are quite problematic because of their biggest exposure to the split brain problem. That is why often in the two node clusters additional devices or systems are added to make sure that this split brain does not happen. For example one can add third node without any resources or services just as a ‘witness’ role. Other way is to add a shared disk resource that will serve the same purpose and often its a raw volume with SCSI-3 Persistent Reservation mechanism used.

Lab Setup

As usual it will be entirely VirtualBox based and it will consist of 3 hosts. To not create 3 same FreeBSD installations I used 12.1-RELEASE virtual machine image available from the FreeBSD Project directly:

• https://download.freebsd.org/ftp/releases/VM-IMAGES/12.1-RELEASE/

There are several formats available – qcow2/raw/vhd/vmdk – but as I will be using VirtualBox I used the VMDK one.

Here is the list of the machines for the GlusterFS cluster:

• 10.0.10.111 node1
• 10.0.10.112 node2
• 10.0.10.113 node3

Each VirtualBox virtual machine for FreeBSD is the default one (as suggested in the VirtualBox wizard) with 512 MB RAM and NAT Network as shown on the image below.

Here is the configuration of the NAT Network on VirtualBox.

Before we will try connect to our FreeBSD machines we need to make the minimal network configuration inside each VM. Each FreeBSD machine will have such minimal /etc/rc.conf file as shown example for node1 host.

root@node1:~ # cat /etc/rc.conf
hostname=node1
ifconfig_em0="inet 10.0.10.111/24 up"
defaultrouter=10.0.10.1
sshd_enable=YES

For the setup purposes we will need to allow root login on these FreeBSD machines with PermitRootLogin yes option in the /etc/ssh/sshd_config file. You will also need to restart the sshd(8) service after the changes.

root@node1:~ # grep PermitRootLogin /etc/ssh/sshd_config
PermitRootLogin yes

root@node1:~ # service sshd restart

By using NAT Network with Port Forwarding the FreeBSD machines will be accessible on the localhost ports. For example the node1 machine will be available on port 2211, the node2 machine will be available on port 2212 and so on. This is shown in the sockstat utility output below.

To connect to such machine from the VirtualBox host system you will need this command:

vboxhost % ssh -l root localhost -p 2211

Packages

As we now have ssh(1) connectivity we need to add needed packages. To make our VMs resolve DNS queries we need to add one last thing. We will also switch to ‘quarterly’ branch of the pkg(8) packages.

root@node1:~ # echo 'nameserver 1.1.1.1' > /etc/resolv.conf
root@node1:~ # sed -i '' s/quarterly/latest/g /etc/pkg/FreeBSD.conf

Remember to repeat these two upper commands on node2 and node3 systems.

Now we will add Pacemaker and Corosync packages.

root@node1:~ # pkg install pacemaker2 corosync2 crmsh

root@node2:~ # pkg install pacemaker2 corosync2 crmsh

root@node3:~ # pkg install pacemaker2 corosync2 crmsh

These are messages both from pacemaker2 and corosync2 that we need to address.

Message from pacemaker2-2.0.4:

--
For correct operation, maximum socket buffer size must be tuned
by performing the following command as root :

# sysctl kern.ipc.maxsockbuf=18874368

To preserve this setting across reboots, append the following
to /etc/sysctl.conf :

kern.ipc.maxsockbuf=18874368

======================================================================

Message from corosync2-2.4.5_1:

--
For correct operation, maximum socket buffer size must be tuned
by performing the following command as root :

# sysctl kern.ipc.maxsockbuf=18874368

To preserve this setting across reboots, append the following
to /etc/sysctl.conf :

kern.ipc.maxsockbuf=18874368

We need to change the kern.ipc.maxsockbuf parameter. Lets do it then.

root@node1:~ # echo 'kern.ipc.maxsockbuf=18874368' >> /etc/sysctl.conf
root@node1:~ # service sysctl restart

root@node2:~ # echo 'kern.ipc.maxsockbuf=18874368' >> /etc/sysctl.conf
root@node2:~ # service sysctl restart

root@node3:~ # echo 'kern.ipc.maxsockbuf=18874368' >> /etc/sysctl.conf
root@node3:~ # service sysctl restart

Lets check what binaries come with these packages.

root@node1:~ # pkg info -l pacemaker2 | grep bin
        /usr/local/sbin/attrd_updater
        /usr/local/sbin/cibadmin
        /usr/local/sbin/crm_attribute
        /usr/local/sbin/crm_diff
        /usr/local/sbin/crm_error
        /usr/local/sbin/crm_failcount
        /usr/local/sbin/crm_master
        /usr/local/sbin/crm_mon
        /usr/local/sbin/crm_node
        /usr/local/sbin/crm_report
        /usr/local/sbin/crm_resource
        /usr/local/sbin/crm_rule
        /usr/local/sbin/crm_shadow
        /usr/local/sbin/crm_simulate
        /usr/local/sbin/crm_standby
        /usr/local/sbin/crm_ticket
        /usr/local/sbin/crm_verify
        /usr/local/sbin/crmadmin
        /usr/local/sbin/fence_legacy
        /usr/local/sbin/iso8601
        /usr/local/sbin/pacemaker-remoted
        /usr/local/sbin/pacemaker_remoted
        /usr/local/sbin/pacemakerd
        /usr/local/sbin/stonith_admin

root@node1:~ # pkg info -l corosync2 | grep bin
        /usr/local/bin/corosync-blackbox
        /usr/local/sbin/corosync
        /usr/local/sbin/corosync-cfgtool
        /usr/local/sbin/corosync-cmapctl
        /usr/local/sbin/corosync-cpgtool
        /usr/local/sbin/corosync-keygen
        /usr/local/sbin/corosync-notifyd
        /usr/local/sbin/corosync-quorumtool

root@node1:~ # pkg info -l crmsh | grep bin
        /usr/local/bin/crm

Cluster Initialization

Now we will initialize our FreeBSD cluster.

First we need to make sure that names of the nodes are DNS resolvable.

root@node1:~ # tail -3 /etc/hosts

10.0.10.111 node1
10.0.10.112 node2
10.0.10.113 node3

root@node2:~ # tail -3 /etc/hosts

10.0.10.111 node1
10.0.10.112 node2
10.0.10.113 node3

root@node3:~ # tail -3 /etc/hosts

10.0.10.111 node1
10.0.10.112 node2
10.0.10.113 node3

Now we will generate the Corosync key.

root@node1:~ # corosync-keygen
Corosync Cluster Engine Authentication key generator.
Gathering 1024 bits for key from /dev/random.
Press keys on your keyboard to generate entropy.
Writing corosync key to /usr/local/etc/corosync/authkey.

root@node1:~ # echo $?
0

root@node1:~ # ls -l /usr/local/etc/corosync/authkey
-r--------  1 root  wheel  128 Sep  2 20:37 /usr/local/etc/corosync/authkey

Now the Corosync configuration file. For sure some examples were provided by the package maintainer.

root@node1:~ # pkg info -l corosync2 | grep example
        /usr/local/etc/corosync/corosync.conf.example
        /usr/local/etc/corosync/corosync.conf.example.udpu

We will take the second one as a base for our config.

root@node1:~ # cp /usr/local/etc/corosync/corosync.conf.example.udpu /usr/local/etc/corosync/corosync.conf

root@node1:~ # vi /usr/local/etc/corosync/corosync.conf
               /* LOTS OF EDITS HERE */

root@node1:~ # cat /usr/local/etc/corosync/corosync.conf

totem {
  version: 2
  crypto_cipher: aes256
  crypto_hash: sha256
  transport: udpu

  interface {
    ringnumber: 0
    bindnetaddr: 10.0.10.0
    mcastport: 5405
    ttl: 1
  }
}

logging {
  fileline: off
  to_logfile: yes
  to_syslog: no
  logfile: /var/log/cluster/corosync.log
  debug: off
  timestamp: on

  logger_subsys {
    subsys: QUORUM
    debug: off
  }
}

nodelist {

  node {
    ring0_addr: 10.0.10.111
    nodeid: 1
  }

  node {
    ring0_addr: 10.0.10.112
    nodeid: 2
  }

  node {
    ring0_addr: 10.0.10.113
    nodeid: 3
  }

}

quorum {
  provider: corosync_votequorum
  expected_votes: 2
}

Now we need to propagate both Corosync key and config across the nodes in the cluster.

We can use some simple tools created exactly for that like net/csync2 cluster synchronization tool for example but plain old net/rsync will serve as well.

root@node1:~ # pkg install -y rsync

root@node1:~ # rsync -av /usr/local/etc/corosync/ node2:/usr/local/etc/corosync/
The authenticity of host 'node2 (10.0.10.112)' can't be established.
ECDSA key fingerprint is SHA256:/ZDmln7GKi6n0kbad73TIrajPjGfQqJJX+ReSf3NMvc.
No matching host key fingerprint found in DNS.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'node2' (ECDSA) to the list of known hosts.
Password for root@node2:
sending incremental file list
./
authkey
corosync.conf
service.d/
uidgid.d/

sent 1,100 bytes  received 69 bytes  259.78 bytes/sec
total size is 4,398  speedup is 3.76

root@node1:~ # rsync -av /usr/local/etc/corosync/ node3:/usr/local/etc/corosync/
The authenticity of host 'node2 (10.0.10.112)' can't be established.
ECDSA key fingerprint is SHA256:/ZDmln7GKi6n0kbad73TIrajPjGfQqJJX+ReSf3NMvc.
No matching host key fingerprint found in DNS.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'node3' (ECDSA) to the list of known hosts.
Password for root@node3:
sending incremental file list
./
authkey
corosync.conf
service.d/
uidgid.d/

sent 1,100 bytes  received 69 bytes  259.78 bytes/sec
total size is 4,398  speedup is 3.76

Now lets check that they are the same.

root@node1:~ # cksum /usr/local/etc/corosync/{authkey,corosync.conf}
2277171666 128 /usr/local/etc/corosync/authkey
1728717329 622 /usr/local/etc/corosync/corosync.conf

root@node2:~ # cksum /usr/local/etc/corosync/{authkey,corosync.conf}
2277171666 128 /usr/local/etc/corosync/authkey
1728717329 622 /usr/local/etc/corosync/corosync.conf

root@node3:~ # cksum /usr/local/etc/corosync/{authkey,corosync.conf}
2277171666 128 /usr/local/etc/corosync/authkey
1728717329 622 /usr/local/etc/corosync/corosync.conf

Same.

We can now add corosync_enable=YES and pacemaker_enable=YES to the /etc/rc.conf file.

root@node1:~ # sysrc corosync_enable=YES
corosync_enable:  -> YES

root@node1:~ # sysrc pacemaker_enable=YES
pacemaker_enable:  -> YES

root@node2:~ # sysrc corosync_enable=YES
corosync_enable:  -> YES

root@node2:~ # sysrc pacemaker_enable=YES
pacemaker_enable:  -> YES

root@node3:~ # sysrc corosync_enable=YES
corosync_enable:  -> YES

root@node3:~ # sysrc pacemaker_enable=YES
pacemaker_enable:  -> YES

Lets start these services then.

root@node1:~ # service corosync start
Starting corosync.
Sep 02 20:55:35 notice  [MAIN  ] Corosync Cluster Engine ('2.4.5'): started and ready to provide service.
Sep 02 20:55:35 info    [MAIN  ] Corosync built-in features:
Sep 02 20:55:35 warning [MAIN  ] interface section bindnetaddr is used together with nodelist. Nodelist one is going to be used.
Sep 02 20:55:35 warning [MAIN  ] Please migrate config file to nodelist.

root@node1:~ # ps aux | grep corosync
root  1695   0.0  7.9 38340 38516  -  S    20:55    0:00.40 /usr/local/sbin/corosync
root  1699   0.0  0.1   524   336  0  R+   20:57    0:00.00 grep corosync

Do the same on the node2 and node3 systems.

The Pacemaker is not yet running so that will fail.

root@node1:~ # crm status
Could not connect to the CIB: Socket is not connected
crm_mon: Error: cluster is not available on this node
ERROR: status: crm_mon (rc=102): 

We will start it now.

root@node1:~ # service pacemaker start
Starting pacemaker.

root@node2:~ # service pacemaker start
Starting pacemaker.

root@node3:~ # service pacemaker start
Starting pacemaker.

You need to give it little time to start because if you will execute crm status command right away you will get 0 nodes configured message as shown below.

root@node1:~ # crm status
Cluster Summary:
  * Stack: unknown
  * Current DC: NONE
  * Last updated: Wed Sep  2 20:58:51 2020
  * Last change:  
  * 0 nodes configured
  * 0 resource instances configured


Full List of Resources:
  * No resources

… but after a while everything is detected and works as desired.

root@node1:~ # crm status
Cluster Summary:
  * Stack: corosync
  * Current DC: node2 (version 2.0.4-2deceaa3ae) - partition with quorum
  * Last updated: Wed Sep  2 21:02:49 2020
  * Last change:  Wed Sep  2 20:59:00 2020 by hacluster via crmd on node2
  * 3 nodes configured
  * 0 resource instances configured

Node List:
  * Online: [ node1 node2 node3 ]

Full List of Resources:
  * No resources

The Pacemaker runs properly.

root@node1:~ # ps aux | grep pacemaker
root      1716   0.0  0.5 10844   2396  -  Is   20:58     0:00.00 daemon: /usr/local/sbin/pacemakerd[1717] (daemon)
root      1717   0.0  5.2 49264  25284  -  S    20:58     0:00.27 /usr/local/sbin/pacemakerd
hacluster 1718   0.0  6.1 48736  29708  -  Ss   20:58     0:00.75 /usr/local/libexec/pacemaker/pacemaker-based
root      1719   0.0  4.5 40628  21984  -  Ss   20:58     0:00.28 /usr/local/libexec/pacemaker/pacemaker-fenced
root      1720   0.0  2.8 25204  13688  -  Ss   20:58     0:00.20 /usr/local/libexec/pacemaker/pacemaker-execd
hacluster 1721   0.0  3.9 38148  19100  -  Ss   20:58     0:00.25 /usr/local/libexec/pacemaker/pacemaker-attrd
hacluster 1722   0.0  2.9 25460  13864  -  Ss   20:58     0:00.17 /usr/local/libexec/pacemaker/pacemaker-schedulerd
hacluster 1723   0.0  5.4 49304  26300  -  Ss   20:58     0:00.41 /usr/local/libexec/pacemaker/pacemaker-controld
root      1889   0.0  0.6 11348   2728  0  S+   21:56     0:00.00 grep pacemaker

We can check how Corosync sees its members.

root@node1:~ # corosync-cmapctl | grep members
runtime.totem.pg.mrp.srp.members.1.config_version (u64) = 0
runtime.totem.pg.mrp.srp.members.1.ip (str) = r(0) ip(10.0.10.111) 
runtime.totem.pg.mrp.srp.members.1.join_count (u32) = 1
runtime.totem.pg.mrp.srp.members.1.status (str) = joined
runtime.totem.pg.mrp.srp.members.2.config_version (u64) = 0
runtime.totem.pg.mrp.srp.members.2.ip (str) = r(0) ip(10.0.10.112) 
runtime.totem.pg.mrp.srp.members.2.join_count (u32) = 1
runtime.totem.pg.mrp.srp.members.2.status (str) = joined
runtime.totem.pg.mrp.srp.members.3.config_version (u64) = 0
runtime.totem.pg.mrp.srp.members.3.ip (str) = r(0) ip(10.0.10.113) 
runtime.totem.pg.mrp.srp.members.3.join_count (u32) = 1
runtime.totem.pg.mrp.srp.members.3.status (str) = joined

… or the quorum information.

root@node1:~ # corosync-quorumtool
Quorum information
------------------
Date:             Wed Sep  2 21:00:38 2020
Quorum provider:  corosync_votequorum
Nodes:            3
Node ID:          1
Ring ID:          1/12
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   3
Highest expected: 3
Total votes:      3
Quorum:           2  
Flags:            Quorate 

Membership information
----------------------
    Nodeid      Votes Name
         1          1 10.0.10.111 (local)
         2          1 10.0.10.112
         3          1 10.0.10.113

The Corosync log file is filled with the following information.

root@node1:~ # cat /var/log/cluster/corosync.log
Sep 02 20:55:35 [1694] node1 corosync notice  [MAIN  ] Corosync Cluster Engine ('2.4.5'): started and ready to provide service.
Sep 02 20:55:35 [1694] node1 corosync info    [MAIN  ] Corosync built-in features:
Sep 02 20:55:35 [1694] node1 corosync warning [MAIN  ] interface section bindnetaddr is used together with nodelist. Nodelist one is going to be used.
Sep 02 20:55:35 [1694] node1 corosync warning [MAIN  ] Please migrate config file to nodelist.
Sep 02 20:55:35 [1694] node1 corosync notice  [TOTEM ] Initializing transport (UDP/IP Unicast).
Sep 02 20:55:35 [1694] node1 corosync notice  [TOTEM ] Initializing transmit/receive security (NSS) crypto: aes256 hash: sha256
Sep 02 20:55:35 [1694] node1 corosync notice  [TOTEM ] The network interface [10.0.10.111] is now up.
Sep 02 20:55:35 [1694] node1 corosync notice  [SERV  ] Service engine loaded: corosync configuration map access [0]
Sep 02 20:55:35 [1694] node1 corosync info    [QB    ] server name: cmap
Sep 02 20:55:35 [1694] node1 corosync notice  [SERV  ] Service engine loaded: corosync configuration service [1]
Sep 02 20:55:35 [1694] node1 corosync info    [QB    ] server name: cfg
Sep 02 20:55:35 [1694] node1 corosync notice  [SERV  ] Service engine loaded: corosync cluster closed process group service v1.01 [2]
Sep 02 20:55:35 [1694] node1 corosync info    [QB    ] server name: cpg
Sep 02 20:55:35 [1694] node1 corosync notice  [SERV  ] Service engine loaded: corosync profile loading service [4]
Sep 02 20:55:35 [1694] node1 corosync notice  [QUORUM] Using quorum provider corosync_votequorum
Sep 02 20:55:35 [1694] node1 corosync notice  [SERV  ] Service engine loaded: corosync vote quorum service v1.0 [5]
Sep 02 20:55:35 [1694] node1 corosync info    [QB    ] server name: votequorum
Sep 02 20:55:35 [1694] node1 corosync notice  [SERV  ] Service engine loaded: corosync cluster quorum service v0.1 [3]
Sep 02 20:55:35 [1694] node1 corosync info    [QB    ] server name: quorum
Sep 02 20:55:35 [1694] node1 corosync notice  [TOTEM ] adding new UDPU member {10.0.10.111}
Sep 02 20:55:35 [1694] node1 corosync notice  [TOTEM ] adding new UDPU member {10.0.10.112}
Sep 02 20:55:35 [1694] node1 corosync notice  [TOTEM ] adding new UDPU member {10.0.10.113}
Sep 02 20:55:35 [1694] node1 corosync notice  [TOTEM ] A new membership (10.0.10.111:4) was formed. Members joined: 1
Sep 02 20:55:35 [1694] node1 corosync warning [CPG   ] downlist left_list: 0 received
Sep 02 20:55:35 [1694] node1 corosync notice  [QUORUM] Members[1]: 1
Sep 02 20:55:35 [1694] node1 corosync notice  [MAIN  ] Completed service synchronization, ready to provide service.
Sep 02 20:58:14 [1694] node1 corosync notice  [TOTEM ] A new membership (10.0.10.111:8) was formed. Members joined: 2
Sep 02 20:58:14 [1694] node1 corosync warning [CPG   ] downlist left_list: 0 received
Sep 02 20:58:14 [1694] node1 corosync warning [CPG   ] downlist left_list: 0 received
Sep 02 20:58:14 [1694] node1 corosync notice  [QUORUM] This node is within the primary component and will provide service.
Sep 02 20:58:14 [1694] node1 corosync notice  [QUORUM] Members[2]: 1 2
Sep 02 20:58:14 [1694] node1 corosync notice  [MAIN  ] Completed service synchronization, ready to provide service.
Sep 02 20:58:19 [1694] node1 corosync notice  [TOTEM ] A new membership (10.0.10.111:12) was formed. Members joined: 3
Sep 02 20:58:19 [1694] node1 corosync warning [CPG   ] downlist left_list: 0 received
Sep 02 20:58:19 [1694] node1 corosync warning [CPG   ] downlist left_list: 0 received
Sep 02 20:58:19 [1694] node1 corosync warning [CPG   ] downlist left_list: 0 received
Sep 02 20:58:19 [1694] node1 corosync notice  [QUORUM] Members[3]: 1 2 3
Sep 02 20:58:19 [1694] node1 corosync notice  [MAIN  ] Completed service synchronization, ready to provide service.

Here is the configuration.

root@node1:~ # crm configure show
node 1: node1
node 2: node2
node 3: node3
property cib-bootstrap-options: \
        have-watchdog=false \
        dc-version=2.0.4-2deceaa3ae \
        cluster-infrastructure=corosync

As we will not be configuring the STONITH mechanism we will disable it.

root@node1:~ # crm configure property stonith-enabled=false

New configuraion with STONITH disabled.

root@node1:~ # crm configure show
node 1: node1
node 2: node2
node 3: node3
property cib-bootstrap-options: \
        have-watchdog=false \
        dc-version=2.0.4-2deceaa3ae \
        cluster-infrastructure=corosync \
        stonith-enabled=false

The STONITH configuration is out of scope of this article but properly configured STONITH looks like that.

First Service

We will now configure our first highly available service – a classic – a floating IP address 🙂

root@node1:~ # crm configure primitive IP ocf:heartbeat:IPaddr2 params ip=10.0.10.200 cidr_netmask="24" op monitor interval="30s"

Lets check how it behaves.

root@node1:~ # crm configure show
node 1: node1
node 2: node2
node 3: node3
primitive IP IPaddr2 \
        params ip=10.0.10.200 cidr_netmask=24 \
        op monitor interval=30s
property cib-bootstrap-options: \
        have-watchdog=false \
        dc-version=2.0.4-2deceaa3ae \
        cluster-infrastructure=corosync \
        stonith-enabled=false

Looks good – lets check the cluster status.

root@node1:~ # crm status
Cluster Summary:
  * Stack: corosync
  * Current DC: node2 (version 2.0.4-2deceaa3ae) - partition with quorum
  * Last updated: Wed Sep  2 22:03:35 2020
  * Last change:  Wed Sep  2 22:02:53 2020 by root via cibadmin on node1
  * 3 nodes configured
  * 1 resource instance configured

Node List:
  * Online: [ node1 node2 node3 ]

Full List of Resources:
  * IP  (ocf::heartbeat:IPaddr2):        Stopped

Failed Resource Actions:
  * IP_monitor_0 on node3 'not installed' (5): call=5, status='complete', exitreason='Setup problem: couldn't find command: ip', last-rc-change='2020-09-02 22:02:53Z', queued=0ms, exec=132ms
  * IP_monitor_0 on node2 'not installed' (5): call=5, status='complete', exitreason='Setup problem: couldn't find command: ip', last-rc-change='2020-09-02 22:02:54Z', queued=0ms, exec=120ms
  * IP_monitor_0 on node1 'not installed' (5): call=5, status='complete', exitreason='Setup problem: couldn't find command: ip', last-rc-change='2020-09-02 22:02:53Z', queued=0ms, exec=110ms

Crap. Linuxism. The ip(8) command is expected to be present in the system. This is FreeBSD and as any UNIX system it comes with ifconfig(8) command instead.

We will have to figure something else. For now we will delete our useless IP service.

root@node1:~ # crm configure delete IP

Status after deletion.

root@node1:~ # crm status
Cluster Summary:
  * Stack: corosync
  * Current DC: node2 (version 2.0.4-2deceaa3ae) - partition with quorum
  * Last updated: Wed Sep  2 22:04:34 2020
  * Last change:  Wed Sep  2 22:04:31 2020 by root via cibadmin on node1
  * 3 nodes configured
  * 0 resource instances configured

Node List:
  * Online: [ node1 node2 node3 ]

Full List of Resources:
  * No resources

Custom Resource

Lets check what resources are available by stock Pacemaker installation.

root@node1:~ # ls -l /usr/local/lib/ocf/resource.d/pacemaker
total 144
-r-xr-xr-x  1 root  wheel   7484 Aug 29 01:22 ClusterMon
-r-xr-xr-x  1 root  wheel   9432 Aug 29 01:22 Dummy
-r-xr-xr-x  1 root  wheel   5256 Aug 29 01:22 HealthCPU
-r-xr-xr-x  1 root  wheel   5342 Aug 29 01:22 HealthIOWait
-r-xr-xr-x  1 root  wheel   9450 Aug 29 01:22 HealthSMART
-r-xr-xr-x  1 root  wheel   6186 Aug 29 01:22 Stateful
-r-xr-xr-x  1 root  wheel  11370 Aug 29 01:22 SysInfo
-r-xr-xr-x  1 root  wheel   5856 Aug 29 01:22 SystemHealth
-r-xr-xr-x  1 root  wheel   7382 Aug 29 01:22 attribute
-r-xr-xr-x  1 root  wheel   7854 Aug 29 01:22 controld
-r-xr-xr-x  1 root  wheel  16134 Aug 29 01:22 ifspeed
-r-xr-xr-x  1 root  wheel  11040 Aug 29 01:22 o2cb
-r-xr-xr-x  1 root  wheel  11696 Aug 29 01:22 ping
-r-xr-xr-x  1 root  wheel   6356 Aug 29 01:22 pingd
-r-xr-xr-x  1 root  wheel   3702 Aug 29 01:22 remote

Not many … we will try to modify the Dummy service into an IP changer on FreeBSD.

root@node1:~ # cp /usr/local/lib/ocf/resource.d/pacemaker/Dummy /usr/local/lib/ocf/resource.d/pacemaker/ifconfig

root@node1:~ # vi /usr/local/lib/ocf/resource.d/pacemaker/ifconfig
               /* LOTS OF TYPING */

Because of the WordPress blogging system limitations I am forced to post this ifconfig resource as an image … but fear not – the text version is also available here – ifconfig.odt – for download.

Also the first version did not went that well …

root@node1:~ # setenv OCF_ROOT /usr/local/lib/ocf
root@node1:~ # ocf-tester -n resourcename /usr/local/lib/ocf/resource.d/pacemaker/ifconfig
Beginning tests for /usr/local/lib/ocf/resource.d/pacemaker/ifconfig...
* rc=3: Your agent has too restrictive permissions: should be 755
-:1: parser error : Start tag expected, '<' not found
usage: /usr/local/lib/ocf/resource.d/pacemaker/ifconfig {start|stop|monitor}
^
* rc=1: Your agent produces meta-data which does not conform to ra-api-1.dtd
* rc=3: Your agent does not support the meta-data action
* rc=3: Your agent does not support the validate-all action
* rc=0: Monitoring a stopped resource should return 7
* rc=0: The initial probe for a stopped resource should return 7 or 5 even if all binaries are missing
* Your agent does not support the notify action (optional)
* Your agent does not support the demote action (optional)
* Your agent does not support the promote action (optional)
* Your agent does not support master/slave (optional)
* rc=0: Monitoring a stopped resource should return 7
* rc=0: Monitoring a stopped resource should return 7
* rc=0: Monitoring a stopped resource should return 7
* Your agent does not support the reload action (optional)
Tests failed: /usr/local/lib/ocf/resource.d/pacemaker/ifconfig failed 9 tests

But after adding 755 mode to it and making several (hundred) changes it become usable.

root@node1:~ # vi /usr/local/lib/ocf/resource.d/pacemaker/ifconfig
             /* LOTS OF NERVOUS TYPING */
root@node1:~ # chmod 755 /usr/local/lib/ocf/resource.d/pacemaker/ifconfig
root@node1:~ # setenv OCF_ROOT /usr/local/lib/ocf
root@node1:~ # ocf-tester -n resourcename /usr/local/lib/ocf/resource.d/pacemaker/ifconfig
Beginning tests for /usr/local/lib/ocf/resource.d/pacemaker/ifconfig...
* Your agent does not support the notify action (optional)
* Your agent does not support the demote action (optional)
* Your agent does not support the promote action (optional)
* Your agent does not support master/slave (optional)
* Your agent does not support the reload action (optional)
/usr/local/lib/ocf/resource.d/pacemaker/ifconfig passed all tests

Looks usable.

The ifconfig resource. Its pretty limited and with hardcoded IP address as for now.

Lets try to add new IP resource to our FreeBSD cluster.

Tests

root@node1:~ # crm configure primitive IP ocf:pacemaker:ifconfig op monitor interval="30"

Added.

Lets see what status command now shows.

root@node1:~ # crm status
Cluster Summary:
  * Stack: corosync
  * Current DC: node2 (version 2.0.4-2deceaa3ae) - partition with quorum
  * Last updated: Wed Sep  2 22:44:52 2020
  * Last change:  Wed Sep  2 22:44:44 2020 by root via cibadmin on node1
  * 3 nodes configured
  * 1 resource instance configured

Node List:
  * Online: [ node1 node2 node3 ]

Full List of Resources:
  * IP  (ocf::pacemaker:ifconfig):       Started node1

Failed Resource Actions:
  * IP_monitor_0 on node3 'not installed' (5): call=24, status='Not installed', exitreason='', last-rc-change='2020-09-02 22:42:52Z', queued=0ms, exec=5ms
  * IP_monitor_0 on node2 'not installed' (5): call=24, status='Not installed', exitreason='', last-rc-change='2020-09-02 22:42:53Z', queued=0ms, exec=2ms

Crap. I forgot to copy this new ifconfig resource to the other nodes. Lets fix that now.

root@node1:~ # rsync -av /usr/local/lib/ocf/resource.d/pacemaker/ node2:/usr/local/lib/ocf/resource.d/pacemaker/
Password for root@node2:
sending incremental file list
./
ifconfig

sent 3,798 bytes  received 38 bytes  1,534.40 bytes/sec
total size is 128,003  speedup is 33.37

root@node1:~ # rsync -av /usr/local/lib/ocf/resource.d/pacemaker/ node3:/usr/local/lib/ocf/resource.d/pacemaker/
Password for root@node3:
sending incremental file list
./
ifconfig

sent 3,798 bytes  received 38 bytes  1,534.40 bytes/sec
total size is 128,003  speedup is 33.37

Lets stop, delete and re-add our precious resource now.

root@node1:~ # crm resource stop IP
root@node1:~ # crm configure delete IP
root@node1:~ # crm configure primitive IP ocf:pacemaker:ifconfig op monitor interval="30"

Fingers crossed.

root@node1:~ # crm status
Cluster Summary:
  * Stack: corosync
  * Current DC: node2 (version 2.0.4-2deceaa3ae) - partition with quorum
  * Last updated: Wed Sep  2 22:45:46 2020
  * Last change:  Wed Sep  2 22:45:43 2020 by root via cibadmin on node1
  * 3 nodes configured
  * 1 resource instance configured

Node List:
  * Online: [ node1 node2 node3 ]

Full List of Resources:
  * IP  (ocf::pacemaker:ifconfig):       Started node1

Looks like running properly.

Lets verify that its really up where it should be.

root@node1:~ # ifconfig em0
em0: flags=8843 metric 0 mtu 1500
        options=81009b
        ether 08:00:27:2a:78:60
        inet 10.0.10.111 netmask 0xffffff00 broadcast 10.0.10.255
        inet 10.0.10.200 netmask 0xffffff00 broadcast 10.0.10.255
        media: Ethernet autoselect (1000baseT )
        status: active
        nd6 options=29

root@node2:~ # ifconfig em0
em0: flags=8843 metric 0 mtu 1500
        options=81009b
        ether 08:00:27:80:50:05
        inet 10.0.10.112 netmask 0xffffff00 broadcast 10.0.10.255
        media: Ethernet autoselect (1000baseT )
        status: active
        nd6 options=29

root@node3:~ # ifconfig em0
em0: flags=8843 metric 0 mtu 1500
        options=81009b
        ether 08:00:27:74:5e:b9
        inet 10.0.10.113 netmask 0xffffff00 broadcast 10.0.10.255
        media: Ethernet autoselect (1000baseT )
        status: active
        nd6 options=29

Seems to be working.

Now lets try to move it to the other node in the cluster.

root@node1:~ # crm resource move IP node3
INFO: Move constraint created for IP to node3

root@node1:~ # crm status
Cluster Summary:
  * Stack: corosync
  * Current DC: node2 (version 2.0.4-2deceaa3ae) - partition with quorum
  * Last updated: Wed Sep  2 22:47:31 2020
  * Last change:  Wed Sep  2 22:47:28 2020 by root via crm_resource on node1
  * 3 nodes configured
  * 1 resource instance configured

Node List:
  * Online: [ node1 node2 node3 ]

Full List of Resources:
  * IP  (ocf::pacemaker:ifconfig):       Started node3

Switched properly to node3 system.

root@node3:~ # ifconfig em0
em0: flags=8843 metric 0 mtu 1500
        options=81009b
        ether 08:00:27:74:5e:b9
        inet 10.0.10.113 netmask 0xffffff00 broadcast 10.0.10.255
        inet 10.0.10.200 netmask 0xffffff00 broadcast 10.0.10.255
        media: Ethernet autoselect (1000baseT )
        status: active
        nd6 options=29

root@node1:~ # ifconfig em0
em0: flags=8843 metric 0 mtu 1500
        options=81009b
        ether 08:00:27:2a:78:60
        inet 10.0.10.111 netmask 0xffffff00 broadcast 10.0.10.255
        media: Ethernet autoselect (1000baseT )
        status: active
        nd6 options=29

Now we will poweroff the node3 system to check it that IP is really highly available.

root@node2:~ # crm status
Cluster Summary:
  * Stack: corosync
  * Current DC: node2 (version 2.0.4-2deceaa3ae) - partition with quorum
  * Last updated: Wed Sep  2 22:49:57 2020
  * Last change:  Wed Sep  2 22:47:29 2020 by root via crm_resource on node1
  * 3 nodes configured
  * 1 resource instance configured

Node List:
  * Online: [ node1 node2 node3 ]

Full List of Resources:
  * IP  (ocf::pacemaker:ifconfig):       Started node3

root@node3:~ # poweroff

root@node2:~ # crm status
Cluster Summary:
  * Stack: corosync
  * Current DC: node2 (version 2.0.4-2deceaa3ae) - partition with quorum
  * Last updated: Wed Sep  2 22:50:16 2020
  * Last change:  Wed Sep  2 22:47:29 2020 by root via crm_resource on node1
  * 3 nodes configured
  * 1 resource instance configured

Node List:
  * Online: [ node1 node2 ]
  * OFFLINE: [ node3 ]

Full List of Resources:
  * IP  (ocf::pacemaker:ifconfig):       Started node1

Seems that failover went well.

The crm command also colors various sections of its output.

Good to know that Pacemaker and Corosync cluster runs well on FreeBSD.

Some work is needed to write the needed resource files but one with some time and determination can surely put FreeBSD into a very capable highly available cluster.

EOF

Run broot on FreeBSD

The broot file manager is quite fresh and nice approach to files and directories filtering/searching/view/manipulation/… and whatever else you call messing with files 🙂

The broot tools is not yet available on the FreeBSD systems (as package or port).

This guide will show you how to built and install it on your FreeBSD system.

Here is how it looks in action.

Filter for jails.

Filter for zfs.

It has ‘size mode’ when started with -s option similar to ncdu(1) tool.

You can also check the Feature Showcase section on their GitHub page – https://github.com/Canop/broot – available here.

Build

There are three steps to make it happen.

1. You need to install the rust package.

# pkg install rust

Then you need to type (as regular user) the cargo install broot command.

% cargo install broot

It will fail here:

You will need to apply this patch below:

% diff -u \
  /home/vermaden/.cargo/registry/src/github.com-1ecc6299db9ec823/crossterm-0.14.1/src/terminal/sys/unix.rs.ORG \
  /home/vermaden/.cargo/registry/src/github.com-1ecc6299db9ec823/crossterm-0.14.1/src/terminal/sys/unix.rs
--- /home/vermaden/.cargo/registry/src/github.com-1ecc6299db9ec823/crossterm-0.14.1/src/terminal/sys/unix.rs.ORG  2020-01-10 23:41:29.825912000 +0100
+++ /home/vermaden/.cargo/registry/src/github.com-1ecc6299db9ec823/crossterm-0.14.1/src/terminal/sys/unix.rs      2020-01-10 23:41:07.703471000 +0100
@@ -33,7 +33,7 @@
         ws_ypixel: 0,
     };
 
-    if let Ok(true) = wrap_with_result(unsafe { ioctl(STDOUT_FILENO, TIOCGWINSZ, &mut size) }) {
+    if let Ok(true) = wrap_with_result(unsafe { ioctl(STDOUT_FILENO, TIOCGWINSZ.into(), &mut size) }) {
         Ok((size.ws_col, size.ws_row))
     } else {
         tput_size().ok_or_else(|| std::io::Error::last_os_error().into())

Then type cargo install broot command again. It will now properly compile.

% cargo install broot
    Updating crates.io index
  Downloaded broot v0.11.6
  Downloaded 1 crate (1.6 MB) in 2.89s
  Installing broot v0.11.6
   Compiling libc v0.2.66
   Compiling cfg-if v0.1.10
   Compiling lazy_static v1.4.0
   Compiling autocfg v0.1.7
   Compiling semver-parser v0.7.0
   Compiling autocfg v1.0.0
   Compiling proc-macro2 v1.0.7
   Compiling log v0.4.8
   Compiling scopeguard v1.0.0
   Compiling unicode-xid v0.2.0
   Compiling bitflags v1.2.1
   Compiling syn v1.0.13
   Compiling memchr v2.2.1
   Compiling arc-swap v0.4.4
   Compiling slab v0.4.2
   Compiling smallvec v1.1.0
   Compiling serde v1.0.104
   Compiling unicode-width v0.1.7
   Compiling regex-syntax v0.6.13
   Compiling ansi_term v0.11.0
   Compiling strsim v0.8.0
   Compiling vec_map v0.8.1
   Compiling id-arena v2.2.1
   Compiling custom_error v1.7.1
   Compiling glob v0.3.0
   Compiling open v1.3.2
   Compiling umask v0.1.8
   Compiling thread_local v1.0.0
   Compiling minimad v0.6.3
   Compiling lazy-regex v0.1.2
   Compiling semver v0.9.0
   Compiling lock_api v0.3.3
   Compiling crossbeam-utils v0.7.0
   Compiling crossbeam-epoch v0.8.0
   Compiling num-traits v0.2.11
   Compiling num-integer v0.1.42
   Compiling textwrap v0.11.0
   Compiling rustc_version v0.2.3
   Compiling memoffset v0.5.3
   Compiling iovec v0.1.4
   Compiling net2 v0.2.33
   Compiling dirs-sys v0.3.4
   Compiling parking_lot_core v0.7.0
   Compiling signal-hook-registry v1.2.0
   Compiling time v0.1.42
   Compiling atty v0.2.14
   Compiling users v0.9.1
   Compiling quote v1.0.2
   Compiling aho-corasick v0.7.6
   Compiling mio v0.6.21
   Compiling dirs v2.0.2
   Compiling directories v2.0.2
   Compiling parking_lot v0.10.0
   Compiling clap v2.33.0
   Compiling crossbeam-queue v0.2.1
   Compiling crossbeam-channel v0.4.0
   Compiling toml v0.5.5
   Compiling term v0.6.1
   Compiling regex v1.3.3
   Compiling signal-hook v0.1.12
   Compiling chrono v0.4.10
   Compiling crossterm v0.14.1
   Compiling simplelog v0.7.4
   Compiling crossbeam-deque v0.7.2
   Compiling thiserror-impl v1.0.9
   Compiling crossbeam v0.7.3
   Compiling thiserror v1.0.9
   Compiling termimad v0.8.9
   Compiling broot v0.11.6
    Finished release [optimized] target(s) in 4m 56s
  Installing /home/vermaden/.cargo/bin/broot
   Installed package `broot v0.11.6` (executable `broot`)
warning: be sure to add `/home/vermaden/.cargo/bin` to your PATH to be able to run the installed binaries

% echo $?
0

Install

Now go to the ~/.cargo/bin directory and copy the broot binary to some place that is set in your ${PATH} variable.

Then start new terminal (updated ${PATH} variable) and type broot command.

% cp ~/.cargo/bin/broot ~/scripts
% rehash
% broot

You will be asked if automatic setup of the br function should tool place. I agreed with y answer.

Here are things generated by this process.

% find ~/.config/broot
/home/vermaden/.config/broot
/home/vermaden/.config/broot/conf.toml
/home/vermaden/.config/broot/launcher
/home/vermaden/.config/broot/launcher/installed-v1
/home/vermaden/.config/broot/launcher/bash
/home/vermaden/.config/broot/launcher/bash/br

% find ~/.local/share/broot
/home/vermaden/.local/share/broot
/home/vermaden/.local/share/broot/launcher
/home/vermaden/.local/share/broot/launcher/fish
/home/vermaden/.local/share/broot/launcher/fish/1.fish
/home/vermaden/.local/share/broot/launcher/bash
/home/vermaden/.local/share/broot/launcher/bash/1

As I use ZSH shell it also updates my ~/.zshrc file.

% tail -3 ~/.zshrc

source /home/vermaden/.config/broot/launcher/bash/br

Finished. You now have broot installed and ready to use.

UPDATE 1 – Now No Patches Are Needed

Thanks to the broot author any patches are now not needed.

It builds and works out of the box.

UPDATE 2 – Its in Ports/Packages Now

The broot file manager is now available via usual FreeBSD Ports and packages which makes this guide pointless 🙂

Its available as misc/broot port.

EOF

 

List Block Devices on FreeBSD lsblk(8) Style

When I have to work on Linux systems I usually miss many nice FreeBSD tools such as these for example to name the few:

• sockstat
• gstat
• top -b -o res
• top -m io -o total
• usbconfig
• rcorder
• beadm/bectl
• idprio/rtprio

… but sometimes – which rarely happens – Linux has some very useful tool that is not available on FreeBSD. An example of such tool is lsblk(8) that does one thing and does it quite well – lists block devices and their contents. It has some problems like listing a disk that is entirely used under ZFS pool on which lsblk(8) displays two partitions instead of information about ZFS just being there – but we all know how much in some circles the CDDL licensed ZFS is unloved in that GPL world.

Example lsblk(8) output from Linux system:

$ lsblk
NAME                         MAJ:MIN RM   SIZE RO TYPE   MOUNTPOINT
sr0                           11:0    1  1024M  0 rom
sda                            8:0    0 931.5G  0 disk
|-sda1                         8:1    0   500M  0 part   /boot
`-sda2                         8:2    0   931G  0 part
  |-vg_local-lv_root (dm-0)  253:0    0    50G  0 lvm    /
  |-vg_local-lv_swap (dm-1)  253:1    0  17.7G  0 lvm    [SWAP]
  `-vg_local-lv_home (dm-2)  253:2    0   1.8T  0 lvm    /home
sdc                            8:32   0 232.9G  0 disk
`-sdc1                         8:33   0 232.9G  0 part
  `-md1                        9:1    0 232.9G  0 raid10 /data
sdd                            8:48   0 232.9G  0 disk
`-sdd1                         8:49   0 232.9G  0 part
  `-md1                        9:1    0 232.9G  0 raid10 /data

What FreeBSD offers in this department? The camcontrol(8) and geom(8) commands are available. You can also use gpart(8) command to list partitions. Below you will find output of these commands from my single disk laptop. Please note that because of WordPress limitations I need to change all > < characters to ] [ ones in the commands outputs.

# camcontrol devlist
[Samsung SSD 860 EVO mSATA 1TB RVT41B6Q]  at scbus1 target 0 lun 0 (ada0,pass0)

% geom disk list
Geom name: ada0
Providers:
1. Name: ada0
   Mediasize: 1000204886016 (932G)
   Sectorsize: 512
   Mode: r1w1e2
   descr: Samsung SSD 860 EVO mSATA 1TB
   lunid: 5002538e402b4ddd
   ident: S41PNB0K303632D
   rotationrate: 0
   fwsectors: 63
   fwheads: 1

# gpart show
=>        40  1953525088  ada0  GPT  (932G)
          40      409600     1  efi  (200M)
      409640        1024     2  freebsd-boot  (512K)
      410664         984        - free -  (492K)
      411648  1953112064     3  freebsd-zfs  (931G)
  1953523712        1416        - free -  (708K)

They provide needed information in acceptable manner but only on systems with small amount of disks. What if you would like to display a summary of all system drives contents? This is where lsblk.sh comes handy. While lsblk(8) has many interesting features like --perms/--scsi/--inverse modes I focused to provide only the basic feature – to list the system block devices and their contents. As I have long and pleasing experience with writing shell scripts such as sysutils/beadm or sysutils/automount I though that writing lsblk.sh may be a good idea. I actually ‘open-sourced’ or should I say shared that project/idea in 2016 in this thread lsblk(8) Command for FreeBSD on FreeBSD Forums but lack of time really slowed that ‘side project’ development pace. I finally got back to it to finish it.

The lsblk.sh is generally small and simple shell script which tales less then 400 SLOC.

Here is example output of lsblk.sh command from my single disk laptop.

% lsblk.sh
DEVICE         MAJ:MIN  SIZE TYPE                      LABEL MOUNT
ada0             0:5b  932G GPT                           - -
  ada0p1         0:64  200M efi                    efiboot0 [UNMOUNTED]
  ada0p2         0:65  512K freebsd-boot           gptboot0 -
  [FREE]         -:-   492K -                             - -
  ada0p3         0:66  931G freebsd-zfs                zfs0 [ZFS]
  [FREE]         -:-   708K -                             - -

Same output in graphical window.

Below you will find an example lsblk.sh output from server with two system SSD drives (da0/da1) and two HDD data drives (da2/da3).

# lsblk.sh
DEVICE         MAJ:MIN SIZE TYPE                      LABEL MOUNT
da0              0:be  224G GPT                           - -
  da0p1          0:15a 200M efi                    efiboot0 [UNMOUNTED]
  da0p2          0:15b 512K freebsd-boot           gptboot0 -
  [FREE]         -:-   492K -                             - -
  da0p3          0:15c 2.0G freebsd-swap              swap0 [UNMOUNTED]
  da0p4          0:15d 221G freebsd-zfs                zfs0 [ZFS]
  [FREE]         -:-   580K -                             - -
da1              0:bf  224G GPT                           - -
  da1p1          0:16a 200M efi                    efiboot1 [UNMOUNTED]
  da1p2          0:16b 512K freebsd-boot           gptboot1 -
  [FREE]         -:-   492K -                             - -
  da1p3          0:16c 2.0G freebsd-swap              swap1 [UNMOUNTED]
  da1p4          0:16d 221G freebsd-zfs                zfs1 [ZFS]
  [FREE]         -:-   580K -                             - -
da2              0:c0   11T GPT                           - -
  da2p1          0:16e  11T freebsd-zfs                   - [ZFS]
  [FREE]         -:-   1.0G -                             - -
da3              0:c1   11T GPT                           - -
  da3p1          0:16f  11T freebsd-zfs                   - [ZFS]
  [FREE]         -:-   1.0G -                             - -

Below you will find other examples from other systems I have tested lsblk.sh on.

While lsblk.sh is not the fastest script on Earth (because of all the needed parsing) it does its job quite well. If you would like to install it in your system just type the command below:

# fetch -o /usr/local/bin/lsblk https://raw.githubusercontent.com/vermaden/scripts/master/lsblk.sh
# chmod +x /usr/local/bin/lsblk
# hash -r || rehash
# lsblk

If I got time which other original Linux lsblk(8) subcommand/option/argument is worth adding to the lsblk.sh script? 🙂

Regards.

UPDATE 1 – Added USAGE/HELP Information

Just added some usage information that can be displayed by specifying one of these as argument:

• h
• -h
• --h
• help
• -help
• --help

IMHO writing man page for such simple utility is needless. I think I will create dedicated man page when lsblk.sh tool will grow in size and options to comparable with the Linux lsblk(8) equivalent. Here is how it looks.

# lsblk.sh --help
usage:

  BASIC USAGE INFORMATION
  =======================
  # lsblk.sh [DISK]

example(s):

  LIST ALL BLOCK DEVICES IN SYSTEM
  --------------------------------
  # lsblk.sh
  DEVICE         MAJ:MIN SIZE TYPE                      LABEL MOUNT
  ada0             0:5b  932G GPT                           - -
    ada0p1         0:64  200M efi                    efiboot0 [UNMOUNTED]
    ada0p2         0:65  512K freebsd-boot           gptboot0 -
    [FREE]         -:-   492K -                             - -
    ada0p3         0:66  931G freebsd-zfs                zfs0 [ZFS]

  LIST ONLY da1 BLOCK DEVICE
  --------------------------
  # lsblk.sh da1
  DEVICE         MAJ:MIN SIZE TYPE                      LABEL MOUNT
  da1              0:80  2.0G MBR                           - -
    da1s1          0:80  2.0G freebsd                       - -
      da1s1a       0:81  1.0G freebsd-ufs                root /
      da1s1b       0:82  1.0G freebsd-swap               swap SWAP

hint(s):

  DISPLAY ALL DISKS IN SYSTEM
  ---------------------------
  # sysctl kern.disks
  kern.disks: ada0 da0 da1

Regards.

UPDATE 2 – Code Reorganization and 75% Rewrite

… at least this is what git(1) tries to tell me after commit message.

% git commit (...)
[master 12fd4aa] Rework entire flow. Split code into functions. Add many useful comments. In other words its 2.0 version.
 1 file changed, 494 insertions(+), 505 deletions(-)
 rewrite lsblk.sh (75%)

After several productive hours new incarnation of lsblk.sh is now available.

It has similar SLOC but its now smaller by a quarter … while doing more and with better accuracy. Great example why “less is more.”

% wc scripts/lsblk.sh.OLD
     491    2201   19721 scripts/lsblk.sh.OLD

% wc scripts/lsblk.sh
     494    1871   15472 scripts/lsblk.sh

Things that does not have simple solution are described below.

One of them is ‘double’ label for FAT filesystems. We have both /dev/gpt/efiboot0 label and FAT label is named EFISYS. We have to choose something here. As not all FAT filesystems have label I have chosen the GPT label.

% glabel status | grep ada0p1
  gpt/efiboot0     N/A  ada0p1
msdosfs/EFISYS     N/A  ada0p1

I was also not able to cover FUSE mounts. When you mount – for example – the /dev/da0 device as NTFS (with ntfs-3g) or exFAT (with mount.exfat) there is no visible difference in mount(8) output.

% mount -t fusefs
/dev/fuse on /mnt/ntfs (fusefs)
/dev/fuse on /mnt/exfat (fusefs)

When I mount such filesystem by my daemon (like sysutils/automount) I keep track of what device have been mounted to which directory in the /var/run/automount.state file. Then when I get the detach event for /dev/da0 device I know what to u(n)mount … but when I only have /dev/fuse device its just not possible.

… or maybe YOU know any way of extracting information from /dev/fuse (or generally from FUSE) what device is mounted where?

Now little presentation after update.

Here are various non ZFS filesystems mounted.

% mount -t nozfs
devfs on /dev (devfs, local, multilabel)
linprocfs on /compat/linux/proc (linprocfs, local)
tmpfs on /compat/linux/dev/shm (tmpfs, local)
/dev/label/ASD on /mnt/tmp (msdosfs, local)
/dev/fuse on /mnt/ntfs (fusefs)
/dev/md0s1f on /mnt/ufs.other (ufs, local)
/dev/gpt/OTHER on /mnt/fat.other (msdosfs, local)
/dev/md0s1a on /mnt/ufs (ufs, local)

… and here is how now lsblk.sh displays them.

% lsblk.sh
DEVICE         MAJ:MIN SIZE TYPE                      LABEL MOUNT
ada0             0:56  932G GPT                           - -
  ada0p1         0:64  200M efi                gpt/efiboot0 -
  ada0p2         0:65  512K freebsd-boot       gpt/gptboot0 -
  [FREE]         -:-   492K -                             - -
  ada0p3         0:66  931G freebsd-zfs                   - [ZFS]
  [FREE]         -:-   708K -                             - -
md0              0:28f 1.0G MBR                           - -
  md0s1          0:294 512M freebsd                       - -
    md0s1a       0:29a 100M freebsd-ufs                root /mnt/ufs
    md0s1b       0:29b  32M freebsd-swap         label/swap SWAP
    md0s1e       0:29c  64M freebsd-ufs                   - -
    md0s1f       0:29d 316M freebsd-ufs                   - /mnt/ufs.other
  md0s2          0:296 256M ntfs                          - -
  md0s3          0:297 256M fat32               msdosfs/ONE -
md1              0:2a4 1.0G msdosfs                   LARGE 
md2              0:298 2.0G GPT                           - -
  md2p1          0:29f 2.0G ms-basic-data         gpt/OTHER /mnt/fat.other

I used some file based memory devices for this. Now by default lsblk.sh also displays memory disks contents.

% mdconfig.sh -l
md0     vnode    1024M  /home/vermaden/FILE     
md2     vnode    2048M  /home/vermaden/FILE.GPT 
md1     vnode    1024M  /home/vermaden/FILER    

Here is how it looks in the xterm(1) terminal.

Regards.

UPDATE 3 – Added geli(8) Support

I thought that adding geli(8) support may be useful. The latest lsblk.sh now avoids code duplication for MOUNT and LABEL detection (moved into single unified function). Also added more comments for code readability and some minor fixes … and its again smaller 🙂

% wc lsblk.sh.1.0
     491    2201   19721 lsblk.sh.1.0

% wc lsblk.sh.2.0
     493    1861   15415 lsblk.sh.2.0

% wc lsblk.sh
     488    1820   15332 lsblk.sh

About 40% (according to git commit was changed this time (191 insertions and 196 deletions).

# git commit (...)
[master ec9985a] Add geli(8) support. Avoid code duplication and move MOUNT/LABEL detection into function. More comments. Minor fixes.
 1 file changed, 191 insertions(+), 196 deletions(-)

Also forgot to mention that now lsblk.sh thanks to smart optimizations (like not doing things twice and aggregating grep(1) | awk(1) pipes into single awk(1) queries) runs 3 times faster then the initial version 🙂

New output with geli(8) support below.

Regards.

UPDATE 4 – Added fuse(8) Support

As I wrote in the UPDATE 2 keeping track of what is mounted and where under fuse(8) is very hard as all mounted devices magically become /dev/fuse after mount is done.

After little research I found that this information (what really is mounted where by using fuse(8) interface under FreeBSD) is available after mounting procfs filesystem under /proc. You just need to cat cmdline entry for all PIDs of ntfs-3g. Its not perfect but the information at least is available.

# mount -t procfs proc /proc

# ps ax | grep ntfs-3g
45995  -  Is      0:00.00 ntfs-3g /dev/md1s2 /mnt/ntfs
59607  -  Is      0:00.00 ntfs-3g /dev/md3 /mnt/ntfs.another
83323  -  Is      0:00.00 ntfs-3g /dev/md3 /mnt/ntfs.another

# pgrep ntfs-3g
59607
83323
45995

% pgrep ntfs-3g | while read I; do cat /proc/$I/cmdline; echo; done
ntfs-3g/dev/md3/mnt/ntfs.another
ntfs-3g/dev/md3/mnt/ntfs.another
ntfs-3g/dev/md1s2/mnt/ntfs

This was the code prototype that worked for fuse(8) mountpoints detection.

    if [ -e /proc/0/status ]
    then
      FUSE_MOUNTS=$(
        while read PID
        do
          cat /proc/${PID}/cmdline
          echo
        done << ________EOF
          $( pgrep ntfs-3g )
________EOF
)
      FUSE_MOUNTS=$( echo "${FUSE_MOUNTS}" | sort -u )
      FUSE_MOUNTS=$( echo "${FUSE_MOUNTS}" | sed 's|ntfs-3g||g' )
      FUSE_CHECKS=$( echo "${FUSE_MOUNTS}" | grep /dev/${TARGET}/ )
      if [ "${FUSE_CHECKS}" != "" ]
      then
        MOUNT=$( echo "${FUSE_CHECKS}" | sed "s|/dev/${TARGET}||g" )
      fi
    fi
  fi

… and I have just realized that I found new (better) way of getting that information without mounting /proc filesystem – all you need to do is to display the ntfs-3g processes with their command line arguments, for example like that:

% ps -p $( pgrep ntfs-3g | tr '\n' ',' | sed '$s/.$//' ) -o command | sed 1d
ntfs-3g /dev/md1s2 /mnt/ntfs
ntfs-3g /dev/md3 /mnt/ntfs.another
ntfs-3g /dev/md3 /mnt/ntfs.another

So after I also thought that its only for NTFS (ntfs-3g(8) process) I also added exFAT support by searching for mount.exfat PIDs as well. The fuse(8) mount point detection works now for both NTFS and exFAT filesystems … and code to support it is even shorter.

  # TRY fuse(8) MOUNTS FROM PROCESSES
  if [ "${MOUNT_FOUND}" != "1" ]
  then
    FUSE_PIDS=$( pgrep mount.exfat ntfs-3g | tr '\n' ',' | sed '$s/.$//' )
    FUSE_MOUNTS=$( ps -p "${FUSE_PIDS}" -o command | sed 1d | sort -u )
    MOUNT=$( echo "${FUSE_MOUNTS}" |  grep "/dev/${TARGET} " | awk '{print $3}' )
  fi

I also changed how MAJOR and MINOR numbers are displayed – from HEX to DEC – as it is on Linux. The FreeBSD’s ls(1) from Base System displays these as HEX – for example you will get 0x2af value:

% ls -l /dev/md4
crw-rw----  1 root  operator  0x2af 2019.09.29 05:18 /dev/md4

But do the same with GNU equivalent by using gls(1) from FreeBSD Ports (from sysutils/coreutils package) and it shows MAJOR and MINOR in DEC values. The gls(1) is just ls(1) from the Linux world but as ls(1) name is already ‘taken’ by FreeBSD’s Base System tool the FreeBSD developers/maintainers add ‘g’ letter (for GNU) to distinguish them.

% gls -l /dev/md4
crw-rw---- 1 root 2, 175 2019-09-29 05:18 /dev/md4

… and they are also easier/faster to get with stat(1) tool.

  MAJ=$( stat -f "%Hr" /dev/${DEV} )
  MIN=$( stat -f "%Lr" /dev/${DEV} )

Latest lsblk.sh looks like that now.

… that is why I did not (yet) added lsblk.sh to the FreeBSD Ports. Several new versions with important features span across just two days 🙂

Regards.

UPDATE 5 – Another 69% Rewrite

After messing with gpart(8) more I found that using its -p flag which is a game changer. The difference is that with -p flag it displays names along partitions – its no longer needed to find the PREFIX and ‘create’ partition names.

Default gpart(8) output.

# gpart show md0
=>     63  2097089  md0  MBR  (1.0G)
       63  1048576    1  freebsd  (512M)
  1048639   524288    2  ntfs  (256M)
  1572927   524225    3  fat32  (256M)

Output of gpart(8) with -p flag.

# gpart show -p md0
=>     63  2097089    md0  MBR  (1.0G)
       63  1048576  md0s1  freebsd  (512M)
  1048639   524288  md0s2  ntfs  (256M)
  1572927   524225  md0s3  fat32  (256M)

That discovery implicated a quite large rewrite of lsblk.sh. The git commit estimates this as 69% code rewrite.

# git commit (...)
(...)
 1 file changed, 487 insertions(+), 501 deletions(-)
 rewrite lsblk.sh (69%)

The latest lsblk.sh has now these features:

• Previous bugs fixed.
• Detects exFAT labels.
• Is now 20% faster.
• Has less 10% SLOC.
• Has less 15% of code.
• Handles bsdlabel(8) on entire device properly.
• Handles exFAT on entire device properly.

The difference in code is shown below.

# wc lsblk.sh
     487    1791   13705 lsblk.sh

# wc lsblk.sh.OLD
     544    1931   16170 lsblk.sh.OLD

Latest lsblk.sh looks as usual but I now use ‘-‘ instead of ‘[UNMOUNTED]‘ one.

UPDATE 6 – New Updated and Fixed Version

The lsblk.sh has been updated to 3.4 version – also already updated in the FreeBSD Ports tree – in the sysutils/lsblk port.

The Changelog for this version is below:

• Add sysctl -n kern.disks to disk listing.
• Reset LABEL in __gpart_present function.
• Fix behavior with [bootme] and [bootonce] flags from gpart(8).
• Disable GPTID for label display.
• Add -d|–disks option to list entire disks only.

Keep in mind that lsblk.sh uses diskinfo(8) which – to function properly – need you to be in the operator group. You can add yourself to that group like that:

# pw groupmod operator -m yourself

… or by editing the /etc/group file.

Here is example output.

UPDATE 7 – More Fixes

The lsblk.sh has been updated to 3.5 version – also already updated in the FreeBSD Ports tree – in the sysutils/lsblk port.

The Changelog for this version is below:

• Remove control sequences and colors from output when listing disks.
• Use diskinfo(8) only for md(4) disks as geom(4) does not support them.
• Add new comments and rework some of the older ones.
• Add additional checks for SIZE gathering and printing.
• Properly print exFAT filesystem label when on entire device without partitions.
• Fix display of NTFS-3G mountpoints.
• Check automount(8) /var/run/automount.state for fusefs(5) filesystems.

Presence in the operator group is only needed for size for md(4)disks. All other disks size is gathered using geom(8)command now.

EOF

SMB/CIFS on FreeBSD

If you use FreeBSD/Illumos/Linux (or other UNIX/Unix-like system) there is big chance that you do not like – to say the least – the Windows world, but sometimes there is need to share some files with the Windows world. This is where Samba project comes handy. Today I would like to share minimalistic and simple Samba configuration and also a way to access SMB/CIFS shares from a FreeBSD machine.

On the naming side CIFS (Common Internet File System) is just particular version/dialect of the SMB (Server Message Block) protocol.

As usual I will use FreeBSD as a server. For the setup I used FreeBSD 12.0-RELEASE virtual machine image available from the project location:

• https://download.freebsd.org/ftp/releases/VM-IMAGES/12.0-RELEASE/

There are several formats available – qcow2/raw/vhd/vmdk – but as I will be using VirtualBox I used the VMDK one.

The main FreeBSD configuration file on the server can be as small and simple as the one bellow.

# cat /etc/rc.conf
hostname="samba"
ifconfig_em0="inet 10.0.10.40/24"
defaultrouter="10.0.10.1"
sshd_enable="YES"

You of course do not need SSH to server SMB/CIFS shares with Samba.

Serve SMB/CIFS Share on FreeBSD with Samba

There are several versions of Samba available on FreeBSD, but if you do not have exact reason to use the older version then just go ahead with the latest one.

# pkg search samba
p5-Samba-LDAP-0.05_2           Manage a Samba PDC with an LDAP Backend
p5-Samba-SIDhelper-0.0.0_3     Create SIDs based on G/UIDs
samba-nsupdate-9.13.3_1        nsupdate utility with GSS-TSIG support
samba46-4.6.16_1               Free SMB/CIFS and AD/DC server and client for Unix
samba47-4.7.12                 Free SMB/CIFS and AD/DC server and client for Unix
samba48-4.8.7                  Free SMB/CIFS and AD/DC server and client for Unix

First You will need to add Samba package.

# pkg install samba48

Then we need to create configuration file for Samba. I will assume here that you would like to share two things as examples. The /data directory with write permissions only to my vermaden user and also my home directory /home/vermaden with read permissions for me and all people on my vermaden group. The so called public read is disabled entirely. Only after passing user and password the access will be possible to these shares. I also added several performance related options. Below is the /usr/local/etc/smb4.conf configuration file.

# cat /usr/local/etc/smb4.conf
[global]
workgroup          = workgroup
netbios name       = smb
server string      = samba
security           = user
max smbd processes = 3
encrypt passwords  = yes
socket options     = TCP_NODELAY IPTOS_LOWDELAY IPTOS_THROUGHPUT SO_KEEPALIVE SO_RCVBUF=65536 SO_SNDBUF=65536
aio read size      = 16384
aio write size     = 16384
strict locking     = no
strict sync        = no

# DISABLE PRINTING
load printers           = no
disable spoolss         = yes
show add printer wizard = no

[data]
  path       = /data
  public     = no
  writable   = yes
  browsable  = no
  write list = vermaden

[vermaden]
  path       = /home/vermaden
  public     = no
  writable   = no
  browsable  = no
  write list = @vermaden

We will also need vermaden user, let’s create one with pw(8) command.

First the vermaden group with GID of 1000. The -N flag just shows what will be done instead of doing actual changes to the system. Let’s try that and then execute the command without the -N flag to actually add the group.

# pw groupadd -n vermaden -g 1000 -N
vermaden:*:1000:
# pw groupadd -n vermaden -g 1000
# pw groupshow vermaden
vermaden:*:1000:

As we have the group its time to create vermaden user with UID of 1000. Like with group let’s first try with -N flag to check what will be made.

# pw useradd -n vermaden -c '' -u 1000 -g 1000 -m -N
vermaden:*:1000:1000::0:0::/home/vermaden:/bin/sh
# pw useradd -n vermaden -c '' -u 1000 -g 1000 -m
# pw usershow vermaden
vermaden:*:1000:1000::0:0::/home/vermaden:/bin/sh

Let’s verify our vermaden user again.

# id vermaden
uid=1000(vermaden) gid=1000(vermaden) groups=1000(vermaden)

# su - vermaden
By pressing "Scroll Lock" you can use the arrow keys to scroll backward
through the console output.  Press "Scroll Lock" again to turn it off.
Don't have a "Scroll Lock" key? The "Pause / Break" key acts alike.

Now let’s create password for this new vermaden user.

# passwd vermaden
Changing local password for vermaden
New Password:
Retype New Password:

Now we need to add the vermaden user with pdbedit command from the Samba package.

# which pdbedit
/usr/local/bin/pdbedit

# pkg which `which pdbedit`
/usr/local/bin/pdbedit was installed by package samba48-4.8.7

# pdbedit -a -u vermaden
new password:
retype new password:
Unix username:        vermaden
NT username:
Account Flags:        [U          ]
User SID:             S-1-5-21-1751207453-560213463-1759912891-1000
Primary Group SID:    S-1-5-21-1751207453-560213463-1759912891-513
Full Name:
Home Directory:       \\smb\vermaden
HomeDir Drive:
Logon Script:
Profile Path:         \\smb\vermaden\profile
Domain:               SMB
Account desc:
Workstations:
Munged dial:
Logon time:           0
Logoff time:          9223372036854775807 seconds since the Epoch
Kickoff time:         9223372036854775807 seconds since the Epoch
Password last set:    Fri, 21 Dec 2018 16:49:29 UTC
Password can change:  Fri, 21 Dec 2018 16:49:29 UTC
Password must change: never
Last bad password   : 0
Bad password count  : 0
Logon hours         : FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF

To list all users with the pdbedit command use the -L argument.

# pdbedit -L
vermaden:1000:

We now need to add Samba to the FreeBSD system services automatic startup.

# sysrc samba_server_enable=YES
samba_server_enable:  -> YES

# sysrc samba_server_enable
samba_server_enable: YES

# cat /etc/rc.conf
hostname="samba"
ifconfig_em0="inet 10.0.10.40/24"
defaultrouter="10.0.10.1"
sshd_enable="YES"
samba_server_enable="YES"

Now we can start the Samba service.

# service samba_server start
Performing sanity check on Samba configuration: OK
Starting nmbd.
Starting smbd.

Let’s check which Samba daemons listen on which ports.

# sockstat -l -4
USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS
root     smbd       599   33 tcp4   *:445                 *:*
root     smbd       599   34 tcp4   *:139                 *:*
root     nmbd       595   15 udp4   *:137                 *:*
root     nmbd       595   16 udp4   *:138                 *:*
(...)

Now let’s try to access the /data share from the Windows system.

Open explorer.exe on Windows machine and type //smb/data into location field and then type smb\vermaden as username.

You should be able to access the share now as shown below.

Let’s put some text into that test.txt file.

Let’s verify that it works on the FreeBSD side.

# cat /data/test.txt
Input from Windows.

So we are able to access/modify files from FreeBSD machine on the Windows world.

Access SMB/CIFS Share from FreeBSD

Let’s try the other way around.

By default there are several shares already served on Windows.

C:\>net share

Share name   Resource                        Remark

-------------------------------------------------------------------------------
C$           C:\                             Default share
IPC$                                         Remote IPC
ADMIN$       C:\Windows                      Remote Admin
Users        C:\Users
The command completed successfully.


C:\>

You can share a directory from Windows by using graphical interface as shown below.

… or by using CLI interface within cmd.exe interpreter with net commands.

The test share is now exported for vuser user with FULL access rights which means read/write in the Windows world.

Here are the same commands in text so you may copy/paste them as needed.

C:\Windows\system32>cd \

C:\>mkdir asd

C:\>net share test=C:\asd /grant:vuser,FULL
test was shared successfully.


C:\>net share

Share name   Resource                        Remark

-------------------------------------------------------------------------------
C$           C:\                             Default share
IPC$                                         Remote IPC
ADMIN$       C:\Windows                      Remote Admin
test         C:\asd
Users        C:\Users
The command completed successfully.


C:\>

Let’s try to mount it using the mount_smbfs command on FreeBSD system. The 10.0.10.4 address is the IP of the Windows machine.

# mount_smbfs -I 10.0.10.4 //vuser@vbox/test /mnt
Password:
#

# mount
/dev/gpt/rootfs on / (ufs, local, soft-updates)
devfs on /dev (devfs, local, multilabel)
//VUSER@VBOX/TEST on /mnt (smbfs)

It also works the other way.

After your job is done you may remove the test share also with net command as shown below.

… and also the same commands in text so you may copy/paste them as needed.

C:\>net share test /delete
test was deleted successfully.


C:\>net share

Share name   Resource                        Remark

-------------------------------------------------------------------------------
C$           C:\                             Default share
IPC$                                         Remote IPC
ADMIN$       C:\Windows                      Remote Admin
Users        C:\Users
The command completed successfully.


C:\>

This sentence concludes this article 😉

UPDATE 1

The SMB/CIFS on FreeBSD article was featured in the BSD Now 279 – Future of ZFS episode.

Thanks for mentioning!

EOF