NFS Server Inside FreeBSD VNET Jail

FreeBSD Jails is a great piece of container technology pioneered several years before Solaris Zones. Not to mention 15 years before Docker was born. Today they still work great and offer some new features like entire network stack for each Jail called VNET. Unfortunately they also have downsides. For example anything related to NFS is broken inside FreeBSD Jails (especially when they are VNET based Jails) and the relevant 251347 Bug Report remains unsolved.

There is however a way to run NFS server inside VNET based FreeBSD Jail – we will use userspace NFS server implementation instead of using the FreeBSD base system kernel space NFS server. Its available as net/unfs3 package and this is exactly what we will gonna use for this guide.

Same in plain text below.

/ % cd /usr/ports/net/unfs3

/usr/ports/net/unfs3 % cat pkg-descr
UNFS3 is a user-space implementation of the NFSv3 server specification. It
provides a daemon for the MOUNT and NFS protocols, which are used by NFS
clients for accessing files on the server.
Since it runs in user-space, you can use it in a jail.

WWW: https://unfs3.github.io/

/usr/ports/net/unfs3 % pkg info -l unfs3           
unfs3-0.9.22_2:
        /usr/local/man/man7/tags.7.gz
        /usr/local/man/man8/unfsd.8.gz
        /usr/local/sbin/unfsd
        /usr/local/share/licenses/unfs3-0.9.22_2/BSD3CLAUSE
        /usr/local/share/licenses/unfs3-0.9.22_2/LICENSE
        /usr/local/share/licenses/unfs3-0.9.22_2/catalog.mk

Its also pity that VNET feature for FreeBSD Jails is not well documented. Search the FreeBSD Handbook or FreeBSD FAQ for the VNET or VIMAGE keywords. Not a single match. There are only man pages and some stuff left in the /usr/share/examples/jails dir. There is also FreeBSD Mastery: Jails book by Michael W. Lucas but its 3 years old already.

Setup

Below you will find the list of systems we will use in this guide.

10.0.10.250  host
10.0.10.251  nfs_server

The host is a common FreeBSD server installed on a physical or virtual machine. We will also use it as out NFS client and mount the NFS share there. The nfs_server is a FreeBSD Jail with VNET separate network stack enabled. We will run NFS server from this host nfs_server system. Both of them run latest FreeBSD 13.1-RELEASE but I suspect that it should also work the same on older versions.

FreeBSD Host and NFS Client (host)

First we will setup the host machine. Its typical default ZFS FreeBSD install – nothing special about that. To use the VNET enabled Jails we will use jib tool from the /usr/share/examples/jails directory as we will need it to automate epair(4) interfaces management.

root@host:/ # install -o root -g wheel -m 0555 /usr/share/examples/jails/jib /usr/sbin/jib

Our next step would be to fetch and setup the nfs_server FreeBSD Jail. We will not waste time in compilation – we will fetch the base.txz directly from FreeBSD page.

root@host:/ # mkdir -p /jail/BASE
root@host:/jail/BASE # cd /jail/BASE
root@host:/jail/BASE # fetch http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/13.1-RELEASE/base.txz
root@host:/jail/BASE # mv base.txz 13.1-base.txz

Now the nfs_server FreeBSD Jail.

root@host:/ # mkdir -p /jail/nfs_server
root@host:/jail/nfs_server # cd /jail/nfs_server
root@host:/jail/nfs_server # tar -xzf /jail/BASE/13.1-base.txz --unlink

The main FreeBSD /etc/rc.conf configuration file does not hold any special setting – pretty usual stuff.

root@host:/ # cat /etc/rc.conf
# NETWORK
  hostname="host"
  ifconfig_em0="inet 10.0.10.250/24 up"
  defaultrouter="10.0.10.1"
  gateway_enable="YES"

# DAEMONS
  dumpdev="AUTO"
  sshd_enable="YES"
  zfs_enable="YES"
  sendmail_enable="NO"
  sendmail_submit_enable="NO"
  sendmail_outbound_enable="NO"
  sendmail_msp_queue_enable="NO"
  update_motd="NO"

# JAILS
  jail_enable="YES"
  jail_parallel_start="YES"
  jail_list="nfs_server"

The nfs_server FreeBSD Jail as configured in the /etc/jail.conf config file.

root@host:/ # cat /etc/jail.conf
nfs_server {
  path = "/jail/${name}";
  host.hostname = "${name}";
  allow.raw_sockets = 1;
  allow.set_hostname = 1;
  allow.sysvipc = 1;
  mount.devfs;
  exec.clean;
  vnet;
  vnet.interface = "e0b_${name}";
  exec.prestart += "/usr/sbin/jib addm -b _bridge0 ${name} em0";
  exec.poststop += "/usr/sbin/jib destroy ${name}";
  exec.start += "/bin/sh /etc/rc";
  exec.stop = "/bin/sh /etc/rc.shutdown";
  exec.consolelog = "/var/log/jail_${name}_console.log";
}

… and last but not least lets make sure the following DNS haiku will not bother us 🙂

Setup the /etc/hosts on the host system.

root@host:/ # tail -3 /etc/hosts
10.0.10.250 host
10.0.10.251 nfs_server

FreeBSD NFS Server VNET Jail (nfs_server)

As our FreeBSD Jail is installed we will now start it and configure it.

root@host:/ # service jail onestart nfs_server

root@host:/ # jls
   JID  IP Address      Hostname                      Path
     1                  nfs_server                    /jail/nfs_server

root@host:/ # jexec 1

root@nfs_server:/ # 

First we will install latest net/unfs3 package – this userspace NFS server is also very minimal and does not have any dependencies.

root@nfs_server:/ # echo nameserver 1.1.1.1 > /etc/resolv.conf

root@nfs_server:/ # sed -i '' s/quarterly/latest/g /etc/pkg/FreeBSD.conf

root@nfs_server:/ # pkg install unfs3

root@nfs_server:/ # pkg info -qoa
ports-mgmt/pkg
net/unfs3

Now we will configure our NFS share under /share dir and start the unfsd(8) userspace NFS server.

root@nfs_server:/ # mkdir /share

root@nfs_server:/ # cat /etc/exports
/share  10.0.10.250(rw,no_root_squash,no_all_squash)

… last but not least – DNS 🙂

root@nfs_server:/ # tail -3 /etc/hosts
10.0.10.250 host
10.0.10.251 nfs_server

As we are using VNET network stack in a FreeBSD Jail we will have to address the network interface in the Jails /etc/rc.conf file. The unfsd(8) daemon does not start without rpcbind service so we will also enable it.

root@nfs_server:/ # cat /etc/rc.conf
# NETWORK
  hostname="nfs_server"
  ifconfig_e0b_nfs_server="10.0.10.251/24 up"
  defaultrouter="10.0.10.1"

# DAEMONS
  sshd_enable="YES"
  rpcbind_enable="YES"
  sendmail_enable="NO"
  sendmail_submit_enable="NO"
  sendmail_outbound_enable="NO"
  sendmail_msp_queue_enable="NO"

We will make unfsd(8) start automatically at Jails start with plain old /etc/rc.local file.

root@nfs_server:/ # cat /etc/rc.local 
/usr/local/sbin/unfsd &

We will not restart our FreeBSD Jail to make these changes take effect.

root@host:/ # service jail onerestart nfs_server

root@host:/ # jls
   JID  IP Address      Hostname                      Path
     2                  nfs_server                    /jail/nfs_server

root@host:/ # jexec 2

root@nfs_server:/ # 

After startup we can see that unfsd(8) is listening on a NFS 2049 port.

root@nfs_server:/ # sockstat -l4
USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS      
root     sshd       1261  4  tcp4   *:22                  *:*
root     sendmail   1241  5  tcp4   127.0.0.1:25          *:*
root     unfsd      1223  3  udp4   *:2049                *:*
root     unfsd      1223  4  tcp4   *:2049                *:*
root     rpcbind    1196  9  udp4   *:111                 *:*
root     rpcbind    1196  10 udp4   *:842                 *:*
root     rpcbind    1196  11 tcp4   *:111                 *:*
root     syslogd    1188  6  udp4   *:514                 *:*
  

We should have our epair(4) interface called e0b_nfs_server addressed properly.

root@nfs_server:/ # ifconfig 
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
e0b_nfs_server: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 0e:27:dd:b3:81:88
        hwaddr 02:30:0d:9f:57:0b
        inet 10.0.10.251 netmask 0xffffff00 broadcast 10.0.10.255
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

Mount /share on NFS Client

I added that NFS entry to the /etc/fstab file on the host machine.

root@host:~ # cat /etc/fstab 
#DEV         #MNT       #TYPE    #OPT  #DUMP/PASS
/dev/ada0p1  /boot/efi  msdosfs  rw    2 2
/dev/ada0p3  none       swap     sw    0 0

#DEV                #MNT  #TYPE  #OPT       #DUMP/PASS
10.0.10.251:/share  /mnt  nfs    rw,noauto  0 0

We will now attempt to mount the /share NFS export on the host machine.

root@host:/ # mount /mnt

root@host:/ # mount | grep share
10.0.10.251:/share on /mnt (nfs)

root@host:/ # cd /mnt

root@host:/mnt # :> FILE

root@host:/mnt # ls -l FILE
-rw-r--r-- 1 root root 0 2022-05-21 22:53 FILE

root@host:/mnt # rm FILE

Seems to work properly.

Here are also network interfaces on the host machine.

root@host:/ # ifconfig 
em0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4810099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER,NOMAP>
        ether 08:00:27:b3:81:88
        inet 10.0.10.250 netmask 0xffffff00 broadcast 10.0.10.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
em0_bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 58:9c:fc:10:ff:dd
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: e0a_nfs_server flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 4 priority 128 path cost 2000
        member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 20000
        groups: bridge
        nd6 options=9<PERFORMNUD,IFDISABLED>
e0a_nfs_server: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 02:27:dd:b3:81:88
        hwaddr 02:30:0d:9f:57:0a
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

Future of NFS Server in Jails

This setup – while allowing to run the NFS server inside FreeBSD Jail with even VNET enabled has its drawbacks unfortunately. First is that it is run in userspace instead of kernel space – which means its slower. Second is that the unfsd(8) only implements NFS version 3 – so no version 4 is not possible.

Where we can go from here? Like with WiFi stuff IMHO the FreeBSD Foundation could step in to sponsor the missing bits of NFS server and VNET to make these native tools work they should. Its up to you to put the pressure on the FreeBSD Foundation when they as what are you missing from the FreeBSD UNIX system that could be improved with one of their projects. You may also join the discussion at the 251347 Bug Report of course.

I think its a big loss that native kernel space NFS server is not currently possible with VNET FreeBSD Jails.

EOF