FreeBSD Jails is a great piece of container technology pioneered several years before Solaris Zones. Not to mention 15 years before Docker was born. Today they still work great and offer some new features like entire network stack for each Jail called VNET. Unfortunately they also have downsides. For example anything related to NFS is broken inside FreeBSD Jails (especially when they are VNET based Jails) and the relevant 251347 Bug Report remains unsolved.
There is however a way to run NFS server inside VNET based FreeBSD Jail – we will use userspace NFS server implementation instead of using the FreeBSD base system kernel space NFS server. Its available as net/unfs3 package and this is exactly what we will gonna use for this guide.
Same in plain text below.
/ % cd /usr/ports/net/unfs3 /usr/ports/net/unfs3 % cat pkg-descr UNFS3 is a user-space implementation of the NFSv3 server specification. It provides a daemon for the MOUNT and NFS protocols, which are used by NFS clients for accessing files on the server. Since it runs in user-space, you can use it in a jail. WWW: https://unfs3.github.io/ /usr/ports/net/unfs3 % pkg info -l unfs3 unfs3-0.9.22_2: /usr/local/man/man7/tags.7.gz /usr/local/man/man8/unfsd.8.gz /usr/local/sbin/unfsd /usr/local/share/licenses/unfs3-0.9.22_2/BSD3CLAUSE /usr/local/share/licenses/unfs3-0.9.22_2/LICENSE /usr/local/share/licenses/unfs3-0.9.22_2/catalog.mk
Its also pity that VNET feature for FreeBSD Jails is not well documented. Search the FreeBSD Handbook or FreeBSD FAQ for the VNET or VIMAGE keywords. Not a single match. There are only man pages and some stuff left in the /usr/share/examples/jails dir. There is also FreeBSD Mastery: Jails book by Michael W. Lucas but its 3 years old already.
Setup
Below you will find the list of systems we will use in this guide.
10.0.10.250 host 10.0.10.251 nfs_server
The host is a common FreeBSD server installed on a physical or virtual machine. We will also use it as out NFS client and mount the NFS share there. The nfs_server is a FreeBSD Jail with VNET separate network stack enabled. We will run NFS server from this host nfs_server system. Both of them run latest FreeBSD 13.1-RELEASE but I suspect that it should also work the same on older versions.
FreeBSD Host and NFS Client (host)
First we will setup the host machine. Its typical default ZFS FreeBSD install – nothing special about that. To use the VNET enabled Jails we will use jib tool from the /usr/share/examples/jails directory as we will need it to automate epair(4) interfaces management.
root@host:/ # install -o root -g wheel -m 0555 /usr/share/examples/jails/jib /usr/sbin/jib
Our next step would be to fetch and setup the nfs_server FreeBSD Jail. We will not waste time in compilation – we will fetch the base.txz directly from FreeBSD page.
root@host:/ # mkdir -p /jail/BASE root@host:/jail/BASE # cd /jail/BASE root@host:/jail/BASE # fetch http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/13.1-RELEASE/base.txz root@host:/jail/BASE # mv base.txz 13.1-base.txz
Now the nfs_server FreeBSD Jail.
root@host:/ # mkdir -p /jail/nfs_server root@host:/jail/nfs_server # cd /jail/nfs_server root@host:/jail/nfs_server # tar -xzf /jail/BASE/13.1-base.txz --unlink
The main FreeBSD /etc/rc.conf configuration file does not hold any special setting – pretty usual stuff.
root@host:/ # cat /etc/rc.conf
# NETWORK
hostname="host"
ifconfig_em0="inet 10.0.10.250/24 up"
defaultrouter="10.0.10.1"
gateway_enable="YES"
# DAEMONS
dumpdev="AUTO"
sshd_enable="YES"
zfs_enable="YES"
sendmail_enable="NO"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"
update_motd="NO"
# JAILS
jail_enable="YES"
jail_parallel_start="YES"
jail_list="nfs_server"
The nfs_server FreeBSD Jail as configured in the /etc/jail.conf config file.
root@host:/ # cat /etc/jail.conf
nfs_server {
path = "/jail/${name}";
host.hostname = "${name}";
allow.raw_sockets = 1;
allow.set_hostname = 1;
allow.sysvipc = 1;
mount.devfs;
exec.clean;
vnet;
vnet.interface = "e0b_${name}";
exec.prestart += "/usr/sbin/jib addm -b _bridge0 ${name} em0";
exec.poststop += "/usr/sbin/jib destroy ${name}";
exec.start += "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.consolelog = "/var/log/jail_${name}_console.log";
}
… and last but not least lets make sure the following DNS haiku will not bother us π
Setup the /etc/hosts on the host system.
root@host:/ # tail -3 /etc/hosts
10.0.10.250 host
10.0.10.251 nfs_server
FreeBSD NFS Server VNET Jail (nfs_server)
As our FreeBSD Jail is installed we will now start it and configure it.
root@host:/ # service jail onestart nfs_server root@host:/ # jls JID IP Address Hostname Path 1 nfs_server /jail/nfs_server root@host:/ # jexec 1 root@nfs_server:/ #
First we will install latest net/unfs3 package – this userspace NFS server is also very minimal and does not have any dependencies.
root@nfs_server:/ # echo nameserver 1.1.1.1 > /etc/resolv.conf root@nfs_server:/ # sed -i '' s/quarterly/latest/g /etc/pkg/FreeBSD.conf root@nfs_server:/ # pkg install unfs3 root@nfs_server:/ # pkg info -qoa ports-mgmt/pkg net/unfs3
Now we will configure our NFS share under /share dir and start the unfsd(8) userspace NFS server.
root@nfs_server:/ # mkdir /share root@nfs_server:/ # cat /etc/exports /share 10.0.10.250(rw,no_root_squash,no_all_squash)
… last but not least – DNS π
root@nfs_server:/ # tail -3 /etc/hosts
10.0.10.250 host
10.0.10.251 nfs_server
As we are using VNET network stack in a FreeBSD Jail we will have to address the network interface in the Jails /etc/rc.conf file. The unfsd(8) daemon does not start without rpcbind service so we will also enable it.
root@nfs_server:/ # cat /etc/rc.conf
# NETWORK
hostname="nfs_server"
ifconfig_e0b_nfs_server="10.0.10.251/24 up"
defaultrouter="10.0.10.1"
# DAEMONS
sshd_enable="YES"
rpcbind_enable="YES"
sendmail_enable="NO"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"
We will make unfsd(8) start automatically at Jails start with plain old /etc/rc.local file.
root@nfs_server:/ # cat /etc/rc.local
/usr/local/sbin/unfsd &
We will not restart our FreeBSD Jail to make these changes take effect.
root@host:/ # service jail onerestart nfs_server root@host:/ # jls JID IP Address Hostname Path 2 nfs_server /jail/nfs_server root@host:/ # jexec 2 root@nfs_server:/ #
After startup we can see that unfsd(8) is listening on a NFS 2049 port.
root@nfs_server:/ # sockstat -l4 USER COMMAND PID FD PROTO LOCAL ADDRESS FOREIGN ADDRESS root sshd 1261 4 tcp4 *:22 *:* root sendmail 1241 5 tcp4 127.0.0.1:25 *:* root unfsd 1223 3 udp4 *:2049 *:* root unfsd 1223 4 tcp4 *:2049 *:* root rpcbind 1196 9 udp4 *:111 *:* root rpcbind 1196 10 udp4 *:842 *:* root rpcbind 1196 11 tcp4 *:111 *:* root syslogd 1188 6 udp4 *:514 *:*
We should have our epair(4) interface called e0b_nfs_server addressed properly.
root@nfs_server:/ # ifconfig lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> e0b_nfs_server: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=8<VLAN_MTU> ether 0e:27:dd:b3:81:88 hwaddr 02:30:0d:9f:57:0b inet 10.0.10.251 netmask 0xffffff00 broadcast 10.0.10.255 groups: epair media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>) status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
Mount /share on NFS Client
I added that NFS entry to the /etc/fstab file on the host machine.
root@host:~ # cat /etc/fstab #DEV #MNT #TYPE #OPT #DUMP/PASS /dev/ada0p1 /boot/efi msdosfs rw 2 2 /dev/ada0p3 none swap sw 0 0 #DEV #MNT #TYPE #OPT #DUMP/PASS 10.0.10.251:/share /mnt nfs rw,noauto 0 0
We will now attempt to mount the /share NFS export on the host machine.
root@host:/ # mount /mnt root@host:/ # mount | grep share 10.0.10.251:/share on /mnt (nfs) root@host:/ # cd /mnt root@host:/mnt # :> FILE root@host:/mnt # ls -l FILE -rw-r--r-- 1 root root 0 2022-05-21 22:53 FILE root@host:/mnt # rm FILE
Seems to work properly.
Here are also network interfaces on the host machine.
root@host:/ # ifconfig em0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=4810099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER,NOMAP> ether 08:00:27:b3:81:88 inet 10.0.10.250 netmask 0xffffff00 broadcast 10.0.10.255 media: Ethernet autoselect (1000baseT <full-duplex>) status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> em0_bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 ether 58:9c:fc:10:ff:dd id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: e0a_nfs_server flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 4 priority 128 path cost 2000 member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 1 priority 128 path cost 20000 groups: bridge nd6 options=9<PERFORMNUD,IFDISABLED> e0a_nfs_server: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=8<VLAN_MTU> ether 02:27:dd:b3:81:88 hwaddr 02:30:0d:9f:57:0a groups: epair media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>) status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
Future of NFS Server in Jails
This setup – while allowing to run the NFS server inside FreeBSD Jail with even VNET enabled has its drawbacks unfortunately. First is that it is run in userspace instead of kernel space – which means its slower. Second is that the unfsd(8) only implements NFS version 3 – so no version 4 is not possible.
Where we can go from here? Like with WiFi stuff IMHO the FreeBSD Foundation could step in to sponsor the missing bits of NFS server and VNET to make these native tools work they should. Its up to you to put the pressure on the FreeBSD Foundation when they as what are you missing from the FreeBSD UNIX system that could be improved with one of their projects. You may also join the discussion at the 251347 Bug Report of course.
I think its a big loss that native kernel space NFS server is not currently possible with VNET FreeBSD Jails.
EOF
ππππππππ 