NFS Server Inside FreeBSD VNET Jail

FreeBSD Jails is a great piece of container technology pioneered several years before Solaris Zones. Not to mention 15 years before Docker was born. Today they still work great and offer some new features like entire network stack for each Jail called VNET. Unfortunately they also have downsides. For example anything related to NFS is broken inside FreeBSD Jails (especially when they are VNET based Jails) and the relevant 251347 Bug Report remains unsolved.

There is however a way to run NFS server inside VNET based FreeBSD Jail – we will use userspace NFS server implementation instead of using the FreeBSD base system kernel space NFS server. Its available as net/unfs3 package and this is exactly what we will gonna use for this guide.


Same in plain text below.

/ % cd /usr/ports/net/unfs3

/usr/ports/net/unfs3 % cat pkg-descr
UNFS3 is a user-space implementation of the NFSv3 server specification. It
provides a daemon for the MOUNT and NFS protocols, which are used by NFS
clients for accessing files on the server.
Since it runs in user-space, you can use it in a jail.


/usr/ports/net/unfs3 % pkg info -l unfs3           

Its also pity that VNET feature for FreeBSD Jails is not well documented. Search the FreeBSD Handbook or FreeBSD FAQ for the VNET or VIMAGE keywords. Not a single match. There are only man pages and some stuff left in the /usr/share/examples/jails dir. There is also FreeBSD Mastery: Jails book by Michael W. Lucas but its 3 years old already.


Below you will find the list of systems we will use in this guide.  host  nfs_server

The host is a common FreeBSD server installed on a physical or virtual machine. We will also use it as out NFS client and mount the NFS share there. The nfs_server is a FreeBSD Jail with VNET separate network stack enabled. We will run NFS server from this host nfs_server system. Both of them run latest FreeBSD 13.1-RELEASE but I suspect that it should also work the same on older versions.

FreeBSD Host and NFS Client (host)

First we will setup the host machine. Its typical default ZFS FreeBSD install – nothing special about that. To use the VNET enabled Jails we will use jib tool from the /usr/share/examples/jails directory as we will need it to automate epair(4) interfaces management.

root@host:/ # install -o root -g wheel -m 0555 /usr/share/examples/jails/jib /usr/sbin/jib

Our next step would be to fetch and setup the nfs_server FreeBSD Jail. We will not waste time in compilation – we will fetch the base.txz directly from FreeBSD page.

root@host:/ # mkdir -p /jail/BASE
root@host:/jail/BASE # cd /jail/BASE
root@host:/jail/BASE # fetch
root@host:/jail/BASE # mv base.txz 13.1-base.txz

Now the nfs_server FreeBSD Jail.

root@host:/ # mkdir -p /jail/nfs_server
root@host:/jail/nfs_server # cd /jail/nfs_server
root@host:/jail/nfs_server # tar -xzf /jail/BASE/13.1-base.txz --unlink

The main FreeBSD /etc/rc.conf configuration file does not hold any special setting – pretty usual stuff.

root@host:/ # cat /etc/rc.conf
  ifconfig_em0="inet up"



The nfs_server FreeBSD Jail as configured in the /etc/jail.conf config file.

root@host:/ # cat /etc/jail.conf
nfs_server {
  path = "/jail/${name}";
  host.hostname = "${name}";
  allow.raw_sockets = 1;
  allow.set_hostname = 1;
  allow.sysvipc = 1;
  vnet.interface = "e0b_${name}";
  exec.prestart += "/usr/sbin/jib addm -b _bridge0 ${name} em0";
  exec.poststop += "/usr/sbin/jib destroy ${name}";
  exec.start += "/bin/sh /etc/rc";
  exec.stop = "/bin/sh /etc/rc.shutdown";
  exec.consolelog = "/var/log/jail_${name}_console.log";

… and last but not least lets make sure the following DNS haiku will not bother us πŸ™‚


Setup the /etc/hosts on the host system.

root@host:/ # tail -3 /etc/hosts host nfs_server

FreeBSD NFS Server VNET Jail (nfs_server)

As our FreeBSD Jail is installed we will now start it and configure it.

root@host:/ # service jail onestart nfs_server

root@host:/ # jls
   JID  IP Address      Hostname                      Path
     1                  nfs_server                    /jail/nfs_server

root@host:/ # jexec 1

root@nfs_server:/ # 

First we will install latest net/unfs3 package – this userspace NFS server is also very minimal and does not have any dependencies.

root@nfs_server:/ # echo nameserver > /etc/resolv.conf

root@nfs_server:/ # sed -i '' s/quarterly/latest/g /etc/pkg/FreeBSD.conf

root@nfs_server:/ # pkg install unfs3

root@nfs_server:/ # pkg info -qoa

Now we will configure our NFS share under /share dir and start the unfsd(8) userspace NFS server.

root@nfs_server:/ # mkdir /share

root@nfs_server:/ # cat /etc/exports

… last but not least – DNS πŸ™‚

root@nfs_server:/ # tail -3 /etc/hosts host nfs_server

As we are using VNET network stack in a FreeBSD Jail we will have to address the network interface in the Jails /etc/rc.conf file. The unfsd(8) daemon does not start without rpcbind service so we will also enable it.

root@nfs_server:/ # cat /etc/rc.conf
  ifconfig_e0b_nfs_server=" up"


We will make unfsd(8) start automatically at Jails start with plain old /etc/rc.local file.

root@nfs_server:/ # cat /etc/rc.local 
/usr/local/sbin/unfsd &

We will not restart our FreeBSD Jail to make these changes take effect.

root@host:/ # service jail onerestart nfs_server

root@host:/ # jls
   JID  IP Address      Hostname                      Path
     2                  nfs_server                    /jail/nfs_server

root@host:/ # jexec 2

root@nfs_server:/ # 

After startup we can see that unfsd(8) is listening on a NFS 2049 port.

root@nfs_server:/ # sockstat -l4
root     sshd       1261  4  tcp4   *:22                  *:*
root     sendmail   1241  5  tcp4          *:*
root     unfsd      1223  3  udp4   *:2049                *:*
root     unfsd      1223  4  tcp4   *:2049                *:*
root     rpcbind    1196  9  udp4   *:111                 *:*
root     rpcbind    1196  10 udp4   *:842                 *:*
root     rpcbind    1196  11 tcp4   *:111                 *:*
root     syslogd    1188  6  udp4   *:514                 *:*

We should have our epair(4) interface called e0b_nfs_server addressed properly.

root@nfs_server:/ # ifconfig 
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
        inet netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
e0b_nfs_server: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 0e:27:dd:b3:81:88
        hwaddr 02:30:0d:9f:57:0b
        inet netmask 0xffffff00 broadcast
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active

Mount /share on NFS Client

I added that NFS entry to the /etc/fstab file on the host machine.

root@host:~ # cat /etc/fstab 
#DEV         #MNT       #TYPE    #OPT  #DUMP/PASS
/dev/ada0p1  /boot/efi  msdosfs  rw    2 2
/dev/ada0p3  none       swap     sw    0 0

#DEV                #MNT  #TYPE  #OPT       #DUMP/PASS  /mnt  nfs    rw,noauto  0 0

We will now attempt to mount the /share NFS export on the host machine.

root@host:/ # mount /mnt

root@host:/ # mount | grep share on /mnt (nfs)

root@host:/ # cd /mnt

root@host:/mnt # :> FILE

root@host:/mnt # ls -l FILE
-rw-r--r-- 1 root root 0 2022-05-21 22:53 FILE

root@host:/mnt # rm FILE

Seems to work properly.

Here are also network interfaces on the host machine.

root@host:/ # ifconfig 
        ether 08:00:27:b3:81:88
        inet netmask 0xffffff00 broadcast
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        inet netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
em0_bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 58:9c:fc:10:ff:dd
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: e0a_nfs_server flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 4 priority 128 path cost 2000
        member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 20000
        groups: bridge
        nd6 options=9<PERFORMNUD,IFDISABLED>
e0a_nfs_server: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:27:dd:b3:81:88
        hwaddr 02:30:0d:9f:57:0a
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active

Future of NFS Server in Jails

This setup – while allowing to run the NFS server inside FreeBSD Jail with even VNET enabled has its drawbacks unfortunately. First is that it is run in userspace instead of kernel space – which means its slower. Second is that the unfsd(8) only implements NFS version 3 – so no version 4 is not possible.


Where we can go from here? Like with WiFi stuff IMHO the FreeBSD Foundation could step in to sponsor the missing bits of NFS server and VNET to make these native tools work they should. Its up to you to put the pressure on the FreeBSD Foundation when they as what are you missing from the FreeBSD UNIX system that could be improved with one of their projects. You may also join the discussion at the 251347 Bug Report of course.

I think its a big loss that native kernel space NFS server is not currently possible with VNET FreeBSD Jails.


4 thoughts on “NFS Server Inside FreeBSD VNET Jail

  1. Pingback: NFS Server Inside FreeBSD VNET Jail - SLRED - All The Latest News, Tips and Job Post!

  2. CSS

    I’m currently not even getting unfs3 to work outside a jail – any incantation results in “mount_nfs: nmount: /mnt/logtest: Stale NFS file handle”12.4 client – 12.4 and 13.1 servers.

    I’m wondering if unfs3 maybe does not play well with ZFS?


      1. CSS

        I should have mentioned, gave it a try with 13.1 server and client, same deal. I feel like I’m going crazy… I even started ktrace-ing the mount and unfsd commands, nothing glaring, but I do wish unfsd had more debugging output.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s