FreeBSD networking and containers (Jails) stacks are very mature and provide lots of useful features … yet for some reason these features are not properly advertised by the FreeBSD project … or not even documented at all. I remember when Solaris was still under Sun before ‘fatal’ 2008 Oracle acquisition and one of the advertised Solaris features were its networking capabilities – along with virtual switches etc. that were administrated with the ipadm(1M) and dladm(1M) commands. FreeBSD – while having technologies like Netgraph or Jails lightweight containers – along with VNET Jails that have full independent of the host virtual network stack … are almost not advertised at all. The VNET Jails – while being production ready and used by thousands of sysadmins for about a decade – are still not documented in the FreeBSD Handbook or FreeBSD FAQ at all … you will not be able to find a single VNET mention in the FreeBSD Handbook. Even the FreeBSD Man Pages like jail.conf(5) does not mention it – only jail(8)partially mentions VNET feature.
There are however two FreeBSD Books dedicated to Jails … one is free – FreeBSD Jails Using VNETs (from 2020) and one is non-free – FreeBSD Mastery – Jails (from 2019).
The Table of Contents for this article is:
- FreeBSD Host Setup
- Classic Jails
- VNET Jails
- Thin Provisioning Jails
- Single Process Jails
- Removing Jails
- Summary
This guide aims to make VNET Jails networking little closer and simple. While one can use Netgraph bridge for this purpose – we will use the simpler and more obvious classic network bridge supported by if_bridge(4) driver on FreeBSD. I also encourage you to check the FreeBSD Handbook – Jails chapter.
FreeBSD Host Setup
The first thing we will do is to prepare the host networking. This is the typical static network configuration on FreeBSD with single IPv4 IP address without any VLANs in the /etc/rc.conf file. We will also enable the Jails subsystem.
host # cat /etc/rc.conf
# NETWORK
hostname="host"
ifconfig_em0="inet 20.0.0.20/24"
defaultrouter="20.0.0.1"
# JAILS
jail_enable="YES"
jail_parallel_start="YES"
jail_list="classic vnet shadow"
With that setup – your classic Jails will be able to connect to the outside World.
It can be visualized more or less like that.
+--------+ +-----------------+
|GATEWAY | |20.0.0.20 HOST|
(Internet)<==>| |<==>|em0 |
|20.0.0.1| | |
+--------+ +-----------------+
Below we will create a classic Jail for a start. Each such setup requires certain decisions to be made – one of them will be using /jail as our Jails root.
We will create one with ZFS datasets now.
host # zfs create -o mountpoint=/jail -p zroot/jail host # zfs create -p zroot/jail/BASE host # zfs create -p zroot/jail/classic
I have also created the /jail/BASE and /jail/classic dirs. The first one will be used as a placeholder for various FreeBSD versions *-base.txz files. The latter will be used for our first ‘classic’ FreeBSD Jail.
host # find /jail -maxdepth 1
/jail
/jail/BASE
/jail/classic
Classic Jails
I will be using FreeBSD 13.2-RELEASE for the host system so this is the Jail version I would use for ‘classic’ Jail.
host # fetch -o /jail/BASE/13.2-RELEASE-base.txz http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/13.2-RELEASE/base.txz
The FreeBSD host can run any other FreeBSD version in a Jail as long as its not newer then the host system version. That means that while we use FreeBSD 13.2-RELEASE you are able to run and consolidate a farm of any older FreeBSD versions – along with legendary 4.11-RELEASE … or one of the most problematic 5.0-RELEASE that was the first one that introduced the SMP with its M:N threading model.
We will now create the ‘classic’ Jail.
host # tar -xf /jail/BASE/13.2-RELEASE-base.txz -C /jail/classic --unlink
… and now its configuration. Usually the jail.conf(5) file is used for that … but as you grow more Jails it becomes less practical to scroll through this file to ‘find’ your desired Jail that you want to modify. This is where the /etc/jail.conf.d dir comes handy. You will be able to place each Jail config as /etc/jail.conf.d/JAILNAME.conf file. This is what we will use here – leaving /etc/jail.conf file empty or non-existent.
This is the config we will use for the ‘classic’ Jail.
host # cat /etc/jail.conf.d/classic.conf
classic {
# STARTUP/LOGGING
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.consolelog = "/var/log/jail_console_${name}.log";
# PERMISSIONS
allow.raw_sockets;
exec.clean;
# PATH/HOSTNAME
path = "/jail/${name}";
host.hostname = "${name}";
# NETWORK
ip4.addr = 20.0.0.50;
interface = em0;
}
Now we will start that Jail.
host # service jail start classic Starting jails: classic. host # jls JID IP Address Hostname Path 31 20.0.0.50 classic /jail/classic
Our ‘classic’ Jail successfully started.
The 20.0.0.50 IP address was added to the host em0 interface as shown below.
host # ifconfig em0 em0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=481009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER,NOMAP> ether 08:00:27:09:cc:a8 inet 20.0.0.20/24 broadcast 20.0.0.255 inet 20.0.0.50/32 broadcast 20.0.0.50 media: Ethernet autoselect (1000baseT <full-duplex>) status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
The ‘classic’ Jails networking can be visualized like that.
+--------+ +-----------------------------+
|GATEWAY | |20.0.0.20 HOST|
(Internet)<==>| |<==>|em0 +-------------+ |
|20.0.0.1| |20.0.0.50<==>|____jail0____| |
+--------+ |20.0.0.51<==>|____jail1____| |
|(.......)<==>| (...) | |
| +-------------+ |
+-----------------------------+
We can also ping(8) the ‘classic’ Jail IP address from the host system.
host # ping -c 3 20.50
PING 20.50 (20.0.0.50): 56 data bytes
64 bytes from 20.0.0.50: icmp_seq=0 ttl=64 time=0.114 ms
64 bytes from 20.0.0.50: icmp_seq=1 ttl=64 time=0.049 ms
64 bytes from 20.0.0.50: icmp_seq=2 ttl=64 time=0.046 ms
--- 20.50 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.046/0.069/0.114/0.031 ms
Lets login into the ‘classic’ Jail.
host # jls JID IP Address Hostname Path 1 20.0.0.50 classic /jail/classic host # jexec classic root@classic:/ # hostname classic
Inside the ‘classic’ Jail we are able to ping(8) the host gateway.
root@classic:/ # ping -c 3 20.1
PING 20.1 (20.0.0.1): 56 data bytes
64 bytes from 20.0.0.1: icmp_seq=0 ttl=255 time=0.083 ms
64 bytes from 20.0.0.1: icmp_seq=1 ttl=255 time=0.314 ms
64 bytes from 20.0.0.1: icmp_seq=2 ttl=255 time=0.256 ms
--- 20.1 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.083/0.218/0.314/0.098 ms
We can also reach outside World popular DNS public servers.
root@classic:/ # nc -v -u 1.1.1.1 53
Connection to 1.1.1.1 53 port [udp/domain] succeeded!
^C
Lets configure one of those DNS servers for our ‘classic’ Jail and test it.
root@classic:/ # echo nameserver 1.1.1.1 > /etc/resolv.conf root@classic:/ # drill freebsd.org | grep '^[^;]' freebsd.org. 240 IN A 96.47.72.84
Seems to work properly. Lets enable sshd(8) service on it.
root@classic:/ # service sshd enable sshd enabled in /etc/rc.conf root@classic:/ # service sshd start Generating RSA host key. 3072 SHA256:mSVNDUSi14S+GiaFJgNHNLCqQi6ndFG9JaSyA/wev1k root@classic (RSA) Generating ECDSA host key. 256 SHA256:Hij315+3C/IMVJ1RX+hNJynGtVSU7ALYN0AS9/lxpJY root@classic (ECDSA) Generating ED25519 host key. 256 SHA256:qzQnJCHjhHB7jQzmimSLayBfOc3dkLzIVhmrL2r9qxM root@classic (ED25519) Performing sanity check on sshd configuration. Starting sshd. root@classic:/ # sockstat -l4 USER COMMAND PID FD PROTO LOCAL ADDRESS FOREIGN ADDRESS root sshd 15049 3 tcp4 20.0.0.50:22 *:* root sendmail 18689 3 tcp4 20.0.0.50:25 *:* root syslogd 84689 5 udp4 20.0.0.50:514 *:*
Seems to work properly. We will now try to login to it from other host on the 20.0.0.0/24 network.
laptop % ssh 20.50 The authenticity of host '20.0.0.50 (20.0.0.50)' can't be established. ED25519 key fingerprint is SHA256:qzQnJCHjhHB7jQzmimSLayBfOc3dkLzIVhmrL2r9qxM. This key is not known by any other names. Are you sure you want to continue connecting (yes/no/[fingerprint])? yes Warning: Permanently added '20.0.0.50' (ED25519) to the list of known hosts. (root@20.0.0.50) Password for root@classic: (root@20.0.0.50) Password for root@classic: (root@20.0.0.50) Password for root@classic: root@20.0.0.50: Permission denied (publickey,keyboard-interactive).
Works. I was not able to login as by default logins to root account are disabled.
VNET Jails
On the other hand the VNET Jails (with options VIMAGE in the kernel) are quite different beasts. While on the file/dir/config level they work the same – on the network part they are a lot different. They come with separate network stack and do not add their IP to the host network interface.
Also – the current configuration of the host system (as repeated below) would also not work for the VNET Jails network connectivity with the outside World.
host # cat /etc/rc.conf
# NETWORK
hostname="host"
ifconfig_em0="inet 20.0.0.20/24"
defaultrouter="20.0.0.1"
To allow VNET Jails entry to the World outside of the host system we need to use – for example – if_bridge(4) interface – and move our host IP address there. Below is the not working host network configuration – VNET Jails will not be able to access outside World.
host # ifconfig em0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=481009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER,NOMAP> ether 08:00:27:09:cc:a8 inet 20.0.0.20/24 broadcast 20.0.0.255 inet 20.0.0.50/32 broadcast 20.0.0.50 media: Ethernet autoselect (1000baseT <full-duplex>) status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 inet 127.0.0.1/8 groups: lo nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> host # netstat -Win -f inet Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll em0 - 20.0.0.0/24 20.0.0.20 164406 - - 91934 - - em0 - 20.0.0.50/32 20.0.0.50 207 - - 168 - - lo0 - 127.0.0.0/8 127.0.0.1 6 - - 6 - - host # route get 0 route to: default destination: default mask: default gateway: 20.0.0.1 fib: 0 interface: em0 flags: <UP,GATEWAY,DONE,STATIC> recvpipe sendpipe ssthresh rtt,msec mtu weight expire 0 0 0 0 1500 1 0
This is how the rc.conf(5) file needs to look like now. This config will allow VNET Jails to leave host system and access outside World.
host # cat /etc/rc.conf
# NETWORK
hostname="host"
cloned_interfaces="bridge0"
ifconfig_em0="up"
ifconfig_bridge0="inet 20.0.0.20/24 up addm em0"
defaultrouter="20.0.0.1"
gateway_enable=YES
# JAILS
jail_enable="YES"
jail_parallel_start="YES"
jail_list="classic"
… and this is how it looks in the ifconfig(8) and netstat(8) commands.
host # ifconfig em0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=481009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER,NOMAP> ether 08:00:27:09:cc:a8 inet 20.0.0.50/32 broadcast 20.0.0.50 media: Ethernet autoselect (1000baseT <full-duplex>) status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 inet 127.0.0.1/8 groups: lo nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 ether 58:9c:fc:10:ff:b6 inet 20.0.0.20/24 broadcast 20.0.0.255 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 1 priority 128 path cost 20000 groups: bridge nd6 options=9<PERFORMNUD,IFDISABLED> host # netstat -Win -f inet Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll em0 - 20.0.0.50/32 20.0.0.50 207 - - 168 - - lo0 - 127.0.0.0/8 127.0.0.1 6 - - 6 - - bridge0 - 20.0.0.0/24 20.0.0.20 164576 - - 92109 - - host # route get 0 route to: default destination: default mask: default gateway: 20.0.0.1 fib: 0 interface: bridge0 flags: <UP,GATEWAY,DONE,STATIC> recvpipe sendpipe ssthresh rtt,msec mtu weight expire 0 0 0 0 1500 1 0
The ‘classic’ Jail IP address is still bound to the em0 interface while the host IP address has been moved to the bridge0 bridge. If needed you can also move all the ‘classic’ Jails IP addresses to the bridge0 interface but its not mandatory.
It can be visualized like that. Keep in mind that em0 and epair*a interfaces are members of bridge0 interface.
+--------------------------------------------+
| +-------------+ HOST|
| em0 20.0.0.50<==>|____jail0____| |
| / 20.0.0.51<==>|____jail1____| |
+--------+ | / (.......)<==>| (...) | |
|GATEWAY | |20.0.0.20 +-------------+ |
(Internet)<==>| |<==>|bridge0 |
|20.0.0.1| | \ \ \ +-----------------------+ |
+--------+ | \ \ epairXa<==>|epairXb 20.0.0.60 vnet0| |
| \ \ +-----------------------+ |
| \ epairYa<==>|epairYb 20.0.0.61 vnet1| |
| \ +-----------------------+ |
| (....)a<==>|(....)b (.......) (...)| |
| +-----------------------+ |
+--------------------------------------------+
Lets create now our first VNET Jail. The beginning is the same – we will extract base.txz file from the 13.2-RELEASE system.
host # zfs create -p zroot/jail/vnet host # tar -xf /jail/BASE/13.2-RELEASE-base.txz -C /jail/vnet --unlink
We will now also need the VNET Jail config.
host # cat /etc/jail.conf.d/vnet.conf
vnet {
# STARTUP/LOGGING
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.consolelog = "/var/log/jail_console_${name}.log";
# PERMISSIONS
allow.raw_sockets;
exec.clean;
# PATH/HOSTNAME
path = "/jail/${name}";
host.hostname = "${name}";
# VNET/VIMAGE
vnet;
vnet.interface = "${if}b";
# NETWORKS/INTERFACES
$id = "60";
$ip = "20.0.0.${id}/24";
$gw = "20.0.0.1";
$br = "bridge0";
$if = "epair${id}";
# ADD TO bridge0 INTERFACE
exec.prestart += "ifconfig ${if} create up";
exec.prestart += "ifconfig ${if}a up descr jail:${name}";
exec.prestart += "ifconfig ${br} addm ${if}a up";
exec.start += "ifconfig ${if}b ${ip} up";
exec.start += "route add default ${gw}";
exec.poststop += "ifconfig ${if}a destroy";
}
Lets now start the VNET Jail.
host # service jail start vnet Starting jails: vnet. host # jls JID IP Address Hostname Path 1 20.0.0.50 classic /jail/classic 2 vnet /jail/vnet
One thing to notice here is that the jls(8) tool does not show the VNET Jails IP addresses.
You can of course overcome that limitation with jexec(8) – but IMHO after 10+ years of VNET Jails being production ready its PITA to say the least.
host # jexec vnet ifconfig | grep 'inet '
inet 127.0.0.1/8
inet 20.0.0.60/24 broadcast 20.0.0.255
Lets now try our VNET Jail network connectivity to the outside World.
host # jexec vnet root@vnet:/ # ping -c 3 20.1 PING 20.1 (20.0.0.1): 56 data bytes 64 bytes from 20.0.0.1: icmp_seq=0 ttl=255 time=0.853 ms 64 bytes from 20.0.0.1: icmp_seq=1 ttl=255 time=0.474 ms 64 bytes from 20.0.0.1: icmp_seq=2 ttl=255 time=1.355 ms --- 20.1 ping statistics --- 3 packets transmitted, 3 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 0.474/0.894/1.355/0.361 ms root@vnet:/ # nc -v -u 1.1.1.1 53 Connection to 1.1.1.1 53 port [udp/domain] succeeded! ^C
Works as desired.
This is how the networking looks like inside the VNET Jail.
root@vnet:/ # ifconfig lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 inet 127.0.0.1/8 groups: lo nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> epair60b: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=8<VLAN_MTU> ether 02:47:e8:2e:6b:0b inet 20.0.0.60/24 broadcast 20.0.0.255 groups: epair media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>) status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> root@vnet:/ # netstat -Win -f inet Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll lo0 - 127.0.0.0/8 127.0.0.1 2848 - - 2848 - - epair60b - 20.0.0.0/24 20.0.0.60 5 - - 9 - - root@vnet:/ # route get 0 route to: default destination: default mask: default gateway: 20.0.0.1 fib: 0 interface: epair60b flags: <UP,GATEWAY,DONE,STATIC> recvpipe sendpipe ssthresh rtt,msec mtu weight expire 0 0 0 0 1500 1 0
Same as with ‘classic’ Jail – we will enable the sshd(8) service.
root@vnet:/ # service sshd enable sshd enabled in /etc/rc.conf root@vnet:/ # service sshd start Generating RSA host key. 3072 SHA256:n9sGBV1bmz3bT4+qKuFE5fZHjnBcYlOMxXCq98z7/r0 root@vnet (RSA) Generating ECDSA host key. 256 SHA256:A+gDtzkkrhNnGRGR4Yf27cqME8/NZk5NCHrxwyEO9oM root@vnet (ECDSA) Generating ED25519 host key. 256 SHA256:aojc9Kbyd32HkllG9+noKL8GvKMjObuLrUNiq24+OFk root@vnet (ED25519) Performing sanity check on sshd configuration. Starting sshd. root@vnet:/ # sockstat -l4 USER COMMAND PID FD PROTO LOCAL ADDRESS FOREIGN ADDRESS root sshd 37145 4 tcp4 *:22 *:* root sendmail 99914 4 tcp4 127.0.0.1:25 *:* root syslogd 71123 6 udp4 *:514 *:*
We will try to connect to it from a system other then the host.
laptop % ssh 20.60 The authenticity of host '20.0.0.60 (20.0.0.60)' can't be established. ED25519 key fingerprint is SHA256:aojc9Kbyd32HkllG9+noKL8GvKMjObuLrUNiq24+OFk. This key is not known by any other names. Are you sure you want to continue connecting (yes/no/[fingerprint])? yes Warning: Permanently added '20.0.0.60' (ED25519) to the list of known hosts. (root@20.0.0.60) Password for root@vnet: (root@20.0.0.60) Password for root@vnet: (root@20.0.0.60) Password for root@vnet: root@20.0.0.60: Permission denied (publickey,keyboard-interactive).
Same as with the ‘classic’ Jail – we will not login with root user as its disabled by default – but that is not the task of this exercise.
Thin Provisioning Jails
While this will not be possible on the UFS filesystem – the ZFS allows that without any hassle.
We will now create a template for the ‘classic’ Jails on ZFS.
host # zfs snapshot zroot/jail/classic@template
The only things we did with the ‘classic’ Jail was to setup DNS server in /etc/resolv.conf file and we enabled sshd(8) service. We generally want that from any of our future Jails – so its a good candidate for a template Jail.
As we created the classic@template ZFS snapshot – we can not create unlimited thin provisioning ‘classic’ Jails from it. Just make sure to create needed /etc/jail.conf.d Jail config files.
Lets create a new thin provisioning Jail from it now.
host # zfs list -t snapshot NAME USED AVAIL REFER MOUNTPOINT zroot/jail/classic@template 224K - 503M - host # zfs clone zroot/jail/classic@template zroot/jail/shadow host # zfs list -t all NAME USED AVAIL REFER MOUNTPOINT zroot 3.31G 15.6G 96K none zroot/ROOT 700M 15.6G 96K none zroot/ROOT/default 699M 15.6G 699M / zroot/jail 1.86G 15.6G 194M /jail zroot/jail/BASE 191M 15.6G 191M /jail/BASE zroot/jail/classic 503M 15.6G 503M /jail/classic zroot/jail/classic@template 224K - 503M - zroot/jail/shadow 8K 15.6G 503M /jail/shadow zroot/jail/vnet 503M 15.6G 503M /jail/vnet zroot/tmp 96K 15.6G 96K /tmp zroot/usr 780M 15.6G 96K /usr zroot/usr/home 96K 15.6G 96K /usr/home zroot/usr/ports 96K 15.6G 96K /usr/ports zroot/usr/src 780M 15.6G 780M /usr/src zroot/var 684K 15.6G 96K /var zroot/var/audit 96K 15.6G 96K /var/audit zroot/var/crash 96K 15.6G 96K /var/crash zroot/var/log 204K 15.6G 204K /var/log zroot/var/mail 96K 15.6G 96K /var/mail zroot/var/tmp 96K 15.6G 96K /var/tmp host # zfs list -r -o space zroot/jail NAME AVAIL USED USEDSNAP USEDDS USEDREFRESERV USEDCHILD zroot/jail 15.6G 1.88G 0B 212M 0B 1.67G zroot/jail/BASE 15.6G 191M 0B 191M 0B 0B zroot/jail/classic 15.6G 503M 260K 503M 0B 0B zroot/jail/nfsd 15.6G 513M 0B 513M 0B 0B zroot/jail/shadow 15.6G 392K 0B 392K 0B 0B zroot/jail/vnet 15.6G 505M 0B 505M 0B 0B host # cp /etc/jail.conf.d/classic.conf /etc/jail.conf.d/shadow.conf host # sed -i '' -e 's|classic|shadow|g' -e 's|20.0.0.50|20.0.0.51|g' /etc/jail.conf.d/shadow.conf host # cat /etc/jail.conf.d/shadow.conf shadow { # STARTUP/LOGGING exec.start = "/bin/sh /etc/rc"; exec.stop = "/bin/sh /etc/rc.shutdown"; exec.consolelog = "/var/log/jail_console_${name}.log"; # PERMISSIONS allow.raw_sockets; exec.clean; # PATH/HOSTNAME path = "/jail/${name}"; host.hostname = "${name}"; # NETWORK ip4.addr = 20.0.0.51; interface = em0; } host # service jail start shadow Starting jails: shadow. host # jls JID IP Address Hostname Path 1 20.0.0.50 classic /jail/classic 2 vnet /jail/vnet 3 20.0.0.51 shadow /jail/shadow host # netstat -Win -f inet Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll em0 - 20.0.0.50/32 20.0.0.50 2854 - - 2746 - - em0 - 20.0.0.51/32 20.0.0.51 3 - - 0 - - lo0 - 127.0.0.0/8 127.0.0.1 6 - - 6 - - bridge0 - 20.0.0.0/24 20.0.0.20 195544 - - 150555 - -
While the ‘classic’ Jail uses about 500 MB of space the Thin Provisioning shadow Jail uses less then 1 MB of space.
We can now test its sshd(8) daemon connection.
laptop % ssh 20.51 The authenticity of host '20.0.0.51 (20.0.0.51)' can't be established. ED25519 key fingerprint is SHA256:qzQnJCHjhHB7jQzmimSLayBfOc3dkLzIVhmrL2r9qxM. This key is not known by any other names. Are you sure you want to continue connecting (yes/no/[fingerprint])? yes Warning: Permanently added '20.0.0.51' (ED25519) to the list of known hosts. (root@20.0.0.51) Password for root@shadow: (root@20.0.0.51) Password for root@shadow: (root@20.0.0.51) Password for root@shadow: root@20.0.0.51: Permission denied (publickey,keyboard-interactive).
Yep. Works as it should.
You can create a template for VNET Jail in the same manner.
Single Process Jails
The Docker is mostly know from its single process execution. You choose a process/daemon that you want to run isolated – copy it into the Docker container with its dependencies and you start it.
Exactly the same can be achieved with FreeBSD Jails containers.
For the purposes of this guide we will execute /bin/sh POSIX shell interpreter in a separated FreeBSD Jail. To omit copying additional dependency files we will use the statically compiled version from the well known FreeBSD Rescue System.
host # mkdir -p /jail/shell/dev host # cp /rescue/sh /rescue/hostname /jail/shell/ host # jail -n shell \ -c path=/jail/shell \ mount.devfs \ host.hostname=shell \ ip4.addr=20.0.0.111 \ command=/sh shell # /hostname shell shell # /sh Cannot read termcap database; using dumb terminal settings. shell # for I in 1 2 3; do echo ${I}; done 1 2 3 shell # echo /* /dev /hostname /sh
You can login to the host system from other SSH session to see that Thin Provisioning shell Jail is running among our other FreeBSD Jails.
host # jls JID IP Address Hostname Path 1 20.0.0.50 classic /jail/classic 2 vnet /jail/vnet 3 20.0.0.51 shadow /jail/shadow 4 20.0.0.111 shell /jail/shell
Removing Jails
For that purpose we will remove the ‘classic’ Jail.
We will first stop our Jail.
host # service jail stop classic
host # rm -rf /jail/classic
rm: /jail/classic/var/empty: Operation not permitted
rm: /jail/classic/var: Directory not empty
rm: /jail/classic/libexec/ld-elf32.so.1: Operation not permitted
rm: /jail/classic/libexec/ld-elf.so.1: Operation not permitted
rm: /jail/classic/libexec: Directory not empty
rm: /jail/classic/lib/libthr.so.3: Operation not permitted
rm: /jail/classic/lib/libcrypt.so.5: Operation not permitted
rm: /jail/classic/lib/libc.so.7: Operation not permitted
rm: /jail/classic/lib: Directory not empty
(...)
Wait … you are root and you can not delete files? 🙂 This is just another FreeBSD feature. The File Flags. They can also be used with FreeBSD Secure Levels feature. By default some files and directories are marked with some of these flags. To be able to remove these files you first need to remove these flags. We can do that with chflags(1) command.
host # chflags -R 0 /jail/classic host # rm -rf /jail/classic host # echo ${?} 0
Now as the File Flags have been removed we can delete all files and dirs with usual rm(1) command. As our ‘classic’ Jail files are gone we should also delete the Jail config at /etc/jail.conf.d/classic.conf file.
host # rm -f /etc/jail.conf.d/classic.conf
Summary
I am not sure what else I should add here … but I am sure that you will let me know in the comments section 🙂
Regards.
EOF
𝚟𝚎𝚛𝚖𝚊𝚍𝚎𝚗 
Pingback: Recipientes de prisões FreeBSD – linux-BR.org
Pingback: FreeBSD Jails コンテナ – 世界の話題を日本語でザックリ素早く確認！
Thanks for posting this up. You’re right, there’s not a lot of information available about FreeBSD jails. This was a nice pretty easy to digest overview. Would be an interesting follow-up to see a specific task like setting up a jail nuts-to-bolts for running apache in one jail, database in another. Regardless, thank you again for sharing your article.
LikeLike
Thanks.
IMHO example You are asking for is here for example:
– https://blog.uidrafter.com/freebsd-jails-network-setup
Regards,
vermaden
LikeLike
Good day.
Is it possible for VNET Jails to use not a single interface em0, but an aggregated one of the lagg(4) type (lasp, loadbalance, …).
LikeLike
Yes, just configure the lagg0 interface on the FreeBSD host and then pass it into FreeBSD Jail.
LikeLike
Pingback: FreeBSD Jails Containers – Breaking News
Thank you Vermaden, your writing style is always instructive, clarifying and much enjoyable.
I come from OmniOS, I was a bit spoiled from illumos’ zones. With VNET things are starting to get a bit more similar.
Would you consider showing a bare-bone approach dealing with multiple externals IPs and many jails using VNET + pfnat?
LikeLike
Thanks.
IMHO the example You are looking for is here for example:
– https://blog.uidrafter.com/freebsd-jails-network-setup
Regards,
vermaden
LikeLike
Wonderful, thanks!
LikeLike
Pingback: FreeBSD监狱容器 - 偏执的码农
Pingback: FreeBSD Jails Containers - PROJIN NEWS
Indeed, the under-advertisement of FreeBSD’s mature networking and container capabilities is quite puzzling. Your detailed analysis and guide on leveraging VNET Jails networking do a fantastic job at shedding light on the subject. It’s interesting to think about how many potentially transformative tools or technologies are out there, underused simply due to a lack of awareness or sufficient documentation.
LikeLike
Obvious links spam is obvious.
LikeLike
Unrelated Question: what window manager are you running? (its beautiful)
LikeLike
Its Openbox.
You can check its configuration here:
… and the rest of that desktop setup here:
Regards,
vermaden
LikeLike
Pingback: Links 28/06/2023: NVK Update and Ubuntu EoL | Techrights
Pingback: FreeBSD Jails Containers – Cyber Geeks Global
This was very helpful – quite clear and concise. It would be great addition to the handbook.
If you were to do a follow on, one place that trips me up a bit with jails and bhyve is with firewalls. More specifically, IPFW gets a bit gnarly when dealing with a bridge on the machines external facing interface. Making use of the available sysctl knobs relating to how bridges and IPFW interact never seems to work quite the way I expect.
A completely internal bridge is easy, we can just treat it as an internal network segment with nat and it works quite nicely.
LikeLike
Thanks mate.
LikeLike
This was so much fun to read! I can’t wait to get home and start playing with Jails!
Thank you mate for such a great explanation!
LikeLike
Thanks mate 🙂
LikeLike
Didn’t know you could run single-process Jails, like containers, I’m impressed.
LikeLiked by 1 person
The beautiful artice, as usual. Thanks, vermaden!
Hope to see one more article about using jails on single-public-IP hosts, when we have to use firewall/NAT.
LikeLiked by 1 person
Great article, thanks! Would it be also possible to write an example for IPv6, please? It’s quite different from IPv4, which uses NAT.
Also, another mystery of FreeBSD – netgraph. It sounds like such a cool technology, but the only proper documentation I could find are man pages. One could build a whole network with multiple bridges and I don’t know what else just using it. Yet almost zero documentation for it…
LikeLike
Pingback: NFSv4 Server Inside FreeBSD VNET Jail | 𝚟𝚎𝚛𝚖𝚊𝚍𝚎𝚗
Pingback: Valuable News – 2023/07/10 | 𝚟𝚎𝚛𝚖𝚊𝚍𝚎𝚗
Pingback: Quare FreeBSD? | 𝚟𝚎𝚛𝚖𝚊𝚍𝚎𝚗
Excellent article Vermaden, thank you. VNET jails by definition allow raw sockets so no need to include that line in the vnet.conf file.
LikeLike
Thanks, did not knew that 🙂
LikeLike
Hi,
follow up questions:
1) How to configure VNET Jails to belong to different IP network?
2) Can I configure multiple subnets via the same bridge?
LikeLike
Hi.
You mean different the the host network?
For example You can pass a real physical em1 interface into your VNET Jail and address it in a network that host does not even know.
Does that answer your question?
You can add as many IPs/subnets as you want but the outgoing traffic will still land on a default gateway – not sure if that helps 🙂
Regards.
LikeLike