Addendum – MongoDB Replica Set Cluster on FreeBSD

As I recently created a guide on how to setup 5-node MongoDB Replica Set Cluster on Oracle Linux I would like to add notes about setting the same setup on FreeBSD system.

 

I Will only post the differences between Oracle Linux and FreeBSD setup here, check original post for MongoDB instructions.

To distinguish commands I type on the host system and mongoX node I use two different prompts, this way it should be obvious what command to execute and where.

Command on the host system.

host # command

Command on the mongoX node.

root@mongoX:/ # command

VirtualBox

The machines are generally the same, here is their list.

For FreeBSD setup we would still use VirtualBox NAT Network connectivity for the virtual machines communication.

FreeBSD Install

The FreeBSD install would be generally the same as in the FreeBSD Desktop – Part 2 – Install guide. To not do the same thing 5 times I installed the first node, then after reboot I connected it to the Internet with dhclient em0 command, then added the MongoDB 3.6 package with pkg install -y mongodb36 command and then I cloned that machine into the remaining nodes. Remember to regenerate the MAC address in VirtualBox interfaces for these machines because when two or more machines share the same MAC address funny things will happen 🙂

The table below lists all MongoDB nodes and their IP addresses and roles that we will use.

NODE    ADDRESS        ROLE
mongo0  10.0.10.10/24  DATA
mongo1  10.0.10.11/24  DATA
mongo2  10.0.10.12/24  DATA
mongo3  10.0.10.13/24  DATA
mongo4  10.0.10.14/24  ARBITER (does not contain data)

The ‘last’ mongo4 node will be have the ARBITER role while mongo0 to mongo3 nodes will have DATA role.

After FreeBSD installation is finished the tuned /etc/rc.conf file on mongo0 – mongo4 machines looks as follows where X is the node number from 0 to 4.

Main FreeBSD configuration file.

root@mongoX:~ # cat /etc/rc.conf
# NETWORK
  hostname=mongoX
  ifconfig_em0="inet 10.0.10.1X/24 up"

# DAEMONS
  zfs_enable=YES
  sshd_enable=YES
  moused_enable=YES
  syslogd_flags="-ss"
  sendmail_enable=NONE

# OTHER
  clear_tmp_enable=YES
  dumpdev=NO

Boot parameters.

root@mongoX:~ # cat /boot/loader.conf
# INSTALLER
  kern.geom.label.disk_ident.enable=0
  kern.geom.label.gptid.enable=0
  vfs.zfs.min_auto_ashift=12
  zfs_load=YES

Parameters to be set at run time.

root@mongoX:~ # cat /etc/sysctl.conf
# INSTALLER
  security.bsd.see_other_uids=0
  security.bsd.see_other_gids=0
  security.bsd.unprivileged_read_msgbuf=0
  security.bsd.unprivileged_proc_debug=0
  kern.randompid=50X0
  security.bsd.stack_guard_page=1

FreeBSD Limits

MongoDB prefers that number of files limit should be at least 231471 and number of processes limit should be at least 0.5 times number of files limit. Below is the warning that will be displayed by MongoDB.

2018-04-19T17:12:43.465+0200 I CONTROL  [initandlisten] ** WARNING: soft rlimits too low. rlimits set to 12085 processes, 231471 files. Number of processes should be at least 115736 : 0.5 times number of files.

To make numbers aligned we will use 256 * 1024 for files and 128 * 1024 for processes.

host # expr 256 \* 1024
262144

host # expr 128 \* 1024
131072

Set the boot parameters.

host # for I in 0 1 2 3 4; do ssh -p 220${I} root@localhost 'cat >> /boot/loader.conf << __EOF

# FASTBOOT
  autoboot_delay=1

# MONGODB LIMITS
  kern.maxproc=131072

__EOF'
done

… and the run time parameters.

host # for I in 0 1 2 3 4; do ssh -p 220${I} root@localhost 'cat >> /etc/sysctl.conf << __EOF

# MONGODB LIMITS
  kern.maxfiles=262144
  kern.maxfilesperproc=262144
  kern.maxprocperuid=131072

__EOF'
done
host #

Reboot to make the boot parameters take effect.

host # for I in 0 1 2 3 4; do ssh -p 220${I} root@localhost reboot; done

MongoDB Startup Script

In order to MongoDB work correctly on FreeBSD one parameter needs to be added to the MongoDB – mongod – startup scipt. The parameter --setParameter=disabledSecureAllocatorDomains=\* needs to be added to the mongod_flags parameter. You may modify the /usr/local/etc/rc.d/mongod script or You can specify ‘custom’ mongod_flags parameter in /etc/rc.conf file which we will do here.

In case You would want to patch the /usr/local/etc/rc.d/mongod script here is the diff(1).

root@mongo0:~ # diff -u /root/mongod.ORG /usr/local/etc/rc.d/mongod 
--- /root/mongod.ORG    2018-04-25 20:50:04.326644000 +0200
+++ /usr/local/etc/rc.d/mongod  2018-04-25 20:50:14.019943000 +0200
@@ -30,7 +30,7 @@
 : ${mongod_enable="NO"}
 : ${mongod_limits="NO"}
 : ${mongod_dbpath="/var/db/mongodb"}
-: ${mongod_flags="--logpath ${mongod_dbpath}/mongod.log --logappend"}
+: ${mongod_flags="--logpath ${mongod_dbpath}/mongod.log --logappend --setParameter=disabledSecureAllocatorDomains=\*"}
 : ${mongod_user="mongodb"}
 : ${mongod_group="mongodb"}
 : ${mongod_config="/usr/local/etc/mongodb.conf"}

To put the changes together, here is the /etc/rc.conf file after needed changes on all MongoDB nodes.

root@mongoX:~ # cat /etc/rc.conf
# NETWORK
  hostname=mongoX
  ifconfig_em0="inet 10.0.10.1X/24 up"

# DAEMONS
  zfs_enable=YES
  sshd_enable=YES
  moused_enable=YES
  syslogd_flags="-ss"
  sendmail_enable=NONE

# OTHER
  clear_tmp_enable=YES
  dumpdev=NO

# MONGODB
  mongod_enable=YES
  mongod_flags="--logpath /var/db/mongodb/mongod.log --logappend --setParameter=disabledSecureAllocatorDomains=\*"

Time Daemon

As with every cluster we need to synchronize time between cluster nodes, so we will enable ntpd(8) here.

root@mongoX:~ # sysrc ntpd_enable=YES
ntpd_enable: NO -> YES

root@mongoX:~ # service ntpd start
Starting ntpd.

Here is final /etc/rc.conf file after adding the ntpd(8) time daemon.

root@mongoX:~ # cat /etc/rc.conf
# NETWORK
  hostname=mongoX
  ifconfig_em0="inet 10.0.10.1X/24 up"

# DAEMONS
  zfs_enable=YES
  sshd_enable=YES
  ntpd_enable=YES
  moused_enable=YES
  syslogd_flags="-ss"
  sendmail_enable=NONE

# OTHER
  clear_tmp_enable=YES
  dumpdev=NO

# MONGODB
  mongod_enable=YES
  mongod_flags="--logpath /var/db/mongodb/mongod.log --logappend --setParameter=disabledSecureAllocatorDomains=\*"

… and thats about the changes, You may now start the MongoDB instances on each node and create the cluster as specified in original article.

EOF