Monthly Archives: April 2018

Valuable News – 2018/04/30

Setting up Buildbot in FreeBSD Jails.
https://andidog.de/blog/2018-04-22-buildbot-setup-freebsd-jails

Bringing Objective-C to the Amiga.
https://heap.zone/blog/?76

Patch Meltdown and Spectre on FreeBSD.
https://www.cyberciti.biz/faq/how-to-patch-meltdown-and-spectre-vulnerabilities-on-freebsd/

Free Command for OpenBSD.
https://github.com/NanXiao/free

… and FreeBSD Free Command Equivalent.
https://www.freshports.org/sysutils/freecolor/
http://www.rkeene.org/oss/freecolor/

BCHS is hipster-free OpenBSD based open source software stack for web applications.
https://learnbchs.org/

It’s Impossible to Prove Your Laptop Hasn’t Been Hacked.
https://theintercept.com/2018/04/28/computer-malware-tampering/

This ‘Demonically Clever’ Backdoor Hides In Tiny Slice of Computer Chip
https://www.wired.com/2016/06/demonically-clever-backdoor-hides-inside-computer-chip/

Edward Snowden’s New App Uses Your Smartphone to Physically Guard Your Laptop
https://theintercept.com/2017/12/22/snowdens-new-app-uses-your-smartphone-to-physically-guard-your-laptop/

Netflix Tips for High Availability
https://medium.com/@NetflixTechBlog/tips-for-high-availability-be0472f2599c

OSM – Object Store Manipulator – The curl for Cloud Storage.
https://github.com/appscode/osm

In Other BSDs for 2018/04/28.
https://www.dragonflydigest.com/2018/04/28/21156.html

EOF.

.
.
.
.
.

MongoDB Replica Set Cluster on Oracle Linux

Meet MongoDB.

mongodb-logo

MongoDB is a free and open-source cross-platform document database with scalability and flexibility. Classified as NoSQL database MongoDB uses JSON like documents with schemas. MongoDB is a distributed database at its core, so high availability, horizontal scaling, and geographic distribution are built in and easy to use.

Today I will show you how to install and configure MongoDB Cluster Replica Set with 4 data nodes and 1 arbiter node. Minimal replica set configuration is three members and largest replica set can support only 12 members in total. The replica set must have odd number of voting members. As I always used FreeBSD or its forks for various setups I will today use latest Oracle Linux 7.5 for this example.

Architecture

Below is the POOR MAN’S ASCII ARCHITECT diagram showing that five node MongoDB replica set cluster installation.

mongo0 [DATA]       |   |
/var/lib/mongo -- > |   |
                    |   |
mongo1 [DATA]       | M |
/var/lib/mongo -- > | o |
                    | n |
mongo2 [DATA]       | g |
/var/lib/mongo -- > | o |
                    |   |
mongo3 [DATA]       | D |
/var/lib/mongo -- > | B |
                    |   |
mongo4 [ARBITER]    |   |
/var/lib/mongo -- x |   |

The MongoDB project visualizes this little differently, as show below.

mongodb-replica-set-four-members-one-arbiter

VirtualBox

For the convenience of the setup we will use VirtualBox virtual machines for our MongoDB replica set cluster setup. Below is list of VirtualBox virtual machines used in the setup.

virtualbox-mongodb-list

We will use VirtualBox NAT Network connectivity for the virtual machines communication. Below are settings for the NAT Network we will use here.

virtualbox-mongodb-nat-01

virtualbox-mongodb-nat-02

virtualbox-mongodb-nat-03-forward

virtualbox-mongodb-nat-04-vm

We can verify that ports forwarding is working with sockstat command from the host FreeBSD system.

host % sockstat -l4
USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS      
vermaden VBoxNetNAT 13138 17 udp4   *:*                   *:*
vermaden VBoxNetNAT 13138 19 tcp4   *:2200                *:*
vermaden VBoxNetNAT 13138 20 tcp4   *:2201                *:*
vermaden VBoxNetNAT 13138 21 tcp4   *:2202                *:*
vermaden VBoxNetNAT 13138 22 tcp4   *:2203                *:*
vermaden VBoxNetNAT 13138 23 tcp4   *:2204                *:*
root     sshd       986   4  tcp4   *:22                  *:*

The table below lists all MongoDB nodes and their IP addresses and roles that we will use.

NODE    ADDRESS        ROLE
mongo0  10.0.10.10/24  DATA
mongo1  10.0.10.11/24  DATA
mongo2  10.0.10.12/24  DATA
mongo3  10.0.10.13/24  DATA
mongo4  10.0.10.14/24  ARBITER (does not contain data)

The ‘last’ mongo4 node will be have the ARBITER role while mongo0 to mongo3 nodes will have DATA role. Similarly like with the Distributed Object Storage with Minio on FreeBSD You can place two nodes (mongo0 and mongo2 for example) in primary datacenter, other two nodes (mongo1 and mongo3 for example) in secondary datacenter and mongo4 with ARBITER node in the third datacenter or other location available from the primary and secondary datacenters.

To not do the same thing five times I installed the first node (mongo0) then updated it and made some preconfigurations, then powered it off and I cloned into the remaining nodes. Remember to regenerate the MAC addresses in VirtualBox interface in the cloning process for these machines to omit ‘strange’ connectivity problems πŸ™‚

After cloning the only files that needs to be modified are these:

  • /etc/sysconfig/network-scripts/ifcfg-eth0
  • /etc/hostname

Below is an example for mongo4 machine.

[root@mongo4 ~]# grep 4$ /etc/sysconfig/network-scripts/ifcfg-eth0 /etc/hostname 
/etc/sysconfig/network-scripts/ifcfg-eth0:IPADDR=10.0.10.14
/etc/sysconfig/network-scripts/ifcfg-eth0:PREFIX=24
/etc/hostname:mongo4

To distinguish commands I type on the host system and mongoX virtual machines I use two different prompts, this way it should be obvious what command to execute and where.

Command on the host system.

host % command

Command on the mongoX virtual machine.

[root@mongoX ~]# command

Linux

I have installed Oracle Linux 7.5 on a single primary / partition on XFS filesystem as show on the images below, this is Minimal install with statically configured network connection in VirtualBox NAT Network mode.

oracle-linux-7.5-install-01

oracle-linux-7.5-install-02

oracle-linux-7.5-install-03

If that will make life easier for anybody, here is the /root/anaconda-ks.cfg file.

[root@mongo0 ~]# cat /root/anaconda-ks.cfg 
#version=DEVEL
# System authorization information
auth --enableshadow --passalgo=sha512
repo --name="Server-HighAvailability" --baseurl=file:///run/install/repo/addons/HighAvailability
repo --name="Server-ResilientStorage" --baseurl=file:///run/install/repo/addons/ResilientStorage
# Use CDROM installation media
cdrom
# Use graphical install
graphical
# Run the Setup Agent on first boot
firstboot --enable
ignoredisk --only-use=sda
# Keyboard layouts
keyboard --vckeymap=us --xlayouts='us'
# System language
lang en_US.UTF-8

# Network information
network  --bootproto=static --device=enp0s3 --gateway=10.0.10.1 --ip=10.0.10.10 --nameserver=1.1.1.1 --netmask=255.255.255.0 --ipv6=auto --activate
network  --hostname=mongo0

# Root password
rootpw --iscrypted $6$EzciOQdLpJD8IJTv$wnAvxjgP.JluqsRAPu/mbTv8Upvg02AAb4.T5zBi6VMGdNfNsiRw7Gp0FyRtwAGW5Orpqc1nRwtRFwLQDJU/l.
# System services
services --disabled="chronyd"
# System timezone
timezone Europe/Warsaw --isUtc --nontp
# System bootloader configuration
bootloader --location=mbr --boot-drive=sda
# Partition clearing information
clearpart --none --initlabel
# Disk partitioning information
part / --fstype="xfs" --ondisk=sda --size=16383 --label=ROOT

%packages
@^minimal
@core

%end

%addon com_redhat_kdump --disable --reserve-mb='auto'

%end

%anaconda
pwpolicy root --minlen=6 --minquality=1 --notstrict --nochanges --notempty
pwpolicy user --minlen=6 --minquality=1 --notstrict --nochanges --emptyok
pwpolicy luks --minlen=6 --minquality=1 --notstrict --nochanges --notempty
%end

After the first boot I will yum update the system to the latest version.

[root@mongo0 ~]# yum update
Loaded plugins: ulninfo
Resolving Dependencies
--> Running transaction check
---> Package initscripts.x86_64 0:9.49.41-1.0.1.el7 will be updated
---> Package initscripts.x86_64 0:9.49.41-1.0.3.el7 will be an update
---> Package kernel-uek.x86_64 0:4.1.12-124.14.1.el7uek will be installed
---> Package kernel-uek-firmware.noarch 0:4.1.12-124.14.1.el7uek will be installed
---> Package krb5-libs.x86_64 0:1.15.1-18.el7 will be updated
---> Package krb5-libs.x86_64 0:1.15.1-19.el7 will be an update
---> Package selinux-policy.noarch 0:3.13.1-192.0.1.el7 will be updated
---> Package selinux-policy.noarch 0:3.13.1-192.0.1.el7_5.3 will be an update
---> Package selinux-policy-targeted.noarch 0:3.13.1-192.0.1.el7 will be updated
---> Package selinux-policy-targeted.noarch 0:3.13.1-192.0.1.el7_5.3 will be an update
---> Package tzdata.noarch 0:2018c-1.el7 will be updated
---> Package tzdata.noarch 0:2018d-1.el7 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

====================================================================================================
 Package                        Arch          Version                       Repository         Size
====================================================================================================
Installing:
 kernel-uek                     x86_64        4.1.12-124.14.1.el7uek        ol7_UEKR4          46 M
 kernel-uek-firmware            noarch        4.1.12-124.14.1.el7uek        ol7_UEKR4         2.5 M
Updating:
 initscripts                    x86_64        9.49.41-1.0.3.el7             ol7_latest        437 k
 krb5-libs                      x86_64        1.15.1-19.el7                 ol7_latest        747 k
 selinux-policy                 noarch        3.13.1-192.0.1.el7_5.3        ol7_latest        452 k
 selinux-policy-targeted        noarch        3.13.1-192.0.1.el7_5.3        ol7_latest        6.6 M
 tzdata                         noarch        2018d-1.el7                   ol7_latest        480 k

Transaction Summary
====================================================================================================
Install  2 Packages
Upgrade  5 Packages

Total download size: 57 M
Is this ok [y/d/N]: y
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
warning: /var/cache/yum/x86_64/7Server/ol7_latest/packages/initscripts-9.49.41-1.0.3.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID ec551f03: NOKEY
Public key for initscripts-9.49.41-1.0.3.el7.x86_64.rpm is not installed
(1/7): initscripts-9.49.41-1.0.3.el7.x86_64.rpm                              | 437 kB  00:00:02     
(2/7): selinux-policy-3.13.1-192.0.1.el7_5.3.noarch.rpm                      | 452 kB  00:00:01     
(3/7): krb5-libs-1.15.1-19.el7.x86_64.rpm                                    | 747 kB  00:00:04     
(4/7): tzdata-2018d-1.el7.noarch.rpm                                         | 480 kB  00:00:02     
Public key for kernel-uek-firmware-4.1.12-124.14.1.el7uek.noarch.rpm is not installed  00:01:09 ETA 
(5/7): kernel-uek-firmware-4.1.12-124.14.1.el7uek.noarch.rpm                 | 2.5 MB  00:00:13     
(6/7): selinux-policy-targeted-3.13.1-192.0.1.el7_5.3.noarch.rpm             | 6.6 MB  00:00:22     
(7/7): kernel-uek-4.1.12-124.14.1.el7uek.x86_64.rpm                          |  46 MB  00:01:19     
----------------------------------------------------------------------------------------------------
Total                                                               732 kB/s |  57 MB  00:01:19     
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
Importing GPG key 0xEC551F03:
 Userid     : "Oracle OSS group (Open Source Software group) "
 Fingerprint: 4214 4123 fecf c55b 9086 313d 72f9 7b74 ec55 1f03
 Package    : 7:oraclelinux-release-7.5-1.0.3.el7.x86_64 (@anaconda/7.5)
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
Is this ok [y/N]: y
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Updating   : selinux-policy-3.13.1-192.0.1.el7_5.3.noarch                                    1/12 
  Updating   : initscripts-9.49.41-1.0.3.el7.x86_64                                            2/12 
  Installing : kernel-uek-firmware-4.1.12-124.14.1.el7uek.noarch                               3/12 
  Installing : kernel-uek-4.1.12-124.14.1.el7uek.x86_64                                        4/12 
  Updating   : selinux-policy-targeted-3.13.1-192.0.1.el7_5.3.noarch                           5/12 
  Updating   : tzdata-2018d-1.el7.noarch                                                       6/12 
  Updating   : krb5-libs-1.15.1-19.el7.x86_64                                                  7/12 
  Cleanup    : selinux-policy-targeted-3.13.1-192.0.1.el7.noarch                               8/12 
  Cleanup    : selinux-policy-3.13.1-192.0.1.el7.noarch                                        9/12 
  Cleanup    : tzdata-2018c-1.el7.noarch                                                      10/12 
  Cleanup    : krb5-libs-1.15.1-18.el7.x86_64                                                 11/12 
  Cleanup    : initscripts-9.49.41-1.0.1.el7.x86_64                                           12/12 
  Verifying  : kernel-uek-4.1.12-124.14.1.el7uek.x86_64                                        1/12 
  Verifying  : selinux-policy-targeted-3.13.1-192.0.1.el7_5.3.noarch                           2/12 
  Verifying  : kernel-uek-firmware-4.1.12-124.14.1.el7uek.noarch                               3/12 
  Verifying  : initscripts-9.49.41-1.0.3.el7.x86_64                                            4/12 
  Verifying  : selinux-policy-3.13.1-192.0.1.el7_5.3.noarch                                    5/12 
  Verifying  : krb5-libs-1.15.1-19.el7.x86_64                                                  6/12 
  Verifying  : tzdata-2018d-1.el7.noarch                                                       7/12 
  Verifying  : initscripts-9.49.41-1.0.1.el7.x86_64                                            8/12 
  Verifying  : tzdata-2018c-1.el7.noarch                                                       9/12 
  Verifying  : krb5-libs-1.15.1-18.el7.x86_64                                                 10/12 
  Verifying  : selinux-policy-3.13.1-192.0.1.el7.noarch                                       11/12 
  Verifying  : selinux-policy-targeted-3.13.1-192.0.1.el7.noarch                              12/12 

Installed:
  kernel-uek.x86_64 0:4.1.12-124.14.1.el7uek   kernel-uek-firmware.noarch 0:4.1.12-124.14.1.el7uek  

Updated:
  initscripts.x86_64 0:9.49.41-1.0.3.el7                                                            
  krb5-libs.x86_64 0:1.15.1-19.el7                                                                  
  selinux-policy.noarch 0:3.13.1-192.0.1.el7_5.3                                                    
  selinux-policy-targeted.noarch 0:3.13.1-192.0.1.el7_5.3                                           
  tzdata.noarch 0:2018d-1.el7                                                                       

Complete!
[root@mongo0 ~]#

As on of the packages was kernel I will now reboot the system.

[root@mongo0 ~]# reboot

After reboot there will be two kernels 4.x kernels installed, the original one that came on the ISO image and the latest one, lets remove the unneeded older version.

[root@mongo0 ~]# rpm -qa | grep kernel | sort
kernel-3.10.0-862.el7.x86_64
kernel-tools-3.10.0-862.el7.x86_64
kernel-tools-libs-3.10.0-862.el7.x86_64
kernel-uek-4.1.12-112.16.4.el7uek.x86_64
kernel-uek-4.1.12-124.14.1.el7uek.x86_64
kernel-uek-firmware-4.1.12-112.16.4.el7uek.noarch
kernel-uek-firmware-4.1.12-124.14.1.el7uek.noarch

[root@mongo0 ~]# uname -r
4.1.12-124.14.1.el7uek.x86_64

[root@mongo0 ~]# rpm -e kernel-uek-firmware-4.1.12-112.16.4.el7uek.noarch kernel-uek-4.1.12-112.16.4.el7uek.x86_64

[root@mongo0 ~]# rpm -qa | grep kernel | sort
kernel-3.10.0-862.el7.x86_64
kernel-tools-3.10.0-862.el7.x86_64
kernel-tools-libs-3.10.0-862.el7.x86_64
kernel-uek-4.1.12-124.14.1.el7uek.x86_64
kernel-uek-firmware-4.1.12-124.14.1.el7uek.noarch

Now we will add the MongoDB repository.

[root@mongo0 ~]# cat > /etc/yum.repos.d/mongodb-org-3.6.repo << __EOF
> [mongodb-org-3.6]
> name=MongoDB Repository
> baseurl=https://repo.mongodb.org/yum/redhat/\$releasever/mongodb-org/3.6/x86_64/
> gpgcheck=1
> enabled=1
> gpgkey=https://www.mongodb.org/static/pgp/server-3.6.asc
> __EOF

[root@mongo0 ~]# cat /etc/yum.repos.d/mongodb-org-3.6.repo
[mongodb-org-3.6]
name=MongoDB Repository
baseurl=https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/3.6/x86_64/
gpgcheck=1
enabled=1
gpgkey=https://www.mongodb.org/static/pgp/server-3.6.asc

We will then install the MongoDB package.

[root@mongo0 ~]# yum install mongodb-org
Loaded plugins: ulninfo
Resolving Dependencies
--> Running transaction check
---> Package mongodb-org.x86_64 0:3.6.4-1.el7 will be installed
--> Processing Dependency: mongodb-org-tools = 3.6.4 for package: mongodb-org-3.6.4-1.el7.x86_64
--> Processing Dependency: mongodb-org-shell = 3.6.4 for package: mongodb-org-3.6.4-1.el7.x86_64
--> Processing Dependency: mongodb-org-server = 3.6.4 for package: mongodb-org-3.6.4-1.el7.x86_64
--> Processing Dependency: mongodb-org-mongos = 3.6.4 for package: mongodb-org-3.6.4-1.el7.x86_64
--> Running transaction check
---> Package mongodb-org-mongos.x86_64 0:3.6.4-1.el7 will be installed
---> Package mongodb-org-server.x86_64 0:3.6.4-1.el7 will be installed
---> Package mongodb-org-shell.x86_64 0:3.6.4-1.el7 will be installed
---> Package mongodb-org-tools.x86_64 0:3.6.4-1.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

====================================================================================================
 Package                     Arch            Version                 Repository                Size
====================================================================================================
Installing:
 mongodb-org                 x86_64          3.6.4-1.el7             mongodb-org-3.6          5.8 k
Installing for dependencies:
 mongodb-org-mongos          x86_64          3.6.4-1.el7             mongodb-org-3.6           12 M
 mongodb-org-server          x86_64          3.6.4-1.el7             mongodb-org-3.6           20 M
 mongodb-org-shell           x86_64          3.6.4-1.el7             mongodb-org-3.6           12 M
 mongodb-org-tools           x86_64          3.6.4-1.el7             mongodb-org-3.6           46 M

Transaction Summary
====================================================================================================
Install  1 Package (+4 Dependent packages)

Total download size: 90 M
Installed size: 265 M
Is this ok [y/d/N]: y
Downloading packages:
warning: /var/cache/yum/x86_64/7Server/mongodb-org-3.6/packages/mongodb-org-3.6.4-1.el7.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID 91fa4ad5: NOKEY
Public key for mongodb-org-3.6.4-1.el7.x86_64.rpm is not installed
(1/5): mongodb-org-3.6.4-1.el7.x86_64.rpm                                    | 5.8 kB  00:00:01     
(2/5): mongodb-org-mongos-3.6.4-1.el7.x86_64.rpm                             |  12 MB  00:00:32     
(3/5): mongodb-org-server-3.6.4-1.el7.x86_64.rpm                             |  20 MB  00:00:57     
(4/5): mongodb-org-shell-3.6.4-1.el7.x86_64.rpm                              |  12 MB  00:00:30     
(5/5): mongodb-org-tools-3.6.4-1.el7.x86_64.rpm                              |  46 MB  00:01:06     
----------------------------------------------------------------------------------------------------
Total                                                               740 kB/s |  90 MB  00:02:04     
Retrieving key from https://www.mongodb.org/static/pgp/server-3.6.asc
Importing GPG key 0x91FA4AD5:
 Userid     : "MongoDB 3.6 Release Signing Key "
 Fingerprint: 2930 adae 8caf 5059 ee73 bb4b 5871 2a22 91fa 4ad5
 From       : https://www.mongodb.org/static/pgp/server-3.6.asc
Is this ok [y/N]: y
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : mongodb-org-shell-3.6.4-1.el7.x86_64                                             1/5 
  Installing : mongodb-org-tools-3.6.4-1.el7.x86_64                                             2/5 
  Installing : mongodb-org-mongos-3.6.4-1.el7.x86_64                                            3/5 
  Installing : mongodb-org-server-3.6.4-1.el7.x86_64                                            4/5 
Created symlink from /etc/systemd/system/multi-user.target.wants/mongod.service to /usr/lib/systemd/system/mongod.service.
  Installing : mongodb-org-3.6.4-1.el7.x86_64                                                   5/5 
  Verifying  : mongodb-org-3.6.4-1.el7.x86_64                                                   1/5 
  Verifying  : mongodb-org-server-3.6.4-1.el7.x86_64                                            2/5 
  Verifying  : mongodb-org-mongos-3.6.4-1.el7.x86_64                                            3/5 
  Verifying  : mongodb-org-tools-3.6.4-1.el7.x86_64                                             4/5 
  Verifying  : mongodb-org-shell-3.6.4-1.el7.x86_64                                             5/5 

Installed:
  mongodb-org.x86_64 0:3.6.4-1.el7                                                                  

Dependency Installed:
  mongodb-org-mongos.x86_64 0:3.6.4-1.el7          mongodb-org-server.x86_64 0:3.6.4-1.el7         
  mongodb-org-shell.x86_64 0:3.6.4-1.el7           mongodb-org-tools.x86_64 0:3.6.4-1.el7          

Complete!
[root@mongo0 ~]#

Network Manager

As we do not need Network Manager we will disable it entirely.

[root@mongo0 ~]# systemctl list-unit-files | grep -i network
dbus-org.freedesktop.NetworkManager.service   enabled 
NetworkManager-dispatcher.service             enabled 
NetworkManager-wait-online.service            enabled 
NetworkManager.service                        enabled 
network-online.target                         static  
network-pre.target                            static  
network.target                                static 

[root@mongo0 ~]# systemctl stop NetworkManager

[root@mongo0 ~]# systemctl disable NetworkManager 
Removed symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service.
Removed symlink /etc/systemd/system/dbus-org.freedesktop.NetworkManager.service.
Removed symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service.

[root@mongo0 ~]# systemctl stop NetworkManager-wait-online

[root@mongo0 ~]# systemctl disable NetworkManager-wait-online 
Removed symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service.

[root@mongo0 ~]# systemctl stop NetworkManager-dispatcher

[root@mongo0 ~]# systemctl disable NetworkManager-dispatcher

[root@mongo0 ~]# systemctl list-unit-files | grep -i network
NetworkManager-dispatcher.service             disabled
NetworkManager-wait-online.service            disabled
NetworkManager.service                        disabled
network-online.target                         static  
network-pre.target                            static  
network.target                                static

SELinux

We do not need SELinux either.

[root@mongo0 ~]# sestatus 
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   enforcing
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Max kernel policy version:      29

[root@mongo0 ~]# setenforce 0

[root@mongo0 ~]# sestatus 
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   permissive
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Max kernel policy version:      29

[root@mongo0 ~]# cat /etc/sysconfig/selinux 

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=enforcing
# SELINUXTYPE= can take one of three two values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected. 
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted 


[root@mongo0 ~]# sed -i -e 's@^SELINUX=enforcing$@SELINUX=disabled@g' /etc/selinux/config

[root@mongo0 ~]# cat /etc/sysconfig/selinux 

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of three two values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected. 
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted

Firewall

… and iptables to the disabled state.

[root@mongo0 ~]# systemctl stop firewalld

[root@mongo0 ~]# systemctl disable firewalld
Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.

[root@mongo0 ~]# iptables -nvL
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Old Deterministic Naming Scheme

With the introduction of RHEL 7.x (as Oracle Linux and CentOS systems are just ‘dumb’ clones) the old network interfaces naming scheme eth0, eth1 is gone. In 7.x the interfaces will now be named in a “Predictable Interface Names” which makes these names very unpredictable … fortunately there is a way to move back to old ‘unpredictable’ RHEL 6.x naming scheme with net.ifnames=0 biosdevname=0 options in the GRUB_CMDLINE_LINUX variable in the /etc/default/grub file. Lets do it then.

[root@mongo0 ~]# cat /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="rhgb quiet"
GRUB_DISABLE_RECOVERY="true"

[root@mongo0 ~]# cp /etc/default/grub /etc/default/grub.ORG

[root@mongo0 ~]# vi /etc/default/grub

[root@mongo0 ~]# diff -u /etc/default/grub.ORG /etc/default/grub
--- /etc/default/grub.ORG       2018-04-24 10:56:03.094000000 +0200
+++ /etc/default/grub   2018-04-24 10:56:13.668000000 +0200
@@ -3,5 +3,5 @@
 GRUB_DEFAULT=saved
 GRUB_DISABLE_SUBMENU=true
 GRUB_TERMINAL_OUTPUT="console"
-GRUB_CMDLINE_LINUX="rhgb quiet"
+GRUB_CMDLINE_LINUX="rhgb quiet net.ifnames=0 biosdevname=0"
 GRUB_DISABLE_RECOVERY="true"

[root@mongo0 ~]# grub2-mkconfig
Generating grub configuration file ...
#
# DO NOT EDIT THIS FILE
#
# It is automatically generated by grub2-mkconfig using templates
# from /etc/grub.d and settings from /etc/default/grub
#

### BEGIN /etc/grub.d/00_header ###
set pager=1

if [ -s $prefix/grubenv ]; then
  load_env
fi
if [ "${next_entry}" ] ; then
   set default="${next_entry}"
   set next_entry=
   save_env next_entry
   set boot_once=true
else
   set default="${saved_entry}"
fi

if [ x"${feature_menuentry_id}" = xy ]; then
  menuentry_id_option="--id"
else
  menuentry_id_option=""
fi

export menuentry_id_option

if [ "${prev_saved_entry}" ]; then
  set saved_entry="${prev_saved_entry}"
  save_env saved_entry
  set prev_saved_entry=
  save_env prev_saved_entry
  set boot_once=true
fi

function savedefault {
  if [ -z "${boot_once}" ]; then
    saved_entry="${chosen}"
    save_env saved_entry
  fi
}

function load_video {
  if [ x$feature_all_video_module = xy ]; then
    insmod all_video
  else
    insmod efi_gop
    insmod efi_uga
    insmod ieee1275_fb
    insmod vbe
    insmod vga
    insmod video_bochs
    insmod video_cirrus
  fi
}

terminal_output console
if [ x$feature_timeout_style = xy ] ; then
  set timeout_style=menu
  set timeout=5
# Fallback normal timeout code in case the timeout_style feature is
# unavailable.
else
  set timeout=5
fi
### END /etc/grub.d/00_header ###

### BEGIN /etc/grub.d/00_tuned ###
set tuned_params=""
set tuned_initrd=""
### END /etc/grub.d/00_tuned ###

### BEGIN /etc/grub.d/01_users ###
if [ -f ${prefix}/user.cfg ]; then
  source ${prefix}/user.cfg
  if [ -n "${GRUB2_PASSWORD}" ]; then
    set superusers="root"
    export superusers
    password_pbkdf2 root ${GRUB2_PASSWORD}
  fi
fi
### END /etc/grub.d/01_users ###

### BEGIN /etc/grub.d/10_linux ###
Found linux image: /boot/vmlinuz-4.1.12-124.14.1.el7uek.x86_64
Found initrd image: /boot/initramfs-4.1.12-124.14.1.el7uek.x86_64.img
menuentry 'Oracle Linux Server (4.1.12-124.14.1.el7uek.x86_64 with Unbreakable Enterprise Kernel) 7.5' --class oracle --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-4.1.12-124.14.1.el7uek.x86_64-advanced-621c9873-8ad4-4a24-9a2f-14763bb1b77f' {
        load_video
        set gfxpayload=keep
        insmod gzio
        insmod part_msdos
        insmod xfs
        set root='hd0,msdos1'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1 --hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1 --hint='hd0,msdos1'  621c9873-8ad4-4a24-9a2f-14763bb1b77f
        else
          search --no-floppy --fs-uuid --set=root 621c9873-8ad4-4a24-9a2f-14763bb1b77f
        fi
        linux16 /boot/vmlinuz-4.1.12-124.14.1.el7uek.x86_64 root=UUID=621c9873-8ad4-4a24-9a2f-14763bb1b77f ro rhgb quiet net.ifnames=0 biosdevname=0 
        initrd16 /boot/initramfs-4.1.12-124.14.1.el7uek.x86_64.img
}
Found linux image: /boot/vmlinuz-3.10.0-862.el7.x86_64
Found initrd image: /boot/initramfs-3.10.0-862.el7.x86_64.img
menuentry 'Oracle Linux Server (3.10.0-862.el7.x86_64 with Linux) 7.5' --class oracle --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-3.10.0-862.el7.x86_64-advanced-621c9873-8ad4-4a24-9a2f-14763bb1b77f' {
        load_video
        set gfxpayload=keep
        insmod gzio
        insmod part_msdos
        insmod xfs
        set root='hd0,msdos1'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1 --hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1 --hint='hd0,msdos1'  621c9873-8ad4-4a24-9a2f-14763bb1b77f
        else
          search --no-floppy --fs-uuid --set=root 621c9873-8ad4-4a24-9a2f-14763bb1b77f
        fi
        linux16 /boot/vmlinuz-3.10.0-862.el7.x86_64 root=UUID=621c9873-8ad4-4a24-9a2f-14763bb1b77f ro rhgb quiet net.ifnames=0 biosdevname=0 
        initrd16 /boot/initramfs-3.10.0-862.el7.x86_64.img
}
Found linux image: /boot/vmlinuz-0-rescue-141943d2370a45fe9230ea2413f80d41
Found initrd image: /boot/initramfs-0-rescue-141943d2370a45fe9230ea2413f80d41.img
menuentry 'Oracle Linux Server (0-rescue-141943d2370a45fe9230ea2413f80d41 with Linux) 7.5' --class oracle --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-0-rescue-141943d2370a45fe9230ea2413f80d41-advanced-621c9873-8ad4-4a24-9a2f-14763bb1b77f' {
        load_video
        insmod gzio
        insmod part_msdos
        insmod xfs
        set root='hd0,msdos1'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1 --hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1 --hint='hd0,msdos1'  621c9873-8ad4-4a24-9a2f-14763bb1b77f
        else
          search --no-floppy --fs-uuid --set=root 621c9873-8ad4-4a24-9a2f-14763bb1b77f
        fi
        linux16 /boot/vmlinuz-0-rescue-141943d2370a45fe9230ea2413f80d41 root=UUID=621c9873-8ad4-4a24-9a2f-14763bb1b77f ro rhgb quiet net.ifnames=0 biosdevname=0 
        initrd16 /boot/initramfs-0-rescue-141943d2370a45fe9230ea2413f80d41.img
}

### END /etc/grub.d/10_linux ###

### BEGIN /etc/grub.d/20_linux_xen ###

### END /etc/grub.d/20_linux_xen ###

### BEGIN /etc/grub.d/20_ppc_terminfo ###
### END /etc/grub.d/20_ppc_terminfo ###

### BEGIN /etc/grub.d/30_os-prober ###
### END /etc/grub.d/30_os-prober ###

### BEGIN /etc/grub.d/40_custom ###
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.
### END /etc/grub.d/40_custom ###

### BEGIN /etc/grub.d/41_custom ###
if [ -f  ${config_directory}/custom.cfg ]; then
  source ${config_directory}/custom.cfg
elif [ -z "${config_directory}" -a -f  $prefix/custom.cfg ]; then
  source $prefix/custom.cfg;
fi
### END /etc/grub.d/41_custom ###
done
[root@mongo0 ~]# 

[root@mongo0 ~]# grub2-mkconfig -o /boot/grub2/grub.cfg
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.1.12-124.14.1.el7uek.x86_64
Found initrd image: /boot/initramfs-4.1.12-124.14.1.el7uek.x86_64.img
Found linux image: /boot/vmlinuz-3.10.0-862.el7.x86_64
Found initrd image: /boot/initramfs-3.10.0-862.el7.x86_64.img
Found linux image: /boot/vmlinuz-0-rescue-141943d2370a45fe9230ea2413f80d41
Found initrd image: /boot/initramfs-0-rescue-141943d2370a45fe9230ea2413f80d41.img
done
[root@mongo0 ~]#

Network

As Anaconda installer got the award for the worst installer [Citation Needed] we will now have to clean up the installer generated configuration files. The interface is still enp0s3 instead eth0 because we haven’t done reboot yet.

Below are files generated by Anaconda installer.

[root@mongo0 ~]# ip li 
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp0s3:  mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether 08:00:27:dd:93:cf brd ff:ff:ff:ff:ff:ff

[root@mongo0 ~]# cat /etc/sysconfig/network
# Created by anaconda

[root@mongo0 ~]# cat /etc/sysconfig/network-scripts/ifcfg-enp0s3
TYPE="Ethernet"
PROXY_METHOD="none"
BROWSER_ONLY="no"
BOOTPROTO="none"
DEFROUTE="yes"
IPV4_FAILURE_FATAL="no"
IPV6INIT="yes"
IPV6_AUTOCONF="yes"
IPV6_DEFROUTE="yes"
IPV6_FAILURE_FATAL="no"
IPV6_ADDR_GEN_MODE="stable-privacy"
NAME="enp0s3"
UUID="3eba4d78-3392-49ed-807e-70fe6bc134b7"
DEVICE="enp0s3"
ONBOOT="yes"
IPADDR="10.0.10.10"
PREFIX="24"
IPV6_PRIVACY="no"
GATEWAY="10.0.10.1"
DNS1="1.1.1.1"

Lets make some cleanup and ‘migration’ to the old ethX naming scheme.

[root@mongo0 ~]# mv /etc/sysconfig/network-scripts/ifcfg-enp0s3 /etc/sysconfig/network-scripts/ifcfg-eth0

[root@mongo0 ~]# cat !$ | tr -d \" > ASD

[root@mongo0 ~]# mv -f !$ /etc/sysconfig/network-scripts/ifcfg-eth0

[root@mongo0 ~]# grep GATEWAY /etc/sysconfig/network-scripts/ifcfg-enp0s3 > /etc/sysconfig/network

[root@mongo0 ~]# cat /etc/sysconfig/network
GATEWAY=10.0.10.1

[root@mongo0 ~]# vi /etc/sysconfig/network-scripts/ifcfg-eth0

[root@mongo0 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
TYPE=Ethernet
BOOTPROTO=none
IPV6INIT=no
NAME=eth0
DEVICE=eth0
ONBOOT=yes
IPADDR=10.0.10.10
PREFIX=24

[root@mongo0 ~]# echo nameserver 1.1.1.1 > /etc/resolv.conf 

[root@mongo0 ~]# diff -u /root/ifcfg-eth0.ORG /etc/sysconfig/network-scripts/ifcfg-eth0
--- /root/ifcfg-eth0.ORG        2018-04-24 11:00:17.493000000 +0200
+++ /etc/sysconfig/network-scripts/ifcfg-eth0   2018-04-24 11:00:57.914000000 +0200
@@ -1,20 +1,8 @@
 TYPE=Ethernet
-PROXY_METHOD=none
-BROWSER_ONLY=no
 BOOTPROTO=none
-DEFROUTE=yes
-IPV4_FAILURE_FATAL=no
-IPV6INIT=yes
-IPV6_AUTOCONF=yes
-IPV6_DEFROUTE=yes
-IPV6_FAILURE_FATAL=no
-IPV6_ADDR_GEN_MODE=stable-privacy
-NAME=enp0s3
-UUID=3eba4d78-3392-49ed-807e-70fe6bc134b7
-DEVICE=enp0s3
+IPV6INIT=no
+NAME=eth0
+DEVICE=eth0
 ONBOOT=yes
 IPADDR=10.0.10.10
 PREFIX=24
-IPV6_PRIVACY=no
-GATEWAY=10.0.10.1
-DNS1=1.1.1.1

We will now reboot the system to get the eth0 interface.

[root@mongo0 ~]# reboot

After the reboot the interface is plain old eth0 device.

[root@mongo0 ~]# ip li
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0:  mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether 08:00:27:dd:93:cf brd ff:ff:ff:ff:ff:ff

Vim

If You will work with PuTTY with these hosts this may (or not) make your work more pleasant.

[root@mongo0 ~]# echo 'set mouse-=a' >> /root/.vimrc

Filesystem

We will disable atime for performance reasons in the /etc/fstab file.

[root@mongo0 ~]# cat /etc/fstab 

#
# /etc/fstab
# Created by anaconda on Tue Apr 24 00:23:14 2018
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=621c9873-8ad4-4a24-9a2f-14763bb1b77f /                       xfs     defaults        0 0

[root@mongo0 ~]# sed -i -e s@defaults@rw,noatime,nodiratime@g /etc/fstab

[root@mongo0 ~]# cat /etc/fstab 

#
# /etc/fstab
# Created by anaconda on Tue Apr 24 00:23:14 2018
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=621c9873-8ad4-4a24-9a2f-14763bb1b77f /                       xfs     rw,noatime,nodiratime        0 0

Lets see what output will give us the mount command on a modern Linux system with just one single / filesystem …

[root@mongo0 ~]# mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,size=746928k,nr_inodes=186732,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
configfs on /sys/kernel/config type configfs (rw,relatime)
/dev/sda1 on / type xfs (rw,noatime,nodiratime,attr2,inode64,noquota)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=32,pgrp=1,timeout=0,minproto=5,maxproto=5,direct)
mqueue on /dev/mqueue type mqueue (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=153040k,mode=700)

Horrible mess.

We can limit that output to something readable.

[root@mongo0 ~]# mount -t xfs
/dev/sda1 on / type xfs (rw,noatime,nodiratime,attr2,inode64,noquota)

[root@mongo0 ~]# mount | grep ^/
/dev/sda1 on / type xfs (rw,noatime,nodiratime,attr2,inode64,noquota)

Better.

Time Daemon

As with every cluster we will have to install and configure the time daemon, ntp for example.

First installation …

[root@mongo0 ~]# yum install ntp
Loaded plugins: ulninfo
mongodb-org-3.6                                                                         | 2.5 kB  00:00:00     
ol7_UEKR4                                                                               | 1.2 kB  00:00:00     
ol7_latest                                                                              | 1.4 kB  00:00:00     
Resolving Dependencies
--> Running transaction check
---> Package ntp.x86_64 0:4.2.6p5-28.0.1.el7 will be installed
--> Processing Dependency: ntpdate = 4.2.6p5-28.0.1.el7 for package: ntp-4.2.6p5-28.0.1.el7.x86_64
--> Processing Dependency: libopts.so.25()(64bit) for package: ntp-4.2.6p5-28.0.1.el7.x86_64
--> Running transaction check
---> Package autogen-libopts.x86_64 0:5.18-5.el7 will be installed
---> Package ntpdate.x86_64 0:4.2.6p5-28.0.1.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

===========================================================================================================
 Package                       Arch                Version                      Repository            Size
===========================================================================================================
Installing:
 ntp                           x86_64              4.2.6p5-28.0.1.el7           ol7_latest           548 k
Installing for dependencies:
 autogen-libopts               x86_64              5.18-5.el7                   ol7_latest            65 k
 ntpdate                       x86_64              4.2.6p5-28.0.1.el7           ol7_latest            85 k

Transaction Summary
===========================================================================================================
Install  1 Package (+2 Dependent packages)

Total download size: 698 k
Installed size: 1.6 M
Is this ok [y/d/N]: y
Downloading packages:
(1/3): autogen-libopts-5.18-5.el7.x86_64.rpm                                        |  65 kB  00:00:03     
(2/3): ntpdate-4.2.6p5-28.0.1.el7.x86_64.rpm                                        |  85 kB  00:00:00     
(3/3): ntp-4.2.6p5-28.0.1.el7.x86_64.rpm                                            | 548 kB  00:00:05     
-----------------------------------------------------------------------------------------------------------
Total                                                                      136 kB/s | 698 kB  00:00:05     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Warning: RPMDB altered outside of yum.                               
  Installing : autogen-libopts-5.18-5.el7.x86_64                                                        1/3 
  Installing : ntpdate-4.2.6p5-28.0.1.el7.x86_64                                                        2/3 
  Installing : ntp-4.2.6p5-28.0.1.el7.x86_64                                                            3/3 
  Verifying  : ntpdate-4.2.6p5-28.0.1.el7.x86_64                                                        1/3 
  Verifying  : autogen-libopts-5.18-5.el7.x86_64                                                        2/3 
  Verifying  : ntp-4.2.6p5-28.0.1.el7.x86_64                                                            3/3 

Installed:
  ntp.x86_64 0:4.2.6p5-28.0.1.el7                                                                                                   

Dependency Installed:
  autogen-libopts.x86_64 0:5.18-5.el7                              ntpdate.x86_64 0:4.2.6p5-28.0.1.el7                             

Complete!
[root@mongo0 ~]#

… and configuration.

[root@mongo0 ~]# cat /etc/sysconfig/ntpd
# Command line options for ntpd
OPTIONS="-g"

[root@mongo0 ~]# cp /etc/sysconfig/ntpd /etc/sysconfig/ntpd.ORG

[root@mongo0 ~]# vi /etc/sysconfig/ntpd

[root@mongo0 ~]# cat /etc/sysconfig/ntpd
# Command line options for ntpd
OPTIONS="-g -x"

[root@mongo0 ~]# diff -u /etc/sysconfig/ntpd.ORG /etc/sysconfig/ntpd
--- /etc/sysconfig/ntpd.ORG     2018-04-24 15:22:46.215788131 +0200
+++ /etc/sysconfig/ntpd 2018-04-24 15:22:31.464368114 +0200
@@ -1,2 +1,2 @@
 # Command line options for ntpd
-OPTIONS="-g"
+OPTIONS="-g -x"
[root@mongo0 ~]# 

[root@mongo0 ~]# systemctl start ntpd

[root@mongo0 ~]# systemctl enable ntpd
Created symlink from /etc/systemd/system/multi-user.target.wants/ntpd.service to /usr/lib/systemd/system/ntpd.service.

[root@mongo0 ~]# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
+jamesl.tk       130.149.17.8     2 u    3   64    1   99.361  -66.467  38.578
+sunsite.icm.edu 194.146.251.100  2 u    2   64    1   79.675  -55.486  12.338
+maggo.info      124.216.164.14   2 u    3   64    1   84.604  -65.630  14.804
*91-211-101-141. 5.226.98.186     2 u    2   64    1   74.071  -62.252  14.619
[root@mongo0 ~]#

Attack of the Clones

As our mongo0 machine install is finished we can now power it off and clone it into the remainng mongo1/mongo2/mongo3/mongo4 nodes.

SSH

Lets setup the keys to not have to type password everytime we want to do anything.

host % ssh-copy-id -i ~/.ssh/id_rsa.pub -p 2200 root@localhost
Password for root@mongo0:

host % ssh -p 2200 root@localhost
[root@mongo0 ~]#

Cluster SSH

For the convenience you may wish to use Cluster SSH to connect to all nodes for the tasks that are the same on all nodes.

Here is the Cluster SSH cssh command used to connect to our MongoDB cluster.

host % cssh \
  root@localhost:2200 \
  root@localhost:2201 \
  root@localhost:2202 \
  root@localhost:2203 \
  root@localhost:2204 \

… or like that.

host % cssh root@localhost:220{0,1,2,3,4}

… and here is how it looks like.

clusterssh-mongodb
If there are taks to be made only on DATA nodes you may connect only to 4 nodes with Cluster SSH of course.

Environment

As we have our clones ready lets start them.

host % for I in 0 1 2 3 4; do VBoxManage startvm mongo${I} --type headless; done
Waiting for VM "mongo0" to power on...
VM "mongo0" has been successfully started.
Waiting for VM "mongo1" to power on...
VM "mongo1" has been successfully started.
Waiting for VM "mongo2" to power on...
VM "mongo2" has been successfully started.
Waiting for VM "mongo3" to power on...
VM "mongo3" has been successfully started.
Waiting for VM "mongo4" to power on...
VM "mongo4" has been successfully started.

As we have our nodes installed and started lets check the connectivity between them.

[root@mongo0 ~]# awk '/mongo/ {print $1}' /etc/hosts | xargs -n1 ping -c 1 -t 3 | grep loss
1 packets transmitted, 1 received, 0% packet loss, time 0ms
1 packets transmitted, 1 received, 0% packet loss, time 0ms
1 packets transmitted, 1 received, 0% packet loss, time 0ms
1 packets transmitted, 1 received, 0% packet loss, time 0ms
1 packets transmitted, 1 received, 0% packet loss, time 0ms

Lets verify that MongoDB is installed.

host % for I in 0 1 2 3 4; do ssh -p 220${I} root@localhost which mongod; done
/usr/bin/mongod
/usr/bin/mongod
/usr/bin/mongod
/usr/bin/mongod
/usr/bin/mongod

Now we will configure /etc/hosts file.

host % for I in 0 1 2 3 4; do ssh -p 220${I} root@localhost 'cat >> /etc/hosts << __EOF
10.0.10.10 mongo0
10.0.10.11 mongo1
10.0.10.12 mongo2
10.0.10.13 mongo3
10.0.10.14 mongo4
__EOF'
done

Lets verify it.

host % for I in 0 1 2 3 4; do ssh -p 220${I} root@localhost "grep mongo${I} /etc/hosts"; done
10.0.10.10 mongo0
10.0.10.11 mongo1
10.0.10.12 mongo2
10.0.10.13 mongo3
10.0.10.14 mongo4

MongoDB

It is now (at last) time to configure MongoDB, lets start with the configuration files.

Configuration Files

Create the config files for the MongoDB data nodes.

host % for I in 0 1 2 3; do ssh -p 220${I} root@localhost "cat > /etc/mongod.conf << __EOF
systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log

storage:
  dbPath: /var/lib/mongo
  journal.enabled: true  # ONLY DIFFERENCE BETWEEN DATA AND ARBITER NODE #

processManagement:
  fork: true
  pidFilePath: /var/run/mongodb/mongod.pid
  timeZoneInfo: /usr/share/zoneinfo

net:
  port: 27017
  bindIp: localhost,10.0.10.1${I}

replication:
   replSetName: \"replica0\"

__EOF"
done

Create the config file for the MongoDB arbiter node.

host % for I in 4; do ssh -p 220${I} root@localhost "cat > /etc/mongod.conf << __EOF
systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log

storage:
  dbPath: /var/lib/mongo
  journal.enabled: false # ONLY DIFFERENCE BETWEEN DATA AND ARBITER NODE #

processManagement:
  fork: true
  pidFilePath: /var/run/mongodb/mongod.pid
  timeZoneInfo: /usr/share/zoneinfo

net:
  port: 27017
  bindIp: localhost,10.0.10.1${I}

replication:
   replSetName: \"replica0\"

__EOF"
done

Lets verify these configuration files.

host % for I in 0 1 2 3 4; do ssh -p 220${I} root@localhost grep -H bindIp /etc/mongod.conf; done  
/etc/mongod.conf:  bindIp: localhost,10.0.10.10
/etc/mongod.conf:  bindIp: localhost,10.0.10.11
/etc/mongod.conf:  bindIp: localhost,10.0.10.12
/etc/mongod.conf:  bindIp: localhost,10.0.10.13
/etc/mongod.conf:  bindIp: localhost,10.0.10.14
host % for I in 0 1 2 3 4; do ssh -p 220${I} root@localhost grep -H /var /etc/mongod.conf; echo; done | column -t 
/etc/mongod.conf:  path:         /var/log/mongodb/mongod.log
/etc/mongod.conf:  dbPath:       /var/lib/mongo
/etc/mongod.conf:  pidFilePath:  /var/run/mongodb/mongod.pid

/etc/mongod.conf:  path:         /var/log/mongodb/mongod.log
/etc/mongod.conf:  dbPath:       /var/lib/mongo
/etc/mongod.conf:  pidFilePath:  /var/run/mongodb/mongod.pid

/etc/mongod.conf:  path:         /var/log/mongodb/mongod.log
/etc/mongod.conf:  dbPath:       /var/lib/mongo
/etc/mongod.conf:  pidFilePath:  /var/run/mongodb/mongod.pid

/etc/mongod.conf:  path:         /var/log/mongodb/mongod.log
/etc/mongod.conf:  dbPath:       /var/lib/mongo
/etc/mongod.conf:  pidFilePath:  /var/run/mongodb/mongod.pid

/etc/mongod.conf:  path:         /var/log/mongodb/mongod.log
/etc/mongod.conf:  dbPath:       /var/lib/mongo
/etc/mongod.conf:  pidFilePath:  /var/run/mongodb/mongod.pid
host % for I in 0 1 2 3 4; do ssh -p 220${I} root@localhost grep -H DIFFERENCE /etc/mongod.conf; done 
/etc/mongod.conf:  journal.enabled: true  # ONLY DIFFERENCE BETWEEN DATA AND ARBITER NODE #
/etc/mongod.conf:  journal.enabled: true  # ONLY DIFFERENCE BETWEEN DATA AND ARBITER NODE #
/etc/mongod.conf:  journal.enabled: true  # ONLY DIFFERENCE BETWEEN DATA AND ARBITER NODE #
/etc/mongod.conf:  journal.enabled: true  # ONLY DIFFERENCE BETWEEN DATA AND ARBITER NODE #
/etc/mongod.conf:  journal.enabled: false # ONLY DIFFERENCE BETWEEN DATA AND ARBITER NODE #

Lets start the MongoDB nodes, if MongoDB is already running with the default config (not ours) then restart it.

host % for I in 0 1 2 3 4; do ssh -p 220${I} root@localhost service mongod stop; done
Redirecting to /bin/systemctl stop mongod.service
Redirecting to /bin/systemctl stop mongod.service
Redirecting to /bin/systemctl stop mongod.service
Redirecting to /bin/systemctl stop mongod.service
Redirecting to /bin/systemctl stop mongod.service
host % for I in 0 1 2 3 4; do ssh -p 220${I} root@localhost service mongod start; done     
Redirecting to /bin/systemctl start mongod.service
Redirecting to /bin/systemctl start mongod.service
Redirecting to /bin/systemctl start mongod.service
Redirecting to /bin/systemctl start mongod.service
Redirecting to /bin/systemctl start mongod.service

Lets verify that MongoDB is running on our nodes with the new onfiguration.

host % for I in 0 1 2 3 4; do ssh -p 220${I} root@localhost pgrep mongod; done                             
735
744
738
736
748
host % for I in 0 1 2 3 4; do ssh -p 220${I} root@localhost ss -ln4; echo; done
Netid  State      Recv-Q Send-Q Local Address:Port               Peer Address:Port              
udp    UNCONN     0      0      10.0.10.10:123                   *:*                  
udp    UNCONN     0      0      127.0.0.1:123                   *:*                  
udp    UNCONN     0      0         *:123                   *:*                  
tcp    LISTEN     0      128    127.0.0.1:27017                 *:*                  
tcp    LISTEN     0      128       *:22                    *:*                  
tcp    LISTEN     0      100    127.0.0.1:25                    *:*                  

Netid  State      Recv-Q Send-Q Local Address:Port               Peer Address:Port              
udp    UNCONN     0      0      10.0.10.11:123                   *:*                  
udp    UNCONN     0      0      127.0.0.1:123                   *:*                  
udp    UNCONN     0      0         *:123                   *:*                  
tcp    LISTEN     0      128    127.0.0.1:27017                 *:*                  
tcp    LISTEN     0      128       *:22                    *:*                  
tcp    LISTEN     0      100    127.0.0.1:25                    *:*                  

Netid  State      Recv-Q Send-Q Local Address:Port               Peer Address:Port              
udp    UNCONN     0      0      10.0.10.12:123                   *:*                  
udp    UNCONN     0      0      127.0.0.1:123                   *:*                  
udp    UNCONN     0      0         *:123                   *:*                  
tcp    LISTEN     0      128    127.0.0.1:27017                 *:*                  
tcp    LISTEN     0      128       *:22                    *:*                  
tcp    LISTEN     0      100    127.0.0.1:25                    *:*                  

Netid  State      Recv-Q Send-Q Local Address:Port               Peer Address:Port              
udp    UNCONN     0      0      10.0.10.13:123                   *:*                  
udp    UNCONN     0      0      127.0.0.1:123                   *:*                  
udp    UNCONN     0      0         *:123                   *:*                  
tcp    LISTEN     0      128    127.0.0.1:27017                 *:*                  
tcp    LISTEN     0      128       *:22                    *:*                  
tcp    LISTEN     0      100    127.0.0.1:25                    *:*                  

Netid  State      Recv-Q Send-Q Local Address:Port               Peer Address:Port              
udp    UNCONN     0      0      10.0.10.14:123                   *:*                  
udp    UNCONN     0      0      127.0.0.1:123                   *:*                  
udp    UNCONN     0      0         *:123                   *:*                  
tcp    LISTEN     0      128    127.0.0.1:27017                 *:*                  
tcp    LISTEN     0      128       *:22                    *:*                  
tcp    LISTEN     0      100    127.0.0.1:25                    *:*

Replica Set

We may now configure our MongoDB Replica Set Cluster.

We will use replica0 name for the replica set.

We will paste these instructions into the MongoDB prompt on the first node (mongo0) to configure replica set.

use admin
rs.initiate(
  {
    _id : "replica0",
    members: [
      { _id: 0, host: "mongo0:27017" },
      { _id: 1, host: "mongo1:27017" },
      { _id: 2, host: "mongo2:27017" },
      { _id: 3, host: "mongo3:27017" }
    ]
  }
)

Lets do it then. As You will paste it you will see that prompt changed to replica0:SECONDARY> string. Hit [ENTER] once a second and after about 15-20 seconds it should change to replica0:PRIMARY> as this will be currently the node role in the cluster after forming it.

% ssh root@localhost -p 2200
Last login: Tue Apr 24 14:39:06 2018 from 10.0.10.2
[root@mongo0 ~]# mongo
MongoDB shell version v3.6.4
connecting to: mongodb://127.0.0.1:27017
MongoDB server version: 3.6.4
Server has startup warnings: 
2018-04-24T14:39:33.161+0200 I CONTROL  [initandlisten] 
2018-04-24T14:39:33.162+0200 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2018-04-24T14:39:33.162+0200 I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2018-04-24T14:39:33.162+0200 I CONTROL  [initandlisten] 
> use admin
switched to db admin
> rs.initiate(
...   {
...     _id : "replica0",
...     members: [
...       { _id: 0, host: "mongo0:27017" },
...       { _id: 1, host: "mongo1:27017" },
...       { _id: 2, host: "mongo2:27017" },
...       { _id: 3, host: "mongo3:27017" }
...     ]
...   }
... )
{
        "ok" : 1,
        "operationTime" : Timestamp(1524574334, 1),
        "$clusterTime" : {
                "clusterTime" : Timestamp(1524574334, 1),
                "signature" : {
                        "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
                        "keyId" : NumberLong(0)
                }
        }
}
replica0:SECONDARY> 
replica0:SECONDARY> 
replica0:SECONDARY> 
replica0:SECONDARY> 
replica0:SECONDARY> 
replica0:SECONDARY> 
replica0:SECONDARY> 
replica0:SECONDARY> 
replica0:SECONDARY> 
replica0:PRIMARY> 
replica0:PRIMARY>

We will not create admin (less powerful) and root (as the name suggests can do anything) users on our new MongoDB cluster.

We will paste these instructions into the MongoDB prompt on the PRIMARY node (currnetly mongo0) to add users.

use admin
db.createUser(
  {
    user: "admin",
    pwd: "ADMIN-PASSWORD",
    roles: [ { role: "userAdminAnyDatabase", db: "admin" } ]
  }
)
use admin
db.createUser(
  {
    user: "root",
    pwd: "ROOT-PASSWORD",
    roles:["root"]
  }
)

Lets do it then.

replica0:PRIMARY> use admin
switched to db admin
replica0:PRIMARY> db.createUser(
...   {
...     user: "admin",
...     pwd: "ADMIN-PASSWORD",
...     roles: [ { role: "userAdminAnyDatabase", db: "admin" } ]
...   }
... )
Successfully added user: {
        "user" : "admin",
        "roles" : [
                {
                        "role" : "userAdminAnyDatabase",
                        "db" : "admin"
                }
        ]
}
replica0:PRIMARY>
replica0:PRIMARY> use admin
switched to db admin
replica0:PRIMARY> db.createUser(
...   {
...     user: "root",
...     pwd: "ROOT-PASSWORD",
...     roles:["root"]
...   }
... )
Successfully added user: { "user" : "root", "roles" : [ "root" ] }
replica0:PRIMARY>

We can now exit from the MongoDB prompt.

replica0:PRIMARY> exit
[root@mongo0 ~]#

We will not stop the MongoDB services and enable authorization and also configure shared keyfile.

host % for I in 0 1 2 3 4; do ssh -p 220${I} root@localhost service mongod stop; done   
Redirecting to /bin/systemctl stop mongod.service
Redirecting to /bin/systemctl stop mongod.service
Redirecting to /bin/systemctl stop mongod.service
Redirecting to /bin/systemctl stop mongod.service
Redirecting to /bin/systemctl stop mongod.service

Lets add needed configuration files settings.

host % for I in 0 1 2 3 4; do ssh -p 220${I} root@localhost "cat >> /etc/mongod.conf << __EOF
security:
  authorization: enabled
  keyFile: /etc/mongod.conf.key

__EOF"
done

Now lets generate a new key …

host % dd  /dev/null | sha256
66700abfea54b9f07e9767acd912f4ab17f9153fa0718984fe3b0c4fe2116baf

… and put it into the nodes as /etc/mongod.conf.key file.

host % for I in 0 1 2 3 4; do ssh -p 220${I} root@localhost 'echo 66700abfea54b9f07e9767acd912f4ab17f9153fa0718984fe3b0c4fe2116baf > /etc/mongod.conf.key'; done
host % for I in 0 1 2 3 4; do ssh -p 220${I} root@localhost chmod 600 /etc/mongod.conf.key; done
host % for I in 0 1 2 3 4; do ssh -p 220${I} root@localhost chown mongod:mongod /etc/mongod.conf.key; done

Lets verify out new key is there.

% for I in 0 1 2 3 4; do ssh -p 220${I} root@localhost cat /etc/mongod.conf.key; done
66700abfea54b9f07e9767acd912f4ab17f9153fa0718984fe3b0c4fe2116baf
66700abfea54b9f07e9767acd912f4ab17f9153fa0718984fe3b0c4fe2116baf
66700abfea54b9f07e9767acd912f4ab17f9153fa0718984fe3b0c4fe2116baf
66700abfea54b9f07e9767acd912f4ab17f9153fa0718984fe3b0c4fe2116baf
66700abfea54b9f07e9767acd912f4ab17f9153fa0718984fe3b0c4fe2116baf

We can now start the MongoDB with new settings.

host % for I in 0 1 2 3 4; do ssh -p 220${I} root@localhost service mongod start; done                    
Redirecting to /bin/systemctl start mongod.service
Redirecting to /bin/systemctl start mongod.service
Redirecting to /bin/systemctl start mongod.service
Redirecting to /bin/systemctl start mongod.service
Redirecting to /bin/systemctl start mongod.service

We can now connect to our MongoDB cluster with root user.

[root@mongo0 ~]# mongo --port 27017 -u root -p ROOT-PASSWORD --authenticationDatabase admin
MongoDB shell version v3.6.4
connecting to: mongodb://127.0.0.1:27017/
MongoDB server version: 3.6.4
replica0:PRIMARY>

Lets see how MongoDB rs.conf() function shows our configuration (yet before ARBITER node role added).

[root@mongo0 ~]# mongo --port 27017 -u root -p ROOT-PASSWORD --authenticationDatabase admin
MongoDB shell version v3.6.4
connecting to: mongodb://127.0.0.1:27017/
MongoDB server version: 3.6.4
replica0:PRIMARY> rs.conf()
{
        "_id" : "replica0",
        "version" : 1,
        "protocolVersion" : NumberLong(1),
        "members" : [
                {
                        "_id" : 0,
                        "host" : "mongo0:27017",
                        "arbiterOnly" : false,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 1,
                        "tags" : {

                        },
                        "slaveDelay" : NumberLong(0),
                        "votes" : 1
                },
                {
                        "_id" : 1,
                        "host" : "mongo1:27017",
                        "arbiterOnly" : false,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 1,
                        "tags" : {

                        },
                        "slaveDelay" : NumberLong(0),
                        "votes" : 1
                },
                {
                        "_id" : 2,
                        "host" : "mongo2:27017",
                        "arbiterOnly" : false,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 1,
                        "tags" : {

                        },
                        "slaveDelay" : NumberLong(0),
                        "votes" : 1
                },
                {
                        "_id" : 3,
                        "host" : "mongo3:27017",
                        "arbiterOnly" : false,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 1,
                        "tags" : {

                        },
                        "slaveDelay" : NumberLong(0),
                        "votes" : 1
                }
        ],
        "settings" : {
                "chainingAllowed" : true,
                "heartbeatIntervalMillis" : 2000,
                "heartbeatTimeoutSecs" : 10,
                "electionTimeoutMillis" : 10000,
                "catchUpTimeoutMillis" : -1,
                "catchUpTakeoverDelayMillis" : 30000,
                "getLastErrorModes" : {

                },
                "getLastErrorDefaults" : {
                        "w" : 1,
                        "wtimeout" : 0
                },
                "replicaSetId" : ObjectId("5adf287d597df99256d11280")
        }
}
replica0:PRIMARY>

Lets see how MongoDB rs.status() function shows our configuration (yet before ARBITER node role added).

[root@mongo0 ~]# mongo --port 27017 -u root -p ROOT-PASSWORD --authenticationDatabase admin
MongoDB shell version v3.6.4
connecting to: mongodb://127.0.0.1:27017/
MongoDB server version: 3.6.4
replica0:PRIMARY> rs.status()
{
        "set" : "replica0",
        "date" : ISODate("2018-04-24T13:14:14.653Z"),
        "myState" : 1,
        "term" : NumberLong(2),
        "heartbeatIntervalMillis" : NumberLong(2000),
        "optimes" : {
                "lastCommittedOpTime" : {
                        "ts" : Timestamp(1524575648, 1),
                        "t" : NumberLong(2)
                },
                "readConcernMajorityOpTime" : {
                        "ts" : Timestamp(1524575648, 1),
                        "t" : NumberLong(2)
                },
                "appliedOpTime" : {
                        "ts" : Timestamp(1524575648, 1),
                        "t" : NumberLong(2)
                },
                "durableOpTime" : {
                        "ts" : Timestamp(1524575648, 1),
                        "t" : NumberLong(2)
                }
        },
        "members" : [
                {
                        "_id" : 0,
                        "name" : "mongo0:27017",
                        "health" : 1,
                        "state" : 1,
                        "stateStr" : "PRIMARY",
                        "uptime" : 111,
                        "optime" : {
                                "ts" : Timestamp(1524575648, 1),
                                "t" : NumberLong(2)
                        },
                        "optimeDate" : ISODate("2018-04-24T13:14:08Z"),
                        "infoMessage" : "could not find member to sync from",
                        "electionTime" : Timestamp(1524575557, 1),
                        "electionDate" : ISODate("2018-04-24T13:12:37Z"),
                        "configVersion" : 1,
                        "self" : true
                },
                {
                        "_id" : 1,
                        "name" : "mongo1:27017",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 104,
                        "optime" : {
                                "ts" : Timestamp(1524575648, 1),
                                "t" : NumberLong(2)
                        },
                        "optimeDurable" : {
                                "ts" : Timestamp(1524575648, 1),
                                "t" : NumberLong(2)
                        },
                        "optimeDate" : ISODate("2018-04-24T13:14:08Z"),
                        "optimeDurableDate" : ISODate("2018-04-24T13:14:08Z"),
                        "lastHeartbeat" : ISODate("2018-04-24T13:14:13.546Z"),
                        "lastHeartbeatRecv" : ISODate("2018-04-24T13:14:13.892Z"),
                        "pingMs" : NumberLong(1),
                        "syncingTo" : "mongo0:27017",
                        "configVersion" : 1
                },
                {
                        "_id" : 2,
                        "name" : "mongo2:27017",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 101,
                        "optime" : {
                                "ts" : Timestamp(1524575648, 1),
                                "t" : NumberLong(2)
                        },
                        "optimeDurable" : {
                                "ts" : Timestamp(1524575648, 1),
                                "t" : NumberLong(2)
                        },
                        "optimeDate" : ISODate("2018-04-24T13:14:08Z"),
                        "optimeDurableDate" : ISODate("2018-04-24T13:14:08Z"),
                        "lastHeartbeat" : ISODate("2018-04-24T13:14:13.546Z"),
                        "lastHeartbeatRecv" : ISODate("2018-04-24T13:14:13.863Z"),
                        "pingMs" : NumberLong(1),
                        "syncingTo" : "mongo0:27017",
                        "configVersion" : 1
                },
                {
                        "_id" : 3,
                        "name" : "mongo3:27017",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 99,
                        "optime" : {
                                "ts" : Timestamp(1524575648, 1),
                                "t" : NumberLong(2)
                        },
                        "optimeDurable" : {
                                "ts" : Timestamp(1524575648, 1),
                                "t" : NumberLong(2)
                        },
                        "optimeDate" : ISODate("2018-04-24T13:14:08Z"),
                        "optimeDurableDate" : ISODate("2018-04-24T13:14:08Z"),
                        "lastHeartbeat" : ISODate("2018-04-24T13:14:13.548Z"),
                        "lastHeartbeatRecv" : ISODate("2018-04-24T13:14:13.725Z"),
                        "pingMs" : NumberLong(1),
                        "syncingTo" : "mongo0:27017",
                        "configVersion" : 1
                }
        ],
        "ok" : 1,
        "operationTime" : Timestamp(1524575648, 1),
        "$clusterTime" : {
                "clusterTime" : Timestamp(1524575648, 1),
                "signature" : {
                        "hash" : BinData(0,"v6kIdsFS93nZcf2hJ/EYTrVjsso="),
                        "keyId" : NumberLong("6547996956390588417")
                }
        }
}
replica0:PRIMARY>

Arbiter

We can now add the ARBITER role on mongo4 node.

[root@mongo0 ~]# mongo --port 27017 -u root -p ROOT-PASSWORD --authenticationDatabase admin
MongoDB shell version v3.6.4
connecting to: mongodb://127.0.0.1:27017/
MongoDB server version: 3.6.4
replica0:PRIMARY> rs.addArb("mongo4:27017")
{
        "ok" : 1,
        "operationTime" : Timestamp(1524575694, 1),
        "$clusterTime" : {
                "clusterTime" : Timestamp(1524575694, 1),
                "signature" : {
                        "hash" : BinData(0,"Bkvnl5fskD4NLvA1qhaU+BYLFNo="),
                        "keyId" : NumberLong("6547996956390588417")
                }
        }
}
replica0:PRIMARY>

The ARBITER role has different prompt also.

[root@mongo4 ~]# mongo
MongoDB shell version v3.6.4
connecting to: mongodb://127.0.0.1:27017
MongoDB server version: 3.6.4
Welcome to the MongoDB shell.
For interactive help, type "help".
For more comprehensive documentation, see
        http://docs.mongodb.org/
Questions? Try the support group
        http://groups.google.com/group/mongodb-user
replica0:ARBITER>

Lets see how MongoDB rs.config() function shows our configuration after adding the ARBITER node.

[root@mongo0 ~]# mongo --port 27017 -u root -p ROOT-PASSWORD --authenticationDatabase admin
MongoDB shell version v3.6.4
connecting to: mongodb://127.0.0.1:27017/
MongoDB server version: 3.6.4
replica0:PRIMARY> rs.config()
{
        "_id" : "replica0",
        "version" : 2,
        "protocolVersion" : NumberLong(1),
        "members" : [
                {
                        "_id" : 0,
                        "host" : "mongo0:27017",
                        "arbiterOnly" : false,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 1,
                        "tags" : {

                        },
                        "slaveDelay" : NumberLong(0),
                        "votes" : 1
                },
                {
                        "_id" : 1,
                        "host" : "mongo1:27017",
                        "arbiterOnly" : false,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 1,
                        "tags" : {

                        },
                        "slaveDelay" : NumberLong(0),
                        "votes" : 1
                },
                {
                        "_id" : 2,
                        "host" : "mongo2:27017",
                        "arbiterOnly" : false,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 1,
                        "tags" : {

                        },
                        "slaveDelay" : NumberLong(0),
                        "votes" : 1
                },
                {
                        "_id" : 3,
                        "host" : "mongo3:27017",
                        "arbiterOnly" : false,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 1,
                        "tags" : {

                        },
                        "slaveDelay" : NumberLong(0),
                        "votes" : 1
                },
                {
                        "_id" : 4,
                        "host" : "mongo4:27017",
                        "arbiterOnly" : true,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 0,
                        "tags" : {

                        },
                        "slaveDelay" : NumberLong(0),
                        "votes" : 1
                }
        ],
        "settings" : {
                "chainingAllowed" : true,
                "heartbeatIntervalMillis" : 2000,
                "heartbeatTimeoutSecs" : 10,
                "electionTimeoutMillis" : 10000,
                "catchUpTimeoutMillis" : -1,
                "catchUpTakeoverDelayMillis" : 30000,
                "getLastErrorModes" : {

                },
                "getLastErrorDefaults" : {
                        "w" : 1,
                        "wtimeout" : 0
                },
                "replicaSetId" : ObjectId("5adf287d597df99256d11280")
        }
}
replica0:PRIMARY>

Lets see how MongoDB rs.status() function shows our configuration after adding the ARBITER node.

[root@mongo0 ~]# mongo --port 27017 -u root -p ROOT-PASSWORD --authenticationDatabase admin
MongoDB shell version v3.6.4
connecting to: mongodb://127.0.0.1:27017/
MongoDB server version: 3.6.4
replica0:PRIMARY> rs.status()
{
        "set" : "replica0",
        "date" : ISODate("2018-04-24T13:19:31.989Z"),
        "myState" : 1,
        "term" : NumberLong(2),
        "heartbeatIntervalMillis" : NumberLong(2000),
        "optimes" : {
                "lastCommittedOpTime" : {
                        "ts" : Timestamp(1524575969, 1),
                        "t" : NumberLong(2)
                },
                "readConcernMajorityOpTime" : {
                        "ts" : Timestamp(1524575969, 1),
                        "t" : NumberLong(2)
                },
                "appliedOpTime" : {
                        "ts" : Timestamp(1524575969, 1),
                        "t" : NumberLong(2)
                },
                "durableOpTime" : {
                        "ts" : Timestamp(1524575969, 1),
                        "t" : NumberLong(2)
                }
        },
        "members" : [
                {
                        "_id" : 0,
                        "name" : "mongo0:27017",
                        "health" : 1,
                        "state" : 1,
                        "stateStr" : "PRIMARY",
                        "uptime" : 428,
                        "optime" : {
                                "ts" : Timestamp(1524575969, 1),
                                "t" : NumberLong(2)
                        },
                        "optimeDate" : ISODate("2018-04-24T13:19:29Z"),
                        "electionTime" : Timestamp(1524575557, 1),
                        "electionDate" : ISODate("2018-04-24T13:12:37Z"),
                        "configVersion" : 2,
                        "self" : true
                },
                {
                        "_id" : 1,
                        "name" : "mongo1:27017",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 421,
                        "optime" : {
                                "ts" : Timestamp(1524575969, 1),
                                "t" : NumberLong(2)
                        },
                        "optimeDurable" : {
                                "ts" : Timestamp(1524575969, 1),
                                "t" : NumberLong(2)
                        },
                        "optimeDate" : ISODate("2018-04-24T13:19:29Z"),
                        "optimeDurableDate" : ISODate("2018-04-24T13:19:29Z"),
                        "lastHeartbeat" : ISODate("2018-04-24T13:19:31.139Z"),
                        "lastHeartbeatRecv" : ISODate("2018-04-24T13:19:30.455Z"),
                        "pingMs" : NumberLong(1),
                        "syncingTo" : "mongo0:27017",
                        "configVersion" : 2
                },
                {
                        "_id" : 2,
                        "name" : "mongo2:27017",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 418,
                        "optime" : {
                                "ts" : Timestamp(1524575969, 1),
                                "t" : NumberLong(2)
                        },
                        "optimeDurable" : {
                                "ts" : Timestamp(1524575969, 1),
                                "t" : NumberLong(2)
                        },
                        "optimeDate" : ISODate("2018-04-24T13:19:29Z"),
                        "optimeDurableDate" : ISODate("2018-04-24T13:19:29Z"),
                        "lastHeartbeat" : ISODate("2018-04-24T13:19:31.145Z"),
                        "lastHeartbeatRecv" : ISODate("2018-04-24T13:19:30.571Z"),
                        "pingMs" : NumberLong(2),
                        "syncingTo" : "mongo0:27017",
                        "configVersion" : 2
                },
                {
                        "_id" : 3,
                        "name" : "mongo3:27017",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 416,
                        "optime" : {
                                "ts" : Timestamp(1524575969, 1),
                                "t" : NumberLong(2)
                        },
                        "optimeDurable" : {
                                "ts" : Timestamp(1524575969, 1),
                                "t" : NumberLong(2)
                        },
                        "optimeDate" : ISODate("2018-04-24T13:19:29Z"),
                        "optimeDurableDate" : ISODate("2018-04-24T13:19:29Z"),
                        "lastHeartbeat" : ISODate("2018-04-24T13:19:31.145Z"),
                        "lastHeartbeatRecv" : ISODate("2018-04-24T13:19:30.445Z"),
                        "pingMs" : NumberLong(2),
                        "syncingTo" : "mongo0:27017",
                        "configVersion" : 2
                },
                {
                        "_id" : 4,
                        "name" : "mongo4:27017",
                        "health" : 1,
                        "state" : 7,
                        "stateStr" : "ARBITER",
                        "uptime" : 215,
                        "lastHeartbeat" : ISODate("2018-04-24T13:19:31.111Z"),
                        "lastHeartbeatRecv" : ISODate("2018-04-24T13:19:30.730Z"),
                        "pingMs" : NumberLong(2),
                        "configVersion" : 2
                }
        ],
        "ok" : 1,
        "operationTime" : Timestamp(1524575969, 1),
        "$clusterTime" : {
                "clusterTime" : Timestamp(1524575969, 1),
                "signature" : {
                        "hash" : BinData(0,"+nUr+dY6LufEIZIjfzwKRw4cQpM="),
                        "keyId" : NumberLong("6547996956390588417")
                }
        }
}
replica0:PRIMARY>

We can also check what roles are configured by default on our MongoDB cluster.

[root@mongo0 ~]# mongo --port 27017 -u root -p ROOT-PASSWORD --authenticationDatabase admin
MongoDB shell version v3.6.4
connecting to: mongodb://127.0.0.1:27017/
MongoDB server version: 3.6.4
replica0:PRIMARY> show roles
{
        "role" : "dbAdmin",
        "db" : "test",
        "isBuiltin" : true,
        "roles" : [ ],
        "inheritedRoles" : [ ]
}
{
        "role" : "dbOwner",
        "db" : "test",
        "isBuiltin" : true,
        "roles" : [ ],
        "inheritedRoles" : [ ]
}
{
        "role" : "enableSharding",
        "db" : "test",
        "isBuiltin" : true,
        "roles" : [ ],
        "inheritedRoles" : [ ]
}
{
        "role" : "read",
        "db" : "test",
        "isBuiltin" : true,
        "roles" : [ ],
        "inheritedRoles" : [ ]
}
{
        "role" : "readWrite",
        "db" : "test",
        "isBuiltin" : true,
        "roles" : [ ],
        "inheritedRoles" : [ ]
}
{
        "role" : "userAdmin",
        "db" : "test",
        "isBuiltin" : true,
        "roles" : [ ],
        "inheritedRoles" : [ ]
}
replica0:PRIMARY>

… and users.

[root@mongo0 ~]# mongo --port 27017 -u root -p ROOT-PASSWORD --authenticationDatabase admin
MongoDB shell version v3.6.4
connecting to: mongodb://127.0.0.1:27017/
MongoDB server version: 3.6.4
replica0:PRIMARY> use admin
switched to db admin
replica0:PRIMARY> show users
{
        "_id" : "admin.admin",
        "user" : "admin",
        "db" : "admin",
        "roles" : [
                {
                        "role" : "userAdminAnyDatabase",
                        "db" : "admin"
                }
        ]
}
{
        "_id" : "admin.root",
        "user" : "root",
        "db" : "admin",
        "roles" : [
                {
                        "role" : "root",
                        "db" : "admin"
                }
        ]
}
replica0:PRIMARY>

Backup

Below are simple backup commands for the completeness of the article.

As ‘local‘ database is not being backed up by default we will have to backup it exclusively with separate command.

backup % mongodump \
           --host "rs0/mongo0:27017,mongo1:27017,mongo2:27017,mongo3:27017" \
           --username root \
           --password ROOT-PASSWORD \
           --authenticationDatabase admin \
           --db local \
           --out /backup/replica0-local

backup % mongodump \
           --host "rs0/mongo0:27017,mongo1:27017,mongo2:27017,mongo3:27017" \
           --username root \
           --password ROOT-PASSWORD \
           --authenticationDatabase admin \
           --out /backup/replica0-dat

Pretty

To auto format the query response add this to your ~/.mongorc.js file.

DBQuery.prototype._prettyShell = true
DBQuery.prototype.unpretty = function () {
  this._prettyShell = false;
  return this;
}

I would also set how much results will the .find() print before asking to type the result in the ~/.mongorc.js file.

DBQuery.shellBatchSize = 50

Final ~/.mongorc.js file.

% cat ~/.mongorc.js
DBQuery.shellBatchSize = 50
DBQuery.prototype._prettyShell = true
DBQuery.prototype.unpretty = function () {
  this._prettyShell = false;
  return this;
}

You will find other useful tips in the MongoDB: Tips & Tricks blog post.

Performance

As it seems MongoDB is not always the fastest option as PostgreSQL database also can work with JSON data type.

benchmark-mongodb24-postgresql94

benchmark-persister-timeline

Check these two below for more information and insight and decide which database is best for your needs.

Management

For the convenience of management I would also suggest adding MongoDB Ops Manager, but this is not covered in this (already big) article.

mongodb-ops-manager-deploy

EOF

Β 

Valuable News – 2018/04/23

Sortix rw(1) program is designed to be replacement for dd(1) command.
https://sortix.org/rw/
https://twitter.com/sortiecat/status/985622258071887872

As we are with Sortix it is a small self-hosting operating-system aiming to be a clean and modern POSIX implementation.
https://sortix.org/

FreeBSD and Intel Video Cards.
http://srobb.net/freebsdintel.html

FreeBSD on a Laptop.
https://www.c0ffee.net/blog/freebsd-on-a-laptop

Running FreeBSD on the Lenovo T470s ThinkPad.
https://blog.grem.de/pages/t470s.html

ThinkPad X220 macOS High Sierra 10.13 Installation Guide.
http://x220.mcdonnelltech.com/

In Other BSDs for 2018/04/21.
https://www.dragonflydigest.com/2018/04/21/21128.html

Initial OmniOS impressions by a BSD user.
https://www.linuxquestions.org/questions/solaris-opensolaris-20/initial-omnios-impressions-by-a-bsd-user-4175626757/

Download any file from WebDAV using replace.exe on Windows.
https://twitter.com/elceef/status/986334113941655553

Linux ‘Innovation’.
https://twitter.com/vermaden/status/987399057202130949

As DragonFly BSD 5.2 was recently released with HAMMER2 filesystem, here is the HAMMER2 DESIGN DOCUMENT.
https://gitweb.dragonflybsd.org/dragonfly.git/blob_plain/b93cc2e0815ec1ad6d6f8e60cc0becbdee247679:/sys/vfs/hammer2/DESIGN

SSG – Static Site Generator with rsync and lowdown.
https://twitter.com/romanzolotarev/status/988165297080340480

My beadm tool has been ported to Linux.
https://github.com/b333z/beadm

EOF.

.
.
.
.
.

Distributed Object Storage with Minio on FreeBSD

Meet Minio.

minio-logo-arch-32

Free and open source distributed object storage server compatible with Amazon S3 v2/v4 API. Offers data protection against hardware failures using erasure code and bitrot detection. Supports highly available distributed setup. Provides confidentiality, integrity and authenticity assurances for encrypted data with negligible performance overhead. Both server side and client side encryption are supported. Below is the image of example Minio setup.

Web

The Minio identifies itself as the ZFS of Cloud Object Storage. This guide will show You how to setup highly available distributed Minio storage on the FreeBSD operating system with ZFS as backend for Minio data. For convenience we will use FreeBSD Jails operating system level virtualization.

Setup

The setup will assume that You have 3 datacenters and assumption that you have two datacenters in whose the most of the data must reside and that the third datacenter is used as a ‘quorum/witness’ role. Distributed Minio supports up to 16 nodes/drives total, so we may juggle with that number to balance data between desired datacenters. As we have 16 drives to allocate resources on 3 sites we will use 7 + 7 + 2 approach here. The datacenters where most of the data must reside have 7/16 ratio while the ‘quorum/witness’ datacenter have only 2/16 ratio. Thanks to built in Minio redundancy we may loose (turn off for example) any one of those machines and our object storage will still be available and ready to use for any purpose.

Jails

First we will create 3 jails for our proof of concept Minio setup, storage1 will have the ‘quorum/witness’ role while storage2 and storage3 will have the ‘data’ role. To distinguish commands I type on the host system and storageX Jail I use two different prompts, this way it should be obvious what command to execute and where.

Command on the host system.

host # command

Command on the storageX Jail.

root@storageX:/ # command

First we will create the base Jails for our setup.

host # mkdir -p /jail/BASE /jail/storage1 /jail/storage2 /jail/storage3
host # cd /jail/BASE
host # fetch http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/11.1-RELEASE/base.txz
host # for I in 1 2 3; do echo ${I}; tar --unlink -xpJf /jail/BASE/base.txz -C /jail/storage${I}; done
1
2
3
host #

We will now add Jails configuration the the /etc/jail.conf file.

I have used my laptop for the Jail host. This is why Jail will configured to use the wireless wlan0 interface and 192.168.43.10X addresses.

host # for I in 1 2 3
do
  cat >> /etc/jail.conf << __EOF
storage${I} {
  host.hostname = storage${I}.local;
  ip4.addr = 192.168.43.10${I};
  interface = wlan0;
  path = /jail/storage${I};
  exec.start = "/bin/sh /etc/rc";
  exec.stop = "/bin/sh /etc/rc.shutdown";
  exec.clean;
  mount.devfs;
  allow.raw_sockets;
}

__EOF
done
host #

Lets verify that /etc/jail.conf file is configured as desired.

host # cat /etc/jail.conf
storage1 {
  host.hostname = storage1.local;
  ip4.addr = 192.168.43.101;
  interface = wlan0;
  path = /jail/storage1;
  exec.start = "/bin/sh /etc/rc";
  exec.stop = "/bin/sh /etc/rc.shutdown";
  exec.clean;
  mount.devfs;
  allow.raw_sockets;
}

storage2 {
  host.hostname = storage2.local;
  ip4.addr = 192.168.43.102;
  interface = wlan0;
  path = /jail/storage2;
  exec.start = "/bin/sh /etc/rc";
  exec.stop = "/bin/sh /etc/rc.shutdown";
  exec.clean;
  mount.devfs;
  allow.raw_sockets;
}

storage3 {
  host.hostname = storage3.local;
  ip4.addr = 192.168.43.103;
  interface = wlan0;
  path = /jail/storage3;
  exec.start = "/bin/sh /etc/rc";
  exec.stop = "/bin/sh /etc/rc.shutdown";
  exec.clean;
  mount.devfs;
  allow.raw_sockets;
}

host #

Now we will start our Jails.

host # for I in 1 2 3; do service jail onestart storage${I}; done
Starting jails: storage1.
Starting jails: storage2.
Starting jails: storage3.

Lets see how they work.

host # jls
   JID  IP Address      Hostname                      Path
     1  192.168.43.101  storage1.local                /jail/storage1
     2  192.168.43.102  storage2.local                /jail/storage2
     3  192.168.43.103  storage3.local                /jail/storage3

Now lets add DNS server so they will have Internet connectivity.

host # for I in 1 2 3; do echo nameserver 1.1.1.1 > /jail/storage${I}/etc/resolv.conf; done

We can now install Minio package.

host # for I in 1 2 3; do jexec storage${I} env ASSUME_ALWAYS_YES=yes pkg install -y minio; echo; done
Bootstrapping pkg from pkg+http://pkg.FreeBSD.org/FreeBSD:11:amd64/quarterly, please wait...
Verifying signature with trusted certificate pkg.freebsd.org.2013102301... done
[storage1.local] Installing pkg-1.10.5...
[storage1.local] Extracting pkg-1.10.5: 100%
Updating FreeBSD repository catalogue...
pkg: Repository FreeBSD load error: access repo file(/var/db/pkg/repo-FreeBSD.sqlite) failed: No such file or directory
[storage1.local] Fetching meta.txz: 100%    944 B   0.9kB/s    00:01    
[storage1.local] Fetching packagesite.txz: 100%    6 MiB 637.1kB/s    00:10    
Processing entries: 100%
FreeBSD repository update completed. 31143 packages processed.
All repositories are up to date.
Updating database digests format: 100%
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        minio: 2018.03.19.19.22.06

Number of packages to be installed: 1

The process will require 22 MiB more space.
6 MiB to be downloaded.
[storage1.local] [1/1] Fetching minio-2018.03.19.19.22.06.txz: 100%    6 MiB 305.6kB/s    00:19    
Checking integrity... done (0 conflicting)
[storage1.local] [1/1] Installing minio-2018.03.19.19.22.06...
===> Creating groups.
Creating group 'minio' with gid '473'.
===> Creating users
Creating user 'minio' with uid '473'.
[storage1.local] [1/1] Extracting minio-2018.03.19.19.22.06: 100%

Bootstrapping pkg from pkg+http://pkg.FreeBSD.org/FreeBSD:11:amd64/quarterly, please wait...
Verifying signature with trusted certificate pkg.freebsd.org.2013102301... done
[storage2.local] Installing pkg-1.10.5...
[storage2.local] Extracting pkg-1.10.5: 100%
Updating FreeBSD repository catalogue...
pkg: Repository FreeBSD load error: access repo file(/var/db/pkg/repo-FreeBSD.sqlite) failed: No such file or directory
[storage2.local] Fetching meta.txz: 100%    944 B   0.9kB/s    00:01    
[storage2.local] Fetching packagesite.txz: 100%    6 MiB 637.1kB/s    00:10    
Processing entries: 100%
FreeBSD repository update completed. 31143 packages processed.
All repositories are up to date.
Updating database digests format: 100%
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        minio: 2018.03.19.19.22.06

Number of packages to be installed: 1

The process will require 22 MiB more space.
6 MiB to be downloaded.
[storage2.local] [1/1] Fetching minio-2018.03.19.19.22.06.txz: 100%    6 MiB 305.6kB/s    00:19    
Checking integrity... done (0 conflicting)
[storage2.local] [1/1] Installing minio-2018.03.19.19.22.06...
===> Creating groups.
Creating group 'minio' with gid '473'.
===> Creating users
Creating user 'minio' with uid '473'.
[storage2.local] [1/1] Extracting minio-2018.03.19.19.22.06: 100%

Bootstrapping pkg from pkg+http://pkg.FreeBSD.org/FreeBSD:11:amd64/quarterly, please wait...
Verifying signature with trusted certificate pkg.freebsd.org.2013102301... done
[storage3.local] Installing pkg-1.10.5...
[storage3.local] Extracting pkg-1.10.5: 100%
Updating FreeBSD repository catalogue...
pkg: Repository FreeBSD load error: access repo file(/var/db/pkg/repo-FreeBSD.sqlite) failed: No such file or directory
[storage3.local] Fetching meta.txz: 100%    944 B   0.9kB/s    00:01    
[storage3.local] Fetching packagesite.txz: 100%    6 MiB 637.1kB/s    00:10    
Processing entries: 100%
FreeBSD repository update completed. 31143 packages processed.
All repositories are up to date.
Updating database digests format: 100%
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        minio: 2018.03.19.19.22.06

Number of packages to be installed: 1

The process will require 22 MiB more space.
6 MiB to be downloaded.
[storage3.local] [1/1] Fetching minio-2018.03.19.19.22.06.txz: 100%    6 MiB 305.6kB/s    00:19    
Checking integrity... done (0 conflicting)
[storage3.local] [1/1] Installing minio-2018.03.19.19.22.06...
===> Creating groups.
Creating group 'minio' with gid '473'.
===> Creating users
Creating user 'minio' with uid '473'.
[storage3.local] [1/1] Extracting minio-2018.03.19.19.22.06: 100%

host #

Lets verify that Minio package has installed successfully.

host # for I in 1 2 3; do jexec storage${I} which minio; done
/usr/local/bin/minio
/usr/local/bin/minio
/usr/local/bin/minio
host #

Now we will configure /etc/hosts file.

root@storage1:/ # cat >> /etc/hosts << __EOF
192.168.43.101 storage1
192.168.43.102 storage2
192.168.43.103 storage3
__EOF
root@storage1:/ # cat >> /etc/hosts << __EOF
192.168.43.101 storage1
192.168.43.102 storage2
192.168.43.103 storage3
__EOF
root@storage1:/ # cat >> /etc/hosts << __EOF
192.168.43.101 storage1
192.168.43.102 storage2
192.168.43.103 storage3
__EOF

We will create directories for Minio data.

host # for DIR in 1 2 3 4 5 6 7
do
  for I in 2 3
  do
    jexec storage${I} mkdir -p /data${DIR}
  done
done
host # for DIR in 1 2
do
  for I in 1
  do
    jexec storage${I} mkdir -p /data${DIR}
  done
done

Lets verify that that our data directories created successfully.

host # for I in 1 2 3
  do
    echo storage${I}
    jexec storage${I} ls -1 / | grep data
    echo
  done


storage1
data1
data2

storage2
data1
data2
data3
data4
data5
data6
data7

storage3
data1
data2
data3
data4
data5
data6
data7

Basic minio command example.

root@storage1:/ # minio
NAME:
  minio - Cloud Storage Server.

DESCRIPTION:
  Minio is an Amazon S3 compatible object storage server. Use it to store photos, videos, VMs, containers, log files, or any blob of data as objects.

USAGE:
  minio [FLAGS] COMMAND [ARGS...]

COMMANDS:
  server   Start object storage server.
  gateway  Start object storage gateway.
  update   Check for a new software update.
  version  Print version.
  
FLAGS:
  --config-dir value, -C value  Path to configuration directory. (default: "/root/.minio")
  --quiet                       Disable startup information.
  --json                        Output server logs and startup information in json format.
  --help, -h                    Show help.
  
VERSION:
  2018-03-19T19:22:06Z

Now we can generate the list of directories on servers to add as argument for Minio.

host # for DIR in 1 2
do
  for I in 1 
  do
    echo -n http://
    jls | grep storage${I} | awk '{printf $3}' | sed s/.local//g
    echo ":9000/data${DIR} \\"
  done
done | sort -n

host # for DIR in 1 2 3 4 5 6 7
do
  for I in 2 3
  do
    echo -n http://
    jls | grep storage${I} | awk '{printf $3}' | sed s/.local//g
    echo ":9000/data${DIR} \\"
  done
done | sort -n
http://storage1:9000/data1 \
http://storage1:9000/data2 \
http://storage2:9000/data1 \
http://storage2:9000/data2 \
http://storage2:9000/data3 \
http://storage2:9000/data4 \
http://storage2:9000/data5 \
http://storage2:9000/data6 \
http://storage2:9000/data7 \
http://storage3:9000/data1 \
http://storage3:9000/data2 \
http://storage3:9000/data3 \
http://storage3:9000/data4 \
http://storage3:9000/data5 \
http://storage3:9000/data6 \
http://storage3:9000/data7 \

We can as well just write it down by hand of course πŸ™‚

host # for DIR in 1 2
do
  for I in 1 
  do
    echo -n http://
    jls | grep storage${I} | awk '{printf $3}' | sed s/.local//g
    echo -n ":9000/data${DIR} "
  done
done | sort -n

host # for DIR in 1 2 3 4 5 6 7
do
  for I in 2 3
  do
    echo -n http://
    jls | grep storage${I} | awk '{printf $3}' | sed s/.local//g
    echo -n ":9000/data${DIR} "
  done
done | sort -n

This is out list of data directories that we will use to configure Minio in FreeBSD’s main configuration /etc/rc.conf file.

http://storage1:9000/data1 http://storage1:9000/data2 http://storage2:9000/data1 http://storage2:9000/data2 http://storage2:9000/data3 http://storage2:9000/data4 http://storage2:9000/data5 http://storage2:9000/data6 http://storage2:9000/data7 http://storage3:9000/data1 http://storage3:9000/data2 http://storage3:9000/data3 http://storage3:9000/data4 http://storage3:9000/data5 http://storage3:9000/data6 http://storage3:9000/data7

Now, lets put Minio settings into the /etc/rc.conf file.

root@storageX:~ # cat > /etc/rc.conf << __EOF 
minio_enable=YES
minio_disks="http://storage1:9000/data1 http://storage1:9000/data2 http://storage2:9000/data1 http://storage2:9000/data2 http://storage2:9000/data3 http://storage2:9000/data4 http://storage2:9000/data5 http://storage2:9000/data6 http://storage2:9000/data7 http://storage3:9000/data1 http://storage3:9000/data2 http://storage3:9000/data3 http://storage3:9000/data4 http://storage3:9000/data5 http://storage3:9000/data6 http://storage3:9000/data7"
__EOF
root@storageX:~ # 
root@storageX:~ # cat /etc/rc.conf
minio_enable=YES
minio_disks="http://storage1:9000/data1 http://storage1:9000/data2 http://storage2:9000/data1 http://storage2:9000/data2 http://storage2:9000/data3 http://storage2:9000/data4 http://storage2:9000/data5 http://storage2:9000/data6 http://storage2:9000/data7 http://storage3:9000/data1 http://storage3:9000/data2 http://storage3:9000/data3 http://storage3:9000/data4 http://storage3:9000/data5 http://storage3:9000/data6 http://storage3:9000/data7"
root@storageX:~ #

Now we will start and configure Minio for the first time.

On each storageX server run the following set of commands.

host # jexec storage3
root@storage3:~ # 
root@storage3:/ # rm -rf /http:\*
root@storage3:/ # rm -rf /usr/local/etc/minio
root@storage3:/ # rm -rf /data?/* /data?/.minio.sys
root@storage3:/ # touch                /var/log/minio.log
root@storage3:/ # chown    minio:minio /var/log/minio.log
root@storage3:/ # mkdir -p             /usr/local/etc/minio
root@storage3:/ # chown -R minio:minio /usr/local/etc/minio
root@storage3:/ # mkdir -p             /http::
root@storage3:/ # chown -R minio:minio /http::
root@storage3:/ # mkdir -p             /http:
root@storage3:/ # chown -R minio:minio /http:
root@storage3:/ # su -m minio -c 'env \\
?   MINIO_ACCESS_KEY=alibaba \\
?   MINIO_SECRET_KEY=0P3NS3S4M3 \\
?   minio server \\
?     --config-dir /usr/local/etc/minio \\
?     http://storage1:9000/data1 \\
?     http://storage1:9000/data2 \\
?     http://storage2:9000/data1 \\
?     http://storage2:9000/data2 \\
?     http://storage2:9000/data3 \\
?     http://storage2:9000/data4 \\
?     http://storage2:9000/data5 \\
?     http://storage2:9000/data6 \\
?     http://storage2:9000/data7 \\
?     http://storage3:9000/data1 \\
?     http://storage3:9000/data2 \\
?     http://storage3:9000/data3 \\
?     http://storage3:9000/data4 \\
?     http://storage3:9000/data5 \\
?     http://storage3:9000/data6 \\
?     http://storage3:9000/data7'
Created minio configuration file successfully at /usr/local/etc/minio
Waiting for the first server to format the disks.
Waiting for the first server to format the disks.
Drive Capacity: 504 GiB Free, 515 GiB Total
Status:         16 Online, 0 Offline. 

Endpoint:  http://192.168.43.103:9000
AccessKey: alibaba 
SecretKey: 0P3NS3S4M3 

Browser Access:
   http://192.168.43.103:9000

Command-line Access: https://docs.minio.io/docs/minio-client-quickstart-guide
   $ mc config host add myminio http://192.168.43.103:9000 alibaba 0P3NS3S4M3

Object API (Amazon S3 compatible):
   Go:         https://docs.minio.io/docs/golang-client-quickstart-guide
   Java:       https://docs.minio.io/docs/java-client-quickstart-guide
   Python:     https://docs.minio.io/docs/python-client-quickstart-guide
   JavaScript: https://docs.minio.io/docs/javascript-client-quickstart-guide
   .NET:       https://docs.minio.io/docs/dotnet-client-quickstart-guide
host # jexec storage2
root@storage2:~ # 
root@storage2:/ # rm -rf /http:\*
root@storage2:/ # rm -rf /usr/local/etc/minio
root@storage2:/ # rm -rf /data?/* /data?/.minio.sys
root@storage2:/ # touch                /var/log/minio.log
root@storage2:/ # chown    minio:minio /var/log/minio.log
root@storage2:/ # mkdir -p             /usr/local/etc/minio
root@storage2:/ # chown -R minio:minio /usr/local/etc/minio
root@storage2:/ # mkdir -p             /http::
root@storage2:/ # chown -R minio:minio /http::
root@storage2:/ # mkdir -p             /http:
root@storage2:/ # chown -R minio:minio /http:
root@storage2:/ # su -m minio -c 'env \\
?   MINIO_ACCESS_KEY=alibaba \\
?   MINIO_SECRET_KEY=0P3NS3S4M3 \\
?   minio server \\
?     --config-dir /usr/local/etc/minio \\
?     http://storage1:9000/data1 \\
?     http://storage1:9000/data2 \\
?     http://storage2:9000/data1 \\
?     http://storage2:9000/data2 \\
?     http://storage2:9000/data3 \\
?     http://storage2:9000/data4 \\
?     http://storage2:9000/data5 \\
?     http://storage2:9000/data6 \\
?     http://storage2:9000/data7 \\
?     http://storage3:9000/data1 \\
?     http://storage3:9000/data2 \\
?     http://storage3:9000/data3 \\
?     http://storage3:9000/data4 \\
?     http://storage3:9000/data5 \\
?     http://storage3:9000/data6 \\
?     http://storage3:9000/data7'
Created minio configuration file successfully at /usr/local/etc/minio
Waiting for the first server to format the disks.
Waiting for the first server to format the disks.
Drive Capacity: 504 GiB Free, 515 GiB Total
Status:         16 Online, 0 Offline. 

Endpoint:  http://192.168.43.102:9000
AccessKey: alibaba 
SecretKey: 0P3NS3S4M3 

Browser Access:
   http://192.168.43.102:9000

Command-line Access: https://docs.minio.io/docs/minio-client-quickstart-guide
   $ mc config host add myminio http://192.168.43.102:9000 alibaba 0P3NS3S4M3

Object API (Amazon S3 compatible):
   Go:         https://docs.minio.io/docs/golang-client-quickstart-guide
   Java:       https://docs.minio.io/docs/java-client-quickstart-guide
   Python:     https://docs.minio.io/docs/python-client-quickstart-guide
   JavaScript: https://docs.minio.io/docs/javascript-client-quickstart-guide
   .NET:       https://docs.minio.io/docs/dotnet-client-quickstart-guide
host # jexec storage1
root@storage1:~ # 
root@storage1:/ # rm -rf /http:\*
root@storage1:/ # rm -rf /usr/local/etc/minio
root@storage1:/ # rm -rf /data?/* /data?/.minio.sys
root@storage1:/ # touch                /var/log/minio.log
root@storage1:/ # chown    minio:minio /var/log/minio.log
root@storage1:/ # mkdir -p             /usr/local/etc/minio
root@storage1:/ # chown -R minio:minio /usr/local/etc/minio
root@storage1:/ # mkdir -p             /http::
root@storage1:/ # chown -R minio:minio /http::
root@storage1:/ # mkdir -p             /http:
root@storage1:/ # chown -R minio:minio /http:
root@storage1:/ # su -m minio -c 'env \\
?   MINIO_ACCESS_KEY=alibaba \\
?   MINIO_SECRET_KEY=0P3NS3S4M3 \\
?   minio server \\
?     --config-dir /usr/local/etc/minio \\
?     http://storage1:9000/data1 \\
?     http://storage1:9000/data2 \\
?     http://storage2:9000/data1 \\
?     http://storage2:9000/data2 \\
?     http://storage2:9000/data3 \\
?     http://storage2:9000/data4 \\
?     http://storage2:9000/data5 \\
?     http://storage2:9000/data6 \\
?     http://storage2:9000/data7 \\
?     http://storage3:9000/data1 \\
?     http://storage3:9000/data2 \\
?     http://storage3:9000/data3 \\
?     http://storage3:9000/data4 \\
?     http://storage3:9000/data5 \\
?     http://storage3:9000/data6 \\
?     http://storage3:9000/data7'
Created minio configuration file successfully at /usr/local/etc/minio
Waiting for the first server to format the disks.
Waiting for the first server to format the disks.
Drive Capacity: 504 GiB Free, 515 GiB Total
Status:         16 Online, 0 Offline. 

Endpoint:  http://192.168.43.101:9000
AccessKey: alibaba 
SecretKey: 0P3NS3S4M3 

Browser Access:
   http://192.168.43.101:9000

Command-line Access: https://docs.minio.io/docs/minio-client-quickstart-guide
   $ mc config host add myminio http://192.168.43.101:9000 alibaba 0P3NS3S4M3

Object API (Amazon S3 compatible):
   Go:         https://docs.minio.io/docs/golang-client-quickstart-guide
   Java:       https://docs.minio.io/docs/java-client-quickstart-guide
   Python:     https://docs.minio.io/docs/python-client-quickstart-guide
   JavaScript: https://docs.minio.io/docs/javascript-client-quickstart-guide
   .NET:       https://docs.minio.io/docs/dotnet-client-quickstart-guide

Here is how it looks in the xterm terminal.

minio-first-run-setup

We can now verify in the browser that it actually works.

minio-browser-01

Now hit [CTRL]+[C] in each of these windows to stop the Minio cluster.

We will now start Minio with FreeBSD rc(8) subsystem as a service.

root@storage1:/ # service minio start
Starting minio.
root@storage1:/ # cat /var/log/minio.log 
root@storage1:/ # service minio status
minio is running as pid 50309.

Lets check if it works.

root@storage1:/ # ps -U minio
  PID TT  STAT    TIME COMMAND
50308  -  IsJ  0:00.00 daemon: /usr/bin/env[50309] (daemon)
50309  -  IJ   0:00.27 /usr/local/bin/minio -C /usr/local/etc/minio server (...)

Now we will do some basic operations, login into Minio distributed storage, create new bucket and upload some file to it.

minio-browser-02

This is how empty Minio cluster looks like.

minio-browser-03

Select Create Bucket option from the button below.

minio-browser-04-create-bucket

We will use name test for our new bucket.

minio-browser-05-create-bucket

It is created and we can access it.

minio-browser-06-bucket

Lets Upload File using same menu as previously.

minio-browser-07-file-upload

The upload progress shown by Minio.

minio-browser-08-file-upload

File has been indeed uploaded.

minio-browser-09-file-upload

By clicking on it we may access it directly from the browser.

minio-browser-10-file-display

We can also share link to that file by using the File Menu as shown below.

minio-browser-10-file-link

The link creation dialog is shown below.

minio-browser-11-file-link

minio-browser-12-file-link

Lets see how Minio distributes the data – the ThinkPad Design – Spirit and Essence.pdf file in out case – over its data directories spread across the servers.

host # jexec storage1
root@storage1:/ # find /data?/test
/data1/test
/data1/test/ThinkPad Design - Spirit and Essence.pdf
/data1/test/ThinkPad Design - Spirit and Essence.pdf/xl.json
/data1/test/ThinkPad Design - Spirit and Essence.pdf/part.1
/data2/test
/data2/test/ThinkPad Design - Spirit and Essence.pdf
/data2/test/ThinkPad Design - Spirit and Essence.pdf/xl.json
/data2/test/ThinkPad Design - Spirit and Essence.pdf/part.1
root@storage1:/ # exit
host # jexec storage2
root@storage2:/ # find /data?/test
/data1/test
/data1/test/ThinkPad Design - Spirit and Essence.pdf
/data1/test/ThinkPad Design - Spirit and Essence.pdf/part.1
/data1/test/ThinkPad Design - Spirit and Essence.pdf/xl.json
/data2/test
/data2/test/ThinkPad Design - Spirit and Essence.pdf
/data2/test/ThinkPad Design - Spirit and Essence.pdf/xl.json
/data2/test/ThinkPad Design - Spirit and Essence.pdf/part.1
/data3/test
/data3/test/ThinkPad Design - Spirit and Essence.pdf
/data3/test/ThinkPad Design - Spirit and Essence.pdf/part.1
/data3/test/ThinkPad Design - Spirit and Essence.pdf/xl.json
/data4/test
/data4/test/ThinkPad Design - Spirit and Essence.pdf
/data4/test/ThinkPad Design - Spirit and Essence.pdf/part.1
/data4/test/ThinkPad Design - Spirit and Essence.pdf/xl.json
/data5/test
/data5/test/ThinkPad Design - Spirit and Essence.pdf
/data5/test/ThinkPad Design - Spirit and Essence.pdf/part.1
/data5/test/ThinkPad Design - Spirit and Essence.pdf/xl.json
/data6/test
/data6/test/ThinkPad Design - Spirit and Essence.pdf
/data6/test/ThinkPad Design - Spirit and Essence.pdf/part.1
/data6/test/ThinkPad Design - Spirit and Essence.pdf/xl.json
/data7/test
/data7/test/ThinkPad Design - Spirit and Essence.pdf
/data7/test/ThinkPad Design - Spirit and Essence.pdf/xl.json
/data7/test/ThinkPad Design - Spirit and Essence.pdf/part.1
root@storage2:/ # exit
host # jexec storage3
root@storage3:/ # find /data?/test
/data1/test
/data1/test/ThinkPad Design - Spirit and Essence.pdf
/data1/test/ThinkPad Design - Spirit and Essence.pdf/part.1
/data1/test/ThinkPad Design - Spirit and Essence.pdf/xl.json
/data2/test
/data2/test/ThinkPad Design - Spirit and Essence.pdf
/data2/test/ThinkPad Design - Spirit and Essence.pdf/xl.json
/data2/test/ThinkPad Design - Spirit and Essence.pdf/part.1
/data3/test
/data3/test/ThinkPad Design - Spirit and Essence.pdf
/data3/test/ThinkPad Design - Spirit and Essence.pdf/xl.json
/data3/test/ThinkPad Design - Spirit and Essence.pdf/part.1
/data4/test
/data4/test/ThinkPad Design - Spirit and Essence.pdf
/data4/test/ThinkPad Design - Spirit and Essence.pdf/part.1
/data4/test/ThinkPad Design - Spirit and Essence.pdf/xl.json
/data5/test
/data5/test/ThinkPad Design - Spirit and Essence.pdf
/data5/test/ThinkPad Design - Spirit and Essence.pdf/part.1
/data5/test/ThinkPad Design - Spirit and Essence.pdf/xl.json
/data6/test
/data6/test/ThinkPad Design - Spirit and Essence.pdf
/data6/test/ThinkPad Design - Spirit and Essence.pdf/part.1
/data6/test/ThinkPad Design - Spirit and Essence.pdf/xl.json
/data7/test
/data7/test/ThinkPad Design - Spirit and Essence.pdf
/data7/test/ThinkPad Design - Spirit and Essence.pdf/xl.json
/data7/test/ThinkPad Design - Spirit and Essence.pdf/part.1
root@storage3:/ # exit

We can also see what Minio configuration file /usr/local/etc/minio/config.json has been generated.

host # jexec storage1
root@storage1:/ # cat /usr/local/etc/minio/config.json 
{
        "version": "22",
        "credential": {
                "accessKey": "alibaba",
                "secretKey": "0P3NS3S4M3"
        },
        "region": "",
        "browser": "on",
        "domain": "",
        "storageclass": {
                "standard": "",
                "rrs": ""
        },
        "notify": {
                "amqp": {
                        "1": {
                                "enable": false,
                                "url": "",
                                "exchange": "",
                                "routingKey": "",
                                "exchangeType": "",
                                "deliveryMode": 0,
                                "mandatory": false,
                                "immediate": false,
                                "durable": false,
                                "internal": false,
                                "noWait": false,
                                "autoDeleted": false
                        }
                },
                "elasticsearch": {
                        "1": {
                                "enable": false,
                                "format": "",
                                "url": "",
                                "index": ""
                        }
                },
                "kafka": {
                        "1": {
                                "enable": false,
                                "brokers": null,
                                "topic": ""
                        }
                },
                "mqtt": {
                        "1": {
                                "enable": false,
                                "broker": "",
                                "topic": "",
                                "qos": 0,
                                "clientId": "",
                                "username": "",
                                "password": "",
                                "reconnectInterval": 0,
                                "keepAliveInterval": 0
                        }
                },
                "mysql": {
                        "1": {
                                "enable": false,
                                "format": "",
                                "dsnString": "",
                                "table": "",
                                "host": "",
                                "port": "",
                                "user": "",
                                "password": "",
                                "database": ""
                        }
                },
                "nats": {
                        "1": {
                                "enable": false,
                                "address": "",
                                "subject": "",
                                "username": "",
                                "password": "",
                                "token": "",
                                "secure": false,
                                "pingInterval": 0,
                                "streaming": {
                                        "enable": false,
                                        "clusterID": "",
                                        "clientID": "",
                                        "async": false,
                                        "maxPubAcksInflight": 0
                                }
                        }
                },
                "postgresql": {
                        "1": {
                                "enable": false,
                                "format": "",
                                "connectionString": "",
                                "table": "",
                                "host": "",
                                "port": "",
                                "user": "",
                                "password": "",
                                "database": ""
                        }
                },
                "redis": {
                        "1": {
                                "enable": false,
                                "format": "",
                                "address": "",
                                "password": "",
                                "key": ""
                        }
                },
                "webhook": {
                        "1": {
                                "enable": false,
                                "endpoint": ""
                        }
                }
        }

S3FS

We can also mount that test bucket from out distributed Minio object storage cluster as filesystem using the S3FS project. Lets add s3fs package and mount our bucket.

host # pkg install -y fusefs-s3fs

Now we will configure password for our bucket.

host # echo test:alibaba:0P3NS3S4M3 > /root/.passwd-s3fs
host # chmod 600 /root/.passwd-s3fs
host # cat /root/.passwd-s3fs 
test:alibaba:0P3NS3S4M3

Now lets do the actual mount.

host # mkdir /tmp/test
host # s3fs \
  -o allow_other \
  -o use_path_request_style \
  -o url=http://192.168.43.101:9000 \
  -o passwd_file=/root/.passwd-s3fs \
  test /tmp/test

The file ThinkPad Design – Spirit and Essence.pdf that we put through web interface should be here.

host # exa -l /tmp/test
.--------- 10M root 2018-04-16 14:15 ThinkPad Design - Spirit and Essence.pdf

host # file /tmp/test/ThinkPad\ Design\ -\ Spirit\ and\ Essence.pdf 
/tmp/test/ThinkPad Design - Spirit and Essence.pdf: PDF document, version 1.4

host # stat /tmp/test/ThinkPad\ Design\ -\ Spirit\ and\ Essence.pdf
3976265496 2 ---------- 1 root wheel 0 10416953 "Jan  1 01:00:00 1970" "Apr 16 14:35:35 2018" "Jan  1 01:00:00 1970" "Jan  1 00:59:59 1970" 4096 20346 0 /tmp/test/ThinkPad Design - Spirit and Essence.pdf

We can now upload other file into that bucket using s3fs mount.

host # cp -v /home/vermaden/On\ the\ Shortness\ of\ Life\ -\ Lucius\ Seneca.pdf /tmp/test
/home/vermaden/On the Shortness of Life - Lucius Seneca.pdf -> /tmp/test/On the Shortness of Life - Lucius Seneca.pdf

host # file /tmp/test/On\ the\ Shortness\ of\ Life\ -\ Lucius\ Seneca.pdf 
On the Shortness of Life - Lucius Seneca.pdf: PDF document, version 1.4

We can also verify that our file put through s3fs is visible on the web interface.

minio-browser-13-s3fs-upload

Real Hardware

Now, as we have working Proof of Concept for the distributed Minio setup how about putting it on a real hardware for real storage purposes? I would setup a 16 node Minio distributed server on a Supermicro SSG-5018D8-AR12L hardware. Supermicro even suggests using that kind of servers for object storage, here is their white paper on that topic – Object Storage Solution for Data Archive using Supermicro SSG-5018D8-AR12L and OpenIO SDS – but they use OpenIO not Minio for distributed object storage solution.

This server features the Supermicro X10SDV-7TP4F motherboard. This is important as this motherboard officially supports FreeBSD 11.x operating system on their Supermicro OS Compatibility page.

Motherboard specification has these features.

 1 x Intel Xeon D-1537 8-Core / 16-Threads TDP 35W
 4 x UDIMM for up to 128GB ECC RDIMM DDR4 2133MHz
12 x 3.5" SAS2/SATA3 Hot-Swap HDD Bays
 4 x 2.5" Cold-Swap HDD Bays
 1 x Controller Intel SoC for 4 SATA3 (6Gbps) Ports
 1 x Controller Broadcom 2116 for 16 SATA3 (6Gbps) Ports
 1 x Expansion Slot PCI-E 3.0 x8 
 1 x Expansion Slot M.2 PCIe 3.0 x4
 1 x Expansion Slot Mini-PCIe w/ mSATA Support
 2 x 10G SFP+ Port
 2 x 1GbE LAN Port
 2 x External USB 3.0 Port
 1 x Interlal USB 2.0 Port
 2 x 400W High-Rfficiency Redundant Power Supplies

You can configure your own and get approximated price using the Thinkmate site from here:
https://www.thinkmate.com/system/superstorage-server-5018d8-ar12l

I would add this components to the basic setup:

 4 x UDIMM FULL 128 GB ECC RDIMM DDR4
 2 x 240GB Micron 5100 MAX 2.5" SATA 6.0Gb/s SSD
 2 x 7.68TB Micron 5200 ECO Series 2.5" SATA 6.0Gb/s SSD
12 x 12TB SATA 6.0Gb/s 7200RPM 3.5" Hitachi Ultrastarβ„’ He12
 3 x SanDisk Cruzer Fit 32GB USB 3.0

Now, I will use the 3 x SanDisk Cruzer Fit 32GB USB 3.0 disks to install FreeBSD as a ZFS root/boot pool with mirror + spare on these disks. We do not need performance here.

Then, the 12 x 12TB SATA 6.0Gb/s 7200RPM 3.5″ Hitachi Ultrastarβ„’ He12 drives will be used as RAIDZ (RAID5 equivalent in ZFS without the write hole) for the Minio data, wich 11 + 1 setup, which means 11 drives for data and 1 drive for parity. As we can lose HALF of the Minio servers I would not waste 12 TB drive for spare here. Then, I would use 2 x 240GB Micron 5100 MAX 2.5″ SATA 6.0Gb/s SSD in mirror for the ZFS ZIL (ZFS Intent Log) to accelerate writes and 2 x 7.68TB Micron 5200 ECO Series 2.5″ SATA 6.0Gb/s SSD for the ZFS read cache (L2ARC).

The network would be setup on 2 x 10G SFP+ Port with LACP as lagg0 interface so each server would have 20 Gbit connectivity. This will give us a total of 320 Gbit theoretical network throughput.

This setup would give as 132 TB ZFS pool space with 15 TB for read cache and 240 GB for writes for single 1U server. Making the calculations this will give as 2112 TB (more then 2 PB) of space for Minio data.

With Minio algorithm for data redundancy we will have about 1 PB of usable storage space in our 16U Object Storage FreeBSD Appliance.

Not bad for my taste πŸ™‚

UPDATE 1

The Distributed Object Storage with Minio on FreeBSD article was included in the BSD Now 246 – Disclosure episode.

Thanks for mentioning!

EOF

Valuable News – 2018/04/14

New version of DragonFly BSD has been released.
https://www.dragonflybsd.org/release52/

How to fast create content filtering proxy on OpenBSD.
https://twitter.com/ogmaconnect1/status/983813448504094721

Nextcloud 13.0.1 can check new passwords against HaveIBeenPwned database.
https://twitter.com/Nextclouders/status/983669931467329536
https://nextcloud.com/blog/nextcloud-will-check-passwords-against-database-of-haveibeenpwned/

The pass is a command line password manager powered by LibreSSL.
https://www.romanzolotarev.com/pass.html

HP 9000 and PA-RISC Computers Story.
https://www.openpa.net/systems/hp-9000_pa-risc_story.html

How to overclock the microSD card reader on a Raspberry Pi.
https://www.jeffgeerling.com/blog/2016/how-overclock-microsd-card-reader-on-raspberry-pi-3

FreeBSD’s pkg will now accept -y and --yes from arguments
https://svnweb.freebsd.org/base?view=revision&revision=332484

2nd Generation AMD Ryzen CPUs – meet 3.2Ghz octa core at 65W TDP for $299.
https://www.anandtech.com/show/12642/amd-ryzen-2nd-gen-details-4-skus-reviews-19th

We Are Amiga!
http://blog.a-eon.biz/blog/?p=10986

Backblaze Hard Drive Stats for 2017
TL;DR Hitachi is still most reliable, Seagate still with most failures.
https://www.backblaze.com/blog/hard-drive-stats-for-2017/

Run OpenBSD on your web server.
https://www.romanzolotarev.com/openbsd/webserver.html

In Other BSDs for 2018/04/14.
https://www.dragonflydigest.com/2018/04/14/21108.html

EOF.

.
.
.
.
.

FreeBSD Desktop – Part 2 – Install

This is the second post in the FreeBSD Desktop series.

You may want to check other articles in the FreeBSD Desktop series on the FreeBSD Desktop – Global Page where you will find links to all episodes of the series along with table of contents for each episode’s contents.

How about actually installing FreeBSD? On a real hardware? You may first want to mess with the FreeBSD installer in a virtualized hardware (Bhyve/KVM/XEN/Virtualbox/…) so You will gain some confidence. In these posts series we will create a complete step by step FreeBSD desktop installation on USB 2.0 pendrive. 8 GB size or more is required. This way You will not have to ‘touch’ you installed system and as You gain confidence in FreeBSD You may want to switch to it if it suits your needs better then the currently installed system on your hardware. This way You can also try it on various computers without the need of (re)install or You may use it as You ‘go/mobile’ system on a stick.

For the series I will use one of the best laptops ever made – Lenovo ThinkPad X220 – shown below.

thinkpad-x220-photo

Its a 2011 12.5″ screen laptop with best keyboard layout on the world. As the series will progress I will also show the differences to other laptop – Lenovo ThinkPad T420s – another great machine. Besides being one of the best laptops ever made they also today have another advantage – they are dirt cheap. You can get any of them in a really good condition for about $150-180.

BIOS

First we need to setup our machine, I mean the BIOS settings in the laptop. We will disable unused (Express Card/Bluetooth/Camera/Fingerprint Reader/…) and unwanted options (Intel AMT/Security Chip/UEFI/…). Below are screens with these settings.

thinkpad-x220-bios-01

thinkpad-x220-bios-02

thinkpad-x220-bios-03

thinkpad-x220-bios-04

thinkpad-x220-bios-05

thinkpad-x220-bios-06

thinkpad-x220-bios-07

thinkpad-x220-bios-08

thinkpad-x220-bios-09

thinkpad-x220-bios-10

thinkpad-x220-bios-11

thinkpad-x220-bios-12

thinkpad-x220-bios-13

thinkpad-x220-bios-14

thinkpad-x220-bios-15

thinkpad-x220-bios-16

thinkpad-x220-bios-17

Install

The install is very similar to the HardenedBSD install made in earlier post – Introduction to HardenedBSD World.

You will need:
* 8GB or larger USB pendrive for the installation (or install directly to hard disk)
* 1GB or larger USB pendrive or 700MB CD-R(W) disk for the FreeBSD installer to boot/install from

Besides all Lenovo ThinkPad XY20 (X220/T420/T420s/T520/W520) laptops being one of the best ever made they have one amusing problem – the broken GPT boot πŸ™‚ The installer – on these Lenovo laptops – will ask You if You want to apply that fix – answer ‘yes’ there. Later in the setup process this option is called GPT + Lenovo Fix (BIOS) as a Partition Scheme option. You can read more about GPT issue on this FreeBSD Mailing Lists post – using GPT / EFI booting on Lenovo BIOSeshttps://lists.freebsd.org/pipermail/freebsd-i386/2013-March/010437.html.

One more thing to mention, this GPT BUG is present only when You want to boot FreeBSD from GPT partition in BIOS/Legacy mode. If You make UEFI install (which we will do in one of the next articles in the series) this problem is not present and FreeBSD boots properly on Lenovo ThinkPad X220 (and other from XY20 series) from GPT partition in UEFI mode.

Now the install process. First You need the FreeBSD 11.1-RELEASE install image, download it from here – FreeBSD-11.1-RELEASE-amd64-memstick.imghttp://ftp.freebsd.org/pub/FreeBSD/releases/ISO-IMAGES/11.1/FreeBSD-11.1-RELEASE-amd64-memstick.img.

To put the image on the pendrive we will use the dd(1) tool available on almost any Mac OS X (macOS) and Linux system – You can read more about that command on following Wikipedia article on dd command.

For Windows You will have to download it from here – dd for windowshttp://www.chrysocome.net/dd.

On FreeBSD such command will look like that:

# dd if=FreeBSD-11.1-RELEASE-amd64-memstick.img of=/dev/da1

… but for performance reasons its better to add bs=1m (bs=1M on Linux/Windows) to the command.

# dd if=FreeBSD-11.1-RELEASE-amd64-memstick.img of=/dev/da1 bs=1m

Alternatively You may want to burn the ISO image to the CD-R(W) disc and boot from it instead of USB pendrive, in such case here is the ISO image link – FreeBSD-11.1-RELEASE-amd64-disc1.isohttp://ftp.freebsd.org/pub/FreeBSD/releases/ISO-IMAGES/11.1/FreeBSD-11.1-RELEASE-amd64-disc1.iso

Now to install. Attach the USB pendrive or CD disc with FreeBSD installer and boot from it, on most laptops You may show the temporary boot list selection with [F11] or [F12] key.

After You have booted to the screen below with Welcome message, leave the ‘focus’ on the Install button and attach the 8GB or larger USB pendrive for installation.

freebsd-11.1-laptop-install-01

You will see kernel buffer messages messing the screen and showing that new drive has been attached, but as the ‘focus’ is still on the Install button so just hit [ENTER] key and proceed with the installation process as shown on the images below.

You can select your desired keyboard layout, it can also be changed later, your choice.

freebsd-11.1-laptop-install-02

When we have a new machine there is always a problem with new name for it. The RFC 1178 Choosing a Name for Your Computer from 1990 year tries to address that issue – https://tools.ietf.org/html/rfc1178. For machine hostname we will use Slavic word for the wind which would be vetru – in normalized form – as original as You can imagine is not in ASCII compatible – Π²Ρ£Ρ‚Ρ€ΡŠ (vΔ›trΕ­) πŸ™‚

freebsd-11.1-laptop-install-03

We do not need any additional components, with all deselected as shown below only the (not shown) base and kernel will be installed.

freebsd-11.1-laptop-install-04

We will use ZFS because we want to use Boot Environments with sysutils/beadm port.

freebsd-11.1-laptop-install-05

Hit [ENTER] on the Pool Type/Disks to select target disk to install FreeBSD on.

freebsd-11.1-laptop-install-06

As we will be using only single disk without redundancy select stripe here.

freebsd-11.1-laptop-install-07

This may vary, select the pendrive (or disk) that You want to install FreeBSD on.

freebsd-11.1-laptop-install-08

We will set SWAP size to ‘0‘ (no SWAP) as it will not be needed. If we will need SWAP in the future, then we will create ZVOL on ZFS and use it as a SWAP device.

freebsd-11.1-laptop-install-09

freebsd-11.1-laptop-install-10

As we are installing it on a Lenovo ThnikPad X220 with GPT BUG then we need to select GPT + Lenovo FIX (BIOS) here.

freebsd-11.1-laptop-install-11

Now move ‘focus’ to Install and hit [ENTER] to proceed with installation.

freebsd-11.1-laptop-install-12

This is the last chance to not overwrite your precious data.

freebsd-11.1-laptop-install-13

Then You will see the install progress.

freebsd-11.1-laptop-install-14

Fill in your root user password.

freebsd-11.1-laptop-install-15

Select any interface here as we will not be making any configuration here.

freebsd-11.1-laptop-install-16

Leave the interface unconfigured for now, we will take care about that later.

freebsd-11.1-laptop-install-17

freebsd-11.1-laptop-install-18

Now we will configure the time zone, I have configured Europe/Poland as shown below.

freebsd-11.1-laptop-install-19

freebsd-11.1-laptop-install-20

Now you may correct the date and time.

freebsd-11.1-laptop-install-21

freebsd-11.1-laptop-install-22

Select services as shown below.

freebsd-11.1-laptop-install-23

Enable all security hardening features as shown below.

freebsd-11.1-laptop-install-24

We want to create new ‘regular’ user so answer Yes here.

freebsd-11.1-laptop-install-25

For the username we will use Slavic word for wolf which is Vuk.

freebsd-11.1-laptop-install-26

freebsd-11.1-laptop-install-27

Our job is done here, hit [ENTER] key with ‘focus’ on Exit name,

freebsd-11.1-laptop-install-28

Installer will ask You if You want to modify the installed system before rebooting into it. We do not have any needs now so hit [ENTER] key on No option.

freebsd-11.1-laptop-install-29

As we finished hit [ENTER] on Reboot button.

freebsd-11.1-laptop-install-30

Boot

After the boot process of the newly installed FreeBSD 11.1-RELEASE USB pendrive You will see an image similar to below one.

freebsd-11.1-laptop-install-00

freebsd-11.1-laptop-first-boot

What to do next? Lets make some basic network connectivity. Later in the FreeBSD Desktop series we will switch to network.sh for networking – FreeBSD Network Management with network.sh Script but now lets see how its done the raw way. If You will have attached LAN cable and your interface is em0 (check ifconfig(8) command output) then dhclient em0 command should grant You the working connection to the Internet – assuming that You have DHCP server on that network. Here are the commands.

# ifconfig em0 up

# dhclient em0
DHCPREQUEST on em0 to 255.255.255.255 port 67
DHCPACK from 10.254.17.1
bound to 10.254.17.3 -- renewal in 2147483647 seconds.

To test the network connectivity use the ping(8) command.

# ping -c 3 freebsd.org
PING freebsd.org (8.8.178.110): 56 data bytes
64 bytes from 8.8.178.110: icmp_seq=0 ttl=52 time=242.421 ms
64 bytes from 8.8.178.110: icmp_seq=1 ttl=52 time=234.535 ms
64 bytes from 8.8.178.110: icmp_seq=2 ttl=52 time=245.120 ms

--- freebsd.org ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 234.535/240.692/245.120/4.491 ms

Also check LAN network chapter in the FreeBSD Handbookhttps://freebsd.org/handbook/config-network-setup.html

If You would like to connect to the World with wireless connection then here are the needed commands.

First lets check what wireless card You have.

# sysctl net.wlan.devices
net.wlan.devices: iwn0

We will now create wlan(4) virtual device on top of our iwn0 device and bring it up.

# ifconfig wlan0 create wlandev iwn0

# ifconfig wlan0 up

We can scan for existing nearby WiFi access points if needed.

# ifconfig wlan0 scan
SSID/MESH ID    BSSID              CHAN RATE    S:N     INT CAPS
xxxxxxx         xx:xx:xx:xx:xx:xx    7   54M  -85:-95   100 EPS  HTCAP RSN WME BSSLOAD WPS
WIFI-NETWOR...  xx:xx:xx:xx:xx:xx    8   54M  -71:-95   100 EPS  HTCAP APCHANREP WME ATH RSN WPA
xxxxxxx         xx:xx:xx:xx:xx:xx    1   54M  -90:-95   100 EPS  RSN HTCAP WME ATH WPS
xxxxxxx         xx:xx:xx:xx:xx:xx    1   54M  -90:-95   100 EP   RSN WME WPS

In case the name of the network does not fit into the first column, then use -v flag for verbose (and wider) output.

# ifconfig -v wlan0 scan

Now we need to add the desired WiFi network to the /etc/wpa_supplicant.conf file as shown below.

# cat > /etc/wpa_supplicant.conf << __EOF
network={
  ssid="WIFI-NETWORK-NAME"
  psk="PASSWORD"
}
__EOF

The You may connect to it using the wpa_supplicant(8) daemon. We will connect to the WiFi hot spot, and as the message ‘CTRL-EVENT-CONNECTED‘ will be displayed we hit the [CTRL]+[Z] key combination to put the process into suspended state (SIGTSTP signal). Then we type the bg(1) command to put it back into running state, but in the background (SIGCONT signal) so we can continue to type next commands.

# wpa_supplicant -i wlan0 -c /etc/wpa_supplicant.conf
Successfully initialized wpa_supplicant
wlan0: Trying to associate with xx:xx:xx:xx:xx:xx (SSID='WIFI-NETWORK-NAME' freq=2447 MHz)
wlan0: Associated with xx:xx:xx:xx:xx:xx
wlan0: WPA: Key negotiation completed with xx:xx:xx:xx:xx:xx [PTK=CCMP GTK=TKIP]
wlan0: CTRL-EVENT-CONNECTED - Connection to xx:xx:xx:xx:xx:xx completed [id=17 id_str=]
^Z
[1]  + Suspended  sudo wpa_supplicant -i wlan0 -c /etc/wpa_supplicant.conf
# bg
[1]    sudo wpa_supplicant -i wlan0 -c /etc/wpa_supplicant.conf &

Now we will request for the IP address from the access point DHCP server.

# dhclient wlan0
DHCPREQUEST on wlan0 to 255.255.255.255 port 67
DHCPACK from 10.254.17.1
bound to 10.254.17.8 -- renewal in 2147483647 seconds.

To test the network connectivity use the ping(8) command.

# ping -c 3 freebsd.org
PING freebsd.org (8.8.178.110): 56 data bytes
64 bytes from 8.8.178.110: icmp_seq=0 ttl=52 time=242.421 ms
64 bytes from 8.8.178.110: icmp_seq=1 ttl=52 time=234.535 ms
64 bytes from 8.8.178.110: icmp_seq=2 ttl=52 time=245.120 ms

--- freebsd.org ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 234.535/240.692/245.120/4.491 ms

Also check WLAN chapter in the FreeBSD Handbookhttps://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/network-wireless.html

Knowledge

In case You want to learn more about FreeBSD the FreeBSD Handbook and FreeBSD FAQ are good place to start.

GPT Fix

In case the installer would not made its job properly You can make the fix by yourself, you just need a FreeBSD system for it (You already have it on the USB pendrive or CD disk). Boot into the first question and move the ‘focus’ to the Live CD button on the right but do not hit the [ENTER] yet. Now, attach the USB pendrive with FreeBSD installed. Screen will be flooded with kernel buffer about attaching new da device, note the da device number (for example da1) and now hit [ENTER] key.

You will be welcomed with login: prompt, type root and hit [ENTER] (no password is required).

Remember the kernel buffer messages screen earlier? They started with the device name (like da0 or da1), we will use that device name to apply the fix. I will assume its da0. First, lets dump partition table to FDISK file.

# fdisk -p /dev/da0 > FDISK

# cat FDISK
# /dev/da0
g c1886 h255 s63
p 1 0xee 63 30298527

Now we need to make the fix, after the 'p 1 (...)' line add 'a 1' an below that line copy the line that starts with 'p 1 (...)' and name it 'p 2 (...)'. Now in the ‘first’ line with 'p 1 (...)' change the 0xee into 0x00. Final result is shown below.

# vi FDISK
# /dev/da0
g c1886 h255 s63
p 1 0x00 63 30298527
a 1
p 2 0xee 63 30298527
:wq

After writing the file, apply this new partition table into the da0 device with fdisk(8) command.

# fdisk -f FDISK /dev/da0

Verify that fix is applied.

# fdisk -p /dev/da0
# /dev/da0
g c1886 h255 s63
p 2 0xee 63 30298527

Indeed it is. Now You have Lenovo GPT fix applied. Important to mention that this ‘fix’ does not break the USB pendrive for computers that do not have GPT problem, so with GPT FIX applied you may use it on computers other then Lenovo.

Image

In case You may want to start after the installation I have uploaded the installed image here – http://www.mediafire.com/file/ywcnw99ghqezd6u/usb-freebsd-11.1-lenovo.raw.xz – keep in mind that its compressed by xz(1) so in order to apply it to USB pendrive You must first unpack it. You may do that on the fly with following command assuming that we will be writing that image to da1 device:

# xz -c -d usb-freebsd-11.1-lenovo.raw.xz | dd > /dev/da1 bs=1m

For the record, this image contains the Lenovo GPT FIX. The passwords are set to ‘asd‘ for convenience.

Enough for one post πŸ˜‰

UPDATE 1 – Lenovo GPT Fix with gpart(8)

Initially I described how to apply a fix for broken Lenovo GPT bug with fdisk(8) tool. This is very tricky and error prone but there is a lot easier way to make the same fix. Its made with other FreeBSD tool called gpart(8) and its one line only.

# gpart set -a lenovofix /dev/ada0
lenovofix set on ada0

Done. No longer the need to dance with fdisk(8) tool.

EOF

Valuable News – 2018/04/08

First some news about storage. You think Dell EMC XtremeIO is fast? Try Datrium DVX which is at least 3.7 times faster then biggest Dell EMC XtremeIO cluster money can buy.

What Separates the Storage Industry’s Men from Boys? Random Writes!
https://www.datrium.com/blog/what-separates-storage-industrys-men-boys/

Why should I use DuckDuckGo instead of Google?
https://www.quora.com/Why-should-I-use-DuckDuckGo-instead-of-Google

Fifty (or Sixty) Years of Processor Development … for This?
https://www.eejournal.com/article/fifty-or-sixty-years-of-processor-developmentfor-this/

… a complete list of all FreeBSD Port maintainers.
https://portscout.freebsd.org/

Repository statistics from Repology with FreeBSD on top πŸ˜‰
https://repology.org/statistics/newest

Note on SYSVIPC and Jails on FreeBSD
https://www.skyforge.at/posts/a-note-in-sysvipc-and-jails-on-freebsd/

Introduction to Jails and Jail Networking
https://www.skyforge.at/posts/an-introduction-to-jails-and-jail-networking/

Life is Short
http://www.paulgraham.com/vb.html

In Other BSDs for 2018/04/07
https://www.dragonflydigest.com/2018/04/07/21080.html

Using FreeBSD Text Dumps
http://www.etinc.com/122/Using-FreeBSD-Text-Dumps

EOF.

.
.
.
.
.

Introduction to HardenedBSD World

HardenedBSD is a security enhanced fork of FreeBSD which happened in 2014. HardenedBSD is implementing many exploit mitigation and security technologies on top of FreeBSD which all started with implementation of Address Space Layout Randomization (ASLR). The fork has been created for ease of development.

To cite the https://hardenedbsd.org/content/about page – “HardenedBSD aims to implement innovative exploit mitigation and security solutions for the FreeBSD community. (…) HardenedBSD takes a holistic approach to security by hardening the system and implementing exploit mitigation technologies.”

Most FreeBSD enthusiasts know mfsBSD project by Martin Matuska – http://mfsbsd.vx.sk/ – FreeBSD system loaded completely into memory. The mfsBSD synonym for the HardenedBSD world is SoloBSD – https://www.solobsd.org/ – which is based on HardenedBSD sources.

SoloBSD.Boot.Menu

One may ask how HardenedBSD project compared to more well know for its security OpenBSD system and it is very important question. The OpenBSD developers try to write ‘good’ code without dirty hacks for performance or other reasons. Clean and secure code is most important in OpenBSD world. The OpenBSD project even made security audit of all OpenBSD code available, line by line. This was easier to achieve in FreeBSD or HardenedBSD because OpenBSD code base its about ten times smaller. This has also other implications, possibilities. While FreeBSD (and HardenedBSD) offer many new features like mature SMP subsystem even with some NUMA support, ZFS filesystem, GEOM storage framework, Bhyve virtualization, Virtualbox option and many other new modern features the OpenBSD remains classic UNIX system with UFS filesystem and with very ‘theoretical’ SMP support. The vmm project tried to implement new hypervisor in OpenBSD world, but because of lack of support for graphics its for OpenBSD, Illumos and Linux currently, You will not virtualize Windows or Mac OS X there. This is also only virtualization option for OpenBSD as there are no Jails on OpenBSD. Current Bhyve implementation allows one even to boot latest Windows 2019 Technology Preview.

A HardenedBSD project is FreeBSD system code base with LOTS of security mechanisms and mitigations that are not available on FreeBSD system. For example entire lib32 tree has been disabled by default on HardenedBSD to make it more secure. Also LibreSSL is the default SSL library on HardenedBSD, same as OpenBSD while FreeBSD uses OpenSSL for compatibility reasons.

Comparison between LibreSSL and OpenSSL vulnerabilities.

One may see HardenedBSD as FreeBSD being successfully pulled up to the OpenBSD level (at least that is the goal), but as FreeBSD has tons more code and features it will be harder and longer process to achieve the goal.

As I do not have that much competence on the security field I will just repost the comparison from the HardenedBSD project versus other BSD systems. The comparison is also available here – https://hardenedbsd.org/content/easy-feature-comparison – on the HardenedBSD website.

HardenedBSD.Easy.Feature.Comparison

Install

The installation is almost identical to the FreeBSD system, an example installation of HardenedBSD system (on ZFS with Boot Environments) is shown below.

HardenedBSD-install-00

HardenedBSD-install-01

HardenedBSD-install-02

HardenedBSD-install-03

HardenedBSD-install-04

HardenedBSD-install-05

HardenedBSD-install-06

HardenedBSD-install-07

HardenedBSD-install-08

HardenedBSD-install-09

HardenedBSD-install-10

HardenedBSD-install-11

HardenedBSD-install-12

HardenedBSD-install-13

HardenedBSD-install-14

HardenedBSD-install-15

HardenedBSD-install-16

HardenedBSD-install-17

HardenedBSD-install-18

HardenedBSD-install-19

HardenedBSD-install-20

HardenedBSD-install-21

HardenedBSD-install-22

HardenedBSD-install-23

HardenedBSD-install-24

HardenedBSD-install-25

HardenedBSD-install-26

HardenedBSD-install-31

First Login

This is how just installed HardenedBSD system looks like.

% ssh notme@localhost
Password for root@hardenedbsd.local:
FreeBSD 11.1-STABLE-HBSD (HARDENEDBSD) #0 [STABLE:HardenedBSD-11-STABLE-v1100054.1]: Thu Nov 30 03:11:44 UTC 2017

+------------------------------------------------------------------------------+
|                                                                              |
|                             Welcome to HardenedBSD!                          |
|                                                                              |
|       _    _               _                     _ ____   _____ _____        |
|      | |  | |             | |                   | |  _ \ / ____|  __ \       |
|      | |__| | __ _ _ __ __| | ___ _ __   ___  __| | |_) | (___ | |  | |      |
|      |  __  |/ _` | '__/ _` |/ _ \ '_ \ / _ \/ _` |  _ < \___ \| |  | |      |
|      | |  | | (_| | | | (_| |  __/ | | |  __/ (_| | |_) |____) | |__| |      |
|      |_|  |_|\__,_|_|  \__,_|\___|_| |_|\___|\__,_|____/|_____/|_____/       |
|                                                                              |
+------------------------------------------------------------------------------+
|     keyword: sysctl, secadm, git, github.com/hardenedbsd hardenedbsd.org     |
+------------------------------------------------------------------------------+
                Edit /etc/motd to change this login announcement.               

root@hardenedbsd:~ # 

ZFS Boot Environments

We can use pkg(8) as usual and as I intentionally not installed the latest version of HardenedBSD the pkg(8) warns about possible compatibility issues. As sysutils/beadm is just a shell script I would install it anyway.

root@hardenedbsd:~ # pkg install beadm
The package management tool is not yet installed on your system.
Do you want to fetch and install it now? [y/N]: y
Bootstrapping pkg from pkg+http://pkgs.hardenedbsd.org/HardenedBSD/pkg/FreeBSD:11:amd64, please wait...
Verifying signature with trusted certificate pkg.hardenedbsd.org.2014-09-04... done
Installing pkg-1.10.5...
Newer FreeBSD version for package pkg:
To ignore this error set IGNORE_OSVERSION=yes
- package: 1101512
- running kernel: 1101506
Allow missmatch now?[Y/n]: y
Extracting pkg-1.10.5: 100%
Updating HardenedBSD repository catalogue...
pkg: Repository HardenedBSD load error: access repo file(/var/db/pkg/repo-HardenedBSD.sqlite) failed: No such file or directory
Fetching meta.txz: 100%    1 KiB   1.5kB/s    00:01    
Fetching packagesite.txz: 100%    6 MiB 628.8kB/s    00:10    
Processing entries:   0%
Newer FreeBSD version for package p5-Statistics-Frequency:
To ignore this error set IGNORE_OSVERSION=yes
- package: 1101512
- running kernel: 1101506
Allow missmatch now?[Y/n]: y
Processing entries: 100%
HardenedBSD repository update completed. 30459 packages processed.
All repositories are up to date.
Updating database digests format: 100%
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        beadm: 1.2.7_4 [HardenedBSD]

Number of packages to be installed: 1

9 KiB to be downloaded.

Proceed with this action? [y/N]: y
[1/1] Fetching beadm-1.2.7_4.txz: 100%    9 KiB   9.6kB/s    00:01    
Checking integrity... done (0 conflicting)
[1/1] Installing beadm-1.2.7_4...
[1/1] Extracting beadm-1.2.7_4: 100%
root@hardenedbsd:~ # beadm list
BE      Active Mountpoint  Space Created
default NR     /          426.0M 2018-04-05 20:24

Same as FreeBSD the HardenedBSD system comes with ‘crippled’ system layout when it comes to its usability under ZFS Boot Environments. The problem is in /var and /usr filesystems/datasets NOT being placed under the pool/ROOT/bename path so they will be omitted when new Boot Environments is created. This makes beadm (and whole Boot Environments idea) quite uselss as packages and base system userspace under /usr/local and /usr respectively along with /var/db/pkg installed packages information and other ‘databases’ are not protected by it. But with two commands it is very easy to fix that ‘crippled’ setup even on a running system without unmounting anything.

The default HardenedBSD (and FreeBSD) ‘crippled’ system layout looks as follows.

root@hardenedbsd:~ # zfs list
NAME                 USED  AVAIL  REFER  MOUNTPOINT
zroot                428M  15.0G    88K  /zroot
zroot/ROOT           426M  15.0G    88K  none
zroot/ROOT/default   426M  15.0G   426M  /
zroot/tmp             88K  15.0G    88K  /tmp
zroot/usr            352K  15.0G    88K  /usr
zroot/usr/home        88K  15.0G    88K  /usr/home
zroot/usr/ports       88K  15.0G    88K  /usr/ports
zroot/usr/src         88K  15.0G    88K  /usr/src
zroot/var            572K  15.0G    88K  /var
zroot/var/audit       88K  15.0G    88K  /var/audit
zroot/var/crash       88K  15.0G    88K  /var/crash
zroot/var/log        132K  15.0G   132K  /var/log
zroot/var/mail        88K  15.0G    88K  /var/mail
zroot/var/tmp         88K  15.0G    88K  /var/tmp

With these two commands we move the /usr and /var filesystems/datasets under the pool/ROOT/bename so when new Boot Environments are created they will be covered and protected by Boot Environment.

root@hardenedbsd:~ # zfs rename -u zroot/usr zroot/ROOT/default/usr
root@hardenedbsd:~ # zfs rename -u zroot/var zroot/ROOT/default/var

New layout after the fix is shown below.

root@hardenedbsd:~ # zfs list
NAME                           USED  AVAIL  REFER  MOUNTPOINT
zroot                          428M  15.0G    88K  /zroot
zroot/ROOT                     427M  15.0G    88K  none
zroot/ROOT/default             426M  15.0G   426M  /
zroot/ROOT/default/usr         352K  15.0G    88K  /usr
zroot/ROOT/default/usr/home     88K  15.0G    88K  /usr/home
zroot/ROOT/default/usr/ports    88K  15.0G    88K  /usr/ports
zroot/ROOT/default/usr/src      88K  15.0G    88K  /usr/src
zroot/ROOT/default/var         572K  15.0G    88K  /var
zroot/ROOT/default/var/audit    88K  15.0G    88K  /var/audit
zroot/ROOT/default/var/crash    88K  15.0G    88K  /var/crash
zroot/ROOT/default/var/log     132K  15.0G   132K  /var/log
zroot/ROOT/default/var/mail     88K  15.0G    88K  /var/mail
zroot/ROOT/default/var/tmp      88K  15.0G    88K  /var/tmp
zroot/tmp                       88K  15.0G    88K  /tmp

Base System Update

While FreeBSD uses freebsd-update for base system updates HardenedBSD project uses its own hbsd-update tool that does not rely on delta patches.

root@hardenedbsd:~ # freebsd-update
freebsd-update: Command not found.

The hbsd-update tools has nice feature to make update in a new separate Boot Environment to which you can reboot while leaving the running system untouched. That way You can go back to not upgraded system anytime if anything would went wrong in the update procedure or after the update itself.

As I installed older 1100054.1 version we will now make an update to the latest 1100055 version.

root@hardenedbsd:~ # hbsd-update -I -V -b updated
[*] Latest build: hbsd-v1100055-069a9206df22a498095e5e20f5ee28b9fd859080
/tmp/tmp.wP3U86Ci/update.tar                  100% of  299 MB  434 kBps 11m46s
[*] Verified hash: 39e2c6a4c8a8387bf4091584bd029aad615378e49206ad1f863bd3602a77cdb7 = 39e2c6a4c8a8387bf4091584bd029aad615378e49206ad1f863bd3602a77cdb7
[*] Checking validity of the public key
[*] Checking the validity of base.txz
[*] Checking the validity of etcupdate.tbz
[*] Checking the validity of skip.txt
[*] Checking the validity of kernel-HARDENEDBSD.txz
[*] Checking the validity of ObsoleteFiles.txt
[*] Checking the validity of ObsoleteDirs.txt
[*] Checking the validity of script.sh
[*] Checking the validity of secadm.integriforce.rules
******************
* IMPORTANT NOTE *
******************

This update includes the PTI patch. Third-party kernel modules (such
as x11/nvidia-driver and hardenedbsd/secadm-kmod) will need to be
recompiled/reinstalled.

If you wish to postpone installing this update, please hit Control-C
within the next ten (10) seconds.
Created successfully
Mounted successfully on '/tmp/tmp.tFMmByeO'
[*] Applying base
[*] Updating /etc
[*] Manual merges need to be done.
Resolving conflict in '/etc/motd':
Select: (p) postpone, (df) diff-full, (e) edit,
        (h) help for more options: h
  (p)  postpone    - ignore this conflict for now
  (df) diff-full   - show all changes made to merged file
  (e)  edit        - change merged file in an editor
  (r)  resolved    - accept merged version of file
  (mf) mine-full   - accept local version of entire file (ignore new changes)
  (tf) theirs-full - accept new version of entire file (lose local changes)
  (h)  help        - show this list
Select: (p) postpone, (df) diff-full, (e) edit,
        (h) help for more options: mf
Resolving conflict in '/etc/periodic/daily/200.backup-passwd':
Select: (p) postpone, (df) diff-full, (e) edit,
        (h) help for more options: tf
Resolving conflict in '/etc/pf.os':
Select: (p) postpone, (df) diff-full, (e) edit,
        (h) help for more options: tf
Resolving conflict in '/etc/rc.initdiskless':
Select: (p) postpone, (df) diff-full, (e) edit,
        (h) help for more options: tf
Resolving conflict in '/etc/devd/usb.conf':
Select: (p) postpone, (df) diff-full, (e) edit,
        (h) help for more options: tf
Resolving conflict in '/etc/rc.firewall':
Select: (p) postpone, (df) diff-full, (e) edit,
        (h) help for more options: tf
Resolving conflict in '/etc/mtree/BSD.root.dist':
Select: (p) postpone, (df) diff-full, (e) edit,
        (h) help for more options: tf
Resolving conflict in '/etc/mtree/BSD.debug.dist':
Select: (p) postpone, (df) diff-full, (e) edit,
        (h) help for more options: tf
Resolving conflict in '/etc/mtree/BSD.usr.dist':
Select: (p) postpone, (df) diff-full, (e) edit,
        (h) help for more options: tf
Resolving conflict in '/etc/mtree/BSD.tests.dist':
Select: (p) postpone, (df) diff-full, (e) edit,
        (h) help for more options: tf
Resolving conflict in '/etc/mtree/BSD.include.dist':
Select: (p) postpone, (df) diff-full, (e) edit,
        (h) help for more options: tf
Resolving conflict in '/etc/rc.d/ntpd':
Select: (p) postpone, (df) diff-full, (e) edit,
        (h) help for more options: tf
Resolving conflict in '/etc/rc.d/fsck':
Select: (p) postpone, (df) diff-full, (e) edit,
        (h) help for more options: tf
Resolving conflict in '/etc/rc.d/ipsec':
Select: (p) postpone, (df) diff-full, (e) edit,
        (h) help for more options: tf
Resolving conflict in '/etc/rc.d/pf':
Select: (p) postpone, (df) diff-full, (e) edit,
        (h) help for more options: tf
Resolving conflict in '/etc/rc.d/sendmail':
Select: (p) postpone, (df) diff-full, (e) edit,
        (h) help for more options: tf
Resolving conflict in '/etc/rc.d/ipfw':
Select: (p) postpone, (df) diff-full, (e) edit,
        (h) help for more options: tf
Resolving conflict in '/etc/services':
Select: (p) postpone, (df) diff-full, (e) edit,
        (h) help for more options: tf
Resolving conflict in '/etc/regdomain.xml':
Select: (p) postpone, (df) diff-full, (e) edit,
        (h) help for more options: tf
Resolving conflict in '/etc/sysctl.conf':
Select: (p) postpone, (df) diff-full, (e) edit,
        (h) help for more options: tf
Resolving conflict in '/etc/rc.subr':
Select: (p) postpone, (df) diff-full, (e) edit,
        (h) help for more options: tf
Resolving conflict in '/etc/mail/mailer.conf':
Select: (p) postpone, (df) diff-full, (e) edit,
        (h) help for more options: tf
Resolving conflict in '/etc/ssh/sshd_config':
Select: (p) postpone, (df) diff-full, (e) edit,
        (h) help for more options: tf
Resolving conflict in '/etc/master.passwd':
Select: (p) postpone, (df) diff-full, (e) edit,
        (h) help for more options: h
  (p)  postpone    - ignore this conflict for now
  (df) diff-full   - show all changes made to merged file
  (e)  edit        - change merged file in an editor
  (r)  resolved    - accept merged version of file
  (mf) mine-full   - accept local version of entire file (ignore new changes)
  (tf) theirs-full - accept new version of entire file (lose local changes)
  (h)  help        - show this list
Select: (p) postpone, (df) diff-full, (e) edit,
        (h) help for more options: df
--- /tmp/tmp.tFMmByeO/etc/master.passwd 2018-04-05 20:25:29.869447000 +0200
+++ /tmp/tmp.tFMmByeO/var/db/etcupdate/conflicts/etc/master.passwd      2018-04-06 09:36:06.231096000 +0200
@@ -1,6 +1,10 @@
 # $FreeBSD$
 #
+ (stock)
 toor:*:0:0::0:0:Bourne-again Superuser:/root:
 daemon:*:1:1::0:0:Owner of many system processes:/root:/usr/sbin/nologin
 operator:*:2:5::0:0:System &:/:/usr/sbin/nologin
Select: (p) postpone, (df) diff-full, (e) edit,
        (h) help for more options: mf
Resolving conflict in '/etc/devd.conf':
Select: (p) postpone, (df) diff-full, (e) edit,
        (h) help for more options: tf
Resolving conflict in '/etc/defaults/rc.conf':
Select: (p) postpone, (df) diff-full, (e) edit,
        (h) help for more options: tf
[*] Updating the password database
[*] Applying kernel HARDENEDBSD
[*] Removing obsolete files
Remove /tmp/tmp.tFMmByeO/boot/pcibios.4th (Y/n)? y
    [+] Removing /tmp/tmp.tFMmByeO/boot/pcibios.4th
Remove /tmp/tmp.tFMmByeO/usr/lib/librt_p.a (Y/n)? y
    [+] Removing /tmp/tmp.tFMmByeO/usr/lib/librt_p.a
[*] Applying Integriforce rules
Unmounted successfully
Activated successfully

The update procedure is now finished, new Boot Environment is present and already set as Activated upon reboot.

root@hardenedbsd:~ # beadm list
BE      Active Mountpoint  Space Created
default N      /           95.4M 2018-04-05 20:24
updated R      -          887.4M 2018-04-06 09:35

Before rebooting into the updated system we may want to modify some files. Lets mount that Boot Environment for that purpose.

root@hardenedbsd:~ # beadm mount updated
Mounted successfully on '/tmp/BE-updated.8sbzisOh'

root@hardenedbsd:~ # beadm list
BE      Active Mountpoint                Space Created
default N      /                         95.5M 2018-04-05 20:24
updated R      /tmp/BE-updated.8sbzisOh 887.4M 2018-04-06 09:35

root@hardenedbsd:~ # cd /tmp/BE-updated.8sbzisOh

// MAKE NEEDED CHANGES BEFORE REBOOT

root@hardenedbsd:/tmp/BE-updated.8sbzisOh # cd

root@hardenedbsd:~ # beadm umount updated
Unmounted successfully

root@hardenedbsd:~ # shutdown -r now

After the reboot we can see that our HardenedBSD system indeed is upgraded to newer version.

root@hardenedbsd:~ # sysctl hardening.version
hardening.version: 1100055

The pkg(8) does not warn now about possible incompatibilities because we were using older HardenedBSD version.

root@hardenedbsd:~ # pkg install beadm
The package management tool is not yet installed on your system.
Do you want to fetch and install it now? [y/N]: y
Bootstrapping pkg from pkg+http://pkgs.hardenedbsd.org/HardenedBSD/pkg/FreeBSD:11:amd64, please wait...
Verifying signature with trusted certificate pkg.hardenedbsd.org.2014-09-04... done
Installing pkg-1.10.5...
Extracting pkg-1.10.5: 100%
Updating HardenedBSD repository catalogue...
pkg: Repository HardenedBSD load error: access repo file(/var/db/pkg/repo-HardenedBSD.sqlite) failed: No such file or directory
Fetching meta.txz: 100%    1 KiB   1.5kB/s    00:01    
Fetching packagesite.txz: 100%    6 MiB 224.6kB/s    00:28    
Processing entries: 100%
HardenedBSD repository update completed. 30459 packages processed.
All repositories are up to date.
Updating database digests format: 100%
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        beadm: 1.2.7_4 [HardenedBSD]

Number of packages to be installed: 1

9 KiB to be downloaded.

Proceed with this action? [y/N]: y
[1/1] Fetching beadm-1.2.7_4.txz: 100%    9 KiB   9.6kB/s    00:01    
Checking integrity... done (0 conflicting)
[1/1] Installing beadm-1.2.7_4...
[1/1] Extracting beadm-1.2.7_4: 100%
root@hardenedbsd:~ # beadm list
BE      Active Mountpoint  Space Created
default -      -           95.6M 2018-04-05 20:24
updated NR     /          928.1M 2018-04-06 09:35

As we see we still can get back to older Boot Environment with out 1100054.1 HardenedBSD system if needed.

Here are datasets for both systems.

root@hardenedbsd:~ # zfs list
NAME                           USED  AVAIL  REFER  MOUNTPOINT
zroot                          930M  14.5G    88K  /zroot
zroot/ROOT                     928M  14.5G    88K  none
zroot/ROOT/default             364K  14.5G   426M  /
zroot/ROOT/default/usr            0  14.5G    88K  /usr
zroot/ROOT/default/usr/home       0  14.5G    88K  /usr/home
zroot/ROOT/default/usr/ports      0  14.5G    88K  /usr/ports
zroot/ROOT/default/usr/src        0  14.5G    88K  /usr/src
zroot/ROOT/default/var         132K  14.5G    88K  /var
zroot/ROOT/default/var/audit      0  14.5G    88K  /var/audit
zroot/ROOT/default/var/crash      0  14.5G    88K  /var/crash
zroot/ROOT/default/var/log      76K  14.5G   132K  /var/log
zroot/ROOT/default/var/mail       0  14.5G    88K  /var/mail
zroot/ROOT/default/var/tmp      56K  14.5G    88K  /var/tmp
zroot/ROOT/updated             927M  14.5G   497M  /
zroot/ROOT/updated/usr         300M  14.5G   300M  /usr
zroot/ROOT/updated/usr/home     88K  14.5G    88K  /usr/home
zroot/ROOT/updated/usr/ports    88K  14.5G    88K  /usr/ports
zroot/ROOT/updated/usr/src     144K  14.5G    88K  /usr/src
zroot/ROOT/updated/var        36.0M  14.5G  35.2M  /var
zroot/ROOT/updated/var/audit   144K  14.5G    88K  /var/audit
zroot/ROOT/updated/var/crash   144K  14.5G    88K  /var/crash
zroot/ROOT/updated/var/log     216K  14.5G   132K  /var/log
zroot/ROOT/updated/var/mail    144K  14.5G    88K  /var/mail
zroot/ROOT/updated/var/tmp     144K  14.5G    88K  /var/tmp
zroot/tmp                       88K  14.5G    88K  /tmp

We will now destroy the older 1100054.1 HardenedBSD system as its no longer needed.

root@hardenedbsd:~ # beadm destroy default
Are you sure you want to destroy 'default'?
This action cannot be undone (y/[n]): y
Boot environment 'default' was created from existing snapshot
Destroy 'updated@2018-04-06-09:35:01' snapshot? (y/[n]): y
Destroyed successfully

Lets check if we are running the latest version.

root@hardenedbsd:~ # hbsd-update -I -V -b new
[*] This system is already on the latest version.

HardenedBSD Ports

To get general idea, if software is in FreeBSD Ports, then it would be in HardenedBSD Ports. Both FreeBSD and HardenedBSD provide packages built from their ports trees, but not all FreeBSD packages available on FreeBSD are available on HardenedBSD because some fail to build against LibreSSL (FreeBSD still has OpenSSL in base) and other fail to build because of HardenedBSD security mechanisms and mitigations enabled. There is also one counter example to that rule, there is audio/lame package in HardenedBSD while there is none on FreeBSD.

The good thing about HardenedBSD community is their response speed. For example I had a problem with sysutils/bareos-client port that failed to built – bareos-client fails to buildhttps://groups.google.com/a/hardenedbsd.org/forum/#!topic/users/xop4rVGVRC4 – and within hours they modified the port to allow me built it against GnuTLS – Bug 227318 – sysutils/bareos-server: Add GNUTLS optionhttps://bugs.freebsd.org/bugzilla/show_bug.cgi?id=227318 – which worked like a charm. It was that fast and accurate, that even overpriced enterprise support from Oracle or Dell EMC does not work that fast and that well.

Same as with freebsd-update tool the HardenedBSD project does not use the portsnap tool.

root@hardenedbsd:~ # portsnap fetch extract
portsnap: Command not found.

The git tool is used instead. You first need to add it from pkg(8) repository as any other package, as shown below.

root@hardenedbsd:~ # pkg install git-lite

Now You may want to use git to fetch the HardenedBSD Ports tree, one thing that I see not convenient is that You must KNOW the GitHub page to type, same as with OpenBSD for packages. I like the FreeBSD approach here more that I do not have to remember that.

root@hardenedbsd:~ # git clone --depth=1 https://github.com/HardenedBSD/hardenedbsd-ports.git /usr/ports
Cloning into '/usr/ports'...
remote: Counting objects: 170097, done.
remote: Compressing objects: 100% (155861/155861), done.
remote: Total 170097 (delta 11341), reused 115842 (delta 9658), pack-reused 0
Receiving objects: 100% (170097/170097), 64.51 MiB | 335.00 KiB/s, done.
Resolving deltas: 100% (11341/11341), done.
Checking out files: 100% (135035/135035), done.

Lets verify the contents.

root@hardenedbsd:~ # find /usr/ports | head -20
.
./www
./www/ocaml-net
./www/ocaml-net/files
./www/ocaml-net/files/patch-Makefile.rules
./www/ocaml-net/distinfo
./www/ocaml-net/Makefile
./www/ocaml-net/pkg-descr
./www/pear-Services_TinyURL
./www/pear-Services_TinyURL/distinfo
./www/pear-Services_TinyURL/Makefile
./www/pear-Services_TinyURL/pkg-descr
./www/grafana5
./www/grafana5/files
./www/grafana5/files/grafana.in
./www/grafana5/files/grafana.conf.in
./www/grafana5/pkg-plist
./www/grafana5/pkg-descr
./www/grafana5/Makefile
./www/grafana5/distinfo

Next updates will be done using the git pull command.

root@hardenedbsd:/usr/ports # git pull
Already up to date.

The configured HardenedBSD system I use here has 1 GB RAM, but that with ZFS ARC default size is not not be enough for git to run.

root@hardenedbsd:~ # grep git /var/log/messages 
Apr  6 14:47:42 hardenedbsd kernel: [18059] pid 32538 (git), uid 0, was killed: out of swap space

To increase amount of RAM for ‘processes’ and to take away some ZFS cache (ARC) put these into the /boot/loader.conf file and reboot the system for the changes to take effect. The size of RAM that is known to not have issues with git memory usage is 2 GB.

root@hardenedbsd:~ # cat >> /boot/loader.conf << __EOF
# ZFS
vfs.zfs.arc_min: 1M
vfs.zfs.arc_max: 32M
__EOF

root@hardenedbsd:~ # shutdown -r now

HardenedBSD Base System Sources

The HardenedBSD base system sources are also fetched with git tool. Below is an example command that fetches the sources.

root@hardenedbsd:~ # git clone --depth=1 --single-branch --branch hardened/11-stable/master https://github.com/hardenedbsd/hardenedbsd-stable/ /usr/src
Cloning into '/usr/src'...
remote: Counting objects: 2314758, done.
remote: Compressing objects: 100% (4207/4207), done.
remote: Total 2314758 (delta 53355), reused 51865 (delta 51526), pack-reused 2258987
Receiving objects: 100% (2314758/2314758), 1.26 GiB | 75.00 KiB/s, done.
Resolving deltas: 100% (1769344/1769344), done.
root@hardenedbsd:~ #

Security Administration (secadm)

The HardenedBSD secadm tool allows users to toggle exploit mitigations on a per application and per jail basis. Users will typically use secadm to disable PAGEEXEC and/or MPROTECT restrictions or to use Integriforce – implementation of verified execution. It enforces hash based signatures for binaries and their dependent shared objects.

First, lets add secadm tool from the pkg(8) repository.

root@hardenedbsd:~ # pkg search secadm
secadm-0.5.1                   HardenedBSD Security Administration
secadm-kmod-0.5.1              HardenedBSD Security Administration

root@hardenedbsd:~ # pkg install secadm secadm-kmod
Updating HardenedBSD repository catalogue...
HardenedBSD repository is up to date.
All repositories are up to date.
The following 3 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        secadm: 0.5.1 [HardenedBSD]
        secadm-kmod: 0.5.1 [HardenedBSD]
        libucl: 0.8.0 [HardenedBSD]

Number of packages to be installed: 3

166 KiB to be downloaded.

Proceed with this action? [y/N]: y
[1/3] Fetching secadm-0.5.1.txz: 100%   53 KiB  54.6kB/s    00:01    
[2/3] Fetching secadm-kmod-0.5.1.txz: 100%   12 KiB  12.7kB/s    00:01    
[3/3] Fetching libucl-0.8.0.txz: 100%  100 KiB 102.6kB/s    00:01    
Checking integrity... done (0 conflicting)
[1/3] Installing libucl-0.8.0...
[1/3] Extracting libucl-0.8.0: 100%
[2/3] Installing secadm-0.5.1...
[2/3] Extracting secadm-0.5.1: 100%
[3/3] Installing secadm-kmod-0.5.1...
[3/3] Extracting secadm-kmod-0.5.1: 100%
Message from secadm-0.5.1:

======================================================================

When you running on custom kernel config, please consult with the 
kernel's source tree, especially with UPDATING-HardenedBSD file.

If you have any other question, you can inform on FreeNode's IRC
#hardenedbsd channel.

Keywords:

 options PAX_CONTROL_ACL
 options PAX_CONTROL_ACL_OVERRIDE_SUPPORT
 options PAX_CONTROL_EXTATTR

 hbsdcontrol secadm

 https://github.com/HardenedBSD/hardenedBSD/blob/hardened/current/master/UPDATING-HardenedBSD#L1

======================================================================
Message from secadm-kmod-0.5.1:

======================================================================

When you running on custom kernel config, please consult with the 
kernel's source tree, especially with UPDATING-HardenedBSD file.

If you have any other question, you can inform on FreeNode's IRC
#hardenedbsd channel.

Keywords:

 options PAX_CONTROL_ACL
 options PAX_CONTROL_ACL_OVERRIDE_SUPPORT
 options PAX_CONTROL_EXTATTR

 hbsdcontrol secadm

 https://github.com/HardenedBSD/hardenedBSD/blob/hardened/current/master/UPDATING-HardenedBSD#L1

======================================================================

Now lets create a simple Integriforce rule for the /bin/dd command.

root@hardenedbsd:~ # cat >> /usr/local/etc/secadm.rules << __EOF
secadm {
  integriforce {
    path: "/bin/dd",
    hash: "72be7c66d4b0a7b776bfac314310edc7423fc251666d548ccde8bf3f9b5b37af",
    type: "sha256",
    mode: "hard",
  }
}
__EOF

Lets enable the secadm in /etc/rc.conf file.

root@hardenedbsd:~ # sysrc secadm_enable=YES

… and finally lets start security administration mechainsm.

root@hardenedbsd:~ # /usr/local/etc/rc.d/secadm start
Starting secadm.

Now lets verify that it really works. Lets try to modify the /bin/dd command.

root@hardenedbsd:~ # echo 1 >> /bin/dd
/bin/dd: Operation not permitted.

root@hardenedbsd:~ # tail -1 /var/log/messages 
Apr 6 12:44:28 hardenedbsd kernel: [10664] [SECADM] Prevented modification of (/bin/dd): protected by a SECADM rule.

But the /bin/dd command works as usual.

root@hardenedbsd:~ # /bin/dd  FILE bs=1m count=1
1+0 records in
1+0 records out
1048576 bytes transferred in 0.039543 secs (26517139 bytes/sec)

root@hardenedbsd:~ # ls -lh FILE
-rw-r--r--  1 root  wheel   1.0M Apr  6 12:06 FILE

Resources

The HardenedBSD Handbook is mainly copied FreeBSD Handbook with Chapter 14. HardenedBSD which is related to HardenedBSD. This is good because HardenedBSD is generally a modified FreeBSD system, so most things work the same way.
https://hardenedbsd.org/~shawn/hbsd_handbook/book.html#hardenedbsd-secadm

The HardenedBSD Forum is available on Google Groups.
https://groups.google.com/a/hardenedbsd.org/forum/#!forum/users

List of HardenedBSD applications that need custom secadm rules is available here.
https://github.com/HardenedBSD/hardenedBSD/wiki/Non-Compliant-Applications

There is also whole github page with secadm rules.
https://github.com/HardenedBSD/secadm-rules

The Twitter accounts for both HardenedBSD and SoloBSD.
https://twitter.com/HardenedBSD
https://twitter.com/SoloBSD

OPNsense Connection

One last thing to notice is the OPNsense connection broadly described here.
https://wiki.opnsense.org/relations/hardenedBSD.html
https://hardenedbsd.org/article/shawn-webb/2015-06-10/first-official-opnsense-images-hardenedbsd

UPDATE 1 – HardenedBSD Switching Back to OpenSSL

To cite the HardenedBSD project site:

Over a year ago, HardenedBSD switched to LibreSSL as the default cryptographic library in base for 12-CURRENT. 11-STABLE followed suit later on. Bernard Spil has done an excellent job at keeping our users up-to-date with the latest security patches from LibreSSL.

After recently updating 12-CURRENT to LibreSSL 2.7.2 from 2.6.4, it has become increasingly clear to us that performing major upgrades requires a team larger than a single person. Upgrading to 2.7.2 caused a lot of fallout in our ports tree. As of 28 Apr 2018, several ports we consider high priority are still broken. As it stands right now, it would take Bernard a significant amount of his spare personal time to fix these issues.

Until we have a multi-person team dedicated to maintaining LibreSSL in base along with the patches required in ports, HardenedBSD will use OpenSSL going forward as the default crypographic library in base. LibreSSL will co-exist with OpenSSL in the source tree, as it does now. However, MK_LIBRESSL will default to “no” instead of the current “yes”. Bernard will continue maintaining LibreSSL in base along with addressing the various problematic ports entries.

To provide our users with ample time to plan and perform updates, we will wait a period of two months prior to making the switch. The switch will occur on 01 Jul 2018 and will be performed simultaneously in 12-CURRENT and 11-STABLE. HardenedBSD will archive a copy of the LibreSSL-centric package repositories and binary updates for base for a period of six months after the switch (expiring the package repos on 01 Jan 2019). This essentially gives our users eight full months for an upgrade path.

As part of the switch back to OpenSSL, the default NTP daemon in base will switch back from OpenNTPd to ISC NTP. Users who have local_openntpd_enable=”YES” set in rc.conf will need to switch back to ntpd_enable=”YES”.

Users who build base from source will want to fully clean their object directories. Any and all packages that link with libcrypto or libssl will need to be rebuilt or reinstalled.

With the community’s help, we look forward to the day when we can make the switch back to LibreSSL. We at HardenedBSD believe that providing our users options to rid themselves of software monocultures can better increase security and manage risk.

UPDATE 2

The Introduction to HardenedBSD World article was included in the BSD Now 245 – ZFS User Conf 2018 episode.

Thanks for mentioning!

UPDATE 3

The Chapter 14. HardenedBSD of the HardenedBSD Handbook has been migrated/ported to wiki page available here – https://github.com/HardenedBSD/hardenedBSD/wiki – enjoy.

EOF

Nextcloud 13 on FreeBSD

Today I would like to share a setup of Nextcloud 13 running on a FreeBSD system. To make things more interesting it would be running inside a FreeBSD Jail. I will not describe the Nextcloud setup itself here as its large enough for several blog posts.

Official Nextcloud 13 documentation recommends following setup:

  • MySQL/MariaDB
  • PHP 7.0 (or newer)
  • Apache 2.4 (with mod_php)

I prefer PostgreSQL database to MySQL/MariaDB and I prefer fast and lean Nginx web server to Apache, so my setup is based on these components:

  • PostgreSQL 10.3
  • PHP 7.2.4
  • Nginx 1.12.2 (with php-fpm)
  • Memcached 1.5.7

The Memcached subsystem is least important, it can be easily changed into something more modern like Redis for example. I prefer not to use any third party tools for FreeBSD Jails management. Not because they are bad or something like that. There are just many choices for good FreeBSD Jails management and I want to provide a GENERIC example for Nextcloud 13 in a Jail, not for a specific management tool.

Host

Lets start with preparing the FreeBSD Host with needed settings. We need to allow using raw sockets in Jails. For the future optional upgrades of the Jail we will also allow using chflags(1) in Jails.

host # cat >> /etc/sysctl.conf << __EOF

# ALLOW JAIL RAW SOCKETS
security.jail.allow_raw_sockets=1

# ALLOW UPGRADES IN JAIL 
security.jail.chflags_allowed=1
__EOF

host # sysctl security.jail.allow_raw_sockets=1
security.jail.allow_raw_sockets: 0 -> 1

host # sysctl security.jail.chflags_allowed=1
security.jail.chflags_allowed: 0 -> 1

I would also enable rctl(8) limits for convenient resource limitations on the host system.

host # cat >> /boot/loader.conf << __EOF

# RACCT/RCTL RESOURCE LIMITS
kern.racct.enable=1
__EOF

The complete Jail after finished installation takes less size then 800 MB if You remove not needed parts after installation is finished. With complete FreeBSD Ports tree and current portsnap(8) information it takes about 1.6 GB of space.

  MB PATH                            DESC
1670 /jail/nextcloud                 (complete Nextcloud 13 Jail)
 726 /jail/nextcloud/usr/ports       (can be removed after install)
 178 /jail/nextcloud/var/db/portsnap (can be removed after install)

I have used my laptop for the Jail host. This is why Jail will configured to use the wireless wlan0 interface and 192.168.43.100 address.

To distinguish the commands I type on the host system and nextcloud.local Jail I use two different prompts, this way it should be obvious what command to execute and where.

Command on the host system.

host # command

Command on the nextcloud.local Jail.

root@nextcloud:/ # command

Here is the running Jail and its processes.

host # jls
   JID  IP Address      Hostname                      Path
    10  192.168.43.100  nextcloud.local               /jail/nextcloud
host # ps axwww -o %cpu,rss,time,command -J nextcloud
%CPU   RSS    TIME COMMAND
 0.0  2032 0:00.01 /usr/sbin/syslogd -s -s
 0.0  5504 0:00.00 /usr/sbin/sshd
 0.0  2056 0:00.01 /usr/sbin/cron -s
 0.0 24196 0:00.04 postgres: checkpointer process    (postgres)
 0.0 23040 0:00.04 postgres: writer process    (postgres)
 0.0 23036 0:00.07 postgres: wal writer process    (postgres)
 0.0 23328 0:00.06 postgres: autovacuum launcher process    (postgres)
 0.0 12764 0:00.24 postgres: stats collector process    (postgres)
 0.0 23204 0:00.00 postgres: bgworker: logical replication launcher    (postgres)
 0.0 23036 0:00.23 /usr/local/bin/postgres -D /var/db/postgres/data
 0.0  6072 0:00.00 nginx: master process /usr/local/sbin/nginx
 0.0  6548 0:00.00 nginx: worker process (nginx)
 0.0  7604 0:00.15 nginx: worker process (nginx)
 0.0  6548 0:00.00 nginx: worker process (nginx)
 0.0  6544 0:00.00 nginx: worker process (nginx)
 0.0 17600 0:01.25 /usr/local/bin/memcached -l 192.168.43.100 -d -P /var/run/memcached/memcached.pid
 0.0 31372 0:00.01 php-fpm: master process (/usr/local/etc/php-fpm.conf) (php-fpm)
 0.0 31388 0:00.00 php-fpm: pool www (php-fpm)
 0.0 31388 0:00.00 php-fpm: pool www (php-fpm)
 0.0 31388 0:00.00 php-fpm: pool www (php-fpm)
 0.0 31388 0:00.00 php-fpm: pool www (php-fpm)

Jail

First we will prepare the Jail for our Nextcloud 13 installation. Lets create some ZFS datasets for that purpose. I will use my local ZFS pool. I will also create the ZFS dataset for PostgreSQL data with 8k record size.

host # zfs create -o mountpoint=/jail                                                 local/jail
host # zfs create -o mountpoint=/jail/nextcloud                                       local/jail/nextcloud
host # zfs create -o mountpoint=/jail/nextcloud/var/db/postgres/data -o recordsize=8k local/jail/nextcloud/pgsql
host # zfs get -r recordsize local/jail
NAME                        PROPERTY    VALUE    SOURCE
local/jail                  recordsize  128K     default
local/jail/nextcloud        recordsize  128K     default
local/jail/nextcloud/pgsql  recordsize  8K       local

Now lets fetch the FreeBSD base into /jail/nextcloud path.

host # cd /jail/nextcloud
host # fetch -o - http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/11.1-RELEASE/base.txz | tar --unlink -xpJf - -C /jail/nextcloud
-                                             100% of   99 MB  689 kBps 02m28s
host # ls /jail/nextcloud
.cshrc     bin/       COPYRIGHT  etc/       libexec/   mnt/       proc/      root/      sys        usr/
.profile   boot/      dev/       lib/       media/     net/       rescue/    sbin/      tmp/       var/

We have base FreeBSD Jail fetched into /jail/nextcloud path, lets configure host for that Jail.

We will only have one Jail configured (for simplicity), the nextcloud.local Jail.

host # cat >> /etc/jail.conf << __EOF
nextcloud {
  host.hostname = nextcloud.local;
  ip4.addr = 192.168.43.100;
  interface = wlan0;
  path = /jail/nextcloud;
  exec.start = "/bin/sh /etc/rc";
  exec.stop = "/bin/sh /etc/rc.shutdown";
  exec.clean;
  mount.devfs;
  allow.raw_sockets;
  allow.sysvipc;
}
__EOF

After creating/modifying the /etc/jail.conf file there should not be any running Jails.

host # jls
   JID  IP Address      Hostname                      Path

Lets enable Jails on the host system.

host # cat >> /etc/rc.conf << __EOF
# JAILS
  jail_enable=YES
__EOF

We can now start out nextcloud.local Jail for the first time.

host # service jail start nextcloud
Starting jails: nextcloud.
host #Β jls
   JID  IP Address      Hostname                      Path
     1  192.168.43.100  nextcloud.local               /jail/nextcloud

Now lets configure the nextcloud.local name on both host and Jail.

host # cat >> /etc/hosts << __EOF

# NEXTCLOUD
192.168.43.100 nextcloud.local nextcloud
__EOF
host # jexec 1 /bin/csh

root@nextcloud:/ # cat >> /etc/hosts << __EOF

# NEXTCLOUD
192.168.43.100 nextcloud.local nextcloud
__EOF

One has to remember that there is no localhost (127.0.0.1) in the FreeBSD Jail. The Jail only has itself configure IP address for listening purposes (192.168.43.100 in our example). This is important because if You configure services on the host that listen on localhost (127.0.0.1) they will work as usual, when You do the same in a FreeBSD Jail you will not able to connect to them (even from this very Jail).

Lets make some basic configuration of the Nextcloud Jail.

host # jexec 1 /bin/csh

root@nextcloud:/ # newaliases -v
WARNING: local host name (nextcloud) is not qualified; see cf/README: WHO AM I?
/etc/mail/aliases: 29 aliases, longest 10 bytes, 297 bytes total

root@nextcloud:/ # cp /usr/share/zoneinfo/Europe/Warsaw /etc/localtime

Lets create basic /etc/rc.conf file for out Jail. I will leave some services commented out as they are not yet configured to run, we do not want to imitate Debian here and start services with default configs πŸ™‚

root@nextcloud:/ # cat >> /etc/rc.conf << __EOF
# DAEMONS | yes
  syslogd_flags="-s -s"
  sshd_enable=YES
# php_fpm_enable=YES
# postgresql_enable=YES
# postgresql_class=postgres
# postgresql_data=/var/db/postgres/data
# memcached_enable=YES
# memcached_flags="-l 192.168.43.100"
# nginx_enable=YES

# DAEMONS | no
  sendmail_enable=NONE
  sendmail_submit_enable=NO
  sendmail_outbound_enable=NO
  sendmail_msp_queue_enable=NO

# OTHER
  clear_tmp_enable=YES
  clear_tmp_X=YES
  extra_netfs_types=NFS
  dumpdev=NO
  update_motd=NO
  keyrate=fast
__EOF

As we will disable sendmail(8) we need to make sure that the /var/spool/clientmqueue would not fill up with time. Lets configure simple cron job for that.

root@nextcloud:/ # cat > /etc/cron.d/sendmail-clean-clientmqueue << __EOF
 # CLEAN SENDMAIL
 0 * * * * root /bin/rm -r -f /var/spool/clientmqueue/*
 __EOF

As we have some basic configuration lets restart our Jail.

root@nextcloud:/ # exit

host # service jail restart nextcloud
Stopping jails: nextcloud.
Starting jails: nextcloud.

host # jls
   JID  IP Address      Hostname                      Path
     2  192.168.43.100  nextcloud.local               /jail/nextcloud

host # jexec nextcloud /bin/csh

After restart we only have sshd(8) daemon listening for connections.

root@nextcloud:/ # sockstat -l4
USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS
root     sshd       97823 3  tcp4   192.168.43.100:22         *:*

Packages

Lets configure network connectivity on the Jail as it will be needed to get the packages from Internet.

root@nextcloud:/ # echo nameserver 1.1.1.1 > /etc/resolv.conf

root@nextcloud:/ # ping -c 3 -t 3 freebsd.org
PING freebsd.org (8.8.178.110): 56 data bytes
64 bytes from 8.8.178.110: icmp_seq=0 ttl=52 time=180.860 ms
64 bytes from 8.8.178.110: icmp_seq=1 ttl=52 time=180.373 ms
64 bytes from 8.8.178.110: icmp_seq=2 ttl=52 time=181.363 ms

--- freebsd.org ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 180.373/180.865/181.363/0.404 ms

As we want the latest packages lets set that in the pkg(8) repository config file.

root@nextcloud:/ # grep quarterly /etc/pkg/FreeBSD.conf
  url: "pkg+http://pkg.FreeBSD.org/${ABI}/quarterly",

root@nextcloud:/ # sed -i '' s/quarterly/latest/g /etc/pkg/FreeBSD.conf

root@nextcloud:/ # grep latest /etc/pkg/FreeBSD.conf
  url: "pkg+http://pkg.FreeBSD.org/${ABI}/latest",

Now lets setup pkg(8) and fetch latest repository metadata.

root@nextcloud:/ # pkg update -f
The package management tool is not yet installed on your system.
Do you want to fetch and install it now? [y/N]: y
Bootstrapping pkg from pkg+http://pkg.FreeBSD.org/FreeBSD:11:amd64/latest, please wait...
Verifying signature with trusted certificate pkg.freebsd.org.2013102301... done
[nextcloud] Installing pkg-1.10.5...
[nextcloud] Extracting pkg-1.10.5: 100%
Updating FreeBSD repository catalogue...
pkg: Repository FreeBSD load error: access repo file(/var/db/pkg/repo-FreeBSD.sqlite) failed: No such file or directory
[nextcloud] Fetching meta.txz: 100%    944 B   0.9kB/s    00:01
[nextcloud] Fetching packagesite.txz: 100%    6 MiB 530.8kB/s    00:12
Processing entries: 100%
FreeBSD repository update completed. 31134 packages processed.
All repositories are up to date.

… and up to date FreeBSD Ports tree.

root@nextcloud:/ # portsnap fetch extract
Looking up portsnap.FreeBSD.org mirrors... 6 mirrors found.
Fetching public key from ec2-eu-west-1.portsnap.freebsd.org... done.
Fetching snapshot tag from ec2-eu-west-1.portsnap.freebsd.org... done.
Fetching snapshot metadata... done.
Fetching snapshot generated at Mon Apr  2 02:06:03 CEST 2018:
7cd019f9e1af8a9d637a56ba3d2bbc2025f54d9931cd8b100% of   79 MB  624 kBps 02m10s
Extracting snapshot... done.
Verifying snapshot integrity...
(...)
Building new INDEX files... done.

root@nextcloud:/ # portsnap fetch update
Looking up portsnap.FreeBSD.org mirrors... 6 mirrors found.
Fetching snapshot tag from ec2-eu-west-1.portsnap.freebsd.org... done.
Ports tree hasn't changed since last snapshot.
No updates needed.
Ports tree is already up to date.

By default Nextcloud 13 package in repository is built with MySQL 5.6 and older PHP 5.6, thus we can not use packages for everything, some (automated) compilation is unavoidable.

root@nextcloud:/ # cd /usr/ports/www/nextcloud

root@nextcloud:/usr/ports/www/nextcloud # make run-depends-list | grep -m 1 php
/usr/ports/lang/php56

root@nextcloud:/usr/ports/www/nextcloud # make run-depends-list | grep -m 1 database
/usr/ports/databases/mysql56-client

Lets check what are the FreeBSD Ports default packages versions.

root@nextcloud:/ # grep -E '^[A-Z]+_DEFAULT' /usr/ports/Mk/bsd.default-versions.mk | column -t
APACHE_DEFAULT?=       2.4
BDB_DEFAULT?=          5
FIREBIRD_DEFAULT?=     2.5
FORTRAN_DEFAULT?=      gfortran
FPC_DEFAULT?=          3.0.4
GCC_DEFAULT?=          6
GHOSTSCRIPT_DEFAULT?=  agpl
LAZARUS_DEFAULT?=      1.8.2
LINUX_DEFAULT?=        c6_64
LINUX_DEFAULT?=        c6
LUA_DEFAULT?=          5.2
MYSQL_DEFAULT?=        5.6
PGSQL_DEFAULT?=        9.5
PHP_DEFAULT?=          5.6
PYTHON_DEFAULT?=       2.7
RUBY_DEFAULT?=         2.4
SAMBA_DEFAULT?=        4.6
SSL_DEFAULT=           base
SSL_DEFAULT:=          ${OPENSSL_INSTALLED:T}
SSL_DEFAULT?=          base
TCLTK_DEFAULT?=        8.6
VARNISH_DEFAULT?=      4

Its PostgreSQL 9.5 and PHP 5.6. We will override that in /etc/make.conf file with the following settings. We will also force using PGSQL option and disable MYSQL option for all Ports.

root@nextcloud:/ # cat >> /etc/make.conf << __EOF
WRKDIRPREFIX=\${PORTSDIR}/obj
DEFAULT_VERSIONS+= php=7.2
DEFAULT_VERSIONS+= pgsql=10
OPTIONS_UNSET+=    MYSQL
OPTIONS_SET+=      PGSQL
__EOF

Now, lets display the default Nextcloud port configuration.

root@nextcloud:/usr/ports/www/nextcloud # make showconfig
===> The following configuration options are available for nextcloud-13.0.0:
     EXIF=on: Image rotation support
     LDAP=on: LDAP protocol support
     SMB=on: SMB network protocol support
     SSL=on: SSL protocol support
====> Database backend(s): you have to choose at least one of them
     MYSQL=on: MySQL database support
     PGSQL=off: PostgreSQL database support
     SQLITE=off: SQLite database support
===> Use 'make config' to modify these settings

PostgreSQL support is not even enabled. Lets configure the Nextcloud port to our needs.

root@nextcloud:/usr/ports/www/nextcloud # make config

nextcloud-06-xterm-make-config
Lets check current port configuration.

root@nextcloud:/usr/ports/www/nextcloud # make showconfig
===> The following configuration options are available for nextcloud-13.0.0:
     EXIF=on: Image rotation support
     LDAP=on: LDAP protocol support
     SMB=on: SMB network protocol support
     SSL=on: SSL protocol support
====> Database backend(s): you have to choose at least one of them
     MYSQL=off: MySQL database support
     PGSQL=on: PostgreSQL database support
     SQLITE=off: SQLite database support
===> Use 'make config' to modify these settings

Good. In case You wandered where these settings are stored below is the answer. Yes, if you delete /var/db/ports/www_nextcloud directory, they will be brought back to defaults without PostgreSQL and with MySQL.

root@nextcloud:/ # cat /var/db/ports/www_nextcloud/options
# This file is auto-generated by 'make config'.
# Options for nextcloud-13.0.0
_OPTIONS_READ=nextcloud-13.0.0
_FILE_COMPLETE_OPTIONS_LIST=EXIF LDAP SMB SSL MYSQL PGSQL SQLITE
OPTIONS_FILE_SET+=EXIF
OPTIONS_FILE_SET+=LDAP
OPTIONS_FILE_SET+=SMB
OPTIONS_FILE_SET+=SSL
OPTIONS_FILE_UNSET+=MYSQL
OPTIONS_FILE_SET+=PGSQL
OPTIONS_FILE_UNSET+=SQLITE

Now lets check again for the run-depends-list after our configuration.

root@nextcloud:/ # make -C /usr/ports/www/nextcloud run-depends-list | grep -m 1 php
/usr/ports/lang/php72

root@nextcloud:/ # make -C /usr/ports/www/nextcloud run-depends-list | grep -m 1 sql
/usr/ports/databases/postgresql10-client

Better.

To make things little faster and to not build everything from FreeBSD Ports we may add most of needed software from packages. We will have to switch for /bin/sh shell for that purpose.

root@nextcloud:/usr/ports/www/nextcloud # exit

host # jexec nextcloud /bin/sh

root@nextcloud:/ # make -C /usr/ports/www/nextcloud run-depends-list | while read I; do echo pkg install -y $( basename $I );done
pkg install -y gettext-runtime
pkg install -y postgresql10-client
pkg install -y pecl-smbclient
pkg install -y php72
pkg install -y php72-bz2
pkg install -y php72-ctype
pkg install -y php72-curl
pkg install -y php72-dom
pkg install -y php72-fileinfo
pkg install -y php72-filter
pkg install -y php72-gd
pkg install -y php72-hash
pkg install -y php72-iconv
pkg install -y php72-json
pkg install -y php72-mbstring
pkg install -y php72-pdo
pkg install -y php72-posix
pkg install -y php72-session
pkg install -y php72-simplexml
pkg install -y php72-xml
pkg install -y php72-xmlreader
pkg install -y php72-xmlwriter
pkg install -y php72-xsl
pkg install -y php72-wddx
pkg install -y php72-zip
pkg install -y php72-zlib
pkg install -y php72-exif
pkg install -y php72-ldap
pkg install -y php72-openssl
pkg install -y php72-pdo_pgsql
pkg install -y php72-pgsql

So we have our list of commands to install most packages, lets paste them one by one into the nextcloud.local prompt. As packages are build, they sometimes get a prefix against what version of ‘upstream’ port it has been built, the ‘pecl-smbclient’ port is a good example here:

root@nextcloud:/ # pkg install -y pecl-smbclient
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
pkg: No packages available to install matching 'pecl-smbclient' have been found in the repositories

root@nextcloud:/ # pkg search pecl-smbclient
php56-pecl-smbclient-0.9.0_3   Smbclient wrapper extension
php70-pecl-smbclient-0.9.0_3   Smbclient wrapper extension
php71-pecl-smbclient-0.9.0_3   Smbclient wrapper extension
php72-pecl-smbclient-0.9.0_3   Smbclient wrapper extension

Here is complete list of packages that we need to install.

root@nextcloud:/ # pkg install -y gettext-runtime
root@nextcloud:/ # pkg install -y postgresql10-client
root@nextcloud:/ # pkg install -y pecl-smbclient
root@nextcloud:/ # pkg install -y php72
root@nextcloud:/ # pkg install -y php72-bz2
root@nextcloud:/ # pkg install -y php72-ctype
root@nextcloud:/ # pkg install -y php72-curl
root@nextcloud:/ # pkg install -y php72-dom
root@nextcloud:/ # pkg install -y php72-fileinfo
root@nextcloud:/ # pkg install -y php72-filter
root@nextcloud:/ # pkg install -y php72-gd
root@nextcloud:/ # pkg install -y php72-hash
root@nextcloud:/ # pkg install -y php72-iconv
root@nextcloud:/ # pkg install -y php72-json
root@nextcloud:/ # pkg install -y php72-mbstring
root@nextcloud:/ # pkg install -y php72-pdo
root@nextcloud:/ # pkg install -y php72-posix
root@nextcloud:/ # pkg install -y php72-session
root@nextcloud:/ # pkg install -y php72-simplexml
root@nextcloud:/ # pkg install -y php72-xml
root@nextcloud:/ # pkg install -y php72-xmlreader
root@nextcloud:/ # pkg install -y php72-xmlwriter
root@nextcloud:/ # pkg install -y php72-xsl
root@nextcloud:/ # pkg install -y php72-wddx
root@nextcloud:/ # pkg install -y php72-zip
root@nextcloud:/ # pkg install -y php72-zlib
root@nextcloud:/ # pkg install -y php72-exif
root@nextcloud:/ # pkg install -y php72-ldap
root@nextcloud:/ # pkg install -y php72-openssl
root@nextcloud:/ # pkg install -y php72-pdo_pgsql
root@nextcloud:/ # pkg install -y php72-pgsql
root@nextcloud:/ # pkg install -y php72-pecl-smbclient
root@nextcloud:/ # pkg install -y nginx
root@nextcloud:/ # pkg install -y memcached
root@nextcloud:/ # pkg install -y portmaster
root@nextcloud:/ # pkg install -y sudo
root@nextcloud:/ # pkg install -y php72-pecl-memcached
root@nextcloud:/ # pkg install -y php72-pcntl
root@nextcloud:/ # pkg install -y postgresql10-server

Some packages like postgresql10-server will remove packages build against the PostgreSQL 9.5 version like below.

root@nextcloud:/ # pkg install postgresql10-server
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
Checking integrity... done (2 conflicting)
  - postgresql10-client-10.3 conflicts with postgresql95-client-9.5.12 on /usr/local/bin/clusterdb
  - postgresql10-client-10.3 conflicts with postgresql95-client-9.5.12 on /usr/local/bin/clusterdb
Checking integrity... done (0 conflicting)
The following 5 package(s) will be affected (of 0 checked):

Installed packages to be REMOVED:
        postgresql95-client-9.5.12
        php72-pdo_pgsql-7.2.4
        php72-pgsql-7.2.4

New packages to be INSTALLED:
        postgresql10-server: 10.3
        postgresql10-client: 10.3

Number of packages to be removed: 3
Number of packages to be installed: 2

The process will require 21 MiB more space.

Proceed with this action? [y/N]: y

Now we need to build the rest.

root@nextcloud:/ # portmaster databases/php72-pgsql databases/php72-pdo_pgsql www/nextcloud www/php72-opcache devel/php72-intl mail/cclient mail/php72-imap math/php72-gmp ftp/php72-ftp
(...)
===>>> The following actions were performed:
        Installation of devel/gmake (gmake-4.2.1_2)
        Installation of devel/gettext-tools (gettext-tools-0.19.8.1)
        Installation of devel/p5-Locale-gettext (p5-Locale-gettext-1.07)
        Installation of misc/help2man (help2man-1.47.6)
        Installation of print/texinfo (texinfo-6.5,1)
        Installation of devel/m4 (m4-1.4.18,1)
        Installation of devel/autoconf-wrapper (autoconf-wrapper-20131203)
        Installation of devel/autoconf (autoconf-2.69_1)
        Installation of databases/php72-pgsql (php72-pgsql-7.2.4)
        Installation of databases/php72-pdo_pgsql (php72-pdo_pgsql-7.2.4)
        Installation of www/nextcloud (nextcloud-13.0.0)
        Installation of www/php72-opcache (php72-opcache-7.2.4)
        Installation of devel/php72-intl (php72-intl-7.2.4)
        Installation of mail/cclient (cclient-2007f_3,1)
        Installation of mail/php72-imap (php72-imap-7.2.4)
        Installation of math/php72-gmp (php72-gmp-7.2.4)
        Installation of ftp/php72-ftp (php72-ftp-7.2.4)

Alternatively to not juggle between packages and ports you may build everything from the FreeBSD Ports tree with command below.

root@nextcloud:/ # portmaster -y www/nextcloud www/nginx databases/memcached security/sudo databases/postgresql10-server www/php72-opcache devel/php72-intl mail/cclient mail/php72-imap math/php72-gmp ftp/php72-ftp

Whichever method you choose, You must end up with these packages installed. This list does not contain dependencies.

root@nextcloud:/ # pkg info | grep -E 'php|nginx|memcached|postgresql|sudo|nextcloud|portmaster'
libmemcached-1.0.18_6          C and C++ client library to the memcached server
memcached-1.5.7                High-performance distributed memory object cache system
nextcloud-13.0.0               Personal cloud which runs on your own server
nginx-1.12.2_11,2              Robust and small WWW server
php72-7.2.4                    PHP Scripting Language
php72-bz2-7.2.4                The bz2 shared extension for php
php72-ctype-7.2.4              The ctype shared extension for php
php72-curl-7.2.4               The curl shared extension for php
php72-dom-7.2.4                The dom shared extension for php
php72-exif-7.2.4               The exif shared extension for php
php72-fileinfo-7.2.4           The fileinfo shared extension for php
php72-filter-7.2.4             The filter shared extension for php
php72-ftp-7.2.4                The ftp shared extension for php
php72-gd-7.2.4                 The gd shared extension for php
php72-gmp-7.2.4                The gmp shared extension for php
php72-hash-7.2.4               The hash shared extension for php
php72-iconv-7.2.4              The iconv shared extension for php
php72-imap-7.2.4               The imap shared extension for php
php72-intl-7.2.4               The intl shared extension for php
php72-json-7.2.4               The json shared extension for php
php72-ldap-7.2.4               The ldap shared extension for php
php72-mbstring-7.2.4           The mbstring shared extension for php
php72-opcache-7.2.4            The opcache shared extension for php
php72-openssl-7.2.4            The openssl shared extension for php
php72-pcntl-7.2.4              The pcntl shared extension for php
php72-pdo-7.2.4                The pdo shared extension for php
php72-pdo_pgsql-7.2.4          The pdo_pgsql shared extension for php
php72-pecl-memcached-3.0.4     PHP extension for interfacing with memcached via libmemcached library
php72-pecl-smbclient-0.9.0_3   Smbclient wrapper extension
php72-pgsql-7.2.4              The pgsql shared extension for php
php72-posix-7.2.4              The posix shared extension for php
php72-session-7.2.4            The session shared extension for php
php72-simplexml-7.2.4          The simplexml shared extension for php
php72-wddx-7.2.4               The wddx shared extension for php
php72-xml-7.2.4                The xml shared extension for php
php72-xmlreader-7.2.4          The xmlreader shared extension for php
php72-xmlwriter-7.2.4          The xmlwriter shared extension for php
php72-xsl-7.2.4                The xsl shared extension for php
php72-zip-7.2.4                The zip shared extension for php
php72-zlib-7.2.4               The zlib shared extension for php
portmaster-3.19_7              Manage your ports without external databases or languages
postgresql10-client-10.3       PostgreSQL database (client)
postgresql10-server-10.3       PostgreSQL is the most advanced open-source database available anywhere
sudo-1.8.22                    Allow others to run commands as root

PostgreSQL Database

Now we have to configure the PostgreSQL database. First lets start with FreeBSD login class.

root@nextcloud:/ # cat >> /etc/login.conf << __EOF
postgres:\
        :lang=en_US.UTF-8:\
        :setenv=LC_COLLATE=C:\
        :tc=default:
__EOF

root@nextcloud:/ # grep -A 4 postgres /etc/login.conf
postgres:\
        :lang=en_US.UTF-8:\
        :setenv=LC_COLLATE=C:\
        :tc=default:

root@nextcloud:/ # cap_mkdb /etc/login.conf

Lets make sure that PostgreSQL data directory belongs to the postgres user.

root@nextcloud:/ # chown postgres:postgres /var/db/postgres/data

Lets enable the PostgreSQL service in the /etc/rc.conf file. It will look like that for now.

root@nextcloud:/ # cat /etc/rc.conf
# DAEMONS | yes
  syslogd_flags="-s -s"
  sshd_enable=YES
  postgresql_enable=YES
  postgresql_class=postgres
  postgresql_data=/var/db/postgres/data
# php_fpm_enable=YES
# memcached_enable=YES
# memcached_flags="-l 192.168.43.100"
# nginx_enable=YES

# DAEMONS | no
  sendmail_enable=NONE
  sendmail_submit_enable=NO
  sendmail_outbound_enable=NO
  sendmail_msp_queue_enable=NO

# OTHER
  clear_tmp_enable=YES
  clear_tmp_X=YES
  extra_netfs_types=NFS
  dumpdev=NO
  update_motd=NO
  keyrate=fast

Now we may initialize and start the database.

root@nextcloud:/ # /usr/local/etc/rc.d/postgresql initdb
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locales
  COLLATE:  C
  CTYPE:    en_US.UTF-8
  MESSAGES: en_US.UTF-8
  MONETARY: en_US.UTF-8
  NUMERIC:  en_US.UTF-8
  TIME:     en_US.UTF-8
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/db/postgres/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.

Success. You can now start the database server using:

    /usr/local/bin/pg_ctl -D /var/db/postgres/data -l logfile start

Now lets start it.

root@nextcloud:/ # /usr/local/etc/rc.d/postgresql start
2018-04-03 13:41:49.289 CEST [14522] LOG:  could not create IPv6 socket for address "::1": Protocol not supported
2018-04-03 13:41:49.291 CEST [14522] LOG:  listening on IPv4 address "127.0.0.1", port 5432
2018-04-03 13:41:49.297 CEST [14522] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"
2018-04-03 13:41:49.328 CEST [14522] LOG:  ending log output to stderr
2018-04-03 13:41:49.328 CEST [14522] HINT:  Future log output will go to log destination "syslog".

root@nextcloud:/ # sockstat -l4
USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS
postgres postgres   14522 4  tcp4   192.168.43.100:5432   *:*
root     sshd       14178 3  tcp4   192.168.43.100:22     *:*

Ok, its working and listening for connections.

Next we will have to connect to create user and database for the Nextcloud server.

root@nextcloud:/ # psql -h nextcloud.local -U postgres
psql: FATAL:  no pg_hba.conf entry for host "192.168.43.100", user "postgres", database "postgres", SSL off

Remmeber the rule about localhost in a Jail? There is no localhost in a Jail. We have to add 192.168.43.100/32 address to the /var/db/postgres/data/pg_hba.conf file as shown below.

root@nextcloud:/ # grep -C 1 192.168.43.100 /var/db/postgres/data/pg_hba.conf
# IPv4 local connections:
host    all             all             192.168.43.100/32       trust
host    all             all             127.0.0.1/32            trust

Now lets restart the PostgreSQL database.

root@nextcloud:/ # /usr/local/etc/rc.d/postgresql restart
2018-04-03 13:44:01.264 CEST [14692] LOG:  could not create IPv6 socket for address "::1": Protocol not supported
2018-04-03 13:44:01.266 CEST [14692] LOG:  listening on IPv4 address "127.0.0.1", port 5432
2018-04-03 13:44:01.271 CEST [14692] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"
2018-04-03 13:44:01.296 CEST [14692] LOG:  ending log output to stderr
2018-04-03 13:44:01.296 CEST [14692] HINT:  Future log output will go to log destination "syslog".

Now we can connect and create needed user and database for Nextcloud 13 installation.

root@nextcloud:/ # psql -h nextcloud.local -U postgres
psql (10.3)
Type "help" for help.

postgres=# CREATE USER nextcloud WITH PASSWORD '{NEXTCLOUD_DB_PASSWORD}';
CREATE ROLE
postgres=# CREATE DATABASE nextcloud TEMPLATE template0 ENCODING 'UNICODE';
CREATE DATABASE
postgres=# ALTER DATABASE nextcloud OWNER TO nextcloud;
ALTER DATABASE
postgres-# \q
root@nextcloud:/ #

I will also create ‘daily maintenance’ script for PostgreSQL database.

root@nextcloud:/ # cat >> /var/db/postgres/data/vacuum.sh  /dev/null
/usr/local/bin/reindexdb -a 1> /dev/null 2> /dev/null
/usr/local/bin/reindexdb -s 1> /dev/null 2> /dev/null
__EOF

root@nextcloud:/ # chmod +x /var/db/postgres/data/vacuum.sh
root@nextcloud:/ # chown postgres:postgres /var/db/postgres/data/vacuum.sh

Lets add it as a cron job on the postgres user.

root@nextcloud:/ # su - postgres -c 'crontab -e'
/tmp/crontab.ruG73E5ivZ: 1 lines, 42 characters.
crontab: installing new crontab

root@nextcloud:/ # su - postgres -c 'crontab -l'
0 0 * * * /var/db/postgres/data/vacuum.sh

Nginx Webserver

Now we have to configure Nginx, lets start by creating the self signed certificate. If You do not want to see warnings that this certificate is not signed then You may want to use service such as letsencrypt.org for example.

root@nextcloud:/ # mkdir -p /usr/local/etc/nginx/ssl
root@nextcloud:/ # cd /usr/local/etc/nginx/ssl
root@nextcloud:/usr/local/etc/nginx/ssl # openssl req -x509 -nodes -days 3650 -newkey rsa:4096 -keyout nginx.key -out nginx.crt
Enter pass phrase for server.key: {NEXTCLOUD_SERVER_PASSWORD}
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:PL
State or Province Name (full name) [Some-State]:lodzkie
Locality Name (eg, city) []:Lodz
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Vermaden Enterprises Ltd.
Organizational Unit Name (eg, section) []:Nextcloud Departament
Common Name (e.g. server FQDN or YOUR name) []:nextcloud.local
Email Address []:vermaden@nextcloud.com

root@nextcloud:/ # chmod 400 /usr/local/etc/nginx/ssl/nginx.key
root@nextcloud:/ # ls -l /usr/local/etc/nginx/ssl
total 14
-rw-r--r--  1 root  wheel  2220 Apr  3 14:43 nginx.crt
-rw-------  1 root  wheel  3272 Apr  3 14:43 nginx.key

Lets tak care of rights for Nginx log files.

root@nextcloud:/ # ls -l /var/log | grep nginx
drwxr-xr-x  2 root  wheel        2 Apr  3 01:10 nginx

root@nextcloud:/ # chown -R www:www /var/log/nginx

root@nextcloud:/ # ls -l /var/log/ | grep nginx
drwxr-xr-x  2 www   www          2 Apr  3 01:10 nginx

… and last but not least, the Nginx main configuration file.

root@nextcloud:/ # cat /usr/local/etc/nginx/nginx.conf
user                 www;
worker_processes     4;
worker_rlimit_nofile 51200;
error_log            /var/log/nginx/error.log;

events {
  worker_connections 1024;
}

http {
  include           mime.types;
  default_type      application/octet-stream;
  log_format        main  '$remote_addr - $remote_user [$time_local] "$request" ';
  access_log        /var/log/nginx/access.log main;
  sendfile          on;
  keepalive_timeout 65;

  upstream php-handler {
    server unix:/var/run/php-fpm.sock;
  }

  server {
    # ENFORCE HTTPS
    listen      80;
    server_name nextcloud.local;
    return      301 https://$server_name$request_uri;
  }

  server {
    listen              443 ssl http2;
    server_name         nextcloud.local;
    ssl_certificate     /usr/local/etc/nginx/ssl/nginx.crt;
    ssl_certificate_key /usr/local/etc/nginx/ssl/nginx.key;

    # HEADERS SECURITY RELATED
    add_header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload;";

    # HEADERS
    add_header X-Content-Type-Options nosniff;
    add_header X-XSS-Protection "1; mode=block";
    add_header X-Robots-Tag none;
    add_header X-Download-Options noopen;
    add_header X-Permitted-Cross-Domain-Policies none;

    # PATH TO THE ROOT OF YOUR INSTALLATION
    root /usr/local/www/nextcloud/;

    location = /robots.txt {
      allow all;
      log_not_found off;
      access_log off;
    }

    location = /.well-known/carddav {
      return 301 $scheme://$host/remote.php/dav;
    }

    location = /.well-known/caldav {
      return 301 $scheme://$host/remote.php/dav;
    }

    # BUFFERS TIMEOUTS UPLOAD SIZES
    client_max_body_size    16400M;
    client_body_buffer_size 1048576k;
    send_timeout            3000;

    # ENABLE GZIP BUT DO NOT REMOVE ETag HEADERS
    gzip on;
    gzip_vary on;
    gzip_comp_level 4;
    gzip_min_length 256;
    gzip_proxied expired no-cache no-store private no_last_modified no_etag auth;
    gzip_types application/atom+xml application/javascript application/json application/ld+json application/manifest+json application/rss+xml application/vnd.geo+json application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/bmp image/svg+xml image/x-icon text/cache-manifest text/css text/plain text/vcard text/vnd.rim.location.xloc text/vtt text/x-component text/x-cross-domain-policy;

    location / {
      rewrite ^ /index.php$uri;
    }

    location ~ ^/(?:build|tests|config|lib|3rdparty|templates|data)/ {
      deny all;
    }

    location ~ ^/(?:\.|autotest|occ|issue|indie|db_|console) {
      deny all;
    }

    location ~ ^/(?:index|remote|public|cron|core/ajax/update|status|ocs/v[12]|updater/.+|ocs-provider/.+)\.php(?:$|/) {
      fastcgi_split_path_info ^(.+\.php)(/.*)$;
      include fastcgi_params;
      fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
      fastcgi_param PATH_INFO $fastcgi_path_info;
      fastcgi_param HTTPS on;
      fastcgi_param modHeadersAvailable true;
      fastcgi_param front_controller_active true;
      fastcgi_pass php-handler;
      fastcgi_intercept_errors on;
      fastcgi_request_buffering off;
      fastcgi_keep_conn       off;
      fastcgi_buffers         16 256K;
      fastcgi_buffer_size     256k;
      fastcgi_busy_buffers_size 256k;
      fastcgi_temp_file_write_size 256k;
      fastcgi_send_timeout    3000s;
      fastcgi_read_timeout    3000s;
      fastcgi_connect_timeout 3000s;
    }

    location ~ ^/(?:updater|ocs-provider)(?:$|/) {
      try_files $uri/ =404;
      index index.php;
    }

    # ADDING THE CACHE CONTROL HEADER FOR JS AND CSS FILES
    # MAKE SURE IT IS BELOW PHP BLOCK
    location ~ \.(?:css|js|woff|svg|gif)$ {
      try_files $uri /index.php$uri$is_args$args;
      add_header Cache-Control "public, max-age=15778463";
      # HEADERS SECURITY RELATED
      # IT IS INTENDED TO HAVE THOSE DUPLICATED TO ONES ABOVE
      add_header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload;";
      # HEADERS
      add_header X-Content-Type-Options nosniff;
      add_header X-XSS-Protection "1; mode=block";
      add_header X-Robots-Tag none;
      add_header X-Download-Options noopen;
      add_header X-Permitted-Cross-Domain-Policies none;
      # OPTIONAL: DONT LOG ACCESS TO ASSETS
      access_log off;
    }

    location ~ \.(?:png|html|ttf|ico|jpg|jpeg)$ {
      try_files $uri /index.php$uri$is_args$args;
      # OPTIONAL: DONT LOG ACCESS TO OTHER ASSETS
      access_log off;
    }
  }
}

If at any point later You would get following error in the browser then there is a problem between Nginx and PHP (php-fpm) configuration.

502 Bad Gateway
---------------
nginx/1.12.2

PHP

Now we have to configure PHP for our needs. First PostgreSQL related settings.

root@nextcloud:/ # cat /usr/local/etc/php/ext-20-pgsql.ini
extension=pgsql.so

root@nextcloud:/ # cat >> /usr/local/etc/php/ext-20-pgsql.ini << __EOF

[PostgresSQL]
pgsql.allow_persistent = On
pgsql.auto_reset_persistent = Off
pgsql.max_persistent = -1
pgsql.max_links = -1
pgsql.ignore_notice = 0
pgsql.log_notice = 0
__EOF

root@nextcloud:/ # cat /usr/local/etc/php/ext-20-pgsql.ini
extension=pgsql.so

[PostgresSQL]
pgsql.allow_persistent = On
pgsql.auto_reset_persistent = Off
pgsql.max_persistent = -1
pgsql.max_links = -1
pgsql.ignore_notice = 0
pgsql.log_notice = 0
root@nextcloud:/ # cat /usr/local/etc/php/ext-30-pdo_pgsql.ini
extension=pdo_pgsql.so


root@nextcloud:/ # cat >> /usr/local/etc/php/ext-30-pdo_pgsql.ini << __EOF

[PostgresSQL]
pgsql.allow_persistent = On
pgsql.auto_reset_persistent = Off
pgsql.max_persistent = -1
pgsql.max_links = -1
pgsql.ignore_notice = 0
pgsql.log_notice = 0
__EOF

root@nextcloud:/ # cat /usr/local/etc/php/ext-30-pdo_pgsql.ini
extension=pdo_pgsql.so

[PostgresSQL]
pgsql.allow_persistent = On
pgsql.auto_reset_persistent = Off
pgsql.max_persistent = -1
pgsql.max_links = -1
pgsql.ignore_notice = 0
pgsql.log_notice = 0

Lets make sure php-fpm log file exists and has right owner.

root@nextcloud:/ # :> /var/log/php-fpm.log
root@nextcloud:/ # chown www:www /var/log/php-fpm.log

No modifications needed to the /usr/local/etc/php-fpm.conf file.

root@nextcloud:/ # grep '^[^;]' /usr/local/etc/php-fpm.conf
[global]
pid = run/php-fpm.pid
include=/usr/local/etc/php-fpm.d/*.conf

Lets create www profile for php-fpm daemon.

root@nextcloud:/ # cat /usr/local/etc/php-fpm.d/www.conf
[www]
user = www
group = www
listen = /var/run/php-fpm.sock
listen.backlog = -1
listen.owner = www
listen.group = www
listen.mode=0660
pm = static
pm.max_children = 4
pm.start_servers = 2
pm.min_spare_servers = 2
pm.max_spare_servers = 4
pm.process_idle_timeout = 1000s;
pm.max_requests = 500
request_terminate_timeout = 0
rlimit_files = 51200
env[HOSTNAME] = $HOSTNAME
env[PATH] = /usr/local/bin:/usr/bin:/bin
env[TMP] = /tmp
env[TMPDIR] = /tmp
env[TEMP] = /tmp

… and the main PHP /usr/local/etc/php.ini configuration file.

root@nextcloud:/ # cat /usr/local/etc/php.ini
[PHP]
max_input_time=3600
engine = On
short_open_tag = On
precision = 14
output_buffering = OFF
zlib.output_compression = Off
implicit_flush = Off
unserialize_callback_func =
serialize_precision = 17
disable_functions =
disable_classes =
zend.enable_gc = On
expose_php = On
max_execution_time = 3600
max_input_time = 30000
memory_limit = 1024M
error_reporting = E_ALL & ~E_DEPRECATED & ~E_STRICT
display_errors = Off
display_startup_errors = Off
log_errors = On
log_errors_max_len = 1024
ignore_repeated_errors = Off
ignore_repeated_source = Off
report_memleaks = On
track_errors = Off
html_errors = On
error_log = /var/log/php.log
variables_order = "GPCS"
request_order = "GP"
register_argc_argv = Off
auto_globals_jit = On
post_max_size = 16400M
auto_prepend_file =
auto_append_file =
default_mimetype = "text/html"
default_charset = "UTF-8"
doc_root =
user_dir =
enable_dl = Off
file_uploads = On
upload_max_filesize = 16400M
max_file_uploads = 64
allow_url_fopen = On
allow_url_include = Off
default_socket_timeout = 300
[CLI Server]
cli_server.color = On
[Date]
date.timezone = Europe/Warsaw
[filter]
[iconv]
[intl]
[sqlite3]
[Pcre]
[Pdo]
[Pdo_mysql]
pdo_mysql.cache_size = 2000
pdo_mysql.default_socket=
[Phar]
[mail function]
SMTP = localhost
smtp_port = 25
mail.add_x_header = On
[SQL]
sql.safe_mode = Off
[ODBC]
odbc.allow_persistent = On
odbc.check_persistent = On
odbc.max_persistent = -1
odbc.max_links = -1
odbc.defaultlrl = 4096
odbc.defaultbinmode = 1
[Interbase]
ibase.allow_persistent = 1
ibase.max_persistent = -1
ibase.max_links = -1
ibase.timestampformat = "%Y-%m-%d %H:%M:%S"
ibase.dateformat = "%Y-%m-%d"
ibase.timeformat = "%H:%M:%S"
[MySQLi]
mysqli.max_persistent = -1
mysqli.allow_persistent = On
mysqli.max_links = -1
mysqli.cache_size = 2000
mysqli.default_port = 3306
mysqli.default_socket =
mysqli.default_host =
mysqli.default_user =
mysqli.default_pw =
mysqli.reconnect = Off
[mysqlnd]
mysqlnd.collect_statistics = On
mysqlnd.collect_memory_statistics = Off
[OCI8]
[PostgreSQL]
pgsql.allow_persistent = On
pgsql.auto_reset_persistent = Off
pgsql.max_persistent = -1
pgsql.max_links = -1
pgsql.ignore_notice = 0
pgsql.log_notice = 0
[bcmath]
bcmath.scale = 0
[browscap]
[Session]
session.save_handler = files
session.save_path = "/tmp"
session.use_strict_mode = 0
session.use_cookies = 1
session.use_only_cookies = 1
session.name = PHPSESSID
session.auto_start = 0
session.cookie_lifetime = 0
session.cookie_path = /
session.cookie_domain =
session.cookie_httponly =
session.serialize_handler = php
session.gc_probability = 1
session.gc_divisor = 1000
session.gc_maxlifetime = 1440
session.referer_check =
session.cache_limiter = nocache
session.cache_expire = 180
session.use_trans_sid = 0
session.hash_function = 0
session.hash_bits_per_character = 5
url_rewriter.tags = "a=href,area=href,frame=src,input=src,form=fakeentry"
[Assertion]
zend.assertions = -1
[COM]
[mbstring]
[gd]
[exif]
[Tidy]
tidy.clean_output = Off
[soap]
soap.wsdl_cache_enabled=1
soap.wsdl_cache_dir="/tmp"
soap.wsdl_cache_ttl=86400
soap.wsdl_cache_limit = 5
[sysvshm]
[ldap]
ldap.max_links = -1
[mcrypt]
[dba]
[opcache]
opcache.enable=1
opcache.enable_cli=1
opcache.interned_strings_buffer=8
opcache.max_accelerated_files=10000
opcache.memory_consumption=128
opcache.save_comments=1
opcache.revalidate_freq=1
[curl]
[openssl]

Daemons

Now, we should enable all daemons and start them, here is the final /etc/rc.conf file.

root@nextcloud:/ # cat /etc/rc.conf
# DAEMONS | yes
  syslogd_flags="-s -s"
  sshd_enable=YES
  postgresql_enable=YES
  postgresql_class=postgres
  postgresql_data=/var/db/postgres/data
  php_fpm_enable=YES 
  memcached_enable=YES
  memcached_flags="-l 192.168.43.100"
  nginx_enable=YES

# DAEMONS | no
  sendmail_enable=NONE
  sendmail_submit_enable=NO
  sendmail_outbound_enable=NO
  sendmail_msp_queue_enable=NO

# OTHER
  clear_tmp_enable=YES
  clear_tmp_X=YES
  extra_netfs_types=NFS
  dumpdev=NO
  update_motd=NO
  keyrate=fast

Lets start the services then.

Memcached.

root@nextcloud:/ # /usr/local/etc/rc.d/memcached start
Starting memcached.

The PHP php-fpm daemon.

root@nextcloud:/ # /usr/local/etc/rc.d/php-fpm start
Performing sanity check on php-fpm configuration:
[03-Apr-2018 14:28:22] NOTICE: configuration file /usr/local/etc/php-fpm.conf test is successful

Starting php_fpm.

PostgreSQL database sohuld be running already.

root@nextcloud:/ # /usr/local/etc/rc.d/postgresql status
pg_ctl: server is running (PID: 17751)
/usr/local/bin/postgres "-D" "/var/db/postgres/data"

… and the Nginx webserver.

root@nextcloud:/ # /usr/local/etc/rc.d/nginx start
Performing sanity check on nginx configuration:
nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok
nginx: configuration file /usr/local/etc/nginx/nginx.conf test is successful
Starting nginx.

Lets see what daemon is listening on what port.

root@nextcloud:/ # sockstat -l4
USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS
www      nginx      28583 6  tcp4   192.168.43.100:80     *:*
www      nginx      28583 7  tcp4   192.168.43.100:443    *:*
www      nginx      28582 6  tcp4   192.168.43.100:80     *:*
www      nginx      28582 7  tcp4   192.168.43.100:443    *:*
www      nginx      28581 6  tcp4   192.168.43.100:80     *:*
www      nginx      28581 7  tcp4   192.168.43.100:443    *:*
www      nginx      28580 6  tcp4   192.168.43.100:80     *:*
www      nginx      28580 7  tcp4   192.168.43.100:443    *:*
root     nginx      28579 6  tcp4   192.168.43.100:80     *:*
root     nginx      28579 7  tcp4   192.168.43.100:443    *:*
nobody   memcached  28239 16 tcp4   192.168.43.100:11211  *:*
postgres postgres   28211 4  tcp4   192.168.43.100:5432   *:*
root     sshd       28205 3  tcp4   192.168.43.100:22     *:*

Remember that php-fpm daemon uses /var/run/php-fpm.sock socket.

root@nextcloud:/ # ls -l /var/run/php-fpm.sock
srw-rw----  1 www  www  0 Apr  3 18:27 /var/run/php-fpm.sock

Nextcloud

Now lets prepare the directory for Nextcloud data.

root@nextcloud:/ # mkdir -p /var/db/nextcloud/data
root@nextcloud:/ # chown -R www:www /var/db/nextcloud

We also need to make sure that whole Nextcloud installation directory is owned by www user.

root@nextcloud:/ # chown -R www:www /usr/local/www/nextcloud

Now we should be able to access the Nextcloud 13 using a browser, lets type the https://nextcloud.local/ on the host in the browser of your choice.

nextcloud-00

Viola! Its alive.

Here are last configuration bits etered directly in the browser.

   ADMIN USER: admin
   ADMIN PASS: {NEXTCLOUD_ADMIN_PASSWORD}
  DATA FOLDER: /var/db/nextcloud/data
DATABASE USER: nextcloud
DATABASE PASS: {NEXTCLOUD_DB_PASSWORD} 
DATABASE NAME: nextcloud
DATABASE HOST: nextcloud.local

Here is how it looks in the browser.

nextcloud-01-setup

After we click the Finish setup button we should see the Nextcloud welcome message as shown below.

nextcloud-02-welcome

We may close this message and we will see our files.

nextcloud-03-files

The Nextcloud settings page yelds about lack of cache daemon.

nextcloud-04-nocache

Lets configure our memcached daemon into the Nextcloud configuration file.

nextcloud-05-cache

Here are added lines to the /usr/local/www/nextcloud/config/config.php file.

root@nextcloud:/usr/local/www/nextcloud/config # diff -u config.php.ORG config.php
--- config.php.ORG      2018-04-03 16:39:04.531258000 +0200
+++ config.php  2018-04-03 16:40:01.509956000 +0200
@@ -18,4 +18,14 @@
   'dbuser' => 'nextcloud',
   'dbpassword' => '',
   'installed' => true,
+  'memcache.local' => '\\OC\\Memcache\\Memcached',
+  'memcache.distributed' => '\\OC\\Memcache\\Memcached',
+  'memcached_servers' =>
+  array (
+    0 =>
+    array (
+      0 => 'nextcloud.local',
+      1 => 11211,
+    ),
+  ),
 );

Here is complete Nextcloud 13 main configuration file /usr/local/www/nextcloud/config/config.php after changes.

root@nextcloud:/ # cat /usr/local/www/nextcloud/config/config.php
 'oc70jc009i5e',
  'passwordsalt' => 'anVkM4F5kJwhInurq0N6eq65JmL3xZ',
  'secret' => '2RjnOfiMfrdW6rJEcpxORL39+E1gvS38+sys+G0uI6vZOOSc',
  'trusted_domains' =>
  array (
    0 => 'nextcloud.local',
  ),
  'datadirectory' => '/var/db/nextcloud/data',
  'overwrite.cli.url' => 'https://nextcloud.local',
  'dbtype' => 'pgsql',
  'version' => '13.0.0.14',
  'dbname' => 'nextcloud',
  'dbhost' => 'nextcloud.local',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'dbuser' => 'nextcloud',
  'dbpassword' => '',
  'installed' => true,
  'memcache.local' => '\\OC\\Memcache\\Memcached',
  'memcache.distributed' => '\\OC\\Memcache\\Memcached',
  'memcached_servers' =>
  array (
    0 =>
    array (
      0 => 'nextcloud.local',
      1 => 11211,
    ),
  )
);

Alternatively You may want to get that config directly from the Nextcloud application using occ command.

root@nextcloud:/ # sudo -u www php /usr/local/www/nextcloud/occ config:list system
{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "nextcloud.local"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "overwrite.cli.url": "https:\/\/nextcloud.local",
        "dbtype": "pgsql",
        "version": "13.0.0.14",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "memcache.local": "\\OC\\Memcache\\Memcached",
        "memcache.distributed": "\\OC\\Memcache\\Memcached",
        "memcached_servers": [
            [
                "nextcloud.local",
                11211
            ]
        ]
    }
}

Logs

To not end with filled /var/log directory with tons of logs we need to configure their rotation.

Here are lines add to the /etc/newsyslog.conf file.

root@nextcloud:/ # cat >> /etc/newsyslog.conf << __EOF
/var/db/nextcloud/data/nextcloud.log     www:www     640  7     *    @T00  JC
/var/log/php-fpm.log                     www:www     640  7     *    @T00  JC
/var/log/nginx/error.log                 www:www     640  7     *    @T00  JC
/var/log/nginx/access.log                www:www     640  7     *    @T00  JC
__EOF

Lets verify the rotation.

root@nextcloud:/ # newsyslog -v | tail -4
/var/db/nextcloud/data/nextcloud.log : --> will trim at Fri Jul 21 00:00:00 2017
/var/log/php-fpm.log : --> will trim at Fri Jul 21 00:00:00 2017
/var/log/nginx/error.log : --> will trim at Fri Jul 21 00:00:00 2017
/var/log/nginx/access.log : --> will trim at Fri Jul 21 00:00:00 2017

Yep. Works like a charm.

Cleanup (Optional)

We may now remove not needed parts of the Jail, for example downloaded packages and distfiles.

root@nextcloud:/ # rm -rf /var/cache/pkg

root@nextcloud:/ # rm -rf /usr/ports/obj

root@nextcloud:/ # rm -rf usr/ports/distfiles

root@nextcloud:/ # pkg autoremove
Checking integrity... done (0 conflicting)
Deinstallation has been requested for the following 8 packages:

Installed packages to be REMOVED:
        autoconf-2.69_1
        autoconf-wrapper-20131203
        gettext-tools-0.19.8.1
        gmake-4.2.1_2
        help2man-1.47.6
        m4-1.4.18,1
        p5-Locale-gettext-1.07
        texinfo-6.5,1

Number of packages to be removed: 8

The operation will free 26 MiB.

Proceed with deinstalling packages? [y/N]: y

You have reached the end, good luck with Your Nextcloud setup πŸ˜‰

UPDATE 1 – SysV IPC in Jails

Since FreeBSD 11.0-RELEASE and FreeBSD 10.4-RELEASE the allow.sysvipc Jail parameter in /etc/jail.conf has been deprecated in favor of sysvmsg/sysvsem/sysvshm parameters. This information is available in man 8 jail manual page for example.

host # man 8 jail
(...)
             allow.sysvipc
                     A process within the jail has access to System V IPC
                     primitives.  This is deprecated in favor of the per-
                     module parameters (see below).  When this parameter is
                     set, it is equivalent to setting sysvmsg, sysvsem, and
                     sysvshm all to β€œinherit”.

(...)

Before this change there was problem with SysV IPC calls because each PostgreSQL server user postgres need to have different UID for each Jail running on that host. Now You can have 100 Jails with each postgres user UID 500 and everything works like a charm.

Its described broadly in this blog post – Postgres in FreeBSD Jailshttps://planet.freebsd.org/brd/2017/11/07/postgres-in-freebsd-jails/.

Using newer approach these are the changes in the configuration in the /etc/jail.conf file.

host # diff -u /etc/jail.conf.OLD /etc/jail.conf
--- /etc/jail.conf.OLD  2018-04-05 13:04:18.556904000 +0200
+++ /etc/jail.conf      2018-04-05 13:04:10.828127000 +0200
@@ -8,5 +8,7 @@
   exec.clean;
   mount.devfs;
   allow.raw_sockets;
-  allow.sysvipc;
+  sysvsem = new;
+  sysvshm = new;
+  sysvmsg = new;
 }

… and the whole /etc/jail.conf file after modification.

host # cat /etc/jail.conf
nextcloud {
  host.hostname = nextcloud.local;
  ip4.addr = 192.168.43.100;
  interface = wlan0;
  path = /jail/nextcloud;
  exec.start = "/bin/sh /etc/rc";
  exec.stop = "/bin/sh /etc/rc.shutdown";
  exec.clean;
  mount.devfs;
  allow.raw_sockets;
  sysvsem = new;
  sysvshm = new;
  sysvmsg = new;
}

UPDATE 2 – Setup without Sockets

To run this setup without sockets you may want to modify the PHP php-fpm daemon to listen in IPv4 address instead of using /var/run/php-fpm.sock socket for communication. These are the changes in /usr/local/etc/php-fpm.d/www.conf php-rpm profile and Nginx webserver main configuration /usr/local/etc/nginx/nginx.conf file.

root@nextcloud:/ # diff -u /usr/local/etc/php-fpm.d/www.conf.SOCKET /usr/local/etc/php-fpm.d/www.conf
--- /usr/local/etc/php-fpm.d/www.conf.SOCKET    2018-04-05 12:46:06.351550000 +0200
+++ /usr/local/etc/php-fpm.d/www.conf   2018-04-05 12:46:09.324277000 +0200
@@ -1,7 +1,8 @@
 [www]
 user = www
 group = www
-listen = /var/run/php-fpm.sock
+listen = 192.168.43.100:9000
+listen.allowed_clients = 192.168.43.100
 listen.backlog = -1
 listen.owner = www
 listen.group = www
root@nextcloud:/ # diff -u /usr/local/etc/nginx/nginx.conf.SOCKET /usr/local/etc/nginx/nginx.conf
--- /usr/local/etc/nginx/nginx.conf.SOCKET      2018-04-05 12:42:09.051583000 +0200
+++ /usr/local/etc/nginx/nginx.conf     2018-04-05 12:42:30.491819000 +0200
@@ -16,7 +16,7 @@
   keepalive_timeout 65;
 
   upstream php-handler {
-    server unix:/var/run/php-fpm.sock;
+    server 192.168.43.100:9000;
   }
 
   server {

UPDATE 3 – Nextcloud 13.0.1 Update

After update to latest Nextcloud 13.0.1 the setup is broken. The symptom is that after the login page it keeps redirecting. The fix has been described in the /usr/ports/UPDATING file, here is the message with the fix itself.

20180404:
  AFFECTS: users of www/nextcloud
  AUTHOR: brnrd@FreeBSD.org

  With the 13.0.1 update the path for Apps bundled with the package has
  changed from "apps" to "apps-pkg". You must add an entry to the
  "apps_paths" array in config/config.php of your nextcloud installation,
  a patch for the default installation can be applied with:

  # cd /usr/local/www/nextcloud
  # su -m www -c "php ./occ config:import < /usr/local/share/nextcloud/fix-apps_paths.json"

So to fix the installation after eventual upgrade to 13.0.1 these instructions need to be executed in the Jail.

root@nextcloud:/ # su -m www -c "php /usr/local/www/nextcloud/occ config:import < /usr/local/share/nextcloud/fix-apps_paths.json"

Hope that helps to resolve the issue.

UPDATE 4

The Nextcloud 13 on FreeBSD article was featured in the BSD Now 245 – ZFS User Conf 2018 episode.

Thanks for mentioning!

EOF

Valuable News – 2018/04/03

IBM released new free font named IBM Plex.
https://en.wikipedia.org/wiki/IBM_Plex
https://github.com/IBM/plex

Difference between Little Endian and Big Endian in one simple picture.
https://pbs.twimg.com/media/DKPVGu-UIAASyDZ.jpg:large

From 2010 Apple plans to use its own chips in Macs instead of Intel. Seems that iOS will be merged with Mac OS X (macOS) into one single system to run on custom ARM64 chips from Apple.
https://www.bloomberg.com/news/articles/2018-04-02/apple-is-said-to-plan-move-from-intel-to-own-mac-chips-from-2020

Install X11 and Desktop Environment on OmniOSce/Illumos system.
https://geekblood.wordpress.com/2017/10/26/installing-x11-and-a-desktop-environment-on-omnios/

OpenBSD 6.3 has need released.
https://marc.info/?l=openbsd-announce&m=152267725618055

The gron utility makes JSON files greppable.
https://github.com/tomnomnom/gron/

The gron is not yet in the FreeBSD Ports but FreeBSD binary is available for download.
https://github.com/tomnomnom/gron/releases

The end of Windows has finally happened, its just not what You expected.
https://stratechery.com/2018/the-end-of-windows/

The new part of the arstechnica.com series – History of the AmigaPart 12: Red vs. Blue has been published.
https://arstechnica.com/gadgets/2018/03/a-history-of-the-amiga-part-12-red-vs-blue/

Earlier parts of the History of the Amiga series are available here:
https://arstechnica.com/series/history-of-the-amiga/

The zrepl – ZFS replication tool – has new 0.3 version.
https://zrepl.github.io/changelog.html
https://github.com/zrepl/zrepl/releases/tag/0.0.3

… and last but not least, the freebsdwiki.net page seeks for new maintainer as the current owner is not really active in the FreeBSD community anymore.
http://freebsdwiki.net/index.php/Main_Page

EOF.

.
.
.
.
.