Tag Archives: jail

New ZFS Boot Environments Tool

About a month ago I was honored to give talk about ZFS Boot Environments on PBUG. At the end of the presentation I mentioned the history of tools to manage ZFS Boot Environments on FreeBSD.

zfs-boot-environments-history.png

Pawel Jakub Dawidek – which also was on this PBUG #3 meeting – suggested that I should try to add beadm into the FreeBSD base system. I also heard that idea from many beadm users which repetitively asked why beadm is not in the FreeBSD base system. So after finished PBUG #3 that is exactly what I did. I created new PR – Bug 230323 – Idea/Feature Request – include beadm in the base – and to my (positive) surprise they included new bectl tool into the FreeBSD base! We now have new member of that ZFS Boot Environment tools family – the bectl tool.

I will of course maintain and update beadm tool and it will still be available in the FreeBSD Ports under sysutils/beadm category as having such tool written in POSIX /bin/sh allows fast debugging and easy changes to such tool. In short (TLDR) the bectl tool is beadm implemented in C language and as it has just been imported into FreeBSD base which means that it will be part of the FreeBSD 12.0-RELEASE. Currently bectl is already available in the 12.0-ALPHA2 image.

Comparison

The new bectl tool is at very early stage and does not (yet) offer full replacement for the beadm tool. Here is quick comparision of the usage information between bectl and beadm tools.

root@fbsd12:~ # beadm
usage:
  beadm activate 
  beadm create [-e nonActiveBe | -e beName@snapshot] 
  beadm create 
  beadm destroy [-F] 
  beadm list [-a] [-s] [-D] [-H]
  beadm rename  
  beadm mount  [mountpoint]
  beadm { umount | unmount } [-f] 
  beadm version

… and new bectl tool.

root@fbsd12:~ # bectl
missing command
usage:  bectl ( -h | -? | subcommand [args...] )
        bectl activate [-t] beName
        bectl create [-e nonActiveBe | -e beName@snapshot] beName
        bectl create beName@snapshot
        bectl destroy [-F] beName | beName@snapshot⟩
        bectl export sourceBe
        bectl import targetBe
        bectl jail [ -o key=value | -u key ]... bootenv
        bectl list [-a] [-D] [-H] [-s]
        bectl mount beName [mountpoint]
        bectl rename origBeName newBeName
        bectl { ujail | unjail } ⟨jailID | jailName | bootenv)
        bectl { umount | unmount } [-f] beName

For example bectl is not able to rename currently used/mounted boot environment while beadm can.

root@fbsd12:~ # bectl rename safe new
boot environment is already mounted
failed to rename bootenv safe to new

Its possible to rename such ZFS dataset mounted as / with zfs rename -u ... command (this is exactly what beadm does under the hood) as a workaround for bectl tool.

root@fbsd12:~ # bectl list
BE      Active Mountpoint Space Created
safe    NR     /          188K  2018-08-18 02:32
default -      -          427M  2018-08-18 02:26

root@fbsd12:~ # zfs list | grep safe
zroot/ROOT/safe      108K  6.85G   427M  /

root@fbsd12:~ # zfs rename -u zroot/ROOT/safe zroot/ROOT/new

Its then listed as usual under new name in bectl as shown below:

root@fbsd12:~ # bectl list
BE      Active Mountpoint Space Created
new     NR     /          188K  2018-08-18 02:32
default -      -          427M  2018-08-18 02:26

One nice addition that bectl has that beadm lacks is dynamic FreeBSD Jail creation in specified boot environment.

Here is bectl FreeBSD Jail creation in action.

root@fbsd12:~ # bectl list
BE      Active Mountpoint Space Created
new     NR     /          188K  2018-08-18 02:32
default -      -          427M  2018-08-18 02:26

root@fbsd12:~ # bectl jail default
# pwd
/
# ls /
.cshrc          bin             entropy         libexec         net             root            usr
.profile        boot            etc             media           proc            sbin            var
COPYRIGHT       dev             lib             mnt             rescue          tmp             zroot
# exit
root@fbsd12:~ # jls
   JID  IP Address      Hostname                      Path
     1                  default                       /tmp/be_mount.OnRc

root@fbsd12:~ # mount | grep default
zroot/ROOT/default on /tmp/be_mount.OnRc (zfs, local, noatime, nfsv4acls)

root@fbsd12:~ # bectl unjail default

root@fbsd12:~ # jls
   JID  IP Address      Hostname                      Path

If you move/migrate to bectl from beadm you will also have to be more careful as bectl does not ask questions πŸ™‚

For example beadm tool asks if you are sure that you want to destroy specified boot environment. The bectl tool will just remove it without even writing anything on the screen.

root@fbsd12:~ # bectl list
BE      Active Mountpoint Space Created
new     NR     /          188K  2018-08-18 02:32
default -      -          427M  2018-08-18 02:26

root@fbsd12:~ # beadm destroy safe
Are you sure you want to destroy 'safe'?
This action cannot be undone (y/[n]): n

root@fbsd12:~ # bectl destroy safe

root@fbsd12:~ # bectl list
BE      Active Mountpoint Space Created
new     NR     /          188K  2018-08-18 02:32

One of the things that bectl lacks is also the Ansible plugin, beadm is supported by the Ansible plugin so if you prefer to use that configuration management tool, then bectl will ‘backport’ you to raw Ansible module πŸ™‚

The good information is that beadm and bectl can work together on the same host, so you do not have to choose. You may still use beadm tool for daily tasks (or for Ansible module) and bectl for the jail/unjail options for example.

But I think in time bectl will have needed features added and having such tool in FreeBSD base system is a welcome addition.

UPDATE 1

The New ZFS Boot Environments Tool article was featured in the BSD Now 262 – OpenBSD Surfacing episode.

Thanks for mentioning!

UPDATE 2

Finally I had time to check new bectl command again in the newer FreeBSD-12.0-ALPHA6 release for possible improvements.

Now bectl does not display missing command when invoked without arguments.

It is now possible to rename currently used Boot Environment with bectl command.

The last thing I noticed is that bectl jail command does not leave enabled/running Jail after you exit from it, cosmetic but important.

… and last but not least, the easiest path of migration is to create simple alias.

# alias beadm=bectl

… or for (T)CSH shell.

# alias beadm bectl
EOF
Advertisements

Distributed Object Storage with Minio on FreeBSD

Meet Minio.

minio-logo-arch-32

Free and open source distributed object storage server compatible with Amazon S3 v2/v4 API. Offers data protection against hardware failures using erasure code and bitrot detection. Supports highly available distributed setup. Provides confidentiality, integrity and authenticity assurances for encrypted data with negligible performance overhead. Both server side and client side encryption are supported. Below is the image of example Minio setup.

Web

The Minio identifies itself as the ZFS of Cloud Object Storage. This guide will show You how to setup highly available distributed Minio storage on the FreeBSD operating system with ZFS as backend for Minio data. For convenience we will use FreeBSD Jails operating system level virtualization.

Setup

The setup will assume that You have 3 datacenters and assumption that you have two datacenters in whose the most of the data must reside and that the third datacenter is used as a ‘quorum/witness’ role. Distributed Minio supports up to 16 nodes/drives total, so we may juggle with that number to balance data between desired datacenters. As we have 16 drives to allocate resources on 3 sites we will use 7 + 7 + 2 approach here. The datacenters where most of the data must reside have 7/16 ratio while the ‘quorum/witness’ datacenter have only 2/16 ratio. Thanks to built in Minio redundancy we may loose (turn off for example) any one of those machines and our object storage will still be available and ready to use for any purpose.

Jails

First we will create 3 jails for our proof of concept Minio setup, storage1 will have the ‘quorum/witness’ role while storage2 and storage3 will have the ‘data’ role. To distinguish commands I type on the host system and storageX Jail I use two different prompts, this way it should be obvious what command to execute and where.

Command on the host system.

host # command

Command on the storageX Jail.

root@storageX:/ # command

First we will create the base Jails for our setup.

host # mkdir -p /jail/BASE /jail/storage1 /jail/storage2 /jail/storage3
host # cd /jail/BASE
host # fetch http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/11.1-RELEASE/base.txz
host # for I in 1 2 3; do echo ${I}; tar --unlink -xpJf /jail/BASE/base.txz -C /jail/storage${I}; done
1
2
3
host #

We will now add Jails configuration the the /etc/jail.conf file.

I have used my laptop for the Jail host. This is why Jail will configured to use the wireless wlan0 interface and 192.168.43.10X addresses.

host # for I in 1 2 3
do
  cat >> /etc/jail.conf << __EOF
storage${I} {
  host.hostname = storage${I}.local;
  ip4.addr = 192.168.43.10${I};
  interface = wlan0;
  path = /jail/storage${I};
  exec.start = "/bin/sh /etc/rc";
  exec.stop = "/bin/sh /etc/rc.shutdown";
  exec.clean;
  mount.devfs;
  allow.raw_sockets;
}

__EOF
done
host #

Lets verify that /etc/jail.conf file is configured as desired.

host # cat /etc/jail.conf
storage1 {
  host.hostname = storage1.local;
  ip4.addr = 192.168.43.101;
  interface = wlan0;
  path = /jail/storage1;
  exec.start = "/bin/sh /etc/rc";
  exec.stop = "/bin/sh /etc/rc.shutdown";
  exec.clean;
  mount.devfs;
  allow.raw_sockets;
}

storage2 {
  host.hostname = storage2.local;
  ip4.addr = 192.168.43.102;
  interface = wlan0;
  path = /jail/storage2;
  exec.start = "/bin/sh /etc/rc";
  exec.stop = "/bin/sh /etc/rc.shutdown";
  exec.clean;
  mount.devfs;
  allow.raw_sockets;
}

storage3 {
  host.hostname = storage3.local;
  ip4.addr = 192.168.43.103;
  interface = wlan0;
  path = /jail/storage3;
  exec.start = "/bin/sh /etc/rc";
  exec.stop = "/bin/sh /etc/rc.shutdown";
  exec.clean;
  mount.devfs;
  allow.raw_sockets;
}

host #

Now we will start our Jails.

host # for I in 1 2 3; do service jail onestart storage${I}; done
Starting jails: storage1.
Starting jails: storage2.
Starting jails: storage3.

Lets see how they work.

host # jls
   JID  IP Address      Hostname                      Path
     1  192.168.43.101  storage1.local                /jail/storage1
     2  192.168.43.102  storage2.local                /jail/storage2
     3  192.168.43.103  storage3.local                /jail/storage3

Now lets add DNS server so they will have Internet connectivity.

host # for I in 1 2 3; do echo nameserver 1.1.1.1 > /jail/storage${I}/etc/resolv.conf; done

We can now install Minio package.

host # for I in 1 2 3; do jexec storage${I} env ASSUME_ALWAYS_YES=yes pkg install -y minio; echo; done
Bootstrapping pkg from pkg+http://pkg.FreeBSD.org/FreeBSD:11:amd64/quarterly, please wait...
Verifying signature with trusted certificate pkg.freebsd.org.2013102301... done
[storage1.local] Installing pkg-1.10.5...
[storage1.local] Extracting pkg-1.10.5: 100%
Updating FreeBSD repository catalogue...
pkg: Repository FreeBSD load error: access repo file(/var/db/pkg/repo-FreeBSD.sqlite) failed: No such file or directory
[storage1.local] Fetching meta.txz: 100%    944 B   0.9kB/s    00:01    
[storage1.local] Fetching packagesite.txz: 100%    6 MiB 637.1kB/s    00:10    
Processing entries: 100%
FreeBSD repository update completed. 31143 packages processed.
All repositories are up to date.
Updating database digests format: 100%
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        minio: 2018.03.19.19.22.06

Number of packages to be installed: 1

The process will require 22 MiB more space.
6 MiB to be downloaded.
[storage1.local] [1/1] Fetching minio-2018.03.19.19.22.06.txz: 100%    6 MiB 305.6kB/s    00:19    
Checking integrity... done (0 conflicting)
[storage1.local] [1/1] Installing minio-2018.03.19.19.22.06...
===> Creating groups.
Creating group 'minio' with gid '473'.
===> Creating users
Creating user 'minio' with uid '473'.
[storage1.local] [1/1] Extracting minio-2018.03.19.19.22.06: 100%

Bootstrapping pkg from pkg+http://pkg.FreeBSD.org/FreeBSD:11:amd64/quarterly, please wait...
Verifying signature with trusted certificate pkg.freebsd.org.2013102301... done
[storage2.local] Installing pkg-1.10.5...
[storage2.local] Extracting pkg-1.10.5: 100%
Updating FreeBSD repository catalogue...
pkg: Repository FreeBSD load error: access repo file(/var/db/pkg/repo-FreeBSD.sqlite) failed: No such file or directory
[storage2.local] Fetching meta.txz: 100%    944 B   0.9kB/s    00:01    
[storage2.local] Fetching packagesite.txz: 100%    6 MiB 637.1kB/s    00:10    
Processing entries: 100%
FreeBSD repository update completed. 31143 packages processed.
All repositories are up to date.
Updating database digests format: 100%
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        minio: 2018.03.19.19.22.06

Number of packages to be installed: 1

The process will require 22 MiB more space.
6 MiB to be downloaded.
[storage2.local] [1/1] Fetching minio-2018.03.19.19.22.06.txz: 100%    6 MiB 305.6kB/s    00:19    
Checking integrity... done (0 conflicting)
[storage2.local] [1/1] Installing minio-2018.03.19.19.22.06...
===> Creating groups.
Creating group 'minio' with gid '473'.
===> Creating users
Creating user 'minio' with uid '473'.
[storage2.local] [1/1] Extracting minio-2018.03.19.19.22.06: 100%

Bootstrapping pkg from pkg+http://pkg.FreeBSD.org/FreeBSD:11:amd64/quarterly, please wait...
Verifying signature with trusted certificate pkg.freebsd.org.2013102301... done
[storage3.local] Installing pkg-1.10.5...
[storage3.local] Extracting pkg-1.10.5: 100%
Updating FreeBSD repository catalogue...
pkg: Repository FreeBSD load error: access repo file(/var/db/pkg/repo-FreeBSD.sqlite) failed: No such file or directory
[storage3.local] Fetching meta.txz: 100%    944 B   0.9kB/s    00:01    
[storage3.local] Fetching packagesite.txz: 100%    6 MiB 637.1kB/s    00:10    
Processing entries: 100%
FreeBSD repository update completed. 31143 packages processed.
All repositories are up to date.
Updating database digests format: 100%
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        minio: 2018.03.19.19.22.06

Number of packages to be installed: 1

The process will require 22 MiB more space.
6 MiB to be downloaded.
[storage3.local] [1/1] Fetching minio-2018.03.19.19.22.06.txz: 100%    6 MiB 305.6kB/s    00:19    
Checking integrity... done (0 conflicting)
[storage3.local] [1/1] Installing minio-2018.03.19.19.22.06...
===> Creating groups.
Creating group 'minio' with gid '473'.
===> Creating users
Creating user 'minio' with uid '473'.
[storage3.local] [1/1] Extracting minio-2018.03.19.19.22.06: 100%

host #

Lets verify that Minio package has installed successfully.

host # for I in 1 2 3; do jexec storage${I} which minio; done
/usr/local/bin/minio
/usr/local/bin/minio
/usr/local/bin/minio
host #

Now we will configure /etc/hosts file.

root@storage1:/ # cat >> /etc/hosts << __EOF
192.168.43.101 storage1
192.168.43.102 storage2
192.168.43.103 storage3
__EOF
root@storage1:/ # cat >> /etc/hosts << __EOF
192.168.43.101 storage1
192.168.43.102 storage2
192.168.43.103 storage3
__EOF
root@storage1:/ # cat >> /etc/hosts << __EOF
192.168.43.101 storage1
192.168.43.102 storage2
192.168.43.103 storage3
__EOF

We will create directories for Minio data.

host # for DIR in 1 2 3 4 5 6 7
do
  for I in 2 3
  do
    jexec storage${I} mkdir -p /data${DIR}
  done
done
host # for DIR in 1 2
do
  for I in 1
  do
    jexec storage${I} mkdir -p /data${DIR}
  done
done

Lets verify that that our data directories created successfully.

host # for I in 1 2 3
  do
    echo storage${I}
    jexec storage${I} ls -1 / | grep data
    echo
  done


storage1
data1
data2

storage2
data1
data2
data3
data4
data5
data6
data7

storage3
data1
data2
data3
data4
data5
data6
data7

Basic minio command example.

root@storage1:/ # minio
NAME:
  minio - Cloud Storage Server.

DESCRIPTION:
  Minio is an Amazon S3 compatible object storage server. Use it to store photos, videos, VMs, containers, log files, or any blob of data as objects.

USAGE:
  minio [FLAGS] COMMAND [ARGS...]

COMMANDS:
  server   Start object storage server.
  gateway  Start object storage gateway.
  update   Check for a new software update.
  version  Print version.
  
FLAGS:
  --config-dir value, -C value  Path to configuration directory. (default: "/root/.minio")
  --quiet                       Disable startup information.
  --json                        Output server logs and startup information in json format.
  --help, -h                    Show help.
  
VERSION:
  2018-03-19T19:22:06Z

Now we can generate the list of directories on servers to add as argument for Minio.

host # for DIR in 1 2
do
  for I in 1 
  do
    echo -n http://
    jls | grep storage${I} | awk '{printf $3}' | sed s/.local//g
    echo ":9000/data${DIR} \\"
  done
done | sort -n

host # for DIR in 1 2 3 4 5 6 7
do
  for I in 2 3
  do
    echo -n http://
    jls | grep storage${I} | awk '{printf $3}' | sed s/.local//g
    echo ":9000/data${DIR} \\"
  done
done | sort -n
http://storage1:9000/data1 \
http://storage1:9000/data2 \
http://storage2:9000/data1 \
http://storage2:9000/data2 \
http://storage2:9000/data3 \
http://storage2:9000/data4 \
http://storage2:9000/data5 \
http://storage2:9000/data6 \
http://storage2:9000/data7 \
http://storage3:9000/data1 \
http://storage3:9000/data2 \
http://storage3:9000/data3 \
http://storage3:9000/data4 \
http://storage3:9000/data5 \
http://storage3:9000/data6 \
http://storage3:9000/data7 \

We can as well just write it down by hand of course πŸ™‚

host # for DIR in 1 2
do
  for I in 1 
  do
    echo -n http://
    jls | grep storage${I} | awk '{printf $3}' | sed s/.local//g
    echo -n ":9000/data${DIR} "
  done
done | sort -n

host # for DIR in 1 2 3 4 5 6 7
do
  for I in 2 3
  do
    echo -n http://
    jls | grep storage${I} | awk '{printf $3}' | sed s/.local//g
    echo -n ":9000/data${DIR} "
  done
done | sort -n

This is out list of data directories that we will use to configure Minio in FreeBSD’s main configuration /etc/rc.conf file.

http://storage1:9000/data1 http://storage1:9000/data2 http://storage2:9000/data1 http://storage2:9000/data2 http://storage2:9000/data3 http://storage2:9000/data4 http://storage2:9000/data5 http://storage2:9000/data6 http://storage2:9000/data7 http://storage3:9000/data1 http://storage3:9000/data2 http://storage3:9000/data3 http://storage3:9000/data4 http://storage3:9000/data5 http://storage3:9000/data6 http://storage3:9000/data7

Now, lets put Minio settings into the /etc/rc.conf file.

root@storageX:~ # cat > /etc/rc.conf << __EOF 
minio_enable=YES
minio_disks="http://storage1:9000/data1 http://storage1:9000/data2 http://storage2:9000/data1 http://storage2:9000/data2 http://storage2:9000/data3 http://storage2:9000/data4 http://storage2:9000/data5 http://storage2:9000/data6 http://storage2:9000/data7 http://storage3:9000/data1 http://storage3:9000/data2 http://storage3:9000/data3 http://storage3:9000/data4 http://storage3:9000/data5 http://storage3:9000/data6 http://storage3:9000/data7"
__EOF
root@storageX:~ # 
root@storageX:~ # cat /etc/rc.conf
minio_enable=YES
minio_disks="http://storage1:9000/data1 http://storage1:9000/data2 http://storage2:9000/data1 http://storage2:9000/data2 http://storage2:9000/data3 http://storage2:9000/data4 http://storage2:9000/data5 http://storage2:9000/data6 http://storage2:9000/data7 http://storage3:9000/data1 http://storage3:9000/data2 http://storage3:9000/data3 http://storage3:9000/data4 http://storage3:9000/data5 http://storage3:9000/data6 http://storage3:9000/data7"
root@storageX:~ #

Now we will start and configure Minio for the first time.

On each storageX server run the following set of commands.

host # jexec storage3
root@storage3:~ # 
root@storage3:/ # rm -rf /http:\*
root@storage3:/ # rm -rf /usr/local/etc/minio
root@storage3:/ # rm -rf /data?/* /data?/.minio.sys
root@storage3:/ # touch                /var/log/minio.log
root@storage3:/ # chown    minio:minio /var/log/minio.log
root@storage3:/ # mkdir -p             /usr/local/etc/minio
root@storage3:/ # chown -R minio:minio /usr/local/etc/minio
root@storage3:/ # mkdir -p             /http::
root@storage3:/ # chown -R minio:minio /http::
root@storage3:/ # mkdir -p             /http:
root@storage3:/ # chown -R minio:minio /http:
root@storage3:/ # su -m minio -c 'env \\
?   MINIO_ACCESS_KEY=alibaba \\
?   MINIO_SECRET_KEY=0P3NS3S4M3 \\
?   minio server \\
?     --config-dir /usr/local/etc/minio \\
?     http://storage1:9000/data1 \\
?     http://storage1:9000/data2 \\
?     http://storage2:9000/data1 \\
?     http://storage2:9000/data2 \\
?     http://storage2:9000/data3 \\
?     http://storage2:9000/data4 \\
?     http://storage2:9000/data5 \\
?     http://storage2:9000/data6 \\
?     http://storage2:9000/data7 \\
?     http://storage3:9000/data1 \\
?     http://storage3:9000/data2 \\
?     http://storage3:9000/data3 \\
?     http://storage3:9000/data4 \\
?     http://storage3:9000/data5 \\
?     http://storage3:9000/data6 \\
?     http://storage3:9000/data7'
Created minio configuration file successfully at /usr/local/etc/minio
Waiting for the first server to format the disks.
Waiting for the first server to format the disks.
Drive Capacity: 504 GiB Free, 515 GiB Total
Status:         16 Online, 0 Offline. 

Endpoint:  http://192.168.43.103:9000
AccessKey: alibaba 
SecretKey: 0P3NS3S4M3 

Browser Access:
   http://192.168.43.103:9000

Command-line Access: https://docs.minio.io/docs/minio-client-quickstart-guide
   $ mc config host add myminio http://192.168.43.103:9000 alibaba 0P3NS3S4M3

Object API (Amazon S3 compatible):
   Go:         https://docs.minio.io/docs/golang-client-quickstart-guide
   Java:       https://docs.minio.io/docs/java-client-quickstart-guide
   Python:     https://docs.minio.io/docs/python-client-quickstart-guide
   JavaScript: https://docs.minio.io/docs/javascript-client-quickstart-guide
   .NET:       https://docs.minio.io/docs/dotnet-client-quickstart-guide
host # jexec storage2
root@storage2:~ # 
root@storage2:/ # rm -rf /http:\*
root@storage2:/ # rm -rf /usr/local/etc/minio
root@storage2:/ # rm -rf /data?/* /data?/.minio.sys
root@storage2:/ # touch                /var/log/minio.log
root@storage2:/ # chown    minio:minio /var/log/minio.log
root@storage2:/ # mkdir -p             /usr/local/etc/minio
root@storage2:/ # chown -R minio:minio /usr/local/etc/minio
root@storage2:/ # mkdir -p             /http::
root@storage2:/ # chown -R minio:minio /http::
root@storage2:/ # mkdir -p             /http:
root@storage2:/ # chown -R minio:minio /http:
root@storage2:/ # su -m minio -c 'env \\
?   MINIO_ACCESS_KEY=alibaba \\
?   MINIO_SECRET_KEY=0P3NS3S4M3 \\
?   minio server \\
?     --config-dir /usr/local/etc/minio \\
?     http://storage1:9000/data1 \\
?     http://storage1:9000/data2 \\
?     http://storage2:9000/data1 \\
?     http://storage2:9000/data2 \\
?     http://storage2:9000/data3 \\
?     http://storage2:9000/data4 \\
?     http://storage2:9000/data5 \\
?     http://storage2:9000/data6 \\
?     http://storage2:9000/data7 \\
?     http://storage3:9000/data1 \\
?     http://storage3:9000/data2 \\
?     http://storage3:9000/data3 \\
?     http://storage3:9000/data4 \\
?     http://storage3:9000/data5 \\
?     http://storage3:9000/data6 \\
?     http://storage3:9000/data7'
Created minio configuration file successfully at /usr/local/etc/minio
Waiting for the first server to format the disks.
Waiting for the first server to format the disks.
Drive Capacity: 504 GiB Free, 515 GiB Total
Status:         16 Online, 0 Offline. 

Endpoint:  http://192.168.43.102:9000
AccessKey: alibaba 
SecretKey: 0P3NS3S4M3 

Browser Access:
   http://192.168.43.102:9000

Command-line Access: https://docs.minio.io/docs/minio-client-quickstart-guide
   $ mc config host add myminio http://192.168.43.102:9000 alibaba 0P3NS3S4M3

Object API (Amazon S3 compatible):
   Go:         https://docs.minio.io/docs/golang-client-quickstart-guide
   Java:       https://docs.minio.io/docs/java-client-quickstart-guide
   Python:     https://docs.minio.io/docs/python-client-quickstart-guide
   JavaScript: https://docs.minio.io/docs/javascript-client-quickstart-guide
   .NET:       https://docs.minio.io/docs/dotnet-client-quickstart-guide
host # jexec storage1
root@storage1:~ # 
root@storage1:/ # rm -rf /http:\*
root@storage1:/ # rm -rf /usr/local/etc/minio
root@storage1:/ # rm -rf /data?/* /data?/.minio.sys
root@storage1:/ # touch                /var/log/minio.log
root@storage1:/ # chown    minio:minio /var/log/minio.log
root@storage1:/ # mkdir -p             /usr/local/etc/minio
root@storage1:/ # chown -R minio:minio /usr/local/etc/minio
root@storage1:/ # mkdir -p             /http::
root@storage1:/ # chown -R minio:minio /http::
root@storage1:/ # mkdir -p             /http:
root@storage1:/ # chown -R minio:minio /http:
root@storage1:/ # su -m minio -c 'env \\
?   MINIO_ACCESS_KEY=alibaba \\
?   MINIO_SECRET_KEY=0P3NS3S4M3 \\
?   minio server \\
?     --config-dir /usr/local/etc/minio \\
?     http://storage1:9000/data1 \\
?     http://storage1:9000/data2 \\
?     http://storage2:9000/data1 \\
?     http://storage2:9000/data2 \\
?     http://storage2:9000/data3 \\
?     http://storage2:9000/data4 \\
?     http://storage2:9000/data5 \\
?     http://storage2:9000/data6 \\
?     http://storage2:9000/data7 \\
?     http://storage3:9000/data1 \\
?     http://storage3:9000/data2 \\
?     http://storage3:9000/data3 \\
?     http://storage3:9000/data4 \\
?     http://storage3:9000/data5 \\
?     http://storage3:9000/data6 \\
?     http://storage3:9000/data7'
Created minio configuration file successfully at /usr/local/etc/minio
Waiting for the first server to format the disks.
Waiting for the first server to format the disks.
Drive Capacity: 504 GiB Free, 515 GiB Total
Status:         16 Online, 0 Offline. 

Endpoint:  http://192.168.43.101:9000
AccessKey: alibaba 
SecretKey: 0P3NS3S4M3 

Browser Access:
   http://192.168.43.101:9000

Command-line Access: https://docs.minio.io/docs/minio-client-quickstart-guide
   $ mc config host add myminio http://192.168.43.101:9000 alibaba 0P3NS3S4M3

Object API (Amazon S3 compatible):
   Go:         https://docs.minio.io/docs/golang-client-quickstart-guide
   Java:       https://docs.minio.io/docs/java-client-quickstart-guide
   Python:     https://docs.minio.io/docs/python-client-quickstart-guide
   JavaScript: https://docs.minio.io/docs/javascript-client-quickstart-guide
   .NET:       https://docs.minio.io/docs/dotnet-client-quickstart-guide

Here is how it looks in the xterm terminal.

minio-first-run-setup

We can now verify in the browser that it actually works.

minio-browser-01

Now hit [CTRL]+[C] in each of these windows to stop the Minio cluster.

We will now start Minio with FreeBSD rc(8) subsystem as a service.

root@storage1:/ # service minio start
Starting minio.
root@storage1:/ # cat /var/log/minio.log 
root@storage1:/ # service minio status
minio is running as pid 50309.

Lets check if it works.

root@storage1:/ # ps -U minio
  PID TT  STAT    TIME COMMAND
50308  -  IsJ  0:00.00 daemon: /usr/bin/env[50309] (daemon)
50309  -  IJ   0:00.27 /usr/local/bin/minio -C /usr/local/etc/minio server (...)

Now we will do some basic operations, login into Minio distributed storage, create new bucket and upload some file to it.

minio-browser-02

This is how empty Minio cluster looks like.

minio-browser-03

Select Create Bucket option from the button below.

minio-browser-04-create-bucket

We will use name test for our new bucket.

minio-browser-05-create-bucket

It is created and we can access it.

minio-browser-06-bucket

Lets Upload File using same menu as previously.

minio-browser-07-file-upload

The upload progress shown by Minio.

minio-browser-08-file-upload

File has been indeed uploaded.

minio-browser-09-file-upload

By clicking on it we may access it directly from the browser.

minio-browser-10-file-display

We can also share link to that file by using the File Menu as shown below.

minio-browser-10-file-link

The link creation dialog is shown below.

minio-browser-11-file-link

minio-browser-12-file-link

Lets see how Minio distributes the data – the ThinkPad Design – Spirit and Essence.pdf file in out case – over its data directories spread across the servers.

host # jexec storage1
root@storage1:/ # find /data?/test
/data1/test
/data1/test/ThinkPad Design - Spirit and Essence.pdf
/data1/test/ThinkPad Design - Spirit and Essence.pdf/xl.json
/data1/test/ThinkPad Design - Spirit and Essence.pdf/part.1
/data2/test
/data2/test/ThinkPad Design - Spirit and Essence.pdf
/data2/test/ThinkPad Design - Spirit and Essence.pdf/xl.json
/data2/test/ThinkPad Design - Spirit and Essence.pdf/part.1
root@storage1:/ # exit
host # jexec storage2
root@storage2:/ # find /data?/test
/data1/test
/data1/test/ThinkPad Design - Spirit and Essence.pdf
/data1/test/ThinkPad Design - Spirit and Essence.pdf/part.1
/data1/test/ThinkPad Design - Spirit and Essence.pdf/xl.json
/data2/test
/data2/test/ThinkPad Design - Spirit and Essence.pdf
/data2/test/ThinkPad Design - Spirit and Essence.pdf/xl.json
/data2/test/ThinkPad Design - Spirit and Essence.pdf/part.1
/data3/test
/data3/test/ThinkPad Design - Spirit and Essence.pdf
/data3/test/ThinkPad Design - Spirit and Essence.pdf/part.1
/data3/test/ThinkPad Design - Spirit and Essence.pdf/xl.json
/data4/test
/data4/test/ThinkPad Design - Spirit and Essence.pdf
/data4/test/ThinkPad Design - Spirit and Essence.pdf/part.1
/data4/test/ThinkPad Design - Spirit and Essence.pdf/xl.json
/data5/test
/data5/test/ThinkPad Design - Spirit and Essence.pdf
/data5/test/ThinkPad Design - Spirit and Essence.pdf/part.1
/data5/test/ThinkPad Design - Spirit and Essence.pdf/xl.json
/data6/test
/data6/test/ThinkPad Design - Spirit and Essence.pdf
/data6/test/ThinkPad Design - Spirit and Essence.pdf/part.1
/data6/test/ThinkPad Design - Spirit and Essence.pdf/xl.json
/data7/test
/data7/test/ThinkPad Design - Spirit and Essence.pdf
/data7/test/ThinkPad Design - Spirit and Essence.pdf/xl.json
/data7/test/ThinkPad Design - Spirit and Essence.pdf/part.1
root@storage2:/ # exit
host # jexec storage3
root@storage3:/ # find /data?/test
/data1/test
/data1/test/ThinkPad Design - Spirit and Essence.pdf
/data1/test/ThinkPad Design - Spirit and Essence.pdf/part.1
/data1/test/ThinkPad Design - Spirit and Essence.pdf/xl.json
/data2/test
/data2/test/ThinkPad Design - Spirit and Essence.pdf
/data2/test/ThinkPad Design - Spirit and Essence.pdf/xl.json
/data2/test/ThinkPad Design - Spirit and Essence.pdf/part.1
/data3/test
/data3/test/ThinkPad Design - Spirit and Essence.pdf
/data3/test/ThinkPad Design - Spirit and Essence.pdf/xl.json
/data3/test/ThinkPad Design - Spirit and Essence.pdf/part.1
/data4/test
/data4/test/ThinkPad Design - Spirit and Essence.pdf
/data4/test/ThinkPad Design - Spirit and Essence.pdf/part.1
/data4/test/ThinkPad Design - Spirit and Essence.pdf/xl.json
/data5/test
/data5/test/ThinkPad Design - Spirit and Essence.pdf
/data5/test/ThinkPad Design - Spirit and Essence.pdf/part.1
/data5/test/ThinkPad Design - Spirit and Essence.pdf/xl.json
/data6/test
/data6/test/ThinkPad Design - Spirit and Essence.pdf
/data6/test/ThinkPad Design - Spirit and Essence.pdf/part.1
/data6/test/ThinkPad Design - Spirit and Essence.pdf/xl.json
/data7/test
/data7/test/ThinkPad Design - Spirit and Essence.pdf
/data7/test/ThinkPad Design - Spirit and Essence.pdf/xl.json
/data7/test/ThinkPad Design - Spirit and Essence.pdf/part.1
root@storage3:/ # exit

We can also see what Minio configuration file /usr/local/etc/minio/config.json has been generated.

host # jexec storage1
root@storage1:/ # cat /usr/local/etc/minio/config.json 
{
        "version": "22",
        "credential": {
                "accessKey": "alibaba",
                "secretKey": "0P3NS3S4M3"
        },
        "region": "",
        "browser": "on",
        "domain": "",
        "storageclass": {
                "standard": "",
                "rrs": ""
        },
        "notify": {
                "amqp": {
                        "1": {
                                "enable": false,
                                "url": "",
                                "exchange": "",
                                "routingKey": "",
                                "exchangeType": "",
                                "deliveryMode": 0,
                                "mandatory": false,
                                "immediate": false,
                                "durable": false,
                                "internal": false,
                                "noWait": false,
                                "autoDeleted": false
                        }
                },
                "elasticsearch": {
                        "1": {
                                "enable": false,
                                "format": "",
                                "url": "",
                                "index": ""
                        }
                },
                "kafka": {
                        "1": {
                                "enable": false,
                                "brokers": null,
                                "topic": ""
                        }
                },
                "mqtt": {
                        "1": {
                                "enable": false,
                                "broker": "",
                                "topic": "",
                                "qos": 0,
                                "clientId": "",
                                "username": "",
                                "password": "",
                                "reconnectInterval": 0,
                                "keepAliveInterval": 0
                        }
                },
                "mysql": {
                        "1": {
                                "enable": false,
                                "format": "",
                                "dsnString": "",
                                "table": "",
                                "host": "",
                                "port": "",
                                "user": "",
                                "password": "",
                                "database": ""
                        }
                },
                "nats": {
                        "1": {
                                "enable": false,
                                "address": "",
                                "subject": "",
                                "username": "",
                                "password": "",
                                "token": "",
                                "secure": false,
                                "pingInterval": 0,
                                "streaming": {
                                        "enable": false,
                                        "clusterID": "",
                                        "clientID": "",
                                        "async": false,
                                        "maxPubAcksInflight": 0
                                }
                        }
                },
                "postgresql": {
                        "1": {
                                "enable": false,
                                "format": "",
                                "connectionString": "",
                                "table": "",
                                "host": "",
                                "port": "",
                                "user": "",
                                "password": "",
                                "database": ""
                        }
                },
                "redis": {
                        "1": {
                                "enable": false,
                                "format": "",
                                "address": "",
                                "password": "",
                                "key": ""
                        }
                },
                "webhook": {
                        "1": {
                                "enable": false,
                                "endpoint": ""
                        }
                }
        }

S3FS

We can also mount that test bucket from out distributed Minio object storage cluster as filesystem using the S3FS project. Lets add s3fs package and mount our bucket.

host # pkg install -y fusefs-s3fs

Now we will configure password for our bucket.

host # echo test:alibaba:0P3NS3S4M3 > /root/.passwd-s3fs
host # chmod 600 /root/.passwd-s3fs
host # cat /root/.passwd-s3fs 
test:alibaba:0P3NS3S4M3

Now lets do the actual mount.

host # mkdir /tmp/test
host # s3fs \
  -o allow_other \
  -o use_path_request_style \
  -o url=http://192.168.43.101:9000 \
  -o passwd_file=/root/.passwd-s3fs \
  test /tmp/test

The file ThinkPad Design – Spirit and Essence.pdf that we put through web interface should be here.

host # exa -l /tmp/test
.--------- 10M root 2018-04-16 14:15 ThinkPad Design - Spirit and Essence.pdf

host # file /tmp/test/ThinkPad\ Design\ -\ Spirit\ and\ Essence.pdf 
/tmp/test/ThinkPad Design - Spirit and Essence.pdf: PDF document, version 1.4

host # stat /tmp/test/ThinkPad\ Design\ -\ Spirit\ and\ Essence.pdf
3976265496 2 ---------- 1 root wheel 0 10416953 "Jan  1 01:00:00 1970" "Apr 16 14:35:35 2018" "Jan  1 01:00:00 1970" "Jan  1 00:59:59 1970" 4096 20346 0 /tmp/test/ThinkPad Design - Spirit and Essence.pdf

We can now upload other file into that bucket using s3fs mount.

host # cp -v /home/vermaden/On\ the\ Shortness\ of\ Life\ -\ Lucius\ Seneca.pdf /tmp/test
/home/vermaden/On the Shortness of Life - Lucius Seneca.pdf -> /tmp/test/On the Shortness of Life - Lucius Seneca.pdf

host # file /tmp/test/On\ the\ Shortness\ of\ Life\ -\ Lucius\ Seneca.pdf 
On the Shortness of Life - Lucius Seneca.pdf: PDF document, version 1.4

We can also verify that our file put through s3fs is visible on the web interface.

minio-browser-13-s3fs-upload

Real Hardware

Now, as we have working Proof of Concept for the distributed Minio setup how about putting it on a real hardware for real storage purposes? I would setup a 16 node Minio distributed server on a Supermicro SSG-5018D8-AR12L hardware. Supermicro even suggests using that kind of servers for object storage, here is their white paper on that topic – Object Storage Solution for Data Archive using Supermicro SSG-5018D8-AR12L and OpenIO SDS – but they use OpenIO not Minio for distributed object storage solution.

This server features the Supermicro X10SDV-7TP4F motherboard. This is important as this motherboard officially supports FreeBSD 11.x operating system on their Supermicro OS Compatibility page.

Motherboard specification has these features.

 1 x Intel Xeon D-1537 8-Core / 16-Threads TDP 35W
 4 x UDIMM for up to 128GB ECC RDIMM DDR4 2133MHz
12 x 3.5" SAS2/SATA3 Hot-Swap HDD Bays
 4 x 2.5" Cold-Swap HDD Bays
 1 x Controller Intel SoC for 4 SATA3 (6Gbps) Ports
 1 x Controller Broadcom 2116 for 16 SATA3 (6Gbps) Ports
 1 x Expansion Slot PCI-E 3.0 x8 
 1 x Expansion Slot M.2 PCIe 3.0 x4
 1 x Expansion Slot Mini-PCIe w/ mSATA Support
 2 x 10G SFP+ Port
 2 x 1GbE LAN Port
 2 x External USB 3.0 Port
 1 x Interlal USB 2.0 Port
 2 x 400W High-Rfficiency Redundant Power Supplies

You can configure your own and get approximated price using the Thinkmate site from here:
https://www.thinkmate.com/system/superstorage-server-5018d8-ar12l

I would add this components to the basic setup:

 4 x UDIMM FULL 128 GB ECC RDIMM DDR4
 2 x 240GB Micron 5100 MAX 2.5" SATA 6.0Gb/s SSD
 2 x 7.68TB Micron 5200 ECO Series 2.5" SATA 6.0Gb/s SSD
12 x 12TB SATA 6.0Gb/s 7200RPM 3.5" Hitachi Ultrastarβ„’ He12
 3 x SanDisk Cruzer Fit 32GB USB 3.0

Now, I will use the 3 x SanDisk Cruzer Fit 32GB USB 3.0 disks to install FreeBSD as a ZFS root/boot pool with mirror + spare on these disks. We do not need performance here.

Then, the 12 x 12TB SATA 6.0Gb/s 7200RPM 3.5″ Hitachi Ultrastarβ„’ He12 drives will be used as RAIDZ (RAID5 equivalent in ZFS without the write hole) for the Minio data, wich 11 + 1 setup, which means 11 drives for data and 1 drive for parity. As we can lose HALF of the Minio servers I would not waste 12 TB drive for spare here. Then, I would use 2 x 240GB Micron 5100 MAX 2.5″ SATA 6.0Gb/s SSD in mirror for the ZFS ZIL (ZFS Intent Log) to accelerate writes and 2 x 7.68TB Micron 5200 ECO Series 2.5″ SATA 6.0Gb/s SSD for the ZFS read cache (L2ARC).

The network would be setup on 2 x 10G SFP+ Port with LACP as lagg0 interface so each server would have 20 Gbit connectivity. This will give us a total of 320 Gbit theoretical network throughput.

This setup would give as 132 TB ZFS pool space with 15 TB for read cache and 240 GB for writes for single 1U server. Making the calculations this will give as 2112 TB (more then 2 PB) of space for Minio data.

With Minio algorithm for data redundancy we will have about 1 PB of usable storage space in our 16U Object Storage FreeBSD Appliance.

Not bad for my taste πŸ™‚

UPDATE 1

The Distributed Object Storage with Minio on FreeBSD article was included in the BSD Now 246 – Disclosure episode.

Thanks for mentioning!

EOF

Nextcloud 13 on FreeBSD

Today I would like to share a setup of Nextcloud 13 running on a FreeBSD system. To make things more interesting it would be running inside a FreeBSD Jail. I will not describe the Nextcloud setup itself here as its large enough for several blog posts.

Official Nextcloud 13 documentation recommends following setup:

  • MySQL/MariaDB
  • PHP 7.0 (or newer)
  • Apache 2.4 (with mod_php)

I prefer PostgreSQL database to MySQL/MariaDB and I prefer fast and lean Nginx web server to Apache, so my setup is based on these components:

  • PostgreSQL 10.3
  • PHP 7.2.4
  • Nginx 1.12.2 (with php-fpm)
  • Memcached 1.5.7

The Memcached subsystem is least important, it can be easily changed into something more modern like Redis for example. I prefer not to use any third party tools for FreeBSD Jails management. Not because they are bad or something like that. There are just many choices for good FreeBSD Jails management and I want to provide a GENERIC example for Nextcloud 13 in a Jail, not for a specific management tool.

Host

Lets start with preparing the FreeBSD Host with needed settings. We need to allow using raw sockets in Jails. For the future optional upgrades of the Jail we will also allow using chflags(1) in Jails.

host # cat >> /etc/sysctl.conf << __EOF

# ALLOW JAIL RAW SOCKETS
security.jail.allow_raw_sockets=1

# ALLOW UPGRADES IN JAIL 
security.jail.chflags_allowed=1
__EOF

host # sysctl security.jail.allow_raw_sockets=1
security.jail.allow_raw_sockets: 0 -> 1

host # sysctl security.jail.chflags_allowed=1
security.jail.chflags_allowed: 0 -> 1

I would also enable rctl(8) limits for convenient resource limitations on the host system.

host # cat >> /boot/loader.conf << __EOF

# RACCT/RCTL RESOURCE LIMITS
kern.racct.enable=1
__EOF

The complete Jail after finished installation takes less size then 800 MB if You remove not needed parts after installation is finished. With complete FreeBSD Ports tree and current portsnap(8) information it takes about 1.6 GB of space.

  MB PATH                            DESC
1670 /jail/nextcloud                 (complete Nextcloud 13 Jail)
 726 /jail/nextcloud/usr/ports       (can be removed after install)
 178 /jail/nextcloud/var/db/portsnap (can be removed after install)

I have used my laptop for the Jail host. This is why Jail will configured to use the wireless wlan0 interface and 192.168.43.100 address.

To distinguish the commands I type on the host system and nextcloud.local Jail I use two different prompts, this way it should be obvious what command to execute and where.

Command on the host system.

host # command

Command on the nextcloud.local Jail.

root@nextcloud:/ # command

Here is the running Jail and its processes.

host # jls
   JID  IP Address      Hostname                      Path
    10  192.168.43.100  nextcloud.local               /jail/nextcloud
host # ps axwww -o %cpu,rss,time,command -J nextcloud
%CPU   RSS    TIME COMMAND
 0.0  2032 0:00.01 /usr/sbin/syslogd -s -s
 0.0  5504 0:00.00 /usr/sbin/sshd
 0.0  2056 0:00.01 /usr/sbin/cron -s
 0.0 24196 0:00.04 postgres: checkpointer process    (postgres)
 0.0 23040 0:00.04 postgres: writer process    (postgres)
 0.0 23036 0:00.07 postgres: wal writer process    (postgres)
 0.0 23328 0:00.06 postgres: autovacuum launcher process    (postgres)
 0.0 12764 0:00.24 postgres: stats collector process    (postgres)
 0.0 23204 0:00.00 postgres: bgworker: logical replication launcher    (postgres)
 0.0 23036 0:00.23 /usr/local/bin/postgres -D /var/db/postgres/data
 0.0  6072 0:00.00 nginx: master process /usr/local/sbin/nginx
 0.0  6548 0:00.00 nginx: worker process (nginx)
 0.0  7604 0:00.15 nginx: worker process (nginx)
 0.0  6548 0:00.00 nginx: worker process (nginx)
 0.0  6544 0:00.00 nginx: worker process (nginx)
 0.0 17600 0:01.25 /usr/local/bin/memcached -l 192.168.43.100 -d -P /var/run/memcached/memcached.pid
 0.0 31372 0:00.01 php-fpm: master process (/usr/local/etc/php-fpm.conf) (php-fpm)
 0.0 31388 0:00.00 php-fpm: pool www (php-fpm)
 0.0 31388 0:00.00 php-fpm: pool www (php-fpm)
 0.0 31388 0:00.00 php-fpm: pool www (php-fpm)
 0.0 31388 0:00.00 php-fpm: pool www (php-fpm)

Jail

First we will prepare the Jail for our Nextcloud 13 installation. Lets create some ZFS datasets for that purpose. I will use my local ZFS pool. I will also create the ZFS dataset for PostgreSQL data with 8k record size.

host # zfs create -o mountpoint=/jail                                                 local/jail
host # zfs create -o mountpoint=/jail/nextcloud                                       local/jail/nextcloud
host # zfs create -o mountpoint=/jail/nextcloud/var/db/postgres/data -o recordsize=8k local/jail/nextcloud/pgsql
host # zfs get -r recordsize local/jail
NAME                        PROPERTY    VALUE    SOURCE
local/jail                  recordsize  128K     default
local/jail/nextcloud        recordsize  128K     default
local/jail/nextcloud/pgsql  recordsize  8K       local

Now lets fetch the FreeBSD base into /jail/nextcloud path.

host # cd /jail/nextcloud
host # fetch -o - http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/11.1-RELEASE/base.txz | tar --unlink -xpJf - -C /jail/nextcloud
-                                             100% of   99 MB  689 kBps 02m28s
host # ls /jail/nextcloud
.cshrc     bin/       COPYRIGHT  etc/       libexec/   mnt/       proc/      root/      sys        usr/
.profile   boot/      dev/       lib/       media/     net/       rescue/    sbin/      tmp/       var/

We have base FreeBSD Jail fetched into /jail/nextcloud path, lets configure host for that Jail.

We will only have one Jail configured (for simplicity), the nextcloud.local Jail.

host # cat >> /etc/jail.conf << __EOF
nextcloud {
  host.hostname = nextcloud.local;
  ip4.addr = 192.168.43.100;
  interface = wlan0;
  path = /jail/nextcloud;
  exec.start = "/bin/sh /etc/rc";
  exec.stop = "/bin/sh /etc/rc.shutdown";
  exec.clean;
  mount.devfs;
  allow.raw_sockets;
  allow.sysvipc;
}
__EOF

After creating/modifying the /etc/jail.conf file there should not be any running Jails.

host # jls
   JID  IP Address      Hostname                      Path

Lets enable Jails on the host system.

host # cat >> /etc/rc.conf << __EOF
# JAILS
  jail_enable=YES
__EOF

We can now start out nextcloud.local Jail for the first time.

host # service jail start nextcloud
Starting jails: nextcloud.
host #Β jls
   JID  IP Address      Hostname                      Path
     1  192.168.43.100  nextcloud.local               /jail/nextcloud

Now lets configure the nextcloud.local name on both host and Jail.

host # cat >> /etc/hosts << __EOF

# NEXTCLOUD
192.168.43.100 nextcloud.local nextcloud
__EOF
host # jexec 1 /bin/csh

root@nextcloud:/ # cat >> /etc/hosts << __EOF

# NEXTCLOUD
192.168.43.100 nextcloud.local nextcloud
__EOF

One has to remember that there is no localhost (127.0.0.1) in the FreeBSD Jail. The Jail only has itself configure IP address for listening purposes (192.168.43.100 in our example). This is important because if You configure services on the host that listen on localhost (127.0.0.1) they will work as usual, when You do the same in a FreeBSD Jail you will not able to connect to them (even from this very Jail).

Lets make some basic configuration of the Nextcloud Jail.

host # jexec 1 /bin/csh

root@nextcloud:/ # newaliases -v
WARNING: local host name (nextcloud) is not qualified; see cf/README: WHO AM I?
/etc/mail/aliases: 29 aliases, longest 10 bytes, 297 bytes total

root@nextcloud:/ # cp /usr/share/zoneinfo/Europe/Warsaw /etc/localtime

Lets create basic /etc/rc.conf file for out Jail. I will leave some services commented out as they are not yet configured to run, we do not want to imitate Debian here and start services with default configs πŸ™‚

root@nextcloud:/ # cat >> /etc/rc.conf << __EOF
# DAEMONS | yes
  syslogd_flags="-s -s"
  sshd_enable=YES
# php_fpm_enable=YES
# postgresql_enable=YES
# postgresql_class=postgres
# postgresql_data=/var/db/postgres/data
# memcached_enable=YES
# memcached_flags="-l 192.168.43.100"
# nginx_enable=YES

# DAEMONS | no
  sendmail_enable=NONE
  sendmail_submit_enable=NO
  sendmail_outbound_enable=NO
  sendmail_msp_queue_enable=NO

# OTHER
  clear_tmp_enable=YES
  clear_tmp_X=YES
  extra_netfs_types=NFS
  dumpdev=NO
  update_motd=NO
  keyrate=fast
__EOF

As we will disable sendmail(8) we need to make sure that the /var/spool/clientmqueue would not fill up with time. Lets configure simple cron job for that.

root@nextcloud:/ # cat > /etc/cron.d/sendmail-clean-clientmqueue << __EOF
 # CLEAN SENDMAIL
 0 * * * * root /bin/rm -r -f /var/spool/clientmqueue/*
 __EOF

As we have some basic configuration lets restart our Jail.

root@nextcloud:/ # exit

host # service jail restart nextcloud
Stopping jails: nextcloud.
Starting jails: nextcloud.

host # jls
   JID  IP Address      Hostname                      Path
     2  192.168.43.100  nextcloud.local               /jail/nextcloud

host # jexec nextcloud /bin/csh

After restart we only have sshd(8) daemon listening for connections.

root@nextcloud:/ # sockstat -l4
USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS
root     sshd       97823 3  tcp4   192.168.43.100:22         *:*

Packages

Lets configure network connectivity on the Jail as it will be needed to get the packages from Internet.

root@nextcloud:/ # echo nameserver 1.1.1.1 > /etc/resolv.conf

root@nextcloud:/ # ping -c 3 -t 3 freebsd.org
PING freebsd.org (8.8.178.110): 56 data bytes
64 bytes from 8.8.178.110: icmp_seq=0 ttl=52 time=180.860 ms
64 bytes from 8.8.178.110: icmp_seq=1 ttl=52 time=180.373 ms
64 bytes from 8.8.178.110: icmp_seq=2 ttl=52 time=181.363 ms

--- freebsd.org ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 180.373/180.865/181.363/0.404 ms

As we want the latest packages lets set that in the pkg(8) repository config file.

root@nextcloud:/ # grep quarterly /etc/pkg/FreeBSD.conf
  url: "pkg+http://pkg.FreeBSD.org/${ABI}/quarterly",

root@nextcloud:/ # sed -i '' s/quarterly/latest/g /etc/pkg/FreeBSD.conf

root@nextcloud:/ # grep latest /etc/pkg/FreeBSD.conf
  url: "pkg+http://pkg.FreeBSD.org/${ABI}/latest",

Now lets setup pkg(8) and fetch latest repository metadata.

root@nextcloud:/ # pkg update -f
The package management tool is not yet installed on your system.
Do you want to fetch and install it now? [y/N]: y
Bootstrapping pkg from pkg+http://pkg.FreeBSD.org/FreeBSD:11:amd64/latest, please wait...
Verifying signature with trusted certificate pkg.freebsd.org.2013102301... done
[nextcloud] Installing pkg-1.10.5...
[nextcloud] Extracting pkg-1.10.5: 100%
Updating FreeBSD repository catalogue...
pkg: Repository FreeBSD load error: access repo file(/var/db/pkg/repo-FreeBSD.sqlite) failed: No such file or directory
[nextcloud] Fetching meta.txz: 100%    944 B   0.9kB/s    00:01
[nextcloud] Fetching packagesite.txz: 100%    6 MiB 530.8kB/s    00:12
Processing entries: 100%
FreeBSD repository update completed. 31134 packages processed.
All repositories are up to date.

… and up to date FreeBSD Ports tree.

root@nextcloud:/ # portsnap fetch extract
Looking up portsnap.FreeBSD.org mirrors... 6 mirrors found.
Fetching public key from ec2-eu-west-1.portsnap.freebsd.org... done.
Fetching snapshot tag from ec2-eu-west-1.portsnap.freebsd.org... done.
Fetching snapshot metadata... done.
Fetching snapshot generated at Mon Apr  2 02:06:03 CEST 2018:
7cd019f9e1af8a9d637a56ba3d2bbc2025f54d9931cd8b100% of   79 MB  624 kBps 02m10s
Extracting snapshot... done.
Verifying snapshot integrity...
(...)
Building new INDEX files... done.

root@nextcloud:/ # portsnap fetch update
Looking up portsnap.FreeBSD.org mirrors... 6 mirrors found.
Fetching snapshot tag from ec2-eu-west-1.portsnap.freebsd.org... done.
Ports tree hasn't changed since last snapshot.
No updates needed.
Ports tree is already up to date.

By default Nextcloud 13 package in repository is built with MySQL 5.6 and older PHP 5.6, thus we can not use packages for everything, some (automated) compilation is unavoidable.

root@nextcloud:/ # cd /usr/ports/www/nextcloud

root@nextcloud:/usr/ports/www/nextcloud # make run-depends-list | grep -m 1 php
/usr/ports/lang/php56

root@nextcloud:/usr/ports/www/nextcloud # make run-depends-list | grep -m 1 database
/usr/ports/databases/mysql56-client

Lets check what are the FreeBSD Ports default packages versions.

root@nextcloud:/ # grep -E '^[A-Z]+_DEFAULT' /usr/ports/Mk/bsd.default-versions.mk | column -t
APACHE_DEFAULT?=       2.4
BDB_DEFAULT?=          5
FIREBIRD_DEFAULT?=     2.5
FORTRAN_DEFAULT?=      gfortran
FPC_DEFAULT?=          3.0.4
GCC_DEFAULT?=          6
GHOSTSCRIPT_DEFAULT?=  agpl
LAZARUS_DEFAULT?=      1.8.2
LINUX_DEFAULT?=        c6_64
LINUX_DEFAULT?=        c6
LUA_DEFAULT?=          5.2
MYSQL_DEFAULT?=        5.6
PGSQL_DEFAULT?=        9.5
PHP_DEFAULT?=          5.6
PYTHON_DEFAULT?=       2.7
RUBY_DEFAULT?=         2.4
SAMBA_DEFAULT?=        4.6
SSL_DEFAULT=           base
SSL_DEFAULT:=          ${OPENSSL_INSTALLED:T}
SSL_DEFAULT?=          base
TCLTK_DEFAULT?=        8.6
VARNISH_DEFAULT?=      4

Its PostgreSQL 9.5 and PHP 5.6. We will override that in /etc/make.conf file with the following settings. We will also force using PGSQL option and disable MYSQL option for all Ports.

root@nextcloud:/ # cat >> /etc/make.conf << __EOF
WRKDIRPREFIX=\${PORTSDIR}/obj
DEFAULT_VERSIONS+= php=7.2
DEFAULT_VERSIONS+= pgsql=10
OPTIONS_UNSET+=    MYSQL
OPTIONS_SET+=      PGSQL
__EOF

Now, lets display the default Nextcloud port configuration.

root@nextcloud:/usr/ports/www/nextcloud # make showconfig
===> The following configuration options are available for nextcloud-13.0.0:
     EXIF=on: Image rotation support
     LDAP=on: LDAP protocol support
     SMB=on: SMB network protocol support
     SSL=on: SSL protocol support
====> Database backend(s): you have to choose at least one of them
     MYSQL=on: MySQL database support
     PGSQL=off: PostgreSQL database support
     SQLITE=off: SQLite database support
===> Use 'make config' to modify these settings

PostgreSQL support is not even enabled. Lets configure the Nextcloud port to our needs.

root@nextcloud:/usr/ports/www/nextcloud # make config

nextcloud-06-xterm-make-config
Lets check current port configuration.

root@nextcloud:/usr/ports/www/nextcloud # make showconfig
===> The following configuration options are available for nextcloud-13.0.0:
     EXIF=on: Image rotation support
     LDAP=on: LDAP protocol support
     SMB=on: SMB network protocol support
     SSL=on: SSL protocol support
====> Database backend(s): you have to choose at least one of them
     MYSQL=off: MySQL database support
     PGSQL=on: PostgreSQL database support
     SQLITE=off: SQLite database support
===> Use 'make config' to modify these settings

Good. In case You wandered where these settings are stored below is the answer. Yes, if you delete /var/db/ports/www_nextcloud directory, they will be brought back to defaults without PostgreSQL and with MySQL.

root@nextcloud:/ # cat /var/db/ports/www_nextcloud/options
# This file is auto-generated by 'make config'.
# Options for nextcloud-13.0.0
_OPTIONS_READ=nextcloud-13.0.0
_FILE_COMPLETE_OPTIONS_LIST=EXIF LDAP SMB SSL MYSQL PGSQL SQLITE
OPTIONS_FILE_SET+=EXIF
OPTIONS_FILE_SET+=LDAP
OPTIONS_FILE_SET+=SMB
OPTIONS_FILE_SET+=SSL
OPTIONS_FILE_UNSET+=MYSQL
OPTIONS_FILE_SET+=PGSQL
OPTIONS_FILE_UNSET+=SQLITE

Now lets check again for the run-depends-list after our configuration.

root@nextcloud:/ # make -C /usr/ports/www/nextcloud run-depends-list | grep -m 1 php
/usr/ports/lang/php72

root@nextcloud:/ # make -C /usr/ports/www/nextcloud run-depends-list | grep -m 1 sql
/usr/ports/databases/postgresql10-client

Better.

To make things little faster and to not build everything from FreeBSD Ports we may add most of needed software from packages. We will have to switch for /bin/sh shell for that purpose.

root@nextcloud:/usr/ports/www/nextcloud # exit

host # jexec nextcloud /bin/sh

root@nextcloud:/ # make -C /usr/ports/www/nextcloud run-depends-list | while read I; do echo pkg install -y $( basename $I );done
pkg install -y gettext-runtime
pkg install -y postgresql10-client
pkg install -y pecl-smbclient
pkg install -y php72
pkg install -y php72-bz2
pkg install -y php72-ctype
pkg install -y php72-curl
pkg install -y php72-dom
pkg install -y php72-fileinfo
pkg install -y php72-filter
pkg install -y php72-gd
pkg install -y php72-hash
pkg install -y php72-iconv
pkg install -y php72-json
pkg install -y php72-mbstring
pkg install -y php72-pdo
pkg install -y php72-posix
pkg install -y php72-session
pkg install -y php72-simplexml
pkg install -y php72-xml
pkg install -y php72-xmlreader
pkg install -y php72-xmlwriter
pkg install -y php72-xsl
pkg install -y php72-wddx
pkg install -y php72-zip
pkg install -y php72-zlib
pkg install -y php72-exif
pkg install -y php72-ldap
pkg install -y php72-openssl
pkg install -y php72-pdo_pgsql
pkg install -y php72-pgsql

So we have our list of commands to install most packages, lets paste them one by one into the nextcloud.local prompt. As packages are build, they sometimes get a prefix against what version of ‘upstream’ port it has been built, the ‘pecl-smbclient’ port is a good example here:

root@nextcloud:/ # pkg install -y pecl-smbclient
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
pkg: No packages available to install matching 'pecl-smbclient' have been found in the repositories

root@nextcloud:/ # pkg search pecl-smbclient
php56-pecl-smbclient-0.9.0_3   Smbclient wrapper extension
php70-pecl-smbclient-0.9.0_3   Smbclient wrapper extension
php71-pecl-smbclient-0.9.0_3   Smbclient wrapper extension
php72-pecl-smbclient-0.9.0_3   Smbclient wrapper extension

Here is complete list of packages that we need to install.

root@nextcloud:/ # pkg install -y gettext-runtime
root@nextcloud:/ # pkg install -y postgresql10-client
root@nextcloud:/ # pkg install -y pecl-smbclient
root@nextcloud:/ # pkg install -y php72
root@nextcloud:/ # pkg install -y php72-bz2
root@nextcloud:/ # pkg install -y php72-ctype
root@nextcloud:/ # pkg install -y php72-curl
root@nextcloud:/ # pkg install -y php72-dom
root@nextcloud:/ # pkg install -y php72-fileinfo
root@nextcloud:/ # pkg install -y php72-filter
root@nextcloud:/ # pkg install -y php72-gd
root@nextcloud:/ # pkg install -y php72-hash
root@nextcloud:/ # pkg install -y php72-iconv
root@nextcloud:/ # pkg install -y php72-json
root@nextcloud:/ # pkg install -y php72-mbstring
root@nextcloud:/ # pkg install -y php72-pdo
root@nextcloud:/ # pkg install -y php72-posix
root@nextcloud:/ # pkg install -y php72-session
root@nextcloud:/ # pkg install -y php72-simplexml
root@nextcloud:/ # pkg install -y php72-xml
root@nextcloud:/ # pkg install -y php72-xmlreader
root@nextcloud:/ # pkg install -y php72-xmlwriter
root@nextcloud:/ # pkg install -y php72-xsl
root@nextcloud:/ # pkg install -y php72-wddx
root@nextcloud:/ # pkg install -y php72-zip
root@nextcloud:/ # pkg install -y php72-zlib
root@nextcloud:/ # pkg install -y php72-exif
root@nextcloud:/ # pkg install -y php72-ldap
root@nextcloud:/ # pkg install -y php72-openssl
root@nextcloud:/ # pkg install -y php72-pdo_pgsql
root@nextcloud:/ # pkg install -y php72-pgsql
root@nextcloud:/ # pkg install -y php72-pecl-smbclient
root@nextcloud:/ # pkg install -y nginx
root@nextcloud:/ # pkg install -y memcached
root@nextcloud:/ # pkg install -y portmaster
root@nextcloud:/ # pkg install -y sudo
root@nextcloud:/ # pkg install -y php72-pecl-memcached
root@nextcloud:/ # pkg install -y php72-pcntl
root@nextcloud:/ # pkg install -y postgresql10-server

Some packages like postgresql10-server will remove packages build against the PostgreSQL 9.5 version like below.

root@nextcloud:/ # pkg install postgresql10-server
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
Checking integrity... done (2 conflicting)
  - postgresql10-client-10.3 conflicts with postgresql95-client-9.5.12 on /usr/local/bin/clusterdb
  - postgresql10-client-10.3 conflicts with postgresql95-client-9.5.12 on /usr/local/bin/clusterdb
Checking integrity... done (0 conflicting)
The following 5 package(s) will be affected (of 0 checked):

Installed packages to be REMOVED:
        postgresql95-client-9.5.12
        php72-pdo_pgsql-7.2.4
        php72-pgsql-7.2.4

New packages to be INSTALLED:
        postgresql10-server: 10.3
        postgresql10-client: 10.3

Number of packages to be removed: 3
Number of packages to be installed: 2

The process will require 21 MiB more space.

Proceed with this action? [y/N]: y

Now we need to build the rest.

root@nextcloud:/ # portmaster databases/php72-pgsql databases/php72-pdo_pgsql www/nextcloud www/php72-opcache devel/php72-intl mail/cclient mail/php72-imap math/php72-gmp ftp/php72-ftp
(...)
===>>> The following actions were performed:
        Installation of devel/gmake (gmake-4.2.1_2)
        Installation of devel/gettext-tools (gettext-tools-0.19.8.1)
        Installation of devel/p5-Locale-gettext (p5-Locale-gettext-1.07)
        Installation of misc/help2man (help2man-1.47.6)
        Installation of print/texinfo (texinfo-6.5,1)
        Installation of devel/m4 (m4-1.4.18,1)
        Installation of devel/autoconf-wrapper (autoconf-wrapper-20131203)
        Installation of devel/autoconf (autoconf-2.69_1)
        Installation of databases/php72-pgsql (php72-pgsql-7.2.4)
        Installation of databases/php72-pdo_pgsql (php72-pdo_pgsql-7.2.4)
        Installation of www/nextcloud (nextcloud-13.0.0)
        Installation of www/php72-opcache (php72-opcache-7.2.4)
        Installation of devel/php72-intl (php72-intl-7.2.4)
        Installation of mail/cclient (cclient-2007f_3,1)
        Installation of mail/php72-imap (php72-imap-7.2.4)
        Installation of math/php72-gmp (php72-gmp-7.2.4)
        Installation of ftp/php72-ftp (php72-ftp-7.2.4)

Alternatively to not juggle between packages and ports you may build everything from the FreeBSD Ports tree with command below.

root@nextcloud:/ # portmaster -y www/nextcloud www/nginx databases/memcached security/sudo databases/postgresql10-server www/php72-opcache devel/php72-intl mail/cclient mail/php72-imap math/php72-gmp ftp/php72-ftp

Whichever method you choose, You must end up with these packages installed. This list does not contain dependencies.

root@nextcloud:/ # pkg info | grep -E 'php|nginx|memcached|postgresql|sudo|nextcloud|portmaster'
libmemcached-1.0.18_6          C and C++ client library to the memcached server
memcached-1.5.7                High-performance distributed memory object cache system
nextcloud-13.0.0               Personal cloud which runs on your own server
nginx-1.12.2_11,2              Robust and small WWW server
php72-7.2.4                    PHP Scripting Language
php72-bz2-7.2.4                The bz2 shared extension for php
php72-ctype-7.2.4              The ctype shared extension for php
php72-curl-7.2.4               The curl shared extension for php
php72-dom-7.2.4                The dom shared extension for php
php72-exif-7.2.4               The exif shared extension for php
php72-fileinfo-7.2.4           The fileinfo shared extension for php
php72-filter-7.2.4             The filter shared extension for php
php72-ftp-7.2.4                The ftp shared extension for php
php72-gd-7.2.4                 The gd shared extension for php
php72-gmp-7.2.4                The gmp shared extension for php
php72-hash-7.2.4               The hash shared extension for php
php72-iconv-7.2.4              The iconv shared extension for php
php72-imap-7.2.4               The imap shared extension for php
php72-intl-7.2.4               The intl shared extension for php
php72-json-7.2.4               The json shared extension for php
php72-ldap-7.2.4               The ldap shared extension for php
php72-mbstring-7.2.4           The mbstring shared extension for php
php72-opcache-7.2.4            The opcache shared extension for php
php72-openssl-7.2.4            The openssl shared extension for php
php72-pcntl-7.2.4              The pcntl shared extension for php
php72-pdo-7.2.4                The pdo shared extension for php
php72-pdo_pgsql-7.2.4          The pdo_pgsql shared extension for php
php72-pecl-memcached-3.0.4     PHP extension for interfacing with memcached via libmemcached library
php72-pecl-smbclient-0.9.0_3   Smbclient wrapper extension
php72-pgsql-7.2.4              The pgsql shared extension for php
php72-posix-7.2.4              The posix shared extension for php
php72-session-7.2.4            The session shared extension for php
php72-simplexml-7.2.4          The simplexml shared extension for php
php72-wddx-7.2.4               The wddx shared extension for php
php72-xml-7.2.4                The xml shared extension for php
php72-xmlreader-7.2.4          The xmlreader shared extension for php
php72-xmlwriter-7.2.4          The xmlwriter shared extension for php
php72-xsl-7.2.4                The xsl shared extension for php
php72-zip-7.2.4                The zip shared extension for php
php72-zlib-7.2.4               The zlib shared extension for php
portmaster-3.19_7              Manage your ports without external databases or languages
postgresql10-client-10.3       PostgreSQL database (client)
postgresql10-server-10.3       PostgreSQL is the most advanced open-source database available anywhere
sudo-1.8.22                    Allow others to run commands as root

PostgreSQL Database

Now we have to configure the PostgreSQL database. First lets start with FreeBSD login class.

root@nextcloud:/ # cat >> /etc/login.conf << __EOF
postgres:\
        :lang=en_US.UTF-8:\
        :setenv=LC_COLLATE=C:\
        :tc=default:
__EOF

root@nextcloud:/ # grep -A 4 postgres /etc/login.conf
postgres:\
        :lang=en_US.UTF-8:\
        :setenv=LC_COLLATE=C:\
        :tc=default:

root@nextcloud:/ # cap_mkdb /etc/login.conf

Lets make sure that PostgreSQL data directory belongs to the postgres user.

root@nextcloud:/ # chown postgres:postgres /var/db/postgres/data

Lets enable the PostgreSQL service in the /etc/rc.conf file. It will look like that for now.

root@nextcloud:/ # cat /etc/rc.conf
# DAEMONS | yes
  syslogd_flags="-s -s"
  sshd_enable=YES
  postgresql_enable=YES
  postgresql_class=postgres
  postgresql_data=/var/db/postgres/data
# php_fpm_enable=YES
# memcached_enable=YES
# memcached_flags="-l 192.168.43.100"
# nginx_enable=YES

# DAEMONS | no
  sendmail_enable=NONE
  sendmail_submit_enable=NO
  sendmail_outbound_enable=NO
  sendmail_msp_queue_enable=NO

# OTHER
  clear_tmp_enable=YES
  clear_tmp_X=YES
  extra_netfs_types=NFS
  dumpdev=NO
  update_motd=NO
  keyrate=fast

Now we may initialize and start the database.

root@nextcloud:/ # /usr/local/etc/rc.d/postgresql initdb
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locales
  COLLATE:  C
  CTYPE:    en_US.UTF-8
  MESSAGES: en_US.UTF-8
  MONETARY: en_US.UTF-8
  NUMERIC:  en_US.UTF-8
  TIME:     en_US.UTF-8
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/db/postgres/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.

Success. You can now start the database server using:

    /usr/local/bin/pg_ctl -D /var/db/postgres/data -l logfile start

Now lets start it.

root@nextcloud:/ # /usr/local/etc/rc.d/postgresql start
2018-04-03 13:41:49.289 CEST [14522] LOG:  could not create IPv6 socket for address "::1": Protocol not supported
2018-04-03 13:41:49.291 CEST [14522] LOG:  listening on IPv4 address "127.0.0.1", port 5432
2018-04-03 13:41:49.297 CEST [14522] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"
2018-04-03 13:41:49.328 CEST [14522] LOG:  ending log output to stderr
2018-04-03 13:41:49.328 CEST [14522] HINT:  Future log output will go to log destination "syslog".

root@nextcloud:/ # sockstat -l4
USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS
postgres postgres   14522 4  tcp4   192.168.43.100:5432   *:*
root     sshd       14178 3  tcp4   192.168.43.100:22     *:*

Ok, its working and listening for connections.

Next we will have to connect to create user and database for the Nextcloud server.

root@nextcloud:/ # psql -h nextcloud.local -U postgres
psql: FATAL:  no pg_hba.conf entry for host "192.168.43.100", user "postgres", database "postgres", SSL off

Remmeber the rule about localhost in a Jail? There is no localhost in a Jail. We have to add 192.168.43.100/32 address to the /var/db/postgres/data/pg_hba.conf file as shown below.

root@nextcloud:/ # grep -C 1 192.168.43.100 /var/db/postgres/data/pg_hba.conf
# IPv4 local connections:
host    all             all             192.168.43.100/32       trust
host    all             all             127.0.0.1/32            trust

Now lets restart the PostgreSQL database.

root@nextcloud:/ # /usr/local/etc/rc.d/postgresql restart
2018-04-03 13:44:01.264 CEST [14692] LOG:  could not create IPv6 socket for address "::1": Protocol not supported
2018-04-03 13:44:01.266 CEST [14692] LOG:  listening on IPv4 address "127.0.0.1", port 5432
2018-04-03 13:44:01.271 CEST [14692] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"
2018-04-03 13:44:01.296 CEST [14692] LOG:  ending log output to stderr
2018-04-03 13:44:01.296 CEST [14692] HINT:  Future log output will go to log destination "syslog".

Now we can connect and create needed user and database for Nextcloud 13 installation.

root@nextcloud:/ # psql -h nextcloud.local -U postgres
psql (10.3)
Type "help" for help.

postgres=# CREATE USER nextcloud WITH PASSWORD '{NEXTCLOUD_DB_PASSWORD}';
CREATE ROLE
postgres=# CREATE DATABASE nextcloud TEMPLATE template0 ENCODING 'UNICODE';
CREATE DATABASE
postgres=# ALTER DATABASE nextcloud OWNER TO nextcloud;
ALTER DATABASE
postgres-# \q
root@nextcloud:/ #

I will also create ‘daily maintenance’ script for PostgreSQL database.

root@nextcloud:/ # cat >> /var/db/postgres/data/vacuum.sh  /dev/null
/usr/local/bin/reindexdb -a 1> /dev/null 2> /dev/null
/usr/local/bin/reindexdb -s 1> /dev/null 2> /dev/null
__EOF

root@nextcloud:/ # chmod +x /var/db/postgres/data/vacuum.sh
root@nextcloud:/ # chown postgres:postgres /var/db/postgres/data/vacuum.sh

Lets add it as a cron job on the postgres user.

root@nextcloud:/ # su - postgres -c 'crontab -e'
/tmp/crontab.ruG73E5ivZ: 1 lines, 42 characters.
crontab: installing new crontab

root@nextcloud:/ # su - postgres -c 'crontab -l'
0 0 * * * /var/db/postgres/data/vacuum.sh

Nginx Webserver

Now we have to configure Nginx, lets start by creating the self signed certificate. If You do not want to see warnings that this certificate is not signed then You may want to use service such as letsencrypt.org for example.

root@nextcloud:/ # mkdir -p /usr/local/etc/nginx/ssl
root@nextcloud:/ # cd /usr/local/etc/nginx/ssl
root@nextcloud:/usr/local/etc/nginx/ssl # openssl req -x509 -nodes -days 3650 -newkey rsa:4096 -keyout nginx.key -out nginx.crt
Enter pass phrase for server.key: {NEXTCLOUD_SERVER_PASSWORD}
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:PL
State or Province Name (full name) [Some-State]:lodzkie
Locality Name (eg, city) []:Lodz
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Vermaden Enterprises Ltd.
Organizational Unit Name (eg, section) []:Nextcloud Departament
Common Name (e.g. server FQDN or YOUR name) []:nextcloud.local
Email Address []:vermaden@nextcloud.com

root@nextcloud:/ # chmod 400 /usr/local/etc/nginx/ssl/nginx.key
root@nextcloud:/ # ls -l /usr/local/etc/nginx/ssl
total 14
-rw-r--r--  1 root  wheel  2220 Apr  3 14:43 nginx.crt
-rw-------  1 root  wheel  3272 Apr  3 14:43 nginx.key

Lets tak care of rights for Nginx log files.

root@nextcloud:/ # ls -l /var/log | grep nginx
drwxr-xr-x  2 root  wheel        2 Apr  3 01:10 nginx

root@nextcloud:/ # chown -R www:www /var/log/nginx

root@nextcloud:/ # ls -l /var/log/ | grep nginx
drwxr-xr-x  2 www   www          2 Apr  3 01:10 nginx

… and last but not least, the Nginx main configuration file.

root@nextcloud:/ # cat /usr/local/etc/nginx/nginx.conf
user                 www;
worker_processes     4;
worker_rlimit_nofile 51200;
error_log            /var/log/nginx/error.log;

events {
  worker_connections 1024;
}

http {
  include           mime.types;
  default_type      application/octet-stream;
  log_format        main  '$remote_addr - $remote_user [$time_local] "$request" ';
  access_log        /var/log/nginx/access.log main;
  sendfile          on;
  keepalive_timeout 65;

  upstream php-handler {
    server unix:/var/run/php-fpm.sock;
  }

  server {
    # ENFORCE HTTPS
    listen      80;
    server_name nextcloud.local;
    return      301 https://$server_name$request_uri;
  }

  server {
    listen              443 ssl http2;
    server_name         nextcloud.local;
    ssl_certificate     /usr/local/etc/nginx/ssl/nginx.crt;
    ssl_certificate_key /usr/local/etc/nginx/ssl/nginx.key;

    # HEADERS SECURITY RELATED
    add_header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload;";

    # HEADERS
    add_header X-Content-Type-Options nosniff;
    add_header X-XSS-Protection "1; mode=block";
    add_header X-Robots-Tag none;
    add_header X-Download-Options noopen;
    add_header X-Permitted-Cross-Domain-Policies none;

    # PATH TO THE ROOT OF YOUR INSTALLATION
    root /usr/local/www/nextcloud/;

    location = /robots.txt {
      allow all;
      log_not_found off;
      access_log off;
    }

    location = /.well-known/carddav {
      return 301 $scheme://$host/remote.php/dav;
    }

    location = /.well-known/caldav {
      return 301 $scheme://$host/remote.php/dav;
    }

    # BUFFERS TIMEOUTS UPLOAD SIZES
    client_max_body_size    16400M;
    client_body_buffer_size 1048576k;
    send_timeout            3000;

    # ENABLE GZIP BUT DO NOT REMOVE ETag HEADERS
    gzip on;
    gzip_vary on;
    gzip_comp_level 4;
    gzip_min_length 256;
    gzip_proxied expired no-cache no-store private no_last_modified no_etag auth;
    gzip_types application/atom+xml application/javascript application/json application/ld+json application/manifest+json application/rss+xml application/vnd.geo+json application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/bmp image/svg+xml image/x-icon text/cache-manifest text/css text/plain text/vcard text/vnd.rim.location.xloc text/vtt text/x-component text/x-cross-domain-policy;

    location / {
      rewrite ^ /index.php$uri;
    }

    location ~ ^/(?:build|tests|config|lib|3rdparty|templates|data)/ {
      deny all;
    }

    location ~ ^/(?:\.|autotest|occ|issue|indie|db_|console) {
      deny all;
    }

    location ~ ^/(?:index|remote|public|cron|core/ajax/update|status|ocs/v[12]|updater/.+|ocs-provider/.+)\.php(?:$|/) {
      fastcgi_split_path_info ^(.+\.php)(/.*)$;
      include fastcgi_params;
      fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
      fastcgi_param PATH_INFO $fastcgi_path_info;
      fastcgi_param HTTPS on;
      fastcgi_param modHeadersAvailable true;
      fastcgi_param front_controller_active true;
      fastcgi_pass php-handler;
      fastcgi_intercept_errors on;
      fastcgi_request_buffering off;
      fastcgi_keep_conn       off;
      fastcgi_buffers         16 256K;
      fastcgi_buffer_size     256k;
      fastcgi_busy_buffers_size 256k;
      fastcgi_temp_file_write_size 256k;
      fastcgi_send_timeout    3000s;
      fastcgi_read_timeout    3000s;
      fastcgi_connect_timeout 3000s;
    }

    location ~ ^/(?:updater|ocs-provider)(?:$|/) {
      try_files $uri/ =404;
      index index.php;
    }

    # ADDING THE CACHE CONTROL HEADER FOR JS AND CSS FILES
    # MAKE SURE IT IS BELOW PHP BLOCK
    location ~ \.(?:css|js|woff|svg|gif)$ {
      try_files $uri /index.php$uri$is_args$args;
      add_header Cache-Control "public, max-age=15778463";
      # HEADERS SECURITY RELATED
      # IT IS INTENDED TO HAVE THOSE DUPLICATED TO ONES ABOVE
      add_header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload;";
      # HEADERS
      add_header X-Content-Type-Options nosniff;
      add_header X-XSS-Protection "1; mode=block";
      add_header X-Robots-Tag none;
      add_header X-Download-Options noopen;
      add_header X-Permitted-Cross-Domain-Policies none;
      # OPTIONAL: DONT LOG ACCESS TO ASSETS
      access_log off;
    }

    location ~ \.(?:png|html|ttf|ico|jpg|jpeg)$ {
      try_files $uri /index.php$uri$is_args$args;
      # OPTIONAL: DONT LOG ACCESS TO OTHER ASSETS
      access_log off;
    }
  }
}

If at any point later You would get following error in the browser then there is a problem between Nginx and PHP (php-fpm) configuration.

502 Bad Gateway
---------------
nginx/1.12.2

PHP

Now we have to configure PHP for our needs. First PostgreSQL related settings.

root@nextcloud:/ # cat /usr/local/etc/php/ext-20-pgsql.ini
extension=pgsql.so

root@nextcloud:/ # cat >> /usr/local/etc/php/ext-20-pgsql.ini << __EOF

[PostgresSQL]
pgsql.allow_persistent = On
pgsql.auto_reset_persistent = Off
pgsql.max_persistent = -1
pgsql.max_links = -1
pgsql.ignore_notice = 0
pgsql.log_notice = 0
__EOF

root@nextcloud:/ # cat /usr/local/etc/php/ext-20-pgsql.ini
extension=pgsql.so

[PostgresSQL]
pgsql.allow_persistent = On
pgsql.auto_reset_persistent = Off
pgsql.max_persistent = -1
pgsql.max_links = -1
pgsql.ignore_notice = 0
pgsql.log_notice = 0
root@nextcloud:/ # cat /usr/local/etc/php/ext-30-pdo_pgsql.ini
extension=pdo_pgsql.so


root@nextcloud:/ # cat >> /usr/local/etc/php/ext-30-pdo_pgsql.ini << __EOF

[PostgresSQL]
pgsql.allow_persistent = On
pgsql.auto_reset_persistent = Off
pgsql.max_persistent = -1
pgsql.max_links = -1
pgsql.ignore_notice = 0
pgsql.log_notice = 0
__EOF

root@nextcloud:/ # cat /usr/local/etc/php/ext-30-pdo_pgsql.ini
extension=pdo_pgsql.so

[PostgresSQL]
pgsql.allow_persistent = On
pgsql.auto_reset_persistent = Off
pgsql.max_persistent = -1
pgsql.max_links = -1
pgsql.ignore_notice = 0
pgsql.log_notice = 0

Lets make sure php-fpm log file exists and has right owner.

root@nextcloud:/ # :> /var/log/php-fpm.log
root@nextcloud:/ # chown www:www /var/log/php-fpm.log

No modifications needed to the /usr/local/etc/php-fpm.conf file.

root@nextcloud:/ # grep '^[^;]' /usr/local/etc/php-fpm.conf
[global]
pid = run/php-fpm.pid
include=/usr/local/etc/php-fpm.d/*.conf

Lets create www profile for php-fpm daemon.

root@nextcloud:/ # cat /usr/local/etc/php-fpm.d/www.conf
[www]
user = www
group = www
listen = /var/run/php-fpm.sock
listen.backlog = -1
listen.owner = www
listen.group = www
listen.mode=0660
pm = static
pm.max_children = 4
pm.start_servers = 2
pm.min_spare_servers = 2
pm.max_spare_servers = 4
pm.process_idle_timeout = 1000s;
pm.max_requests = 500
request_terminate_timeout = 0
rlimit_files = 51200
env[HOSTNAME] = $HOSTNAME
env[PATH] = /usr/local/bin:/usr/bin:/bin
env[TMP] = /tmp
env[TMPDIR] = /tmp
env[TEMP] = /tmp

… and the main PHP /usr/local/etc/php.ini configuration file.

root@nextcloud:/ # cat /usr/local/etc/php.ini
[PHP]
max_input_time=3600
engine = On
short_open_tag = On
precision = 14
output_buffering = OFF
zlib.output_compression = Off
implicit_flush = Off
unserialize_callback_func =
serialize_precision = 17
disable_functions =
disable_classes =
zend.enable_gc = On
expose_php = On
max_execution_time = 3600
max_input_time = 30000
memory_limit = 1024M
error_reporting = E_ALL & ~E_DEPRECATED & ~E_STRICT
display_errors = Off
display_startup_errors = Off
log_errors = On
log_errors_max_len = 1024
ignore_repeated_errors = Off
ignore_repeated_source = Off
report_memleaks = On
track_errors = Off
html_errors = On
error_log = /var/log/php.log
variables_order = "GPCS"
request_order = "GP"
register_argc_argv = Off
auto_globals_jit = On
post_max_size = 16400M
auto_prepend_file =
auto_append_file =
default_mimetype = "text/html"
default_charset = "UTF-8"
doc_root =
user_dir =
enable_dl = Off
file_uploads = On
upload_max_filesize = 16400M
max_file_uploads = 64
allow_url_fopen = On
allow_url_include = Off
default_socket_timeout = 300
[CLI Server]
cli_server.color = On
[Date]
date.timezone = Europe/Warsaw
[filter]
[iconv]
[intl]
[sqlite3]
[Pcre]
[Pdo]
[Pdo_mysql]
pdo_mysql.cache_size = 2000
pdo_mysql.default_socket=
[Phar]
[mail function]
SMTP = localhost
smtp_port = 25
mail.add_x_header = On
[SQL]
sql.safe_mode = Off
[ODBC]
odbc.allow_persistent = On
odbc.check_persistent = On
odbc.max_persistent = -1
odbc.max_links = -1
odbc.defaultlrl = 4096
odbc.defaultbinmode = 1
[Interbase]
ibase.allow_persistent = 1
ibase.max_persistent = -1
ibase.max_links = -1
ibase.timestampformat = "%Y-%m-%d %H:%M:%S"
ibase.dateformat = "%Y-%m-%d"
ibase.timeformat = "%H:%M:%S"
[MySQLi]
mysqli.max_persistent = -1
mysqli.allow_persistent = On
mysqli.max_links = -1
mysqli.cache_size = 2000
mysqli.default_port = 3306
mysqli.default_socket =
mysqli.default_host =
mysqli.default_user =
mysqli.default_pw =
mysqli.reconnect = Off
[mysqlnd]
mysqlnd.collect_statistics = On
mysqlnd.collect_memory_statistics = Off
[OCI8]
[PostgreSQL]
pgsql.allow_persistent = On
pgsql.auto_reset_persistent = Off
pgsql.max_persistent = -1
pgsql.max_links = -1
pgsql.ignore_notice = 0
pgsql.log_notice = 0
[bcmath]
bcmath.scale = 0
[browscap]
[Session]
session.save_handler = files
session.save_path = "/tmp"
session.use_strict_mode = 0
session.use_cookies = 1
session.use_only_cookies = 1
session.name = PHPSESSID
session.auto_start = 0
session.cookie_lifetime = 0
session.cookie_path = /
session.cookie_domain =
session.cookie_httponly =
session.serialize_handler = php
session.gc_probability = 1
session.gc_divisor = 1000
session.gc_maxlifetime = 1440
session.referer_check =
session.cache_limiter = nocache
session.cache_expire = 180
session.use_trans_sid = 0
session.hash_function = 0
session.hash_bits_per_character = 5
url_rewriter.tags = "a=href,area=href,frame=src,input=src,form=fakeentry"
[Assertion]
zend.assertions = -1
[COM]
[mbstring]
[gd]
[exif]
[Tidy]
tidy.clean_output = Off
[soap]
soap.wsdl_cache_enabled=1
soap.wsdl_cache_dir="/tmp"
soap.wsdl_cache_ttl=86400
soap.wsdl_cache_limit = 5
[sysvshm]
[ldap]
ldap.max_links = -1
[mcrypt]
[dba]
[opcache]
opcache.enable=1
opcache.enable_cli=1
opcache.interned_strings_buffer=8
opcache.max_accelerated_files=10000
opcache.memory_consumption=128
opcache.save_comments=1
opcache.revalidate_freq=1
[curl]
[openssl]

Daemons

Now, we should enable all daemons and start them, here is the final /etc/rc.conf file.

root@nextcloud:/ # cat /etc/rc.conf
# DAEMONS | yes
  syslogd_flags="-s -s"
  sshd_enable=YES
  postgresql_enable=YES
  postgresql_class=postgres
  postgresql_data=/var/db/postgres/data
  php_fpm_enable=YES 
  memcached_enable=YES
  memcached_flags="-l 192.168.43.100"
  nginx_enable=YES

# DAEMONS | no
  sendmail_enable=NONE
  sendmail_submit_enable=NO
  sendmail_outbound_enable=NO
  sendmail_msp_queue_enable=NO

# OTHER
  clear_tmp_enable=YES
  clear_tmp_X=YES
  extra_netfs_types=NFS
  dumpdev=NO
  update_motd=NO
  keyrate=fast

Lets start the services then.

Memcached.

root@nextcloud:/ # /usr/local/etc/rc.d/memcached start
Starting memcached.

The PHP php-fpm daemon.

root@nextcloud:/ # /usr/local/etc/rc.d/php-fpm start
Performing sanity check on php-fpm configuration:
[03-Apr-2018 14:28:22] NOTICE: configuration file /usr/local/etc/php-fpm.conf test is successful

Starting php_fpm.

PostgreSQL database sohuld be running already.

root@nextcloud:/ # /usr/local/etc/rc.d/postgresql status
pg_ctl: server is running (PID: 17751)
/usr/local/bin/postgres "-D" "/var/db/postgres/data"

… and the Nginx webserver.

root@nextcloud:/ # /usr/local/etc/rc.d/nginx start
Performing sanity check on nginx configuration:
nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok
nginx: configuration file /usr/local/etc/nginx/nginx.conf test is successful
Starting nginx.

Lets see what daemon is listening on what port.

root@nextcloud:/ # sockstat -l4
USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS
www      nginx      28583 6  tcp4   192.168.43.100:80     *:*
www      nginx      28583 7  tcp4   192.168.43.100:443    *:*
www      nginx      28582 6  tcp4   192.168.43.100:80     *:*
www      nginx      28582 7  tcp4   192.168.43.100:443    *:*
www      nginx      28581 6  tcp4   192.168.43.100:80     *:*
www      nginx      28581 7  tcp4   192.168.43.100:443    *:*
www      nginx      28580 6  tcp4   192.168.43.100:80     *:*
www      nginx      28580 7  tcp4   192.168.43.100:443    *:*
root     nginx      28579 6  tcp4   192.168.43.100:80     *:*
root     nginx      28579 7  tcp4   192.168.43.100:443    *:*
nobody   memcached  28239 16 tcp4   192.168.43.100:11211  *:*
postgres postgres   28211 4  tcp4   192.168.43.100:5432   *:*
root     sshd       28205 3  tcp4   192.168.43.100:22     *:*

Remember that php-fpm daemon uses /var/run/php-fpm.sock socket.

root@nextcloud:/ # ls -l /var/run/php-fpm.sock
srw-rw----  1 www  www  0 Apr  3 18:27 /var/run/php-fpm.sock

Nextcloud

Now lets prepare the directory for Nextcloud data.

root@nextcloud:/ # mkdir -p /var/db/nextcloud/data
root@nextcloud:/ # chown -R www:www /var/db/nextcloud

We also need to make sure that whole Nextcloud installation directory is owned by www user.

root@nextcloud:/ # chown -R www:www /usr/local/www/nextcloud

Now we should be able to access the Nextcloud 13 using a browser, lets type the https://nextcloud.local/ on the host in the browser of your choice.

nextcloud-00

Viola! Its alive.

Here are last configuration bits etered directly in the browser.

   ADMIN USER: admin
   ADMIN PASS: {NEXTCLOUD_ADMIN_PASSWORD}
  DATA FOLDER: /var/db/nextcloud/data
DATABASE USER: nextcloud
DATABASE PASS: {NEXTCLOUD_DB_PASSWORD} 
DATABASE NAME: nextcloud
DATABASE HOST: nextcloud.local

Here is how it looks in the browser.

nextcloud-01-setup

After we click the Finish setup button we should see the Nextcloud welcome message as shown below.

nextcloud-02-welcome

We may close this message and we will see our files.

nextcloud-03-files

The Nextcloud settings page yelds about lack of cache daemon.

nextcloud-04-nocache

Lets configure our memcached daemon into the Nextcloud configuration file.

nextcloud-05-cache

Here are added lines to the /usr/local/www/nextcloud/config/config.php file.

root@nextcloud:/usr/local/www/nextcloud/config # diff -u config.php.ORG config.php
--- config.php.ORG      2018-04-03 16:39:04.531258000 +0200
+++ config.php  2018-04-03 16:40:01.509956000 +0200
@@ -18,4 +18,14 @@
   'dbuser' => 'nextcloud',
   'dbpassword' => '',
   'installed' => true,
+  'memcache.local' => '\\OC\\Memcache\\Memcached',
+  'memcache.distributed' => '\\OC\\Memcache\\Memcached',
+  'memcached_servers' =>
+  array (
+    0 =>
+    array (
+      0 => 'nextcloud.local',
+      1 => 11211,
+    ),
+  ),
 );

Here is complete Nextcloud 13 main configuration file /usr/local/www/nextcloud/config/config.php after changes.

root@nextcloud:/ # cat /usr/local/www/nextcloud/config/config.php
 'oc70jc009i5e',
  'passwordsalt' => 'anVkM4F5kJwhInurq0N6eq65JmL3xZ',
  'secret' => '2RjnOfiMfrdW6rJEcpxORL39+E1gvS38+sys+G0uI6vZOOSc',
  'trusted_domains' =>
  array (
    0 => 'nextcloud.local',
  ),
  'datadirectory' => '/var/db/nextcloud/data',
  'overwrite.cli.url' => 'https://nextcloud.local',
  'dbtype' => 'pgsql',
  'version' => '13.0.0.14',
  'dbname' => 'nextcloud',
  'dbhost' => 'nextcloud.local',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'dbuser' => 'nextcloud',
  'dbpassword' => '',
  'installed' => true,
  'memcache.local' => '\\OC\\Memcache\\Memcached',
  'memcache.distributed' => '\\OC\\Memcache\\Memcached',
  'memcached_servers' =>
  array (
    0 =>
    array (
      0 => 'nextcloud.local',
      1 => 11211,
    ),
  )
);

Alternatively You may want to get that config directly from the Nextcloud application using occ command.

root@nextcloud:/ # sudo -u www php /usr/local/www/nextcloud/occ config:list system
{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "nextcloud.local"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "overwrite.cli.url": "https:\/\/nextcloud.local",
        "dbtype": "pgsql",
        "version": "13.0.0.14",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "memcache.local": "\\OC\\Memcache\\Memcached",
        "memcache.distributed": "\\OC\\Memcache\\Memcached",
        "memcached_servers": [
            [
                "nextcloud.local",
                11211
            ]
        ]
    }
}

Logs

To not end with filled /var/log directory with tons of logs we need to configure their rotation.

Here are lines add to the /etc/newsyslog.conf file.

root@nextcloud:/ # cat >> /etc/newsyslog.conf << __EOF
/var/db/nextcloud/data/nextcloud.log     www:www     640  7     *    @T00  JC
/var/log/php-fpm.log                     www:www     640  7     *    @T00  JC
/var/log/nginx/error.log                 www:www     640  7     *    @T00  JC
/var/log/nginx/access.log                www:www     640  7     *    @T00  JC
__EOF

Lets verify the rotation.

root@nextcloud:/ # newsyslog -v | tail -4
/var/db/nextcloud/data/nextcloud.log : --> will trim at Fri Jul 21 00:00:00 2017
/var/log/php-fpm.log : --> will trim at Fri Jul 21 00:00:00 2017
/var/log/nginx/error.log : --> will trim at Fri Jul 21 00:00:00 2017
/var/log/nginx/access.log : --> will trim at Fri Jul 21 00:00:00 2017

Yep. Works like a charm.

Cleanup (Optional)

We may now remove not needed parts of the Jail, for example downloaded packages and distfiles.

root@nextcloud:/ # rm -rf /var/cache/pkg

root@nextcloud:/ # rm -rf /usr/ports/obj

root@nextcloud:/ # rm -rf usr/ports/distfiles

root@nextcloud:/ # pkg autoremove
Checking integrity... done (0 conflicting)
Deinstallation has been requested for the following 8 packages:

Installed packages to be REMOVED:
        autoconf-2.69_1
        autoconf-wrapper-20131203
        gettext-tools-0.19.8.1
        gmake-4.2.1_2
        help2man-1.47.6
        m4-1.4.18,1
        p5-Locale-gettext-1.07
        texinfo-6.5,1

Number of packages to be removed: 8

The operation will free 26 MiB.

Proceed with deinstalling packages? [y/N]: y

You have reached the end, good luck with Your Nextcloud setup πŸ˜‰

UPDATE 1 – SysV IPC in Jails

Since FreeBSD 11.0-RELEASE and FreeBSD 10.4-RELEASE the allow.sysvipc Jail parameter in /etc/jail.conf has been deprecated in favor of sysvmsg/sysvsem/sysvshm parameters. This information is available in man 8 jail manual page for example.

host # man 8 jail
(...)
             allow.sysvipc
                     A process within the jail has access to System V IPC
                     primitives.  This is deprecated in favor of the per-
                     module parameters (see below).  When this parameter is
                     set, it is equivalent to setting sysvmsg, sysvsem, and
                     sysvshm all to β€œinherit”.

(...)

Before this change there was problem with SysV IPC calls because each PostgreSQL server user postgres need to have different UID for each Jail running on that host. Now You can have 100 Jails with each postgres user UID 500 and everything works like a charm.

Its described broadly in this blog post – Postgres in FreeBSD Jailshttps://planet.freebsd.org/brd/2017/11/07/postgres-in-freebsd-jails/.

Using newer approach these are the changes in the configuration in the /etc/jail.conf file.

host # diff -u /etc/jail.conf.OLD /etc/jail.conf
--- /etc/jail.conf.OLD  2018-04-05 13:04:18.556904000 +0200
+++ /etc/jail.conf      2018-04-05 13:04:10.828127000 +0200
@@ -8,5 +8,7 @@
   exec.clean;
   mount.devfs;
   allow.raw_sockets;
-  allow.sysvipc;
+  sysvsem = new;
+  sysvshm = new;
+  sysvmsg = new;
 }

… and the whole /etc/jail.conf file after modification.

host # cat /etc/jail.conf
nextcloud {
  host.hostname = nextcloud.local;
  ip4.addr = 192.168.43.100;
  interface = wlan0;
  path = /jail/nextcloud;
  exec.start = "/bin/sh /etc/rc";
  exec.stop = "/bin/sh /etc/rc.shutdown";
  exec.clean;
  mount.devfs;
  allow.raw_sockets;
  sysvsem = new;
  sysvshm = new;
  sysvmsg = new;
}

UPDATE 2 – Setup without Sockets

To run this setup without sockets you may want to modify the PHP php-fpm daemon to listen in IPv4 address instead of using /var/run/php-fpm.sock socket for communication. These are the changes in /usr/local/etc/php-fpm.d/www.conf php-rpm profile and Nginx webserver main configuration /usr/local/etc/nginx/nginx.conf file.

root@nextcloud:/ # diff -u /usr/local/etc/php-fpm.d/www.conf.SOCKET /usr/local/etc/php-fpm.d/www.conf
--- /usr/local/etc/php-fpm.d/www.conf.SOCKET    2018-04-05 12:46:06.351550000 +0200
+++ /usr/local/etc/php-fpm.d/www.conf   2018-04-05 12:46:09.324277000 +0200
@@ -1,7 +1,8 @@
 [www]
 user = www
 group = www
-listen = /var/run/php-fpm.sock
+listen = 192.168.43.100:9000
+listen.allowed_clients = 192.168.43.100
 listen.backlog = -1
 listen.owner = www
 listen.group = www
root@nextcloud:/ # diff -u /usr/local/etc/nginx/nginx.conf.SOCKET /usr/local/etc/nginx/nginx.conf
--- /usr/local/etc/nginx/nginx.conf.SOCKET      2018-04-05 12:42:09.051583000 +0200
+++ /usr/local/etc/nginx/nginx.conf     2018-04-05 12:42:30.491819000 +0200
@@ -16,7 +16,7 @@
   keepalive_timeout 65;
 
   upstream php-handler {
-    server unix:/var/run/php-fpm.sock;
+    server 192.168.43.100:9000;
   }
 
   server {

UPDATE 3 – Nextcloud 13.0.1 Update

After update to latest Nextcloud 13.0.1 the setup is broken. The symptom is that after the login page it keeps redirecting. The fix has been described in the /usr/ports/UPDATING file, here is the message with the fix itself.

20180404:
  AFFECTS: users of www/nextcloud
  AUTHOR: brnrd@FreeBSD.org

  With the 13.0.1 update the path for Apps bundled with the package has
  changed from "apps" to "apps-pkg". You must add an entry to the
  "apps_paths" array in config/config.php of your nextcloud installation,
  a patch for the default installation can be applied with:

  # cd /usr/local/www/nextcloud
  # su -m www -c "php ./occ config:import < /usr/local/share/nextcloud/fix-apps_paths.json"

So to fix the installation after eventual upgrade to 13.0.1 these instructions need to be executed in the Jail.

root@nextcloud:/ # su -m www -c "php /usr/local/www/nextcloud/occ config:import < /usr/local/share/nextcloud/fix-apps_paths.json"

Hope that helps to resolve the issue.

UPDATE 4

The Nextcloud 13 on FreeBSD article was featured in the BSD Now 245 – ZFS User Conf 2018 episode.

Thanks for mentioning!

EOF