Meet Minio.
Free and open source distributed object storage server compatible with Amazon S3 v2/v4 API. Offers data protection against hardware failures using erasure code and bitrot detection. Supports highly available distributed setup. Provides confidentiality, integrity and authenticity assurances for encrypted data with negligible performance overhead. Both server side and client side encryption are supported. Below is the image of example Minio setup.
The Minio identifies itself as the ZFS of Cloud Object Storage. This guide will show You how to setup highly available distributed Minio storage on the FreeBSD operating system with ZFS as backend for Minio data. For convenience we will use FreeBSD Jails operating system level virtualization.
Setup
The setup will assume that You have 3 datacenters and assumption that you have two datacenters in whose the most of the data must reside and that the third datacenter is used as a ‘quorum/witness’ role. Distributed Minio supports up to 16 nodes/drives total, so we may juggle with that number to balance data between desired datacenters. As we have 16 drives to allocate resources on 3 sites we will use 7 + 7 + 2 approach here. The datacenters where most of the data must reside have 7/16 ratio while the ‘quorum/witness’ datacenter have only 2/16 ratio. Thanks to built in Minio redundancy we may loose (turn off for example) any one of those machines and our object storage will still be available and ready to use for any purpose.
Jails
First we will create 3 jails for our proof of concept Minio setup, storage1 will have the ‘quorum/witness’ role while storage2 and storage3 will have the ‘data’ role. To distinguish commands I type on the host system and storageX Jail I use two different prompts, this way it should be obvious what command to execute and where.
Command on the host system.
host # command
Command on the storageX Jail.
root@storageX:/ # command
First we will create the base Jails for our setup.
host # mkdir -p /jail/BASE /jail/storage1 /jail/storage2 /jail/storage3
host # cd /jail/BASE
host # fetch http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/11.1-RELEASE/base.txz
host # for I in 1 2 3; do echo ${I}; tar --unlink -xpJf /jail/BASE/base.txz -C /jail/storage${I}; done
1
2
3
host #
We will now add Jails configuration the the /etc/jail.conf file.
I have used my laptop for the Jail host. This is why Jail will configured to use the wireless wlan0 interface and 192.168.43.10X addresses.
host # for I in 1 2 3
do
cat >> /etc/jail.conf << __EOF
storage${I} {
host.hostname = storage${I}.local;
ip4.addr = 192.168.43.10${I};
interface = wlan0;
path = /jail/storage${I};
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;
mount.devfs;
allow.raw_sockets;
}
__EOF
done
host #
Lets verify that /etc/jail.conf file is configured as desired.
host # cat /etc/jail.conf
storage1 {
host.hostname = storage1.local;
ip4.addr = 192.168.43.101;
interface = wlan0;
path = /jail/storage1;
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;
mount.devfs;
allow.raw_sockets;
}
storage2 {
host.hostname = storage2.local;
ip4.addr = 192.168.43.102;
interface = wlan0;
path = /jail/storage2;
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;
mount.devfs;
allow.raw_sockets;
}
storage3 {
host.hostname = storage3.local;
ip4.addr = 192.168.43.103;
interface = wlan0;
path = /jail/storage3;
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;
mount.devfs;
allow.raw_sockets;
}
host #
Now we will start our Jails.
host # for I in 1 2 3; do service jail onestart storage${I}; done
Starting jails: storage1.
Starting jails: storage2.
Starting jails: storage3.
Lets see how they work.
host # jls
JID IP Address Hostname Path
1 192.168.43.101 storage1.local /jail/storage1
2 192.168.43.102 storage2.local /jail/storage2
3 192.168.43.103 storage3.local /jail/storage3
Now lets add DNS server so they will have Internet connectivity.
host # for I in 1 2 3; do echo nameserver 1.1.1.1 > /jail/storage${I}/etc/resolv.conf; done
We can now install Minio package.
host # for I in 1 2 3; do jexec storage${I} env ASSUME_ALWAYS_YES=yes pkg install -y minio; echo; done
Bootstrapping pkg from pkg+http://pkg.FreeBSD.org/FreeBSD:11:amd64/quarterly, please wait...
Verifying signature with trusted certificate pkg.freebsd.org.2013102301... done
[storage1.local] Installing pkg-1.10.5...
[storage1.local] Extracting pkg-1.10.5: 100%
Updating FreeBSD repository catalogue...
pkg: Repository FreeBSD load error: access repo file(/var/db/pkg/repo-FreeBSD.sqlite) failed: No such file or directory
[storage1.local] Fetching meta.txz: 100% 944 B 0.9kB/s 00:01
[storage1.local] Fetching packagesite.txz: 100% 6 MiB 637.1kB/s 00:10
Processing entries: 100%
FreeBSD repository update completed. 31143 packages processed.
All repositories are up to date.
Updating database digests format: 100%
The following 1 package(s) will be affected (of 0 checked):
New packages to be INSTALLED:
minio: 2018.03.19.19.22.06
Number of packages to be installed: 1
The process will require 22 MiB more space.
6 MiB to be downloaded.
[storage1.local] [1/1] Fetching minio-2018.03.19.19.22.06.txz: 100% 6 MiB 305.6kB/s 00:19
Checking integrity... done (0 conflicting)
[storage1.local] [1/1] Installing minio-2018.03.19.19.22.06...
===> Creating groups.
Creating group 'minio' with gid '473'.
===> Creating users
Creating user 'minio' with uid '473'.
[storage1.local] [1/1] Extracting minio-2018.03.19.19.22.06: 100%
Bootstrapping pkg from pkg+http://pkg.FreeBSD.org/FreeBSD:11:amd64/quarterly, please wait...
Verifying signature with trusted certificate pkg.freebsd.org.2013102301... done
[storage2.local] Installing pkg-1.10.5...
[storage2.local] Extracting pkg-1.10.5: 100%
Updating FreeBSD repository catalogue...
pkg: Repository FreeBSD load error: access repo file(/var/db/pkg/repo-FreeBSD.sqlite) failed: No such file or directory
[storage2.local] Fetching meta.txz: 100% 944 B 0.9kB/s 00:01
[storage2.local] Fetching packagesite.txz: 100% 6 MiB 637.1kB/s 00:10
Processing entries: 100%
FreeBSD repository update completed. 31143 packages processed.
All repositories are up to date.
Updating database digests format: 100%
The following 1 package(s) will be affected (of 0 checked):
New packages to be INSTALLED:
minio: 2018.03.19.19.22.06
Number of packages to be installed: 1
The process will require 22 MiB more space.
6 MiB to be downloaded.
[storage2.local] [1/1] Fetching minio-2018.03.19.19.22.06.txz: 100% 6 MiB 305.6kB/s 00:19
Checking integrity... done (0 conflicting)
[storage2.local] [1/1] Installing minio-2018.03.19.19.22.06...
===> Creating groups.
Creating group 'minio' with gid '473'.
===> Creating users
Creating user 'minio' with uid '473'.
[storage2.local] [1/1] Extracting minio-2018.03.19.19.22.06: 100%
Bootstrapping pkg from pkg+http://pkg.FreeBSD.org/FreeBSD:11:amd64/quarterly, please wait...
Verifying signature with trusted certificate pkg.freebsd.org.2013102301... done
[storage3.local] Installing pkg-1.10.5...
[storage3.local] Extracting pkg-1.10.5: 100%
Updating FreeBSD repository catalogue...
pkg: Repository FreeBSD load error: access repo file(/var/db/pkg/repo-FreeBSD.sqlite) failed: No such file or directory
[storage3.local] Fetching meta.txz: 100% 944 B 0.9kB/s 00:01
[storage3.local] Fetching packagesite.txz: 100% 6 MiB 637.1kB/s 00:10
Processing entries: 100%
FreeBSD repository update completed. 31143 packages processed.
All repositories are up to date.
Updating database digests format: 100%
The following 1 package(s) will be affected (of 0 checked):
New packages to be INSTALLED:
minio: 2018.03.19.19.22.06
Number of packages to be installed: 1
The process will require 22 MiB more space.
6 MiB to be downloaded.
[storage3.local] [1/1] Fetching minio-2018.03.19.19.22.06.txz: 100% 6 MiB 305.6kB/s 00:19
Checking integrity... done (0 conflicting)
[storage3.local] [1/1] Installing minio-2018.03.19.19.22.06...
===> Creating groups.
Creating group 'minio' with gid '473'.
===> Creating users
Creating user 'minio' with uid '473'.
[storage3.local] [1/1] Extracting minio-2018.03.19.19.22.06: 100%
host #
Lets verify that Minio package has installed successfully.
host # for I in 1 2 3; do jexec storage${I} which minio; done
/usr/local/bin/minio
/usr/local/bin/minio
/usr/local/bin/minio
host #
Now we will configure /etc/hosts file.
root@storage1:/ # cat >> /etc/hosts << __EOF 192.168.43.101 storage1 192.168.43.102 storage2 192.168.43.103 storage3 __EOF
root@storage1:/ # cat >> /etc/hosts << __EOF 192.168.43.101 storage1 192.168.43.102 storage2 192.168.43.103 storage3 __EOF
root@storage1:/ # cat >> /etc/hosts << __EOF 192.168.43.101 storage1 192.168.43.102 storage2 192.168.43.103 storage3 __EOF
We will create directories for Minio data.
host # for DIR in 1 2 3 4 5 6 7
do
for I in 2 3
do
jexec storage${I} mkdir -p /data${DIR}
done
done
host # for DIR in 1 2
do
for I in 1
do
jexec storage${I} mkdir -p /data${DIR}
done
done
Lets verify that that our data directories created successfully.
host # for I in 1 2 3
do
echo storage${I}
jexec storage${I} ls -1 / | grep data
echo
done
storage1
data1
data2
storage2
data1
data2
data3
data4
data5
data6
data7
storage3
data1
data2
data3
data4
data5
data6
data7
Basic minio command example.
root@storage1:/ # minio NAME: minio - Cloud Storage Server. DESCRIPTION: Minio is an Amazon S3 compatible object storage server. Use it to store photos, videos, VMs, containers, log files, or any blob of data as objects. USAGE: minio [FLAGS] COMMAND [ARGS...] COMMANDS: server Start object storage server. gateway Start object storage gateway. update Check for a new software update. version Print version. FLAGS: --config-dir value, -C value Path to configuration directory. (default: "/root/.minio") --quiet Disable startup information. --json Output server logs and startup information in json format. --help, -h Show help. VERSION: 2018-03-19T19:22:06Z
Now we can generate the list of directories on servers to add as argument for Minio.
host # for DIR in 1 2
do
for I in 1
do
echo -n http://
jls | grep storage${I} | awk '{printf $3}' | sed s/.local//g
echo ":9000/data${DIR} \\"
done
done | sort -n
host # for DIR in 1 2 3 4 5 6 7
do
for I in 2 3
do
echo -n http://
jls | grep storage${I} | awk '{printf $3}' | sed s/.local//g
echo ":9000/data${DIR} \\"
done
done | sort -n
http://storage1:9000/data1 \ http://storage1:9000/data2 \ http://storage2:9000/data1 \ http://storage2:9000/data2 \ http://storage2:9000/data3 \ http://storage2:9000/data4 \ http://storage2:9000/data5 \ http://storage2:9000/data6 \ http://storage2:9000/data7 \ http://storage3:9000/data1 \ http://storage3:9000/data2 \ http://storage3:9000/data3 \ http://storage3:9000/data4 \ http://storage3:9000/data5 \ http://storage3:9000/data6 \ http://storage3:9000/data7 \
We can as well just write it down by hand of course π
host # for DIR in 1 2
do
for I in 1
do
echo -n http://
jls | grep storage${I} | awk '{printf $3}' | sed s/.local//g
echo -n ":9000/data${DIR} "
done
done | sort -n
host # for DIR in 1 2 3 4 5 6 7
do
for I in 2 3
do
echo -n http://
jls | grep storage${I} | awk '{printf $3}' | sed s/.local//g
echo -n ":9000/data${DIR} "
done
done | sort -n
This is out list of data directories that we will use to configure Minio in FreeBSD’s main configuration /etc/rc.conf file.
http://storage1:9000/data1 http://storage1:9000/data2 http://storage2:9000/data1 http://storage2:9000/data2 http://storage2:9000/data3 http://storage2:9000/data4 http://storage2:9000/data5 http://storage2:9000/data6 http://storage2:9000/data7 http://storage3:9000/data1 http://storage3:9000/data2 http://storage3:9000/data3 http://storage3:9000/data4 http://storage3:9000/data5 http://storage3:9000/data6 http://storage3:9000/data7
Now, lets put Minio settings into the /etc/rc.conf file.
root@storageX:~ # cat > /etc/rc.conf << __EOF minio_enable=YES minio_disks="http://storage1:9000/data1 http://storage1:9000/data2 http://storage2:9000/data1 http://storage2:9000/data2 http://storage2:9000/data3 http://storage2:9000/data4 http://storage2:9000/data5 http://storage2:9000/data6 http://storage2:9000/data7 http://storage3:9000/data1 http://storage3:9000/data2 http://storage3:9000/data3 http://storage3:9000/data4 http://storage3:9000/data5 http://storage3:9000/data6 http://storage3:9000/data7" __EOF root@storageX:~ #
root@storageX:~ # cat /etc/rc.conf minio_enable=YES minio_disks="http://storage1:9000/data1 http://storage1:9000/data2 http://storage2:9000/data1 http://storage2:9000/data2 http://storage2:9000/data3 http://storage2:9000/data4 http://storage2:9000/data5 http://storage2:9000/data6 http://storage2:9000/data7 http://storage3:9000/data1 http://storage3:9000/data2 http://storage3:9000/data3 http://storage3:9000/data4 http://storage3:9000/data5 http://storage3:9000/data6 http://storage3:9000/data7" root@storageX:~ #
Now we will start and configure Minio for the first time.
On each storageX server run the following set of commands.
host # jexec storage3 root@storage3:~ # root@storage3:/ # rm -rf /http:\* root@storage3:/ # rm -rf /usr/local/etc/minio root@storage3:/ # rm -rf /data?/* /data?/.minio.sys root@storage3:/ # touch /var/log/minio.log root@storage3:/ # chown minio:minio /var/log/minio.log root@storage3:/ # mkdir -p /usr/local/etc/minio root@storage3:/ # chown -R minio:minio /usr/local/etc/minio root@storage3:/ # mkdir -p /http:: root@storage3:/ # chown -R minio:minio /http:: root@storage3:/ # mkdir -p /http: root@storage3:/ # chown -R minio:minio /http: root@storage3:/ # su -m minio -c 'env \\ ? MINIO_ACCESS_KEY=alibaba \\ ? MINIO_SECRET_KEY=0P3NS3S4M3 \\ ? minio server \\ ? --config-dir /usr/local/etc/minio \\ ? http://storage1:9000/data1 \\ ? http://storage1:9000/data2 \\ ? http://storage2:9000/data1 \\ ? http://storage2:9000/data2 \\ ? http://storage2:9000/data3 \\ ? http://storage2:9000/data4 \\ ? http://storage2:9000/data5 \\ ? http://storage2:9000/data6 \\ ? http://storage2:9000/data7 \\ ? http://storage3:9000/data1 \\ ? http://storage3:9000/data2 \\ ? http://storage3:9000/data3 \\ ? http://storage3:9000/data4 \\ ? http://storage3:9000/data5 \\ ? http://storage3:9000/data6 \\ ? http://storage3:9000/data7' Created minio configuration file successfully at /usr/local/etc/minio Waiting for the first server to format the disks. Waiting for the first server to format the disks. Drive Capacity: 504 GiB Free, 515 GiB Total Status: 16 Online, 0 Offline. Endpoint: http://192.168.43.103:9000 AccessKey: alibaba SecretKey: 0P3NS3S4M3 Browser Access: http://192.168.43.103:9000 Command-line Access: https://docs.minio.io/docs/minio-client-quickstart-guide $ mc config host add myminio http://192.168.43.103:9000 alibaba 0P3NS3S4M3 Object API (Amazon S3 compatible): Go: https://docs.minio.io/docs/golang-client-quickstart-guide Java: https://docs.minio.io/docs/java-client-quickstart-guide Python: https://docs.minio.io/docs/python-client-quickstart-guide JavaScript: https://docs.minio.io/docs/javascript-client-quickstart-guide .NET: https://docs.minio.io/docs/dotnet-client-quickstart-guide
host # jexec storage2 root@storage2:~ # root@storage2:/ # rm -rf /http:\* root@storage2:/ # rm -rf /usr/local/etc/minio root@storage2:/ # rm -rf /data?/* /data?/.minio.sys root@storage2:/ # touch /var/log/minio.log root@storage2:/ # chown minio:minio /var/log/minio.log root@storage2:/ # mkdir -p /usr/local/etc/minio root@storage2:/ # chown -R minio:minio /usr/local/etc/minio root@storage2:/ # mkdir -p /http:: root@storage2:/ # chown -R minio:minio /http:: root@storage2:/ # mkdir -p /http: root@storage2:/ # chown -R minio:minio /http: root@storage2:/ # su -m minio -c 'env \\ ? MINIO_ACCESS_KEY=alibaba \\ ? MINIO_SECRET_KEY=0P3NS3S4M3 \\ ? minio server \\ ? --config-dir /usr/local/etc/minio \\ ? http://storage1:9000/data1 \\ ? http://storage1:9000/data2 \\ ? http://storage2:9000/data1 \\ ? http://storage2:9000/data2 \\ ? http://storage2:9000/data3 \\ ? http://storage2:9000/data4 \\ ? http://storage2:9000/data5 \\ ? http://storage2:9000/data6 \\ ? http://storage2:9000/data7 \\ ? http://storage3:9000/data1 \\ ? http://storage3:9000/data2 \\ ? http://storage3:9000/data3 \\ ? http://storage3:9000/data4 \\ ? http://storage3:9000/data5 \\ ? http://storage3:9000/data6 \\ ? http://storage3:9000/data7' Created minio configuration file successfully at /usr/local/etc/minio Waiting for the first server to format the disks. Waiting for the first server to format the disks. Drive Capacity: 504 GiB Free, 515 GiB Total Status: 16 Online, 0 Offline. Endpoint: http://192.168.43.102:9000 AccessKey: alibaba SecretKey: 0P3NS3S4M3 Browser Access: http://192.168.43.102:9000 Command-line Access: https://docs.minio.io/docs/minio-client-quickstart-guide $ mc config host add myminio http://192.168.43.102:9000 alibaba 0P3NS3S4M3 Object API (Amazon S3 compatible): Go: https://docs.minio.io/docs/golang-client-quickstart-guide Java: https://docs.minio.io/docs/java-client-quickstart-guide Python: https://docs.minio.io/docs/python-client-quickstart-guide JavaScript: https://docs.minio.io/docs/javascript-client-quickstart-guide .NET: https://docs.minio.io/docs/dotnet-client-quickstart-guide
host # jexec storage1 root@storage1:~ # root@storage1:/ # rm -rf /http:\* root@storage1:/ # rm -rf /usr/local/etc/minio root@storage1:/ # rm -rf /data?/* /data?/.minio.sys root@storage1:/ # touch /var/log/minio.log root@storage1:/ # chown minio:minio /var/log/minio.log root@storage1:/ # mkdir -p /usr/local/etc/minio root@storage1:/ # chown -R minio:minio /usr/local/etc/minio root@storage1:/ # mkdir -p /http:: root@storage1:/ # chown -R minio:minio /http:: root@storage1:/ # mkdir -p /http: root@storage1:/ # chown -R minio:minio /http: root@storage1:/ # su -m minio -c 'env \\ ? MINIO_ACCESS_KEY=alibaba \\ ? MINIO_SECRET_KEY=0P3NS3S4M3 \\ ? minio server \\ ? --config-dir /usr/local/etc/minio \\ ? http://storage1:9000/data1 \\ ? http://storage1:9000/data2 \\ ? http://storage2:9000/data1 \\ ? http://storage2:9000/data2 \\ ? http://storage2:9000/data3 \\ ? http://storage2:9000/data4 \\ ? http://storage2:9000/data5 \\ ? http://storage2:9000/data6 \\ ? http://storage2:9000/data7 \\ ? http://storage3:9000/data1 \\ ? http://storage3:9000/data2 \\ ? http://storage3:9000/data3 \\ ? http://storage3:9000/data4 \\ ? http://storage3:9000/data5 \\ ? http://storage3:9000/data6 \\ ? http://storage3:9000/data7' Created minio configuration file successfully at /usr/local/etc/minio Waiting for the first server to format the disks. Waiting for the first server to format the disks. Drive Capacity: 504 GiB Free, 515 GiB Total Status: 16 Online, 0 Offline. Endpoint: http://192.168.43.101:9000 AccessKey: alibaba SecretKey: 0P3NS3S4M3 Browser Access: http://192.168.43.101:9000 Command-line Access: https://docs.minio.io/docs/minio-client-quickstart-guide $ mc config host add myminio http://192.168.43.101:9000 alibaba 0P3NS3S4M3 Object API (Amazon S3 compatible): Go: https://docs.minio.io/docs/golang-client-quickstart-guide Java: https://docs.minio.io/docs/java-client-quickstart-guide Python: https://docs.minio.io/docs/python-client-quickstart-guide JavaScript: https://docs.minio.io/docs/javascript-client-quickstart-guide .NET: https://docs.minio.io/docs/dotnet-client-quickstart-guide
Here is how it looks in the xterm terminal.
We can now verify in the browser that it actually works.
Now hit [CTRL]+[C] in each of these windows to stop the Minio cluster.
We will now start Minio with FreeBSD rc(8) subsystem as a service.
root@storage1:/ # service minio start Starting minio. root@storage1:/ # cat /var/log/minio.log root@storage1:/ # service minio status minio is running as pid 50309.
Lets check if it works.
root@storage1:/ # ps -U minio PID TT STAT TIME COMMAND 50308 - IsJ 0:00.00 daemon: /usr/bin/env[50309] (daemon) 50309 - IJ 0:00.27 /usr/local/bin/minio -C /usr/local/etc/minio server (...)
Now we will do some basic operations, login into Minio distributed storage, create new bucket and upload some file to it.
This is how empty Minio cluster looks like.
Select Create Bucket option from the button below.
We will use name test for our new bucket.
It is created and we can access it.
Lets Upload File using same menu as previously.
The upload progress shown by Minio.
File has been indeed uploaded.
By clicking on it we may access it directly from the browser.
We can also share link to that file by using the File Menu as shown below.
The link creation dialog is shown below.
Lets see how Minio distributes the data – the ThinkPad Design – Spirit and Essence.pdf file in out case – over its data directories spread across the servers.
host # jexec storage1 root@storage1:/ # find /data?/test /data1/test /data1/test/ThinkPad Design - Spirit and Essence.pdf /data1/test/ThinkPad Design - Spirit and Essence.pdf/xl.json /data1/test/ThinkPad Design - Spirit and Essence.pdf/part.1 /data2/test /data2/test/ThinkPad Design - Spirit and Essence.pdf /data2/test/ThinkPad Design - Spirit and Essence.pdf/xl.json /data2/test/ThinkPad Design - Spirit and Essence.pdf/part.1 root@storage1:/ # exit
host # jexec storage2 root@storage2:/ # find /data?/test /data1/test /data1/test/ThinkPad Design - Spirit and Essence.pdf /data1/test/ThinkPad Design - Spirit and Essence.pdf/part.1 /data1/test/ThinkPad Design - Spirit and Essence.pdf/xl.json /data2/test /data2/test/ThinkPad Design - Spirit and Essence.pdf /data2/test/ThinkPad Design - Spirit and Essence.pdf/xl.json /data2/test/ThinkPad Design - Spirit and Essence.pdf/part.1 /data3/test /data3/test/ThinkPad Design - Spirit and Essence.pdf /data3/test/ThinkPad Design - Spirit and Essence.pdf/part.1 /data3/test/ThinkPad Design - Spirit and Essence.pdf/xl.json /data4/test /data4/test/ThinkPad Design - Spirit and Essence.pdf /data4/test/ThinkPad Design - Spirit and Essence.pdf/part.1 /data4/test/ThinkPad Design - Spirit and Essence.pdf/xl.json /data5/test /data5/test/ThinkPad Design - Spirit and Essence.pdf /data5/test/ThinkPad Design - Spirit and Essence.pdf/part.1 /data5/test/ThinkPad Design - Spirit and Essence.pdf/xl.json /data6/test /data6/test/ThinkPad Design - Spirit and Essence.pdf /data6/test/ThinkPad Design - Spirit and Essence.pdf/part.1 /data6/test/ThinkPad Design - Spirit and Essence.pdf/xl.json /data7/test /data7/test/ThinkPad Design - Spirit and Essence.pdf /data7/test/ThinkPad Design - Spirit and Essence.pdf/xl.json /data7/test/ThinkPad Design - Spirit and Essence.pdf/part.1 root@storage2:/ # exit
host # jexec storage3 root@storage3:/ # find /data?/test /data1/test /data1/test/ThinkPad Design - Spirit and Essence.pdf /data1/test/ThinkPad Design - Spirit and Essence.pdf/part.1 /data1/test/ThinkPad Design - Spirit and Essence.pdf/xl.json /data2/test /data2/test/ThinkPad Design - Spirit and Essence.pdf /data2/test/ThinkPad Design - Spirit and Essence.pdf/xl.json /data2/test/ThinkPad Design - Spirit and Essence.pdf/part.1 /data3/test /data3/test/ThinkPad Design - Spirit and Essence.pdf /data3/test/ThinkPad Design - Spirit and Essence.pdf/xl.json /data3/test/ThinkPad Design - Spirit and Essence.pdf/part.1 /data4/test /data4/test/ThinkPad Design - Spirit and Essence.pdf /data4/test/ThinkPad Design - Spirit and Essence.pdf/part.1 /data4/test/ThinkPad Design - Spirit and Essence.pdf/xl.json /data5/test /data5/test/ThinkPad Design - Spirit and Essence.pdf /data5/test/ThinkPad Design - Spirit and Essence.pdf/part.1 /data5/test/ThinkPad Design - Spirit and Essence.pdf/xl.json /data6/test /data6/test/ThinkPad Design - Spirit and Essence.pdf /data6/test/ThinkPad Design - Spirit and Essence.pdf/part.1 /data6/test/ThinkPad Design - Spirit and Essence.pdf/xl.json /data7/test /data7/test/ThinkPad Design - Spirit and Essence.pdf /data7/test/ThinkPad Design - Spirit and Essence.pdf/xl.json /data7/test/ThinkPad Design - Spirit and Essence.pdf/part.1 root@storage3:/ # exit
We can also see what Minio configuration file /usr/local/etc/minio/config.json has been generated.
host # jexec storage1
root@storage1:/ # cat /usr/local/etc/minio/config.json
{
"version": "22",
"credential": {
"accessKey": "alibaba",
"secretKey": "0P3NS3S4M3"
},
"region": "",
"browser": "on",
"domain": "",
"storageclass": {
"standard": "",
"rrs": ""
},
"notify": {
"amqp": {
"1": {
"enable": false,
"url": "",
"exchange": "",
"routingKey": "",
"exchangeType": "",
"deliveryMode": 0,
"mandatory": false,
"immediate": false,
"durable": false,
"internal": false,
"noWait": false,
"autoDeleted": false
}
},
"elasticsearch": {
"1": {
"enable": false,
"format": "",
"url": "",
"index": ""
}
},
"kafka": {
"1": {
"enable": false,
"brokers": null,
"topic": ""
}
},
"mqtt": {
"1": {
"enable": false,
"broker": "",
"topic": "",
"qos": 0,
"clientId": "",
"username": "",
"password": "",
"reconnectInterval": 0,
"keepAliveInterval": 0
}
},
"mysql": {
"1": {
"enable": false,
"format": "",
"dsnString": "",
"table": "",
"host": "",
"port": "",
"user": "",
"password": "",
"database": ""
}
},
"nats": {
"1": {
"enable": false,
"address": "",
"subject": "",
"username": "",
"password": "",
"token": "",
"secure": false,
"pingInterval": 0,
"streaming": {
"enable": false,
"clusterID": "",
"clientID": "",
"async": false,
"maxPubAcksInflight": 0
}
}
},
"postgresql": {
"1": {
"enable": false,
"format": "",
"connectionString": "",
"table": "",
"host": "",
"port": "",
"user": "",
"password": "",
"database": ""
}
},
"redis": {
"1": {
"enable": false,
"format": "",
"address": "",
"password": "",
"key": ""
}
},
"webhook": {
"1": {
"enable": false,
"endpoint": ""
}
}
}
S3FS
We can also mount that test bucket from out distributed Minio object storage cluster as filesystem using the S3FS project. Lets add s3fs package and mount our bucket.
host # pkg install -y fusefs-s3fs
Now we will configure password for our bucket.
host # echo test:alibaba:0P3NS3S4M3 > /root/.passwd-s3fs host # chmod 600 /root/.passwd-s3fs host # cat /root/.passwd-s3fs test:alibaba:0P3NS3S4M3
Now lets do the actual mount.
host # mkdir /tmp/test host # s3fs \ -o allow_other \ -o use_path_request_style \ -o url=http://192.168.43.101:9000 \ -o passwd_file=/root/.passwd-s3fs \ test /tmp/test
The file ThinkPad Design – Spirit and Essence.pdf that we put through web interface should be here.
host # exa -l /tmp/test .--------- 10M root 2018-04-16 14:15 ThinkPad Design - Spirit and Essence.pdf host # file /tmp/test/ThinkPad\ Design\ -\ Spirit\ and\ Essence.pdf /tmp/test/ThinkPad Design - Spirit and Essence.pdf: PDF document, version 1.4 host # stat /tmp/test/ThinkPad\ Design\ -\ Spirit\ and\ Essence.pdf 3976265496 2 ---------- 1 root wheel 0 10416953 "Jan 1 01:00:00 1970" "Apr 16 14:35:35 2018" "Jan 1 01:00:00 1970" "Jan 1 00:59:59 1970" 4096 20346 0 /tmp/test/ThinkPad Design - Spirit and Essence.pdf
We can now upload other file into that bucket using s3fs mount.
host # cp -v /home/vermaden/On\ the\ Shortness\ of\ Life\ -\ Lucius\ Seneca.pdf /tmp/test /home/vermaden/On the Shortness of Life - Lucius Seneca.pdf -> /tmp/test/On the Shortness of Life - Lucius Seneca.pdf host # file /tmp/test/On\ the\ Shortness\ of\ Life\ -\ Lucius\ Seneca.pdf On the Shortness of Life - Lucius Seneca.pdf: PDF document, version 1.4
We can also verify that our file put through s3fs is visible on the web interface.
Real Hardware
Now, as we have working Proof of Concept for the distributed Minio setup how about putting it on a real hardware for real storage purposes? I would setup a 16 node Minio distributed server on a Supermicro SSG-5018D8-AR12L hardware. Supermicro even suggests using that kind of servers for object storage, here is their white paper on that topic – Object Storage Solution for Data Archive using Supermicro SSG-5018D8-AR12L and OpenIO SDS – but they use OpenIO not Minio for distributed object storage solution.
This server features the Supermicro X10SDV-7TP4F motherboard. This is important as this motherboard officially supports FreeBSD 11.x operating system on their Supermicro OS Compatibility page.
Motherboard specification has these features.
1 x Intel Xeon D-1537 8-Core / 16-Threads TDP 35W 4 x UDIMM for up to 128GB ECC RDIMM DDR4 2133MHz 12 x 3.5" SAS2/SATA3 Hot-Swap HDD Bays 4 x 2.5" Cold-Swap HDD Bays 1 x Controller Intel SoC for 4 SATA3 (6Gbps) Ports 1 x Controller Broadcom 2116 for 16 SATA3 (6Gbps) Ports 1 x Expansion Slot PCI-E 3.0 x8 1 x Expansion Slot M.2 PCIe 3.0 x4 1 x Expansion Slot Mini-PCIe w/ mSATA Support 2 x 10G SFP+ Port 2 x 1GbE LAN Port 2 x External USB 3.0 Port 1 x Interlal USB 2.0 Port 2 x 400W High-Rfficiency Redundant Power Supplies
You can configure your own and get approximated price using the Thinkmate site from here:
https://www.thinkmate.com/system/superstorage-server-5018d8-ar12l
I would add this components to the basic setup:
4 x UDIMM FULL 128 GB ECC RDIMM DDR4 2 x 240GB Micron 5100 MAX 2.5" SATA 6.0Gb/s SSD 2 x 7.68TB Micron 5200 ECO Series 2.5" SATA 6.0Gb/s SSD 12 x 12TB SATA 6.0Gb/s 7200RPM 3.5" Hitachi Ultrastarβ’ He12 3 x SanDisk Cruzer Fit 32GB USB 3.0
Now, I will use the 3 x SanDisk Cruzer Fit 32GB USB 3.0 disks to install FreeBSD as a ZFS root/boot pool with mirror + spare on these disks. We do not need performance here.
Then, the 12 x 12TB SATA 6.0Gb/s 7200RPM 3.5″ Hitachi Ultrastarβ’ He12 drives will be used as RAIDZ (RAID5 equivalent in ZFS without the write hole) for the Minio data, wich 11 + 1 setup, which means 11 drives for data and 1 drive for parity. As we can lose HALF of the Minio servers I would not waste 12 TB drive for spare here. Then, I would use 2 x 240GB Micron 5100 MAX 2.5″ SATA 6.0Gb/s SSD in mirror for the ZFS ZIL (ZFS Intent Log) to accelerate writes and 2 x 7.68TB Micron 5200 ECO Series 2.5″ SATA 6.0Gb/s SSD for the ZFS read cache (L2ARC).
The network would be setup on 2 x 10G SFP+ Port with LACP as lagg0 interface so each server would have 20 Gbit connectivity. This will give us a total of 320 Gbit theoretical network throughput.
This setup would give as 132 TB ZFS pool space with 15 TB for read cache and 240 GB for writes for single 1U server. Making the calculations this will give as 2112 TB (more then 2 PB) of space for Minio data.
With Minio algorithm for data redundancy we will have about 1 PB of usable storage space in our 16U Object Storage FreeBSD Appliance.
Not bad for my taste π
UPDATE 1
The Distributed Object Storage with Minio on FreeBSD article was included in the BSD Now 246 – Disclosure episode.
Thanks for mentioning!
ππππππππ 
May I ask what tool do you use to create that network diagram?
LikeLike
I have taken that image from the Minio page, so You will have to ask them for that info:
https://docs.minio.io/docs/distributed-minio-quickstart-guide
LikeLike
Pingback: [How-To] Distributed Object Storage with Minio on FreeBSD - FreeBSDNews.com
Pingback: MongoDB Replica Set Cluster on Oracle Linux | vermaden
Pingback: Home | vermaden
Pingback: Properly Coordinated Disclosure | BSD Now 246 | Jupiter Broadcasting
Pingback: GlusterFS Cluster on FreeBSD with Ansible and GNU Parallel | ππππππππ
Pingback: Strategic thinking, or what IΒ think what we need to do to keep FreeBSD relevant | | Alexander Leidinger
Waoh such gorgeous setup π
And how much does it cost ?
LikeLike
I cost me only 3 jails … and some of my time π
LikeLike
Pingback: RabbitMQ Cluster on FreeBSD Containers | ππππππππ
Pingback: FreeBSD Enterprise 1 PB Storage | ππππππππ
Pingback: [How-To] RabbitMQ Cluster on FreeBSD Containers - FreeBSDNews.com