Tag Archives: network

FreeBSD Desktop – Part 12 – Configuration – Openbox

Time to cut the bullshit and actually make some real configuration. In today’s article of the FreeBSD Desktop series I will describe how to configure the Openbox window manager.

You may want to check other articles in the FreeBSD Desktop series on the FreeBSD Desktop – Global Page where you will find links to all episodes of the series along with table of contents for each episode’s contents.

Features

Comparing to earlier articles in the series it will be HUGE, sorry for that. I could cut it into smaller parts but that would require editing of the Openbox configuration, its shortcuts and menus over and over again, so for the sake of simplicity its better to put it all at once. As it is as that big there will be mistakes, but I will fix them ASAP.

Here is the list of all features that will be available in this Openbox configuration.

  • Nice looking Openbox theme.
  • Openbox Menu (static) with nice looking icons.
  • Openbox Menu for FreeBSD top(1)/ps(1) commands and config files/logs.
  • Openbox Menu for FreeBSD default sound output.
  • Openbox Menu and shortcuts for FreeBSD sound volume increase/decrease.
  • Openbox Menu for FreeBSD for CPU frequency scaling.
  • Openbox Menu for FreeBSD network management with network.sh script.
  • Openbox Menu for screenshots/wallpapers management.
  • Openbox Menu for Recent files.
  • Random wallpaper handling.
  • Random xterm(1) theme at every terminal start with lost of great themes.
  • Openbox shortcuts and script for Aero Snap like behavior.
  • Openbox Dmenu shortcuts and integration.
  • Openbox configured with nice fonts.
  • Openbox shortcuts for most important tasks.
  • Warning for low battery on laptop.
  • I probably forgot about dozen other features – let me know in comments πŸ™‚

Here is how the Openbox menus and window borders and window switching would look like.

openbox-alt-tab

openbox-menu

Β 

Here are all the files with needed configuration.

Doas

To make most scripts work Your user (vuk in the series) needs to be in the wheel, operator and network groups and doas(1) (sudo(8) equivalent) needs to be installed and configured in the following way.

# pkg install doas

# pw groupmod wheel    -m vuk
# pw groupmod operator -m vuk
# pw groupmod network  -m vuk

# cat /usr/local/etc/doas.conf
permit nopass :wheel as root

permit nopass :network as root cmd ifconfig
permit nopass :network as root cmd dhclient
permit nopass :network as root cmd umount
permit nopass :network as root cmd wpa_supplicant
permit nopass :network as root cmd ppp
permit nopass :network as root cmd killall args -9 dhclient
permit nopass :network as root cmd killall args -9 wpa_supplicant
permit nopass :network as root cmd killall args -9 ppp
permit nopass :network as root cmd cat args /etc/ppp/ppp.conf
permit nopass :network as root cmd /etc/rc.d/netif args onerestart
permit nopass :network as root cmd tee args /etc/resolv.conf
permit nopass :network as root cmd tee args -a /etc/resolv.conf

Scripts

In this post I attach scripts I have made and used for about 13 years since I started to use FreeBSD on the desktop. Download them all in the scripts.tar.gz file and unpack them into the ~/scripts to make it look like that.

% find scripts | sort
scripts/__openbox_cpufreq.sh
scripts/__openbox_current_wallpaper.sh
scripts/__openbox_delete_wallpaper.sh
scripts/__openbox_dmenu.sh
scripts/__openbox_edit_screenshot.sh
scripts/__openbox_edit_wallpaper_gimp.sh
scripts/__openbox_freebsd_sound.sh
scripts/__openbox_lock_zzz.sh
scripts/__openbox_lock.sh
scripts/__openbox_recent.sh
scripts/__openbox_reload_wallpaper.sh
scripts/__openbox_restart_conky.sh
scripts/__openbox_restart_dzen2.sh
scripts/__openbox_restart_plank.sh
scripts/__openbox_restart_tint2.sh
scripts/__openbox_show_screenshot.sh
scripts/__openbox_stats_ps_KILLALL.sh
scripts/__openbox_stats_top_cpu_KILL.sh
scripts/__openbox_stats_top_cpu_RENICE.sh
scripts/__openbox_stats_top_mem_KILL.sh
scripts/__openbox_stats_top_mem_RENICE.sh
scripts/aero-snap.sh
scripts/fc-cache.sh
scripts/firefox-clean.sh
scripts/network.sh
scripts/random_wallpaper.sh
scripts/shot.sh
scripts/xterm.sh
scripts/desktop-kill-shit.sh
scripts/desktop-battery-warning.sh

Make sure they remain executable.

% chmod +x ~/scripts/*

To make them work properly add ~/scripts into the ${PATH} variable at the beginning of the ~/.xinitrc file.

# PATH TO SCRIPTS
  export PATH=${PATH}:~/scripts


All of my scripts have this ‘mysterious’ line at the end. Its for statistics to check which scripts are run when (or it at all to which ones to delete).

echo '1' >> ~/scripts/stats/$( basename ${0} )

Thus it is needed to create the ‘stats’ directory.

% mkdir -p ~/scripts/stats

I have implemented that about two months ago and here are the results.

% wc -l ~/scripts/stats/* | sort -n
       1 /home/vermaden/scripts/stats/__openbox_show_screenshot.sh
       2 /home/vermaden/scripts/stats/__openbox_cpufreq.sh
       2 /home/vermaden/scripts/stats/__openbox_current_wallpaper.sh
       2 /home/vermaden/scripts/stats/__openbox_fullscreen.sh
       4 /home/vermaden/scripts/stats/__openbox_restart_dzen2.sh
       4 /home/vermaden/scripts/stats/dzen2-fifo.sh
       5 /home/vermaden/scripts/stats/__openbox_dmenu.sh
       5 /home/vermaden/scripts/stats/__openbox_restart_conky.sh
       5 /home/vermaden/scripts/stats/__openbox_restart_tint2.sh
       6 /home/vermaden/scripts/stats/__openbox_delete_wallpaper.sh
       7 /home/vermaden/scripts/stats/__openbox_freebsd_sound.sh
       8 /home/vermaden/scripts/stats/aero-snap.sh
      12 /home/vermaden/scripts/stats/__openbox_edit_screenshot.sh
      16 /home/vermaden/scripts/stats/__openbox_lock_zzz.sh
      16 /home/vermaden/scripts/stats/__openbox_lock.sh
      22 /home/vermaden/scripts/stats/shot.sh
      24 /home/vermaden/scripts/stats/network.sh
     214 /home/vermaden/scripts/stats/xterm.sh
     960 /home/vermaden/scripts/stats/random_wallpaper.sh
    2767 /home/vermaden/scripts/stats/desktop-battery-warning.sh
   13834 /home/vermaden/scripts/stats/desktop-kill-shit.sh
   17916 total

Of course I limited the output only to scripts that are available in this article, but be patient, more to come later πŸ™‚

Dependencies

To make these scripts work and generally to make all this setup work we will need these dependencies.

  • arandr
  • gfontview
  • qtconfig
  • qt5ct
  • qtconfig-qt4
  • sakura
  • leafpad
  • geany
  • caja
  • thunar
  • libreoffice
  • galculator
  • pidgin
  • firefox
  • chrome
  • deadbeef
  • transmission-gtk
  • gnumeric
  • abiword
  • audacity
  • filezilla
  • midori
  • gimp
  • lupe
  • recorder
  • xvidcap
  • zenity
  • xterm
  • xrdb
  • scrot
  • feh
  • wmctrl
  • xdotool
  • viewnior
  • tint2
  • plank
  • dzen2
  • conky
  • mate-screensaver
  • xlockmore
  • gimp
  • dmenu
  • powerdxx
  • htop
  • galculator

To install them all with pkg(8) just type this line below.

# pkg install \
    geany caja thunar libreoffice galculator pidgin firefox chrome midori \
    abiword deadbeef transmission-gtk gnumeric  audacity filezilla zenity \
    gimp lupe recorder xvidcap  xterm xrdb scrot feh wmctrl xdotool tint2 \
    viewnior plank dzen2 conky mate-screensaver xlockmore powerdxx arandr \
    qt5ct gfontview galculator qtconfig qtconfig-qt4 sakura leafpad dmenu \
    htop 
   

I also assume that wallpapers will be kept under ~/gfx/wallpapers dir and screenshots made under ~/gfx/screenshots directory, so lets create them now.

% mkdir -p ~/gfx/wallpapers
% mkdir -p ~/gfx/screenshots

Crontab

Some of these scripts needs to be put into crontab(1) to work, here are their entries.

% crontab -l
# DESKTOP
  *     *     * * * ~/scripts/desktop-kill-shit.sh                                       1> /dev/null 2> /dev/null
  */5   *     * * * ~/scripts/desktop-battery-warning.sh
  */20  *     * * * ~/scripts/random_wallpaper.sh ~/gfx/wallpapers                       1> /dev/null 2> /dev/null
  12,0  *     * * * /usr/bin/find ~/.cache -mtime +10 -delete                            1> /dev/null 2> /dev/null
  0     */3   * * * /usr/bin/find ~/.local/share/Trash/files -mtime +1 -delete  1> /dev/null 2> /dev/null

Fonts

I use Ubuntu Mono font for the Openbox menus and Fira Sans font for the Openbox window bar titles, thus you will download them in the fonts.tar.gz file and extract them like that into the ~/.fonts directory, if if does not exists, create it.

% find .fonts
.fonts/fira-sans-bold-italic.otf
.fonts/fira-sans-bold.otf
.fonts/fira-sans-italic.otf
.fonts/fira-sans-regular.otf
.fonts/ubuntu-mono-bold-italic.ttf
.fonts/ubuntu-mono-bold.ttf
.fonts/ubuntu-mono-italic.ttf
.fonts/ubuntu-mono-regular.ttf

To make sure that Openbox will ‘see’ them you can verify it using the fc-match(1) command like below.

% fc-match 'Fira Sans'
fira-sans-regular.otf: "Fira Sans" "Regular"

% fc-match 'Ubuntu Mono'
ubuntu-mono-regular.ttf: "Ubuntu Mono" "Regular"

Openbox

Openbox consists mostly of two files.

  • ~/.config/openbox/menu.xml
  • ~/.config/openbox/rc.xml

There are also these two, but its pointless to use them as we set our environment and start our apps/daemons in the ~/.xinitrc file (with ~/.xsession symlink to it), but anyway.

  • ~/.config/openbox/autostart
  • ~/.config/openbox/environment

The icons for the Openbox menu are kept under ~/.config/openbox/icons directory.

Download whole Openbox configuration in the openbox.tar.gz file and unpack it into the ~/.config/openbox to make it look like that.

% find .config/openbox -maxdepth 1
.config/openbox
.config/openbox/rc.xml
.config/openbox/menu.xml
.config/openbox/icons
.config/openbox/environment
.config/openbox/autostart

Openbox Theme

The theme we will use at start is the Openbox Flat made by myself, I do not remember if I put it online on the https://www.box-look.org/ site but that does not matter. Grab it in the openbox-flat-theme.tar.gz file and unpack it like that into the ~/.themes directory, create it if it does not exists.

% find .themes/openbox_flat
.themes/openbox_flat
.themes/openbox_flat/openbox-3
.themes/openbox_flat/openbox-3/iconify.xbm
.themes/openbox_flat/openbox-3/XPM
.themes/openbox_flat/openbox-3/XPM/over.xpm
.themes/openbox_flat/openbox-3/XPM/close.xpm
.themes/openbox_flat/openbox-3/XPM/max.xpm
.themes/openbox_flat/openbox-3/XPM/stick.0.xpm
.themes/openbox_flat/openbox-3/XPM/min.xpm
.themes/openbox_flat/openbox-3/XPM/shade.xpm
.themes/openbox_flat/openbox-3/XPM/stick.1.xpm
.themes/openbox_flat/openbox-3/max.xbm
.themes/openbox_flat/openbox-3/close.xbm
.themes/openbox_flat/openbox-3/bullet.xbm
.themes/openbox_flat/openbox-3/shade.xbm
.themes/openbox_flat/openbox-3/themerc
.themes/openbox_flat/openbox-3/desk.xbm
.themes/openbox_flat/openbox-3/desk_toggled.xbm

Openbox FreeBSD Submenus

The ‘systemOpenbox submenu is for FreeBSD top(1)/ps(1) commands and config files/logs.

openbox-system.jpg

The ‘soundOpenbox submenu is for FreeBSD default sound output selection.

openbox-sound.jpg

The ‘recentOpenbox submenu is for Recent files.

openbox-recent.jpg

Check ‘screenshot:‘ and ‘wallpaper:‘ in the ‘x11Openbox submenu for screenshots/wallpapers management.

Check ‘cpu:‘ in the ‘utilitiesOpenbox submenu for FreeBSD for CPU frequency scaling.

Check ‘NETWORK:‘ in the ‘daemonsOpenbox submenu for FreeBSD network management with network.sh script.

Shortcuts

Lets start with the most basic ones. [SUPER] is the so called Windows key.

Shortcuts – Virtual Desktops

  • [ALT] + [F1] – switch to 1st virtual desktop.
  • [ALT] + [F2] – switch to 2nd virtual desktop.
  • [ALT] + [F3] – switch to 3rd virtual desktop.
  • [ALT] + [F4] – switch to 4th virtual desktop.
  • [SHIFT] + [ALT] + [F1] – move current window to 1st virtual desktop.
  • [SHIFT] + [ALT] + [F2] – move current window to 2nd virtual desktop.
  • [SHIFT] + [ALT] + [F3] – move current window to 3rd virtual desktop.
  • [SHIFT] + [ALT] + [F4] – move current window to 4th virtual desktop.

Shortcuts – Menus

  • [SUPER] + [SPACE] – show Openbox root menu.
  • [SUPER] + [ALT] + [SPACE] – show Openbox window list menu.
  • [ALT] + [SPACE] – show current window options menu (client menu).

Shortcuts – Window Management

  • [ALT] + [TAB] – cycle windows focus forward.
  • [SHIFT] + [ALT] + [TAB] – cycle windows focus backward.
  • [CTRL] + [ALT] + [Q] – close current window.
  • [CTRL] + [ALT] + [F] – put current window info fullscreen.
  • [ALT] + [Up] – shade current window.
  • [ALT] + [Down] – minimize current window.
  • [ALT] + [ESC] – send current window below all other windows.

Shortcuts – Advanced Aero Snap

  • [SUPER] + [Up] – move window to half of the screen from top.
  • [SUPER] + [Down] – move window to half of the screen from bottom.
  • [SUPER] + [Left] – move window to half of the screen from left.
  • [SUPER] + [Right] – move window to half of the screen from right.
  • [SUPER] + [CTRL] + [Up] – move window to top-left part of the screen.
  • [SUPER] + [CTRL] + [Down] – move window to bottom-left part of the screen.
  • [SUPER] + [ALT] + [Up] – move window to top-right part of the screen.
  • [SUPER] + [ALT] + [Down] – move window to bottom-right part of the screen.
  • [SUPER] + [ESC] – move window to center – but without fullscreen.

Shortcuts – Mouse

  • [Scroll Up] on Desktop – previous virtual desktop.
  • [Scroll Down] on Desktop – next virtual desktop.
  • [Scroll Up] on (unshaded) Window Titlebar – shade current window.
  • [Scroll Up] on (shaded) Window Titlebar – unshade current window.
  • [Middle Click] on Window Titlebar – send window to background.
  • [Right Click] on Window Titlebar – show window options menu (client menu).
  • [Left Click] on Window Titlebar Icon – show window options menu (client menu).
  • [Middle Click] on Window Titlebar Icon – close window.

Shortcuts – Various

  • [CTRL] + [SHIFT] + [ESC] – launch xterm(1) with htop(1) started with doas(1) for root provilages.
  • [SUPER] + [E] – start Explorer Caja primary file manager.
  • [SUPER] + [E] – start Thunar secondary file manager.
  • [SUPER] + [D] – show desktop – minimize all windows.
  • [SUPER] + [R] – launch dmenu(1) starter.
  • [SUPER] + [L] – lock the screen.
  • [ALT] + [SHIFT] + [SUPER] + [L] – lock the screen and go to sleep.
  • [CTRL] + [PrintScreen] – make screenshot of the whole screen.
  • [SHIFT] + [CTRL] + [PrintScreen] – make screenshot of current window (click without moving the mouse) or selection (select part of the screen).

Shortcuts – Volume

These two work from keyboard.

  • [SUPER] + [ALT] + [PageUp] – increase volume.
  • [SUPER] + [ALT] + [PageDown] – decrease volume.

These below with mouse.

For those who do not have mouse with buttons on the wheel like the Lenovo ThinkPad Precision Wireless Mouse (0B47163) for example, use [ALT] key with mouse scroll up/scroll down on the desktop to increase/decrease volume.

If you do have such mouse, then left on the wheel to decrease and right on the wheel to increase volume.

Random Wallpaper

The random wallpaper handling is done with the ~/scripts/random_wallpaper.sh script. Be sure to put some images into the ~/gfx/wallpapers directory to make it work and to configure crontab(1) properly as shown earlier.

Random xterm(1) Theme

To have random xterm(1) theme on every startup you need three things, the ~/.Xdefaults default config file which is used by xterm(1), the ~/scripts/xterm.sh script and the ~/.config/Xdefaults directory with xterm(1) themes. I gathered all these themes all over the Internet, only the VERMADEN and VERMADEN-OLD themes are created by me.

Little preview of some of the included xterm(1) themes.

openbox-xterm.jpg

To make xterm(1) icon look better you will also need icons.tar.gz file download and extract with the end result looking as follows.

% find .icons
.icons/vermaden/xterm.xpm

Download and extract the xterm.tar.gz file to make its contents look like that.

% find .Xdefaults .config/Xdefaults 
.config/Xdefaults
.config/Xdefaults/Xdefaults.theme.3024NIGHT
.config/Xdefaults/Xdefaults.theme.ALIENBLOOD
.config/Xdefaults/Xdefaults.theme.ARTHUR
.config/Xdefaults/Xdefaults.theme.EARTHSONG
.config/Xdefaults/Xdefaults.theme.ELEMENTAL
.config/Xdefaults/Xdefaults.theme.ELEMENTARY
.config/Xdefaults/Xdefaults.theme.ELIC
.config/Xdefaults/Xdefaults.theme.FLATUI
.config/Xdefaults/Xdefaults.theme.FREYA
.config/Xdefaults/Xdefaults.theme.FRONTEND-DELIGHT
.config/Xdefaults/Xdefaults.theme.GITHUB
.config/Xdefaults/Xdefaults.theme.GREY
.config/Xdefaults/Xdefaults.theme.GRUVBOX-DARK
.config/Xdefaults/Xdefaults.theme.GRUVBOX-LIGHT
.config/Xdefaults/Xdefaults.theme.GRUVBOX-TILIX
.config/Xdefaults/Xdefaults.theme.HARPER
.config/Xdefaults/Xdefaults.theme.HEMISU-LIGHT
.config/Xdefaults/Xdefaults.theme.HIGHWAY
.config/Xdefaults/Xdefaults.theme.IC-GREEN-PPL
.config/Xdefaults/Xdefaults.theme.MOLOKAI
.config/Xdefaults/Xdefaults.theme.MONOKAI-SODA
.config/Xdefaults/Xdefaults.theme.N0TCH2K
.config/Xdefaults/Xdefaults.theme.NEOPOLITAN
.config/Xdefaults/Xdefaults.theme.PALI
.config/Xdefaults/Xdefaults.theme.PAPERCOLOUR
.config/Xdefaults/Xdefaults.theme.SOLARIZED
.config/Xdefaults/Xdefaults.theme.SOLARIZED-LIGHT
.config/Xdefaults/Xdefaults.theme.SPACEDUST
.config/Xdefaults/Xdefaults.theme.TOMORROW
.config/Xdefaults/Xdefaults.theme.TWILIGHT
.config/Xdefaults/Xdefaults.theme.VERMADEN
.config/Xdefaults/Xdefaults.theme.VERMADEN-OLD

Thats a lot of information for one article, feel free to ask me for anything related or about things that I might forgot to put here.

UPDATE 1 – network.sh Integration

In other article I described how to manage various network sources with the network.sh script – FreeBSD Network Management with network.sh Script – available here.

Below is an example of integration of that network.sh script with Openbox window manager.

network.sh.openbox.menu.jpg

… and here is the code used in the ~/.config/openbox/menu.xml file.

network.sh.openbox.menu.code

Of course you can integrate network.sh script with almost anything – its just a command πŸ™‚

EOF
Advertisements

Distributed Object Storage with Minio on FreeBSD

Meet Minio.

minio-logo-arch-32

Free and open source distributed object storage server compatible with Amazon S3 v2/v4 API. Offers data protection against hardware failures using erasure code and bitrot detection. Supports highly available distributed setup. Provides confidentiality, integrity and authenticity assurances for encrypted data with negligible performance overhead. Both server side and client side encryption are supported. Below is the image of example Minio setup.

Web

The Minio identifies itself as the ZFS of Cloud Object Storage. This guide will show You how to setup highly available distributed Minio storage on the FreeBSD operating system with ZFS as backend for Minio data. For convenience we will use FreeBSD Jails operating system level virtualization.

Setup

The setup will assume that You have 3 datacenters and assumption that you have two datacenters in whose the most of the data must reside and that the third datacenter is used as a ‘quorum/witness’ role. Distributed Minio supports up to 16 nodes/drives total, so we may juggle with that number to balance data between desired datacenters. As we have 16 drives to allocate resources on 3 sites we will use 7 + 7 + 2 approach here. The datacenters where most of the data must reside have 7/16 ratio while the ‘quorum/witness’ datacenter have only 2/16 ratio. Thanks to built in Minio redundancy we may loose (turn off for example) any one of those machines and our object storage will still be available and ready to use for any purpose.

Jails

First we will create 3 jails for our proof of concept Minio setup, storage1 will have the ‘quorum/witness’ role while storage2 and storage3 will have the ‘data’ role. To distinguish commands I type on the host system and storageX Jail I use two different prompts, this way it should be obvious what command to execute and where.

Command on the host system.

host # command

Command on the storageX Jail.

root@storageX:/ # command

First we will create the base Jails for our setup.

host # mkdir -p /jail/BASE /jail/storage1 /jail/storage2 /jail/storage3
host # cd /jail/BASE
host # fetch http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/11.1-RELEASE/base.txz
host # for I in 1 2 3; do echo ${I}; tar --unlink -xpJf /jail/BASE/base.txz -C /jail/storage${I}; done
1
2
3
host #

We will now add Jails configuration the the /etc/jail.conf file.

I have used my laptop for the Jail host. This is why Jail will configured to use the wireless wlan0 interface and 192.168.43.10X addresses.

host # for I in 1 2 3
do
  cat >> /etc/jail.conf << __EOF
storage${I} {
  host.hostname = storage${I}.local;
  ip4.addr = 192.168.43.10${I};
  interface = wlan0;
  path = /jail/storage${I};
  exec.start = "/bin/sh /etc/rc";
  exec.stop = "/bin/sh /etc/rc.shutdown";
  exec.clean;
  mount.devfs;
  allow.raw_sockets;
}

__EOF
done
host #

Lets verify that /etc/jail.conf file is configured as desired.

host # cat /etc/jail.conf
storage1 {
  host.hostname = storage1.local;
  ip4.addr = 192.168.43.101;
  interface = wlan0;
  path = /jail/storage1;
  exec.start = "/bin/sh /etc/rc";
  exec.stop = "/bin/sh /etc/rc.shutdown";
  exec.clean;
  mount.devfs;
  allow.raw_sockets;
}

storage2 {
  host.hostname = storage2.local;
  ip4.addr = 192.168.43.102;
  interface = wlan0;
  path = /jail/storage2;
  exec.start = "/bin/sh /etc/rc";
  exec.stop = "/bin/sh /etc/rc.shutdown";
  exec.clean;
  mount.devfs;
  allow.raw_sockets;
}

storage3 {
  host.hostname = storage3.local;
  ip4.addr = 192.168.43.103;
  interface = wlan0;
  path = /jail/storage3;
  exec.start = "/bin/sh /etc/rc";
  exec.stop = "/bin/sh /etc/rc.shutdown";
  exec.clean;
  mount.devfs;
  allow.raw_sockets;
}

host #

Now we will start our Jails.

host # for I in 1 2 3; do service jail onestart storage${I}; done
Starting jails: storage1.
Starting jails: storage2.
Starting jails: storage3.

Lets see how they work.

host # jls
   JID  IP Address      Hostname                      Path
     1  192.168.43.101  storage1.local                /jail/storage1
     2  192.168.43.102  storage2.local                /jail/storage2
     3  192.168.43.103  storage3.local                /jail/storage3

Now lets add DNS server so they will have Internet connectivity.

host # for I in 1 2 3; do echo nameserver 1.1.1.1 > /jail/storage${I}/etc/resolv.conf; done

We can now install Minio package.

host # for I in 1 2 3; do jexec storage${I} env ASSUME_ALWAYS_YES=yes pkg install -y minio; echo; done
Bootstrapping pkg from pkg+http://pkg.FreeBSD.org/FreeBSD:11:amd64/quarterly, please wait...
Verifying signature with trusted certificate pkg.freebsd.org.2013102301... done
[storage1.local] Installing pkg-1.10.5...
[storage1.local] Extracting pkg-1.10.5: 100%
Updating FreeBSD repository catalogue...
pkg: Repository FreeBSD load error: access repo file(/var/db/pkg/repo-FreeBSD.sqlite) failed: No such file or directory
[storage1.local] Fetching meta.txz: 100%    944 B   0.9kB/s    00:01    
[storage1.local] Fetching packagesite.txz: 100%    6 MiB 637.1kB/s    00:10    
Processing entries: 100%
FreeBSD repository update completed. 31143 packages processed.
All repositories are up to date.
Updating database digests format: 100%
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        minio: 2018.03.19.19.22.06

Number of packages to be installed: 1

The process will require 22 MiB more space.
6 MiB to be downloaded.
[storage1.local] [1/1] Fetching minio-2018.03.19.19.22.06.txz: 100%    6 MiB 305.6kB/s    00:19    
Checking integrity... done (0 conflicting)
[storage1.local] [1/1] Installing minio-2018.03.19.19.22.06...
===> Creating groups.
Creating group 'minio' with gid '473'.
===> Creating users
Creating user 'minio' with uid '473'.
[storage1.local] [1/1] Extracting minio-2018.03.19.19.22.06: 100%

Bootstrapping pkg from pkg+http://pkg.FreeBSD.org/FreeBSD:11:amd64/quarterly, please wait...
Verifying signature with trusted certificate pkg.freebsd.org.2013102301... done
[storage2.local] Installing pkg-1.10.5...
[storage2.local] Extracting pkg-1.10.5: 100%
Updating FreeBSD repository catalogue...
pkg: Repository FreeBSD load error: access repo file(/var/db/pkg/repo-FreeBSD.sqlite) failed: No such file or directory
[storage2.local] Fetching meta.txz: 100%    944 B   0.9kB/s    00:01    
[storage2.local] Fetching packagesite.txz: 100%    6 MiB 637.1kB/s    00:10    
Processing entries: 100%
FreeBSD repository update completed. 31143 packages processed.
All repositories are up to date.
Updating database digests format: 100%
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        minio: 2018.03.19.19.22.06

Number of packages to be installed: 1

The process will require 22 MiB more space.
6 MiB to be downloaded.
[storage2.local] [1/1] Fetching minio-2018.03.19.19.22.06.txz: 100%    6 MiB 305.6kB/s    00:19    
Checking integrity... done (0 conflicting)
[storage2.local] [1/1] Installing minio-2018.03.19.19.22.06...
===> Creating groups.
Creating group 'minio' with gid '473'.
===> Creating users
Creating user 'minio' with uid '473'.
[storage2.local] [1/1] Extracting minio-2018.03.19.19.22.06: 100%

Bootstrapping pkg from pkg+http://pkg.FreeBSD.org/FreeBSD:11:amd64/quarterly, please wait...
Verifying signature with trusted certificate pkg.freebsd.org.2013102301... done
[storage3.local] Installing pkg-1.10.5...
[storage3.local] Extracting pkg-1.10.5: 100%
Updating FreeBSD repository catalogue...
pkg: Repository FreeBSD load error: access repo file(/var/db/pkg/repo-FreeBSD.sqlite) failed: No such file or directory
[storage3.local] Fetching meta.txz: 100%    944 B   0.9kB/s    00:01    
[storage3.local] Fetching packagesite.txz: 100%    6 MiB 637.1kB/s    00:10    
Processing entries: 100%
FreeBSD repository update completed. 31143 packages processed.
All repositories are up to date.
Updating database digests format: 100%
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        minio: 2018.03.19.19.22.06

Number of packages to be installed: 1

The process will require 22 MiB more space.
6 MiB to be downloaded.
[storage3.local] [1/1] Fetching minio-2018.03.19.19.22.06.txz: 100%    6 MiB 305.6kB/s    00:19    
Checking integrity... done (0 conflicting)
[storage3.local] [1/1] Installing minio-2018.03.19.19.22.06...
===> Creating groups.
Creating group 'minio' with gid '473'.
===> Creating users
Creating user 'minio' with uid '473'.
[storage3.local] [1/1] Extracting minio-2018.03.19.19.22.06: 100%

host #

Lets verify that Minio package has installed successfully.

host # for I in 1 2 3; do jexec storage${I} which minio; done
/usr/local/bin/minio
/usr/local/bin/minio
/usr/local/bin/minio
host #

Now we will configure /etc/hosts file.

root@storage1:/ # cat >> /etc/hosts << __EOF
192.168.43.101 storage1
192.168.43.102 storage2
192.168.43.103 storage3
__EOF
root@storage1:/ # cat >> /etc/hosts << __EOF
192.168.43.101 storage1
192.168.43.102 storage2
192.168.43.103 storage3
__EOF
root@storage1:/ # cat >> /etc/hosts << __EOF
192.168.43.101 storage1
192.168.43.102 storage2
192.168.43.103 storage3
__EOF

We will create directories for Minio data.

host # for DIR in 1 2 3 4 5 6 7
do
  for I in 2 3
  do
    jexec storage${I} mkdir -p /data${DIR}
  done
done
host # for DIR in 1 2
do
  for I in 1
  do
    jexec storage${I} mkdir -p /data${DIR}
  done
done

Lets verify that that our data directories created successfully.

host # for I in 1 2 3
  do
    echo storage${I}
    jexec storage${I} ls -1 / | grep data
    echo
  done


storage1
data1
data2

storage2
data1
data2
data3
data4
data5
data6
data7

storage3
data1
data2
data3
data4
data5
data6
data7

Basic minio command example.

root@storage1:/ # minio
NAME:
  minio - Cloud Storage Server.

DESCRIPTION:
  Minio is an Amazon S3 compatible object storage server. Use it to store photos, videos, VMs, containers, log files, or any blob of data as objects.

USAGE:
  minio [FLAGS] COMMAND [ARGS...]

COMMANDS:
  server   Start object storage server.
  gateway  Start object storage gateway.
  update   Check for a new software update.
  version  Print version.
  
FLAGS:
  --config-dir value, -C value  Path to configuration directory. (default: "/root/.minio")
  --quiet                       Disable startup information.
  --json                        Output server logs and startup information in json format.
  --help, -h                    Show help.
  
VERSION:
  2018-03-19T19:22:06Z

Now we can generate the list of directories on servers to add as argument for Minio.

host # for DIR in 1 2
do
  for I in 1 
  do
    echo -n http://
    jls | grep storage${I} | awk '{printf $3}' | sed s/.local//g
    echo ":9000/data${DIR} \\"
  done
done | sort -n

host # for DIR in 1 2 3 4 5 6 7
do
  for I in 2 3
  do
    echo -n http://
    jls | grep storage${I} | awk '{printf $3}' | sed s/.local//g
    echo ":9000/data${DIR} \\"
  done
done | sort -n
http://storage1:9000/data1 \
http://storage1:9000/data2 \
http://storage2:9000/data1 \
http://storage2:9000/data2 \
http://storage2:9000/data3 \
http://storage2:9000/data4 \
http://storage2:9000/data5 \
http://storage2:9000/data6 \
http://storage2:9000/data7 \
http://storage3:9000/data1 \
http://storage3:9000/data2 \
http://storage3:9000/data3 \
http://storage3:9000/data4 \
http://storage3:9000/data5 \
http://storage3:9000/data6 \
http://storage3:9000/data7 \

We can as well just write it down by hand of course πŸ™‚

host # for DIR in 1 2
do
  for I in 1 
  do
    echo -n http://
    jls | grep storage${I} | awk '{printf $3}' | sed s/.local//g
    echo -n ":9000/data${DIR} "
  done
done | sort -n

host # for DIR in 1 2 3 4 5 6 7
do
  for I in 2 3
  do
    echo -n http://
    jls | grep storage${I} | awk '{printf $3}' | sed s/.local//g
    echo -n ":9000/data${DIR} "
  done
done | sort -n

This is out list of data directories that we will use to configure Minio in FreeBSD’s main configuration /etc/rc.conf file.

http://storage1:9000/data1 http://storage1:9000/data2 http://storage2:9000/data1 http://storage2:9000/data2 http://storage2:9000/data3 http://storage2:9000/data4 http://storage2:9000/data5 http://storage2:9000/data6 http://storage2:9000/data7 http://storage3:9000/data1 http://storage3:9000/data2 http://storage3:9000/data3 http://storage3:9000/data4 http://storage3:9000/data5 http://storage3:9000/data6 http://storage3:9000/data7

Now, lets put Minio settings into the /etc/rc.conf file.

root@storageX:~ # cat > /etc/rc.conf << __EOF 
minio_enable=YES
minio_disks="http://storage1:9000/data1 http://storage1:9000/data2 http://storage2:9000/data1 http://storage2:9000/data2 http://storage2:9000/data3 http://storage2:9000/data4 http://storage2:9000/data5 http://storage2:9000/data6 http://storage2:9000/data7 http://storage3:9000/data1 http://storage3:9000/data2 http://storage3:9000/data3 http://storage3:9000/data4 http://storage3:9000/data5 http://storage3:9000/data6 http://storage3:9000/data7"
__EOF
root@storageX:~ # 
root@storageX:~ # cat /etc/rc.conf
minio_enable=YES
minio_disks="http://storage1:9000/data1 http://storage1:9000/data2 http://storage2:9000/data1 http://storage2:9000/data2 http://storage2:9000/data3 http://storage2:9000/data4 http://storage2:9000/data5 http://storage2:9000/data6 http://storage2:9000/data7 http://storage3:9000/data1 http://storage3:9000/data2 http://storage3:9000/data3 http://storage3:9000/data4 http://storage3:9000/data5 http://storage3:9000/data6 http://storage3:9000/data7"
root@storageX:~ #

Now we will start and configure Minio for the first time.

On each storageX server run the following set of commands.

host # jexec storage3
root@storage3:~ # 
root@storage3:/ # rm -rf /http:\*
root@storage3:/ # rm -rf /usr/local/etc/minio
root@storage3:/ # rm -rf /data?/* /data?/.minio.sys
root@storage3:/ # touch                /var/log/minio.log
root@storage3:/ # chown    minio:minio /var/log/minio.log
root@storage3:/ # mkdir -p             /usr/local/etc/minio
root@storage3:/ # chown -R minio:minio /usr/local/etc/minio
root@storage3:/ # mkdir -p             /http::
root@storage3:/ # chown -R minio:minio /http::
root@storage3:/ # mkdir -p             /http:
root@storage3:/ # chown -R minio:minio /http:
root@storage3:/ # su -m minio -c 'env \\
?   MINIO_ACCESS_KEY=alibaba \\
?   MINIO_SECRET_KEY=0P3NS3S4M3 \\
?   minio server \\
?     --config-dir /usr/local/etc/minio \\
?     http://storage1:9000/data1 \\
?     http://storage1:9000/data2 \\
?     http://storage2:9000/data1 \\
?     http://storage2:9000/data2 \\
?     http://storage2:9000/data3 \\
?     http://storage2:9000/data4 \\
?     http://storage2:9000/data5 \\
?     http://storage2:9000/data6 \\
?     http://storage2:9000/data7 \\
?     http://storage3:9000/data1 \\
?     http://storage3:9000/data2 \\
?     http://storage3:9000/data3 \\
?     http://storage3:9000/data4 \\
?     http://storage3:9000/data5 \\
?     http://storage3:9000/data6 \\
?     http://storage3:9000/data7'
Created minio configuration file successfully at /usr/local/etc/minio
Waiting for the first server to format the disks.
Waiting for the first server to format the disks.
Drive Capacity: 504 GiB Free, 515 GiB Total
Status:         16 Online, 0 Offline. 

Endpoint:  http://192.168.43.103:9000
AccessKey: alibaba 
SecretKey: 0P3NS3S4M3 

Browser Access:
   http://192.168.43.103:9000

Command-line Access: https://docs.minio.io/docs/minio-client-quickstart-guide
   $ mc config host add myminio http://192.168.43.103:9000 alibaba 0P3NS3S4M3

Object API (Amazon S3 compatible):
   Go:         https://docs.minio.io/docs/golang-client-quickstart-guide
   Java:       https://docs.minio.io/docs/java-client-quickstart-guide
   Python:     https://docs.minio.io/docs/python-client-quickstart-guide
   JavaScript: https://docs.minio.io/docs/javascript-client-quickstart-guide
   .NET:       https://docs.minio.io/docs/dotnet-client-quickstart-guide
host # jexec storage2
root@storage2:~ # 
root@storage2:/ # rm -rf /http:\*
root@storage2:/ # rm -rf /usr/local/etc/minio
root@storage2:/ # rm -rf /data?/* /data?/.minio.sys
root@storage2:/ # touch                /var/log/minio.log
root@storage2:/ # chown    minio:minio /var/log/minio.log
root@storage2:/ # mkdir -p             /usr/local/etc/minio
root@storage2:/ # chown -R minio:minio /usr/local/etc/minio
root@storage2:/ # mkdir -p             /http::
root@storage2:/ # chown -R minio:minio /http::
root@storage2:/ # mkdir -p             /http:
root@storage2:/ # chown -R minio:minio /http:
root@storage2:/ # su -m minio -c 'env \\
?   MINIO_ACCESS_KEY=alibaba \\
?   MINIO_SECRET_KEY=0P3NS3S4M3 \\
?   minio server \\
?     --config-dir /usr/local/etc/minio \\
?     http://storage1:9000/data1 \\
?     http://storage1:9000/data2 \\
?     http://storage2:9000/data1 \\
?     http://storage2:9000/data2 \\
?     http://storage2:9000/data3 \\
?     http://storage2:9000/data4 \\
?     http://storage2:9000/data5 \\
?     http://storage2:9000/data6 \\
?     http://storage2:9000/data7 \\
?     http://storage3:9000/data1 \\
?     http://storage3:9000/data2 \\
?     http://storage3:9000/data3 \\
?     http://storage3:9000/data4 \\
?     http://storage3:9000/data5 \\
?     http://storage3:9000/data6 \\
?     http://storage3:9000/data7'
Created minio configuration file successfully at /usr/local/etc/minio
Waiting for the first server to format the disks.
Waiting for the first server to format the disks.
Drive Capacity: 504 GiB Free, 515 GiB Total
Status:         16 Online, 0 Offline. 

Endpoint:  http://192.168.43.102:9000
AccessKey: alibaba 
SecretKey: 0P3NS3S4M3 

Browser Access:
   http://192.168.43.102:9000

Command-line Access: https://docs.minio.io/docs/minio-client-quickstart-guide
   $ mc config host add myminio http://192.168.43.102:9000 alibaba 0P3NS3S4M3

Object API (Amazon S3 compatible):
   Go:         https://docs.minio.io/docs/golang-client-quickstart-guide
   Java:       https://docs.minio.io/docs/java-client-quickstart-guide
   Python:     https://docs.minio.io/docs/python-client-quickstart-guide
   JavaScript: https://docs.minio.io/docs/javascript-client-quickstart-guide
   .NET:       https://docs.minio.io/docs/dotnet-client-quickstart-guide
host # jexec storage1
root@storage1:~ # 
root@storage1:/ # rm -rf /http:\*
root@storage1:/ # rm -rf /usr/local/etc/minio
root@storage1:/ # rm -rf /data?/* /data?/.minio.sys
root@storage1:/ # touch                /var/log/minio.log
root@storage1:/ # chown    minio:minio /var/log/minio.log
root@storage1:/ # mkdir -p             /usr/local/etc/minio
root@storage1:/ # chown -R minio:minio /usr/local/etc/minio
root@storage1:/ # mkdir -p             /http::
root@storage1:/ # chown -R minio:minio /http::
root@storage1:/ # mkdir -p             /http:
root@storage1:/ # chown -R minio:minio /http:
root@storage1:/ # su -m minio -c 'env \\
?   MINIO_ACCESS_KEY=alibaba \\
?   MINIO_SECRET_KEY=0P3NS3S4M3 \\
?   minio server \\
?     --config-dir /usr/local/etc/minio \\
?     http://storage1:9000/data1 \\
?     http://storage1:9000/data2 \\
?     http://storage2:9000/data1 \\
?     http://storage2:9000/data2 \\
?     http://storage2:9000/data3 \\
?     http://storage2:9000/data4 \\
?     http://storage2:9000/data5 \\
?     http://storage2:9000/data6 \\
?     http://storage2:9000/data7 \\
?     http://storage3:9000/data1 \\
?     http://storage3:9000/data2 \\
?     http://storage3:9000/data3 \\
?     http://storage3:9000/data4 \\
?     http://storage3:9000/data5 \\
?     http://storage3:9000/data6 \\
?     http://storage3:9000/data7'
Created minio configuration file successfully at /usr/local/etc/minio
Waiting for the first server to format the disks.
Waiting for the first server to format the disks.
Drive Capacity: 504 GiB Free, 515 GiB Total
Status:         16 Online, 0 Offline. 

Endpoint:  http://192.168.43.101:9000
AccessKey: alibaba 
SecretKey: 0P3NS3S4M3 

Browser Access:
   http://192.168.43.101:9000

Command-line Access: https://docs.minio.io/docs/minio-client-quickstart-guide
   $ mc config host add myminio http://192.168.43.101:9000 alibaba 0P3NS3S4M3

Object API (Amazon S3 compatible):
   Go:         https://docs.minio.io/docs/golang-client-quickstart-guide
   Java:       https://docs.minio.io/docs/java-client-quickstart-guide
   Python:     https://docs.minio.io/docs/python-client-quickstart-guide
   JavaScript: https://docs.minio.io/docs/javascript-client-quickstart-guide
   .NET:       https://docs.minio.io/docs/dotnet-client-quickstart-guide

Here is how it looks in the xterm terminal.

minio-first-run-setup

We can now verify in the browser that it actually works.

minio-browser-01

Now hit [CTRL]+[C] in each of these windows to stop the Minio cluster.

We will now start Minio with FreeBSD rc(8) subsystem as a service.

root@storage1:/ # service minio start
Starting minio.
root@storage1:/ # cat /var/log/minio.log 
root@storage1:/ # service minio status
minio is running as pid 50309.

Lets check if it works.

root@storage1:/ # ps -U minio
  PID TT  STAT    TIME COMMAND
50308  -  IsJ  0:00.00 daemon: /usr/bin/env[50309] (daemon)
50309  -  IJ   0:00.27 /usr/local/bin/minio -C /usr/local/etc/minio server (...)

Now we will do some basic operations, login into Minio distributed storage, create new bucket and upload some file to it.

minio-browser-02

This is how empty Minio cluster looks like.

minio-browser-03

Select Create Bucket option from the button below.

minio-browser-04-create-bucket

We will use name test for our new bucket.

minio-browser-05-create-bucket

It is created and we can access it.

minio-browser-06-bucket

Lets Upload File using same menu as previously.

minio-browser-07-file-upload

The upload progress shown by Minio.

minio-browser-08-file-upload

File has been indeed uploaded.

minio-browser-09-file-upload

By clicking on it we may access it directly from the browser.

minio-browser-10-file-display

We can also share link to that file by using the File Menu as shown below.

minio-browser-10-file-link

The link creation dialog is shown below.

minio-browser-11-file-link

minio-browser-12-file-link

Lets see how Minio distributes the data – the ThinkPad Design – Spirit and Essence.pdf file in out case – over its data directories spread across the servers.

host # jexec storage1
root@storage1:/ # find /data?/test
/data1/test
/data1/test/ThinkPad Design - Spirit and Essence.pdf
/data1/test/ThinkPad Design - Spirit and Essence.pdf/xl.json
/data1/test/ThinkPad Design - Spirit and Essence.pdf/part.1
/data2/test
/data2/test/ThinkPad Design - Spirit and Essence.pdf
/data2/test/ThinkPad Design - Spirit and Essence.pdf/xl.json
/data2/test/ThinkPad Design - Spirit and Essence.pdf/part.1
root@storage1:/ # exit
host # jexec storage2
root@storage2:/ # find /data?/test
/data1/test
/data1/test/ThinkPad Design - Spirit and Essence.pdf
/data1/test/ThinkPad Design - Spirit and Essence.pdf/part.1
/data1/test/ThinkPad Design - Spirit and Essence.pdf/xl.json
/data2/test
/data2/test/ThinkPad Design - Spirit and Essence.pdf
/data2/test/ThinkPad Design - Spirit and Essence.pdf/xl.json
/data2/test/ThinkPad Design - Spirit and Essence.pdf/part.1
/data3/test
/data3/test/ThinkPad Design - Spirit and Essence.pdf
/data3/test/ThinkPad Design - Spirit and Essence.pdf/part.1
/data3/test/ThinkPad Design - Spirit and Essence.pdf/xl.json
/data4/test
/data4/test/ThinkPad Design - Spirit and Essence.pdf
/data4/test/ThinkPad Design - Spirit and Essence.pdf/part.1
/data4/test/ThinkPad Design - Spirit and Essence.pdf/xl.json
/data5/test
/data5/test/ThinkPad Design - Spirit and Essence.pdf
/data5/test/ThinkPad Design - Spirit and Essence.pdf/part.1
/data5/test/ThinkPad Design - Spirit and Essence.pdf/xl.json
/data6/test
/data6/test/ThinkPad Design - Spirit and Essence.pdf
/data6/test/ThinkPad Design - Spirit and Essence.pdf/part.1
/data6/test/ThinkPad Design - Spirit and Essence.pdf/xl.json
/data7/test
/data7/test/ThinkPad Design - Spirit and Essence.pdf
/data7/test/ThinkPad Design - Spirit and Essence.pdf/xl.json
/data7/test/ThinkPad Design - Spirit and Essence.pdf/part.1
root@storage2:/ # exit
host # jexec storage3
root@storage3:/ # find /data?/test
/data1/test
/data1/test/ThinkPad Design - Spirit and Essence.pdf
/data1/test/ThinkPad Design - Spirit and Essence.pdf/part.1
/data1/test/ThinkPad Design - Spirit and Essence.pdf/xl.json
/data2/test
/data2/test/ThinkPad Design - Spirit and Essence.pdf
/data2/test/ThinkPad Design - Spirit and Essence.pdf/xl.json
/data2/test/ThinkPad Design - Spirit and Essence.pdf/part.1
/data3/test
/data3/test/ThinkPad Design - Spirit and Essence.pdf
/data3/test/ThinkPad Design - Spirit and Essence.pdf/xl.json
/data3/test/ThinkPad Design - Spirit and Essence.pdf/part.1
/data4/test
/data4/test/ThinkPad Design - Spirit and Essence.pdf
/data4/test/ThinkPad Design - Spirit and Essence.pdf/part.1
/data4/test/ThinkPad Design - Spirit and Essence.pdf/xl.json
/data5/test
/data5/test/ThinkPad Design - Spirit and Essence.pdf
/data5/test/ThinkPad Design - Spirit and Essence.pdf/part.1
/data5/test/ThinkPad Design - Spirit and Essence.pdf/xl.json
/data6/test
/data6/test/ThinkPad Design - Spirit and Essence.pdf
/data6/test/ThinkPad Design - Spirit and Essence.pdf/part.1
/data6/test/ThinkPad Design - Spirit and Essence.pdf/xl.json
/data7/test
/data7/test/ThinkPad Design - Spirit and Essence.pdf
/data7/test/ThinkPad Design - Spirit and Essence.pdf/xl.json
/data7/test/ThinkPad Design - Spirit and Essence.pdf/part.1
root@storage3:/ # exit

We can also see what Minio configuration file /usr/local/etc/minio/config.json has been generated.

host # jexec storage1
root@storage1:/ # cat /usr/local/etc/minio/config.json 
{
        "version": "22",
        "credential": {
                "accessKey": "alibaba",
                "secretKey": "0P3NS3S4M3"
        },
        "region": "",
        "browser": "on",
        "domain": "",
        "storageclass": {
                "standard": "",
                "rrs": ""
        },
        "notify": {
                "amqp": {
                        "1": {
                                "enable": false,
                                "url": "",
                                "exchange": "",
                                "routingKey": "",
                                "exchangeType": "",
                                "deliveryMode": 0,
                                "mandatory": false,
                                "immediate": false,
                                "durable": false,
                                "internal": false,
                                "noWait": false,
                                "autoDeleted": false
                        }
                },
                "elasticsearch": {
                        "1": {
                                "enable": false,
                                "format": "",
                                "url": "",
                                "index": ""
                        }
                },
                "kafka": {
                        "1": {
                                "enable": false,
                                "brokers": null,
                                "topic": ""
                        }
                },
                "mqtt": {
                        "1": {
                                "enable": false,
                                "broker": "",
                                "topic": "",
                                "qos": 0,
                                "clientId": "",
                                "username": "",
                                "password": "",
                                "reconnectInterval": 0,
                                "keepAliveInterval": 0
                        }
                },
                "mysql": {
                        "1": {
                                "enable": false,
                                "format": "",
                                "dsnString": "",
                                "table": "",
                                "host": "",
                                "port": "",
                                "user": "",
                                "password": "",
                                "database": ""
                        }
                },
                "nats": {
                        "1": {
                                "enable": false,
                                "address": "",
                                "subject": "",
                                "username": "",
                                "password": "",
                                "token": "",
                                "secure": false,
                                "pingInterval": 0,
                                "streaming": {
                                        "enable": false,
                                        "clusterID": "",
                                        "clientID": "",
                                        "async": false,
                                        "maxPubAcksInflight": 0
                                }
                        }
                },
                "postgresql": {
                        "1": {
                                "enable": false,
                                "format": "",
                                "connectionString": "",
                                "table": "",
                                "host": "",
                                "port": "",
                                "user": "",
                                "password": "",
                                "database": ""
                        }
                },
                "redis": {
                        "1": {
                                "enable": false,
                                "format": "",
                                "address": "",
                                "password": "",
                                "key": ""
                        }
                },
                "webhook": {
                        "1": {
                                "enable": false,
                                "endpoint": ""
                        }
                }
        }

S3FS

We can also mount that test bucket from out distributed Minio object storage cluster as filesystem using the S3FS project. Lets add s3fs package and mount our bucket.

host # pkg install -y fusefs-s3fs

Now we will configure password for our bucket.

host # echo test:alibaba:0P3NS3S4M3 > /root/.passwd-s3fs
host # chmod 600 /root/.passwd-s3fs
host # cat /root/.passwd-s3fs 
test:alibaba:0P3NS3S4M3

Now lets do the actual mount.

host # mkdir /tmp/test
host # s3fs \
  -o allow_other \
  -o use_path_request_style \
  -o url=http://192.168.43.101:9000 \
  -o passwd_file=/root/.passwd-s3fs \
  test /tmp/test

The file ThinkPad Design – Spirit and Essence.pdf that we put through web interface should be here.

host # exa -l /tmp/test
.--------- 10M root 2018-04-16 14:15 ThinkPad Design - Spirit and Essence.pdf

host # file /tmp/test/ThinkPad\ Design\ -\ Spirit\ and\ Essence.pdf 
/tmp/test/ThinkPad Design - Spirit and Essence.pdf: PDF document, version 1.4

host # stat /tmp/test/ThinkPad\ Design\ -\ Spirit\ and\ Essence.pdf
3976265496 2 ---------- 1 root wheel 0 10416953 "Jan  1 01:00:00 1970" "Apr 16 14:35:35 2018" "Jan  1 01:00:00 1970" "Jan  1 00:59:59 1970" 4096 20346 0 /tmp/test/ThinkPad Design - Spirit and Essence.pdf

We can now upload other file into that bucket using s3fs mount.

host # cp -v /home/vermaden/On\ the\ Shortness\ of\ Life\ -\ Lucius\ Seneca.pdf /tmp/test
/home/vermaden/On the Shortness of Life - Lucius Seneca.pdf -> /tmp/test/On the Shortness of Life - Lucius Seneca.pdf

host # file /tmp/test/On\ the\ Shortness\ of\ Life\ -\ Lucius\ Seneca.pdf 
On the Shortness of Life - Lucius Seneca.pdf: PDF document, version 1.4

We can also verify that our file put through s3fs is visible on the web interface.

minio-browser-13-s3fs-upload

Real Hardware

Now, as we have working Proof of Concept for the distributed Minio setup how about putting it on a real hardware for real storage purposes? I would setup a 16 node Minio distributed server on a Supermicro SSG-5018D8-AR12L hardware. Supermicro even suggests using that kind of servers for object storage, here is their white paper on that topic – Object Storage Solution for Data Archive using Supermicro SSG-5018D8-AR12L and OpenIO SDS – but they use OpenIO not Minio for distributed object storage solution.

This server features the Supermicro X10SDV-7TP4F motherboard. This is important as this motherboard officially supports FreeBSD 11.x operating system on their Supermicro OS Compatibility page.

Motherboard specification has these features.

 1 x Intel Xeon D-1537 8-Core / 16-Threads TDP 35W
 4 x UDIMM for up to 128GB ECC RDIMM DDR4 2133MHz
12 x 3.5" SAS2/SATA3 Hot-Swap HDD Bays
 4 x 2.5" Cold-Swap HDD Bays
 1 x Controller Intel SoC for 4 SATA3 (6Gbps) Ports
 1 x Controller Broadcom 2116 for 16 SATA3 (6Gbps) Ports
 1 x Expansion Slot PCI-E 3.0 x8 
 1 x Expansion Slot M.2 PCIe 3.0 x4
 1 x Expansion Slot Mini-PCIe w/ mSATA Support
 2 x 10G SFP+ Port
 2 x 1GbE LAN Port
 2 x External USB 3.0 Port
 1 x Interlal USB 2.0 Port
 2 x 400W High-Rfficiency Redundant Power Supplies

You can configure your own and get approximated price using the Thinkmate site from here:
https://www.thinkmate.com/system/superstorage-server-5018d8-ar12l

I would add this components to the basic setup:

 4 x UDIMM FULL 128 GB ECC RDIMM DDR4
 2 x 240GB Micron 5100 MAX 2.5" SATA 6.0Gb/s SSD
 2 x 7.68TB Micron 5200 ECO Series 2.5" SATA 6.0Gb/s SSD
12 x 12TB SATA 6.0Gb/s 7200RPM 3.5" Hitachi Ultrastarβ„’ He12
 3 x SanDisk Cruzer Fit 32GB USB 3.0

Now, I will use the 3 x SanDisk Cruzer Fit 32GB USB 3.0 disks to install FreeBSD as a ZFS root/boot pool with mirror + spare on these disks. We do not need performance here.

Then, the 12 x 12TB SATA 6.0Gb/s 7200RPM 3.5″ Hitachi Ultrastarβ„’ He12 drives will be used as RAIDZ (RAID5 equivalent in ZFS without the write hole) for the Minio data, wich 11 + 1 setup, which means 11 drives for data and 1 drive for parity. As we can lose HALF of the Minio servers I would not waste 12 TB drive for spare here. Then, I would use 2 x 240GB Micron 5100 MAX 2.5″ SATA 6.0Gb/s SSD in mirror for the ZFS ZIL (ZFS Intent Log) to accelerate writes and 2 x 7.68TB Micron 5200 ECO Series 2.5″ SATA 6.0Gb/s SSD for the ZFS read cache (L2ARC).

The network would be setup on 2 x 10G SFP+ Port with LACP as lagg0 interface so each server would have 20 Gbit connectivity. This will give us a total of 320 Gbit theoretical network throughput.

This setup would give as 132 TB ZFS pool space with 15 TB for read cache and 240 GB for writes for single 1U server. Making the calculations this will give as 2112 TB (more then 2 PB) of space for Minio data.

With Minio algorithm for data redundancy we will have about 1 PB of usable storage space in our 16U Object Storage FreeBSD Appliance.

Not bad for my taste πŸ™‚

UPDATE 1

The Distributed Object Storage with Minio on FreeBSD article was included in the BSD Now 246 – Disclosure episode.

Thanks for mentioning!

EOF

FreeBSD Network Management with network.sh Script

When You use only one connection on FreeBSD, then the best practice is to just put its whole configuration into the /etc/rc.conf file, for example typical server redundant connection would look like that one below.

cloned_interfaces="lagg0"
ifconfig_igb0="-lro -tso -vlanhwtag mtu 9000 up"
ifconfig_igb1="-lro -tso -vlanhwtag mtu 9000 up"
ifconfig_lagg0="laggproto lacp laggport igb0 laggport igb1 up"
ifconfig_lagg0_alias0="inet 10.254.17.2/24"

If You must use more then one connection and You often switch between them, sometimes several times a day, then using the main FreeBSD’s config file is not the most convenient way for such operations.

For laptops where You often switch between WWAN (usually 3G connection) and WLAN (typical WiFi connection) and even LAN cable.

You can of course use graphical NetworkMgr from GhostBSD project which is described as “Python GTK3 network manager for FreeBSD, GhostBSD, TrueOS and DragonFlyBSD. NetworkMgr support both netif and OpenRC network” citing the project site – https://github.com/GhostBSD/networkmgr – it is also available in FreeBSD Ports and as package – net-mgmt/networkmgr.

GhostBSD-networkmgr

What I miss in NetworkMgr is the WWAN connection management, DNS management, optional random MAC generation and network shares unmount at disconnect from network. With my solution – network.sh – you still need to edit /etc/wpa_supplicant.conf and /etc/ppp/ppp.conf files by hand so it’s also not a perfect solution for typical desktop usage, but you do not edit these files every day.

As I use WWAN, WLAN and LAN connections on my laptop depends on the location I wrote a script to automate this connection management in a deterministic and convenient way, at least for me.

It can also set DNS to some safe/nologging providers or even a random safe DNS and generate legitimate MAC address for both LAN and WLAN if needed, even with real OUI first three octets if You also have additional network.sh.oui.txt file with them inside.

Here is the network.sh script help message.

% network.sh help
USAGE:
 network.sh TYPE [OPTIONS]

TYPES:
 lan
 wlan
 wwan
 dns

OPTIONS:
 start
 start SSID|PROFILE
 stop
 example

EXAMPLES:
 network.sh lan start
 network.sh lan restart
 network.sh wlan start
 network.sh wlan start HOME-NETWORK-SSID
 network.sh wwan example
 network.sh dns onic
 network.sh dns udns
 network.sh dns random
 network.sh doas
 network.sh sudo

If You run network.sh with appreciate arguments to start network connection it will display on the screen what commands it would run to achieve that. It also makes use of sudo(8) or doas(1) assuming that You are in the network group. To add yourself into the network group type this command below.

# pw groupmod network -m yourself

The network.sh doas command will print what rights it needs to work without root privileges, same for network.sh sudo command, an example below.

% network.sh doas
 # pw groupmod network -m YOURUSERNAME
 # cat /usr/local/etc/doas.conf
 permit nopass :network as root cmd /bin/cat args /etc/ppp/ppp.conf
 permit nopass :network as root cmd /etc/rc.d/netif args onerestart
 permit nopass :network as root cmd dhclient
 permit nopass :network as root cmd ifconfig
 permit nopass :network as root cmd killall args -9 dhclient
 permit nopass :network as root cmd killall args -9 ppp
 permit nopass :network as root cmd killall args -9 wpa_supplicant
 permit nopass :network as root cmd ppp
 permit nopass :network as root cmd tee args -a /etc/resolv.conf
 permit nopass :network as root cmd tee args /etc/resolv.conf
 permit nopass :network as root cmd umount
 permit nopass :network as root cmd wpa_supplicant

The network.sh script does not edit /usr/local/etc/doas.conf or /usr/local/etc/sudoers files, You have to put these lines there by yourself. An example doas setup for network.sh script is below.

# pkg install -y doas

# cat >> /usr/local/etc/doas.conf << __EOF
permit nopass :network as root cmd /bin/cat args /etc/ppp/ppp.conf
permit nopass :network as root cmd /etc/rc.d/netif args onerestart
permit nopass :network as root cmd dhclient
permit nopass :network as root cmd ifconfig
permit nopass :network as root cmd killall args -9 dhclient
permit nopass :network as root cmd killall args -9 ppp
permit nopass :network as root cmd killall args -9 wpa_supplicant
permit nopass :network as root cmd ppp
permit nopass :network as root cmd tee args -a /etc/resolv.conf
permit nopass :network as root cmd tee args /etc/resolv.conf
permit nopass :network as root cmd umount
permit nopass :network as root cmd wpa_supplicant
__EOF
# 

# pw groupmod network -m yourself

The network.sh script upon disconnect would also forcefully unmount all network shares.

The idea is that it does only one connection type at a time, When You type network.sh lan start and then type network.sh wlan start, then it will reset entire FreeBSD network stack to defaults (to settings that are in /etc/rc.conf file) and then connect to WiFi in a ‘clean network environment’ as I could say. As I use 3 different methods of connecting to various networks I do not have any network settings in theΒ /etc/rc.conf file, but You may prefer for example to have DHCP for local LAN enabled if that is more convenient for You.

The settings are on the beginning of the network.sh script, You should modify them to your needs and hardware that You own.

# SETTINGS
LAN_IF=em0
LAN_RANDOM_MAC=0
WLAN_IF=wlan0
WLAN_PH=iwn0
WLAN_RANDOM_MAC=1
WWAN_IF=tun0
WWAN_PROFILE=WWAN
NAME=$( basename ${0} )
NETFS="nfs,smbfs,fusefs.sshfs"
TIMEOUT=16
DELAY=0.5
SUDO=0
DOAS=1

You can specify other NETFS filesystems that You want to forcefully unmount during network stop or set different physical WLAN adapter (WLAN_PH option), like ath0 for Atheros chips. similar for LAN interface which also defaults to Intel based network card with em0 driver (LAN_IF option).

If you want to disable random MAC address for LAN with LAN_RANDOM_MAC=0 and enable generation of random MAC address for WiFi networks with WLAN_RANDOM_MAC=1 option.

You should also decide if You want to use sudo (SUDO option) or doas (DOAS option).

Here are network.shΒ script and optional network.sh.oui.txt OUI data.

After downloading please rename them accordingly (WordPress limitation).

% mv network-sh.key         network.sh 
% mv network-sh-oui-txt.key network.sh.oui.txt

Here is example of all network connections stop.

% network.sh stop
doas killall -9 wpa_supplicant
doas killall -9 ppp
doas killall -9 dhclient
doas ifconfig wlan0 destroy
doas ifconfig em0 down
echo | doas tee /etc/resolv.conf
doas /etc/rc.d/netif onerestart
%

Here is example of WWAN network connection start.

% network.sh wwan start
doas killall -9 wpa_supplicant
doas killall -9 ppp
doas killall -9 dhclient
doas ifconfig wlan0 destroy
doas ifconfig em0 down
echo | doas tee /etc/resolv.conf
doas /etc/rc.d/netif onerestart
doas ppp -ddial WWAN
%

Here is example od DNS change.

% network.sh dns onic
echo | doas tee /etc/resolv.conf
echo 'nameserver 87.98.175.85' | doas tee -a /etc/resolv.conf
echo 'nameserver 193.183.98.66' | doas tee -a /etc/resolv.conf

If You have any problems with the network.sh script then let me know, I will try to fix them ASAP.

If You are more into OpenBSD then FreeBSD then Vincent Delft wrote nmctlNetwork Manager Control tool for OpenBSD – available here – http://vincentdelft.be/post/post_20171023.

Ther is also another OpenBSD project by Aaron Poffenberger for network management – netctl –Β cli network-location manager for OpenBSD –Β available here – https://github.com/akpoff/netctl.

UPDATE 1 – Connect to Open/Unsecured WiFi Network

Recently when I was attending the Salt workshop during NLUUG Autumn Conference 2018 at Utrecht, Nederlands I wanted to connect to open unsecured WiFi network called 'Utrecht Hotel'. My phone of course attached to it instantly but on the other hand FreeBSD was not able to connect to it. As it turns out if you want to enable wpa_supplicant(8) to connect to open unsecured network a separate /etc/wpa_supplicant.conf option is needed (on option for all open unsecured
networks – no need to create such rule for each open/unsecured network).

Its these lines in the /etc/wpa_supplicant.conf file:

% grep -C 2 key_mgmt=NONE /etc/wpa_supplicant.conf

network={
        key_mgmt=NONE
        priority=0
}

I also modified the network.sh to contain that information in the examples section and also made little fix to always reset the previously set/forced SSID during earlier usage.

# ifconfig wlan0 ssid -

Now the network.sh should be even more pleasant to use.

UPDATE 2 – Openbox Integration

In on of the FreeBSD Desktop series articles I described how to setup Openbox window manager – FreeBSD Desktop – Part 12 – Configuration – Openbox – available here.

Below is an example of integration of that network.sh script with Openbox window manager.

network.sh.openbox.menu.jpg

… and here is the code used in the ~/.config/openbox/menu.xml file.

network.sh.openbox.menu.code

EOF