Ever heard about Bareos? Probably heard about Bacula. Read what is the difference here – Why Bareos forked from Bacula?
If you are interested in more enterprise backup solution then check IBM TSM (Spectrum Protect) on Veritas Cluster Server article.
Bareos (Backup Archiving Recovery Open Sourced) is a network based open source backup solution. It is 100% open source fork of the backup project from bacula.org site. The fork is in development since late 2010 and it has a lot of new features. The source is published on github and licensed under AGPLv3 license. Bareos supports ‘Always Incremental backup which is interesting especially for users with big data. The time and network capacity consuming full backups only have to be taken once. Bareos comes with WebUI for administration tasks and restore file browser. Bareos can backup data to disk and to tape drives as well as tape libraries. It supports compression and encryption both hardware-based (like on LTO tape drives) and software-based. You can also get professional services and support from Bareos as well as Bareos subscription service that provides you access to special quality assured installation packages.
I started my sysadmin job with backup system as one of the new responsibilities, so it will be like going back to the roots. As I look on the ‘backup’ market it is more and more popular – especially in cloud oriented environments – to implement various levels of protection like GOLD, SILVER and BRONZE for example. They of course have different retention times, number of backups kept, different RTO and RPO. Below is a example implementation of BRONZE level backups in Bareos. I used 3 groups of A, B and C with FULL backup starting on DAY 0 (A group), DAY 1 (B group) and DAY 2 (C group).
This way you still have FULL backups quite often and with 3 groups you can balance the network load. I for the days that we will not be doing FULL backups we will be doing DIFFERENTIAL backups. People often confuse them with INCREMENTAL backups. The difference is that DIFFERENTIAL backups are always against FULL backup, so its always ‘one level of combining’. INCREMENTAL ones are done against last done backup TYPE, so its possible to have 100+ levels of combining against 99 earlier INCREMENTAL backups and the 1 FULL backup. That is why I prefer DIFFERENTIAL ones here, faster recovery. That is all backups is about generally, recovery, some people/companies tend to forget that.
The implementation of BRONZE in these three groups is not perfect, but ‘does the job’. I also made ‘simulation’ how these group will overlap at the end/beginning of the month, here is the result.
Not bad for my taste.
Today I will show you how to install and configure Bareos Server based on FreeBSD operating system. It will be the most simplified setup with all services on single machine:
- bareos-dir
- bareos-sd
- bareos-webui
- bareos-fd
I also assume that in order to provide storage space for the backup data itself You would mount resources from external NFS shares.
To get in touch with Bareos terminology and technology check their great Manual in HTML or PDF version depending which format You prefer for reading documentation. Also their FAQ provides a lot of needed answers.
Also this diagram may be useful for You to get some grip into the Bareos world.
System
As every system needs to have its name we will use latin word closest to backup here – replica – for our FreeBSD system hostname. The install would be generally the same as in the FreeBSD Desktop – Part 2 – Install article. Here is our installed FreeBSD system with login prompt.
Sorry couldn’t resist π
Here are 3 most important configuration files after some time in vi(1)
with them.
root@replica:~ # cat /etc/rc.conf # NETWORK hostname=replica.backup.org ifconfig_em0="inet 10.0.10.30/24 up" defaultrouter="10.0.10.1" # DAEMONS zfs_enable=YES sshd_enable=YES nfs_client_enable=YES syslogd_flags="-ss" sendmail_enable=NONE # OTHER clear_tmp_enable=YES dumpdev=NO # BAREOS # postgresql_enable=YES # postgresql_class=pgsql # bareos_dir_enable=YES # bareos_sd_enable=YES # bareos_fd_enable=YES # php_fpm_enable=YES # nginx_enable=YES
As You can see all ‘core’ services for Bareos are currently disabled on purpose. We will enable them later.
Parameters and modules to be set at boot.
root@replica:~ # cat /boot/loader.conf # BOOT OPTIONS autoboot_delay=2 kern.geom.label.disk_ident.enable=0 kern.geom.label.gptid.enable=0 # MODULES zfs_load=YES # IPC kern.ipc.shmseg=1024 kern.ipc.shmmni=1024 kern.ipc.shmseg=1024
Parameters to be set at runtime.
root@replica:~ # cat /etc/sysctl.conf # SECURITY security.bsd.see_other_uids=0 security.bsd.see_other_gids=0 security.bsd.unprivileged_read_msgbuf=0 security.bsd.unprivileged_proc_debug=0 security.bsd.stack_guard_page=1 kern.randompid=9100 # ZFS vfs.zfs.min_auto_ashift=12 # DISABLE ANNOYING THINGS kern.coredump=0 hw.syscons.bell=0 kern.vt.enable_bell=0 # IPC kern.ipc.shmall=524288 kern.ipc.maxsockbuf=5242880 kern.ipc.shm_allow_removed=1
After install we will disable the /zroot
mounting.
root@replica:/ # zfs set mountpoint=none zroot
As we have sendmail(8)
disabled we will need to take care of its queue.
root@replica:~ # cat > /etc/cron.d/sendmail-clean-clientmqueue << __EOF # CLEAN SENDMAIL 0 * * * * root /bin/rm -r -f /var/spool/clientmqueue/* __EOF
Assuming the NFS servers configured in the /etc/hosts
file the ‘complete’ /etc/hosts
file would look like that.
root@replica:~ # grep '^[^#]' /etc/hosts ::1 localhost localhost.my.domain 127.0.0.1 localhost localhost.my.domain 10.0.10.40 replica.backup.org replica 10.0.10.50 nfs-pri.backup.org nfs-pri 10.0.20.50 nfs-sec.backup.org nfs-sec
Lets verify outside world connectivity – needed for adding the Bareos packages.
root@replica:~ # nc -v bareos.org 443 Connection to bareos.org 443 port [tcp/https] succeeded! ^C root@replica:~ #
Packages
As we want the latest packages we will modify the /etc/pkg/FreeBSD.conf
– the pkg(8)
repository file for the latest packages.
root@replica:~ # grep '^[^#]' /etc/pkg/FreeBSD.conf FreeBSD: { url: "pkg+http://pkg.FreeBSD.org/${ABI}/quarterly", mirror_type: "srv", signature_type: "fingerprints", fingerprints: "/usr/share/keys/pkg", enabled: yes } root@replica:~ # sed -i '' s/quarterly/latest/g /etc/pkg/FreeBSD.conf root@replica:~ # grep '^[^#]' /etc/pkg/FreeBSD.conf FreeBSD: { url: "pkg+http://pkg.FreeBSD.org/${ABI}/latest", mirror_type: "srv", signature_type: "fingerprints", fingerprints: "/usr/share/keys/pkg", enabled: yes }
We will use Bareos packages from pkg(8)
as they are available, no need to waste time and power on compilation.
root@replica:~ # pkg search bareos The package management tool is not yet installed on your system. Do you want to fetch and install it now? [y/N]: y (...) bareos-bat-16.2.7 Backup archiving recovery open sourced (GUI) bareos-client-16.2.7 Backup archiving recovery open sourced (client) bareos-client-static-16.2.7 Backup archiving recovery open sourced (static client) bareos-docs-16.2.7 Bareos document set (PDF) bareos-server-16.2.7 Backup archiving recovery open sourced (server) bareos-traymonitor-16.2.7 Backup archiving recovery open sourced (traymonitor) bareos-webui-16.2.7 PHP-Frontend to manage Bareos over the web
Now we will install Bareos along with all needed components for its environment.
root@replica:~ # pkg install \ bareos-client bareos-server bareos-webui postgresql95-server nginx \ php56 php56-xml php56-session php56-simplexml php56-gd php56-ctype \ php56-mbstring php56-zlib php56-tokenizer php56-iconv php56-mcrypt \ php56-pear-DB_ldap php56-zip php56-dom php56-sqlite3 php56-gettext \ php56-curl php56-json php56-opcache php56-wddx php56-hash php56-soap
The bareos
, pgsql
and www
users have been added by pkg(8)
along with their packages.
root@replica:~ # id bareos uid=997(bareos) gid=997(bareos) groups=997(bareos) root@replica:~ # id pgsql uid=70(pgsql) gid=70(pgsql) groups=70(pgsql) root@replica:~ # id www uid=80(www) gid=80(www) groups=80(www)
PostgreSQL
First we will setup the PostgreSQL database.
We will add separate pgsql
login class for PostgreSQL database user.
root@replica:~ # cat >> /etc/login.conf << __EOF # PostgreSQL pgsql:\ :lang=en_US.UTF-8:\ :setenv=LC_COLLATE=C:\ :tc=default: __EOF
This is one of the rare occasions when I would appreciate the -p
flag from the AIX grep
command to display whole paragraph π
root@replica:~ # grep -B 1 -A 3 pgsql /etc/login.conf # PostgreSQL pgsql:\ :lang=en_US.UTF-8:\ :setenv=LC_COLLATE=C:\ :tc=default:
Lets reload the login database.
root@replica:~ # cap_mkdb /etc/login.conf
Here are PostgreSQL rc(8)
startup script ‘options’ that can be set in /etc/rc.conf
file.
root@replica:~ # grep '# postgresql' /usr/local/etc/rc.d/postgresql # postgresql_enable="YES" # postgresql_data="/usr/local/pgsql/data" # postgresql_flags="-w -s -m fast" # postgresql_initdb_flags="--encoding=utf-8 --lc-collate=C" # postgresql_class="default" # postgresql_profiles=""
We only need postgresql_enable
and postgresql_class
to be set.
We will enable them now in the /etc/rc.conf
file.
root@replica:~ # grep -A 10 BAREOS /etc/rc.conf # BAREOS postgresql_enable=YES postgresql_class=pgsql # bareos_dir_enable=YES # bareos_sd_enable=YES # bareos_fd_enable=YES # php_fpm_enable=YES # nginx_enable=YES
We will now init the PostgreSQL database for Bareos.
root@replica:~ # /usr/local/etc/rc.d/postgresql initdb The files belonging to this database system will be owned by user "pgsql". This user must also own the server process. The database cluster will be initialized with locales COLLATE: C CTYPE: en_US.UTF-8 MESSAGES: en_US.UTF-8 MONETARY: en_US.UTF-8 NUMERIC: en_US.UTF-8 TIME: en_US.UTF-8 The default text search configuration will be set to "english". Data page checksums are disabled. creating directory /usr/local/pgsql/data ... ok creating subdirectories ... ok selecting default max_connections ... 100 selecting default shared_buffers ... 128MB selecting dynamic shared memory implementation ... posix creating configuration files ... ok creating template1 database in /usr/local/pgsql/data/base/1 ... ok initializing pg_authid ... ok initializing dependencies ... ok creating system views ... ok loading system objects' descriptions ... ok creating collations ... ok creating conversions ... ok creating dictionaries ... ok setting privileges on built-in objects ... ok creating information schema ... ok loading PL/pgSQL server-side language ... ok vacuuming database template1 ... ok copying template1 to template0 ... ok copying template1 to postgres ... ok syncing data to disk ... ok WARNING: enabling "trust" authentication for local connections You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb. Success. You can now start the database server using: /usr/local/bin/pg_ctl -D /usr/local/pgsql/data -l logfile start
… and start it.
root@replica:~ # /usr/local/etc/rc.d/postgresql start LOG: ending log output to stderr HINT: Future log output will go to log destination "syslog".
We will now take care of the Bareos server configuration. There are a lot *.sample
files that we do not need. We also need to take care about permissions.
root@replica:~ # chown -R bareos:bareos /usr/local/etc/bareos root@replica:~ # find /usr/local/etc/bareos -type f -exec chmod 640 {} ';' root@replica:~ # find /usr/local/etc/bareos -type d -exec chmod 750 {} ';' root@replica:~ # find /usr/local/etc/bareos -name \*\.sample -delete
We also need to change permissions for the /var/run and /var/db directories for Bareos.
root@replica:~ # chown -R bareos:bareos /var/db/bareos root@replica:~ # chown -R bareos:bareos /var/run/bareos
For the ‘trace’ of our changes we will keep a copy of the original configuration to track what we have changed in the process of configuring our Bareos environment.
root@replica:~ # cp -a /usr/local/etc/bareos /usr/local/etc/bareos.ORG
Now, we would configure the Bareos Catalog in the /usr/local/etc/bareos.ORG/bareos-dir.d/catalog/MyCatalog.conf
file, here are its contents after our modifications.
root@replica:~ # cat /usr/local/etc/bareos/bareos-dir.d/catalog/MyCatalog.conf Catalog { Name = MyCatalog dbdriver = "postgresql" dbname = "bareos" dbuser = "bareos" dbpassword = "BAREOS-DATABASE-PASSWORD" }
Lets make sure that pgsql
and www
users are in the bareos
group, to read its configuration files.
root@replica:~ # pw groupmod bareos -m pgsql root@replica:~ # id pgsql uid=70(pgsql) gid=70(pgsql) groups=70(pgsql),997(bareos) root@replica:~ # pw groupmod bareos -m www root@replica:~ # id www uid=80(www) gid=80(www) groups=80(www),997(bareos)
Now, we will prepare the PostgreSQL database for out Bareos instance. We will use scripts provided by the Bareos package from the /usr/local/lib/bareos/scripts
path.
root@replica:~ # su - pgsql $ whoami pgsql $ /usr/local/lib/bareos/scripts/create_bareos_database Creating postgresql database CREATE DATABASE ALTER DATABASE Database encoding OK Creating of bareos database succeeded. $ /usr/local/lib/bareos/scripts/make_bareos_tables Making postgresql tables CREATE TABLE ALTER TABLE CREATE INDEX CREATE TABLE ALTER TABLE CREATE INDEX CREATE TABLE CREATE INDEX CREATE INDEX CREATE TABLE CREATE INDEX CREATE TABLE CREATE INDEX CREATE TABLE CREATE INDEX CREATE TABLE CREATE TABLE CREATE INDEX CREATE TABLE CREATE INDEX CREATE TABLE CREATE INDEX CREATE INDEX CREATE TABLE CREATE TABLE CREATE TABLE CREATE TABLE CREATE INDEX CREATE TABLE CREATE INDEX CREATE TABLE CREATE INDEX CREATE TABLE CREATE TABLE CREATE TABLE CREATE INDEX CREATE TABLE CREATE TABLE CREATE INDEX CREATE TABLE CREATE INDEX CREATE TABLE CREATE TABLE CREATE TABLE CREATE TABLE CREATE TABLE CREATE TABLE CREATE TABLE CREATE TABLE INSERT 0 1 INSERT 0 1 INSERT 0 1 INSERT 0 1 INSERT 0 1 INSERT 0 1 INSERT 0 1 INSERT 0 1 INSERT 0 1 INSERT 0 1 INSERT 0 1 INSERT 0 1 INSERT 0 1 INSERT 0 1 INSERT 0 1 INSERT 0 1 INSERT 0 1 INSERT 0 1 INSERT 0 1 INSERT 0 1 INSERT 0 1 INSERT 0 1 INSERT 0 1 INSERT 0 1 INSERT 0 1 INSERT 0 1 DELETE 0 INSERT 0 1 Creation of Bareos PostgreSQL tables succeeded. $ /usr/local/lib/bareos/scripts/grant_bareos_privileges Granting postgresql tables CREATE ROLE GRANT GRANT GRANT GRANT GRANT GRANT GRANT GRANT GRANT GRANT GRANT GRANT GRANT GRANT GRANT GRANT GRANT GRANT GRANT GRANT GRANT GRANT GRANT GRANT GRANT GRANT GRANT GRANT GRANT GRANT GRANT GRANT GRANT GRANT GRANT GRANT GRANT GRANT GRANT GRANT GRANT GRANT GRANT GRANT GRANT GRANT GRANT Privileges for user bareos granted ON database bareos.
We can now verify that we have the needed database created.
root@replica:~ # su -m bareos -c 'psql -l' List of databases Name | Owner | Encoding | Collate | Ctype | Access privileges -----------+-------+-----------+---------+-------------+------------------- bareos | pgsql | SQL_ASCII | C | C | postgres | pgsql | UTF8 | C | en_US.UTF-8 | template0 | pgsql | UTF8 | C | en_US.UTF-8 | =c/pgsql + | | | | | pgsql=CTc/pgsql template1 | pgsql | UTF8 | C | en_US.UTF-8 | =c/pgsql + | | | | | pgsql=CTc/pgsql (4 rows)
We will also add housekeeping script for PostgreSQL database and put it into crontab(1)
.
root@replica:~ # su - pgsql $ whoami pgsql $ cat > /usr/local/pgsql/vacuum.sh /dev/null /usr/local/bin/reindexdb -a 1> /dev/null 2> /dev/null /usr/local/bin/reindexdb -s 1> /dev/null 2> /dev/null __EOF $ chmod +x /usr/local/pgsql/vacuum.sh $ cat /usr/local/pgsql/vacuum.sh #! /bin/sh /usr/local/bin/vacuumdb -a -z 1> /dev/null 2> /dev/null /usr/local/bin/reindexdb -a 1> /dev/null 2> /dev/null /usr/local/bin/reindexdb -s 1> /dev/null 2> /dev/null $ crontab -e $ exit root@replica:~ # cat /var/cron/tabs/pgsql # DO NOT EDIT THIS FILE - edit the master and reinstall. # (/tmp/crontab.Be9j9VVCUa installed on Thu Apr 26 21:45:04 2018) # (Cron version -- $FreeBSD$) 0 0 * * * /usr/local/pgsql/vacuum.sh root@replica:~ # su -m pgsql -c 'crontab -l' 0 0 * * * /usr/local/pgsql/vacuum.sh
Storage
I assume that the primary storage would be mounted in the /bareos
directory from one NFS server while Disaster Recovery site would be mounted as /bareos-dr
from another NFS server. Below is example NFS configuration of these mount points.
root@replica:~ # mkdir /bareos /bareos-dr root@replica:~ # mount -t nfs nfs-pri.backup.org:/export/bareos on /bareos (nfs, noatime) nfs-sec.backup.org:/export/bareos-dr on /bareos-dr (nfs, noatime) root@replica:~ # cat >> /etc/fstab << __EOF #DEV #MNT #FS #OPTS #DP nfs-pri.backup.org:/export/bareos /bareos nfs rw,noatime,rsize=1048576,wsize=1048576,readahead=4,soft,intr 0 0 nfs-sec.backup.org:/export/bareos-dr /bareos-dr nfs rw,noatime,rsize=1048576,wsize=1048576,readahead=4,soft,intr 0 0 __EOF root@replica:~ # mkdir -p /bareos/bootstrap root@replica:~ # mkdir -p /bareos/restore root@replica:~ # mkdir -p /bareos/storage/FileStorage root@replica:~ # mkdir -p /bareos-dr/bootstrap root@replica:~ # mkdir -p /bareos-dr/restore root@replica:~ # mkdir -p /bareos-dr/storage/FileStorage root@replica:~ # chown -R bareos:bareos /bareos /bareos-dr root@replica:~ # find /bareos /bareos-dr -ls | column -t 69194 1 drwxr-xr-x 5 bareos bareos 5 Apr 27 00:42 /bareos 72239 1 drwxr-xr-x 2 bareos bareos 2 Apr 27 00:42 /bareos/restore 72240 1 drwxr-xr-x 3 bareos bareos 3 Apr 27 00:42 /bareos/storage 72241 1 drwxr-xr-x 2 bareos bareos 2 Apr 27 00:42 /bareos/storage/FileStorage 72238 1 drwxr-xr-x 2 bareos bareos 2 Apr 27 00:42 /bareos/bootstrap 69195 1 drwxr-xr-x 5 bareos bareos 5 Apr 27 00:43 /bareos-dr 72254 1 drwxr-xr-x 3 bareos bareos 3 Apr 27 00:43 /bareos-dr/storage 72255 1 drwxr-xr-x 2 bareos bareos 2 Apr 27 00:43 /bareos-dr/storage/FileStorage 72253 1 drwxr-xr-x 2 bareos bareos 2 Apr 27 00:42 /bareos-dr/restore 72252 1 drwxr-xr-x 2 bareos bareos 2 Apr 27 00:42 /bareos-dr/bootstrap
Bareos
As we already used BAREOS-DATABASE-PASSWORD
for the bareos
user on PostgreSQL’s Bareos database we will use these passwords for the remaining parts of the Bareos subsystems. I think that these passwords are self explaining for what Bareos components they are π
- BAREOS-DATABASE-PASSWORD
- BAREOS-DIR-PASSWORD
- BAREOS-SD-PASSWORD
- BAREOS-FD-PASSWORD
- BAREOS-MON-PASSWORD
- ADMIN-PASSWORD
We will now configure all these Bareos subsystems.
We already modified the MyCatalog.conf
file, here are its contents.
root@replica:~ # cat /usr/local/etc/bareos/bareos-dir.d/catalog/MyCatalog.conf Catalog { Name = MyCatalog dbdriver = "postgresql" dbname = "bareos" dbuser = "bareos" dbpassword = "BAREOS-DATABASE-PASSWORD" }
Contents of the /usr/local/etc/bareos/bconsole.d/bconsole.conf
file after modifications.
root@replica:~ # cat /usr/local/etc/bareos/bconsole.d/bconsole.conf # # Bareos User Agent (or Console) Configuration File # Director { Name = replica.backup.org address = localhost Password = "BAREOS-DIR-PASSWORD" Description = "Bareos Console credentials for local Director" }
Contents of the /usr/local/etc/bareos/bareos-dir.d/director/bareos-dir.conf
file after modifications.
root@replica:~ # cat /usr/local/etc/bareos/bareos-dir.d/director/bareos-dir.conf Director { Name = replica.backup.org QueryFile = "/usr/local/lib/bareos/scripts/query.sql" Maximum Concurrent Jobs = 100 Password = "BAREOS-DIR-PASSWORD" Messages = Daemon Auditing = yes # Enable the Heartbeat if you experience connection losses # (eg. because of your router or firewall configuration). # Additionally the Heartbeat can be enabled in bareos-sd and bareos-fd. # # Heartbeat Interval = 1 min # remove comment in next line to load dynamic backends from specified directory # Backend Directory = /usr/local/lib # remove comment from "Plugin Directory" to load plugins from specified directory. # if "Plugin Names" is defined, only the specified plugins will be loaded, # otherwise all director plugins (*-dir.so) from the "Plugin Directory". # # Plugin Directory = /usr/local/lib/bareos/plugins # Plugin Names = "" }
Contents of the /usr/local/etc/bareos/bareos-dir.d/job/RestoreFiles.conf
file after modifications.
root@replica:~ # cat /usr/local/etc/bareos/bareos-dir.d/job/RestoreFiles.conf Job { Name = "RestoreFiles" Description = "Standard Restore." Type = Restore Client = Default FileSet = "SelfTest" Storage = File Pool = BR-MO Messages = Standard Where = /bareos/restore Accurate = yes }
New /usr/local/etc/bareos/bareos-dir.d/client/Default.conf
file.
root@replica:~ # cat /usr/local/etc/bareos/bareos-dir.d/client/Default.conf Client { Name = Default address = replica.backup.org Password = "BAREOS-FD-PASSWORD" }
New /usr/local/etc/bareos/bareos-dir.d/client/replica.backup.org.conf
file.
root@replica:~ # cat /usr/local/etc/bareos/bareos-dir.d/client/replica.backup.org.conf Client { Name = replica.backup.org Description = "Client resource of the Director itself." address = replica.backup.org Password = "BAREOS-FD-PASSWORD" }
File below is left unchanged.
root@replica:~ # cat /usr/local/etc/bareos/bareos-dir.d/job/BackupCatalog.conf Job { Name = "BackupCatalog" Description = "Backup the catalog database (after the nightly save)" JobDefs = "DefaultJob" Level = Full FileSet="Catalog" Schedule = "WeeklyCycleAfterBackup" # This creates an ASCII copy of the catalog # Arguments to make_catalog_backup.pl are: # make_catalog_backup.pl RunBeforeJob = "/usr/local/lib/bareos/scripts/make_catalog_backup.pl MyCatalog" # This deletes the copy of the catalog RunAfterJob = "/usr/local/lib/bareos/scripts/delete_catalog_backup" # This sends the bootstrap via mail for disaster recovery. # Should be sent to another system, please change recipient accordingly Write Bootstrap = "|/usr/local/bin/bsmtp -h localhost -f \"\(Bareos\) \" -s \"Bootstrap for Job %j\" root@localhost" # (#01) Priority = 11 # run after main backup }
File below is left unchanged.
root@replica:~ # cat /usr/local/etc/bareos/bareos-dir.d/messages/Standard.conf Messages { Name = Standard Description = "Reasonable message delivery -- send most everything to email address and to the console." operatorcommand = "/usr/local/bin/bsmtp -h localhost -f \"\(Bareos\) \\" -s \"Bareos: Intervention needed for %j\" %r" mailcommand = "/usr/local/bin/bsmtp -h localhost -f \"\(Bareos\) \\" -s \"Bareos: %t %e of %c %l\" %r" operator = root@localhost = mount # (#03) mail = root@localhost = all, !skipped, !saved, !audit # (#02) console = all, !skipped, !saved, !audit append = "/var/log/bareos/bareos.log" = all, !skipped, !saved, !audit catalog = all, !skipped, !saved, !audit }
File below is left unchanged.
root@replica:~ # cat /usr/local/etc/bareos/bareos-dir.d/messages/Daemon.conf Messages { Name = Daemon Description = "Message delivery for daemon messages (no job)." mailcommand = "/usr/local/bin/bsmtp -h localhost -f \"\(Bareos\) \\" -s \"Bareos daemon message\" %r" mail = root@localhost = all, !skipped, !audit # (#02) console = all, !skipped, !saved, !audit append = "/var/log/bareos/bareos.log" = all, !skipped, !audit append = "/var/log/bareos/bareos-audit.log" = audit }
Pools
By default Bareos comes with four pools configured, we would not use them so we will delete their configuration files.
root@replica:~ # ls -l /usr/local/etc/bareos/bareos-dir.d/pool total 14 -rw-rw---- 1 bareos bareos 536 Apr 16 08:14 Differential.conf -rw-rw---- 1 bareos bareos 512 Apr 16 08:14 Full.conf -rw-rw---- 1 bareos bareos 534 Apr 16 08:14 Incremental.conf -rw-rw---- 1 bareos bareos 48 Apr 16 08:14 Scratch.conf root@replica:~ # rm -f /usr/local/etc/bareos/bareos-dir.d/pool/*.conf
We will now create two our pools for the DAILY backups and for the MONTHLY backups.
root@replica:~ # cat /usr/local/etc/bareos/bareos-dir.d/pool/BRONZE-DAILY-POOL.conf Pool { Name = BR-DA Pool Type = Backup Recycle = yes # Bareos can automatically recycle Volumes AutoPrune = yes # Prune expired volumes Volume Retention = 7 days # How long should the Full Backups be kept? (#06) Maximum Volume Bytes = 2G # Limit Volume size to something reasonable Maximum Volumes = 100000 # Limit number of Volumes in Pool Label Format = "BR-DA-" # Volumes will be labeled "BR-DA-" } root@replica:~ # cat /usr/local/etc/bareos/bareos-dir.d/pool/BRONZE-MONTHLY-POOL.conf Pool { Name = BR-MO Pool Type = Backup Recycle = yes # Bareos can automatically recycle Volumes AutoPrune = yes # Prune expired volumes Volume Retention = 120 days # How long should the Full Backups be kept? (#06) Maximum Volume Bytes = 2G # Limit Volume size to something reasonable Maximum Volumes = 100000 # Limit number of Volumes in Pool Label Format = "BR-MO-" # Volumes will be labeled "BR-MO-" }
File below is left unchanged.
root@replica:~ # cat /usr/local/etc/bareos/bareos-dir.d/schedule/WeeklyCycle.conf Schedule { Name = "WeeklyCycle" Run = Full 1st sat at 21:00 # (#04) Run = Differential 2nd-5th sat at 21:00 # (#07) Run = Incremental mon-fri at 21:00 # (#10) }
File below is left unchanged.
root@replica:~ # cat /usr/local/etc/bareos/bareos-dir.d/schedule/WeeklyCycle.conf Schedule { Name = "WeeklyCycle" Run = Full 1st sat at 21:00 # (#04) Run = Differential 2nd-5th sat at 21:00 # (#07) Run = Incremental mon-fri at 21:00 # (#10) }
Contents of the /usr/local/etc/bareos/bareos-dir.d/jobdefs/DefaultJob.conf
file after modifications.
root@replica:~ # cat /usr/local/etc/bareos/bareos-dir.d/jobdefs/DefaultJob.conf JobDefs { Name = "DefaultJob" Type = Backup Level = Differential Client = Default FileSet = "SelfTest" Schedule = "WeeklyCycle" Storage = File Messages = Standard Pool = BR-DA Priority = 10 Write Bootstrap = "/bareos/bootstrap/%c.bsr" }
Contents of the /usr/local/etc/bareos/bareos-dir.d/storage/File.conf
file after modifications.
root@replica:~ # cat /usr/local/etc/bareos/bareos-dir.d/storage/File.conf Storage { Name = File Address = replica.backup.org Password = "BAREOS-SD-PASSWORD" Device = FileStorage Media Type = File }
Contents of the /usr/local/etc/bareos/bareos-dir.d/console/bareos-mon.conf
file after modifications.
root@replica: # cat /usr/local/etc/bareos/bareos-dir.d/console/bareos-mon.conf Console { Name = bareos-mon Description = "Restricted console used by tray-monitor to get the status of the director." Password = "BAREOS-MON-PASSWORD" CommandACL = status, .status JobACL = *all* }
Contents of the /usr/local/etc/bareos/bareos-dir.d/fileset/Catalog.conf
file after modifications.
root@replica:~ # cat /usr/local/etc/bareos/bareos-dir.d/fileset/Catalog.conf FileSet { Name = "Catalog" Description = "Backup the catalog dump and Bareos configuration files." Include { Options { signature = MD5 Compression = lzo } File = "/var/db/bareos" File = "/usr/local/etc/bareos" } }
Contents of the /usr/local/etc/bareos/bareos-dir.d/fileset/SelfTest.conf
file after modifications.
root@replica:~ # cat /usr/local/etc/bareos/bareos-dir.d/fileset/SelfTest.conf FileSet { Name = "SelfTest" Description = "fileset just to backup some files for selftest" Include { Options { Signature = MD5 Compression = lzo } File = "/usr/local/sbin" } }
We do not need bundled LinuxAll.conf
and WindowsAllDrives.conf
filesets so we will delete them.
root@replica:~ # ls -l /usr/local/etc/bareos/bareos-dir.d/fileset/ total 18 -rw-rw---- 1 bareos bareos 250 Apr 27 02:25 Catalog.conf -rw-rw---- 1 bareos bareos 765 Apr 16 08:14 LinuxAll.conf -rw-rw---- 1 bareos bareos 210 Apr 27 02:27 SelfTest.conf -rw-rw---- 1 bareos bareos 362 Apr 16 08:14 WindowsAllDrives.conf root@replica:~ # rm -f /usr/local/etc/bareos/bareos-dir.d/fileset/LinuxAll.conf root@replica:~ # rm -f /usr/local/etc/bareos/bareos-dir.d/fileset/WindowsAllDrives.conf
We will now define two new filesets Windows.conf
and UNIX.conf
files.
New /usr/local/etc/bareos/bareos-dir.d/fileset/Windows.conf
file.
root@replica:~ # cat /usr/local/etc/bareos/bareos-dir.d/fileset/Windows.conf FileSet { Name = Windows Enable VSS = yes Include { Options { Signature = MD5 Drive Type = fixed IgnoreCase = yes WildFile = "[A-Z]:/pagefile.sys" WildDir = "[A-Z]:/RECYCLER" WildDir = "[A-Z]:/$RECYCLE.BIN" WildDir = "[A-Z]:/System Volume Information" Exclude = yes Compression = lzo } File = / } }
New /usr/local/etc/bareos/bareos-dir.d/fileset/UNIX.conf
file.
root@replica:~ # cat /usr/local/etc/bareos/bareos-dir.d/fileset/UNIX.conf FileSet { Name = "UNIX" Include { Options { Signature = MD5 # calculate md5 checksum per file One FS = No # change into other filessytems FS Type = ufs FS Type = btrfs FS Type = ext2 # filesystems of given types will be backed up FS Type = ext3 # others will be ignored FS Type = ext4 FS Type = reiserfs FS Type = jfs FS Type = xfs FS Type = zfs noatime = yes Compression = lzo } File = / } # Things that usually have to be excluded # You have to exclude /tmp # on your bareos server Exclude { File = /var/db/bareos File = /tmp File = /proc File = /sys File = /var/tmp File = /.journal File = /.fsck } }
File below is left unchanged.
root@replica: # cat /usr/local/etc/bareos/bareos-dir.d/profile/operator.conf Profile { Name = operator Description = "Profile allowing normal Bareos operations." Command ACL = !.bvfs_clear_cache, !.exit, !.sql Command ACL = !configure, !create, !delete, !purge, !sqlquery, !umount, !unmount Command ACL = *all* Catalog ACL = *all* Client ACL = *all* FileSet ACL = *all* Job ACL = *all* Plugin Options ACL = *all* Pool ACL = *all* Schedule ACL = *all* Storage ACL = *all* Where ACL = *all* }
Contents of the /usr/local/etc/bareos/bareos-sd.d/messages/Standard.conf
file after modifications.
root@replica:~ # cat /usr/local/etc/bareos/bareos-sd.d/messages/Standard.conf Messages { Name = Standard Director = replica.backup.org = all Description = "Send all messages to the Director." }
We will add /bareos/storage/FileStorage
path as out FileStorage
place for backups.
Contents of the /usr/local/etc/bareos/bareos-sd.d/device/FileStorage.conf
file after modifications.
root@replica:~ # cat /usr/local/etc/bareos/bareos-sd.d/device/FileStorage.conf Device { Name = FileStorage Media Type = File Archive Device = /bareos/storage/FileStorage LabelMedia = yes; # lets Bareos label unlabeled media Random Access = yes; AutomaticMount = yes; # when device opened, read it RemovableMedia = no; AlwaysOpen = no; Description = "File device. A connecting Director must have the same Name and MediaType." }
Contents of the /usr/local/etc/bareos/bareos-sd.d/storage/bareos-sd.conf
file after modifications.
root@replica:~ # cat /usr/local/etc/bareos/bareos-sd.d/storage/bareos-sd.conf Storage { Name = replica.backup.org Maximum Concurrent Jobs = 20 # remove comment from "Plugin Directory" to load plugins from specified directory. # if "Plugin Names" is defined, only the specified plugins will be loaded, # otherwise all storage plugins (*-sd.so) from the "Plugin Directory". # # Plugin Directory = /usr/local/lib/bareos/plugins # Plugin Names = "" }
Contents of the /usr/local/etc/bareos/bareos-sd.d/director/bareos-mon.conf
file after modifications.
root@replica:~ # cat /usr/local/etc/bareos/bareos-sd.d/director/bareos-mon.conf Director { Name = bareos-mon Password = "BAREOS-SD-PASSWORD" Monitor = yes Description = "Restricted Director, used by tray-monitor to get the status of this storage daemon." }
Contents of the /usr/local/etc/bareos/bareos-sd.d/director/bareos-dir.conf
file after modifications.
root@replica:~ # cat /usr/local/etc/bareos/bareos-sd.d/director/bareos-dir.conf Director { Name = replica.backup.org Password = "BAREOS-SD-PASSWORD" Description = "Director, who is permitted to contact this storage daemon." }
Contents of the /usr/local/etc/bareos/bareos-fd.d/messages/Standard.conf
file after modifications.
root@replica:~ # cat /usr/local/etc/bareos/bareos-fd.d/messages/Standard.conf Messages { Name = Standard Director = replica.backup.org = all, !skipped, !restored Description = "Send relevant messages to the Director." }
Contents of the /usr/local/etc/bareos/bareos-fd.d/director/bareos-dir.conf
file after modifications.
root@replica:~ # cat /usr/local/etc/bareos/bareos-fd.d/director/bareos-dir.conf Director { Name = replica.backup.org Password = "BAREOS-FD-PASSWORD" Description = "Allow the configured Director to access this file daemon." }
Contents of the /usr/local/etc/bareos/bareos-fd.d/director/bareos-mon.conf
file after modifications.
root@replica:~ # cat /usr/local/etc/bareos/bareos-fd.d/director/bareos-mon.conf Director { Name = bareos-mon Password = "BAREOS-MON-PASSWORD" Monitor = yes Description = "Restricted Director, used by tray-monitor to get the status of this file daemon." }
Contents of the /usr/local/etc/bareos/bareos-fd.d/client/myself.conf
file after modifications.
root@replica:~ # cat /usr/local/etc/bareos/bareos-fd.d/client/myself.conf Client { Name = replica.backup.org Maximum Concurrent Jobs = 20 # remove comment from "Plugin Directory" to load plugins from specified directory. # if "Plugin Names" is defined, only the specified plugins will be loaded, # otherwise all storage plugins (*-fd.so) from the "Plugin Directory". # # Plugin Directory = /usr/local/lib/bareos/plugins # Plugin Names = "" # if compatible is set to yes, we are compatible with bacula # if set to no, new bareos features are enabled which is the default # compatible = yes }
Contents of the /usr/local/etc/bareos/bareos-dir.d/client/bareos-fd.conf
file after modifications.
root@replica:~ # cat /usr/local/etc/bareos/bareos-dir.d/client/bareos-fd.conf Client { Name = bareos-fd Description = "Client resource of the Director itself." Address = localhost Password = "BAREOS-FD-PASSWORD" }
Lets see which files and Bareos components hold which passwords.
root@replica:~ # cd /usr/local/etc/bareos root@replica:/usr/local/etc/bareos # pwd /usr/local/etc/bareos root@replica:/usr/local/etc/bareos # grep -r Password . | sort -k 4 | column -t ./bareos-dir.d/director/bareos-dir.conf: Password = "BAREOS-DIR-PASSWORD" ./bconsole.d/bconsole.conf: Password = "BAREOS-DIR-PASSWORD" ./bareos-dir.d/client/Default.conf: Password = "BAREOS-FD-PASSWORD" ./bareos-dir.d/client/bareos-fd.conf: Password = "BAREOS-FD-PASSWORD" ./bareos-dir.d/client/replica.backup.org.conf: Password = "BAREOS-FD-PASSWORD" ./bareos-fd.d/director/bareos-dir.conf: Password = "BAREOS-FD-PASSWORD" ./bareos-dir.d/console/bareos-mon.conf: Password = "BAREOS-MON-PASSWORD" ./bareos-fd.d/director/bareos-mon.conf: Password = "BAREOS-MON-PASSWORD" ./bareos-dir.d/storage/File.conf: Password = "BAREOS-SD-PASSWORD" ./bareos-sd.d/director/bareos-dir.conf: Password = "BAREOS-SD-PASSWORD" ./bareos-sd.d/director/bareos-mon.conf: Password = "BAREOS-SD-PASSWORD"
Lets fix the rights after creating all new files.
root@replica:~ # chown -R bareos:bareos /usr/local/etc/bareos root@replica:~ # find /usr/local/etc/bareos -type f -exec chmod 640 {} ';' root@replica:~ # find /usr/local/etc/bareos -type d -exec chmod 750 {} ';'
Bareos WebUI
Now we will add/configure files for the Bareos WebUI interface.
The main Nginx webserver configuration file.
root@replica:~ # cat /usr/local/etc/nginx/nginx.conf user www; worker_processes 4; worker_rlimit_nofile 51200; error_log /var/log/nginx/error.log; events { worker_connections 1024; } http { include mime.types; default_type application/octet-stream; log_format main '$remote_addr - $remote_user [$time_local] "$request" '; access_log /var/log/nginx/access.log main; sendfile on; keepalive_timeout 65; server { listen 9100; server_name replica.backup.org bareos; root /usr/local/www/bareos-webui/public; location / { index index.php; try_files $uri $uri/ /index.php?$query_string; } location ~ .php$ { fastcgi_pass 127.0.0.1:9000; fastcgi_param APPLICATION_ENV production; fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name; include fastcgi_params; try_files $uri =404; } } }
For the PHP we will modify the bundled config file from package /usr/local/etc/php.ini-production
file.
root@replica:~ # cp /usr/local/etc/php.ini-production /usr/local/etc/php.ini root@replica:~ # vi /usr/local/etc/php.ini
We only add the timezone, for my location it is the Europe/Warsaw location.
root@replica:~ # diff -u php.ini-production php.ini --- php.ini-production 2017-08-12 03:23:36.000000000 +0200 +++ php.ini 2017-09-12 18:50:40.513138000 +0200 @@ -934,6 +934,7 @@ ; Defines the default timezone used by the date functions ; http://php.net/date.timezone -;date.timezone = +date.timezone = Europe/Warsaw ; http://php.net/date.default-latitude ;date.default_latitude = 31.7667
Here is the PHP php-fpm
daemon configuration.
root@replica:~ # cat /usr/local/etc/php-fpm.conf [global] pid = run/php-fpm.pid log_level = notice [www] user = www group = www listen = 127.0.0.1:9000 listen.backlog = -1 listen.owner = www listen.group = www listen.mode = 0660 listen.allowed_clients = 127.0.0.1 pm = static pm.max_children = 4 pm.start_servers = 1 pm.min_spare_servers = 0 pm.max_spare_servers = 4 pm.process_idle_timeout = 1000s; pm.max_requests = 500 request_terminate_timeout = 0 rlimit_files = 51200 env[HOSTNAME] = $HOSTNAME env[PATH] = /usr/local/bin:/usr/bin:/bin env[TMP] = /tmp env[TMPDIR] = /tmp env[TEMP] = /tmp
Rest of the Bareos WebUI configuration.
New /usr/local/etc/bareos/bareos-dir.d/console/admin.conf
file.
root@replica:~ # cat /usr/local/etc/bareos/bareos-dir.d/console/admin.conf Console { Name = admin Password = ADMIN-PASSWORD Profile = webui-admin }
New /usr/local/etc/bareos/bareos-dir.d/profile/webui-admin.conf
file.
root@replica:~ # cat /usr/local/etc/bareos/bareos-dir.d/profile/webui-admin.conf Profile { Name = webui-admin CommandACL = !.bvfs_clear_cache, !.exit, !.sql, !configure, !create, !delete, !purge, !sqlquery, !umount, !unmount, *all* Job ACL = *all* Schedule ACL = *all* Catalog ACL = *all* Pool ACL = *all* Storage ACL = *all* Client ACL = *all* FileSet ACL = *all* Where ACL = *all* Plugin Options ACL = *all* }
You may add other directors here as well.
Modified /usr/local/etc/bareos-webui/directors.ini
file.
root@replica:~ # cat /usr/local/etc/bareos-webui/directors.ini ;------------------------------------------------------------------------------ ; Section localhost-dir ;------------------------------------------------------------------------------ [replica.backup.org] enabled = "yes" diraddress = "replica.backup.org" dirport = 9101 catalog = "MyCatalog"
Modified /usr/local/etc/bareos-webui/configuration.ini
file.
root@replica:~ # cat /usr/local/etc/bareos-webui/configuration.ini ;------------------------------------------------------------------------------ ; SESSION SETTINGS ;------------------------------------------------------------------------------ [session] timeout=3600 ;------------------------------------------------------------------------------ ; DASHBOARD SETTINGS ;------------------------------------------------------------------------------ [dashboard] autorefresh_interval=60000 ;------------------------------------------------------------------------------ ; TABLE SETTINGS ;------------------------------------------------------------------------------ [tables] pagination_values=10,25,50,100 pagination_default_value=25 save_previous_state=false ;------------------------------------------------------------------------------ ; VARIOUS SETTINGS ;------------------------------------------------------------------------------ [autochanger] labelpooltype=scratch
Last but not least, we need to set permissions for Bareos WebUI configuration files.
root@replica:~ # chown -R www:www /usr/local/etc/bareos-webui root@replica:~ # chown -R www:www /usr/local/www/bareos-webui
Logs
Lets create the needed log files and fix their permissions.
root@replica:~ # chown -R bareos:bareos /var/log/bareos root@replica:~ # :> /var/log/php-fpm.log root@replica:~ # chown -R www:www /var/log/php-fpm.log root@replica:~ # chown -R www:www /var/log/nginx
We will now add rules to the newsyslog(8)
log rotate daemon, we do not want our filesystem to fill up don’t we?
As newsyslog does cover the *.conf.d
directories we will use them instead of modifying the main /etc/newsyslog.conf
configuration file.
root@replica:~ # grep conf\\.d /etc/newsyslog.conf /etc/newsyslog.conf.d/* /usr/local/etc/newsyslog.conf.d/* root@replica:~ # mkdir -p /usr/local/etc/newsyslog.conf.d root@replica:~ # cat > /usr/local/etc/newsyslog.conf.d/bareos << __EOF # BAREOS /var/log/php-fpm.log www:www 640 7 100 @T00 J /var/log/nginx/access.log www:www 640 7 100 @T00 J /var/log/nginx/error.log www:www 640 7 100 @T00 J /var/log/bareos/bareos.log bareos:bareos 640 7 100 @T00 J /var/log/bareos/bareos-audit.log bareos:bareos 640 7 100 @T00 J __EOF
Lets verify that newsyslog(8)
understands out configuration.
root@replica:~ # newsyslog -v | tail -5 /var/log/php-fpm.log : --> will trim at Tue May 1 00:00:00 2018 /var/log/nginx/access.log : --> will trim at Tue May 1 00:00:00 2018 /var/log/nginx/error.log : --> will trim at Tue May 1 00:00:00 2018 /var/log/bareos/bareos.log : --> will trim at Tue May 1 00:00:00 2018 /var/log/bareos/bareos-audit.log : --> will trim at Tue May 1 00:00:00 2018
Skel
We now need to create so called Bareos skel files for the rc(8)
script to gather all the configuration in one file.
If we do not do that the Bareos services would not stop and we will see an error like that one below.
root@replica:~ # /usr/local/etc/rc.d/bareos-sd onestart Starting bareos_sd. 27-Apr 02:59 bareos-sd JobId 0: Error: parse_conf.c:580 Failed to read config file "/usr/local/etc/bareos/bareos-sd.conf" bareos-sd ERROR TERMINATION parse_conf.c:148 Failed to find config filename. /usr/local/etc/rc.d/bareos-sd: WARNING: failed to start bareos_sd
Lets create them then …
root@replica:~ # cat > /usr/local/etc/bareos/bareos-dir.conf << __EOF @/usr/local/etc/bareos/bareos-dir.d/*/* __EOF root@replica:~ # cat > /usr/local/etc/bareos/bareos-fd.conf << __EOF @/usr/local/etc/bareos/bareos-fd.d/*/* __EOF root@replica:~ # cat > /usr/local/etc/bareos/bareos-sd.conf << __EOF @/usr/local/etc/bareos/bareos-sd.d/*/* __EOF root@replica:~ # cat > /usr/local/etc/bareos/bconsole.conf << __EOF @/usr/local/etc/bareos/bconsole.d/* __EOF
… and verify their contents.
root@replica:~ # cat /usr/local/etc/bareos/bareos-dir.conf @/usr/local/etc/bareos/bareos-dir.d/*/* root@replica:~ # cat /usr/local/etc/bareos/bareos-fd.conf @/usr/local/etc/bareos/bareos-fd.d/*/* root@replica:~ # cat /usr/local/etc/bareos/bareos-sd.conf @/usr/local/etc/bareos/bareos-sd.d/*/* root@replica:~ # cat /usr/local/etc/bareos/bconsole.conf @/usr/local/etc/bareos/bconsole.d/*
After all our modification and added files lefs make sure that /usr/local/etc/bareos
dir permissions are properly set.
root@replica:~ # chown -R bareos:bareos /usr/local/etc/bareos root@replica:~ # find /usr/local/etc/bareos -type f -exec chmod 640 {} ';' root@replica:~ # find /usr/local/etc/bareos -type d -exec chmod 750 {} ';'
Its Alive!
Back to our system settings, we will add service start to the main FreeBSD /etc/rc.conf
file.
After the modifications our final /etc/rc.conf
file will look as follows.
root@replica:~ # cat /etc/rc.conf # NETWORK hostname=replica.backup.org ifconfig_em0="inet 10.0.10.30/24 up" defaultrouter="10.0.10.1" # DAEMONS zfs_enable=YES sshd_enable=YES nfs_client_enable=YES syslogd_flags="-ss" sendmail_enable=NONE # OTHER clear_tmp_enable=YES dumpdev=NO # BAREOS postgresql_enable=YES postgresql_class=pgsql bareos_dir_enable=YES bareos_sd_enable=YES bareos_fd_enable=YES php_fpm_enable=YES nginx_enable=YES
As PostgreSQL server is already running …
root@replica:~ # /usr/local/etc/rc.d/postgresql status pg_ctl: server is running (PID: 15205) /usr/local/bin/postgres "-D" "/usr/local/pgsql/data"
… we will now start rest of our Bareos stack services.
First the PHP php-fpm
daemon.
root@replica:~ # /usr/local/etc/rc.d/php-fpm start Performing sanity check on php-fpm configuration: [27-Apr-2018 02:57:09] NOTICE: configuration file /usr/local/etc/php-fpm.conf test is successful Starting php_fpm.
The Nginx webserver.
root@replica:~ # /usr/local/etc/rc.d/nginx start Performing sanity check on nginx configuration: nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok nginx: configuration file /usr/local/etc/nginx/nginx.conf test is successful Starting nginx.
Bareos Storage Daemon.
root@replica:~ # /usr/local/etc/rc.d/bareos-sd start Starting bareos_sd.
Bareos File Daemon also known as Bareos client.
root@replica:~ # /usr/local/etc/rc.d/bareos-fd start Starting bareos_fd.
… and last but least, the most important daemon of this guide, the Bareos Director.
root@replica:~ # /usr/local/etc/rc.d/bareos-dir start Starting bareos_dir.
We may now see on what ports our daemons are listening.
root@replica:~ # sockstat -l4 USER COMMAND PID FD PROTO LOCAL ADDRESS FOREIGN ADDRESS bareos bareos-dir 89823 4 tcp4 *:9101 *:* root bareos-fd 73066 3 tcp4 *:9102 *:* www nginx 33857 6 tcp4 *:9100 *:* www nginx 28675 6 tcp4 *:9100 *:* www nginx 20960 6 tcp4 *:9100 *:* www nginx 15881 6 tcp4 *:9100 *:* root nginx 14388 6 tcp4 *:9100 *:* www php-fpm 84047 0 tcp4 127.0.0.1:9000 *:* www php-fpm 82285 0 tcp4 127.0.0.1:9000 *:* www php-fpm 80688 0 tcp4 127.0.0.1:9000 *:* www php-fpm 74735 0 tcp4 127.0.0.1:9000 *:* root php-fpm 70518 8 tcp4 127.0.0.1:9000 *:* bareos bareos-sd 5151 3 tcp4 *:9103 *:* pgsql postgres 20009 4 tcp4 127.0.0.1:5432 *:* root sshd 49253 4 tcp4 *:22 *:*
In case You wandered in what order these services will start, below is the answer from rc(8)
subsystem.
root@replica:~ # rcorder /etc/rc.d/* /usr/local/etc/rc.d/* | grep -E '(bareos|php-fpm|nginx|postgresql)' /usr/local/etc/rc.d/postgresql /usr/local/etc/rc.d/php-fpm /usr/local/etc/rc.d/nginx /usr/local/etc/rc.d/bareos-sd /usr/local/etc/rc.d/bareos-fd /usr/local/etc/rc.d/bareos-dir
We can now access http://replica.backup.org:9100 in our browser.
Its indeed alive, we can now login with admin
user and ADMIN-PASSWORD
password.
As we logged in we see empty Bareos dashboard.
Jobs
Now, to make life easier I have prepared two scripts for adding clients to the Bareos server.
The BRONZE-job.sh
and BRONZE-sched.sh
for generate Bareos files for new jobs and schedules. We will put them into /root/bin
dir for convenience.
root@replica:~ # mkdir /root/bin
Both scripts are available below:
After downloading them please rename them accordingly (WordPress limitation).
root@replica:~ # mv BRONZE-sched.sh.key BRONZE-sched.sh root@replica:~ # mv BRONZE-job.sh.key BRONZE-job.sh
Lets make them executable.
root@replica:~ # chmod +x /root/bin/BRONZE-sched.sh root@replica:~ # chmod +x /root/bin/BRONZE-job.sh
Below is ‘help’ message for each of them.
root@replica:~ # /root/bin/BRONZE-sched.sh usage: BRONZE-sched.sh GROUP TIME example: BRONZE-sched.sh 01 21:00
root@replica:~ # /root/bin/BRONZE-job.sh usage: BRONZE-job.sh GROUP TIME CLIENT TYPE GROUP option: 01 | 02 | 03 TIME option: 00:00 - 23:59 CLIENT option: FQDN TYPE option: UNIX | Windows example: BRONZE-job.sh 01 21:00 CLIENT.domain.com UNIX
Client
For the first client we will use the replica.backup.org client – the server itself.
First use the BRONZE-sched.sh
to create new scheduler configuration. The script will echo names of the files it created.
root@replica:~ # /root/bin/BRONZE-sched.sh 01 21:00 /usr/local/etc/bareos/bareos-dir.d/schedule/BRONZE-DAILY-01-2100-SCHED.conf /usr/local/etc/bareos/bareos-dir.d/jobdefs/BRONZE-DAILY-01-2100-UNIX.conf /usr/local/etc/bareos/bareos-dir.d/jobdefs/BRONZE-DAILY-01-2100-Windows.conf /usr/local/etc/bareos/bareos-dir.d/schedule/BRONZE-MONTHLY-01-2100-SCHED.conf /usr/local/etc/bareos/bareos-dir.d/jobdefs/BRONZE-MONTHLY-01-2100-UNIX.conf /usr/local/etc/bareos/bareos-dir.d/jobdefs/BRONZE-MONTHLY-01-2100-Windows.conf
We will not use Windows backups for that client in that schedule so we can remove them.
root@replica:~ # rm -f \ /usr/local/etc/bareos/bareos-dir.d/jobdefs/BRONZE-DAILY-01-2100-Windows.conf \ /usr/local/etc/bareos/bareos-dir.d/jobdefs/BRONZE-MONTHLY-01-2100-Windows.conf
Then use the BRONZE-job.sh
to add client and its type to created earlier schedule. Names of the created files will also be echoed to stdout.
root@replica:~ # /root/bin/BRONZE-job.sh 01 21:00 replica.backup.org UNIX INFO: client DNS check. INFO: DNS 'A' RECORD: Host replica.backup.org not found: 3(NXDOMAIN) INFO: DNS 'PTR' RECORD: Host 3\(NXDOMAIN\) not found: 3(NXDOMAIN) /usr/local/etc/bareos/bareos-dir.d/job/BRONZE-DAILY-01-2100-replica.backup.org.conf /usr/local/etc/bareos/bareos-dir.d/job/BRONZE-MONTHLY-01-2100-replica.backup.org.conf
Now we need to reload the Bareos server configuration.
root@replica:~ # echo reload | bconsole Connecting to Director localhost:9101 1000 OK: replica.backup.org Version: 16.2.7 (09 October 2017) Enter a period to cancel a command. reload reloaded
Lets see how it looks in the browser. We will run that job, then cancel it and then rerun it again.
Client replica.backup.org is configured.
Lets go to Jobs tab to start its backup Job.
Message that backup Job has started.
We can see it in running state on Jobs tab.
… and on the Dashboard.
We can also display its messages by clicking on its number.
The Jobs tab after cancelling the first Job and starting it again till completion.
… and the Dashboard after these activities.
Restore
Lets restore some data, in Bareos its a breeze as its accessed directly in the browser on the Restore tab.
The Restore Job has started.
The Dashboard after restoration.
… and Volumes with our precious data.
Contents of a Volume.
Status of our Bareos Director.
… and Director Messages, an equivalent of query actlog
from IBM TSM or as they call it recently – IBM Spectrum Protect.
… and Bareos Console (bconsole) directly in the browser. Masterpiece!
Confirmation about the restored file.
root@replica:~ # ls -l /tmp/bareos-restores/COPYRIGHT -r--r--r-- 1 root wheel 6199 Jul 21 2017 /tmp/bareos-restores/COPYRIGHT root@replica:~ # sha256 /tmp/bareos-restores/COPYRIGHT /COPYRIGHT | column -t SHA256 (/tmp/bareos-restores/COPYRIGHT) = 79b7aaafa1bc42a1ff03f1f78a667edb9a203dbcadec06aabc875e25a83d23f0 SHA256 (/COPYRIGHT) = 79b7aaafa1bc42a1ff03f1f78a667edb9a203dbcadec06aabc875e25a83d23f0
Remote Replica
We have volumes with backup in the /bareos
directory, we will now configure rsync(1)
to replicate these backups to the /bareos-dr
directory, to NFS server in other location.
root@replica:~ # pkg install rsync
The rsync(1)
command will look like that.
/usr/local/bin/rsync -r -u -l -p -t -S --force --no-whole-file --numeric-ids --delete-after /bareos/ /bareos-dr/
We will put that command into the crontab(1)
root job.
root@replica:~ # crontab -e root@replica:~ # crontab -l 0 7 * * * /usr/local/bin/rsync -r -u -l -p -t -S --force --no-whole-file --numeric-ids --delete-after /bareos/ /bareos-dr/
As all backups have finished before 7:00, the end of backup window, we will start replication by then.
Summary
So we have a configured ready to make backups and restore Bareos Backup Server on a FreeBSD operating system. It can be used as an Appliance on any virtualization platform or also on a physical server with local storage resources without NFS shares.
UPDATE 1 – Die Hard Tribute in 9.2-RC3 Loader
The FreeBSD Developers even made a tribute to the Die Hard movie and actually implemented the Nakatomi Socrates screen in the FreeBSD 9.2-RC3 loader as shown on the images below. Unfortunately it has been removed in later FreeBSD 9.2-RC4 and official FreeBSD 9.2-RELEASE versions.
UPDATE 2
The Bareos Backup Server on FreeBSD article was featured in the BSD Now 254 – Bare the OS episode.
Thanks for mentioning!
UPDATE 3 – Additional Permissions
Thanks to Math user who identified the problem I added this paragraph below in proper place to make the HOWTO complete. Without it many Bareos daemons would not start with permissions error.
Here is the added paragraph.
We also need to change permissions for the /var/run and /var/db directories for Bareos.
root@replica:~ # chown -R bareos:bareos /var/db/bareos root@replica:~ # chown -R bareos:bareos /var/run/bareos
Β
Pingback: Home | vermaden
Pingback: BareOS backup server (FreeBSD) | 0ddn1x: tricks with *nix
Pingback: Bare the OS | BSD Now 254 | Jupiter Broadcasting
Hello ! Thanks a lot for this π
Just a thing. When defining conf files in section Skel :
cat > /usr/local/etc/bareos/bareos-sd.conf << __EOF
@/usr/local/etc/bareos/bareos-sd.d/*/*.conf
__EOF
in order to ignore sample or ~ files and have this sort of error :
Attempt to define second "Director" resource named "bareos-dir" is not permitted.
Attempt to define second "Messages" resource named "Standard" is not permitted.
…
Thanks again !
LikeLike
Thanks π
Removing the *.sample files is one of the steps in the guide:
We will now take care of the Bareos server configuration. There are a lot *.sample files that we do not need. (…)
Hope that helps.
Regards.
LikeLike
Oups π
Another thing. Have you test the truncate action ?
(adding Action On Purge = Truncate in the pool def)
It can be usefull for saving space but I saw on google that it’s probably don’t work with Bareos.
LikeLike
According to the Bareos Manual it is supported.
LikeLiked by 1 person
This confuses me greatly:
After install we will disable the /zroot mounting.
root@replica:/ # zfs set mountpoint=none zroot
So what happens when I reboot my server? it can’t mount the root zfs filesystem called /zroot, so my server will fail to boot?
LikeLike
The / dataset is mounted from the Boot Environment ZFS dataset – zroot/ROOT/default – removal of /zroot mountpoint for the entire zroot pool changes nothing for that π
LikeLike
Thank you for the speedy reply.
I am new to freeBSD and want to clarify.
So Unmounting /zroot should have no effect on any other mounted filesystems under / or /zroot
ie
/storage
zroot/iocage
zroot/poudriere
Why exactly are you unmounting /zroot? What is the benefit?
What happens if I boot using a rescue usb, will unmounting /zroot cause problems using the rescue usb environment?
LikeLike
Yes it does not have effect in other datasets.
Unmounted /zroot will not cause problems in rescue environment.
LikeLike
Again thank you for the prompt response. 8:)
But, why exactly are you unmounting /zroot? What is the benefit?
LikeLike
There is no benefit in mounting zroot pool as /zroot.
LikeLike
This is a great tutorial, many many thanks for putting it out here! I’m finding this post a wonderful tool for learning how bareos works.
Right now I’m doing testing, and troubleshooting issues as they come up (at the moment figuring out what the heck this means: 2018-10-05 03:42:48 nix.catpasswd.net JobId 13: Error: Could not open WriteBootstrap file:
/bareos/bootstrap/nix.catpasswd.net.bsr: ERR=No such file or directory)
At any rate, when I’m finished testing, and am confident I know how things work, how do I get a fresh start? As in, delete all the backups and catalogs, clear the job history, etc – without messing with the configuration files and such?
LikeLike
Hi and thanks for comment.
To reset the Bareos instance use these instructions:
First stop all Bareos daemons.
Then remove its files:
Drop Bareos database:
Create Bareos database:
Start Bareos services:
Done.
You should now have clean Bareos installation.
Regards.
LikeLike
Pingback: Bareos @FreeBSD @Freenas 11.2 | getcom.de
thanks for the great article, but i have a problem after the installation.
i’ve only a blank page, nginx log:
*1 FastCGI sent in stderr: “PHP message: PHP Parse error: syntax error, unexpected ‘@’ in /usr/local/www/bareos/config/autoload/global.php on line 142” while reading response header from upstream, client: 192.168.178.50, server: localhost, request: “GET / HTTP/1.1”, upstream: “fastcgi://127.0.0.1:9000”, host: “192.168.178.211:9100”
LikeLike
Thanks.
I would like to help but I have no idea what is causing the problem.
I would suggest doing/checking the instructions point by point to not omit anything in the process.
Regards.
LikeLike
I have the same problem with bareos-webui-18.2.6
Do you resolve it?
Thank you.
2019/09/17 17:17:26 [error] 14653#100511: *1 FastCGI sent in stderr: “PHP message: PHP Parse error: syntax error, unexpected ‘@’ in /usr/local/www/bareos/config/autoload/global.php on line 142” while reading response header from upstream, client: 192.168.0.181, server: bareos, request: “GET / HTTP/1.1”, upstream: “fastcgi://127.0.0.1:9000”, host: “192.168.0.59:9100”
LikeLiked by 2 people
This is error in bareos-webgui port
bareos-webui v.18.2.6 wasn’t expand @@ macros in global.php
I sent PR
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=240657
You can correct this manually
vi config/autoload/global.php
—
$directors_ini = “/usr/local/etc/bareos/directors.ini”; <——
$configuration_ini = "/usr/local/etc/bareos/configuration.ini"; <——
…
else {
$arr[key($directors)]['port'] = 9101; <—— string 142
}
vi /usr/local/etc/bareos/directors.ini
—
; Default value is @dirport@
dirport = 9101 <——
LikeLiked by 1 person
Hi,
First of all thanks a lot for this “how to”. It is very helfull as there is no official documentations for Bareos on FreeBSD.
My problem is that I can’t start bareos-sd and bareos-dir. After checking a thousand times my configuration, I can’t find the issue. The new version of Bareos only chages the directories of bareos-webui, so everything else is the same.
I tried to debut but there is no logs at all in /var/log/bareos/. I tried to have more informations with the command:/usr/local/sbin/bareos-sd -f -s -d 200 but not really helpfull.
The only errors are those lines:
bareos-sd (8): lib/bsys.cc:693-0 Could not open state file. sfd=-1 size=192: ERR=No such file or directory
bareos-sd (8): lib/crypto_cache.cc:55-0 Could not open crypto cache file. /var/db/bareos/bareos-sd.9103.cryptoc ERR=No such file or directory
Any ideas?
Regards
LikeLike
Hi,
I would check the permissions for /var/db/bareos and /var/run directories – if bareos-* binaries are able to create/modify their files there under bareos user.
Also truss(8) command may be useful to check where the problem is.
Regards.
LikeLike
Hi,
Thanks a lot. It was a permissions issues with the two directories that you’ve listed.
Maybe adding thoses commands in the “How to” :
chown -R bareos:bareos /var/db/bareos/
chown -R bareos:bareos /var/run/bareos/
Regards,
LikeLike
Sure, I will update the howto, I probably have done it but forgot to add it to the (big already) howto.
Thanks,
vermaden
LikeLike
Hi there,
I’m getting following message from my nginx
2020/05/28 06:48:37 [error] 67249#101032: *22 upstream prematurely closed connection while reading response header from upstream, client: 172.16.0.124, server: s-bareos.xxx.de, request: “GET / HTTP/1.1”, upstream: “fastcgi://127.0.0.1:9000”, host: “172.16.0.82:9100”
2020/05/28 06:48:38 [error] 67249#101032: *22 kevent() reported about an closed connection (54: Connection reset by peer) while reading response header from upstream, client: 172.16.0.124, server: s-bareos.six.de, request: “GET /favicon.ico HTTP/1.1”, upstream: “fastcgi://127.0.0.1:9000”, host: “172.16.0.82:9100”, referrer: “http://172.16.0.82:9100/”
Some knews what’s wrong?
bareos-webui-19.2.7
Regards Alexej
LikeLike
Hi,
sorry for late response …
Its probably because of wrong/faulty php-fpm configuration.
Regards,
vermaden
LikeLike
Hi,
the solution for me was to change the loopback IP to “172.16.0.82” in /usr/local/etc/php-fpm.conf
Regards and thanks
Alexej
LikeLike
Thank you for sharing that. Maybe it will be helpful for someone else with similar problem.
Regards,
vermaden
LikeLike