One of the comments after the latter was that I compared non-redundant backup solution (single disk) to redundant backup in the cloud. Today – as this is my main backup system – I would like to show you redundant backup solution with two disks in ZFS mirror along with real power usage measurements. This time I got ASRock J3355B-ITX motherboard with only 10W TDP which includes 2-core Celeron J3355 2.0-2.5 GHz CPU and small shiny REALAN H80 Mini ITX case. It looks very nice and comes from AliExpress at very low $33 price for new unit along with free shipping.
Here is how the REALAN H80 case looks like.
The ASRock J3355B-ITX motherboard.
Same as with the earlier build the internal Seagate BarraCuda 5TB 2.5 SATA drives costs about $200. The same Seagate Backup Plus 5TB 2.5 disk in external case with USB 3.0 port costs nearly half of that price – only $120 – at least in the Europe/Poland location. I took the decision to buy external ones and rip off their cases. That saved me about $160.
Here is the simple performance benchmark of these 2.5 disks.
% which pv pv: aliased to pv -t -r -a -b -W -B 1048576 % pv < /dev/ada0 > /dev/null 1.35GiB 0:00:10 [ 137MiB/s] [ 137MiB/s] ^C % dd if=/dev/ada0 of=/dev/null bs=8M 127+0 records in 127+0 records out 1065353216 bytes transferred in 7.494081 secs (142159287 bytes/sec) ^C
About 135MB/s per disk.
The ripped of parts of Seagate Backup Plus USB cases.
What made me laugh was that as I got different cases colors (silver and gray) the disks inside also had different colors (green and blue) 🙂
… but their part number is the same, here they are mounted on a REALAN H80 disks holder.
For the record – several REALAN H80 case real shots (not renders). First its front.
Side with USB port.
Generally the REALAN H80 looks really nice. Little lower REALAN H60 (without COM slots/holes in the back) looks even better but I wanted to make sure that I will have room and space for hot air in that case – as space was not a problem for me.
The complete price tops at $220 total. Here are the parts used.
PRICE COMPONENT $49 CPU/Motherboard ASRock J3355B-ITX Mini-ITX $10 RAM 4GB DDR3 $13 PSU 12V 7.5A 90W Pico (internal) $2 PSU 12V 2.5A 30W Leader Electronics (external) $33 Supermicro SC101i $3 SanDisk Fit 16GB USB 2.0 Drive (system) $120 Seagate 5TB 2.5 drive (ONE) $120 Seagate 5TB 2.5 drive (TWO) $350 TOTAL
That is $110 for the ‘system’ and additional $240 for ‘data’ drives.
Today I would probably get the ASRock N3150DC-ITX or Gigabyte GA-N3160TN motherboard instead because of builtin DC jack slot (compatible with 19V power adapter) on its back. This will eliminate the need for additional internal Pico PSU power supply …
The ASRock N3150DC-ITX with builtin DC jack.
The Gigabyte GA-N3160TN with builtin DC jack.
The Gigabyte GA-N3160TN is also very low profile motherboard as you can see from the back.
It may be good idea to use this one instead ASRock N3150DC-ITX to get more space above the motherboard.
As in the earlier Silent Fanless FreeBSD Server – DIY Backup article I used small 12V 2.5A 30W compact and cheap external PSU instead of the large 90W PSU from FSP Group. As these low power motherboard does not need a lot of power.
New Leader Electronics PSU label.
The internal power supply is Pico PSU which now tops as 12V 7.5A 90W power.
I also measured the power consumption with power meter.
The whole box with two Seagate BarraCuda 5TB 2.5 drives for data on ZFS mirror and SanDisk 16GB USB 2.0 system drive used about 10.4W in idle state.
I used all needed settings from my earlier The Power to Serve – FreeBSD Power Management article with CPU speed limited between 0.4GHz and 1.2GHz.
The powerd(8) settings in the /etc/rc.conf file are below.
powerd_flags="-n hiadaptive -a hiadaptive -b hiadaptive -m 400 -M 1200"
I used python(1)  to load the CPU and dd(8) to load the drives. I used dd(8) on the ZFS pool so 1 disk thread will read  and write  from/to both 2.5 disks. I temporary disabled LZ4 compression for the write tests.
 # echo '999999999999999999 ** 999999999999999999' | python  # dd if=/dev/ada0 of=/dev/null bs=1M  # dd if=/dev/ada1 of=/dev/null bs=1M  # dd if=/dev/zero of=/data/FILE bs=1M
POWER CPU LOAD I/O LOAD 10.4 W IDLE IDLE 12.9 W IDLE 1 DISK READ Thread(s) 14.3 W IDLE 1 DISK READ Thread(s) + 1 DISK WRITE Thread(s) 17.2 W IDLE 3 DISK READ Thread(s) + 3 DISK WRITE Thread(s) 11.0 W 8 CPU Thread(s) IDLE 13.4 W 8 CPU Thread(s) 1 DISK READ Thread(s) 15.0 W 8 CPU Thread(s) 1 DISK READ Thread(s) + 1 DISK WRITE Thread(s) 17.8 W 8 CPU Thread(s) 3 DISK READ Thread(s) + 3 DISK WRITE Thread(s)
That’s not much remembering that 6W TDP power motherboard ASRock N3150B-ITX with just single Maxtor M3 4TB 2.5 USB 3.0 drive used 16.0W with CPU and I/O loaded. Only 1.8W more (on loaded system) with redundancy on two 2.5 disks.
The crypto FreeBSD kernel module was able to squeeze about 68MB/s of random data from /dev/random as this CPU has built in hardware AES-NI acceleration. Note to Linux users – the /dev/random and /dev/urandom are the same thing on FreeBSD. I used both dd(8) and pv(1) commands for this simple test. I made two tests with powerd(8) enabled and disabled to check the difference between CPU speed at 1.2GHz and at 2.5GHz with Turbo mode.
Full speed with Turbo enabled (note 2001 instead of 2000 for CPU frequency)..
# /etc/rc.d/powerd stop Stopping powerd. Waiting for PIDS: 1486. % sysctl dev.cpu.0.freq dev.cpu.0.freq: 2001 % which pv pv: aliased to pv -t -r -a -b -W -B 1048576 % dd /dev/null 1.91GiB 0:00:31 [68.7MiB/s] [68.1MiB/s] 265+0 records in 265+0 records out 2222981120 bytes transferred in 33.566154 secs (70226864 bytes/sec) ^C
CPU limited to 1.2GHz with powerd(8) daemon was able to squeeze about 24MB/s.
# service powerd start Starting powerd. % which pv pv: aliased to pv -t -r -a -b -W -B 1048576 % dd /dev/null 568MiB 0:00:23 [25.3MiB/s] [24.7MiB/s] 71+0 records in 71+0 records out 595591168 bytes transferred in 23.375588 secs (25479195 bytes/sec ^C
Below I will show you the data from dmesg(8) about the used USB and 2.5 drives.
The dmesg(8) information for the SanDisk Fit USB 2.0 16GB drive.
# grep da0 /var/run/dmesg.boot da0 at umass-sim1 bus 1 scbus3 target 0 lun 0 da0: Removable Direct Access SPC-4 SCSI device da0: Serial Number 4C530002030502100093 da0: 400.000MB/s transfers da0: 14663MB (30031250 512 byte sectors) da0: quirks=0x2
… and two Seagate BarraCuda 5TB 2.5 drives.
# grep ada /var/run/dmesg.boot ada0 at ahcich0 bus 0 scbus0 target 0 lun 0 ada0: ACS-3 ATA SATA 3.x device ada0: Serial Number WCJ0DRJE ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) ada0: Command Queueing enabled ada0: 4769307MB (9767541168 512 byte sectors) ada1 at ahcich1 bus 0 scbus1 target 0 lun 0 ada1: ACS-3 ATA SATA 3.x device ada1: Serial Number WCJ0213S ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) ada1: Command Queueing enabled ada1: 4769307MB (9767541168 512 byte sectors)
The whole /var/run/dmesg.boot content (without disks) is shown below.
# cat /var/run/dmesg.boot Copyright (c) 1992-2018 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 11.2-RELEASE-p7 #0: Tue Dec 18 08:29:33 UTC 2018 firstname.lastname@example.org:/usr/obj/usr/src/sys/GENERIC amd64 FreeBSD clang version 6.0.0 (tags/RELEASE_600/final 326565) (based on LLVM 6.0.0) VT(vga): resolution 640x480 CPU: Intel(R) Celeron(R) CPU J3355 @ 2.00GHz (1996.88-MHz K8-class CPU) Origin="GenuineIntel" Id=0x506c9 Family=0x6 Model=0x5c Stepping=9 Features=0xbfebfbff Features2=0x4ff8ebbf AMD Features=0x2c100800 AMD Features2=0x101 Structured Extended Features=0x2294e283 XSAVE Features=0xf VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr TSC: P-state invariant, performance statistics real memory = 4294967296 (4096 MB) avail memory = 3700518912 (3529 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: WARNING: L1 data cache covers less APIC IDs than a core 0 < 1 FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs FreeBSD/SMP: 1 package(s) x 2 core(s) ioapic0 irqs 0-119 on motherboard SMP: AP CPU #1 Launched! Timecounter "TSC" frequency 1996877678 Hz quality 1000 random: entropy device external interface kbd1 at kbdmux0 netmap: loaded module module_register_init: MOD_LOAD (vesa, 0xffffffff80ff4580, 0) error 19 random: registering fast source Intel Secure Key RNG random: fast provider: "Intel Secure Key RNG" nexus0 vtvga0: on motherboard cryptosoft0: on motherboard acpi0: on motherboard unknown: I/O range not supported cpu0: on acpi0 cpu1: on acpi0 attimer0: port 0x40-0x43,0x50-0x53 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 atrtc0: port 0x70-0x77 on acpi0 atrtc0: Warning: Couldn't map I/O. atrtc0: registered as a time-of-day clock, resolution 1.000000s Event timer "RTC" frequency 32768 Hz quality 0 hpet0: iomem 0xfed00000-0xfed003ff irq 8 on acpi0 Timecounter "HPET" frequency 19200000 Hz quality 950 Event timer "HPET" frequency 19200000 Hz quality 550 Event timer "HPET1" frequency 19200000 Hz quality 440 Event timer "HPET2" frequency 19200000 Hz quality 440 Event timer "HPET3" frequency 19200000 Hz quality 440 Event timer "HPET4" frequency 19200000 Hz quality 440 Event timer "HPET5" frequency 19200000 Hz quality 440 Event timer "HPET6" frequency 19200000 Hz quality 440 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: port 0x408-0x40b on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 vgapci0: port 0xf000-0xf03f mem 0x90000000-0x90ffffff,0x80000000-0x8fffffff irq 19 at device 2.0 on pci0 vgapci0: Boot video device hdac0: mem 0x91210000-0x91213fff,0x91000000-0x910fffff irq 25 at device 14.0 on pci0 pci0: at device 15.0 (no driver attached) ahci0: port 0xf090-0xf097,0xf080-0xf083,0xf060-0xf07f mem 0x91214000-0x91215fff,0x91218000-0x912180ff,0x91217000-0x912177ff irq 19 at device 18.0 on pci0 ahci0: AHCI v1.31 with 2 6Gbps ports, Port Multiplier supported ahcich0: at channel 0 on ahci0 ahcich1: at channel 1 on ahci0 pcib1: irq 22 at device 19.0 on pci0 pci1: on pcib1 pcib2: irq 20 at device 19.2 on pci0 pci2: on pcib2 re0: port 0xe000-0xe0ff mem 0x91104000-0x91104fff,0x91100000-0x91103fff irq 20 at device 0.0 on pci2 re0: Using 1 MSI-X message re0: Chip rev. 0x4c000000 re0: MAC rev. 0x00000000 miibus0: on re0 rgephy0: PHY 1 on miibus0 rgephy0: none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow re0: Using defaults for TSO: 65518/35/2048 re0: Ethernet address: 70:85:c2:3f:53:41 re0: netmap queues/slots: TX 1/256, RX 1/256 xhci0: mem 0x91200000-0x9120ffff irq 17 at device 21.0 on pci0 xhci0: 32 bytes context size, 64-bit DMA usbus0 on xhci0 usbus0: 5.0Gbps Super Speed USB v3.0 isab0: at device 31.0 on pci0 isa0: on isab0 acpi_button0: on acpi0 acpi_tz0: on acpi0 atkbdc0: at port 0x60,0x64 on isa0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] ppc0: cannot reserve I/O port range est0: on cpu0 est1: on cpu1 ZFS filesystem version: 5 ZFS storage pool version: features support (5000) Timecounters tick every 1.000 msec hdacc0: at cad 0 on hdac0 hdaa0: at nid 1 on hdacc0 ugen0.1: at usbus0 uhub0: on usbus0 pcm0: at nid 21 and 24,26 on hdaa0 pcm1: at nid 20 and 25 on hdaa0 pcm2: at nid 27 on hdaa0 hdacc1: at cad 2 on hdac0 hdaa1: at nid 1 on hdacc1 pcm3: at nid 3 on hdaa1 uhub0: 15 ports with 15 removable, self powered ugen0.2: at usbus0 uhub1 on uhub0 uhub1: on usbus0 uhub1: 4 ports with 4 removable, self powered Trying to mount root from zfs:zroot/ROOT/default ... random: unblocking device. re0: link state changed to DOWN
ZFS Pool Configuration
To get higher LZ4 compression ratio I use larger blocksize (1MB) on this ZFS mirror pool. Here is the ZFS pool status.
% zpool status data pool: data state: ONLINE scan: scrub repaired 0 in 44h14m with 0 errors on Mon Feb 11 07:13:42 2019 config: NAME STATE READ WRITE CKSUM data ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 label/WCJ0213S ONLINE 0 0 0 label/WCJ0DRJE ONLINE 0 0 0 errors: No known data errors
I get 4% compression (1.04x) on that ZFS pool. Its about 80% filled with lots of movies and photos so while such compression ratio may not be great it gives a lot of space. For example 4% of 4TB of data is about 160GB of ‘free’ space.
% zfs get compressratio data NAME PROPERTY VALUE SOURCE data compressratio 1.04x -
Here is the ZFS pool configuration.
# zpool history History for 'data': 2018-11-12.01:18:33 zpool create data mirror /dev/label/WCJ0229Z /dev/label/WCJ0DPHF 2018-11-12.01:19:11 zfs set mountpoint=none data 2018-11-12.01:19:16 zfs set compression=lz4 data 2018-11-12.01:19:21 zfs set atime=off data 2018-11-12.01:19:34 zfs set primarycache=metadata data 2018-11-12.01:19:40 zfs set secondarycache=metadata data 2018-11-12.01:19:45 zfs set redundant_metadata=most data 2018-11-12.01:19:51 zfs set recordsize=1m data (...)
We do not need redundant_metadata as we already have two disks, its useful only on single disks configurations.
Self Solution Cost
As in the earlier post I will again calculate how much energy this server would consume. Currently 1kWh of power costs about $0.20 in Europe/Poland (rounded up). This means that running computer with 1000W power usage for 1 hour would cost you $0.20 on electricity bill. This system uses 10.4W idle and 12.9W when single disk read occurs. For most of the time server will be idle so I assume 11.0W average for the pricing purposes.
That would cost us $0.0022 for 11.0W device running for 1 hour.
Below you will also find calculations for 1 day (24x multiplier), 1 year (another 365.25x multiplier) and 3 years (another 3x multiplier).
COST TIME $0.0022 1 HOUR(S) $0.0528 1 DAY(S) $19.285 1 YEAR(S) $57.856 3 YEAR(S) $96.426 5 YEAR(S)
Combining that with server cost ($350) we get TCO for our self hosted 5TB storage service.
COST TIME $369.29 1 YEAR(S) $407.86 3 YEAR(S) $446.43 5 YEAR(S)
Our total 3 years TCO is $407.86 and 5 years is $446.43. Its for running system non-stop. We can also implement features like Wake On LAN to limit that power usage even more.
Cloud Storage Prices
This time after searching for cheapest cloud based storage I found these services.
- Amazon Drive
- Amazon S3 Glacier Storage
- Backblaze B2 Cloud Storage
- Google One
Here is its cost summarized for 1 year period for 5TB of data.
PRICE TIME SERVICE $300 1 YEAR(S) Amazon Drive $310 1 YEAR(S) Google One $240 1 YEAR(S) Amazon S3 Glacier Storage $450 1 YEAR(S) Backblaze B2 Cloud Storage
For the Backblaze B2 Cloud Storage I assumed average between upload/download price because upload is two times cheaper then download.
Here is its cost summarized for 3 year period for 5TB of data.
PRICE TIME SERVICE $900 3 YEAR(S) Amazon Drive $930 3 YEAR(S) Google One $720 3 YEAR(S) Amazon S3 Glacier Storage $1350 3 YEAR(S) Backblaze B2 Cloud Storage
Here is its cost summarized for 5 year period for 5TB of data.
PRICE TIME SERVICE $1500 5 YEAR(S) Amazon Drive $1550 5 YEAR(S) Google One $1200 5 YEAR(S) Amazon S3 Glacier Storage $2250 5 YEAR(S) Backblaze B2 Cloud Storage
Now lets compare costs of our own server to various cloud services.
If we would run our server for just 1 year the price will be similar.
PRICE TIME SERVICE $369 1 YEAR(S) Self Build NAS $300 1 YEAR(S) Amazon Drive $310 1 YEAR(S) Google One $240 1 YEAR(S) Amazon S3 Glacier Storage $450 1 YEAR(S) Backblaze B2 Cloud Storage
It gets interesting when we compare 3 years costs. Its two times cheaper to self host our own server then use cloud services. One may argue that clouds are located in many places but even if we would buy two such boxes and put one – for example in our friends place at Jamaica – or other parts of the world.
PRICE TIME SERVICE $408 3 YEAR(S) Self Build NAS $528 3 YEAR(S) Self Build NAS (assuming one of the drives failed) $900 3 YEAR(S) Amazon Drive $930 3 YEAR(S) Google One $720 3 YEAR(S) Amazon S3 Glacier Storage $1350 3 YEAR(S) Backblaze B2 Cloud Storage
… but with 5 years using cloud service instead of self hosted NAS solution is 3-5 times more expensive … and these were the cheapest cloud services I was able to find. I do not even want to know how much would it cos on Dropbox for example 🙂
PRICE TIME SERVICE $447 5 YEAR(S) Self Build NAS $567 5 YEAR(S) Self Build NAS (assuming one of the drives failed) $1500 5 YEAR(S) Amazon Drive $1550 5 YEAR(S) Google One $1200 5 YEAR(S) Amazon S3 Glacier Storage $2250 5 YEAR(S) Backblaze B2 Cloud Storage
… and ‘anywhere’ access is not an argument for cloud services because you can get external IP address for you NAS or use Dynamic DNS – for free. You may also wonder why I compare such ‘full featured NAS’ with S3 storage … well with rclone (rsync for cloud storage) you are able to synchronize your files with almost anything 🙂
Not to mention how much more privacy you have with keeping all your data to yourself … but that is priceless.
You can also setup a lot more services on such hardware – like FreeNAS with Bhyve/Jails virtualization … or Nextcloud instance … or Syncthing … while cloud storage is only that – a storage in the cloud.
Not sure what else could I include in this article. If you have an idea what else could I cover then let me know.