TrueNAS CORE versus TrueNAS SCALE

I was really disappointed when I got to know that the FreeBSD based TrueNAS CORE storage appliance – owned and developed by iXsystems – will be moved into the ‘maintenance’ mode and that (Debian) Linux based TrueNAS SCALE is the future of their offerings. Later iXsystems after all the backslash started to ‘calm down’ that situation stating that FreeBSD based TrueNAS CORE is feature complete and that they do not have plans to kill the FreeBSD based solution … but we all know that TrueNAS CORE uses – for example – abandoned by creator and by the rest of the World – the iocage FreeBSD Jails management framework – its been 5 or more years since the iocage was last touched for any reason.

One could expect a ‘move’ into plain FreeBSD Jails or something that is alive and kicking such as BastilleBSD or something Pot/Nomad related as Klara posted here – Cluster Provisioning on FreeBSD – not so long ago.

I was the first one to encourage my family and friends to just use TrueNAS CORE for their data – because I knew that ZFS being 1st class citizen in the FreeBSD world – this is the perfect and stable combination. I can not say the same about Linux ecosystem where CDDL license that ZFS uses is treated as a curse to say the least and that even ‘the’ Linux guru such as Linus himself does not understand the difference between OpenZFS and Oracle ZFS – Do Not Use ZFS on Linux: Linus Torvalds – which does not feel right – its really hard to experience all of this.

Having these in mind – Linux (Debian or not) is not (and will never be) a real ZFS ‘home’ with all the consequences. One of the reasons – I believe – the iXsystems is betting on Linux instead of FreeBSD here is the container world. The Docker and/or Podman solutions – the ones that are more then a decade late to the containers game as Docker is with us since 2014 – the FreeBSD Jails are with is since year 2000. You do the math.

There are also other aspects as iXsystems advertises the TrueNAS SCALE as the solution for ‘scale-out’ attitude while the FreeBSD version TrueNAS CORE only as ‘scale-up’ one. I also dig that topic – and to say the least – I was not amused. Lets try to split all these parts into understandable puzzle.

Boot Process

The first image below is the AlmaLinux 9.x RHEL clone boot output – seems pretty tidy and clean – assuming for systemd(1) based system.

Now the TrueNAS SCALE boot output.

I really like that iXsystems implemented netif FreeBSD concept on a Debian Linux system – named as ix-netif.service on the screen 🙂

Finally after some period of time we are able to see the ‘final’ management menu.

… and it works as desired.

Maybe I should not worry about it too much as long as the web/browser management is finally available in the end …

ZFS Boot Environments

While FreeBSD (and as a result TrueNAS CORE) have native ZFS Boot Environments support and managed by bectl(8) and beadm(8) tools the situation is quite different on the TrueNAS SCALE front.

The TrueNAS SCALE system itself – being based on Debian Linux – does indeed is installed as root on ZFS and with support for ZFS Boot Environments … but little differently.

Instead of using command line tools like beadm(8) or bectl(8) as You would do on FreeBSD – with TrueNAS SCALE you have to use WebUI instead.

Maybe in the future they will implement some kind of command line tool.

Security

There was time when I was part of the team that needed to decide if we use the then named FreeNAS or to use plain FreeBSD system instead for the data. That decision was on of the core decision we made as a team for the FreeBSD Enterprise 1 PB Storage project.

After I started to play with – then available FreeNAS 11.2-U3 version – I was far from satisfied with its security – as described in the details – FreeNAS 11.2-U3 Vulnerabilities – here.

I also wanted to update the packages with pkg(8) command to newer versions – so that the security vulnerabilities would be fixed – but apparently that broke entire middlewared(8) and all other stuff and renders a then available FreeNAS appliance unusable. In the end me and my team went with plain FreeBSD system and we do not regret our decision to this day.

I wanted to check if its better for the latest – most up to date version – TrueNAS CORE 13.0-U6.1 – but not – its very similar.

root@truenas[~]# pkg audit -F
vulnxml file up-to-date
apache24-2.4.57 is vulnerable:
  Apache httpd -- multiple vulnerabilities
  CVE: CVE-2024-38709
  CVE: CVE-2024-24795
  CVE: CVE-2024-27316
  WWW: https://vuxml.FreeBSD.org/freebsd/8e6f684b-f333-11ee-a573-84a93843eb75.html

  Apache httpd -- Multiple vulnerabilities
  CVE: CVE-2023-31122
  CVE: CVE-2023-43622
  CVE: CVE-2023-45802
  WWW: https://vuxml.FreeBSD.org/freebsd/f923205f-6e66-11ee-85eb-84a93843eb75.html

rclone-1.57.0_2 is vulnerable:
  rclone -- Multiple vulnerabilities
  CVE: CVE-2023-48795
  CVE: CVE-2023-45286
  WWW: https://vuxml.FreeBSD.org/freebsd/b5e22ec5-bc4b-11ee-b0b5-b42e991fc52e.html

python39-3.9.16 is vulnerable:
  Python -- multiple vulnerabilities
  CVE: CVE-2023-40217
  WWW: https://vuxml.FreeBSD.org/freebsd/a57472ba-4d84-11ee-bf05-000c29de725b.html

  Python -- multiple vulnerabilities
  CVE: CVE-2023-24329
  CVE: CVE-2023-0466
  CVE: CVE-2023-0465
  CVE: CVE-2023-0464
  CVE: CVE-2023-0286
  CVE: CVE-2023-2650
  CVE: CVE-2022-4303
  WWW: https://vuxml.FreeBSD.org/freebsd/d86becfe-05a4-11ee-9d4a-080027eda32c.html

py39-configobj-5.0.6_1 is vulnerable:
  py39-configobj -- vulnerable to Regular Expression Denial of Service
  CVE: CVE-2023-26112
  WWW: https://vuxml.FreeBSD.org/freebsd/de970aef-d60e-466b-8e30-1ae945a047f1.html

git-lite-2.34.1 is vulnerable:
  git -- Multiple vulnerabilities
  CVE: CVE-2023-29007
  CVE: CVE-2023-25652
  WWW: https://vuxml.FreeBSD.org/freebsd/d2c6173f-e43b-11ed-a1d7-002590f2a714.html

  git -- Multiple vulnerabilities
  CVE: CVE-2022-39260
  CVE: CVE-2022-39253
  WWW: https://vuxml.FreeBSD.org/freebsd/2523bc76-4f01-11ed-929b-002590f2a714.html

curl-8.0.1 is vulnerable:
  curl -- multiple vulnerabilities
  CVE: CVE-2023-28322
  CVE: CVE-2023-28321
  CVE: CVE-2023-28320
  CVE: CVE-2023-28319
  WWW: https://vuxml.FreeBSD.org/freebsd/a4f8bb03-f52f-11ed-9859-080027083a05.html

  curl -- OCSP verification bypass with TLS session reuse
  CVE: CVE-2024-0853
  WWW: https://vuxml.FreeBSD.org/freebsd/02e33cd1-c655-11ee-8613-08002784c58d.html

  curl -- SOCKS5 heap buffer overflow
  CVE: CVE-2023-38545
  WWW: https://vuxml.FreeBSD.org/freebsd/d6c19e8c-6806-11ee-9464-b42e991fc52e.html

  curl -- HTTP headers eat all memory
  CVE: CVE-2023-38039
  WWW: https://vuxml.FreeBSD.org/freebsd/833b469b-5247-11ee-9667-080027f5fec9.html

py39-beaker-1.12.1 is vulnerable:
  py-beaker -- arbitrary code execution vulnerability
  CVE: CVE-2013-7489
  WWW: https://vuxml.FreeBSD.org/freebsd/b54abe9d-7024-4d10-98b2-180cf1717766.html

openssh-portable-8.8.p1_1,1 is vulnerable:
  OpenSSH -- remote code execution via a forwarded agent socket
  CVE: CVE-2023-38408
  WWW: https://vuxml.FreeBSD.org/freebsd/887eb570-27d3-11ee-adba-c80aa9043978.html

minio-2021.12.27.07.23.18_1 is vulnerable:
  MinIO -- unprivileged users can create service accounts for admin users
  CVE: CVE-2022-24842
  WWW: https://vuxml.FreeBSD.org/freebsd/8e20430d-a72b-11ed-a04f-40b034455553.html

libxml2-2.9.12 is vulnerable:
  libxml2 -- multiple vulnerabilities
  CVE: CVE-2023-29469
  CVE: CVE-2023-28484
  WWW: https://vuxml.FreeBSD.org/freebsd/0bd7f07b-dc22-11ed-bf28-589cfc0f81b0.html

py39-markdown2-2.3.6 is vulnerable:
  py-markdown2 -- regular expression denial of service vulnerability
  CVE: CVE-2021-26813
  WWW: https://vuxml.FreeBSD.org/freebsd/c9b3324f-8e03-4ae3-89ce-8098cdc5bfa9.html

  py-markdown2 -- XSS vulnerability
  CVE: CVE-2020-11888
  WWW: https://vuxml.FreeBSD.org/freebsd/cf6f3465-e996-4672-9458-ce803f29fdb7.html

squashfs-tools-4.3_1 is vulnerable:
  squashfs-tools -- Integer overflow
  CVE: CVE-2015-4645
  WWW: https://vuxml.FreeBSD.org/freebsd/317487c6-85ca-11eb-80fa-14dae938ec40.html

pixman-0.40.0_1 is vulnerable:
  pixman -- heap overflow
  CVE: CVE-2022-44638
  WWW: https://vuxml.FreeBSD.org/freebsd/b278783f-5c1d-11ed-a21f-001fc69cd6dc.html

py39-sentry-sdk-1.5.12 is vulnerable:
  py39-sentry-sdk -- sensitive cookies leak
  CVE: CVE-2023-28117
  WWW: https://vuxml.FreeBSD.org/freebsd/15dae5cc-9ee6-4577-a93e-2ab57780e707.html

c-ares-1.17.2 is vulnerable:
  dns/c-ares -- malformatted file causes application crash
  CVE: CVE-2024-25629
  WWW: https://vuxml.FreeBSD.org/freebsd/255bf44c-d298-11ee-9c27-40b034429ecf.html

py39-setuptools-57.0.0 is vulnerable:
  py39-setuptools -- denial of service vulnerability
  CVE: CVE-2022-40897
  WWW: https://vuxml.FreeBSD.org/freebsd/1b38aec4-4149-4c7d-851c-3c4de3a1fbd0.html

23 problem(s) in 16 installed package(s) found.
 

Now – lets compare that to up-to-date FreeBSD 14.0-RELEASE system with the same packages installed.

freebsd # pkg audit -F
Fetching vuln.xml.xz: 100% 1 MiB 545.9kB/s 00:02 
  py39-beaker-1.12.1 is vulnerable:
  py-beaker -- arbitrary code execution vulnerability
  CVE: CVE-2013-7489
  WWW: https://vuxml.FreeBSD.org/freebsd/b54abe9d-7024-4d10-98b2-180cf1717766.html

1 problem(s) in 1 installed package(s) found.

Pretty big difference … and after trying to force pkg upgrade command on the TrueNAS CORE host it renders the system unusable as shown below after reboot(8) command.

Its unsupported – so not do not blame iXsystems for it.

There are also places in which You will have convince the Security/Compliance teams that all these vulnerabilities are not applicable. I remember I had to do the same and these discussions were like:

ME: This vulnerable package is just a dependency and is not used in actual solution.
SEC: So just remove it.
ME: I can not remove it because that will break entire package.
SEC: So it is used then?

Discussions like that. The other problem is open listening ports – this is how it looks like for current TrueNAS 13.0-U6.1 version.

root@truenas[~]# sockstat -l -4
USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS      
root     python3.9  1223  5  udp4   239.255.255.250:3702  *:*
root     python3.9  1223  6  udp4   *:63280               *:*
root     python3.9  1223  7  udp4   10.1.1.11:3702        *:*
root     python3.9  1223  8  tcp4   10.1.1.11:5357        *:*
avahi    avahi-daem 1204  13 udp4   *:5353                *:*
avahi    avahi-daem 1204  14 udp4   *:41119               *:*
www      nginx      1086  6  tcp4   *:443                 *:*
www      nginx      1086  8  tcp4   *:80                  *:*
root     nginx      1084  6  tcp4   *:443                 *:*
root     nginx      1084  8  tcp4   *:80                  *:*
ntpd     ntpd       998   21 udp4   *:123                 *:*
ntpd     ntpd       998   22 udp4   10.1.1.11:123         *:*
ntpd     ntpd       998   25 udp4   127.0.0.1:123         *:*
root     syslog-ng  932   19 udp4   127.0.0.1:1031        *:*
root     python3.9  164   28 tcp4   *:6000                *:*

Of course its OK that 80 and 443 are open, but there are also 6000, 63280, 3702, 5357, 5353, 41119 and 123. While 123 can be omitted (for ntpd(8) daemon) the other ones? I could expect one additional open port for (REST) API or for some other features or for TrueCommand some kind of management/connection, etc. but that many?

It would be another backslash of questions from the Security/Compliance team. One of them would be:

SEC: Python have multiple vulnerabilities and these services listen at five additional ports,
     what do they do and can they be disabled? What they are used for?

Maybe iXsystems could do some additional documentation about what they actually do and why they are needed – but that would still left vulnerable Python based services listening on multiple ports …

Difference

Now – for the real question – is there a real difference between FreeBSD based TrueNAS CORE and Linux based TrueNAS SCALE systems?

The ones that I found out were these:

• The FreeBSD version uses Bhyve for virtualization while Linux uses KVM subsystem.
• The FreeBSD version uses Jails for containers while Linux uses Docker subsystem.

Thats it. Nothing more and nothing else … besides that the ‘base’ for TrueNAS CORE is FreeBSD and the ‘base’ for TrueNAS SCALE is Debian Linux with systemd(1) daemon.

What is more interesting – that in the FreeNAS Corral release – based on FreeBSD – it was perfectly fine to have Docker (now Apps) on the FreeBSD platform as shown on the screenshot below 🙂

Features

I really liked the concept of ‘scale-out’ for the TrueNAS SCALE systems – TrueNAS SCALE Clustering Overview – that with more systems You have – You can spread both the data and load across multiple machines … but after I started digging in this topic – it was just that TrueNAS SCALE allowed to have a distributed GlusterFS filesystem spread across TrueNAS SCALE boxes.

My disappointment would probably not be that big if I would earlier not tried the same GlusterFS distributed filesystem successfully on FreeBSD.

My guides of GlusterFS on FreeBSD are listed below.

• GlusterFS Cluster on FreeBSD with Ansible and GNU Parallel [2019]
• GlusterFS 8 on FreeBSD 13 [2021]

… and the best yet to come is next 🙂

Its not a FreeBSD limitation for the GlusterFS filesystem – its the iXsystems limitation.

Before I used TrueNAS SCALE I believed that there is some kinda graphical or GUI way to configure/extend that local storage into distributed state with GlusterFS filesystem … I was so wrong 🙂

There is zero GUI options to configure GlusterFS filesystem over multiple TrueNAS SCALE boxes.

The only possible guide on how to spread that GlusterFS filesystem across TrueNAS SCALE system were … command line based – Six Node Gluster Cluster with TrueNAS SCALE – exactly the same as mine guides on FreeBSD.

I was thinking – there has to be GUI options for such an advertised option as ‘scale-out’ distributed GlusterFS filesystem across TrueNAS SCALE systems … but the answer I found was even more disappointing – You just need to use the manager called TrueCommand – there You have all the needed options – as simple as that. Its free to use to manage up to 50 drives. Beyond that it will require some paid plan.

Kubernetes

As FreeBSD Bhyve hypervisor supports memory ballooning it would be brain dead easy (and cheap on RAM) to just to add another VM (with AlmaLinux for example) to be ‘the’ place for Docker/Podman containers (or Applications as iXsystems is calling it). The 30-50 MB of RAM overhead is negligible as FreeBSD is often more memory efficient on RAM then Linux.

Competition

I think that is exactly what is going to happen also this time – I do not wish them that. When FreeNAS/TrueNAS was based on FreeBSD – then it offered mature and stable OS – along with perfectly integrated ZFS. A unique package that there was almost no competition for. When they will move to Linux – they must compete with the rest of the world with similar/different solutions such as:

• Amahi
• OpenMediaVault
• Rockstor
• Openfiler
• Nexenta (Illumos)
• napp-it (Illumos)
• CryptoNAS
• OpenDedupe
• OpenATTIC
• ESOS
• PetaSAN
• OpenIO
• Open vStorage
• OviOS
• TurnKey File Server

Pretty crowded I must say – and You will have to be REALLY outstanding to make the difference.

Community Forums

While I rarely used them – the TrueNAS Community forums were full features and professional.

This is how they looked alike.

Now … recently iXsystems announced new forums to take place – a simplified to minimum ones which look like this.

Maybe I am oldschool (of course I am) but it just feels like a downgrade.

Summary

The iXsystems had a bumpy road in the FreeBSD land. Besides the BSDi lawsuit there were other attempts on making FreeBSD their ‘base’ for many things. Lets start with PC-BSD for example – the FreeBSD desktop distribution that had some of the very future acknowledged features – such as PBI packages. Today the same functionality is provided as Flatpack or Snap packages … yes PC-BSD provided the same a decade earlier … and it was not that fancy then – and often criticized. Now as Linux started doing that – it feels like another god of computing that we can not live without … exactly like Docker/Podman being ‘late’ to the containers party when FreeBSD Jails delivered since 2000 – yet now Linux is ‘the’ container technology that whole World jerks off too – while FreeBSD Jails is remembered as some kind of museum of history.

I even started in the PC-BSD new logo contest if I recall correctly – but it was way too long ago to remember all the details … or was it just another wallpaper …

Later iXsystems tried to – I believe – move forward with PC-BSD as TrueOS – but that also did not played that well. They later moved to Project Trident – still FreeBSD based – and then moved to Void Linux for the base of Project Trident system … and some time later iXsystems abandoned altogether killing the project.

While this post ‘may’ seem like some kind of attack on the iXsystems company – it is not. Its just I got used to some world view that iXsystems would always be there and that they would support FreeBSD system no matter what … I was wrong. Business is business and iXsystems company needs to make money (that is what companies are for anyway) and they need to pick what is the best for them. I wish them all the best as they brought a lot of attention and interest into the FreeBSD world – but if Linux is their current focus – then its OK with me – the iXsystems are very clear about it – no need to make their new read more bumpy then its already is.

I just hope that the current iXsystems bet on Linux systems and not on FreeBSD will not end the same as it ended with PC-BSD or TrueOS or Project Trident in the past … in forgotten path.

Hope that helps.

EOF