I was really disappointed when I got to know that the FreeBSD based TrueNAS CORE storage appliance – owned and developed by iXsystems – will be moved into the ‘maintenance’ mode and that (Debian) Linux based TrueNAS SCALE is the future of their offerings. Later iXsystems after all the backslash started to ‘calm down’ that situation stating that FreeBSD based TrueNAS CORE is feature complete and that they do not have plans to kill the FreeBSD based solution … but we all know that TrueNAS CORE uses – for example – abandoned by creator and by the rest of the World – the iocage FreeBSD Jails management framework – its been 5 or more years since the iocage was last touched for any reason.
One could expect a ‘move’ into plain FreeBSD Jails or something that is alive and kicking such as BastilleBSD or something Pot/Nomad related as Klara posted here – Cluster Provisioning on FreeBSD – not so long ago.
I was the first one to encourage my family and friends to just use TrueNAS CORE for their data – because I knew that ZFS being 1st class citizen in the FreeBSD world – this is the perfect and stable combination. I can not say the same about Linux ecosystem where CDDL license that ZFS uses is treated as a curse to say the least and that even ‘the’ Linux guru such as Linus himself does not understand the difference between OpenZFS and Oracle ZFS – Do Not Use ZFS on Linux: Linus Torvalds – which does not feel right – its really hard to experience all of this.
Having these in mind – Linux (Debian or not) is not (and will never be) a real ZFS ‘home’ with all the consequences. One of the reasons – I believe – the iXsystems is betting on Linux instead of FreeBSD here is the container world. The Docker and/or Podman solutions – the ones that are more then a decade late to the containers game as Docker is with us since 2014 – the FreeBSD Jails are with is since year 2000. You do the math.
There are also other aspects as iXsystems advertises the TrueNAS SCALE as the solution for ‘scale-out’ attitude while the FreeBSD version TrueNAS CORE only as ‘scale-up’ one. I also dig that topic – and to say the least – I was not amused. Lets try to split all these parts into understandable puzzle.
Boot Process
The first image below is the AlmaLinux 9.x RHEL clone boot output – seems pretty tidy and clean – assuming for systemd(1) based system.
Now the TrueNAS SCALE boot output.
I really like that iXsystems implemented netif FreeBSD concept on a Debian Linux system – named as ix-netif.service on the screen π
Finally after some period of time we are able to see the ‘final’ management menu.
… and it works as desired.
Maybe I should not worry about it too much as long as the web/browser management is finally available in the end …
ZFS Boot Environments
While FreeBSD (and as a result TrueNAS CORE) have native ZFS Boot Environments support and managed by bectl(8) and beadm(8) tools the situation is quite different on the TrueNAS SCALE front.
The TrueNAS SCALE system itself – being based on Debian Linux – does indeed is installed as root on ZFS and with support for ZFS Boot Environments … but little differently.
Instead of using command line tools like beadm(8) or bectl(8) as You would do on FreeBSD – with TrueNAS SCALE you have to use WebUI instead.
Maybe in the future they will implement some kind of command line tool.
Security
There was time when I was part of the team that needed to decide if we use the then named FreeNAS or to use plain FreeBSD system instead for the data. That decision was on of the core decision we made as a team for the FreeBSD Enterprise 1 PB Storage project.
After I started to play with – then available FreeNAS 11.2-U3 version – I was far from satisfied with its security – as described in the details – FreeNAS 11.2-U3 Vulnerabilities – here.
I also wanted to update the packages with pkg(8) command to newer versions – so that the security vulnerabilities would be fixed – but apparently that broke entire middlewared(8) and all other stuff and renders a then available FreeNAS appliance unusable. In the end me and my team went with plain FreeBSD system and we do not regret our decision to this day.
I wanted to check if its better for the latest – most up to date version – TrueNAS CORE 13.0-U6.1 – but not – its very similar.
root@truenas[~]# pkg audit -F vulnxml file up-to-date apache24-2.4.57 is vulnerable: Apache httpd -- multiple vulnerabilities CVE: CVE-2024-38709 CVE: CVE-2024-24795 CVE: CVE-2024-27316 WWW: https://vuxml.FreeBSD.org/freebsd/8e6f684b-f333-11ee-a573-84a93843eb75.html Apache httpd -- Multiple vulnerabilities CVE: CVE-2023-31122 CVE: CVE-2023-43622 CVE: CVE-2023-45802 WWW: https://vuxml.FreeBSD.org/freebsd/f923205f-6e66-11ee-85eb-84a93843eb75.html rclone-1.57.0_2 is vulnerable: rclone -- Multiple vulnerabilities CVE: CVE-2023-48795 CVE: CVE-2023-45286 WWW: https://vuxml.FreeBSD.org/freebsd/b5e22ec5-bc4b-11ee-b0b5-b42e991fc52e.html python39-3.9.16 is vulnerable: Python -- multiple vulnerabilities CVE: CVE-2023-40217 WWW: https://vuxml.FreeBSD.org/freebsd/a57472ba-4d84-11ee-bf05-000c29de725b.html Python -- multiple vulnerabilities CVE: CVE-2023-24329 CVE: CVE-2023-0466 CVE: CVE-2023-0465 CVE: CVE-2023-0464 CVE: CVE-2023-0286 CVE: CVE-2023-2650 CVE: CVE-2022-4303 WWW: https://vuxml.FreeBSD.org/freebsd/d86becfe-05a4-11ee-9d4a-080027eda32c.html py39-configobj-5.0.6_1 is vulnerable: py39-configobj -- vulnerable to Regular Expression Denial of Service CVE: CVE-2023-26112 WWW: https://vuxml.FreeBSD.org/freebsd/de970aef-d60e-466b-8e30-1ae945a047f1.html git-lite-2.34.1 is vulnerable: git -- Multiple vulnerabilities CVE: CVE-2023-29007 CVE: CVE-2023-25652 WWW: https://vuxml.FreeBSD.org/freebsd/d2c6173f-e43b-11ed-a1d7-002590f2a714.html git -- Multiple vulnerabilities CVE: CVE-2022-39260 CVE: CVE-2022-39253 WWW: https://vuxml.FreeBSD.org/freebsd/2523bc76-4f01-11ed-929b-002590f2a714.html curl-8.0.1 is vulnerable: curl -- multiple vulnerabilities CVE: CVE-2023-28322 CVE: CVE-2023-28321 CVE: CVE-2023-28320 CVE: CVE-2023-28319 WWW: https://vuxml.FreeBSD.org/freebsd/a4f8bb03-f52f-11ed-9859-080027083a05.html curl -- OCSP verification bypass with TLS session reuse CVE: CVE-2024-0853 WWW: https://vuxml.FreeBSD.org/freebsd/02e33cd1-c655-11ee-8613-08002784c58d.html curl -- SOCKS5 heap buffer overflow CVE: CVE-2023-38545 WWW: https://vuxml.FreeBSD.org/freebsd/d6c19e8c-6806-11ee-9464-b42e991fc52e.html curl -- HTTP headers eat all memory CVE: CVE-2023-38039 WWW: https://vuxml.FreeBSD.org/freebsd/833b469b-5247-11ee-9667-080027f5fec9.html py39-beaker-1.12.1 is vulnerable: py-beaker -- arbitrary code execution vulnerability CVE: CVE-2013-7489 WWW: https://vuxml.FreeBSD.org/freebsd/b54abe9d-7024-4d10-98b2-180cf1717766.html openssh-portable-8.8.p1_1,1 is vulnerable: OpenSSH -- remote code execution via a forwarded agent socket CVE: CVE-2023-38408 WWW: https://vuxml.FreeBSD.org/freebsd/887eb570-27d3-11ee-adba-c80aa9043978.html minio-2021.12.27.07.23.18_1 is vulnerable: MinIO -- unprivileged users can create service accounts for admin users CVE: CVE-2022-24842 WWW: https://vuxml.FreeBSD.org/freebsd/8e20430d-a72b-11ed-a04f-40b034455553.html libxml2-2.9.12 is vulnerable: libxml2 -- multiple vulnerabilities CVE: CVE-2023-29469 CVE: CVE-2023-28484 WWW: https://vuxml.FreeBSD.org/freebsd/0bd7f07b-dc22-11ed-bf28-589cfc0f81b0.html py39-markdown2-2.3.6 is vulnerable: py-markdown2 -- regular expression denial of service vulnerability CVE: CVE-2021-26813 WWW: https://vuxml.FreeBSD.org/freebsd/c9b3324f-8e03-4ae3-89ce-8098cdc5bfa9.html py-markdown2 -- XSS vulnerability CVE: CVE-2020-11888 WWW: https://vuxml.FreeBSD.org/freebsd/cf6f3465-e996-4672-9458-ce803f29fdb7.html squashfs-tools-4.3_1 is vulnerable: squashfs-tools -- Integer overflow CVE: CVE-2015-4645 WWW: https://vuxml.FreeBSD.org/freebsd/317487c6-85ca-11eb-80fa-14dae938ec40.html pixman-0.40.0_1 is vulnerable: pixman -- heap overflow CVE: CVE-2022-44638 WWW: https://vuxml.FreeBSD.org/freebsd/b278783f-5c1d-11ed-a21f-001fc69cd6dc.html py39-sentry-sdk-1.5.12 is vulnerable: py39-sentry-sdk -- sensitive cookies leak CVE: CVE-2023-28117 WWW: https://vuxml.FreeBSD.org/freebsd/15dae5cc-9ee6-4577-a93e-2ab57780e707.html c-ares-1.17.2 is vulnerable: dns/c-ares -- malformatted file causes application crash CVE: CVE-2024-25629 WWW: https://vuxml.FreeBSD.org/freebsd/255bf44c-d298-11ee-9c27-40b034429ecf.html py39-setuptools-57.0.0 is vulnerable: py39-setuptools -- denial of service vulnerability CVE: CVE-2022-40897 WWW: https://vuxml.FreeBSD.org/freebsd/1b38aec4-4149-4c7d-851c-3c4de3a1fbd0.html 23 problem(s) in 16 installed package(s) found.
Now – lets compare that to up-to-date FreeBSD 14.0-RELEASE system with the same packages installed.
freebsd # pkg audit -F Fetching vuln.xml.xz: 100% 1 MiB 545.9kB/s 00:02 py39-beaker-1.12.1 is vulnerable: py-beaker -- arbitrary code execution vulnerability CVE: CVE-2013-7489 WWW: https://vuxml.FreeBSD.org/freebsd/b54abe9d-7024-4d10-98b2-180cf1717766.html 1 problem(s) in 1 installed package(s) found.
Pretty big difference … and after trying to force pkg upgrade command on the TrueNAS CORE host it renders the system unusable as shown below after reboot(8) command.
Its unsupported – so not do not blame iXsystems for it.
There are also places in which You will have convince the Security/Compliance teams that all these vulnerabilities are not applicable. I remember I had to do the same and these discussions were like:
ME: This vulnerable package is just a dependency and is not used in actual solution. SEC: So just remove it. ME: I can not remove it because that will break entire package. SEC: So it is used then?
Discussions like that. The other problem is open listening ports – this is how it looks like for current TrueNAS 13.0-U6.1 version.
root@truenas[~]# sockstat -l -4
USER COMMAND PID FD PROTO LOCAL ADDRESS FOREIGN ADDRESS
root python3.9 1223 5 udp4 239.255.255.250:3702 *:*
root python3.9 1223 6 udp4 *:63280 *:*
root python3.9 1223 7 udp4 10.1.1.11:3702 *:*
root python3.9 1223 8 tcp4 10.1.1.11:5357 *:*
avahi avahi-daem 1204 13 udp4 *:5353 *:*
avahi avahi-daem 1204 14 udp4 *:41119 *:*
www nginx 1086 6 tcp4 *:443 *:*
www nginx 1086 8 tcp4 *:80 *:*
root nginx 1084 6 tcp4 *:443 *:*
root nginx 1084 8 tcp4 *:80 *:*
ntpd ntpd 998 21 udp4 *:123 *:*
ntpd ntpd 998 22 udp4 10.1.1.11:123 *:*
ntpd ntpd 998 25 udp4 127.0.0.1:123 *:*
root syslog-ng 932 19 udp4 127.0.0.1:1031 *:*
root python3.9 164 28 tcp4 *:6000 *:*
Of course its OK that 80 and 443 are open, but there are also 6000, 63280, 3702, 5357, 5353, 41119 and 123. While 123 can be omitted (for ntpd(8) daemon) the other ones? I could expect one additional open port for (REST) API or for some other features or for TrueCommand some kind of management/connection, etc. but that many?
It would be another backslash of questions from the Security/Compliance team. One of them would be:
SEC: Python have multiple vulnerabilities and these services listen at five additional ports, what do they do and can they be disabled? What they are used for?
Maybe iXsystems could do some additional documentation about what they actually do and why they are needed – but that would still left vulnerable Python based services listening on multiple ports …
Difference
Now – for the real question – is there a real difference between FreeBSD based TrueNAS CORE and Linux based TrueNAS SCALE systems?
The ones that I found out were these:
- The FreeBSD version uses Bhyve for virtualization while Linux uses KVM subsystem.
- The FreeBSD version uses Jails for containers while Linux uses Docker subsystem.
Thats it. Nothing more and nothing else … besides that the ‘base’ for TrueNAS CORE is FreeBSD and the ‘base’ for TrueNAS SCALE is Debian Linux with systemd(1) daemon.
What is more interesting – that in the FreeNAS Corral release – based on FreeBSD – it was perfectly fine to have Docker (now Apps) on the FreeBSD platform as shown on the screenshot below π
Features
I really liked the concept of ‘scale-out’ for the TrueNAS SCALE systems – TrueNAS SCALE Clustering Overview – that with more systems You have – You can spread both the data and load across multiple machines … but after I started digging in this topic – it was just that TrueNAS SCALE allowed to have a distributed GlusterFS filesystem spread across TrueNAS SCALE boxes.
My disappointment would probably not be that big if I would earlier not tried the same GlusterFS distributed filesystem successfully on FreeBSD.
My guides of GlusterFS on FreeBSD are listed below.
… and the best yet to come is next π
Its not a FreeBSD limitation for the GlusterFS filesystem – its the iXsystems limitation.
Before I used TrueNAS SCALE I believed that there is some kinda graphical or GUI way to configure/extend that local storage into distributed state with GlusterFS filesystem … I was so wrong π
There is zero GUI options to configure GlusterFS filesystem over multiple TrueNAS SCALE boxes.
The only possible guide on how to spread that GlusterFS filesystem across TrueNAS SCALE system were … command line based – Six Node Gluster Cluster with TrueNAS SCALE – exactly the same as mine guides on FreeBSD.
I was thinking – there has to be GUI options for such an advertised option as ‘scale-out’ distributed GlusterFS filesystem across TrueNAS SCALE systems … but the answer I found was even more disappointing – You just need to use the manager called TrueCommand – there You have all the needed options – as simple as that. Its free to use to manage up to 50 drives. Beyond that it will require some paid plan.
Kubernetes
As FreeBSD Bhyve hypervisor supports memory ballooning it would be brain dead easy (and cheap on RAM) to just to add another VM (with AlmaLinux for example) to be ‘the’ place for Docker/Podman containers (or Applications as iXsystems is calling it). The 30-50 MB of RAM overhead is negligible as FreeBSD is often more memory efficient on RAM then Linux.
Competition
I think that is exactly what is going to happen also this time – I do not wish them that. When FreeNAS/TrueNAS was based on FreeBSD – then it offered mature and stable OS – along with perfectly integrated ZFS. A unique package that there was almost no competition for. When they will move to Linux – they must compete with the rest of the world with similar/different solutions such as:
- Amahi
- OpenMediaVault
- Rockstor
- Openfiler
- Nexenta (Illumos)
- napp-it (Illumos)
- CryptoNAS
- OpenDedupe
- OpenATTIC
- ESOS
- PetaSAN
- OpenIO
- Open vStorage
- OviOS
- TurnKey File Server
Pretty crowded I must say – and You will have to be REALLY outstanding to make the difference.
Community Forums
While I rarely used them – the TrueNAS Community forums were full features and professional.
This is how they looked alike.
Now … recently iXsystems announced new forums to take place – a simplified to minimum ones which look like this.
Maybe I am oldschool (of course I am) but it just feels like a downgrade.
Summary
The iXsystems had a bumpy road in the FreeBSD land. Besides the BSDi lawsuit there were other attempts on making FreeBSD their ‘base’ for many things. Lets start with PC-BSD for example – the FreeBSD desktop distribution that had some of the very future acknowledged features – such as PBI packages. Today the same functionality is provided as Flatpack or Snap packages … yes PC-BSD provided the same a decade earlier … and it was not that fancy then – and often criticized. Now as Linux started doing that – it feels like another god of computing that we can not live without … exactly like Docker/Podman being ‘late’ to the containers party when FreeBSD Jails delivered since 2000 – yet now Linux is ‘the’ container technology that whole World jerks off too – while FreeBSD Jails is remembered as some kind of museum of history.
I even started in the PC-BSD new logo contest if I recall correctly – but it was way too long ago to remember all the details … or was it just another wallpaper …
Later iXsystems tried to – I believe – move forward with PC-BSD as TrueOS – but that also did not played that well. They later moved to Project Trident – still FreeBSD based – and then moved to Void Linux for the base of Project Trident system … and some time later iXsystems abandoned altogether killing the project.
While this post ‘may’ seem like some kind of attack on the iXsystems company – it is not. Its just I got used to some world view that iXsystems would always be there and that they would support FreeBSD system no matter what … I was wrong. Business is business and iXsystems company needs to make money (that is what companies are for anyway) and they need to pick what is the best for them. I wish them all the best as they brought a lot of attention and interest into the FreeBSD world – but if Linux is their current focus – then its OK with me – the iXsystems are very clear about it – no need to make their new read more bumpy then its already is.
I just hope that the current iXsystems bet on Linux systems and not on FreeBSD will not end the same as it ended with PC-BSD or TrueOS or Project Trident in the past … in forgotten path.
Hope that helps.