Today I will show you how to configure unbound(8) to block spam/malicious/malware domains at DNS level.
I will use FreeBSD for that purpose but you can use any system that unbound(8) runs on.
Earlier I used generated /etc/hosts file but that was limited in several ways. The ZSH shell will autocomplete all these blocked domains to the ssh(1)/scp(1) commands (which takes needless time and shows useless completions). Subdomains are not handled. The malicious.com is blocked but ads.malicious.com is not. You need to duplicate all those domains in the /etc/hosts file.
TL;DR
Not all people have time for my long boring stories so this is meritum of this article.
# rm -rf /var/unbound # mkdir -p /var/unbound/conf.d # chown -R unbound:unbound /var/unbound # service local_unbound setup # service local_unbound enable # service local_unbound start # mkdir /root/bin # cd # fetch -o /root/bin/unbound-blacklist-fetch.sh \ > https://raw.githubusercontent.com/vermaden/scripts/master/unbound-blacklist-fetch.sh # chmod +x /root/bin/unbound-blacklist-fetch.sh # /root/bin/unbound-blacklist-fetch.sh # service local_unbound restart # cat << BSD >> /var/cron/tabs/root > # FETCH FRESH unbound(8) BLACKLIST > 0 0 * * * /root/bin/unbound-blacklist-fetch.sh > BSD
Whole Story
The unbound(8) caching DNS resolver has been added to FreeBSD base system in 2014 with 10.0-RELEASE version so being on FreeBSD you do not need to install anything. We will start with cleaning the any existing unbound(8) configuration which relies at /var/unbound. Keep in mind that /etc/unbound links to it.
# ls -l -d /etc/unbound /var/unbound lrwxr-xr-x 1 root wheel 14 2019.09.21 16:23 /etc/unbound -> ../var/unbound drwxr-xr-x 3 unbound unbound 8 2020.11.17 16:48 /var/unbound # rm -rf /var/unbound # mkdir -p /var/unbound/conf.d # chown -R unbound:unbound /var/unbound
The service local_unbound setup will create all needed configuration.
Just keep in mind that this process will setup all DNS servers that you have in the /etc/resolv.conf file.
You may want to put two of your favorite DNS servers before this process.
Configuration
# cat << BSD > /etc/resolv.conf nameserver 9.9.9.9 nameserver 1.1.1.1 BSD # service local_unbound setup Performing initial setup. destination: Extracting forwarders from /etc/resolv.conf. /var/unbound/forward.conf created /var/unbound/lan-zones.conf created /var/unbound/control.conf created /var/unbound/unbound.conf created /etc/resolvconf.conf created Original /etc/resolv.conf saved as /var/backups/resolv.conf.20201115.235254 # rm /var/backups/resolv.conf.20201115.235254 # find /var/unbound /var/unbound /var/unbound/lan-zones.conf /var/unbound/control.conf /var/unbound/unbound.conf /var/unbound/forward.conf % find /var/unbound -ls 12685 17 drwxr-xr-x 3 unbound unbound 8 Nov 17 16:48 /var/unbound 13072 1 -rw-r--r-- 1 root unbound 98 Nov 17 05:00 /var/unbound/forward.conf 12688 9 -rw-r--r-- 1 root unbound 354 Nov 15 23:56 /var/unbound/unbound.conf 12686 1 drwxr-xr-x 2 unbound unbound 3 Nov 16 00:23 /var/unbound/conf.d 12158 9 -rw-r--r-- 1 root unbound 193 Nov 15 23:56 /var/unbound/control.conf 11732 9 -rw-r--r-- 1 root unbound 189 Nov 15 23:56 /var/unbound/lan-zones.conf # tail -n 999 /var/unbound/* ==> /var/unbound/conf.d <== tail: /var/unbound/conf.d: Is a directory ==> /var/unbound/control.conf <== # This file was generated by local-unbound-setup. # Modifications will be overwritten. remote-control: control-enable: yes control-interface: /var/run/local_unbound.ctl control-use-cert: no ==> /var/unbound/forward.conf <== # Generated by resolvconf forward-zone: name: "." forward-addr: 9.9.9.9 forward-addr: 1.1.1.1 ==> /var/unbound/lan-zones.conf <== # This file was generated by local-unbound-setup. # Modifications will be overwritten. server: # Unblock reverse lookups for LAN addresses unblock-lan-zones: yes insecure-lan-zones: yes ==> /var/unbound/unbound.conf <== # This file was generated by local-unbound-setup. # Modifications will be overwritten. server: username: unbound directory: /var/unbound chroot: /var/unbound pidfile: /var/run/local_unbound.pid auto-trust-anchor-file: /var/unbound/root.key include: /var/unbound/lan-zones.conf include: /var/unbound/control.conf include: /var/unbound/conf.d/*.conf
We will now enable the local_unbound service and start it. At this point without any DNS blocking configuration.
# service local_unbound enable local_unbound enabled in /etc/rc.conf # service local_unbound start Starting local_unbound.
The /etc/resolv.conf will now have hour favorite DNS servers hashed/disabled and 127.0.0.1 address will be specified. You can also use sockstat(8) to check that unbound(8) is indeed listening on port 53.
# cat /etc/resolv.conf # nameserver 9.9.9.9 # nameserver 1.1.1.1 nameserver 127.0.0.1 options edns0 % sockstat -l -4 USER COMMAND PID FD PROTO LOCAL ADDRESS FOREIGN ADDRESS unbound local-unbo 7362 5 udp4 127.0.0.1:53 *:* unbound local-unbo 7362 6 tcp4 127.0.0.1:53 *:*
Test
After unbound(8) has been enabled it should now be visible that first DNS request should be longer and the second one and following requests should be very fast.
% time host ftp.freebsd.org ftp.freebsd.org is an alias for ftp.geo.freebsd.org. ftp.geo.freebsd.org has address 139.178.72.202 ftp.geo.freebsd.org has address 213.138.116.78 ftp.geo.freebsd.org has address 139.178.72.202 ftp.geo.freebsd.org has IPv6 address 2604:1380:2000:9501::15:0 ftp.geo.freebsd.org has IPv6 address 2001:41c8:112:8300::15:0 ftp.geo.freebsd.org has IPv6 address 2604:1380:2000:9501::15:0 ftp.geo.freebsd.org mail is handled by 0 . host ftp.freebsd.org 0.00s user 0.01s system 1% cpu 0.501 total % time host ftp.freebsd.org ftp.freebsd.org is an alias for ftp.geo.freebsd.org. ftp.geo.freebsd.org has address 139.178.72.202 ftp.geo.freebsd.org has address 213.138.116.78 ftp.geo.freebsd.org has address 139.178.72.202 ftp.geo.freebsd.org has IPv6 address 2604:1380:2000:9501::15:0 ftp.geo.freebsd.org has IPv6 address 2001:41c8:112:8300::15:0 ftp.geo.freebsd.org has IPv6 address 2604:1380:2000:9501::15:0 ftp.geo.freebsd.org mail is handled by 0 . host ftp.freebsd.org 0.01s user 0.00s system 88% cpu 0.007 total
Yep. Works.
Blacklist
I have written a simple and short unbound-blacklist-fetch.sh to automate the process of generating up to date DNS blocked domains config.
It uses one unbound(8) source and several hosts(5) sources, then combines them in unbound(8) compatible format while removing the duplicated entries.
We will now fetch it, put it under /root/bin directory (or use your favorite one), make it executable and start it.
# mkdir /root/bin # fetch -o /root/bin/unbound-blacklist-fetch.sh \ > https://raw.githubusercontent.com/vermaden/scripts/master/unbound-blacklist-fetch.sh # chmod +x /root/bin/unbound-blacklist-fetch.sh # /root/bin/unbound-blacklist-fetch.sh # ls -l /var/unbound/conf.d/blacklist.conf -rw-r--r-- 1 root unbound 3003929 2020.11.16 00:23 /var/unbound/conf.d/blacklist.conf # tail /var/unbound/conf.d/blacklist.conf local-zone: "zyrtec.1.p2l.info" always_nxdomain local-zone: "zyrtec.3.p2l.info" always_nxdomain local-zone: "zyrtec.4.p2l.info" always_nxdomain local-zone: "zyski-z-innowacji.pl" always_nxdomain local-zone: "zytpirwai.net" always_nxdomain local-zone: "zz.cqcounter.com" always_nxdomain local-zone: "zzhc.vnet.cn" always_nxdomain local-zone: "zzz.clickbank.net" always_nxdomain local-zone: "zzz.onion.pet" always_nxdomain local-zone: "zzzrtrcm2.com" always_nxdomain
The unbound(8) daemon already includes all /var/unbound/conf.d/*.conf files and we use that here.
You can change where the script generates blocked domains config under the # SETTINGS section directly in the script.
% grep -A 5 SETTINGS scripts/unbound-blacklist-fetch.sh
# SETTINGS
FILE=/var/unbound/conf.d/blacklist.conf
TEMP=/tmp/unbound
TYPE=always_nxdomain
ECHO=0
After the /var/unbound/conf.d/blacklist.conf file is generated you can now restart the unbound(8) service.
# service local_unbound restart Stopping local_unbound. Waiting for PIDS: 87745. Starting local_unbound. Waiting for nameserver to start... good
We will also add that script to crontab(5) so it will fetch fresh information every day.
# cat << BSD >> /var/cron/tabs/root > > # FETCH FRESH unbound(8) BLACKLIST > 0 0 * * * /root/bin/unbound-blacklist-fetch.sh > > BSD # crontab -l | tail -4 # FETCH FRESH unbound(8) BLACKLIST 0 0 * * * /root/bin/unbound-blacklist-fetch.sh
Test Blocked Domains
From 60000+ blocked domains I have chosen ad.track.us.org as target for verification.
% ping ad.track.us.org
ping: cannot resolve ad.track.us.org: Unknown host
% host ad.track.us.org
Host ad.track.us.org not found: 3(NXDOMAIN)
% dog ad.track.us.org
Status: NXDomain
% dog @1.1.1.1 ad.track.us.org
CNAME ad.track.us.org. 11m30s "track.us.org."
A track.us.org. 6m30s 185.59.208.177
As You can see the domain is successfully blocked.
The above blocking configuration does not mean that I will now disable the uBlock Origin plugin from Firefox but its a welcome addition to blocking unwanted information tools workshop.
UPDATE 1 – Reworked Script and Alternatives
After reading comments on Hacker News / Lobsters / Reddit I got a lot of good ideas how to improve my script even more.
Some people suggested that very similar functionality already exists in dns/void-zones-tools package on FreeBSD. One can also use get_unbound_adblock.sh script or lie-to-me solution.
There are also more sophisticated tools like Pi-hole which also include DHCP server and web interface for management and statistics. Unfortunately Pi-hole does not run on FreeBSD.
After reworking and adding additional sources to my unbound-blacklist-fetch.sh script its now twice the amount of blocked unwanted domains. In the first release about 60000 domains were blocked. Now its more then 120000.
Here is the distribution of data between various types of sources.
% wc -lc /tmp/unbound/lists-* 54587 1059592 /tmp/unbound/lists-domains 143553 4115745 /tmp/unbound/lists-hosts 32867 1596409 /tmp/unbound/lists-unbound 231007 6771746 total
Now the /var/unbound/conf.d/blacklist.conf before these changes.
% wc -l blacklist.conf 60009 blacklist.conf % ls -l /var/unbound/conf.d/blacklist.conf -rw-r--r-- 1 root unbound 2907535 2020-11-20 00:00 /var/unbound/conf.d/blacklist.conf
… and after adding additional sources.
% wc -l blacklist.conf 122190 blacklist.conf % ls -l /var/unbound/conf.d/blacklist.conf -rw-r--r-- 1 root unbound 6086623 2020-11-20 15:07 /var/unbound/conf.d/blacklist.conf
Here is also performance summary about which part takes what amount of time.
Combining various sources and generating the final config takes about 5 seconds.
Most of the time is spent in fetching the data from various sources.
The script is already uploaded to the GitHub repo.
Just fetch it and enjoy π
UPDATE 2 – Huge Domains List Version
Thanks to Luca Castagnini from bsd.network who pointed me to https://oisd.nl/ site with HUGE list of domains that can/could/should be blocked I made another variant (or version) of the script unbound-blacklist-fetch-huge.sh with a total of 145 (!) various sources for domains to block.
It of course takes little longer to fetch and generate then the ‘casual’ version.
Its little less then 2 minutes to fetch and generate new config while the longest part is the fetching of those 145 sources. Generation takes about 15 seconds.
These 145 sources provide more then a million domains to block.
% wc -l /tmp/unbound/* 551704 lists-domains 439505 lists-hosts 60835 lists-unbound 1052044 total
The script after removing duplicated entries makes little more then 480000 domains of it.
% wc -l /var/unbound/conf.d/blacklist.conf 484829 /var/unbound/conf.d/blacklist.conf
Unfortunately it comes at a price. In this HUGE variant with domains from 145 sources the unbound(8) server now uses about 150 MB of RAM.
% top -b -o res|grep -E 'RES|unbound' PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 75849 unbound 1 20 0 158M 149M select 4 0:03 0.00% local-unbound
I leave up to you which version to use and which sources to choose for blocking, but as my Firefox with about 20 tabs opened takes little more then 4226 MB of RAM these additional 150 MB from unbound(8) does not hurt that much π
% ./FIREFOX.RAM.sh
4226 MB
% cat FIREFOX.RAM.sh
#! /bin/sh
SUM=0
top -b -o res \
| sed 1,10d \
| grep firefox \
| awk '{print $7}' \
| tr -cd '0-9\n' \
| while read I
do
SUM=$(( ${SUM} + ${I} ))
echo ${SUM}
done | tail -1 | tr -d '\n'
echo " MB"
ππππππππ 