Unbound DNS Blacklist

Today I will show you how to configure unbound(8) to block spam/malicious/malware domains at DNS level.

unbound

I will use FreeBSD for that purpose but you can use any system that unbound(8) runs on.

logo-freebsd

Earlier I used generated /etc/hosts file but that was limited in several ways. The ZSH shell will autocomplete all these blocked domains to the ssh(1)/scp(1) commands (which takes needless time and shows useless completions). Subdomains are not handled. The malicious.com is blocked but ads.malicious.com is not. You need to duplicate all those domains in the /etc/hosts file.

TL;DR

Not all people have time for my long boring stories so this is meritum of this article.

# rm -rf /var/unbound
# mkdir -p /var/unbound/conf.d
# chown -R unbound:unbound /var/unbound
# service local_unbound setup
# service local_unbound enable
# service local_unbound start
# mkdir /root/bin
# cd 
# fetch -o /root/bin/unbound-blacklist-fetch.sh \
> https://raw.githubusercontent.com/vermaden/scripts/master/unbound-blacklist-fetch.sh
# chmod +x /root/bin/unbound-blacklist-fetch.sh
# /root/bin/unbound-blacklist-fetch.sh
# service local_unbound restart
# cat << BSD >> /var/cron/tabs/root
> # FETCH FRESH unbound(8) BLACKLIST
>   0 0 * * * /root/bin/unbound-blacklist-fetch.sh
> BSD

Whole Story

The unbound(8) caching DNS resolver has been added to FreeBSD base system in 2014 with 10.0-RELEASE version so being on FreeBSD you do not need to install anything. We will start with cleaning the any existing unbound(8) configuration which relies at /var/unbound. Keep in mind that /etc/unbound links to it.

# ls -l -d /etc/unbound /var/unbound
lrwxr-xr-x 1 root    wheel   14 2019.09.21 16:23 /etc/unbound -> ../var/unbound
drwxr-xr-x 3 unbound unbound  8 2020.11.17 16:48 /var/unbound

# rm -rf /var/unbound

# mkdir -p /var/unbound/conf.d

# chown -R unbound:unbound /var/unbound

The service local_unbound setup will create all needed configuration.

Just keep in mind that this process will setup all DNS servers that you have in the /etc/resolv.conf file.

You may want to put two of your favorite DNS servers before this process.

Configuration

# cat << BSD > /etc/resolv.conf
nameserver 9.9.9.9
nameserver 1.1.1.1
BSD

# service local_unbound setup
Performing initial setup.
destination: 
Extracting forwarders from /etc/resolv.conf.
/var/unbound/forward.conf created
/var/unbound/lan-zones.conf created
/var/unbound/control.conf created
/var/unbound/unbound.conf created
/etc/resolvconf.conf created
Original /etc/resolv.conf saved as /var/backups/resolv.conf.20201115.235254

# rm /var/backups/resolv.conf.20201115.235254

# find /var/unbound
/var/unbound
/var/unbound/lan-zones.conf
/var/unbound/control.conf
/var/unbound/unbound.conf
/var/unbound/forward.conf

% find /var/unbound -ls
 12685  17  drwxr-xr-x  3  unbound  unbound    8  Nov 17 16:48  /var/unbound
 13072   1  -rw-r--r--  1  root     unbound   98  Nov 17 05:00  /var/unbound/forward.conf
 12688   9  -rw-r--r--  1  root     unbound  354  Nov 15 23:56  /var/unbound/unbound.conf
 12686   1  drwxr-xr-x  2  unbound  unbound    3  Nov 16 00:23  /var/unbound/conf.d
 12158   9  -rw-r--r--  1  root     unbound  193  Nov 15 23:56  /var/unbound/control.conf
 11732   9  -rw-r--r--  1  root     unbound  189  Nov 15 23:56  /var/unbound/lan-zones.conf

# tail -n 999 /var/unbound/*
==> /var/unbound/conf.d <==
tail: /var/unbound/conf.d: Is a directory

==> /var/unbound/control.conf <==
# This file was generated by local-unbound-setup.
# Modifications will be overwritten.
remote-control:
	control-enable: yes
	control-interface: /var/run/local_unbound.ctl
	control-use-cert: no

==> /var/unbound/forward.conf <==
# Generated by resolvconf

forward-zone:
	name: "."
	forward-addr: 9.9.9.9
	forward-addr: 1.1.1.1

==> /var/unbound/lan-zones.conf <==
# This file was generated by local-unbound-setup.
# Modifications will be overwritten.
server:
	# Unblock reverse lookups for LAN addresses
	unblock-lan-zones: yes
	insecure-lan-zones: yes

==> /var/unbound/unbound.conf <==
# This file was generated by local-unbound-setup.
# Modifications will be overwritten.
server:
	username: unbound
	directory: /var/unbound
	chroot: /var/unbound
	pidfile: /var/run/local_unbound.pid
	auto-trust-anchor-file: /var/unbound/root.key

include: /var/unbound/lan-zones.conf
include: /var/unbound/control.conf
include: /var/unbound/conf.d/*.conf

We will now enable the local_unbound service and start it. At this point without any DNS blocking configuration.

# service local_unbound enable
local_unbound enabled in /etc/rc.conf

# service local_unbound start
Starting local_unbound.

The /etc/resolv.conf will now have hour favorite DNS servers hashed/disabled and 127.0.0.1 address will be specified. You can also use sockstat(8) to check that unbound(8) is indeed listening on port 53.

# cat /etc/resolv.conf
# nameserver 9.9.9.9
# nameserver 1.1.1.1
nameserver 127.0.0.1
options edns0

% sockstat -l -4
USER COMMAND PID FD PROTO LOCAL ADDRESS FOREIGN ADDRESS 
unbound local-unbo 7362 5 udp4 127.0.0.1:53 *:*
unbound local-unbo 7362 6 tcp4 127.0.0.1:53 *:*

Test

After unbound(8) has been enabled it should now be visible that first DNS request should be longer and the second one and following requests should be very fast.

% time host ftp.freebsd.org
ftp.freebsd.org is an alias for ftp.geo.freebsd.org.
ftp.geo.freebsd.org has address 139.178.72.202
ftp.geo.freebsd.org has address 213.138.116.78
ftp.geo.freebsd.org has address 139.178.72.202
ftp.geo.freebsd.org has IPv6 address 2604:1380:2000:9501::15:0
ftp.geo.freebsd.org has IPv6 address 2001:41c8:112:8300::15:0
ftp.geo.freebsd.org has IPv6 address 2604:1380:2000:9501::15:0
ftp.geo.freebsd.org mail is handled by 0 .
host ftp.freebsd.org  0.00s user 0.01s system 1% cpu 0.501 total

% time host ftp.freebsd.org
ftp.freebsd.org is an alias for ftp.geo.freebsd.org.
ftp.geo.freebsd.org has address 139.178.72.202
ftp.geo.freebsd.org has address 213.138.116.78
ftp.geo.freebsd.org has address 139.178.72.202
ftp.geo.freebsd.org has IPv6 address 2604:1380:2000:9501::15:0
ftp.geo.freebsd.org has IPv6 address 2001:41c8:112:8300::15:0
ftp.geo.freebsd.org has IPv6 address 2604:1380:2000:9501::15:0
ftp.geo.freebsd.org mail is handled by 0 .
host ftp.freebsd.org  0.01s user 0.00s system 88% cpu 0.007 total

Yep. Works.

Blacklist

I have written a simple and short unbound-blacklist-fetch.sh to automate the process of generating up to date DNS blocked domains config.

It uses one unbound(8) source and several hosts(5) sources, then combines them in unbound(8) compatible format while removing the duplicated entries.

unbound-blacklist-script.256

We will now fetch it, put it under /root/bin directory (or use your favorite one), make it executable and start it.

# mkdir /root/bin

# fetch -o /root/bin/unbound-blacklist-fetch.sh \
> https://raw.githubusercontent.com/vermaden/scripts/master/unbound-blacklist-fetch.sh

# chmod +x /root/bin/unbound-blacklist-fetch.sh

# /root/bin/unbound-blacklist-fetch.sh

# ls -l /var/unbound/conf.d/blacklist.conf
-rw-r--r-- 1 root unbound 3003929 2020.11.16 00:23 /var/unbound/conf.d/blacklist.conf

# tail /var/unbound/conf.d/blacklist.conf
local-zone: "zyrtec.1.p2l.info" always_nxdomain
local-zone: "zyrtec.3.p2l.info" always_nxdomain
local-zone: "zyrtec.4.p2l.info" always_nxdomain
local-zone: "zyski-z-innowacji.pl" always_nxdomain
local-zone: "zytpirwai.net" always_nxdomain
local-zone: "zz.cqcounter.com" always_nxdomain
local-zone: "zzhc.vnet.cn" always_nxdomain
local-zone: "zzz.clickbank.net" always_nxdomain
local-zone: "zzz.onion.pet" always_nxdomain
local-zone: "zzzrtrcm2.com" always_nxdomain

The unbound(8) daemon already includes all /var/unbound/conf.d/*.conf files and we use that here.

You can change where the script generates blocked domains config under the # SETTINGS section directly in the script.

% grep -A 5 SETTINGS scripts/unbound-blacklist-fetch.sh 
# SETTINGS
FILE=/var/unbound/conf.d/blacklist.conf
TEMP=/tmp/unbound
TYPE=always_nxdomain
ECHO=0

After the /var/unbound/conf.d/blacklist.conf file is generated you can now restart the unbound(8) service.

# service local_unbound restart
Stopping local_unbound.
Waiting for PIDS: 87745.
Starting local_unbound.
Waiting for nameserver to start... good

We will also add that script to crontab(5) so it will fetch fresh information every day.

# cat << BSD >> /var/cron/tabs/root
> 
> # FETCH FRESH unbound(8) BLACKLIST
>   0 0 * * * /root/bin/unbound-blacklist-fetch.sh
> 
> BSD

# crontab -l | tail -4

# FETCH FRESH unbound(8) BLACKLIST
  0 0 * * * /root/bin/unbound-blacklist-fetch.sh

Test Blocked Domains

From 60000+ blocked domains I have chosen ad.track.us.org as target for verification.

% ping ad.track.us.org
ping: cannot resolve ad.track.us.org: Unknown host

% host ad.track.us.org
Host ad.track.us.org not found: 3(NXDOMAIN)

% dog ad.track.us.org
Status: NXDomain

% dog @1.1.1.1 ad.track.us.org
CNAME ad.track.us.org. 11m30s   "track.us.org."
    A track.us.org.     6m30s   185.59.208.177


unbound-test.256

As You can see the domain is successfully blocked.

The above blocking configuration does not mean that I will now disable the uBlock Origin plugin from Firefox but its a welcome addition to blocking unwanted information tools workshop.

UPDATE 1 – Reworked Script and Alternatives

After reading comments on Hacker News / Lobsters / Reddit I got a lot of good ideas how to improve my script even more.

Some people suggested that very similar functionality already exists in dns/void-zones-tools package on FreeBSD. One can also use get_unbound_adblock.sh script or lie-to-me solution.

There are also more sophisticated tools like Pi-hole which also include DHCP server and web interface for management and statistics. Unfortunately Pi-hole does not run on FreeBSD.

After reworking and adding additional sources to my unbound-blacklist-fetch.sh script its now twice the amount of blocked unwanted domains. In the first release about 60000 domains were blocked. Now its more then 120000.

Here is the distribution of data between various types of sources.

% wc -lc /tmp/unbound/lists-*
   54587 1059592 /tmp/unbound/lists-domains
  143553 4115745 /tmp/unbound/lists-hosts
   32867 1596409 /tmp/unbound/lists-unbound
  231007 6771746 total

Now the /var/unbound/conf.d/blacklist.conf before these changes.

% wc -l blacklist.conf
   60009 blacklist.conf

% ls -l /var/unbound/conf.d/blacklist.conf
-rw-r--r-- 1 root unbound 2907535 2020-11-20 00:00 /var/unbound/conf.d/blacklist.conf

… and after adding additional sources.

% wc -l blacklist.conf
  122190 blacklist.conf

% ls -l /var/unbound/conf.d/blacklist.conf
-rw-r--r-- 1 root unbound 6086623 2020-11-20 15:07 /var/unbound/conf.d/blacklist.conf

Here is also performance summary about which part takes what amount of time.

Combining various sources and generating the final config takes about 5 seconds.

Most of the time is spent in fetching the data from various sources.

UPDATE1.unbound.script.256

The script is already uploaded to the GitHub repo.

Just fetch it and enjoy πŸ™‚

UPDATE 2 – Huge Domains List Version

Thanks to Luca Castagnini from bsd.network who pointed me to https://oisd.nl/ site with HUGE list of domains that can/could/should be blocked I made another variant (or version) of the script unbound-blacklist-fetch-huge.sh with a total of 145 (!) various sources for domains to block.

It of course takes little longer to fetch and generate then the ‘casual’ version.

UPDATE2.unbound.time

Its little less then 2 minutes to fetch and generate new config while the longest part is the fetching of those 145 sources. Generation takes about 15 seconds.

These 145 sources provide more then a million domains to block.

% wc -l /tmp/unbound/* 
 551704 lists-domains
 439505 lists-hosts
  60835 lists-unbound
1052044 total

The script after removing duplicated entries makes little more then 480000 domains of it.

% wc -l /var/unbound/conf.d/blacklist.conf 
 484829 /var/unbound/conf.d/blacklist.conf

Unfortunately it comes at a price. In this HUGE variant with domains from 145 sources the unbound(8) server now uses about 150 MB of RAM.

% top -b -o res|grep -E 'RES|unbound'
  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C    TIME    WCPU COMMAND
75849 unbound       1  20    0   158M   149M select   4    0:03   0.00% local-unbound

I leave up to you which version to use and which sources to choose for blocking, but as my Firefox with about 20 tabs opened takes little more then 4226 MB of RAM these additional 150 MB from unbound(8) does not hurt that much πŸ™‚

% ./FIREFOX.RAM.sh
4226 MB

% cat FIREFOX.RAM.sh 
#! /bin/sh

SUM=0

top -b -o res \
  | sed 1,10d \
  | grep firefox \
  | awk '{print $7}' \
  | tr -cd '0-9\n' \
  | while read I
    do
      SUM=$(( ${SUM} + ${I} ))
      echo ${SUM}
    done | tail -1 | tr -d '\n'
echo " MB"
One more thing related to Firefox. After checking ‘free’ memory with Firefox running and after closing it the difference was about 2.6 GB which means that above script to calculate Firefox memory usage is not a lot accurate πŸ™‚
EOF

6 thoughts on “Unbound DNS Blacklist

  1. Pingback: Unbound DNS Blacklist - GistTree

  2. tour Γ  patato rΓ©volutionaire. Patatas o la muerte (@ObnoxiousJul)

    You may be interested to merge my script into yours. It may full of gnu dirt (bash), but associative arrays are great for configuration πŸ™‚

    #!/bin/bash
    #
    # TODO: gestion des erreurs πŸ™‚
    # pas casser le prΓ©cΓ©dent fichier abdlock de unbound quand Γ§a foire πŸ˜›
    # faire un truc qui marche aussi sur mon freeBSD πŸ˜€
    # nop je ne ferais pas du ksh <:-)
    # nop je ne fera pas du posix1 awk + sed ;]
    # compter source par source ^_^
    # vΓ©rifier les sources mortes πŸ˜‰
    # utiliser doas ^-^
    # indentation et style : JAMAIS :€
    # MuST HAVE : usage :'(
    export PATH="/bin/:/usr/bin/:/sbin/:/usr/sbin:/usr/local/sbin:/usr/local/bin"
    tmpfile="$(mktemp)" && echo '' > $tmpfile
    tmp_work="$(mktemp)" && echo '' > $tmp_work
    unboundconf="/etc/unbound/unbound.conf.d/_unbound-adhosts.conf"
    set -e
    # personnal list in /etc/adblack.txt
    declare -A BLOCK=(
    [perso]="file:///etc/adblack.txt"
    [adaway]=https://adaway.org/hosts.txt
    [disconad]="https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt"
    [discontrack]="https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt"
    [fademind]="https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.Spam/hosts"
    [malwaredom]="https://mirror1.malwaredomains.com/files/justdomains"
    [stevenblack]="https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts"
    [sysctl]=http://sysctl.org/cameleon/hosts
    [suspicous_low]=https://www.dshield.org/feeds/suspiciousdomains_Low.txt
    [suspicous_medium]=https://www.dshield.org/feeds/suspiciousdomains_Medium.txt
    [suspicous_high]=https://www.dshield.org/feeds/suspiciousdomains_High.txt
    [winhelp]=http://winhelp2002.mvps.org/hosts.txt
    [yoyo]="https://pgl.yoyo.org/as/serverlist.php?hostformat=hosts&showintro=0&mimetype=plaintext"
    [firebog_w3k]=https://v.firebog.net/hosts/static/w3kbl.txt
    [firebog_bill]=https://v.firebog.net/hosts/BillStearns.txt
    [matomo]=https://raw.githubusercontent.com/matomo-org/referrer-spam-blacklist/master/spammers.txt
    [dawsey]=https://raw.githubusercontent.com/Dawsey21/Lists/master/main-blacklist.txt
    [vokins]=https://raw.githubusercontent.com/vokins/yhosts/master/hosts
    [nfz_moe]=https://hosts.nfz.moe/basic/hosts
    [bblck]=https://ssl.bblck.me/blacklists/hosts-file.txt
    )
    function clean_get {
    cat $tmp_work | perl -pe 's/ *#.*$//g' | \
    perl -ane 'if (/^(.+)$/) { @l=split( / /,$1);
    $h = lc(pop(@l));
    $h =~ s/\015//g;
    $h =~ s/^\s+|\s+$//g;
    if ($h=~ /^(?=.{1,255}$)[0-9\-_a-z](([\-_0-9a-z]|\b-){0,61}[\-_a0-9a-z])?(\.[\-_0-9a-z](([\-_0-9a-z]|\b-){0,61}[\-_0-9a-z])?)*\.?$/) {
    print qq{local-zone: "$h" always_nxdomain\n};
    } else {
    $h =~ /^\s*$/ or print STDERR "wtf is $h?\n";
    }
    }'
    }
    RZ="\e[0m"
    GD="\e[32m"
    RD="\e[31m"
    for K in ${!BLOCK[@]}; do
    >&2 echo -n "getting $K "
    curl -s ${BLOCK[$K]} > $tmp_work || >&2Β echo -e ${RD}KO${RZ} && >&2 echo -e ${GD}OK${RZ};
    >&2 echo -n "entries "
    >&2 wc -l < $tmp_work;
    clean_get ${BLOCK[$K]}
    done | sort -r | uniq >> $tmpfile
    echo -n "total entries "
    wc -l < $tmpfile
    sudo install -o unbound -m 600 $tmpfile $unboundconf && \
    sudo -u unbound unbound-checkconf 1>/dev/null && \
    sudo systemctl reload unbound 1>/dev/null && echo DNS reloaded
    # way to build a blacklist of DNS over HTTP from curl sources #noDOH
    #curl -s https://gist.githubusercontent.com/kimbo/dd65d539970e3a28a10628f15398247b/raw/bac0b90ef01b6f9d69462512327fd4ff903a9a3f/scrape-doh-providers.py | python3 | perl -ane 'm!https?://([^/:]+)/! and print "$1\n";'
    exit 0

    Like

    Reply
  3. Pingback: Valuable News – 2020/11/23 | πšŸπšŽπš›πš–πšŠπšπšŽπš—

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s