pilosus/dnseen

simple DNS queries analyzer that works on top of the tcpdump logs

dnseen - DNS queries analyzer

dnseen is a simple DNS queries analyzer that works on top of the tcpdump logs.

• Simple: no GUI/TUI, no modes, easy command-line interface
• Stands on the shoulders of giants: used tcpdump and systemd at its core
• Separation of concerns: logs and a stats report produced by different components and can be used independently
• Filtering: select a datetime range, filter out domains with regex, filter by domain hits, etc.

$ dnseen
|                             :domain | :hits | :blocked |
|-------------------------------------+-------+----------|
|        profile.accounts.firefox.com |   831 |    false |
|                          github.com |   531 |    false |
|                      www.google.com |   351 |    false |
|                    alive.github.com |   332 |    false |
|                      api.github.com |   331 |    false |
|                     www.youtube.com |   326 |    false |
|                           slack.com |   260 |    false |
|                       app.slack.com |   246 |    false |
|                          slackb.com |   232 |    false |
|                clojurians.slack.com |   230 |    false |
...

Install

Dependencies

dnseen requires the following dependencies:

• Linux OS
• tcpdump
• babashka
• (optionally) logrotate

Installer script

Install dnseen with the installer script on Linux:

curl -sLO https://raw.githubusercontent.com/pilosus/dnseen/master/install
chmod +x install
./install

By default, the command will be installed in /usr/local/bin (you may need to use sudo to run the installer script in this case!). You can change installation directory with the option --install-dir:

./install --install-dir <your-dir-under-$PATH>

To install a specific version instead of the latest one use --version option:

./install --version 0.2.0

Installer script downloads a package archive file to a temporary directory under /tmp, you can change it with the option --download-dir:

./install --download-dir <your-dir-under-$PATH>

You can uninstall dnseen and all its corresponding services with the --uninstall option (can be used along with --install-dir):

./install --uninstall

For more options see installer script's help:

./install --help

Manual install

1. Clone the repo and cd to it:

git clone https://github.com/pilosus/dnseen.git
cd dnseen

2. Copy content of the dnseen.service file and paste to a new systemd service:

sudo -E systemctl edit dnseen --full --force

Alternatively, simply copy the service file:

sudo cp dnseen.service /etc/systemd/system/

3. Reload systemd, start the service, enable it to start automatically on system boot, and make sure it works:

sudo systemctl daemon-reload
sudo systemctl start dnseen.service 
sudo systemctl enable dnseen.service
sudo systemctl status dnseen.service 

4. (Optionally) Add logrotate config file to make logs rotated:

sudo cp dnseen.logrotate /etc/logrotate.d/dnseen

Make sure config is valid:

sudo logrotate --debug /etc/logrotate.d/dnseen

If needed, force rotation and restart the service:

sudo logrotate --force /etc/logrotate.d/dnseen
sudo systemctl restart dnseen.service

Use

Basic usage takes the whole log and prints the report without any filters applied, domains ordered by number of hits in descending order:

dnseen

when invoking command that is not under your $PATH (e.g. if you followed the manual installation guide), use:

./dnseen

Apply some filters if needed:

dnseen \
    --from "2023-12-01T00:00:00" \
    --to "2024-01-01T00:00:00" \
    --match '\.(goog|google)$' \
    --exclude '(?i).*domains\.' \
    --hosts '/etc/hosts' \
    --hits 10 \
    --head 20 \
    --no-pretty \
    -vvv

A path to a file or a directory containing hosts file can be provided to get statistics about blocked domains, i.e. domains that resolve to either localhost or 0.0.0.0. Use --totals flag to get aggregation statistics of the report itself:

dnseen \
    --hosts '/etc/hosts.d/' \
    --hosts '/etc/hosts.old' \
    --totals

Configuration parameters can also be defined in a EDN config file. A default path to the config file is either $XDG_CONFIG_HOME/dnseen/config.edn or $HOME/.config/dnseen/config.edn. It can be overriden with the --config option:

dnseen --config ~/.dnseen

Get more help with:

dnseen --help

Filters are applied to the raw logs in the order the corresponding CLI options are shown in the help message (e.g. --match is applied before --exclude).

dnseen's Issues

Option `--hosts` to consume hosts file or directory with hosts files with totals resolved from hosts

I'd like to see what domains were resolved throgh the hosts files and "blocked" (e.g. resolved to 0.0.0.0, ::1, localhost, etc.).
I also would like to see the total stats:

• requests total (#/shares)
• requests blocked (#/shares)

`--totals` option

As per #13 and #14, we want to show some stats like:

• Total queries
• Blocked queries (count, share %)
• Domains blocked (based on the --hosts)

Add tests

The code has become relatively complex, especially config params merge.
If we want to develop the program further, tests are needed

`--match <PCRE>` option

In addition to --exclude opt, we need a --match opt, that will select only domains matching agains given PCRE string

Config file for default options

It would be nice to have a dotfile to store configs for dnseen, e.g. exclude regex string, hits thresholds etc.

We may want to follow XDG base directory spec (1, 2)

Verbose option

We want to support --verbose flag that is combined to set verbosity level, e.g.:

• -v => 1
• -vv => 2
• -vvv => 3
• etc

We also want to hide Query options for default verbosity level 0, and show it for level 1 and above

Automate installation with install script

All the installation steps described in the README can be automated with the simple shell script.
We can take babashka's install script as an inspiration.

Format option

By default we use tabular representation of the stats report. It's nice for humans, but not that nice for machine processing.
We want to support:

• --format <name> option
• make table the default value
• Add plain format that is easy to use with *nix pipelines, e.g. cut & awk