aau-network-security/gollector

Application for retrieving and storing domain names from various sources

gollector's Issues

cache add

add the item to the cache:

• when i find them in the db
• after i wrote them in the db (could be done in the posthook)

Try out GitHub actions

Data Race - goroutine

As suggested from @mrturkmen06 i am reporting this problem.
Running tests on CI with this parameter run: go test -v --race ./... give back the following error :
https://github.com/aau-network-security/gollector/runs/689914296?check_suite_focus=true#step:6:68

Separate cache from other processes

With some high-performant IPC communication between a process and a cache container (high throughput => 100,000 messages per second, low latency => (far) under ms), it would be possible to run multiple measurements simultaneously.

Current situation

+------------------+           +------------------+
|                  |           |                  |
|     Process 1    |           |     Process 2    |
|                  |           |                  |
|  +------------+  |           |  +------------+  |
|  |            |  |           |  |            |  |
|  |   Cache    |  |           |  |    Cache   |  |
|  |            |  |           |  |            |  |
+--+------+-----+--+           +--+-----+------+--+
          |                             |
          |                             |
          |                             |
          |                             |
          |                             |
          +--------------+--------------+
                         |
                 +-------+--------+
                 |                |
                 |   Persistent   |
                 |                |
                 +----------------+

Suggested alternative

+------------------+           +------------------+
|                  |           |                  |
|     Process 1    |           |     Process 2    |
|                  |           |                  |
+-------+----------+           +---------+--------+
        |                                |
        +--------------+    +------------+
                       |    |    High performant IPC
                       |    |
                 +-----+----+-----+
                 |                |
                 |     Cache      |
                 |                |
                 +-------+--------+
                         |
                         |  Asynchronous, but reliable
                 +-------+--------+
                 |                |
                 |   Persistent   |
                 |                |
                 +----------------+

Improve error reporting

Error messages are somewhat difficult to debug otherwise

• Wrapping errors to make error prints more clear
• Report messages to Sentry/Rollbar and send emails when errors are occuring

Mount volume in cache to persist issued certificates

When running the cache over TLS, the certmagic lib automatically obtains certificates. The issued certificates are stored on disk, but because we currently do not mount a volume to persist those certificates, they disappear whenever the container is closed, and with a restart a new cert is issued. As a result, the rate limit of Let's Encrypt may be hit, locking us out of running on TLS for a few days.

The docker-compose config should mount a volume to the correct location where the certs are stored by certmagic

DB query

skip query db if the cache size is not reached

Test FTP over SSH

I am not convinced the current implementation is correct. The only zone file accessible over FTP is the .com one

Correctly handle certificate renewal

Right now, I believe gollector does not handle renewal of certificates by certmagic. Only relevant when the cache is running for a long time

Clarify how zone files are collected

There exists a "zonediffer" and a "zones" collector and one of them is supported whereas the other is not.

Index Start Date

I think there is a problem by retrieving the last stored certificate from the DB. I guess this function dosen't work very well (https://github.com/aau-network-security/gollector/blob/master/app/ct/main.go#L135).

The way ct should work: it should insert in the DB (n) certificates every time is run. the function linked above should get the last entered certificate from the DB in order to scan the next (n) certificates from that one.

The way ct is working right now: the first time i run ct it enters the first 100 certificates. Running it again it dosen't store the next 100 certificates in the DB.

The function linked above give back an error in the tests too
https://github.com/aau-network-security/gollector/pull/39/checks#step:6:111

Properly address CT collection when STH cannot be collected

Add status API call for cache

It might be useful to retrieve the current state of the entries contained in cache process via an gRPC. To go even further, a monitoring tool can read this state and visualize the growth of number of entries over time.

Delete unsed files and functions

• delete previous implementation not used
• delete benchmark code

Add support for anonymized domains

Implement cache of max size

We should probably consider using an LRU cache, instead of caching everything.

Read password settings as environment variables

Somehow Golang does not manage to read environment variables

Collectors

Make sure the implementation about CT logs works for all the collectors component we have

Connections are terminated by remote hosts

This results in the failure to download some zone files, and wrong last_seen dates

Error handling

• fix posthook error
• new domain error

DB Libraries

Use just a library to interact with the DB. pq should be the best on

Better handling of configuration

docker-compose requires all environment variables to be set, even when building the application, which is unnecessary. We should move the environment variable check to the be done in the application itself, rather than in docker-compose

Use full list tlds in CZDS

Instead of relying on a precompiled list of TLDs, use the CZDS api to retrieve all TLDs to scan

Add the notion of a "dataset"

We must be able to distinguish between multiple measurements from different vantage points, knowing exactly which data point belongs to which data set/vantage point.

Pass tests on Github Actions

A couple of significant requirements:

• Some tests are written with data sources (such as an zone file accessed via HTTP) in place, which must either (1) be mocked or (2) these tests must by skipped by GA.
• Some tests require interaction with a postgres database in place, which should be provided by GA.

Ensure that anonymized entities refer correctly to their un-anonymized entities