cds-snc/tracker

Check whether a Government of Canada domain is adhering to best security practices.

tracker's Issues

Domain ownership algorithm documentation

Write documentation outlining what happens in the processing stage to map domains onto their owners.

Add command and argument descriptions to CLI

Currently the descriptions for the cli arguments are fairly spotty, they need to be touched up and detail added.

Formalize domain list

Current domain list has quite a few domains and organizations that likely don't need to be included, since our current list is what will be going into beta, it needs to be cleaned up

Improve error handling of tracker scanning

When tracker crashes, there is limited logging to support troubleshooting. Suggest bundling cached scan results so that if/when tracker crashes, there is a log of what data caused it to crash.

Tracker does not include option to drop old domains and/or owners

Tracker Insert process is used to update the database with a new domains.csv, ciphers.csv or owners.csv. Updates are appended to the existing database. If one of the csv files removes an entry, the data still remains in the database. Thus, there is no option to remove old domains, ciphers, owners that are no longer valid.

A manual process is required to drop the tables before the new files are uploaded using the Tracker Insert process. Scans would then be using the latest information when it is pulled from the database prior to execution.

ITPIN shouldn't be compliant if crypto and cert don't have data i.e N/A

As shown in the picture below

Deploy documentation

We need some

Maintain historical records to measure compliance / improvements over time.

Just throwing this in as a place holder, specifically within tracker/data/cli.py def run,
we're currently nuking the entire dataset at runtime which while great for freshness, is stealing away some valuable historical data.

Suggest we take a look at the backend and see how we could at least set aside previous scan results/reports/etc, even if there is no clean way to present the data currently.

#totallynotworking

Data definition documentation in repo

Update and transfer existing data defintions into the repo where it can be useful

Runtime Error - Update SSLZE

Following issue occurs:

File "/opt/apps/tracker/.venv/lib/python3.6/site-packages/nassl/ssl_client.py", line 165, in set_underlying_socket
raise RuntimeError('A socket was already set')
RuntimeError: A socket was already set

Proposed resolution is update the SSLYZE component.

capture cert expiry date so that reports / alerts can be built off that data

Please capture the cert expiry date as part of the crawl so that reports and alerts can be generated from the info.

owner without matching subdomain in domains.csv causes keyerror

https://github.com/cds-snc/tracker/blob/master/data/processing.py#L298

Probably just want to wrap this in a try/except, and or do a quick check 'if in' to determine whether this even a valid ownership domain before continuing.

reproduce:

canada.ca in owner, and not in the domains list.

Assuming we control the input, I'd say not a huge deal, but since we don't, we probably want to assume faulty inbound data and recover gracefully where we can.

(From cds-snc/pulse#139)

Redirect issue - endpoint choice

Use case:

1. Service has all endpoints live (http, httpwww, https, httpswww)
2. http, httpwww and https endpoints are configure properly and immediately redirect to the secure eventual endpoint
3. httpswww downgrade before reaching the eventual secure endpoint

Even if we feed the scanner with a root endpoint that is secured and properly redirect to the eventual secure endpoint, tracker seems to set the canonical url to httpswww. Where there are httpswww configuration issues, it significantly impact the root domain scan result because the httpswww is chosen.

Question:
should Tracker choose to scan the secure root endpoint if Live instead of httpswww?

hsts max age changed to 18 weeks

HSTS max age check for compliance should be changed to 18 weeks (10886400 seconds)

local-development docs touchup

Incorrect relative path

pip3 install -r ../../domain-scan/requirements.txt
pip3 install -r ../../domain-scan/requirements-scanners.txt

directory user would be at this step would generate the wrong path

export DOMAIN_SCAN_PATH=$(pwd)/domain-scan/scan
export DOMAIN_GATHER_PATH=$(pwd)/domain-scan/gather