esonderegger/dotmil-domains

An incomplete listing of `.mil` domains and the code for the scraper used to build the list

dotmil-domains's Issues

Dedupe root domains and www subdomains

I'd argue that www is more of an implementation detail, and that they could all just be listed as the root domain, with services that use the list knowing to look at www or the root.

But in either case, listing dod.mil and www.dod.mil on two separate rows probably isn't correct, as they are the same site, so the list should just have one of those.

Update Readme to explain role of mil.csv

Some .mil domains and subdomains from Certificate Transparency logs

You can search a a Certificate Transparency search engine built by Comodo, crt.sh, for for any known issued .mil domains:

https://crt.sh/?dNSName=%25.mil

Most of them chain up to a USG root, and are not publicly trusted. There are 78 unique .mil sub/domains who had certs issued that chained to a commercial provider:

https://gist.github.com/konklone/00afdaa2b9c4fe1e7843

But for the purpose of establishing a .mil domain list, you may want to look at (and dedupe) the domains that chain up to a USG root. That said, many of these domains (such as www.mil) do not actually seem to resolve over public DNS, and they may either be old, invalid, or purely internal.

I defer to you on how to best incorporate them -- my suggestion is to incorporate those which have public DNS records of any sort (whether or not they respond over HTTP).