hrbrmstr/certstream-rust

Extract all domains from a CertStream-compatible CTL websockets server to RocksDB

certstream-rust

Small funemployment Rust project to build a CLI program to read the websocket stream from https://certstream.calidog.io and log all certificate domains to a RocksDB database, plus a small utility to dump the gathered domains from RocksDB.

Building

git clone [email protected]:hrbrmstr/certstream-rust
cargo build --release 

Installing

The following will put:

• certstream
• dumpdoms

into ~/.cargo/bin unless you've modified the behaviour of cargo install.

$ cargo install --git https://github.com/hrbrmstr/certstream-rust

Read from CertStream websocket

USAGE:
    certstream [OPTIONS] --dbpath <DBPATH>

OPTIONS:
    -d, --dbpath <DBPATH>        
    -h, --help                   Print help information
    -p, --patience <PATIENCE>    [default: 5]
    -s, --server <SERVER>        [default: wss://certstream.calidog.io/]
    -V, --version                Print version information

e.g.

$ certstream --dbpath=~/Data/cert_doms.1 # kill or ^C to stop

See the domains

USAGE:
    dumpdoms --dbpath <DBPATH>

e.g.

$ dumpdoms --dbpath=~/Data/cert_doms.1 | tail -30
zyw017777.direct.quickconnect.to
zywolapki.polmedia.webd.pl
zyx.dzign4u.com
zyxus.keenetic.pro
zyy.aob.mybluehost.me
zyy.kqv.mybluehost.me
zyyid.muo.cc
zyzfnh.net
zyzyxllc.com
zz188.xyz
zz3wd.com
zzalale.com
zzb.cx
zzbolt.org
zzdzkbuxtcwxiru62jefplj2dy.ap-southeast-2.es.amazonaws.com
zzekasd.xyz
zzgljy.com
zznptabez1jijw5z.myfritz.net
zzsmarketing.com
zzx.admin.thinker.vc
zzx.api.thinker.vc
zzx.oauth.thinker.vc
zzx52.direct.quickconnect.to
zzyhomv.shop
zzyzxvape.synology.me
zzz.com.ar
zzz.org.ua
zzz.zzz.org.ua
zzz4ncfxemkkuulse6ctkwzrau.eu-west-3.es.amazonaws.com
zzzzbw.cn

certstream-rust's Issues

Panic at main (http 521)

After running for a while the certstream program failed with the following output:-

thread 'main' panicked at 'Failed to connect: Http(Response { status: 521, version: HTTP/1.1, headers: {"date": "Sun, 09 Oct 2022 18:28:01 GMT", "content-type": "text/html", "transfer-encoding": "chunked", "connection": "keep-alive", "cache-control": "no-store, no-cache", "report-to": "{\"endpoints\":[{\"url\":\"https:\/\/a.nel.cloudflare.com\/report\/v3?s=45%2BwYJkZC5m3mBQfU8uGv%2FguD%2FpKGNy6MZ4MSuBAJwTH3yxa3DPCiIi38pIXmmYiT%2FV9E9kef4jiy46l4G4b3P31DGeINQmKZdCvxgLXtdCuihUJT8ZNUJx0XKFVEDFSa3ZC3JNFOK0nqMr6DfAhdtoKoLs%3D\"}],\"group\":\"cf-nel\",\"max_age\":604800}", "nel": "{\"success_fraction\":0,\"report_to\":\"cf-nel\",\"max_age\":604800}", "server": "cloudflare", "cf-ray": "75792c774e017774-LHR", "alt-svc": "h3=\":443\"; ma=86400, h3-29=\":443\"; ma=86400"}, body: None })', src/certstream/main.rs:75:71