btrask/hash-archive-js

Site for publicly archiving content hashes - https://hash-archive.org/

hash-archive-js's Issues

Automated system for updating critical URL list

Right now I'm adding stuff to the URL list manually, but keeping it up to date is already looking burdensome. We could really use a tool for quickly fetching the new download URLs from a site.

For scraping, the best option I currently know of is Electron.

Provenance tracking

We need to store and report provenance data for hashes in order to start importing 3rd party datasets. Important for btrask/hash-archive#3 and btrask/hash-archive#4.

This has multiple parts:

• Enhance the database schema to track "provenance" or "origin" for each hash <-> URL mapping
• Come up with an intuitive user interface to display hashes that might not be "trustworthy" (haven't been verified by the site itself)
• Sort of separate, but expand the schema to support non-URL sources

[not-a-bug] third-party hash database

I've been told that your hash-archive project may have some use of my little database - I hashed bunch of semi-random sources and put them into purplehash-datasource. My main interest in this is mapping of (crc32,md5,sha1,sha256,…), file format is pretty stable over the course of time, but checksums included may vary (some have whirlpool+ripe-md and some use tth+aich, currently I'm using tth+aich option).

Hashing is done with rhash, because it is pretty fast and supports custom output templates.