Design of tooling for bulk metadata fixes/updates

[internetarchive]

Design of tooling for bulk metadata fixes/updates

[bnewbold]

The pattern that seems to be emerging has three components:

• free-form elasticsearch queries to identify entities needing fixes
• an entity-specific "want()" function that looks at entity and confirms whether it is worth doing an update
• one or more entity-specific map functions that make fixes/updates to the entity

These would be combined by tooling that executes the elasticsearch query in a streaming fashion, fetches matched entities from the API, confirms that they should be updated, runs any cleanup functions, confirms that the entity actually changed (if not, skip update), then pushes the updates in batches. Variants of this can be dry runs (don't do actual update), limited runs (possibly with random sampling), not auto-accepting the updates (leave for human review), showing stats/examples to the user (eg, how many elastic hits, what transform looks like).

A missing piece is to integrate the want() and cleanups with import and other update code to allow cleanups to happen at the same time as other edits. Also to include in review bots for quality review.

An alternative case is to use some external bulk metadata list and iterate through it. The existing import framework is probably Good Enough for this.

[bnewbold]

We now have several "cleanup" workers implemented in python, as well as the fatcat-cli workflow for simple updates. The current workflow is to document "bulk" metadata updates in this git repo when they happen.