marda-alliance/metadata_extractors_registry

Archive. See Datatractor Yard, below:

metadata_extractors_registry's Issues

Nest API under versioned URL

We should deploy the API with a /v<major> prefix (e.g., /v0).

Allow example files to be provided as remote resources (with bibliographic data)

Committing an example file for everything under the sun into this repo directly is probably not the best way forward. We should have a mechanism for providing persistent links to example files (e.g., archived files with DOIs) that the registry can download and use as test data. Probably this ends up being a registry of example files too, in that case...

Sorting entries in alphabetical order

Now we have a few more entries, we can see that entries are sorted in order of addition, but alphabetical probably makes more sense.

Set registry version with SCM tools

Deployed registry version should match e.g., release-<commit>

Missing registry submission instructions

Make deployment job depend on validation

`registered_extractors` field in `/filetypes` response not populated correctly in API

but somehow magically works in the HTML app.

Time to add proper testing!

Automated ingestion pipelines

Potential idea as discussed: save .marda.yaml as a file in a GitHub repo that outlines the entry in this registry, then simply submit the link to the repo here.

Repo can then be watched by the CI of the registry and entries updated.

Can also make a form/UI for creating the initial yaml file.

Immediate file types and extractors we are interested in registering

This issue can be used to vaguely track specific filetypes and extractors we want in the registry:

• CIF -> https://github.com/cod-developers/cod-tools (@ml-evs)
• NeXuS -> https://github.com/dlce-eva/python-nexus (@ml-evs)
• TIF files from SEM machines (see below) -> https://github.com/kit-data-manager/SEM-Mapping-Tool
  • This is a great example as it takes a generic format (TIF), specialises it into TIF files produced by Zeiss instruments for SEM, and extracts metadata in a way that is beyond what a "casual" parser would do (OCR on the metadata in the image, rather than just reading a file) (developed by @reetuelzajoseph -- hope you don't mind me tagging you!) -- it also has a well-defined set of schemas already!

Fly deployment failures

Intermittent build problems e.g., "remote builder app unavailable", seem to be related to docker layer caching. Destroying the "builder" (not the app) with fly destroy <builder name> seems to work on the next build.

Add lookup endpoint for extractors that support a given file type

This could either be additional data added to the single entry file type endpoint, e.g.,

/registry/filetypes/biologic-mpr also returns

"relationships": {
    "extractors": [
        "yadg"
    ]
}

or it could be a search endpoint for registry/extractors?supported_filetypes=biologic-mpr.

I think I prefer the former to start with.

Enforce that entry filename is the same as ID within the entry

`FileType`: Include example files for the filetype.

The most important thing in our "schema" was perhaps the link to an example file (as simply the name of the instrument and extension oftentimes doesn't describe much). Perhaps one could also consider adding it here.

To make this possible, I once started a "chemical files registry" here, where I use also a yml schema similar to yours: https://github.com/kjappelbaum/chemical-files-registry/blob/master/fileDescriptions/analyticalMethods/thermogravimetricAnalysis/ta-txt/description.yml.

I didn't have any time to work on this, but the idea was to collect example files and link to them (and the filetype schema) from the parser registry.

Originally posted by @kjapplebaum in marda-alliance/metadata_extractors_schema#2 (comment)

Use fly volumes to run a real database

This will enable free-text search over entries (if we use e.g., Mongo). Can also just use sqlite or something if that ends up being easier.

https://fly.io/docs/reference/volumes/

Fly flagged us (me) as high-risk

See Materials-Consortia/optimade-python-tools#2027

and as such, the registry is down. Good time to migrate!

Make use of example files for each type in registry

• Make sure the URLs are resolvable through Fly (probably pointing to GitHub raw links)
• Add the files as examples in the file types models and expose them for validation
• Validate entries in the data folder and make sure they correspond to registered file types