ml4ai/askem-ta1-datamodel

askem-ta1-datamodel's Issues

Add full sentence to provenance info

This will make it easier to debug the information for quality purposes

Implement scenario context

Add temporal and location context to the data models and importers

Processing plain text files outputs from SKEMA TR crashes the importer

Fix this issue.

Indent serialized json outputs

For easier human consumption, export json files with indentation

import_arizona() does not work on the latest Arizona extraction standardized_extractions_documents_5Feb1-s2.0-S2211379721005490-main--COSMOS-data.json

I paste the head sample of the latest Arizona extraction (format A) for 1-s2.0-S2211379721005490-main. This is totally different from the format of extraction sample (format B). I was wondering how to convert from A to B?
{ "variable_statements": [ { "id": "R:-1804348017", "variable": { "name": "LV", "metadata": [], "dkg_groundings": [], "column": [], "paper": { "id": "1-s2.0-S2211379721005490-main.pdf", "file_directory": "1-s2.0-S2211379721005490-main.pdf", "doi": "" }, "equations": [] }, "value": { "value": "LotkaVolterra", "type": "description", "dkg_grounding": null }, "metadata": [ { "type": "text_span", "value": "LotkaVolterra (LV" }, { "type": "cosmos_location", "value": "1,11,2185,2202" }, { "type": "scenario_location", "value": "V. Volterra" }, { "type": "scenario_location", "value": "Lotka-Volterra" } ], "provenance": { "method": "SKEMA", "description": "SKEMA text reading pipeline v0.5" } },

Descriptions and SpecValues missing from output

The output json files are not keeping the descriptions and spec values. Fix it and make sure it appears in the merged output

Automatically build and upload image to DockerHub

Configure GitHub's actions to trigger a build and upload of docker image to Lum's Docker Hub account upon merging to main.

`text_descriptions` and `value_descriptions` not imported correctly by `import_arizona`

Support for Python 3.8

The current state of the code requires python >= 3.9. To bundle this up with the SKEMA services, we need to make it work on python 3.8

include evaluation metric functions in this repo

We would benefit form using the extraction metrics functions directly in our code,

We have some internal tests we run and it’s good for our testing to be able to run without depending on a live service.

It would also help us quickly understand any errors we may trigger (which is harder when using an API) and then work around them.

We're happy to contribute any other changes/variations etc to that function back into this repo.

Fix page and block indices in output

Currently, the indices are always set to 0. It should carry the correct values from the Skema output json file