onto-med/concept-graphs

concept-graphs's Issues

Use `requirements.txt` to manage Python dependencies

All Python dependencies should be listed in a requirements.txt file (see pip documentation).
The file can then be used to simplify the Dockerfile and setting up a development environment.

Permission denied for tmp folder

with the new docker file there are some permission denied issues when saving the results (the docker volume folder needs to be writable by the user inside the docker container. I'm not sure how to handle this the correct way. I changed the access of the volume folder to 777...

pipeline: add skip_steps

when starting the pipeline it should be possible to skip steps (if there are already prerequisite steps available and one would only like to e.g. reprocess the graph creation step).

Model handling

The reason why the image is comparably small now, is probably because only the spacy models for preprocessing are downloaded and not the models for the sentence-transformer, i.e. the embedding step. That might be reasonable because one wouldn't need the english model for if one only wants to process german data.
The only drawback seems to be that a respective model needs to be downloaded during runtime and this could potentially throw a wrench into the pipeline process. It might be best if we include a model check step before any of the requested pipeline steps is started and if a model is missing, it will be downloaded. This would mean that we could remove the download step for the spacy models from the docker file as well.

apply recent changes to singular endpoints as well

all recent improvements/changes were made with the /pipeline endpoint in mind. Make sure that all singular endpoints (e.g. /preprocessing) also use those features

no access to files created from docker container

When I want to remove files from a volume that were created from within the docker container, it is refused: we need an endpoint that allows for deletion of project-folders

elasticsearch: fallback id

add a field to the document_server_config that governs which field in ES is the document id if no id field is given

negspacy config not hardcoded

right now the configuration for the negspacy component is hardcoded. but this should be done in preprocessing_config.