scai-bio/index

Intelligent data steward toolbox using Large Language Model embeddings for automated Data-Harmonization

INDEX – the Intelligent Data Steward Toolbox

INDEX is an intelligent data steward toolbox that leverages Large Language Model embeddings for automated Data-Harmonization.

Installation

Local Development Server

Starting the backend

cd api
pip install -r requirements.txt
uvicorn routes:app --reload --port 5000

Navigate to localhost:5000 to access the backend.

Starting the frontend

cd client
ng serve

Navigate to localhost:4200 to access the frontend.

Docker

You can start both frontend and API using docker-compose:

docker-compose -f docker-compose.local.yaml up

Configuration

TODO: Add configuration instructions

index's Issues

Create (temporary) pypi release

• intex
• datasteward
• datastew

Add frontend tests in ci

@mehmetcanay could you add the angular test to the test ci as in PDataviewer?

Find way to only dynamically load pre-configured (list of) models during build process

Bug: Duplicate entries crash DB

curl -X PUT "[https://index.bio.scai.fraunhofer.de/concepts/id001/mappings?terminology_id=test_ab3&concept_name=cough&text=erkaeltung"](https://index.bio.scai.fraunhofer.de/concepts/id001/mappings?terminology_id=test_ab3&concept_name=cough&text=erkaeltung%22) -H "accept: application/json"
{"detail":"Failed to create or update concept: (sqlite3.IntegrityError) UNIQUE constraint failed: concept.id\n[SQL: INSERT INTO concept (id, name, terminology_id) VALUES (?, ?, ?)]\n[parameters: ('id001', 'cough', 'test_ab3')]\n(Background on this error at: [https://sqlalche.me/e/20/gkpj)"}](https://sqlalche.me/e/20/gkpj)%22%7D)

{"detail":"Failed to create or update terminology: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (sqlite3.IntegrityError) UNIQUE constraint failed: concept.id\n[SQL: INSERT INTO concept (id, name, terminology_id) VALUES (?, ?, ?)]\n[parameters: ('id001', 'cough', 'test_ab3')]\n(Background on this error at: https://sqlalche.me/e/20/gkpj) (Background on this error at: [https://sqlalche.me/e/20/7s2a)"}](https://sqlalche.me/e/20/7s2a)%22%7D)

Add description to excel to excel map

Also add parameter to determine number of closest matches (default = 1) and add those as columns to the resulting dataframe

Move db file to own directory

Having the db file in the same directory as a python package will cause issues when mounting the directory in a data container or as a PVC

Add pagination to get endpoints

Integrate scatter plot visualization into REST API

Update api version in routes.py during release

During the Container build, the api version in routes.py should be uodated by action to the current version tag. This makes sure the resulting docker build will have the current release version as API version.