jamino30/dandi-search-response-ui

Interactive UI to showcase and evaluate performance of Dandi Search

Dandi Search UI

Running Locally

1. Create envfile.txt (in root directory) and fill with the following environment variables.

QDRANT_HOST=
QDRANT_PORT=
QDRANT_API_KEY=
OPENAI_API_KEY=
DANDI_API_KEY=
AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
AWS_DEFAULT_REGION=

2. To launch the services in docker-compose.yml, execute the following:

docker compose up --build

Deployment

In order to deploy a new version, please invoke the deploy_new_version workflow with your desired branch in GitHub Actions.

Stress-testing

In order to stress test the app locally, we include a setup for Locust tooling.

Steps:

1. Edit the stress-test/locustfile.py to reflect simulating endpoints that can be invoked via the Locust test
2. Launch the Locust UI via docker-compose -f docker-compose.locust.yml up --scale worker=4 -- note: you may need to bring up/down the containers locally to enable hot-reloading in your locustfile.py

Purpose

• Dandi's LLM-powered search enhances user experience by enabling precise retrieval of relevant dandisets based on scientific inquiries, surpassing basic text and semantic search methods
• The Dandi Search Response UI empowers users to actively contribute to the evaluation of semantic search performance and the creation of valuable testing datasets for model assessment.
• Through the optimization of semantic search performance and the identification of the most effective LLM-powered processes, an optimal search engine can be tailored for the Dandi Archive.

Todo

• Avoid showing a fixed number of dandisets (but place a limit) to allow for more accurate results/responses (e.g. no relevant dandisets, 2 relevant dandisets). There is no point in forcing a fixed number of dandisets if they have no relevance at all.
• Allow users to upload their own Dandiset IDs that do not appear in the list (but may be relevant to the query or may seem like they are related to the query but are not).
• Show confidence levels of dandisets chosen via similarity search (and place a threshold at which dandisets should be considered "relevant" or "not relevant").
• Improve vector embeddings used for similarity search to allow for identification of dandisets relevant to scientific or quantatitve queries (e.g. "Show me dandisets that contain two or more species.")
• Llama 2 for text embeddings (alongside ada-002)

dandi-search-response-ui's Issues

no response error