zeyadtarekk/intelliquery

A semantic search indexing system designed to efficiently retrieve top matching results from a database of 20 million documents. Given the embedding of a search query, it quickly identifies and returns the most relevant documents

IntelliQuery

📝 Table of Contents

• 📝 Table of Contents
• 📙 Overview
• Get Started
  • Infernce Mode
  • Run Locally
• Methods
  • Inverted File Inedex (IVF)
  • Local Sensitive Hashing (LSH)
  • Product Qunatization (PQ)
  • PQ-LSH
• 🕴 Contributors
• 📃 License

📙 Overview

Given the embedding of the search query we can efficent get the top matching k results form DB with 20M document.The objective of this project is to design and implement an indexing system for a semantic search database.

Get Started

Infernce Mode

Check Final Notebook

https://github.com/ZiadSheriif/IntelliQuery/blob/main/Evaluate_ADB_Project.ipynb

Run Locally

Clone Repo

git clone https://github.com/ZiadSheriif/IntelliQuery.git

Install dependencies

pip install -r requirements.txt

Run Indexer

$ python ./src/evaluation.py

Methods

Inverted File Inedex (IVF)

This is out final Approach with Some Enhancements

1. Changed MiniBatchKMeans to regular KMeans
2. We calculate initial centroids with just the first chunk of data
3. Introduced parallel processing for different regions

Local Sensitive Hashing (LSH)

Product Qunatization (PQ)

PQ-LSH

It Combines both LSH & PQ

🕴 Contributors

Ziad Sherif

Zeyad Tarek

Abdalhameed Emad

Basma Elhoseny

📃 License

This software is licensed under MIT License, See License for more information ©Ziad Sherif.