txtai executes machine-learning workflows to transform data and build AI-powered text indices to perform similarity search.
Summary of txtai features:
- ๐ Large-scale similarity search with multiple index backends (Faiss, Annoy, Hnswlib)
- ๐ Create embeddings for text snippets, documents, audio and images. Supports transformers and word vectors.
- ๐ก Machine-learning pipelines to run extractive question-answering, zero-shot labeling, transcription, translation, summarization and text extraction
- โช๏ธ๏ธ Workflows that join pipelines together to aggregate business logic. txtai processes can be microservices or full-fledged indexing workflows.
- ๐ API bindings for JavaScript, Java, Rust and Go
- โ๏ธ Cloud-native architecture that scales out with container orchestration systems (e.g. Kubernetes)
txtai and/or the concepts behind it has already been used to power the Natural Language Processing (NLP) applications listed below:
Application | Description |
---|---|
paperai | AI-powered literature discovery and review engine for medical/scientific papers |
tldrstory | AI-powered understanding of headlines and story text |
neuspo | Fact-driven, real-time sports event and news site |
codequestion | Ask coding questions directly from the terminal |
txtai is built with Python 3.6+, Hugging Face Transformers, Sentence Transformers and FastAPI
The easiest way to install is via pip and PyPI
pip install txtai
You can also install txtai directly from GitHub. Using a Python Virtual Environment is recommended.
pip install git+https://github.com/neuml/txtai
Python 3.6+ is supported. txtai has the following environment specific prerequisites.
Optional audio transcription requires a system library to be installed
Run brew install libomp
see this link
Install C++ Build Tools
The examples directory has a series of notebooks and applications giving an overview of txtai. See the sections below.
Notebook | Description | |
---|---|---|
Introducing txtai | Overview of the functionality provided by txtai | |
Build an Embeddings index with Hugging Face Datasets | Index and search Hugging Face Datasets | |
Build an Embeddings index from a data source | Index and search a data source with word embeddings | |
Add semantic search to Elasticsearch | Add semantic search to existing search systems | |
Extractive QA with txtai | Introduction to extractive question-answering with txtai | |
Extractive QA with Elasticsearch | Run extractive question-answering queries with Elasticsearch | |
Apply labels with zero shot classification | Use zero shot learning for labeling, classification and topic modeling | |
API Gallery | Using txtai in JavaScript, Java, Rust and Go | |
Building abstractive text summaries | Run abstractive text summarization | |
Extract text from documents | Extract text from PDF, Office, HTML and more | |
Transcribe audio to text | Convert audio files to text | |
Translate text between languages | Streamline machine translation and language detection | |
Similarity search with images | Embed images and text into the same space for search | |
Run pipeline workflows | Simple yet powerful constructs to efficiently process data |
Application | Description |
---|---|
Workflow builder | Build and execute txtai workflows. Connect summarization, text extraction, transcription, translation and similarity search pipelines together to run unified workflows. |
Image search | Image similarity search application. Index a directory of images and run searches to identify images similar to the input query |
Demo query shell | Basic similarity search example. Used in the original txtai demo. |
Full documentation on txtai including configuration settings for pipelines, workflows, indexing and the API.
For those who would like to contribute to txtai, please see this guide.