simonw/llm-clip

Generate embeddings for images and text using CLIP with LLM

llm-clip

LLM plugin for embedding images and text using CLIP

Installation

Install this plugin in the same environment as LLM.

llm install llm-clip

Usage

Once you have installed an embedding model you can use it to embed text like this:

llm embed -m clip -c 'Hello world'

Or an image like this:

llm embed -m clip --binary -i IMG_4801.jpeg

Embeddings are more useful if you store them in a database - see the LLM documentation for details.

To embed every photograph in a folder and save them in a collection called "photos":

llm embed-multi photos -m clip --binary --files photos/ '*.jpg'

You can then search for photos of specific things like this:

llm similar photos -c 'bunny'

Development

To set up this plugin locally, first checkout the code. Then create a new virtual environment:

cd llm-clip
python3 -m venv venv
source venv/bin/activate

Now install the dependencies and test dependencies:

pip install -e '.[test]'

To run the tests:

pytest

llm-clip's Issues

finetuned clip

Hi,

I've learned how to fine tune CLIP with my own images and captions. I end up with a pytorch.bin model, and I expect that the config.json files would more or less be the same as eg https://huggingface.co/sentence-transformers/clip-ViT-B-32/blob/main/0_CLIPModel/config.json

Forgive me for my newbness, but in llm_clip.py I would just point

self._model = SentenceTransformer('clip-ViT-B-32') to the location of my own folder with the .bin and .json files, right?

location of the embeddings

Hey, very interesting plugin for llm and love it.

Just curious, where is the embeddings stored in file system?

Ability to run alternative models

The model ID is hard-coded here:

llm-clip/llm_clip.py

Lines 20 to 23 in ffa91b3

	def embed_batch(self, items):
	# Embeds a mix of text strings and binary images
	if self._model is None:
	self._model = SentenceTransformer('clip-ViT-B-32')

It would be neat if you could register and use alternative models.

Relevant: https://carleton.ca/xlab/2023/archaeclip-or-building-a-visual-search-engine-for-archaeology/ - about fine-tuning a custom CLIP model for use with this plugin.

Initial plugin

Refs:

• simonw/llm#264