Batching for CLIP Explainability

[hila-chefer]

Batching for CLIP Explainability

[Alacarter]

Thank you very much for the batch example in the colab! I am able to replicate your batching code on my end, and the heatmaps look similar to what I had before and generate a lot faster. Appreciate the help and response!

[hila-chefer]

Hi @Alacarter, thanks for your interest in our work!
I updated our colab notebook to demonstrate how batching can be done.
In our notebook, I used a single image with multiple texts. In order to create a "one-hot" vector, I duplicated the image batch_size times and then created a "one-hot" vector for each pair of (image, text).
Similarly, if you have a batch of images and texts, simply create a one hot with the shape batch_zise x batch_size and set it to be the identity matrix.

I hope this helps, please refer to our colab for all the details :)
Hila.