lumenpallidium/jepa

Experiments in Joint Embedding Predictive Architectures (JEPAs).

jepa's Issues

Question not an issue

Hi, this looks amazing. I have a quick question. given that we have the encoder transformer and the predictor transformer . How would you go about testing the KNN? Lets say we have a 224x224 image, if you use patches , the context encoder (no masking) would output many many embeddings for one image. Thank you!

Add more modalities

In particular, audio JEPA (perhaps using a soundstream-like encoder) and a 3D array JEPA (for medical images, point clouds etc). Likely only requires updates to the patcher.