yuedajiong-question #02: same or different encoding space?

[facebookresearch]

yuedajiong-question #02: same or different encoding space?

[MidoAssran]

Hi @yuedajiong ,

1. For the first question, if you were to directly match the output of the context and target encoder, you would learn invariant representations that collapse the masked and unmasked representations to the same point. However, the idea behind I-JEPA is that we want to be able to predict masked regions from visible regions, not collapse them together. Hence, the predictor is essential for computing this mapping.
2. For the second question, this is definitely valid. You can pretrain the visual encoder and then just train the predictor in this frozen latent space, but the idea behind I-JEPA is to learn to represent the world by predicting it, hence we were interested in using this approach to actually train a visual encoder. You could of course still use the same network with unfrozen weights for the context and target, but you would need an explicit regularization to prevent collapse (e.g., using the information maximization terms in the diagram you pointed out). This would be an interesting research questions!