Difference between I-JEPA and data2vec2.0?

[facebookresearch]

Difference between I-JEPA and data2vec2.0?

[ballasnicolas]

Hi rayleizhu, thanks for your interest in our work!

Yes, you are correct, both I-JEPA and data2vec-(v1,v2) learns representation by trying to predict missing information in feature space. Data2vec-v2 shows that you could learn very strong pretrained models for end2end finetuning on given tasks. In contrast, our goal was to demonstrate that this learning principle also can learns strong off-the-shelf representation that are good in frozen evaluation and low-shot scenario.

This led to different choices in the model architecture (conv decoder vs ViT predictor) and masking strategies. You can find comparisons with data2vec in the paper (data2vec-v2 was not yet released during the development of the paper).

[lxysl]

Hi rayleizhu, thanks for your interest in our work!

Yes, you are correct, both I-JEPA and data2vec-(v1,v2) learns representation by trying to predict missing information in feature space. Data2vec-v2 shows that you could learn very strong pretrained models for end2end finetuning on given tasks. In contrast, our goal was to demonstrate that this learning principle also can learns strong off-the-shelf representation that are good in frozen evaluation and low-shot scenario.

This led to different choices in the model architecture (conv decoder vs ViT predictor) and masking strategies. You can find comparisons with data2vec in the paper (data2vec-v2 was not yet released during the development of the paper).

Thanks for your reply! But I still confused about the so called off-the-shelf representation. Could you please give some detailed explanation or insight on that?

[ballasnicolas]

'off-the-shelf' means that we don't want to finetune the encoder after pretraining, i.e. we froze its parameters after pretraining.

In the paper, we mostly focus on linear evaluation where we train a linear classifier on top of a frozen pretrained encoder. Let me know if that answer you question.

[lxysl]

Yes！And thanks for your kindly reply~