How to load ijepa checkpoints?

[facebookresearch]

How to load ijepa checkpoints?

[CUN-bjy]

It seems that the checkpoint is saved as a DDP module, but you tried to load it into a pure encoder.
This can be the solution.

ckpt = torch.load(load_path, map_location=torch.device('cpu'))
pretrained_dict = ckpt['encoder']

# -- loading encoder
for k, v in pretrained_dict.items():
  encoder.state_dict()[k[len("module."):]].copy_(v)

[namrahrehman]

Thank you so much @CUN-bjy, it worked!
However, I couldn't classify the images due to limited computing resources.
Thanks again for your help!

[lazarosgogos]

Hello everyone.
Any insights as to how one can take a checkpoint/pretrained model and use it for some downstream task? As in, load the already trained weights into a model, freeze them and use this to train a classifier for another dataset (e.g. CIFAR 10).

Also, what is the complete answer to the question posed above? For example, where does the encoder variable come from? A complete code snippet would be of great help.

I've figured the steps for loading the checkpoint are the following:

• Take the state_dict
• Initialize the corresponding ViT (e.g. ViT-H with the init_model function from src.helper.py)
• Initialize an optimizer with init_opt
• Then? Which parts of the IJEPA architecture are needed to utilize the embeddings in some other task as described earlier?

This is for research purposes by myself, an undergrad.

Thank you in advance!

[lange4531]

Hello everyone. Any insights as to how one can take a checkpoint/pretrained model and use it for some downstream task? As in, load the already trained weights into a model, freeze them and use this to train a classifier for another dataset (e.g. CIFAR 10).

Also, what is the complete answer to the question posed above? For example, where does the encoder variable come from? A complete code snippet would be of great help.

I've figured the steps for loading the checkpoint are the following:

• Take the state_dict
• Initialize the corresponding ViT (e.g. ViT-H with the init_model function from src.helper.py)
• Initialize an optimizer with init_opt
• Then? Which parts of the IJEPA architecture are needed to utilize the embeddings in some other task as described earlier?

This is for research purposes by myself, an undergrad.

Thank you in advance!

This would be great to have a solution on if someone has managed to get it working!

[VimukthiRandika1997]

Hello everyone. Any insights as to how one can take a checkpoint/pretrained model and use it for some downstream task? As in, load the already trained weights into a model, freeze them and use this to train a classifier for another dataset (e.g. CIFAR 10).

Also, what is the complete answer to the question posed above? For example, where does the encoder variable come from? A complete code snippet would be of great help.

I've figured the steps for loading the checkpoint are the following:

• Take the state_dict
• Initialize the corresponding ViT (e.g. ViT-H with the init_model function from src.helper.py)
• Initialize an optimizer with init_opt
• Then? Which parts of the IJEPA architecture are needed to utilize the embeddings in some other task as described earlier?

This is for research purposes by myself, an undergrad.

Thank you in advance!

You can take pretrained Target Encoder and finetune on your custom datasets. But finetuning would be costly as you can see from the size of encoder: It has 32 blocks as Vit based models require lot of data to be tuned for the task at hand. Also GPU requirement is higher. One possibility would be training a MLP (1 layer, 2 layers, ....N layers) on top the encoder for task of interest.

Possible downstream tasks would be image similarity, classification, etc. Feature extraction is the main component, you can use it anywhere!

[FalsoMoralista]

Hello everyone. Any insights as to how one can take a checkpoint/pretrained model and use it for some downstream task? As in, load the already trained weights into a model, freeze them and use this to train a classifier for another dataset (e.g. CIFAR 10).

Also, what is the complete answer to the question posed above? For example, where does the encoder variable come from? A complete code snippet would be of great help.

I've figured the steps for loading the checkpoint are the following:

* Take the state_dict

* Initialize the corresponding ViT (e.g. ViT-H with the `init_model` function from `src.helper.py`)

* Initialize an optimizer with `init_opt`

* Then? Which parts of the IJEPA architecture are needed to utilize the embeddings in some other task as described earlier?

This is for research purposes by myself, an undergrad.

Thank you in advance!

I have developed a fine-tuning code for the I-JEPA here very based on the ViT-MAE in order to reproduce the experiments conducted here right now it's seeming to work, as the loss is decreasing, but I'm not managing to get much reduction on the test error so I am currently investigating that. If you need help contact me on discord (at falsomoralista) or something.