How could I train ImageBind completely from scratch?

[facebookresearch]

How could I train ImageBind completely from scratch?

[fabawi]

you can train the model without LoRA using ImageBind-LoRA. Simply remove the --lora argument when calling train.py and set --full_model_checkpointing. I don't have the resources to fine-tune it but it should work in theory. Try it out on the toy dataset provided in the repo, and later you can try to implement dataloaders for the original datasets.

On a side note, LoRA adaptation was shown to outperform full model fine-tuning in many instances and for different applications --- this is not specific to ImageBind-LoRA but a general statement about low-rank adaptation. If you would like to see the outcomes of fine-tuning on LoRA, check out the example. The weights are in the repo itself. Just install the requirements and follow the instructions in the README.

[wendellgithub0206]

I want to know how to train and how to apply the model after training is complete?
How to present the results?

[Wolfwjs]

Hi @ChloeL19 ， May I ask if you have been able to train the model from scratch？

[AI-ML-Enthusiast]

you can train the model without LoRA using ImageBind-LoRA. Simply remove the --lora argument when calling train.py and set --full_model_checkpointing. I don't have the resources to fine-tune it but it should work in theory. Try it out on the toy dataset provided in the repo, and later you can try to implement dataloaders for the original datasets.

On a side note, LoRA adaptation was shown to outperform full model fine-tuning in many instances and for different applications --- this is not specific to ImageBind-LoRA but a general statement about low-rank adaptation. If you would like to see the outcomes of fine-tuning on LoRA, check out the example. The weights are in the repo itself. Just install the requirements and follow the instructions in the README.

Why use LoRA? Did ImageBind paper use it?