Unexpected Zero-shot Results

[antoyang]

Unexpected Zero-shot Results

[antoyang]

I would try the following things:

• Distributing on a single GPU
• Trying the text-only variant without pretrained checkpoint, which should already give non-zero performance

[terryyz]

Hi @antoyang,
Thanks for pointing out!
Please let me know if you can get expected results via the non-distribution way. Not too sure if this is due to my environment issue or some bugs in the implementation...
Cheers

[antoyang]

I have just verified that the performance of a pretrained checkpoint is the same without distributing. Some notes:

• It looks like you are trying to use a fine-tuned checkpoint and not the one that actually does zero-shot.
• Make sure that the weights of the language model from Hugging Face are loaded. Many of the checkpoints I host do not include these weights for storage issues, which should not be an issue as they are frozen so unchanged through cross-modal training.