some question about 'ClipCaptionPrefix'

[iendi]

some question about 'ClipCaptionPrefix'

[FeiElysia]

Hey, thanks for your appreciation！

I'd like to answer your first question first. Indeed, hyperparameter frozen_gpt means freezing the whole language model during training.

As indicated in the official document, gpt.eval() indeed has any effect only on certain modules, e.g., Dropout, BatchNorm, etc.

However, it's important to note that when utilizing ClipCaptionPrefix.parameters(), we will not feed the parameters of gpt2 to the optimizer because we've redefined the function ClipCaptionPrefix.parameters(). The parameters of gpt2 therefore will not be updated.

You can also train our ViECap in a minimal epoch (one epoch), using a small dataset like Flickr30k, with frozen_gpt = True. and test by 1) loading ViECap that you've trained yourself, and 2) loading gpt2 from pre-trained weights at huggingface to assess whether the two captioning results are consistent.

[iendi]

ok, I will try it. Thanks for reply!
Again, thank you for sharing this exciting work!