feielysia/viecap

Transferable Decoding with Visual Entities for Zero-Shot Image Captioning, ICCV 2023

viecap's Issues

about infer by batch

Congrats on your paper being accepted by iccv 2023！
looking at the infer_by_batch.py file in your source code, I don't seem to see the inputs that use batch data, or am I just being careless and not understanding?

求解答：语言模型是opt时，遇到问题

怎么爱莉的图不在论文里[dog]

我也天天拿她的图测训出来的多模态模型哈哈

代码写的真好

可读性很强，作为模板学习，感谢休伯利安的舰长程序员hhh

CLIP在batch和非batch情况下编码文本结果不一致

作者您好，我在实验过程中发现，clip的encoder.encoder_text()在用batch text输入和非batch text输入时，同一个text提取到的feature不一样，请问您知道是什么原因吗？

some question about 'ClipCaptionPrefix'

Thank you for sharing this exciting work! The code and comments are pretty standard and I really learned a lot from it.

I would like to know: Does the hyperparameter frozen_gpt mean freezing the whole gpt model during training, I notice that the ClipCaptionPrefix code is as follows：

class ClipCaptionPrefix(ClipCaptionModel):

    def parameters(self, recurse: bool = True):
        return self.mapping_network.parameters()

    def train(self, mode: bool = True):
        super(ClipCaptionPrefix, self).train(mode)
        self.gpt.eval()
        return self

I think gpt.eval() just stop the Batch Normalization and Dropout module, and I print the params of gpt2 after set frozen_gpt=True, which goes as follows:

for name, param in model.gpt.named_parameters():
  print(name, ":", param.requires_grad)

the output:

transformer.wte.weight : True
transformer.wpe.weight : True
transformer.h.0.ln_1.weight : True
...

So im wondering whether the whole gpt2 model is frozen，or just the BN and Dropout layer.

Thanks in advance!

clip_project_length应该怎么确定

学长好，最近在尝试使用ViT-L/14作为backbone时发现直接生成的描述都是缺少主语的不完整句子，看了半天觉得是clip_project_length这里的问题，不知道您是怎么确定这个值的呢，以及对于换成ViT-L/14您有没有什么指教。
感谢学长。

GPT2的预训练模型加载问题

尊敬的作者您好，我在运行train_coco.sh时遇到如下问题，请问该如何解决呢？
我想应该是加载GPT2预训练模型权重的问题，我搜索尝试了很多方法但都无效，希望您能解答一下，谢谢！
Traceback (most recent call last):
File "main.py", line 168, in
main()
File "main.py", line 152, in main
datasets = CaptionsDataset(
File "/private/ViECap-main/CaptionsDataset.py", line 31, in init
tokenizer = AutoTokenizer.from_pretrained(language_model)
File "/root/anaconda3/envs/Viecap/lib/python3.8/site-packages/transformers/models/auto/tokenization_auto.py", line 498, in from_pretrained
tokenizer_config = get_tokenizer_config(pretrained_model_name_or_path, **kwargs)
File "/root/anaconda3/envs/Viecap/lib/python3.8/site-packages/transformers/models/auto/tokenization_auto.py", line 359, in get_tokenizer_config
resolved_config_file = get_file_from_repo(
File "/root/anaconda3/envs/Viecap/lib/python3.8/site-packages/transformers/utils/hub.py", line 678, in get_file_from_repo
resolved_file = cached_path(
File "/root/anaconda3/envs/Viecap/lib/python3.8/site-packages/transformers/utils/hub.py", line 282, in cached_path
output_path = get_from_cache(
File "/root/anaconda3/envs/Viecap/lib/python3.8/site-packages/transformers/utils/hub.py", line 545, in get_from_cache
raise ValueError(
ValueError: Connection error, and we cannot find the requested files in the cached path. Please try again or make sure your Internet connection is on.