ajamjoom/image-captions

BERT + Image Captioning

image-captions's Issues

Checkpoint folder is empty!

@ajamjoom can you please help us run the code for our thesis work?

Nothing inside checkpoint folder

Can you kindly provide the contents inside of the checkpoint folder. There is nothing mentioned about the files inside the checkpoint like encoder _baseline or encoder _ bert .
Please give me some solution . I am unable to run main.py
Because it says no file found in checkpoint named encoder _ baseline

cap = u'[CLS] '+cap ???

       embeddings = []
        for cap_idx in  encoded_captions:
            
            # padd caption to correct size
            while len(cap_idx) < max_dec_len:
                cap_idx.append(PAD)
                
            cap = ' '.join([vocab.idx2word[word_idx.item()] for word_idx in cap_idx])
            cap = u'[CLS] '+cap

Hello, I want to know why here is the code "cap = u'[CLS] '+cap ", is not "cap = u'[CLS]'+cap+u'[SEP]'

Test on Our Images

Thanks for great Repo.
How are we going to use from Bert in test code for testing on our images

Get word embedding

I found when getting word embedding, the embedding matrix's size is changed to (batch_size, max_length+1, embedding_dim). The position of [CLS] is calculated to the embedding matrix. Can I change stack of token embedding to cap_embedding = torch.stack(tokens_embedding[1:])?

May I know the performance of your project?

What is the CiDEr and BLEU-4 score reported? Thanks

where is glove_embeds.py & bug report

Thanks for sharing your project.
Would you tell me where can I find glove_embeds.py.
and bug report
main.py Line 316

• decoder = Decoder(vocab_size=len(vocab),use_glove=use_glove, use_bert=use_bert).to(device)

• decoder = Decoder(vocab_size=len(vocab),use_glove=glove_model, use_bert=bert_model).to(device)

when from_checkpoint set to False, [use_glove=use_glove, use_bert=use_bert] will raises error.

Issue with BERT implementation

Hi,

It seems that you're trying to decode auto-regressively using BERT representations as a drop-in replacement for word embeddings. But BERT is bi-directional; the representation at token i has information about all tokens j > i. So, your model already knows what it needs to predict, before it predicts it.

In order for this to be correct you need to mask attention to all tokens j > i, which I don't think you do currently.