ajamjoom/image-captions

BERT + Image Captioning

image-captions's Issues

Nothing inside checkpoint folder

Can you kindly provide the contents inside of the checkpoint folder. There is nothing mentioned about the files inside the checkpoint like encoder _baseline or encoder _ bert .
Please give me some solution . I am unable to run main.py
Because it says no file found in checkpoint named encoder _ baseline

Issue with BERT implementation

Hi,

It seems that you're trying to decode auto-regressively using BERT representations as a drop-in replacement for word embeddings. But BERT is bi-directional; the representation at token i has information about all tokens j > i. So, your model already knows what it needs to predict, before it predicts it.

In order for this to be correct you need to mask attention to all tokens j > i, which I don't think you do currently.

Get word embedding

I found when getting word embedding, the embedding matrix's size is changed to (batch_size, max_length+1, embedding_dim). The position of [CLS] is calculated to the embedding matrix. Can I change stack of token embedding to cap_embedding = torch.stack(tokens_embedding[1:])?

Test on Our Images

Thanks for great Repo.
How are we going to use from Bert in test code for testing on our images

where is glove_embeds.py & bug report

Thanks for sharing your project.
Would you tell me where can I find glove_embeds.py.
and bug report
main.py Line 316

• decoder = Decoder(vocab_size=len(vocab),use_glove=use_glove, use_bert=use_bert).to(device)

• decoder = Decoder(vocab_size=len(vocab),use_glove=glove_model, use_bert=bert_model).to(device)

when from_checkpoint set to False, [use_glove=use_glove, use_bert=use_bert] will raises error.

Checkpoint folder is empty!

@ajamjoom can you please help us run the code for our thesis work?

cap = u'[CLS] '+cap ???

       embeddings = []
        for cap_idx in  encoded_captions:
            
            # padd caption to correct size
            while len(cap_idx) < max_dec_len:
                cap_idx.append(PAD)
                
            cap = ' '.join([vocab.idx2word[word_idx.item()] for word_idx in cap_idx])
            cap = u'[CLS] '+cap

Hello, I want to know why here is the code "cap = u'[CLS] '+cap ", is not "cap = u'[CLS]'+cap+u'[SEP]'

May I know the performance of your project?

What is the CiDEr and BLEU-4 score reported? Thanks