amaiasalvador/imcap_keras

Image captioning with spatial attention using keras with tensorflow backend

imcap_keras's Issues

IndexError: tuple index out of range

I am experiencing this error when trying to train the model. Do you have any idea what's the issue and how I could fix it ?

Undeclared variables?

Here: https://github.com/amaiasalvador/imcap_keras/blob/master/imcap/test.py#L52

lang_model is not declared and also the language_model module is never used in the file.

PS. I'm working on a similar project and I'm trying to understand how to move from training to test because during training it's necessary to feed the lstm with the word sequence, whereas during inference that's not available. What's your approach?

Pre Trained Model

Could you please upload the weights for a pre-trained model, it would be a huge help thanks

Incorrect indexing in dataloader

It looks like the generator function has an issue with getting images and captions from the data file:
Line 110:
batch_ims = ims[batch_idxs,:,:,:]
Lines 122-123:
label_start_ixs = starts[batch_idxs]
label_end_ixs = ends[batch_idxs]
The indexing in data file differs from indexing in splits, so I think the lines should be revised to get images and captions from the appropriate splits. For instance, it could be done by using the following lines:
Line 110:
batch_ims = ims[[u+sample_list[0] for u in batch_idxs],:,:,:]
Lines 122-123:
label_start_ixs = starts[[u+sample_list[0] for u in batch_idxs]]
label_end_ixs = ends[[u+sample_list[0] for u in batch_idxs]]

Why didn't you use the LSTM provided by Keras?

Hi. I've seen that you have manually created the LSTM? Why didn't you use the LSTM provided by Keras, and used it's memory cell output to construct the sentinel?