jonghwanmun/textguidedatt

The implementation of Text-guided Attention Model for Image Captioning

textguidedatt's Issues

Issue with model checkpoint loading in eval_res_att_knn_test5000.lua

Hi Jonghwan

Thank you for your prompt responses. I have the test features and want to evaluate them. But I face the following issue while running eval_res_att_knn_test5000.lua for loading the model checkpoint. Please see the errors below.

ln: failed to create symbolic link './misc': File exists
ln: failed to create symbolic link './data': File exists
ln: failed to create symbolic link './model': File exists
ln: failed to create symbolic link './layers': File exists
ln: failed to create symbolic link './coco-caption': File exists
Load img info file : data/coco/cocotalk_trainval_img_info.json
Load kNN caps info file : data/coco/10NN_cap_valtrainall_cider.json
Load caption label info file : data/coco/cocotalk_cap_label.h5
Load img feat file : data/resnet101_conv_feat_448/
Load cap feat file : data/skipthought/cocotalk_trainval_skipthought.h5
sIdx (5001) | eIdx (10000)
initializing weights from model/textGuideAtt/res_textGuideAtt.t7
model/textGuideAtt/res_textGuideAtt.t7
/home/ubuntu/src/torch/install/bin/luajit: cannot open <model/textGuideAtt/res_textGuideAtt.t7> in mode r at /home/ubuntu/src/torch/pkg/torch/lib/TH/THDiskFile.c:673
stack traceback:
[C]: at 0x7f453fe5f460
[C]: in function 'DiskFile'
/home/ubuntu/src/torch/install/share/lua/5.1/torch/File.lua:405: in function 'load'
eval_res_att_knn_test5000.lua:105: in main chunk
[C]: in function 'dofile'
.../src/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:150: in main chunk
[C]: at 0x00405d50

Error while running run_inference using pretrained model

I get the following error when i run run_inference bash file : 'resNet/prediction_result/res_predictions_10NN_test5000.json'

Can you tell me how to reslove this ?

skipthought/coco_cap_uni_gru_word2vec.t7

Hi, @JonghwanMun , thanks for your excellent code. I have a little question to ask. Specifically,
Did you obtain the skipthought/coco_cap_uni_gru_word2vec.t7 file by training the skipthought model? Can I just map my subdictionary(COCO) to their corresponding vectors according to the initial vocabulary (930,913 words, used to train the skipthought model by the original paper [https://arxiv.org/abs/1506.06726])

Resnet convnet feature extraction for each image taking too much time

Hi Jonghwan

I am running the extract_resnet_feat.lua. It's taking too much time and I am on a time crunch. I was just wondering if you had these feature extracted files somewhere which you can share. It will be very helpful.

I am on this step and even with GPU configured it's taking about 300 ms for each image.
5614/123286: data/resnet101_conv_feat_448/COCO_val2014_000000276434.t7 (left 569.51s)

Missing code

Too much time for loading image feature

Hi, I have noticed that too much time has been spent on feature loading, because of each t7 file record feature map of an image. I recommend using hdf5 file to record the feature in feature extraction

VGG-FCN t7 file

You had released the Res-101.t7 file to extract image features, can you release the VGG-FCN t7 file ? Thanks.

the train code

Different data process from fb.resnet.torch

Hi, Sorry to bother you, I have a little question about the code that use a different data process method TextGuidedAtt from the standard approach fb.resnet.torch. The input image size of the code is 448x448 while the input image size of fb.resnet.torch is 224x224, which means that the feature map is four times as big as that of fb.resnet.torch . So the time for loading the feature map is very long. Could you explain why should use such a large input image size and is it possible to decrease the size of the input image without affecting the performance. Thank you very much.

Syntax errors in the few files

TextguidedATT/000_data_construction/002_skipthought/extract_cap_feat.lua :
line 11 : change require 'layers.textGuideAtt' to require 'layers.textGuidedAtt'

TextguidedATT/layers/LanguageModel.lua :
line 210 : local beam_size = utils.getopt(opt, 'beam_size', 2 --- ')' is missing