rubickh/image-captioning-with-mad-and-sap

Code for paper "Image Captioning with End-to-End Attribute Detection and Subsequent Attributes Prediction". IEEE Transactions on Image Processing 2020

image-captioning-with-mad-and-sap's Issues

self critical sequence training

Hi @RubickH ,
I installed all the prereqs and also able to run eval.py (without any issues) using your provided checkpoint. But when I run train with the below command, it stops in the self_critical step after epoch 49 I think. Could you please help me to fix this issue.

------------Command-----------------

python train.py --self_critical_after 50 --id MADSAP0509 --caption_model lstm_MAD_SAP --save_checkpoint_every 500 --batch_size 600 --num_gpu 3 --gpu_id 0,1,2 --beam 0

------------ERROR Output-----------------------

iter 9440 (epoch 49), SAP_loss = 2.289, word_loss = 2.080, MAD_loss = 0.261 time/batch = 2.197
current_lr is 6.7108864e-05
Read and process data: 1.49842405319
iter 9441 (epoch 49), SAP_loss = 2.213, word_loss = 2.030, MAD_loss = 0.252 time/batch = 2.823
initializing CIDEr scorer...
initlizing CIDEr scorers in 0.000029s
current_lr is 5.36870912e-05
Read and process data: 0.961675882339
GPU time is : 1.09868407249s
Traceback (most recent call last):
File "train.py", line 399, in
train(opt)
File "train.py", line 244, in train
reward = get_self_critical_reward(gen_result,greedy_res,data['gts'])
File "train.py", line 72, in get_self_critical_reward
, cider_scores = CiderD_scorer.compute_score(gts, res)
File "cider-master/pyciderevalcap/ciderD/ciderD.py", line 48, in compute_score
(score, scores) = cider_scorer.compute_score(self._df)
File "cider-master/pyciderevalcap/ciderD/ciderD_scorer.py", line 199, in compute_score
score = self.compute_cider(df_mode)
File "cider-master/pyciderevalcap/ciderD/ciderD_scorer.py", line 173, in compute_cider
vec, norm, length = counts2vec(test)
File "cider-master/pyciderevalcap/ciderD/ciderD_scorer.py", line 122, in counts2vec
df = np.log(max(1.0, self.document_frequency[ngram]))
KeyError: ('7', '9459')
Terminating BlobFetcher

Thank you,
Srikanth

How can I provide a custom input image to the pretrained model and get output captions for the same?

Training Time

How long it takes to train this model with using 1 GPU machine?

Getting Import error for misc.utils

Hey, I tried to execute your code completed the data preprocessing part but I'm getting error while training as follows

Traceback (most recent call last):
File "train.py", line 16, in
import models
File "/home/tusharghorpade/Image Captioning/models/init.py", line 13, in
from .AttModel_MAD_SAP import *
File "/home/tusharghorpade/Image Captioning/models/AttModel_MAD_SAP.py", line 14, in
from .CaptionModel import CaptionModel
File "/home/tusharghorpade/Image Captioning/models/CaptionModel.py", line 9, in
import misc.utils as utils
ImportError: No module named misc.utils

I guess the CaptionModel.py is throwing me this error. Can you please help me out with this?

How to use this repository on my own dataset?

I have my own data set which I'd like to train on end-to-end image captioning. Looking at the data/README file, I see that bottom-up features are required. Is it possible to train without having these features?

Model Ensemble

Hi Yiqing Huang, Could you please upload the model ensemble of this project?. Thanks a lot in advance.
I already sent you a message on your email , my email: [email protected].