vsubhashini/caption-eval

Sentence/Caption evaluation using automated metrics

Caption Evaluator

Sentence/Caption evaluation using automated metrics.

This code is released as supplementary material with S2VT[1].

This code can be used to

1. evaluate sentences/captions for any dataset,
2. it provides BLEU, METEOR, ROUGE-L and CIDEr scores.

This uses the MSCOCO caption evaluation code [2].

Getting started

1. Get this code. git clone https://github.com/vsubhashini/caption-eval.git
2. Get the coco evaluation scripts. ./get_coco_scripts.sh

To ensure you have all the dependencies for the evaluation scripts, please refer to the COCO Caption Evaluation page.

Evaluating predicted sentences against groundtruth references

Make sure you have the coco scripts

    ./get_coco_scripts.sh

Create your groundtruth references in the desired format

Here's a sample file with several reference sentences: data/references.txt

    python create_json_references.py -i data/references.txt -o data/references.json

Evaluate the model predictions against the references

Sample file with predictions from a model is in data/predicted_sentences.txt

    python run_evaluations.py -i data/predicted_sentences.txt -r data/references.json

References

• Sequence to Sequence - Video to Text
• Microsoft COCO Captions: Data Collection and Evaluation Server
• PTBTokenizer: Stanford Tokenizer which is included in Stanford CoreNLP 3.4.1.
• BLEU: BLEU: a Method for Automatic Evaluation of Machine Translation
• Meteor: Project page with related publications. COCO server uses version (1.5) of the Code.
• Rouge-L: ROUGE: A Package for Automatic Evaluation of Summaries
• CIDEr: [CIDEr: Consensus-based Image Description Evaluation] (http://arxiv.org/pdf/1411.5726.pdf)

[1] Sequence to Sequence - Video to Text

Sequence to Sequence - Video to Text
S. Venugopalan, M. Rohrbach, J. Donahue, T. Darrell, R. Mooney, K. Saenko
The IEEE International Conference on Computer Vision (ICCV) 2015

[2] Microsoft COCO Captions: Data Collection and Evaluation Server

Microsoft COCO Captions: Data Collection and Evaluation Server
X. Chen, H. Fang, T.Y. Lin, R. Vedantam, S. Gupta, P. Dollar, C.L. Zitnick
arXiv preprint arXiv:1504.00325

caption-eval's Issues

beam_size_1

What beam_size_1 means and why it should be there in the .txt file?

python s2vt_captioner.py -m s2vt_vgg_rgb error

When I run the s2vt_captioner.py I get the following error:

mkldnn_concat_layer.cpp:139] Check failed: dim_src == bottom[i]->shape().size() (4 vs. 3)

I am running on intel caffe version 1.1.0 in cpu only mode.

How can I resolve the same?

Unable to compile s2vt

I am using cuda-8 and tried cudnn 5.1 and 6 but was unable to compile
https://github.com/vsubhashini/caffe/tree/recurrent/examples/s2vt.
(Cannot create an issue there so raising it here)

Also, can you elaborate on the caffe model shared on your page is trained on all only MVSD dataset?.

Lastly, is there any specific steps to follow to test on new videos other than extracting VGG features.

Thanks for your work.

Excuse me,where is the code to train the model?

in https://vsubhashini.github.io/noc.html,The code to prepare data and train the model can be found in: is empty

Just curious, why do you put this link on NOC blogpost?