peteanderson80/coco-caption

Adds SPICE metric to coco-caption evaluation server codes

Microsoft COCO Caption Evaluation

Evaluation codes for MS COCO caption generation.

No longer maintained. The SPICE metric has been incorporated into the official COCO caption evaluation code, so this repo is no longer maintained.

Requirements

• java 1.8.0
• python 2.7

Files

./

• cocoEvalCapDemo.py (demo script)

./annotation

• captions_val2014.json (MS COCO 2014 caption validation set)
• Visit MS COCO download page for more details.

./results

• captions_val2014_fakecap_results.json (an example of fake results for running demo)
• Visit MS COCO format page for more details.

./pycocoevalcap: The folder where all evaluation codes are stored.

• evals.py: The file includes COCOEavlCap class that can be used to evaluate results on COCO.
• tokenizer: Python wrapper of Stanford CoreNLP PTBTokenizer
• bleu: Bleu evalutation codes
• meteor: Meteor evaluation codes
• rouge: Rouge-L evaluation codes
• cider: CIDEr evaluation codes
• spice: SPICE evaluation codes

Setup

• You will first need to download the Stanford CoreNLP 3.6.0 code and models for use by SPICE. To do this, run: ./get_stanford_models.sh

References

• Microsoft COCO Captions: Data Collection and Evaluation Server
• PTBTokenizer: We use the Stanford Tokenizer which is included in Stanford CoreNLP 3.4.1.
• BLEU: BLEU: a Method for Automatic Evaluation of Machine Translation
• Meteor: Project page with related publications. We use the latest version (1.5) of the Code. Changes have been made to the source code to properly aggreate the statistics for the entire corpus.
• Rouge-L: ROUGE: A Package for Automatic Evaluation of Summaries
• CIDEr: [CIDEr: Consensus-based Image Description Evaluation] (http://arxiv.org/pdf/1411.5726.pdf)
• SPICE: [SPICE: Semantic Propositional Image Caption Evaluation] (http://panderson.me/images/SPICE.pdf)

Developers

• Xinlei Chen (CMU)
• Hao Fang (University of Washington)
• Tsung-Yi Lin (Cornell)
• Ramakrishna Vedantam (Virgina Tech)

Acknowledgement

• David Chiang (University of Norte Dame)
• Michael Denkowski (CMU)
• Alexander Rush (Harvard University)

coco-caption's Issues

How to contribute

Hi,
Thank you very much for the evaluation script. For my applications, I needed a Python 3.5 version of this script so I converted this code to be compatible with Python 3.5 and was wondering whether I can create a pull request to include it either in this repo or somewhere else.

Thanks again

How to obtain the spice scores in other languages

The code seems only support spice evaluation in English, how can i obtain the spice scores in languages such as Chinese, Japanese, and Spanish.

subprocess.CalledProcessError: when calculating SPICE score

Hi, whenever i run spice.Compute_Score() locally i get this error:

CalledProcessError: Command '['java', '-jar', '-Xmx8G', 'spice-1.0.jar', '/Users/peter/opt/anaconda3/lib/python3.8/site-packages/pycocoevalcap/spice/tmp/tmpbq7g34li', '-cache', '/Users/peter/opt/anaconda3/lib/python3.8/site-packages/pycocoevalcap/spice/cache', '-out', '/Users/peter/opt/anaconda3/lib/python3.8/site-packages/pycocoevalcap/spice/tmp/tmpx6ykjgr7', '-subset', '-silent']' returned non-zero exit status 1.

However it seems to work on google colab? Could you kindly help me out?

SPICE implementation on Google Colab

Hi!
I am having issues implementing SPICE.Please help

Error: Could not score batched file input:
java.lang.ClassCastException: org.json.simple.JSONArray cannot be cast to java.lang.String
at edu.anu.spice.SpiceScorer.scoreBatch(SpiceScorer.java:95)
at edu.anu.spice.SpiceScorer.main(SpiceScorer.java:60)
Traceback (most recent call last):
File "/content/pycocoevalcap/spice/model_evaluation.py", line 238, in
Val_with_MLE()
File "/content/pycocoevalcap/spice/model_evaluation.py", line 236, in Val_with_MLE
spice_class.compute_score(all_usable_ref,all_gen_sents)
File "/content/pycocoevalcap/spice/test.py", line 78, in compute_score
cwd=os.path.dirname(os.path.abspath(file)))
File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['java', '-jar', '-Xmx8G', 'spice-1.0.jar', '/content/pycocoevalcap/spice/tmp/tmpr49ny2ll', '-cache', '/content/pycocoevalcap/spice/cache', '-out', '/content/pycocoevalcap/spice/tmp/tmpl8edcdti', '-subset', '-silent']' returned non-zero exit status 1.

Problem with SPICE

Hi,

I was trying to report the SPICE for our paper; however, we faced multiple problems to make it work. I exactly followed the provided instructions to incorporate the spice into our evaluation procedure.
Anyway, I got the following errors:

Error: Could not score batched file input:
org.fusesource.lmdbjni.LMDBException: MDB_CURSOR_FULL: Internal error - cursor stack limit reached
at org.fusesource.lmdbjni.Util.checkErrorCode(Util.java:44)
at org.fusesource.lmdbjni.Database.get(Database.java:202)
at org.fusesource.lmdbjni.Database.get(Database.java:193)
at org.fusesource.lmdbjni.Database.get(Database.java:186)
at org.fusesource.lmdbjni.Database.get(Database.java:161)
at edu.anu.spice.LmdbTupleDB.getTransaction(LmdbTupleDB.java:96)
at edu.anu.spice.SpiceParser.loadTuplesFromDB(SpiceParser.java:195)
at edu.anu.spice.SpiceParser.loadTuples(SpiceParser.java:245)
at edu.anu.spice.SpiceParser.parseCaptions(SpiceParser.java:251)
at edu.anu.spice.SpiceScorer.scoreBatch(SpiceScorer.java:109)
at edu.anu.spice.SpiceScorer.main(SpiceScorer.java:60)

I was wondering have you ever faced with the problem like above? [This happens once a while] It is worth noting that our system works with MSCOCO evolution server with no problems. I'd be appreciated if you please let me know your comment.

Rasool

Can I merge a fork of this project that has Word Mover's Distance included and is Python 3 compatible?

I have an offline version of this project's code that has Word Mover's Distance included and is Python 3 compatible. If I put it up on GitHub can it be merged here?

Also, is this the new official repository for MSCOCO or is it an unofficial fork?