murari023/neuraltalk2-tensorflow

Inspired by Andrey Karpathy's Neuraltalk 2 VGG16-LSTM image captioning model

Neuraltalk2 in tensorflow

Overview

Tensorflow Implementation of image captioning model, inspired by Andrej Karpathy's neuraltalk2. Also added option to use prepared attribute vectors during training.

Usage

Training:

You will need to download image net weights for VGG16 first:https://yadi.sk/d/V6Rfzfei3TdKCH

Specify your mscoco directory in utils/parameters.py and launch:

python main.py --gpu 'your gpu'

Note: train/validation split can be changed simply by setting gen_val_captions parameter. Default is set to 4000 so we will have ~120000 in training set.

Note2: You will need to launch preprocess.py script first to obtain images hdf5 file. It is done for speed up image loading during fine-tuning the model.

Parameters

Parameters can be set directly in in utils/parameters.py file. (or specify through command line parameters). For example:

python main.py --gpu 0 --embed_dim 256 --dec_hid 512 --epochs 50 --temperature 0.6 --gen_name test --dec_lstm_drop 0.7 --lr 0.001 --checkpoint test1 --coco_dir "/home/username/mscoco/coco/" --optimizer Adam --sample_gen greedy

Generation

Two options:

1. Using main.py

After some training just launch:

python main.py --gpu 'your gpu' --mode inference

If you used fine-tuning will need just to add --fine_tune to the parameters:

python main.py --gpu 'your gpu' --mode inference --fine_tune

It will produce json file ready to use with mscoco evaluation tool

2. Using separate gen_caption.py script. Can be used to generate captions for any images.

For list of required parameters:

python gen_caption.py -h

For example:

python -i gen_caption.py --img_path ./images/COCO_val2014_000000233527.jpg --checkpoint ./checkpoints/gaussian_nocv.ckpt --params_path ./pickles/params_Normal_False_gaussian_nocv_False

Where:

• --params_path: saved Parameters class, can be saved by calling main.py --save_params
• --checkpoint: saved checkpoint
• --img_path: path for image
• -i: for launching python in interactive mode so captions can be generated by calling generator.generate_caption(img_path). This can be also used in ipython notebook

Specific requirements

• tensorflow >= 1.4.1

Other files

• prepare_cluster_vectors_train_val.ipynb - takes MSCOCO dataset json files and generates cluster vectors
• prepare_test_vectors.ipynb - gets test set cluster vector file, prepared using tf.models API and generates cluster vector
• gen_caption_example.ipynb - generate caption for some photo (without cluster vectors inputs)