Rewritten version of tensorflow/models/im2txt
A TensorFlow implementation* of the image-to-text model described in the paper:
"Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge."
Oriol Vinyals, Alexander Toshev, Samy Bengio, Dumitru Erhan.
IEEE transactions on pattern analysis and machine intelligence (2016).
Full text available at: http://arxiv.org/abs/1609.06647
*Some key elements was changed:
Advantages
- CNN removed from training loop (train is ~150 times faster)
- Less memory needed for same batch
- Model written with
tf.estimator.Estimator
andtf.dataset.Dataset
Disadvantages
- Model spited in 2 parts
- CNN removed from training loop (paper's finetun not possible, CNN is not trainable)
python3 preprocess.py
150 epoch training will take 8h with 1024 bath on GTX1080
python3 im2txt.py train --optimizer Adagrad --initial_learning_rate 4 --num_epochs_per_decay 20 --batch_size 1024 --max_train_epochs 150 --save_checkpoints_steps 1000 --log_step_count_steps 500
For other options call
python3 im2txt.py --help
python3 im2txt.py eval
For other options call
python3 im2txt.py --help
Test last model (just past output in Markdown editor)
python3 im2txt.py test --test_urls https://cdn132.picsart.com/245236991011202.jpg,https://cdn141.picsart.com/245236965014202.jpg
force run on CPU
CUDA_VISIBLE_DEVICES=""; python3 im2txt.py test ...
python3 im2txt.py
<S> a woman in a dress standing on a street corner . </S> (-11.629340171813965)
<S> a man sitting on a bench in a room . </S> (-8.756865501403809)