kranthigv/pretrained-show-and-tell-model

This repository contains pretrained Show and Tell: A Neural Image Caption Generator implemented in Tensorflow.

(tl;dr)
2M iterations finetuned checkpoint file | Released under MIT License

1M iterations checkpoint file | Released under MIT License

word_counts.txt (at this repository)

model.ckpt-2000000.index (at this repository. Place it in the same folder as the model checkpoint used.)

model.ckpt-1000000.index (at this repository. Place it in the same folder as the model checkpoint used.)

Show and Tell : A Neural Image Caption Generator

Pretrained model for Tensorflow implementation found at tensorflow/models of the image-to-text paper described at:

"Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge."

Oriol Vinyals, Alexander Toshev, Samy Bengio, Dumitru Erhan.

Full text available at: http://arxiv.org/abs/1609.06647

Contact

Kranthi Kiran GV (KranthiGV | [email protected])

Generating Captions

Steps

0. Follow the steps at im2txt to clone the repository, install bazel, etc.

1. Download the desired model checkpoint:
   2M iterations finetuned checkpoint file | Released under MIT License
   1M iterations checkpoint file | Released under MIT License

2. Clone the repository: git clone https://github.com/KranthiGV/Pretrained-Show-and-Tell-model.git

# Path to checkpoint file.
# Notice there's no data-00000-of-00001 in the CHECKPOINT_PATH environment variable
# Also make sure you place model.ckpt-2000000.index (which is cloned from the repository)
# in the same location as model.ckpt-2000000.data-00000-of-00001
# You can use model.ckpt-1000000.data-00000-of-00001 similarly
CHECKPOINT_PATH="/path/to/model.ckpt-2000000"


# Vocabulary file generated by the preprocessing script.
# Since the tokenizer could be of a different version, use the word_counts.txt file supplied. 
VOCAB_FILE="/path/to/word_counts.txt"

# JPEG image file to caption.
IMAGE_FILE="/path/to/image.jpeg"

# Build the inference binary.
bazel build -c opt im2txt/run_inference

# Run inference to generate captions.
bazel-bin/im2txt/run_inference \
  --checkpoint_path=${CHECKPOINT_PATH} \
  --vocab_file=${VOCAB_FILE} \
  --input_files=${IMAGE_FILE}

Extras

1. Graph.pbtxt is uploaded on request.
2. Training stats are uploaded for use with tensorboard.
   tensorboard --logdir="./extras/tensorboard/"

pretrained-show-and-tell-model's Issues

NotFoundError (see above for traceback): Tensor name "InceptionV3/Mixed_7b/Branch_2/Conv2d_0c_1x3/weights"

Before ask question , Thanks for your sharing.

1. I download your model.ckpt-2000000.index

2. Rename model.ckpt-2000000.index -> model.ckpt-2000000

3. bazel-bin/im2txt/run_inference
   --checkpoint_path=/my/path/model.ckpt-2000000
   --vocab_file=${VOCAB_FILE}
   --input_files=${IMAGE_FILE}

4. I got below error

Traceback (most recent call last):
File "/home/nvidia/models/im2txt/bazel-bin/im2txt/run_inference.runfiles/im2txt/im2txt/run_inference.py", line 85, in
tf.app.run()
File "/home/nvidia/.local/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 44, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "/home/nvidia/models/im2txt/bazel-bin/im2txt/run_inference.runfiles/im2txt/im2txt/run_inference.py", line 65, in main
restore_fn(sess)
File "/home/nvidia/models/im2txt/bazel-bin/im2txt/run_inference.runfiles/im2txt/im2txt/inference_utils/inference_wrapper_base.py", line 96, in _restore_fn
saver.restore(sess, checkpoint_path)
File "/home/nvidia/.local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1428, in restore
{self.saver_def.filename_tensor_name: save_path})
File "/home/nvidia/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 767, in run
run_metadata_ptr)
File "/home/nvidia/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 965, in _run
feed_dict_string, options, run_metadata)
File "/home/nvidia/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1015, in _do_run
target_list, options, run_metadata)
File "/home/nvidia/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1035, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.NotFoundError: Tensor name "InceptionV3/Mixed_7b/Branch_2/Conv2d_0c_1x3/weights" not found in checkpoint files /home/nvidia/models/im2txt/im2txt/model/train/model.ckpt-2000000
[[Node: save/RestoreV2_331 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_recv_save/Const_0, save/RestoreV2_331/tensor_names, save/RestoreV2_331/shape_and_slices)]]

Caused by op u'save/RestoreV2_331', defined at:
File "/home/nvidia/models/im2txt/bazel-bin/im2txt/run_inference.runfiles/im2txt/im2txt/run_inference.py", line 85, in
tf.app.run()
File "/home/nvidia/.local/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 44, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "/home/nvidia/models/im2txt/bazel-bin/im2txt/run_inference.runfiles/im2txt/im2txt/run_inference.py", line 51, in main
FLAGS.checkpoint_path)
File "/home/nvidia/models/im2txt/bazel-bin/im2txt/run_inference.runfiles/im2txt/im2txt/inference_utils/inference_wrapper_base.py", line 116, in build_graph_from_config
saver = tf.train.Saver()
File "/home/nvidia/.local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1040, in init
self.build()
File "/home/nvidia/.local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1070, in build
restore_sequentially=self._restore_sequentially)
File "/home/nvidia/.local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 675, in build
restore_sequentially, reshape)
File "/home/nvidia/.local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 402, in _AddRestoreOps
tensors = self.restore_op(filename_tensor, saveable, preferred_shard)
File "/home/nvidia/.local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 242, in restore_op
[spec.tensor.dtype])[0])
File "/home/nvidia/.local/lib/python2.7/site-packages/tensorflow/python/ops/gen_io_ops.py", line 668, in restore_v2
dtypes=dtypes, name=name)
File "/home/nvidia/.local/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 763, in apply_op
op_def=op_def)
File "/home/nvidia/.local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2327, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/home/nvidia/.local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1226, in init
self._traceback = _extract_stack()

NotFoundError (see above for traceback): Tensor name "InceptionV3/Mixed_7b/Branch_2/Conv2d_0c_1x3/weights" not found in checkpoint files /home/nvidia/models/im2txt/im2txt/model/train/model.ckpt-2000000
[[Node: save/RestoreV2_331 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_recv_save/Const_0, save/RestoreV2_331/tensor_names, save/RestoreV2_331/shape_and_slices)]]

Plz tell me how to solve this error. I want just test Captioning ( not training ) . So i down your pre trained...

TF versin : '1.0.1'
Ubuntu : 16.04 LTS
Python2 : 2.7.12
Architecture : NVIDIA Jetson TX2 ( aarch 64 )

NotFoundError

Hey, First of all thank you for providing checkpoint file for im2text.

while I run the final code :- bazel-bin/im2txt/run_inference --checkpoint_path=${CHECKPOINT_PATH} --vocab_file="word_counts.txt" --input_files=${IMAGE_FILE}

I am getting this error

NotFoundError (see above for traceback): Unsuccessful TensorSliceReader constructor: Failed to find any matching files for model.ckpt-2000000
[[Node: save/RestoreV2_225 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_recv_save/Const_0, save/RestoreV2_225/tensor_names, save/RestoreV2_225/shape_and_slices)]]

H5 file

Is it possible for you to share the h5 file so that we can try it in Python?

Bias not found in checkpoint

I’m having some problems running the Pretrained-Show-and-Tell-model
I am using tensorflow v1.2 and python 2.7 (in addition to checking out the latest versions of your repo and the im2txt model repo).
I am getting the following error:

NotFoundError (see above for traceback): Key lstm/basic_lstm_cell/bias not found in checkpoint
	 [[Node: save/RestoreV2_380 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_380/tensor_names, save/RestoreV2_380/shape_and_slices)]]

My Full Trace is Below:

(venv)  abhi@Abhinavs-MacBook-Pro  ~/Desktop/Various_dev_projects/testing/models/im2txt   master  bazel-bin/im2txt/run_inference \
  --checkpoint_path=${CHECKPOINT_PATH} \
  --vocab_file=${VOCAB_FILE} \
  --input_files=${IMAGE_FILE}


INFO:tensorflow:Building model.
INFO:tensorflow:Initializing vocabulary from file: ptrain/word_counts.txt
INFO:tensorflow:Created vocabulary with 11520 words
INFO:tensorflow:Running caption generation on 1 files matching images/1.jpg
2017-06-19 22:24:00.000062: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2017-06-19 22:24:00.000085: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2017-06-19 22:24:00.000090: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2017-06-19 22:24:00.000094: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
INFO:tensorflow:Loading model from checkpoint: ptrain/model.ckpt-2000000
INFO:tensorflow:Restoring parameters from ptrain/model.ckpt-2000000
2017-06-19 22:24:02.932954: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key lstm/basic_lstm_cell/bias not found in checkpoint
2017-06-19 22:24:02.933737: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key lstm/basic_lstm_cell/kernel not found in checkpoint
Traceback (most recent call last):
  File "/Users/abhi/Desktop/Various_dev_projects/testing/models/im2txt/bazel-bin/im2txt/run_inference.runfiles/im2txt/im2txt/run_inference.py", line 85, in <module>
    tf.app.run()
  File "/Users/abhi/Desktop/Various_dev_projects/testing/models/im2txt/venv/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 48, in run
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File "/Users/abhi/Desktop/Various_dev_projects/testing/models/im2txt/bazel-bin/im2txt/run_inference.runfiles/im2txt/im2txt/run_inference.py", line 65, in main
    restore_fn(sess)
  File "/Users/abhi/Desktop/Various_dev_projects/testing/models/im2txt/bazel-bin/im2txt/run_inference.runfiles/im2txt/im2txt/inference_utils/inference_wrapper_base.py", line 96, in _restore_fn
    saver.restore(sess, checkpoint_path)
  File "/Users/abhi/Desktop/Various_dev_projects/testing/models/im2txt/venv/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1548, in restore
    {self.saver_def.filename_tensor_name: save_path})
  File "/Users/abhi/Desktop/Various_dev_projects/testing/models/im2txt/venv/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 789, in run
    run_metadata_ptr)
  File "/Users/abhi/Desktop/Various_dev_projects/testing/models/im2txt/venv/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 997, in _run
    feed_dict_string, options, run_metadata)
  File "/Users/abhi/Desktop/Various_dev_projects/testing/models/im2txt/venv/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1132, in _do_run
    target_list, options, run_metadata)
  File "/Users/abhi/Desktop/Various_dev_projects/testing/models/im2txt/venv/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1152, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.NotFoundError: Key lstm/basic_lstm_cell/bias not found in checkpoint
	 [[Node: save/RestoreV2_380 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_380/tensor_names, save/RestoreV2_380/shape_and_slices)]]

Caused by op u'save/RestoreV2_380', defined at:
  File "/Users/abhi/Desktop/Various_dev_projects/testing/models/im2txt/bazel-bin/im2txt/run_inference.runfiles/im2txt/im2txt/run_inference.py", line 85, in <module>
    tf.app.run()
  File "/Users/abhi/Desktop/Various_dev_projects/testing/models/im2txt/venv/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 48, in run
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File "/Users/abhi/Desktop/Various_dev_projects/testing/models/im2txt/bazel-bin/im2txt/run_inference.runfiles/im2txt/im2txt/run_inference.py", line 51, in main
    FLAGS.checkpoint_path)
  File "/Users/abhi/Desktop/Various_dev_projects/testing/models/im2txt/bazel-bin/im2txt/run_inference.runfiles/im2txt/im2txt/inference_utils/inference_wrapper_base.py", line 116, in build_graph_from_config
    saver = tf.train.Saver()
  File "/Users/abhi/Desktop/Various_dev_projects/testing/models/im2txt/venv/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1139, in __init__
    self.build()
  File "/Users/abhi/Desktop/Various_dev_projects/testing/models/im2txt/venv/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1170, in build
    restore_sequentially=self._restore_sequentially)
  File "/Users/abhi/Desktop/Various_dev_projects/testing/models/im2txt/venv/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 691, in build
    restore_sequentially, reshape)
  File "/Users/abhi/Desktop/Various_dev_projects/testing/models/im2txt/venv/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 407, in _AddRestoreOps
    tensors = self.restore_op(filename_tensor, saveable, preferred_shard)
  File "/Users/abhi/Desktop/Various_dev_projects/testing/models/im2txt/venv/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 247, in restore_op
    [spec.tensor.dtype])[0])
  File "/Users/abhi/Desktop/Various_dev_projects/testing/models/im2txt/venv/lib/python2.7/site-packages/tensorflow/python/ops/gen_io_ops.py", line 640, in restore_v2
    dtypes=dtypes, name=name)
  File "/Users/abhi/Desktop/Various_dev_projects/testing/models/im2txt/venv/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op
    op_def=op_def)
  File "/Users/abhi/Desktop/Various_dev_projects/testing/models/im2txt/venv/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2506, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/Users/abhi/Desktop/Various_dev_projects/testing/models/im2txt/venv/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1269, in __init__
    self._traceback = _extract_stack()

How to run run captioning on multiple images while not loading the checkpoints everytime ?

How can make this a part of my existing code where i pass the Image Name and the code runs without reloading the checkpoints everytime

Fine-tuning the inception v3 model

Hi,

First, thanks for all the available code for im2txt. I could make it work and generate some first captions. But now, I’m completely lost on a problem about fine-tuning im2txt, hoping it’s the good place to ask.

To summarize, I first fine-tuned an inception model using the method described in (https://github.com/tensorflow/models/tree/master/slim#Tuning). My training generates a bunch of files like model.meta, model.index and model.ckpt-xxx. I think the training is correct as I can evaluate it using eval_image_classifier.py available in slim git repo.

Then , what I would like to do is use this fine-tuned inception model as an input to train im2txt as in https://github.com/tensorflow/models/tree/master/im2txt#training-a-model.

1. But as training im2txt requires something like inception_v3.ckpt, what is the correct file to put as input?

I also tried fine-tuning the inception model in im2txt as in (https://github.com/tensorflow/models/tree/master/im2txt#fine-tune-the-inception-v3-model). But I don’t understand what it concretely does.

2. Does it fine-tune the CNN part of im2txt to adapt it to new classes? If so, how can I give it new data (e.g. the flowers dataset)?

Thanks in advance for your help.
Regards,
Stephane

Not able to test with pretrained checkpoints

Trying running with /model.ckpt-2000000.index getting below error

NotFoundError (see above for traceback): Tensor name "InceptionV3/Conv2d_1a_3x3/BatchNorm/beta" not found in checkpoint files /Users/shrgupta5/WA-projects/Image_captioning/Experiments/ImgtoTxt_show_tell/punny_captions/Pretrained-Show-and-Tell-model-master/model.ckpt-2000000.index
[[Node: save/RestoreV2 = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]

Model does not produce expected results

I have followed the directions in the repository to download and run the model. The code successfully executes for the following image (directly from the Tensorflow im2txt repository)

However, when I run the model against this image, I receive the intended results:

What could be the problem that I am running into?

Key lstm/basic_lstm_cell/kernel not found in checkpoint

The same issue already opened, sorry.

I had tried to run your model on the latest version of TF (1.2.0), but encountered the following error:
Key lstm/basic_lstm_cell/kernel not found in checkpoint
I've found @cshallue answer. The original version didn't work out for me. After some digging, I've changed values in vars_to_rename - did the trick.

OLD_CHECKPOINT_FILE = "model.ckpt-1000000"
NEW_CHECKPOINT_FILE = "model2.ckpt-1000000"

import tensorflow as tf
vars_to_rename = {
    "lstm/basic_lstm_cell/weights": "lstm/basic_lstm_cell/kernel",
    "lstm/basic_lstm_cell/biases": "lstm/basic_lstm_cell/bias",
}
new_checkpoint_vars = {}
reader = tf.train.NewCheckpointReader(OLD_CHECKPOINT_FILE)
for old_name in reader.get_variable_to_shape_map():
  if old_name in vars_to_rename:
    new_name = vars_to_rename[old_name]
  else:
    new_name = old_name
  new_checkpoint_vars[new_name] = tf.Variable(reader.get_tensor(old_name))

init = tf.global_variables_initializer()
saver = tf.train.Saver(new_checkpoint_vars)

with tf.Session() as sess:
  sess.run(init)
  saver.save(sess, NEW_CHECKPOINT_FILE)

One might find it useful.

How to freeze to frozen.pb

I'm trying to freeze 2m iterations checkpoint into a frozen.pb with the following command:

python freeze_graph.py --input_graph=C:/dev/pretrained-show-and-tell-model/extras/graph.pbtxt --input_checkpoint=C:/dev/pretrained-show-and-tell-model/model.ckpt-2000000 --output_graph=C:/dev/im2txt.pb --output_node_names=lstm/initial_state,softmax,lstm/state

freeze_graph.py script gives an error stating that lstm/initial_state is not in the graph, but inference_wrapper.py script fetches from this node. What output nodes should I use to to freeze this checkpoint?

unable to run

I'm getting following error

ankush@Lenovo:/media/Study/Projects/showandtell/models/im2txt$ bazel-bin/im2txt/run_inference --checkpoint_path=${CHECKPOINT_PATH} --vocab_file=${VOCAB_FILE} --input_files=${IMAGE_FILE}
INFO:tensorflow:Building model.
INFO:tensorflow:Initializing vocabulary from file:
Traceback (most recent call last):
File "/media/ankush/Study/Projects/showandtell/models/im2txt/bazel-bin/im2txt/run_inference.runfiles/im2txt/im2txt/run_inference.py", line 85, in
tf.app.run()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "/media/ankush/Study/Projects/showandtell/models/im2txt/bazel-bin/im2txt/run_inference.runfiles/im2txt/im2txt/run_inference.py", line 55, in main
vocab = vocabulary.Vocabulary(FLAGS.vocab_file)
File "/media/ankush/Study/Projects/showandtell/models/im2txt/bazel-bin/im2txt/run_inference.runfiles/im2txt/im2txt/inference_utils/vocabulary.py", line 50, in init
assert start_word in reverse_vocab
AssertionError

Text summarization

tensorflow.python.framework.errors_impl.NotFoundError: Key seq2seq/decoder/attention_decoder/lstm_cell/bias not found in checkpoint
[[Node: save/RestoreV2 = RestoreV2[dtypes=[DT_INT32, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]

tensorflow/models#466 (comment)

can you please tell the steps to fix this issue .Thanks in advance

Mention Tensorflow version in main Readme file

Hi,
I was trying to use your models with TF 1.3 and 1.2 with no success, if it is really limited to 1.0, please mention this in main Readme file to save time for people.

Thanks!

Unable to run bazel-bin coz of errors

Hi,
I am getting some errors while running bazel-bin command:
bazel-bin\im2txt\run_inference --checkpoint_path=%CHECKPOINT_PATH% --vocab_file=%VOCAB_FILE% --input_file=%IMAGE_FILE%

Error:

Traceback (most recent call last):
File "\?\C:\Users\VISHNU~1\AppData\Local\Temp\Bazel.runfiles_d8v5_2mn\runfiles\im2txt\im2txt\run_inference.py", line 27, in
from im2txt import configuration
ImportError: cannot import name 'configuration'

Could u please provide suggestion on how to solve this error?
Thanks

model performance

could you please share the performance of the models in comparison to the paper?

Include SaverDef proto

Similar to #2, it would be nice to be able to use build_graph_from_proto method from inference_utils/inference_wrapper_base.py. So can you please also upload your SaverDef proto file?

Poor performance

First, thanks for sharing your work. Results using these models though seems to be worse than Karpathy's work.

DataLossError

Hi, thank you providing the pre trained model. However, while running the last instruction, i.e
Run inference to generate captions.

bazel-bin/im2txt/run_inference
--checkpoint_path=${CHECKPOINT_PATH}
--vocab_file=${VOCAB_FILE}
--input_files=${IMAGE_FILE}

I am getting the following error:

DataLossError (see above for traceback): Unable to open table file /Users/rishimadhok/Desktop/Quote_my_picture/im2txt/im2txt/data/Pretrained-Show-and-Tell-model/model.ckpt-1000000.data-00000-of-00001: Data loss: not an sstable (bad magic number): perhaps your file is in a different file format and you need to use a different restore operator?
[[Node: save/RestoreV2_382 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_382/tensor_names, save/RestoreV2_382/shape_and_slices)]]

Please let me know what should I do to resolve this error? Thank you for your time!

Using the pretrained model for fine-tuning on new data set

Hello,

I am wondering if anyone has used the pretrained model checkpoints for further fine-tuning the im2txt model with new data?
I am currently working on that and there are still so many open questions.
For example, the word_counts file. Do the counts from MSCOCO just get merged with the new vocabulary from the new dataset?

I only have a small, specific image set and that's why I can't use it for the initial training. But is it even possible to use the pretrained model for new data?

If anyone is trying the same or was already successful, I would be really happy to exchange views on this issue.

vocab file not found

Hi, i've cloned tensorflow and your repos and when i try to run this, i get this error. Apparently the word_counts.txt file doesn't has problems, but don't know what's going on.

(tensorflow) MacBook-Pro-de-Carlos-2:im2txt Carlos$ sudo bazel-bin/im2txt/run_inference \ --checkpoint_path=${CHECKPOINT_PATH} \ --vocab_file=word_counts.txt \ --input_files=${IMAGE_FILE}
INFO:tensorflow:Building model.
CRITICAL:tensorflow:Vocab file  not found.
INFO:tensorflow:Initializing vocabulary from file: 
Traceback (most recent call last):
  File "/Users/Carlos/code/tensorflow/models/research/im2txt/bazel-bin/im2txt/run_inference.runfiles/im2txt/im2txt/run_inference.py", line 85, in <module>
    tf.app.run()
  File "/Users/Carlos/code/tensorflow/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 126, in run
    _sys.exit(main(argv))
  File "/Users/Carlos/code/tensorflow/models/research/im2txt/bazel-bin/im2txt/run_inference.runfiles/im2txt/im2txt/run_inference.py", line 55, in main
    vocab = vocabulary.Vocabulary(FLAGS.vocab_file)
  File "/Users/Carlos/code/tensorflow/models/research/im2txt/bazel-bin/im2txt/run_inference.runfiles/im2txt/im2txt/inference_utils/vocabulary.py", line 48, in __init__
    reverse_vocab = list(f.readlines())
  File "/Users/Carlos/code/tensorflow/lib/python2.7/site-packages/tensorflow/python/lib/io/file_io.py", line 183, in readlines
    self._preread_check()
  File "/Users/Carlos/code/tensorflow/lib/python2.7/site-packages/tensorflow/python/lib/io/file_io.py", line 80, in _preread_check
    compat.as_bytes(self.__name), 1024 * 512, status)
  File "/Users/Carlos/code/tensorflow/lib/python2.7/site-packages/tensorflow/python/framework/errors_impl.py", line 516, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.NotFoundError: ; No such file or directory

Tensorflow Serving

I want to deploy the final model for production.
Tensorflow has an example to do this for inception model.
do you have any idea what all changes need to be made to do serving for show and tell model?

Are there any code to calc the BLEU-4

Hello,I have download your pre-trained model for google's tensorflow/im2txt,Thank you very,I have realized that there is a evaluate.py which allows training progress to be monitored in real-time.But there are many people have loaded there pre-trained models,but I don't know which are better,I can only chage a Image into a txt and see it myself,but you know there are many metrics to evaluate the model such as BLUE-4,I want to know ,are there any code for me to evaluate the models that have been downloaded from you or others WITHOUT RUN THE TRAIN CODE,I just want to run a code to gain a BLUE-4 score for some models,thank you very much !

Downloading Dataset

Is It Necessary to Download the 150 GB Dataset from the img2txt ?

Can this model run on CPU instead of GPU?

Hello @KranthiGV i try to run this model on cpu by editing the gpu into cpu in show_and_tell_model.py and i encountered this error.im not sure either i just change the gpu to cpu will help it run on the cpu but possibly there another way you could suggest me.Thank you

Graph PB File

If possible can you please upload the Graph.pb file associated with this model ? And if not can you tell me how to create it from the checkpoints you have provided ?

And thank you so much for uploading these !

[Feature request] Publish the 2M-step checkpoint

Hi and thanks for this!

Could you maybe also convert the second checkpoint from this comment and upload it?

And maybe this one (after 3M steps).

Request perplexity values

Thank you for the checkpoint files!
Could you please also share the perplexity values from evaluation script, if any? Would also appreciate graphs from tensorboard.

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte

is this an image file problem?

Wrong with checkpoint file

Hello, thanks to provide checkpoint file for im2txt project. However, when i follow your instruction to run the code, i met a question: an error that says some variables are not found in checkpoint.

Concretely, the main error tips are described as follows:
NotFoundError (see above for traceback): Key lstm/basic_lstm_cell/kernel not found in checkpoint [[Node: save/RestoreV2_381 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_381/tensor_names, save/RestoreV2_381/shape_and_slices)]]

The whole error are describes as follows:
2017-09-09 16:58:27.152998: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:193] kernel reported version is: 381.9.0 2017-09-09 16:58:27.153007: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:300] kernel version seems to match DSO: 381.9.0 INFO:tensorflow:Loading model from checkpoint: model.ckpt-1000000 INFO:tensorflow:Restoring parameters from model.ckpt-1000000 2017-09-09 16:58:27.548519: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key lstm/basic_lstm_cell/kernel not found in checkpoint 2017-09-09 16:58:27.548988: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key lstm/basic_lstm_cell/bias not found in checkpoint Traceback (most recent call last): File "/home/server109/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1139, in _do_call return fn(*args) File "/home/server109/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1121, in _run_fn status, run_metadata) File "/home/server109/anaconda3/lib/python3.6/contextlib.py", line 89, in __exit__ next(self.gen) File "/home/server109/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 466, in raise_exception_on_not_ok_status pywrap_tensorflow.TF_GetCode(status)) tensorflow.python.framework.errors_impl.NotFoundError: Key lstm/basic_lstm_cell/kernel not found in checkpoint [[Node: save/RestoreV2_381 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_381/tensor_names, save/RestoreV2_381/shape_and_slices)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/server109/Documents/Show_and_Tell/im2txt/bazel-bin/im2txt/run_inference.runfiles/im2txt/im2txt/run_inference.py", line 85, in <module> tf.app.run() File "/home/server109/anaconda3/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 48, in run _sys.exit(main(_sys.argv[:1] + flags_passthrough)) File "/home/server109/Documents/Show_and_Tell/im2txt/bazel-bin/im2txt/run_inference.runfiles/im2txt/im2txt/run_inference.py", line 65, in main restore_fn(sess) File "/home/server109/Documents/Show_and_Tell/im2txt/bazel-bin/im2txt/run_inference.runfiles/im2txt/im2txt/inference_utils/inference_wrapper_base.py", line 96, in _restore_fn saver.restore(sess, checkpoint_path) File "/home/server109/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1548, in restore {self.saver_def.filename_tensor_name: save_path}) File "/home/server109/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 789, in run run_metadata_ptr) File "/home/server109/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 997, in _run feed_dict_string, options, run_metadata) File "/home/server109/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1132, in _do_run target_list, options, run_metadata) File "/home/server109/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1152, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.NotFoundError: Key lstm/basic_lstm_cell/kernel not found in checkpoint [[Node: save/RestoreV2_381 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_381/tensor_names, save/RestoreV2_381/shape_and_slices)]]

Caused by op 'save/RestoreV2_381', defined at: File "/home/server109/Documents/Show_and_Tell/im2txt/bazel-bin/im2txt/run_inference.runfiles/im2txt/im2txt/run_inference.py", line 85, in <module> tf.app.run() File "/home/server109/anaconda3/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 48, in run _sys.exit(main(_sys.argv[:1] + flags_passthrough)) File "/home/server109/Documents/Show_and_Tell/im2txt/bazel-bin/im2txt/run_inference.runfiles/im2txt/im2txt/run_inference.py", line 51, in main FLAGS.checkpoint_path) File "/home/server109/Documents/Show_and_Tell/im2txt/bazel-bin/im2txt/run_inference.runfiles/im2txt/im2txt/inference_utils/inference_wrapper_base.py", line 116, in build_graph_from_config saver = tf.train.Saver() File "/home/server109/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1139, in __init__ self.build() File "/home/server109/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1170, in build restore_sequentially=self._restore_sequentially) File "/home/server109/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 691, in build restore_sequentially, reshape) File "/home/server109/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 407, in _AddRestoreOps tensors = self.restore_op(filename_tensor, saveable, preferred_shard) File "/home/server109/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 247, in restore_op [spec.tensor.dtype])[0]) File "/home/server109/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/gen_io_ops.py", line 640, in restore_v2 dtypes=dtypes, name=name) File "/home/server109/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op op_def=op_def) File "/home/server109/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2506, in create_op original_op=self._default_original_op, op_def=op_def) File "/home/server109/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1269, in __init__ self._traceback = _extract_stack()

NotFoundError (see above for traceback): Key lstm/basic_lstm_cell/kernel not found in checkpoint [[Node: save/RestoreV2_381 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_381/tensor_names, save/RestoreV2_381/shape_and_slices)]]

How can i fix this? Thanks :)

ValueError: No checkpoint file found in: None

hello, I download the pre-trained model you provide and put it in the same location as model.ckpt-2000000.index or model.ckpt-1000000.index. but when I run

bazel-bin/im2txt/run_inference \
  --checkpoint_path=${CHECKPOINT_PATH} \
  --vocab_file=${VOCAB_FILE} \
  --input_files=${IMAGE_FILE}

it shows error:

NFO:tensorflow:Building model.
Traceback (most recent call last):
  File "/home/ztgong/work/deeplearning-tools/tensorflow/models/im2txt/bazel-bin/im2txt/run_inference.runfiles/im2txt/im2txt/run_inference.py", line 85, in <module>
    tf.app.run()
  File "/home/ztgong/local/anaconda2/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 44, in run
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File "/home/ztgong/work/deeplearning-tools/tensorflow/models/im2txt/bazel-bin/im2txt/run_inference.runfiles/im2txt/im2txt/run_inference.py", line 51, in main
    FLAGS.checkpoint_path)
  File "/home/ztgong/work/deeplearning-tools/tensorflow/models/im2txt/bazel-bin/im2txt/run_inference.runfiles/im2txt/im2txt/inference_utils/inference_wrapper_base.py", line 118, in build_graph_from_config
    return self._create_restore_fn(checkpoint_path, saver)
  File "/home/ztgong/work/deeplearning-tools/tensorflow/models/im2txt/bazel-bin/im2txt/run_inference.runfiles/im2txt/im2txt/inference_utils/inference_wrapper_base.py", line 92, in _create_restore_fn
    raise ValueError("No checkpoint file found in: %s" % checkpoint_path)
ValueError: No checkpoint file found in: None

what went wrong? can you provide some advises?