I cannot get train.py to run. I get this verbose error message. Any suggestions?
I can run evaluate and demo without issue
(py2) anthony@aZ68MA-D2H-B3:~/KittiBox$ python train.py
2017-06-12 13:52:22,864 INFO No environment variable 'TV_PLUGIN_DIR' found. Set to '/home/anthony/tv-plugins'.
2017-06-12 13:52:22,864 INFO No environment variable 'TV_STEP_SHOW' found. Set to '50'.
2017-06-12 13:52:22,864 INFO No environment variable 'TV_STEP_EVAL' found. Set to '250'.
2017-06-12 13:52:22,864 INFO No environment variable 'TV_STEP_WRITE' found. Set to '1000'.
2017-06-12 13:52:22,864 INFO No environment variable 'TV_MAX_KEEP' found. Set to '10'.
2017-06-12 13:52:22,864 INFO No environment variable 'TV_STEP_STR' found. Set to 'Step {step}/{total_steps}: loss = {loss_value:.2f}; lr = {lr_value:.2e}; {sec_per_batch:.3f} sec (per Batch); {examples_per_sec:.1f} imgs/sec'.
2017-06-12 13:52:22,865 INFO f: <open file 'hypes/kittiBox.json', mode 'r' at 0x7feb344ba270>
2017-06-12 13:52:22,865 INFO Initialize training folder
2017-06-12 13:52:22,870 INFO Start training
2017-06-12 13:52:22.872836: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2017-06-12 13:52:22.872856: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2017-06-12 13:52:22.872863: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2017-06-12 13:52:22.986751: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:901] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2017-06-12 13:52:22.987118: I tensorflow/core/common_runtime/gpu/gpu_device.cc:887] Found device 0 with properties:
name: GeForce GTX 1070
major: 6 minor: 1 memoryClockRate (GHz) 1.7845
pciBusID 0000:01:00.0
Total memory: 7.92GiB
Free memory: 6.89GiB
2017-06-12 13:52:22.987143: I tensorflow/core/common_runtime/gpu/gpu_device.cc:908] DMA: 0
2017-06-12 13:52:22.987150: I tensorflow/core/common_runtime/gpu/gpu_device.cc:918] 0: Y
2017-06-12 13:52:22.987163: I tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1070, pci bus id: 0000:01:00.0)
npy file loaded
Layer name: conv1_1
Layer shape: (3, 3, 3, 64)
2017-06-12 13:52:23,261 INFO Creating Summary for: conv1_1/filter
2017-06-12 13:52:23,278 INFO Creating Summary for: conv1_1/biases
Layer name: conv1_2
Layer shape: (3, 3, 64, 64)
2017-06-12 13:52:23,299 INFO Creating Summary for: conv1_2/filter
2017-06-12 13:52:23,315 INFO Creating Summary for: conv1_2/biases
Layer name: conv2_1
Layer shape: (3, 3, 64, 128)
2017-06-12 13:52:23,337 INFO Creating Summary for: conv2_1/filter
2017-06-12 13:52:23,353 INFO Creating Summary for: conv2_1/biases
Layer name: conv2_2
Layer shape: (3, 3, 128, 128)
2017-06-12 13:52:23,375 INFO Creating Summary for: conv2_2/filter
2017-06-12 13:52:23,391 INFO Creating Summary for: conv2_2/biases
Layer name: conv3_1
Layer shape: (3, 3, 128, 256)
2017-06-12 13:52:23,413 INFO Creating Summary for: conv3_1/filter
2017-06-12 13:52:23,430 INFO Creating Summary for: conv3_1/biases
Layer name: conv3_2
Layer shape: (3, 3, 256, 256)
2017-06-12 13:52:23,453 INFO Creating Summary for: conv3_2/filter
2017-06-12 13:52:23,469 INFO Creating Summary for: conv3_2/biases
Layer name: conv3_3
Layer shape: (3, 3, 256, 256)
2017-06-12 13:52:23,491 INFO Creating Summary for: conv3_3/filter
2017-06-12 13:52:23,507 INFO Creating Summary for: conv3_3/biases
Layer name: conv4_1
Layer shape: (3, 3, 256, 512)
2017-06-12 13:52:23,533 INFO Creating Summary for: conv4_1/filter
2017-06-12 13:52:23,550 INFO Creating Summary for: conv4_1/biases
Layer name: conv4_2
Layer shape: (3, 3, 512, 512)
2017-06-12 13:52:23,578 INFO Creating Summary for: conv4_2/filter
2017-06-12 13:52:23,594 INFO Creating Summary for: conv4_2/biases
Layer name: conv4_3
Layer shape: (3, 3, 512, 512)
2017-06-12 13:52:23,621 INFO Creating Summary for: conv4_3/filter
2017-06-12 13:52:23,638 INFO Creating Summary for: conv4_3/biases
Layer name: conv5_1
Layer shape: (3, 3, 512, 512)
2017-06-12 13:52:23,664 INFO Creating Summary for: conv5_1/filter
2017-06-12 13:52:23,681 INFO Creating Summary for: conv5_1/biases
Layer name: conv5_2
Layer shape: (3, 3, 512, 512)
2017-06-12 13:52:23,707 INFO Creating Summary for: conv5_2/filter
2017-06-12 13:52:23,723 INFO Creating Summary for: conv5_2/biases
Layer name: conv5_3
Layer shape: (3, 3, 512, 512)
2017-06-12 13:52:23,750 INFO Creating Summary for: conv5_3/filter
2017-06-12 13:52:23,765 INFO Creating Summary for: conv5_3/biases
Layer name: fc6
Layer shape: [7, 7, 512, 4096]
2017-06-12 13:52:24,032 INFO Creating Summary for: fc6/weights
2017-06-12 13:52:24,049 INFO Creating Summary for: fc6/biases
Layer name: fc7
Layer shape: [1, 1, 4096, 4096]
2017-06-12 13:52:24,117 INFO Creating Summary for: fc7/weights
2017-06-12 13:52:24,134 INFO Creating Summary for: fc7/biases
2017-06-12 13:52:24,163 INFO Creating Summary for: score_fr/weights
2017-06-12 13:52:24,179 INFO Creating Summary for: score_fr/biases
2017-06-12 13:52:24,209 INFO Creating Summary for: upscore2/up_filter
2017-06-12 13:52:24,234 INFO Creating Summary for: score_pool4/weights
2017-06-12 13:52:24,250 INFO Creating Summary for: score_pool4/biases
2017-06-12 13:52:24,298 INFO Creating Summary for: upscore4/up_filter
2017-06-12 13:52:24,333 INFO Creating Summary for: score_pool3/weights
2017-06-12 13:52:24,355 INFO Creating Summary for: score_pool3/biases
2017-06-12 13:52:24,422 INFO Creating Summary for: upscore32/up_filter
INFO:tensorflow:Summary name /delta_hist0_x is illegal; using delta_hist0_x instead.
2017-06-12 13:52:24,909 INFO Summary name /delta_hist0_x is illegal; using delta_hist0_x instead.
INFO:tensorflow:Summary name /delta_hist0_y is illegal; using delta_hist0_y instead.
2017-06-12 13:52:24,912 INFO Summary name /delta_hist0_y is illegal; using delta_hist0_y instead.
INFO:tensorflow:Summary name /delta_hist0_w is illegal; using delta_hist0_w instead.
2017-06-12 13:52:24,915 INFO Summary name /delta_hist0_w is illegal; using delta_hist0_w instead.
INFO:tensorflow:Summary name /delta_hist0_h is illegal; using delta_hist0_h instead.
2017-06-12 13:52:24,918 INFO Summary name /delta_hist0_h is illegal; using delta_hist0_h instead.
INFO:tensorflow:Summary name /pred_boxes is illegal; using pred_boxes instead.
2017-06-12 13:52:26,754 INFO Summary name /pred_boxes is illegal; using pred_boxes instead.
INFO:tensorflow:Summary name /true_boxes is illegal; using true_boxes instead.
2017-06-12 13:52:26,755 INFO Summary name /true_boxes is illegal; using true_boxes instead.
npy file loaded
Layer name: conv1_1
Layer shape: (3, 3, 3, 64)
Layer name: conv1_2
Layer shape: (3, 3, 64, 64)
Layer name: conv2_1
Layer shape: (3, 3, 64, 128)
Layer name: conv2_2
Layer shape: (3, 3, 128, 128)
Layer name: conv3_1
Layer shape: (3, 3, 128, 256)
Layer name: conv3_2
Layer shape: (3, 3, 256, 256)
Layer name: conv3_3
Layer shape: (3, 3, 256, 256)
Layer name: conv4_1
Layer shape: (3, 3, 256, 512)
Layer name: conv4_2
Layer shape: (3, 3, 512, 512)
Layer name: conv4_3
Layer shape: (3, 3, 512, 512)
Layer name: conv5_1
Layer shape: (3, 3, 512, 512)
Layer name: conv5_2
Layer shape: (3, 3, 512, 512)
Layer name: conv5_3
Layer shape: (3, 3, 512, 512)
Layer name: fc6
Layer shape: [7, 7, 512, 4096]
Layer name: fc7
Layer shape: [1, 1, 4096, 4096]
2017-06-12 13:52:39,734 INFO Step 0/140000: loss = 7.10; lr = 1.00e-05; 0.094 sec (per Batch); 31.9 imgs/sec
2017-06-12 13:52:40,000 INFO (raw) Acc.: 0.42,Conf: 0.85,Box: 2.04,Weight: 0.38,Delta: 2.32
2017-06-12 13:52:40,000 INFO (smooth) Acc.: 0.42,Conf: 0.85,Box: 2.04,Weight: 0.38,Delta: 2.32
Traceback (most recent call last):
File "/home/anthony/miniconda3/envs/py2/lib/python2.7/site-packages/tensorflow/python/ops/script_ops.py", line 82, in __call__
ret = func(*args)
File "/home/anthony/KittiBox/hypes/../decoder/fastBox.py", line 460, in log_image
plot_image = data_utils.draw_encoded(
AttributeError: 'module' object has no attribute 'draw_encoded'
2017-06-12 13:52:41.271263: W tensorflow/core/framework/op_kernel.cc:1152] Internal: Failed to run py callback pyfunc_1: see error log.
2017-06-12 13:52:41.540951: W tensorflow/core/framework/op_kernel.cc:1152] Internal: Failed to run py callback pyfunc_1: see error log.
[[Node: Evaluation/PyFunc_1 = PyFunc[Tin=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT32, DT_STRING], Tout=[DT_UINT8], token="pyfunc_1", _device="/job:localhost/replica:0/task:0/cpu:0"](Inputs/Maximum/_519, Inputs/fifo_queue_DequeueMany:1, Inputs/fifo_queue_DequeueMany:3, Optimizer/Variable/read, Evaluation/PyFunc_1/input_4)]]
2017-06-12 13:52:41.541169: W tensorflow/core/framework/op_kernel.cc:1152] Internal: Failed to run py callback pyfunc_1: see error log.
[[Node: Evaluation/PyFunc_1 = PyFunc[Tin=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT32, DT_STRING], Tout=[DT_UINT8], token="pyfunc_1", _device="/job:localhost/replica:0/task:0/cpu:0"](Inputs/Maximum/_519, Inputs/fifo_queue_DequeueMany:1, Inputs/fifo_queue_DequeueMany:3, Optimizer/Variable/read, Evaluation/PyFunc_1/input_4)]]
2017-06-12 13:52:41.541397: W tensorflow/core/framework/op_kernel.cc:1152] Internal: Failed to run py callback pyfunc_1: see error log.
[[Node: Evaluation/PyFunc_1 = PyFunc[Tin=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT32, DT_STRING], Tout=[DT_UINT8], token="pyfunc_1", _device="/job:localhost/replica:0/task:0/cpu:0"](Inputs/Maximum/_519, Inputs/fifo_queue_DequeueMany:1, Inputs/fifo_queue_DequeueMany:3, Optimizer/Variable/read, Evaluation/PyFunc_1/input_4)]]
2017-06-12 13:52:41.541553: W tensorflow/core/framework/op_kernel.cc:1152] Internal: Failed to run py callback pyfunc_1: see error log.
[[Node: Evaluation/PyFunc_1 = PyFunc[Tin=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT32, DT_STRING], Tout=[DT_UINT8], token="pyfunc_1", _device="/job:localhost/replica:0/task:0/cpu:0"](Inputs/Maximum/_519, Inputs/fifo_queue_DequeueMany:1, Inputs/fifo_queue_DequeueMany:3, Optimizer/Variable/read, Evaluation/PyFunc_1/input_4)]]
2017-06-12 13:52:41.541776: W tensorflow/core/framework/op_kernel.cc:1152] Internal: Failed to run py callback pyfunc_1: see error log.
[[Node: Evaluation/PyFunc_1 = PyFunc[Tin=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT32, DT_STRING], Tout=[DT_UINT8], token="pyfunc_1", _device="/job:localhost/replica:0/task:0/cpu:0"](Inputs/Maximum/_519, Inputs/fifo_queue_DequeueMany:1, Inputs/fifo_queue_DequeueMany:3, Optimizer/Variable/read, Evaluation/PyFunc_1/input_4)]]
2017-06-12 13:52:41.541936: W tensorflow/core/framework/op_kernel.cc:1152] Internal: Failed to run py callback pyfunc_1: see error log.
[[Node: Evaluation/PyFunc_1 = PyFunc[Tin=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT32, DT_STRING], Tout=[DT_UINT8], token="pyfunc_1", _device="/job:localhost/replica:0/task:0/cpu:0"](Inputs/Maximum/_519, Inputs/fifo_queue_DequeueMany:1, Inputs/fifo_queue_DequeueMany:3, Optimizer/Variable/read, Evaluation/PyFunc_1/input_4)]]
2017-06-12 13:52:41.612397: W tensorflow/core/framework/op_kernel.cc:1152] Internal: Failed to run py callback pyfunc_1: see error log.
[[Node: Evaluation/PyFunc_1 = PyFunc[Tin=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT32, DT_STRING], Tout=[DT_UINT8], token="pyfunc_1", _device="/job:localhost/replica:0/task:0/cpu:0"](Inputs/Maximum/_519, Inputs/fifo_queue_DequeueMany:1, Inputs/fifo_queue_DequeueMany:3, Optimizer/Variable/read, Evaluation/PyFunc_1/input_4)]]
2017-06-12 13:52:41.612638: W tensorflow/core/framework/op_kernel.cc:1152] Internal: Failed to run py callback pyfunc_1: see error log.
[[Node: Evaluation/PyFunc_1 = PyFunc[Tin=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT32, DT_STRING], Tout=[DT_UINT8], token="pyfunc_1", _device="/job:localhost/replica:0/task:0/cpu:0"](Inputs/Maximum/_519, Inputs/fifo_queue_DequeueMany:1, Inputs/fifo_queue_DequeueMany:3, Optimizer/Variable/read, Evaluation/PyFunc_1/input_4)]]
2017-06-12 13:52:41.613163: W tensorflow/core/framework/op_kernel.cc:1152] Internal: Failed to run py callback pyfunc_1: see error log.
[[Node: Evaluation/PyFunc_1 = PyFunc[Tin=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT32, DT_STRING], Tout=[DT_UINT8], token="pyfunc_1", _device="/job:localhost/replica:0/task:0/cpu:0"](Inputs/Maximum/_519, Inputs/fifo_queue_DequeueMany:1, Inputs/fifo_queue_DequeueMany:3, Optimizer/Variable/read, Evaluation/PyFunc_1/input_4)]]
2017-06-12 13:52:41.613290: W tensorflow/core/framework/op_kernel.cc:1152] Internal: Failed to run py callback pyfunc_1: see error log.
[[Node: Evaluation/PyFunc_1 = PyFunc[Tin=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT32, DT_STRING], Tout=[DT_UINT8], token="pyfunc_1", _device="/job:localhost/replica:0/task:0/cpu:0"](Inputs/Maximum/_519, Inputs/fifo_queue_DequeueMany:1, Inputs/fifo_queue_DequeueMany:3, Optimizer/Variable/read, Evaluation/PyFunc_1/input_4)]]
2017-06-12 13:52:41.613442: W tensorflow/core/framework/op_kernel.cc:1152] Internal: Failed to run py callback pyfunc_1: see error log.
[[Node: Evaluation/PyFunc_1 = PyFunc[Tin=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT32, DT_STRING], Tout=[DT_UINT8], token="pyfunc_1", _device="/job:localhost/replica:0/task:0/cpu:0"](Inputs/Maximum/_519, Inputs/fifo_queue_DequeueMany:1, Inputs/fifo_queue_DequeueMany:3, Optimizer/Variable/read, Evaluation/PyFunc_1/input_4)]]
2017-06-12 13:52:41.664624: W tensorflow/core/framework/op_kernel.cc:1152] Internal: Failed to run py callback pyfunc_1: see error log.
[[Node: Evaluation/PyFunc_1 = PyFunc[Tin=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT32, DT_STRING], Tout=[DT_UINT8], token="pyfunc_1", _device="/job:localhost/replica:0/task:0/cpu:0"](Inputs/Maximum/_519, Inputs/fifo_queue_DequeueMany:1, Inputs/fifo_queue_DequeueMany:3, Optimizer/Variable/read, Evaluation/PyFunc_1/input_4)]]
2017-06-12 13:52:41.664875: W tensorflow/core/framework/op_kernel.cc:1152] Internal: Failed to run py callback pyfunc_1: see error log.
[[Node: Evaluation/PyFunc_1 = PyFunc[Tin=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT32, DT_STRING], Tout=[DT_UINT8], token="pyfunc_1", _device="/job:localhost/replica:0/task:0/cpu:0"](Inputs/Maximum/_519, Inputs/fifo_queue_DequeueMany:1, Inputs/fifo_queue_DequeueMany:3, Optimizer/Variable/read, Evaluation/PyFunc_1/input_4)]]
2017-06-12 13:52:41.665110: W tensorflow/core/framework/op_kernel.cc:1152] Internal: Failed to run py callback pyfunc_1: see error log.
[[Node: Evaluation/PyFunc_1 = PyFunc[Tin=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT32, DT_STRING], Tout=[DT_UINT8], token="pyfunc_1", _device="/job:localhost/replica:0/task:0/cpu:0"](Inputs/Maximum/_519, Inputs/fifo_queue_DequeueMany:1, Inputs/fifo_queue_DequeueMany:3, Optimizer/Variable/read, Evaluation/PyFunc_1/input_4)]]
2017-06-12 13:52:41.665352: W tensorflow/core/framework/op_kernel.cc:1152] Internal: Failed to run py callback pyfunc_1: see error log.
[[Node: Evaluation/PyFunc_1 = PyFunc[Tin=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT32, DT_STRING], Tout=[DT_UINT8], token="pyfunc_1", _device="/job:localhost/replica:0/task:0/cpu:0"](Inputs/Maximum/_519, Inputs/fifo_queue_DequeueMany:1, Inputs/fifo_queue_DequeueMany:3, Optimizer/Variable/read, Evaluation/PyFunc_1/input_4)]]
2017-06-12 13:52:41.665449: W tensorflow/core/framework/op_kernel.cc:1152] Internal: Failed to run py callback pyfunc_1: see error log.
[[Node: Evaluation/PyFunc_1 = PyFunc[Tin=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT32, DT_STRING], Tout=[DT_UINT8], token="pyfunc_1", _device="/job:localhost/replica:0/task:0/cpu:0"](Inputs/Maximum/_519, Inputs/fifo_queue_DequeueMany:1, Inputs/fifo_queue_DequeueMany:3, Optimizer/Variable/read, Evaluation/PyFunc_1/input_4)]]
2017-06-12 13:52:41.665539: W tensorflow/core/framework/op_kernel.cc:1152] Internal: Failed to run py callback pyfunc_1: see error log.
[[Node: Evaluation/PyFunc_1 = PyFunc[Tin=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT32, DT_STRING], Tout=[DT_UINT8], token="pyfunc_1", _device="/job:localhost/replica:0/task:0/cpu:0"](Inputs/Maximum/_519, Inputs/fifo_queue_DequeueMany:1, Inputs/fifo_queue_DequeueMany:3, Optimizer/Variable/read, Evaluation/PyFunc_1/input_4)]]
2017-06-12 13:52:41.665630: W tensorflow/core/framework/op_kernel.cc:1152] Internal: Failed to run py callback pyfunc_1: see error log.
[[Node: Evaluation/PyFunc_1 = PyFunc[Tin=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT32, DT_STRING], Tout=[DT_UINT8], token="pyfunc_1", _device="/job:localhost/replica:0/task:0/cpu:0"](Inputs/Maximum/_519, Inputs/fifo_queue_DequeueMany:1, Inputs/fifo_queue_DequeueMany:3, Optimizer/Variable/read, Evaluation/PyFunc_1/input_4)]]
2017-06-12 13:52:45.969701: W tensorflow/core/kernels/queue_base.cc:294] _0_Queues/fifo_queue: Skipping cancelled enqueue attempt with queue not closed
Traceback (most recent call last):
File "train.py", line 82, in <module>
tf.app.run()
File "/home/anthony/miniconda3/envs/py2/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "train.py", line 78, in main
train.do_training(hypes)
File "incl/tensorvision/train.py", line 397, in do_training
run_training(hypes, modules, tv_graph, tv_sess)
File "incl/tensorvision/train.py", line 266, in run_training
feed_dict=feed_dict)
File "/home/anthony/miniconda3/envs/py2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 778, in run
run_metadata_ptr)
File "/home/anthony/miniconda3/envs/py2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 982, in _run
feed_dict_string, options, run_metadata)
File "/home/anthony/miniconda3/envs/py2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1032, in _do_run
target_list, options, run_metadata)
File "/home/anthony/miniconda3/envs/py2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1052, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InternalError: Failed to run py callback pyfunc_1: see error log.
[[Node: Evaluation/PyFunc_1 = PyFunc[Tin=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT32, DT_STRING], Tout=[DT_UINT8], token="pyfunc_1", _device="/job:localhost/replica:0/task:0/cpu:0"](Inputs/Maximum/_519, Inputs/fifo_queue_DequeueMany:1, Inputs/fifo_queue_DequeueMany:3, Optimizer/Variable/read, Evaluation/PyFunc_1/input_4)]]
[[Node: Evaluation/PyFunc/_549 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_2255_Evaluation/PyFunc", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]()]]
Caused by op u'Evaluation/PyFunc_1', defined at:
File "train.py", line 82, in <module>
tf.app.run()
File "/home/anthony/miniconda3/envs/py2/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "train.py", line 78, in main
train.do_training(hypes)
File "incl/tensorvision/train.py", line 378, in do_training
tv_graph = core.build_training_graph(hypes, queue, modules)
File "incl/tensorvision/core.py", line 105, in build_training_graph
hypes, image, labels, decoded_logits, losses, global_step)
File "/home/anthony/KittiBox/hypes/../decoder/fastBox.py", line 481, in evaluation
[tf.uint8])
File "/home/anthony/miniconda3/envs/py2/lib/python2.7/site-packages/tensorflow/python/ops/script_ops.py", line 189, in py_func
input=inp, token=token, Tout=Tout, name=name)
File "/home/anthony/miniconda3/envs/py2/lib/python2.7/site-packages/tensorflow/python/ops/gen_script_ops.py", line 40, in _py_func
name=name)
File "/home/anthony/miniconda3/envs/py2/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 768, in apply_op
op_def=op_def)
File "/home/anthony/miniconda3/envs/py2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2336, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/home/anthony/miniconda3/envs/py2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1228, in __init__
self._traceback = _extract_stack()
InternalError (see above for traceback): Failed to run py callback pyfunc_1: see error log.
[[Node: Evaluation/PyFunc_1 = PyFunc[Tin=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT32, DT_STRING], Tout=[DT_UINT8], token="pyfunc_1", _device="/job:localhost/replica:0/task:0/cpu:0"](Inputs/Maximum/_519, Inputs/fifo_queue_DequeueMany:1, Inputs/fifo_queue_DequeueMany:3, Optimizer/Variable/read, Evaluation/PyFunc_1/input_4)]]
[[Node: Evaluation/PyFunc/_549 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_2255_Evaluation/PyFunc", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]()]]
(py2)