lossnan/i3d-tensorflow

Train I3D model on ucf101 or hmdb51 by tensorflow

Train I3D model on ucf101 or hmdb51 by tensorflow

This code also for training your own dataset

Setup

First follow the instructions for install kinetics-i3d
Then, clone this repository using

$git clone https://github.com/LossNAN/I3D-Tensorflow.git

How to use our code?

1.Data_process

1>download UCF101 and HMDB51 dataset by yourself
2>extract RGB and FLOW frames by denseFlow_GPU, such as:

• ~PATH/UCF-101/ApplyEyeMakeup/v_ApplyEyeMakeup_g01_c01/i for all rgb frames
  
• ~PATH/UCF-101/ApplyEyeMakeup/v_ApplyEyeMakeup_g01_c01/x for all x_flow frames
  
• ~PATH/UCF-101/ApplyEyeMakeup/v_ApplyEyeMakeup_g01_c01/y for all y_flow frames
  

3>convert images to list for train and test

cd ./list/ucf_list/
bash ./convert_images_to_list.sh ~path/UCF-101 4

• you will get train.list and test.list for your own dataset
  
• such as: ~PATH/UCF-101/ApplyEyeMakeup/v_ApplyEyeMakeup_g01_c01 0
  

2.Train your own dataset(UCF101 as example)

1>if you get path errors, please modify by yourself

cd ./experiments/ucf-101
python train_ucf_rgb.py
python train_ucf_flow.py

2>argues

• learning_rate: Initial learning rate
• max_steps: Number of steps to run trainer
• batch_size: Batch size
• num_frame_per_clib: Nummber of frames per clib
• crop_size: Crop_size
• classics: The num of class

3>models will be stored at ./models, and tensorboard logs will be stored at ./visul_logs

tensorboard --logdir=~path/I3D/experiments/ucf_101/visual_logs/

3.Test your own models

1>if you get path errors, please modify by yourself

cd ./experiments/ucf-101
python test_ucf_rgb.py
python test_ucf_flow.py
python test_ucf_rgb+flow.py

4.Result on my linux

Architecture	Pre_train	ACC/top1
RGB+I3D	Kinetics	86.6
FLOW+I3D	Kinetics	91.8
TWO_STREAM+I3D	Kinetics	95.3
FLOW+I3D	IMAGENET+Kinetics	94.72
RGB+I3D	IMAGENET+Kinetics	95.68
TWO_STREAM+I3D	IMAGENET+Kinetics	97.6

i3d-tensorflow's Issues

Dear author According to the README I have encountered some problems. python train_ucf_flow.py

File "../../input_data.py", line 194, in read_clip_and_label
tmp_label = line[1]
IndexError: list index out of range

Looking forward to your answer

Dear author According to the README I have encountered some problems. python train_ucf_flow.py Is there a problem with my list file format?

Traceback (most recent call last):
File "/home/jhy/3DCNN/I3D-Tensorflow-master/experiments/ucf-101/train_ucf_flow.py", line 183, in
tf.app.run()
File "/home/jhy/yes/envs/I3D/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 125, in run
_sys.exit(main(argv))
File "/home/jhy/3DCNN/I3D-Tensorflow-master/experiments/ucf-101/train_ucf_flow.py", line 179, in main
run_training()
File "/home/jhy/3DCNN/I3D-Tensorflow-master/experiments/ucf-101/train_ucf_flow.py", line 133, in run_training
shuffle=True
File "../../input_data.py", line 194, in read_clip_and_label
tmp_label = line[1]
IndexError: list index out of range

Not all input data has been trained

From the code below it seems that at each step only the number of videos trained at each step is equal to batch size and also these videos are chosen at random as i see next_batch_start value from the input_data.py is not fed back in the next step.
for step in xrange(FLAGS.max_steps):
start_time = time.time()
rgb_train_images, flow_train_images, train_labels, _, _, _ = input_data.read_clip_and_label(
filename='../../list/ucf_list/train.txt',
batch_size=FLAGS.batch_size * gpu_num,
num_frames_per_clip=FLAGS.num_frame_per_clib,
crop_size=FLAGS.crop_size,
shuffle=True
)
To my understanding all the training set should be trained at each step. Can the @author or anyone who is successfull in implementing this repo please help me understand how this is working.

How to save the classification results for each video in the testing list?

Thank you for your excellent work, my question is how to save .h5 or .npz classfication score files as shown in https://github.com/rohitgirdhar/ActionVLAD/tree/master/data/hmdb51/final_logits or https://github.com/youjiangxu/seqvlad-pytorch/tree/master/logits/hmdb51, thank you so much for you kind help, looking forward to your reply, thanks a lot.

Can the .list file in I3D-Tensorflow/list/hmdb_list/ be changed to .txt file?

Can the .list file in https://github.com/LossNAN/I3D-Tensorflow/tree/master/list/hmdb_list be changed to .txt file format? Will it influence the functionality of the program? Thank you very much.

IndexError: too many indices for array

Can you tell me how to fix this problem, thank you very much.

checkpoint cannot be loaded

Hello @LossNAN

thanks for the wonderful repo. I tried to use the multi_gpu_test_kinetics_rgb.py. But it could not load the checkpoints saying that
"Key inception_i3d/Conv3d_1a_7x7/batch_norm/beta not found in checkpoint". Below is the full error dump.

Could you please help pointing out what exactly is causing this error?

---

Traceback (most recent call last):
File "multi_gpu_test_thumos14.py", line 336, in
tf.app.run()
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/platform/app.py", line 125, in run
_sys.exit(main(argv))
File "multi_gpu_test_thumos14.py", line 332, in main
run_training()
File "multi_gpu_test_thumos14.py", line 270, in run_training
saver.restore(sess, _MODEL_CKPT)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 1562, in restore
err, "a Variable name or other graph key that is missing")
tensorflow.python.framework.errors_impl.NotFoundError: Restoring from checkpoint failed. This is most likely due to a Variable name or other graph key that is missing from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

Key inception_i3d/Conv3d_1a_7x7/batch_norm/beta not found in checkpoint
[[node save/RestoreV2 (defined at multi_gpu_test_thumos14.py:269) = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]

Caused by op 'save/RestoreV2', defined at:
File "multi_gpu_test_thumos14.py", line 336, in
tf.app.run()
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/platform/app.py", line 125, in run
_sys.exit(main(argv))
File "multi_gpu_test_thumos14.py", line 332, in main
run_training()
File "multi_gpu_test_thumos14.py", line 269, in run_training
saver = tf.train.Saver()
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 1102, in init
self.build()
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 1114, in build
self._build(self._filename, build_save=True, build_restore=True)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 1151, in _build
build_save=build_save, build_restore=build_restore)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 795, in _build_internal
restore_sequentially, reshape)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 406, in _AddRestoreOps
restore_sequentially)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 862, in bulk_restore
return io_ops.restore_v2(filename_tensor, names, slices, dtypes)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_io_ops.py", line 1466, in restore_v2
shape_and_slices=shape_and_slices, dtypes=dtypes, name=name)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
return func(*args, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 3274, in create_op
op_def=op_def)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 1770, in init
self._traceback = tf_stack.extract_stack()

NotFoundError (see above for traceback): Restoring from checkpoint failed. This is most likely due to a Variable name or other graph key that is missing from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

Key inception_i3d/Conv3d_1a_7x7/batch_norm/beta not found in checkpoint
[[node save/RestoreV2 (defined at multi_gpu_test_thumos14.py:269) = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]

train_ucf_rgb s_index problem

Hi
I was trying to train on UCF101. With python 2.7 and encounter this problem
File "../../input_data.py", line 65, in get_data
s_index = random.randint(0, len(filenames) - num_frames_per_clip)
File "/home/alex/anaconda3/envs/py27/lib/python2.7/random.py", line 244, in randint
return self.randrange(a, b+1)
File "/home/alex/anaconda3/envs/py27/lib/python2.7/random.py", line 220, in randrange
raise ValueError, "empty range for randrange() (%d,%d, %d)" % (istart, istop, width)
ValueError: empty range for randrange() (0,-63, -63)

Since len(filenames) - num_frames_per_clip could be smaller than 0, I am not sure how to solve it. I try to make it a positive number later I encounter

File "../../input_data.py", line 67, in get_data
image_name = str(filename) + '/' + str(filenames[int(i*sample_rate)+s_index])
IndexError: list index out of range

Do you have any suggestions? Thank you.

About the training layer when fine-tuning

Dear author, how many layers did you train during fine-tuning? Sorry, my code ability is bad.

How can i modify the train file to multiple gpu version

hello, i used the train_ucf file and modify it to multi-gpu according to your resnet-3d version, but do not success, can you provide a multi-gpu version for this model? thanks.

train_ucf_rgb ValueError: empty range for randrange() (0,-63, -63)

File "/usr/代码/I3D-Tensorflow-master/input_data.py", line 65, in get_data
s_index = random.randint(0, len(filenames) - num_frames_per_clip)
File "/usr/lib/python2.7/random.py", line 242, in randint
return self.randrange(a, b+1)
File "/usr/lib/python2.7/random.py", line 218, in randrange
raise ValueError, "empty range for randrange() (%d,%d, %d)" % (istart, istop, width)
ValueError: empty range for randrange() (0,-63, -63)

Do you have any suggestions?Thanks.

Problem using deepflow for rgb/flow frame extraction

@LossNAN, are there any environment requirements to run deepflow? I also tried changing header files to opencv3 in denseFlow_gpu.cpp as suggested in issues of yangwangx/denseFlow_gpu. I still get this error.

./denseFlow_gpu: error while loading shared libraries: libopencv_gpu.so.2.4: cannot open shared object file: No such file or directory

Thanks

validation set and test set issue

When I was reading your code, it seems that you put the test data into the validation process during training, which might not be right. shouldn't we split some data from the training list as the validation set ?

How to pretrain 3DCNN on imagenet

Can you tell more details about the imagenet+kinetics pretrain process? Is it to repeat the images of imagenet to form the input?

Can you tell me the accuracy of hmdb51?

Can you tell me the hyperparameter?

Could you tell me how many steps have you trained on the ucf101

I have trained 10000 on ucf101-rgb and 20000 on ucf101-flow,but I get the acc of two streams just 96.5,what's the problem

About data augmentation

Dear Author,

I check your training code on UCF101, and I found that you do not use data argumentation for training, however, you provided the argumentation code in the data_process_pos function.

Could you please tell me what argumentation skills you used in the training process?

About　shape???

Traceback (most recent call last):
File "train_ucf_rgb.py", line 180, in
tf.app.run()
File "/home/feng/anaconda3/lib/python3.7/site-packages/tensorflow/python/platform/app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "/home/feng/anaconda3/lib/python3.7/site-packages/absl/app.py", line 300, in run
_run_main(main, args)
File "/home/feng/anaconda3/lib/python3.7/site-packages/absl/app.py", line 251, in _run_main
sys.exit(main(argv))
File "train_ucf_rgb.py", line 176, in main
run_training()
File "train_ucf_rgb.py", line 136, in run_training
is_training: True
File "/home/feng/anaconda3/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 950, in run
run_metadata_ptr)
File "/home/feng/anaconda3/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1149, in _run
str(subfeed_t.get_shape())))
ValueError: Cannot feed value of shape (8, 16, 224, 224, 3) for Tensor 'Placeholder:0', which has shape '(8, 64, 224, 224, 3)'

我实在找不到错误了，求大神帮着看一下，感谢

Transfer Learning Not Implemented Correctly

I don't believe this approach correctly implements transfer learning/fine-tuning. I believe you are meant to freeze/fix the convolutional layers and train only the logits layer. You can update the code by changing the calls to compute_gradients as such:

rgb_grads = opt_rgb.compute_gradients(rgb_loss)
rgb_grads = opt_rgb.compute_gradients(rgb_loss, var_list=[v for v in tf.trainable_variables() if "Logits" in v.name])

Hi，What is your environment?

Ubuntu 16.04？
Python 2.7？
CUDA8？
CuDNN v6？
Tensorflow 1.4.1？

Problem of training accuracy

Can you offer me a copy of the trained model on all the 3 splits of UCF101 and HMDB ,Thank you!

Have you trained the model on all 3 splits of the UCF101 and HMDB ? If you trained,can you offer me a copy of it,the paper dealine is approaching and I have no time to train all the 3 splits, Thank you!

('i','x','y' folder)

how this data look like?('i','x','y' folder)

请问这个代码来自哪篇论文，谢谢

every step takes 650 sec . Why my training is so slow

what does num_frame_per_clib mean in train.py and test.py

I noticed that the values of num_frame_per_clib parameters in training and prediction are different. The example is RGB flow: why num_frame_per_clib is 16 when predicted, and when training is 64. So can num_frame_per_clib be set at 64 when predict?

Another problem is that num_frame_per_clib value is 64 during training, but if the number of RGB frames extracted from a video is more than 64, for example, the number of RGB frames of the video is more than 128 during training, does that mean that the num_frame_per_clib value can be set to 128?

About multi_gpu_train_kinetics_rgb.py

Hello, I downloaded your recent codes and I tried to use your multi_gpu_train_kinetics_rgb.py for multiple GPU.

I modified i3d_nonlocal to i3d and tried to train. However, I found error.

ValueError: Variable inception_i3d_1/Conv3d_1a_7x7/conv_3d/w does not exist, or was not created with tf.get_variable(). Did you mean to set reuse=None in varCcope?

Did you see that error message? I have not found a solution yet, and I would like to ask for advice if you know the solution.

the arguments of generating optical flow

Hello, LossNAN!

Would you provide the settings of the 'denseFlow' which generates the optical flow?
Thank you so much.

How to obtain optical flow diagrams in batches

FLOW+I3D (kinetics pre-trained) performance on UCF101

Hello,

Thanks for sharing your code, it is very helpful.

I am struggling with retrieving the results with FLOW+I3D (kinetics pre-trained) on UCF101 published in the"Quo Vadis" paper. I know you published a 91.8% performance, but were you able to match their published performance of 96.5%.

With that, I have some questions on some hyperparameters:

• Do you train with sequence of 16 frames?
• How many gpus do you use in parallel?
• What is your batch_size per gpu?

Thanks!