sjenni/temporal-ssl

Video Representation Learning by Recognizing Temporal Transformations. In ECCV, 2020.

Video Representation Learning by Recognizing Temporal Transformations [Project Page]

Simon Jenni, Givi Meishvili, and Paolo Favaro.
In ECCV, 2020.

This repository contains code for self-supervised pre-training on UCF101 and supervised transfer learning on the UCF101 and HMDB51 action recognition benchmarks.

Requirements

The code is based on Python 3.7 and tensorflow 1.15.

How to use it

1. Setup

• Dowload UCF101 and HMDB51.
• Set the paths to the data and log directories in constants.py.
• Run init_datasets.py to convert the UCF101 and HMDB datasets to the TFRecord format:

python init_datasets.py

2. Training and evaluation

• To train and evaluate a model using the C3D architecture, execute train_test_C3D.py. An example usage could look like this:

python train_test_C3D.py --tag='test' --num_gpus=2

temporal-ssl's Issues

About training logs.

Hi Jenni,
I'm the author of VTDL. I'm preparing my manuscript and aims to compare with your work. It seems your R(2+1)D model can achieve 46.4 with 112 resolution input which seems very surprising. Could you provide train logs or pretrain weight for your experiments?

R(2+1)D backbone

We noticed in your paper that you got an amazing result with R(2+1)D backbone. Could you public your implementation of R(2+1)D model as well as the best checkpoint? Thanks!

Sampling Technique

Hello
Thank you friend for sharing your work and knowledge. I am sorry for asking these question but I am not familiar with
tensor flow at all.

Please could you clarify the following questions:

1- During the down stream task (action recognition) training, did you sample one clip from each training video using random
starting index ? If Yes, then at each epoch the total number of training videos would be equal to the size of the training split.

Or
Did you use temporal jittering during training? If Yes how many clips did you sample from each training video ? 
What is the size of one epoch then ?

2- During down stream task evaluation, you mentioned in the paper that you used all the sub sequences of each testing video
in the test split to get the video level prediction.
What if the testing video length is not divisible by the clip length, then there would be extra frames that are not enough
to sample one clip ? What is your approach to over come this issue ?

For example: When the testing video has a 173 frames and the clip length is 16 frames then 10 non overlapping clips 
can be sampled and  13 extra frames that are not enough to sample one clip are left over.

Thanks for your help

About two softmax outputs

Hi,
When SSLtraining In your paper, you claim to use two softmax for the pseudo-task. However, in this code, it seems you use tf.split to split the prediction（eg. 8 class for all, and 4 for skip ,the other 4 for transforms）, which lead to only one softmax. what is the difference about this, or just my misunderstanding.
Looking forward to your reply.

Extremely Slow Training

Thanks for releasing the temporal-ssl codebase. I run the temporal-ssl training with command python train_test_C3D.py --tag='test' --num_gpus=2 and find that the training is extremely slow on a 8-TitanXp server: it takes several minutes for one iteration:

How long does it take to train temporal-ssl on your platform? Would you please share the training log?

Cannot reproduce linear evaluation performance on UCF-101

Dear friend, thank you very much for your work, I really learned a lot from it. It is impressive that after training only on speed prediction and 50 epoch, it got 49.3% acc on UCF-101 with linear evaluation. Nowadays, I have been trying to reproduce this performance on Pytorch following your code. But I just got 15% acc on UCF-101 with linear evaluation. Could you please give me some advice on how to achieve the performance? I have checked a lot of times that I followed your code and I may neglect some important things. Thank you very much.