opengvlab/efficient-video-recognition

Frozen CLIP models are Efficient Video Learners

This is the official implementation of the paper Frozen CLIP models are Efficient Video Learners

@article{lin2022frozen,
  title={Frozen CLIP Models are Efficient Video Learners},
  author={Lin, Ziyi and Geng, Shijie and Zhang, Renrui and Gao, Peng and de Melo, Gerard and Wang, Xiaogang and Dai, Jifeng and Qiao, Yu and Li, Hongsheng},
  journal={arXiv preprint arXiv:2208.03550},
  year={2022}
}

Introduction

The overall architecture of the EVL framework includes a trainable Transformer decoder, trainable local temporal modules and a pretrained, fixed image backbone (CLIP is used for instance).

Using a fixed backbone significantly saves training time, and we managed to train a ViT-B/16 with 8 frames for 50 epochs in 60 GPU-hours (NVIDIA V100).

Despite with a small training computation and memory consumption, EVL models achieves high performance on Kinetics-400. A comparison with state-of-the-art methods are as follows

Installation

We tested the released code with the following conda environment

conda create -n pt1.9.0cu11.1_official -c pytorch -c conda-forge pytorch=1.9.0=py3.9_cuda11.1_cudnn8.0.5_0 cudatoolkit torchvision av

Data Preparation

We expect that --train_list_path and --val_list_path command line arguments to be a data list file of the following format

<path_1> <label_1>
<path_2> <label_2>
...
<path_n> <label_n>

where <path_i> points to a video file, and <label_i> is an integer between 0 and num_classes - 1. --num_classes should also be specified in the command line argument.

Additionally, <path_i> might be a relative path when --data_root is specified, and the actual path will be relative to the path passed as --data_root.

The class mappings in the open-source weights are provided at Kinetics-400 class mappings

Backbone Preparation

CLIP weights need to be downloaded from CLIP official repo and passed to the --backbone_path command line argument.

Script Usage

Training and evaliation scripts are provided in the scripts folder. Scripts should be ready to run once the environment is setup and --backbone_path, --train_list_path and --val_list_path are replaced with your own paths.

For other command line arguments please see the help message for usage.

Kinetics-400 Main Results

This is a re-implementation for open-source use. We are still re-running some models, and their scripts, weights and logs will be released later. In the following table we report the re-run accuracy, which may be slightly different from the original paper (typically +/-0.1%)

Backbone	Decoder Layers	#frames x stride	top-1	top-5	Script	Model	Log
ViT-B/16	4	8 x 16	82.8	95.8	script	google drive	google drive
ViT-B/16	4	16 x 16	83.7	96.2	script	google drive	google drive
ViT-B/16	4	32 x 8	84.3	96.6	script	google drive	google drive
ViT-L/14	4	8 x 16	86.3	97.2	script	google drive	google drive
ViT-L/14	4	16 x 16	86.9	97.4	script	google drive	google drive
ViT-L/14	4	32 x 8	87.7	97.6	script	google drive	google drive
ViT-L/14 (336px)	4	32 x 8	87.7	97.8

Data Loading Speed

As the training process is fast, video frames are consumed at a very high rate. For easier installation, the current version uses PyTorch-builtin data loaders. They are not very efficient and can become a bottleneck when using ViT-B as backbones. We provide a --dummy_dataset option to bypass actual video decoding for training speed measurement. The model accuracy should not be affected. Our internal data loader is pure C++-based and does not bottleneck training by much on a machine with 2x Xeon Gold 6148 CPUs and 4x V100 GPUs.

Acknowledgements

The data loader code is modified from PySlowFast. Thanks for their awesome work!

efficient-video-recognition's Issues

Could you share k600/k700 performance? (EOM)

Visualization of cross-frame attention maps

Hi,

Congratulations on your awesome work. If possible, could you also share the script for visualizing cross-frame attention maps (fig. 6 in the paper)?

Thanks in advance.

fix DDP mode saving checkpoint

in the main.py file 193 line should be changed to
if (i + 1) % args.save_freq == 0 and dist.get_rank() == 0:

only master node save checkpoint saving function

Does action recognition not require the start time and end time?

Hi, I have a question as a beginner for action recognition task.

My question is same with title, does action recognition not use start time(frame) and end time information?

I confirmed kinetics400 dataset has time information that contains when the action starts and ends.

If there is many action in one video file, I think this time information is needed, but in this repo, I understand EVL only requires the label of video, not include time information.

Could you tell me about this?

Thank you!

Is it possible that the Frozen CLIP model saw the validation set of Kinetics

Hi,

Congratulations on your awesome work. It is interesting and meaningful.

I have a little concern about this work. I was wondering if it is possible that the dataset on which the CLIP is pre-trained contains part of Kinetics validation video/images.

Thanks in advance.

any plans for adding more comments?

Thanks a lot for open-sourcing this project!

I would like to mention that adding more comments to the codebase could help a lot of people to understand what is going on!

Missing running scripts

Great works! But where is the 'scripts ' floder?

Enquiry on the accessibility of SSV2 configs and checkpoints

Hi, I just wonder if the configs and checkpoints for SSV2 will be available in the future.

bug

Hello！When I run this code, it shows：
AttributeError: 'dict' object has no attribute 'visual'
Traceback (most recent call last):
File "main.py", line 229, in
if name == 'main': main()
File "main.py", line 114, in main
num_frames=args.num_frames,
File "/home/wyl/efficient-video-recognition-master/model.py", line 170, in init
backbone_config = self._create_backbone(backbone_name, backbone_type, backbone_path, backbone_mode)
File "/home/wyl/efficient-video-recognition-master/model.py", line 202, in _create_backbone
state_dict = weight_loader_fn(backbone_path)
File "/home/wyl/efficient-video-recognition-master/weight_loaders.py", line 13, in load_weights_clip
clip_model = clip_model.visual

Do you konw why？

。

batchsize vs training iterations

I had a couple of very basic questions.

Since you define the training length by number of iterations rather than number of epochs, does that mean increasing the batch size would require decreasing iterations accordingly to stay consistent with the 50 epoch training you report?

Because, from what I understood, 50k iterations at a batchsize of 256 would approximately mean 50 epochs (around 950 iterations/epoch for k400). Am I understanding this correctly or have I missed something?

Also, the number of workers defined is the num of workers per GPU or is it total workers across all?

Thank you in advance.

Ask for finetuning scripts

Hi, could you please provide training scrips for finetuning ? I want to reproduce the results of finetuning reported in your paper.
Thanks!

Using ViT-L-14-336px checkpoint

First of all, congrats on the publication of this paper. And thanks for sharing the code.

I tried using the ViT-L-14-336px model, as you got the best result with this one. However, I have seen that the backbone for this model is not defined:

efficient-video-recognition/vision_transformer.py

Lines 307 to 326 in b282e8b

	vit_presets = {
	'ViT-B/16-lnpre': dict(
	feature_dim=768,
	input_size=(224, 224),
	patch_size=(16, 16),
	num_heads=12,
	num_layers=12,
	mlp_factor=4.0,
	ln_pre=True,
	),
	'ViT-L/14-lnpre': dict(
	feature_dim=1024,
	input_size=(224, 224),
	patch_size=(14, 14),
	num_heads=16,
	num_layers=24,
	mlp_factor=4.0,
	ln_pre=True,
	),
	}

Can you add it? I tried it on my own, but not successfully.

Thanks in advance.

Error when running with finetune backbone

I am getting a strange error when trying to use the finetune backbone option.

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [128, 197, 768]], which is output 0 of torch::autograd::CopySlices, is at version 3; expected version 0 instead.

Do you have any idea why this might be happening?

Talal

Saved checkpoints corrupted?

I used your code as it is to train a VitB16 model. But when I try to load the saved checkpoints, they all give the same error:

"PytorchStreamReader failed reading zip archive: invalid header or archive is corrupted"

Tried to inspect the checkpoint file using pickletools, but got the following error in each case.

"UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 63: ordinal not in range(128)"

Is this something you encountered as well?

when the checkpoints and scripts are is open for us?

Great works! when the checkpoints and scripts are is open for us? Thanks very much!

About params config

Hi, thanks for your good work!
My GPUs are limited, only 4 2080Ti.
I set the batchsize=48, lr = 1e-4, backbone is ViT-L-14. Can I get an acceptable result?
Could you give me some suggestions about the params config?

accuracy of vitB16 and vitL_14

I'm very interested in your work and reproduced it. I used the scripts statements you gave , but the reproduction results on the k400 dataset are a bit lower than the results in the paper . I got the accuracy: vitB16_8f:78.87% (82.17%),vitL14_8f:84.76% (86.3%) . I used 4 rtx3090s , the environment is also configured according to the statement in the github. Can you give me some suggestions？

Wprev and Wnext in the formula of the cross-attention part of the paper don't seem to make sense

Hello, how should I understand Yatten in the formula of the cross-attention part of the paper, where Aprev and Anext is already a learnable tensor used to capture optical flow, and Wprev and Wnext is a re-learned feature, which will wouldn't make sense in my opinion

train error

[17:29:39.025738] Not resuming from a checkpoint.
Traceback (most recent call last):
File "main.py", line 238, in
main()
File "main.py", line 148, in main
for i, (data, labels) in enumerate(train_loader, resume_step):
File "/root/miniconda3/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 521, in next
data = self._next_data()
File "/root/miniconda3/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1203, in _next_data
return self._process_data(data)
File "/root/miniconda3/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1229, in _process_data
data.reraise()
File "/root/miniconda3/lib/python3.8/site-packages/torch/_utils.py", line 425, in reraise
raise self.exc_type(msg)
File "av/error.pyx", line 78, in av.error.FFmpegError.init
TypeError: init() takes at least 3 positional arguments (2 given)

How to use the CLIP backbone with other dataset?

I am trying to use the CLIP backbone with the UCF crime dataset. How should I proceed with that?

ssv2 data sampling

Thanks for your work. But when I experimented on SSv2 dataset, as described in the paper that using the TSN style to sample frames, the sampling rate is set to -1, which will perform wrong sampling result in '_generate_temporal_crops'.

Could you kindly provide the ssv2 runable scripts?