opengvlab/efficient-video-recognition

efficient-video-recognition's Issues

Saved checkpoints corrupted?

I used your code as it is to train a VitB16 model. But when I try to load the saved checkpoints, they all give the same error:

"PytorchStreamReader failed reading zip archive: invalid header or archive is corrupted"

Tried to inspect the checkpoint file using pickletools, but got the following error in each case.

"UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 63: ordinal not in range(128)"

Is this something you encountered as well?

Error when running with finetune backbone

I am getting a strange error when trying to use the finetune backbone option.

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [128, 197, 768]], which is output 0 of torch::autograd::CopySlices, is at version 3; expected version 0 instead.

Do you have any idea why this might be happening?

Talal

batchsize vs training iterations

I had a couple of very basic questions.

Since you define the training length by number of iterations rather than number of epochs, does that mean increasing the batch size would require decreasing iterations accordingly to stay consistent with the 50 epoch training you report?

Because, from what I understood, 50k iterations at a batchsize of 256 would approximately mean 50 epochs (around 950 iterations/epoch for k400). Am I understanding this correctly or have I missed something?

Also, the number of workers defined is the num of workers per GPU or is it total workers across all?

Thank you in advance.

。

Does action recognition not require the start time and end time?

Hi, I have a question as a beginner for action recognition task.

My question is same with title, does action recognition not use start time(frame) and end time information?

I confirmed kinetics400 dataset has time information that contains when the action starts and ends.

If there is many action in one video file, I think this time information is needed, but in this repo, I understand EVL only requires the label of video, not include time information.

Could you tell me about this?

Thank you!

bug

Hello！When I run this code, it shows：
AttributeError: 'dict' object has no attribute 'visual'
Traceback (most recent call last):
File "main.py", line 229, in
if name == 'main': main()
File "main.py", line 114, in main
num_frames=args.num_frames,
File "/home/wyl/efficient-video-recognition-master/model.py", line 170, in init
backbone_config = self._create_backbone(backbone_name, backbone_type, backbone_path, backbone_mode)
File "/home/wyl/efficient-video-recognition-master/model.py", line 202, in _create_backbone
state_dict = weight_loader_fn(backbone_path)
File "/home/wyl/efficient-video-recognition-master/weight_loaders.py", line 13, in load_weights_clip
clip_model = clip_model.visual

Do you konw why？

Enquiry on the accessibility of SSV2 configs and checkpoints

Hi, I just wonder if the configs and checkpoints for SSV2 will be available in the future.

Wprev and Wnext in the formula of the cross-attention part of the paper don't seem to make sense

Hello, how should I understand Yatten in the formula of the cross-attention part of the paper, where Aprev and Anext is already a learnable tensor used to capture optical flow, and Wprev and Wnext is a re-learned feature, which will wouldn't make sense in my opinion

train error

[17:29:39.025738] Not resuming from a checkpoint.
Traceback (most recent call last):
File "main.py", line 238, in
main()
File "main.py", line 148, in main
for i, (data, labels) in enumerate(train_loader, resume_step):
File "/root/miniconda3/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 521, in next
data = self._next_data()
File "/root/miniconda3/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1203, in _next_data
return self._process_data(data)
File "/root/miniconda3/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1229, in _process_data
data.reraise()
File "/root/miniconda3/lib/python3.8/site-packages/torch/_utils.py", line 425, in reraise
raise self.exc_type(msg)
File "av/error.pyx", line 78, in av.error.FFmpegError.init
TypeError: init() takes at least 3 positional arguments (2 given)

Is it possible that the Frozen CLIP model saw the validation set of Kinetics

Hi,

Congratulations on your awesome work. It is interesting and meaningful.

I have a little concern about this work. I was wondering if it is possible that the dataset on which the CLIP is pre-trained contains part of Kinetics validation video/images.

Thanks in advance.

Ask for finetuning scripts

Hi, could you please provide training scrips for finetuning ? I want to reproduce the results of finetuning reported in your paper.
Thanks!

Using ViT-L-14-336px checkpoint

First of all, congrats on the publication of this paper. And thanks for sharing the code.

I tried using the ViT-L-14-336px model, as you got the best result with this one. However, I have seen that the backbone for this model is not defined:

efficient-video-recognition/vision_transformer.py

Lines 307 to 326 in b282e8b

	vit_presets = {
	'ViT-B/16-lnpre': dict(
	feature_dim=768,
	input_size=(224, 224),
	patch_size=(16, 16),
	num_heads=12,
	num_layers=12,
	mlp_factor=4.0,
	ln_pre=True,
	),
	'ViT-L/14-lnpre': dict(
	feature_dim=1024,
	input_size=(224, 224),
	patch_size=(14, 14),
	num_heads=16,
	num_layers=24,
	mlp_factor=4.0,
	ln_pre=True,
	),
	}

Can you add it? I tried it on my own, but not successfully.

Thanks in advance.

accuracy of vitB16 and vitL_14

I'm very interested in your work and reproduced it. I used the scripts statements you gave , but the reproduction results on the k400 dataset are a bit lower than the results in the paper . I got the accuracy: vitB16_8f:78.87% (82.17%),vitL14_8f:84.76% (86.3%) . I used 4 rtx3090s , the environment is also configured according to the statement in the github. Can you give me some suggestions？

Missing running scripts

Great works! But where is the 'scripts ' floder?

About params config

Hi, thanks for your good work!
My GPUs are limited, only 4 2080Ti.
I set the batchsize=48, lr = 1e-4, backbone is ViT-L-14. Can I get an acceptable result?
Could you give me some suggestions about the params config?

Could you share k600/k700 performance? (EOM)

fix DDP mode saving checkpoint

in the main.py file 193 line should be changed to
if (i + 1) % args.save_freq == 0 and dist.get_rank() == 0:

only master node save checkpoint saving function

any plans for adding more comments?

Thanks a lot for open-sourcing this project!

I would like to mention that adding more comments to the codebase could help a lot of people to understand what is going on!

Visualization of cross-frame attention maps

Hi,

Congratulations on your awesome work. If possible, could you also share the script for visualizing cross-frame attention maps (fig. 6 in the paper)?

Thanks in advance.

How to use the CLIP backbone with other dataset?

I am trying to use the CLIP backbone with the UCF crime dataset. How should I proceed with that?

ssv2 data sampling

Thanks for your work. But when I experimented on SSv2 dataset, as described in the paper that using the TSN style to sample frames, the sampling rate is set to -1, which will perform wrong sampling result in '_generate_temporal_crops'.

Could you kindly provide the ssv2 runable scripts?

when the checkpoints and scripts are is open for us?

Great works! when the checkpoints and scripts are is open for us? Thanks very much!