arrowluo/videofeatureextractor

Video Feature Extractor for S3D-HowTo100M

This repo is forked from video_feature_extractor to extract S3D feature (S3D_HowTo100M) pretraied on HowTo100M. Read more details in video_feature_extractor.

This repo is also as a preprocess in video-language pretrain model UniVL.

Requirements

IMPORTANT: The video decode process depends on the FFmpeg (https://www.ffmpeg.org/download.html), install it first and run ffmpeg and ffprobe command derectly to make them work well.

• Python 3
• PyTorch (>= 1.0)
• python-ffmpeg (https://github.com/kkroening/ffmpeg-python)

Downloading pretrained models

This will download the pretrained S3D model:

mkdir -p model
cd model
wget https://www.rocq.inria.fr/cluster-willow/amiech/howto100m/s3d_howto100m.pth
cd ..

Extract S3D Feature

First of all you need to generate a csv containing the list of videos you want to process. For instance, if you have absolute_path_video1.mp4 and absolute_path_video2.webm to process, you will need to generate a csv of this form:

video_path,feature_path
absolute_path_video1.mp4,absolute_path_of_video1_features.npy
absolute_path_video2.webm,absolute_path_of_video2_features.npy

Refer to below command to generate such a csv file:

python preprocess_generate_csv.py --csv=input.csv --video_root_path [VIDEO_PATH] --feature_root_path [FEATURE_PATH] --csv_save_path .

Note: the video file should have a suffix, modify the code for your customization

And then just simply run:

python extract.py --csv=./input.csv --type=s3dg --batch_size=64 --num_decoding_thread=4

This command will extract s3d-g video feature in a form of a numpy array.

If you want to pickle all generated npy files:

python convert_video_feature_to_pickle.py --feature_root_path [FEATURE_PATH] --pickle_root_path . --pickle_name input.pickle

The key is set as the video name in the pickle file

Acknowledgements

The code re-used code from https://github.com/kenshohara/3D-ResNets-PyTorch for 3D CNN. And modified from https://github.com/antoine77340/video_feature_extractor.

videofeatureextractor's Issues

Error when run extract.py

thank you for your work！
when i run the extract.py in command
`python extract.py --csv=./input.csv --type=s3dg --batch_size=64 --num_decoding_thread=1
D:\Github\VideoFeatureExtractor
Loading S3DG ...
Weights from pretrained model not used in S3D:
text_module.word_embd.weight
text_module.fc1.weight
text_module.fc1.bias
text_module.fc2.weight
text_module.fc2.bias
loaded
D:\Github\VideoFeatureExtractor
Decoding video: D:/Github/VideoFeatureExtractor/data/1.mp4
ffprobe failed at: D:/Github/VideoFeatureExtractor/data/1.mp4
Video D:/Github/VideoFeatureExtractor/data/1.mp4 already processed.Decoding video: D:/Github/VideoFeatureExtractor/data/2.mp4

ffprobe failed at: D:/Github/VideoFeatureExtractor/data/2.mp4
Video D:/Github/VideoFeatureExtractor/data/2.mp4 already processed.`
Could you check with it?

Can I use multiple GPU to speed up feature extraction?

@ArrowLuo
Hi! As you mentioned in the README,

And modified from https://github.com/antoine77340/video_feature_extractor.

Can I use multiple GPU to speed up feature extraction, just like https://github.com/antoine77340/video_feature_extractor#can-i-use-multiple-gpu-to-speed-up-feature-extraction-?

CUDA_VISIBLE_DEVICES=0 python extract.py --csv=./input.csv --type=s3dg --batch_size=64 --num_decoding_thread=4
CUDA_VISIBLE_DEVICES=1 python extract.py --csv=./input.csv --type=s3dg --batch_size=64 --num_decoding_thread=4
CUDA_VISIBLE_DEVICES=2 python extract.py --csv=./input.csv --type=s3dg --batch_size=64 --num_decoding_thread=4
CUDA_VISIBLE_DEVICES=3 python extract.py --csv=./input.csv --type=s3dg --batch_size=64 --num_decoding_thread=4

What is the difference between this processed output and the official S3D features

Hello,

Thanks for the great work. Can I know what is the difference between this processed output and the official S3D features (https://www.di.ens.fr/willow/research/howto100m/)?

Thanks