btkforever/video_restoration

Video Restoration Toolbox including FGST (ICML 2022), S2SVR (ICML 2022), etc.

A Toolbox for Video Restoration

Authors

Jing Lin*, Yuanhao Cai*, Xiaowan Hu, Haoqian Wang, Youliang Yan, Xueyi Zou, Henghui Ding, Yulun Zhang, Radu Timofte, and Luc Van Gool

News

• 2022.08.05 : Pretrained model of FGST on GOPRO dataset is released. 🚀
• 2022.05.14 : Our FGST and S2SVR are accepted by ICML2022. 🚀

Super-Resolution	Deblur	Compressed Video Enhancement

Papers

• Flow-Guided Sparse Transformer for Video Deblurring (ICML 2022)
• Unsupervised Flow-Aligned Sequence-to-Sequence Learning for Video Restoration (ICML 2022)

1. Create Environment:

• Python 3 (Recommend to use Anaconda)

• NVIDIA GPU + CUDA

• Python packages:

pip install torchvision==0.9.0  torch==1.8.0  torchaudio==0.8.0
pip install -r requirements.txt
pip install openmim
mim install mmcv-full
pip install -v -e .
pip install cupy-cuda101==7.7.0

2. Prepare Dataset:

Download the datasets (GOPRO,DVD,REDS,VIMEO,MFQE-v2) and and recollect them as the following form:

|--VR-Baseline
    |--data
    	|-- GoPro
    	    |-- test
    	    |-- train
    	|-- DVD
    	    |-- test
    	    |-- train
    	|-- REDS
    	    |-- test
    	    |-- train
    	|-- VIMEO
    	    |-- test
    	    |-- train
    	|-- MFQEV2
    	    |-- test
    	    |-- train

3. Training:

cd VR_Baseline

# training FGST
bash tools/dist_train.sh configs/FGST_deblur_gopro.py 8

# training S2SVR
bash tools/dist_train.sh configs/S2SVR_deblur_gopro.py 8

The training log, trained model will be available in VR-Baseline/experiments/ .

4. Testing:

Download pretrained model and run the following command.

python demo/restoration_video_demo.py ${CONFIG} ${CHKPT} ${IN_PATH} ${OUT_PATH}

5. TODO

These works are mostly done during the internship at HUAWEI Noah's Ark Lab. Due to the limitation of company regulations, the original pre-trained models can not be transferred and published here. We will retrain more models and open-source them when we have enough GPUs as soon as possible.

• More Pretrained Models
• Inference Results

6. Acknowledgement.

We refer to codes from BasicVSR++ and mmediting. Thanks for their awesome works.

7. Citation

If this repo helps you, please consider citing our works:

# FGST
@inproceedings{fgst,
  title={Flow-Guided Sparse Transformer for Video Deblurring},
  author={Lin, Jing and Cai, Yuanhao and Hu, Xiaowan and Wang, Haoqian and Yan, Youliang and Zou, Xueyi and Ding, Henghui and Zhang, Yulun and Timofte, Radu and Van Gool, Luc},
  booktitle={ICML},
  year={2022}
}


# S2SVR
@inproceedings{seq2seq,
  title={Unsupervised Flow-Aligned Sequence-to-Sequence Learning for Video Restoration},
  author={Lin, Jing  and Hu, Xiaowan and Cai, Yuanhao and Wang, Haoqian and Yan, Youliang and Zou, Xueyi and Zhang, Yulun and Van Gool, Luc},
  booktitle={ICML},
  year={2022}
}