thmen/egvsr

Efficient & Generic Video Super-Resolution

EGVSR-PyTorch

GitHub | Gitee码云

VSR x4: EGVSR; Upscale x4: Bicubic Interpolation

Contents

• EGVSR-PyTorch
  • Contents
  • Introduction
  • Features
  • Dependencies
  • Datasets
    • A. Training Dataset
    • B. Testing Datasets
  • Benchmarks
    • A. Test on Vid4 Dataset
    • B. Test on Tos3 Dataset
    • C. Test on Gvt72 Dataset
    • D. Optical-Flow based Motion Compensation
    • E. Comprehensive Performance
  • License & Citations
  • Acknowledgements

Introduction

This is a PyTorch implementation of EGVSR: Efficcient & Generic Video Super-Resolution (VSR), using subpixel convolution to optimize the inference speed of TecoGAN VSR model. Please refer to the official implementation ESPCN and TecoGAN for more information.

Features

• Unified Framework: This repo provides a unified framework for various state-of-the-art DL-based VSR methods, such as VESPCN, SOFVSR, FRVSR, TecoGAN and our EGVSR.
• Multiple Test Datasets: This repo offers three types of video datasets for testing, i.e., standard test dataset -- Vid4, Tos3 used in TecoGAN and our new dataset -- Gvt72 (selected from Vimeo site and including more scenes).
• Better Performance: This repo provides model with faster inferencing speed and better overall performance than prior methods. See more details in Benchmarks section.

Dependencies

• Ubuntu >= 16.04
• NVIDIA GPU + CUDA & CUDNN
• Python 3
• PyTorch >= 1.0.0
• Python packages: numpy, matplotlib, opencv-python, pyyaml, lmdb (requirements.txt & req.txt)
• (Optional) Matlab >= R2016b

Datasets

A. Training Dataset

Download the official training dataset based on the instructions in TecoGAN-TensorFlow, rename to VimeoTecoGAN and then place under ./data.

B. Testing Datasets

• Vid4 -- Four video sequences: city, calendar, foliage and walk;
• Tos3 -- Three video sequences: bridge, face and room;
• Gvt72 -- Generic VSR Test Dataset: 72 video sequences (including natural scenery, culture scenery, streetscape scene, life record, sports photography, etc, as shown below)

You can get them at ⏬ 百度网盘 (提取码:8tqc) and put them into 📁 Datasets. The following shows the structure of the above three datasets.

data
  ├─ Vid4
    ├─ GT                # Ground-Truth (GT) video sequences
      └─ calendar
        ├─ 0001.png
        └─ ...
    ├─ Gaussian4xLR      # Low Resolution (LR) video sequences in gaussian degradation and x4 down-sampling
      └─ calendar
        ├─ 0001.png
        └─ ...
  └─ ToS3
    ├─ GT
    └─ Gaussian4xLR
  └─ Gvt72
    ├─ GT
    └─ Gaussian4xLR

Benchmarks

Experimental Environment

	Version	Info.
System	Ubuntu 18.04.5 LTS	X86_64
CPU	Intel i9-9900	3.10GHz
GPU	Nvidia RTX 2080Ti	11GB GDDR6
Memory	DDR4 2666	32GB×2

A. Test on Vid4 Dataset

1.LR 2.VESPCN 3.SOFVSR 4.DUF 5.Ours:EGVSR 6.GT
Objective metrics for visual quality evaluation[1]

B. Test on Tos3 Dataset

1.VESPCN 2.SOFVSR 3. FRVSR 4.TecoGAN 5.Ours:EGVSR 6.GT

C. Test on Gvt72 Dataset

1.LR 2.VESPCN 3.SOFVSR 4.DUF 5.Ours:EGVSR 6.GT
Objective metrics for visual quality and temporal coherence evaluation[1]

D. Optical-Flow based Motion Compensation

Please refer to FLOW_walk, FLOW_foliage and FLOW_city.

E. Comprehensive Performance

Comparison of various SOTA VSR model on video quality score and speed performance[3]

^[1] ⬇️:smaller value for better performance, ⬆️: on the contrary; Red: stands for Top1, Blue: Top2. ^[2] The calculation formula of video quality score considering both spatial and temporal domain, using lambda1=lambda2=lambda3=1/3. ^[3] FLOPs & speed are computed on RGB with resolution 960x540 to 3840x2160 (4K) on NVIDIA GeForce GTX 2080Ti GPU.

License & Citations

This EGVSR project is released under the MIT license. See more details in LICENSE. The provided implementation is strictly for academic purposes only. If EGVSR helps your research or work, please consider citing EGVSR. The following is a BibTeX reference:

@misc{thmen2021egvsr,
  author =       {Yanpeng Cao, Chengcheng Wang, Changjun Song, Yongming Tang and He Li},
  title =        {EGVSR},
  howpublished = {\url{https://github.com/Thmen/EGVSR}},
  year =         {2021}
}

Yanpeng Cao, Chengcheng Wang, Changjun Song, Yongming Tang and He Li. EGVSR. https://github.com/Thmen/EGVSR, 2021.

Acknowledgements

This code is built on the following projects. We thank the authors for sharing their codes.

1. ESPCN
2. BasicSR
3. VideoSuperResolution
4. TecoGAN-PyTorch

egvsr's Issues

colab demo

do you have a colab demo?

Results in MSU Video Super Resolution Benchmark

Hello,
MSU Graphics & Media Lab Video Group has recently launched Video Super Resolution Benchmark.
We provide a comprehensive comparison of a variety of different VSR models.
You can see our current results: MSU VSR Benchmark.
If you are interested in participating, you can add your algorithm, following the submission steps.
You can read about MSU Graphics & Media Lab Video Group here and here.

Generator model of EGVSR

I don't understand what the symbol (+) is trying to get the resized residual and the SRNET output done.
When implementing on egvsr_nets.py, there is no clue about performing these things.
Can you explain the flow of the model on details?

Do you only get a good result if you have an HQ video unlike RealESRGAN?

Do you only get a good result if you have an HQ video?

I can upscale videos if I have the hq and lq version.
But how can I upscale a video without hq version?

So far I've done it with RealESRGAN, so it's always worked well.
Did I understand it correctly that TecoGan can do it only better, or did I get it vill. wrong?

Missing Resources : B. Testing Datasets

You can get them at 百度网盘 (提取码:8tqc)

Ah oh, you are late, and the shared files have been deleted. Please be earlier next time.

Got extremely low score with Vid4

I tested your code with Vid4 using your Model EGVSR and the pretrained weight EGVSR_iter420000.pth but got extremely low score. And the generated images are really blurry.Why does it happen?

obvious artifact in the last frame of foliage

I used Vid4/BDx4/foliage to test egvsr. My cmd line is as below

python codes/main.py --exp_dir ./experiments_BD/EGVSR/001 --mode test --opt test.yml --gpu_id 0 --model TecoGAN

The issue is obvious artifact in the last frame of foliage (please check attachement, in the tree area

)

Practical usability

It's great that you provide elaborate ways to compare this method against previous models on known datasets, but from a practical usability standpoint those script are not very useful...

It would be amazing if you could provide a sample script like
upsample.py --input.mp4 that simply upscales an input video...

how to run the test program

I put some images into dir /data/Vid4/GT for test, but I don't know what's the mean of "data/Vid4/Gaussian4xLR",

python codes/main.py --exp_dir test --mode test --model EGVSR --opt /opt/EGVSR-master/experiments_BD/EGVSR/001/test.yml

2021-07-22 11:01:23,123 [INFO]: ========================================
2021-07-22 11:01:23,123 [INFO]: Testing model: EGVSR_iter420000
2021-07-22 11:01:23,123 [INFO]: ========================================
2021-07-22 11:01:24,793 [INFO]: Testing on test1: Vid4
Traceback (most recent call last):
File "codes/main.py", line 341, in
test(opt)
File "codes/main.py", line 182, in test
test_loader = create_dataloader(opt, dataset_idx=dataset_idx)
File "/opt/daizhsh/EGVSR-master/codes/data/init.py", line 58, in create_dataloader
dataset = PairedFolderDataset(data_opt, scale=opt['scale'])
File "/opt/daizhsh/EGVSR-master/codes/data/paired_folder_dataset.py", line 21, in init
lr_keys = sorted(os.listdir(self.lr_seq_dir))
FileNotFoundError: [Errno 2] No such file or directory: 'data/Vid4/Gaussian4xLR'

Thank you for you reply!

请问训练需要的数据集：data/VimeoTecoGAN/GT.lmdb 怎么得到

似乎没有制作 lmdb 的脚本？

270p到1080p速度16FPS比论文声称低很多，而且伪影有些大

https://www.bilibili.com/video/BV1rb4y1B7t1/
270p到1080p速度16FPS比论文声称低很多，而且伪影有些大，不知道哪里有问题，还是本身效果实际如此．机器配置RTX2080.

[Download Testset]

Hi :)

Can you please give me more information about how to download the testsets?
I'm not able to understand the language of the website you provided!

Thank you in advice!

Run profile.sh failed

bash profile.sh BI TecoGAN 3x540x960
Traceback (most recent call last):
  File "./codes/main.py", line 13, in <module>
    from metrics.metric_calculator import MetricCalculator
  File "/home/tianqing/pyprojects/EGVSR/codes/metrics/metric_calculator.py", line 11, in <module>
    from .LPIPS.models.dist_model import DistModel
  File "/home/tianqing/pyprojects/EGVSR/codes/metrics/LPIPS/models/__init__.py", line 11, in <module>
    from metrics.LPIPS.models import dist_model
  File "/home/tianqing/pyprojects/EGVSR/codes/metrics/LPIPS/models/dist_model.py", line 21, in <module>
    from . import networks_basic as networks
  File "/home/tianqing/pyprojects/EGVSR/codes/metrics/LPIPS/models/networks_basic.py", line 15, in <module>
    import metrics.LPIPS.models as util
AttributeError: module 'metrics.LPIPS' has no attribute 'models'

Testing Datasets

The link has expired, can you give me a new download address

bash test.sh

Bash test How should I write the parameters after sh-- exp_ dir ./ experiments_$ {degradation}/${model}/${exp_id}
--mode test
--model ${model}
--opt test. yml
--gpu_ id ${gpu_id}
Can you give me an example? There is no problem with my path. It keeps reminding me that I can't find the file

Your Results in New Super-Resolution Benchmarks

Hello,

MSU Graphics & Media Lab Video Group has recently launched two new Super-Resolution Benchmarks.

• Video Upscalers Benchmark: Quality Enhancement determines the best upscaling methods for increasing video resolution and improving visual quality.
• Super-Resolution for Video Compression benchmark aims to test Super-Resolution methods on compressed videos and select the best model for each video codec standard.

Your method achieved 12th place in Super-Resolution for Video Compression Benchmark in 'x265 compression' category. We look forward to your future work!

We would be grateful for your feedback on our work.

How to train in 2x factor?

I want to scale 1920x1080 video to 3840x2160, do I need change the codes?

TrainDatasets

Due to network limitations in mainland China, we cannot connect to vimeo.com to download the training set, so please provide a link to the VimeoTecoGAN website.Thank you ！！

(open-mmlab) smartcity@smartcity-X780-G30:/media/smartcity/E6AA1145AA1113A1/CaiFeifan/TecoGAN$ python3 dataPrepare.py --start_id 2000 --duration 120 --REMOVE --disk_path /media/smartcity/E6AA1145AA1113A1/CaiFeifan/datasets/VimeoTecoGAN/
[Configurations]:
start_id: 2000
duration: 120
disk_path: /media/smartcity/E6AA1145AA1113A1/CaiFeifan/datasets/VimeoTecoGAN/
summary_dir: /media/smartcity/E6AA1145AA1113A1/CaiFeifan/datasets/VimeoTecoGAN/log/
REMOVE: True
TEST: False
End of configuration
Try loading 308x120.
https://vimeo.com/121649159
[vimeo] 121649159: Downloading webpage
ERROR: Unable to download webpage: <urlopen error [Errno 110] Connection timed out> (caused by URLError(TimeoutError(110, 'Connection timed out')))
youtube_dl error:https://vimeo.com/121649159
Skipped invalid link or other error:https://vimeo.com/121649159
https://vimeo.com/40439273
[vimeo] 40439273: Downloading webpage
ERROR: Unable to download webpage: <urlopen error [Errno 110] Connection timed out> (caused by URLError(TimeoutError(110, 'Connection timed out')))
youtube_dl error:https://vimeo.com/40439273
Skipped invalid link or other error:https://vimeo.com/40439273
https://vimeo.com/87389090
[vimeo] 87389090: Downloading webpage
ERROR: Unable to download webpage: <urlopen error [Errno 110] Connection timed out> (caused by URLError(TimeoutError(110, 'Connection timed out')))
youtube_dl error:https://vimeo.com/87389090
Skipped invalid link or other error:https://vimeo.com/87389090
https://vimeo.com/335874600
[vimeo] 335874600: Downloading webpage
ERROR: Unable to download webpage: <urlopen error [Errno 110] Connection timed out> (caused by URLError(TimeoutError(110, 'Connection timed out')))
youtube_dl error:https://vimeo.com/335874600
Skipped invalid link or other error:https://vimeo.com/335874600
https://vimeo.com/114053015
[vimeo] 114053015: Downloading webpage
ERROR: Unable to download webpage: <urlopen error [Errno 110] Connection timed out> (caused by URLError(TimeoutError(110, 'Connection timed out')))
youtube_dl error:https://vimeo.com/114053015
Skipped invalid link or other error:https://vimeo.com/114053015
https://vimeo.com/160578133
[vimeo] 160578133: Downloading webpage
ERROR: Unable to download webpage: <urlopen error [Errno 110] Connection timed out> (caused by URLError(TimeoutError(110, 'Connection timed out')))
youtube_dl error:https://vimeo.com/160578133
Skipped invalid link or other error:https://vimeo.com/160578133
https://vimeo.com/148058982
[vimeo] 148058982: Downloading webpage
ERROR: Unable to download webpage: <urlopen error [Errno 110] Connection timed out> (caused by URLError(TimeoutError(110, 'Connection timed out')))
youtube_dl error:https://vimeo.com/148058982
Skipped invalid link or other error:https://vimeo.com/148058982
https://vimeo.com/150225201
[vimeo] 150225201: Downloading webpage
ERROR: Unable to download webpage: <urlopen error [Errno 110] Connection timed out> (caused by URLError(TimeoutError(110, 'Connection timed out')))
youtube_dl error:https://vimeo.com/150225201
Skipped invalid link or other error:https://vimeo.com/150225201
https://vimeo.com/145096806

Real-time SR?

Is it possible to use this GAN for real-time video super resolution? Maybe for video conferencing?

The infer speed is much slower than paper said

How to upscale to 4K video with 30fps as Fig.7 said?

Testing Datasets download link is not working

Testing Datasets download link is not working

Why do you choose features after maxpool for perceptual loss?

Some papers use features before relu activation like ESRGAN, why do you use that after relu activation.

metric is NaN

I met this problem while I run the test.sh
root@Ic75ba579f00d014e0:/hy-tmp/EGVSR# bash test.sh
./experiments_BD/EGVSR/001/
2022-07-25 15:10:19,482 [INFO]: ========================================
2022-07-25 15:10:19,482 [INFO]: Testing model: EGVSR_iter420000
2022-07-25 15:10:19,482 [INFO]: ========================================
2022-07-25 15:10:23,526 [INFO]: Testing on test1: Vid4
2022-07-25 15:10:24,329 [INFO]: Average
/usr/local/lib/python3.8/dist-packages/numpy/core/fromnumeric.py:3474: RuntimeWarning: Mean of empty slice.
return _methods._mean(a, axis=axis, dtype=dtype,
/usr/local/lib/python3.8/dist-packages/numpy/core/_methods.py:189: RuntimeWarning: invalid value encountered in double_scalars
ret = ret.dtype.type(ret / rcount)
2022-07-25 15:10:24,331 [INFO]: PSNR: nan (x1.0)
2022-07-25 15:10:24,331 [INFO]: LPIPS: nan (x1.0)
2022-07-25 15:10:24,331 [INFO]: tOF: nan (x1.0)
2022-07-25 15:10:24,331 [INFO]: ----------------------------------------
2022-07-25 15:10:24,331 [INFO]: Testing on test2: ToS3
2022-07-25 15:10:25,094 [INFO]: Average
2022-07-25 15:10:25,095 [INFO]: PSNR: nan (x1.0)
2022-07-25 15:10:25,095 [INFO]: LPIPS: nan (x1.0)
2022-07-25 15:10:25,095 [INFO]: tOF: nan (x1.0)
2022-07-25 15:10:25,095 [INFO]: ----------------------------------------
2022-07-25 15:10:25,095 [INFO]: Finish testing
2022-07-25 15:10:25,095 [INFO]: ========================================

So I should do what and I am troubled

Getting results lower than the paper

Hi. I have tried your code as
bash test.sh "BD" "EGVSR"
I am getting results much lower than the paper"Real-Time Super-Resolution System of 4K-Video Based on Deep Learning" mentioned on the Vid4 Dataset. Here is the screen grab.