haewonc/ervsr

[WACV 23] Efficient Reference-based Video Super-Resolution (ERVSR)

WACV23: ERVSR

Efficient Reference-based Video Super-Resolution (ERVSR)

Framework Overview

Results

Minimal test

You can simply check the PSNR and visualized results of exemplary data without any set-ups via minimal_test.ipynb.

Setting

1. Clone this repository

git clone https://github.com/haewonc/ERVSR.git

2. Match the dependencies following RefVSR.
3. Download the RealMCVSR dataset.
4. Download the pretrained SPyNet and place inside the directory net/.

Test

You can reproduce the results in our paper with commands below. The testing with default configuration file should work on a machine with GPU memory larger than 6GB.

1. Download our pretrained ERVSR model and place inside the directory results/saved_models/.
2. Run test.

python test.py --model results/saved_models/final.pth --data_offset $DATA_DIRECTORY$ --save_img True --save_dir results/test_imgs

Train

You can train ERVSR model with the command below. The training with default configuration file should work on a machine with NVIDIA RTX TITAN (Memory: 24GB).

python train.py -b $BATCH_SIZE$ -data_offset $DATA_DIRECTORY$

ervsr's Issues

Welcome update to OpenMMLab 2.0

Welcome update to OpenMMLab 2.0

I am Vansin, the technical operator of OpenMMLab. In September of last year, we announced the release of OpenMMLab 2.0 at the World Artificial Intelligence Conference in Shanghai. We invite you to upgrade your algorithm library to OpenMMLab 2.0 using MMEngine, which can be used for both research and commercial purposes. If you have any questions, please feel free to join us on the OpenMMLab Discord at https://discord.gg/A9dCpjHPfE or add me on WeChat (ID: van-sin) and I will invite you to the OpenMMLab WeChat group.

Here are the OpenMMLab 2.0 repos branches:

	OpenMMLab 1.0 branch	OpenMMLab 2.0 branch
MMEngine		0.x
MMCV	1.x	2.x
MMDetection	0.x 、1.x、2.x	3.x
MMAction2	0.x	1.x
MMClassification	0.x	1.x
MMSegmentation	0.x	1.x
MMDetection3D	0.x	1.x
MMEditing	0.x	1.x
MMPose	0.x	1.x
MMDeploy	0.x	1.x
MMTracking	0.x	1.x
MMOCR	0.x	1.x
MMRazor	0.x	1.x
MMSelfSup	0.x	1.x
MMRotate	0.x	1.x
MMYOLO		0.x

Attention: please create a new virtual environment for OpenMMLab 2.0.

Code error

attention calculation

attn = (q @ k.transpose(-2, -1)) * self.temperature

input LR is 64， Ref is 128，
So，the shape of q is torch.Size([2, 2, 32, 16384])
the shape of k.transpose(-2, -1) is torch.Size([2, 2, 4096, 32])，dimension mismatch results in an error.