sunny2109/safmn

[ICCV 2023] Spatially-Adaptive Feature Modulation for Efficient Image Super-Resolution; runner-up method for the model complexity track in NTIRE2023 Efficient SR challenge

📖 Spatially-Adaptive Feature Modulation for Efficient Image Super-Resolution

[Paper]   [Supp]   [Poster]

Long Sun, Jiangxin Dong, Jinhui Tang, and Jinshan Pan
IMAG Lab, Nanjing University of Science and Technology

---

An overview of the proposed SAFMN. SAFMN first transforms the input LR image into the feature space using a convolutional layer, performs feature extraction using a series of feature mixing modules (FMMs), and then reconstructs these extracted features by an upsampler module. The FMM block is mainly implemented by a spatially-adaptive feature modulation (SAFM) layer and a convolutional channel mixer (CCM).

---

News

• [2024-05-08] Add the light_SAFMN++, which is the 1st place winner of the fidelity track of the Real-time 4K Super Resolution Challenge for compressed AVIF images, and is invited to give an oral presentation at the AIS2024 workshop.
• [2024-03-27] Add the improved ESR model SAFMN++, which ranks Top4 in the Overall Performance track and Runtime track of NTIRE2024 ESR challenge
• [2024-03-16] Add SAFMN_BCIE, which enhances the quality of JPEG images compressed with a large range of quality factors. The corresponding inference code can be found here.
• [2023-11-22] The code for ONNX export is available here
• [2023-09-08] Integrated to 🐼 OpenXLab. Try out online demo!
• [2023-08-31] Update SAFMN_Real_x4.pth
• [2023-08-31] Add SAFMN_Real_x2.pth
• [2023-08-21] Colab demo for SAFMN on x4 real-world image SR is available
• [2023-07-14] Our SAFMN is accepted to ICCV 2023
• [2023-06-06] The report of NTIRE 2023 Challenge on Efficient Super-Resolution is available here
• [2023-04-17] The SAFMN ranks Top6 for overall performance in the NTIRE2023 ESR challenge.
• [2023-04-17] The SAFMN variant ranks Top3 for model complexity in the NTIRE2023 ESR Challenge.
• [2023-03-22] The code and checkpoint for the NTIRE2023 Efficient Super-Resolution Challenge is available here.
• [2023-03-22] The pre-trained model with high-order degradation on the LSDIR dataset is available.
• [2023-03-13] The source codes, checkpoints and visual results are available.
• [2023-02-26] The paper is available here.

---

Requirements

• Python 3.8, PyTorch >= 1.11
• BasicSR 1.4.2
• Platforms: Ubuntu 18.04, cuda-11

Installation

# Clone the repo
git clone https://github.com/sunny2109/SAFMN.git
# Install dependent packages
cd SAFMN
pip install -r requirements.txt
# Install BasicSR
python setup.py develop

You can also refer to this INSTALL.md for installation

Training

Run the following commands for training:

# train SAFMN for x4 effieicnt SR
python basicsr/train.py -opt options/train/SAFMN/train_DF2K_x4.yml
# train SAFMN for x4 classic SR
python basicsr/train.py -opt options/train/SAFMN/train_L_DF2K_x4.yml

Testing

• Download the pretrained models.
• Download the testing dataset.
• Run the following commands:

# test SAFMN for x4 efficient SR
python basicsr/test.py -opt options/test/SAFMN/test_benchmark_x4.yml
# test SAFMN for x4 classic SR
python basicsr/test.py -opt options/test/SAFMN/test_L_benchmark_x4.yml
# test SAFMN for x4 real-world SR (without ground-truth)
python basicsr/test.py -opt options/test/SAFMN/test_real_img_x4.yml
# test SAFMN for x4 real-world SR (large input)
python inference/inference_real_safmn.py --input test_demo --output results/test_demo --scale 4 --large_input 

• The test results will be in './results'.

Results

• Pretrained models and visual results

  We have provided three ways to download our checkpoints.

  • 1.Download via the Google Drive links shown below.
  • 2.Download via the Baidu Netdisk links shown below.
  • 3.Visit our huggingface repo at https://huggingface.co/Meloo/SAFMN/tree/main and click the download icons.

Degradation	Model Zoo	Visual Results
BI-Efficient SR	Google Drive/Baidu Netdisk with code: SAFM	Google Drive/Baidu Netdisk with code: SAFM
BI-Classic SR	Google Drive/Baidu Netdisk with code: SAFM	Google Drive/Baidu Netdisk with code: SAFM
x4 Real-world	Google Drive/Baidu Netdisk with code: SAFM

• Results on Blind Compressed Images

• Results on AIGC Image

• Real-world SR Results

Real-World Image (x4)	Real-ESRGAN	SwinIR	SAFMN (ours)

• Efficient SR Results

• Classic SR Results

Citation

If this work is helpful for your research, please consider citing the following BibTeX entry.

@inproceedings{sun2023safmn,
    title={Spatially-Adaptive Feature Modulation for Efficient Image Super-Resolution},
    author={Sun, Long and Dong, Jiangxin and Tang, Jinhui and Pan, Jinshan},
    booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
    year={2023}
 }

Acknowledgement

This code is based on BasicSR toolbox. Thanks for the awesome work.

Contact

If you have any questions, please feel free to reach me out at [email protected]