xinntao/sftgan

CVPR18 - Recovering Realistic Texture in Image Super-resolution by Deep Spatial Feature Transform

SFTGAN [Paper] [BasicSR]

😃 Training codes are in BasicSR repo.

Recovering Realistic Texture in Image Super-resolution by Deep Spatial Feature Transform

By Xintao Wang, Ke Yu, Chao Dong, Chen Change Loy.

This repo only provides simple testing codes - original torch version used in the paper and a pytorch version. For full training and testing codes, please refer to BasicSR.

BibTeX

@InProceedings{wang2018sftgan,
    author = {Wang, Xintao and Yu, Ke and Dong, Chao and Loy, Chen Change},
    title = {Recovering realistic texture in image super-resolution by deep spatial feature transform},
    booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
    month = {June},
    year = {2018}
}

Table of Contents

1. Quick Test
2. Spatial Feature Modulation
3. Semantic Categorical Prior
4. OST dataset

Quick Test

It provides Torch and PyTorch versions. Recommend the PyTorch version.

PyTorch Dependencies

• Python 3
• PyTorch >= 0.4.0
• Python packages: pip install numpy opencv-python

[OR] Torch Dependencies

• Torch
• Other torch dependencies, e.g. nngraph, paths, image (install them by luarocks install xxx)

Test models

Note that the SFTGAN model is limited to some outdoor scenes. It is an unsatisfying limitation that we need to relax in future.

1. Clone this github repo.

git clone https://github.com/xinntao/SFTGAN
cd SFTGAN

2. There are two sample images in the ./data/samples folder.
3. Download pretrained models from Google Drive or Baidu Drive. Please see model list for more details.
4. First run segmentation test.

[PyTorch]

cd pytorch_test
python test_segmentation.py

[Torch]

cd torch_test
th test_segmentation.lua

The segmentation results are then in ./data with _segprob, _colorimg, _byteimg suffix.

5. Run sftgan test.

[PyTorch]

python test_sftgan.py.

[Torch]

th test_sftgan.lua

The results are in then in ./data with _result suffix.

Spatial Feature Modulation

SFT - Spatial Feature Transform (Modulation).

A Spatial Feature Transform (SFT) layer has been proposed to efficiently incorporate the categorical conditions into a CNN network.

There is a fantastic blog explaining the widely-used feature modulation operation distill - Feature-wise transformations.

Semantic Categorical Prior

We have explored the use of semantic segmentation maps as categorical prior for SR.

OST dataset

• Outdoor Scene Train/Test

OST (Outdoor Scenes),OST Training,7 categories images with rich textures

OST300 300 test images of outdoor scences

Download the OST dataset from Google Drive or Baidu Drive.

😆 Image Viewer - HandyViewer

May try HandyViewer - an image viewer that you can switch image with a fixed zoom ratio, easy for comparing image details.

sftgan's Issues

hi

Hi , i am very interested in your work.

Hi,
Thanks to your very outstanding work, but i just can code in pytorch. Can you give me the pytorch code just for model or SFT layer.
Thanks.
Gundati

Pytorch version run in windows

Hi @xinntao
Thank you very much for your great work on SFTGAN
just want to ask if you plan to make your ( SFTGAN Pytorch version ) run in windows, since Pytorch now officially support windows , it is be great if windows users can used your amazing work, thank you

what's the upscaling factor of SFTGAN model?

why the size of output is same as that of input when I run SFTGAN test code with pretrained models?

Data loading speed of probability map is very slow when training SFTGAN

Hi, xintao.
I find load probability map using torch.load is very slow, leading to very long data loading time.
Have you encountered this issue or my code problem?

I have the following problem when I use my own test image.

RuntimeError: The size of tensor a (236) must match the size of tensor b (238) at non-singleton dimension 3

Evaluate the reconstruction effect of the GAN method

Hi，Xintao. Excuse me,After reading SFTGAN, ESRGAN, and RankGAN papers, i would like to discuss with you about how to evaluate the reconstruction effect of the GAN method.

1)SFTGAN uses the method of user evaluation toevaluate reconstruction effect. This is not as convincing as the objective evaluation criteria, and may be rejected by the reviewers.

2）ESRGAN uses standard test sets to test PSNR and SSIM . And the test results are very high and refreshed. This clearly illustrates the effectiveness of the method used and is more convincing to the reviewer.

3）RankSRGAN uses NIQE and other evaluation metrics that are more suitable for the GAN method.

If I want to use SFTGAN as the baseline (running time considerations), based on the above considerations, should I use the NIQE evaluation method ?
Is subjective evaluation necessary? Are there other evaluation methods?

Best regards.

"manually change the probability map to each category"

What does "manually change the probability map to each category for a certain input" mean in the supplementary materials?

Here is my understanding, but I want to make sure.
Before changes, for one picture, there are 8 category maps (sky, water, building, etc). If we want a building prior only, we just use the "building" channel to replace the others, so now we have 8 same category maps(all buildings). Is that right?

Question

Hi @xinntao how did you insert the segmentation and label data to the super resolution in the code?

training time

Hi, thanks for your wonderful work and opening source.
Could you please tell me how long did you train the model , the kind of GPU and number of GPUs?

Best regards

OST dataset license information

Hi Xintao,
Awesome work with this paper. I wanted to use OST dataset and was wondering under what license are you open sourcing the dataset? Thanks.

Do you have the tensorflow version of the code?

hello, i want to ask you do you have the tensorflow version of the code, thanks.

validation set for SFTGAN

Hi, xintao.
I have seen that you say we can choose some images from OST300 test set as validation set .
Could you tell me what is your choice of test set as validation set?

Best regards.

How to replace a pre-trained BN layer with the SFT layer at the beginning of training

Hi xintao, I'm doing a SR working inspired by your SFTGAN. I would like to know how you replace a pre-trained BN layer with the SFT layer when you start training? How to initialize the convolution layer which outputs gamma or beta? Thanks.

https://github.com/xinntao/SFTGAN/issues/3#issuecomment-393222418

Have you tried using a semantic map instead of a probability map?

A semantic map,I mean,is the output of segmentation instead of the probability.I wonder if you tried using a semantic map and if you did,what was the result?Thank you.

Cannot find the train code in basic-sr.

As the tittle says, i didn't find the train code in basic-sr.

generating segmentation probability mapping

Hi Xintao, I am wondering whether it is possible for you to release the code about segmentation probability mapping.

parameters of pixel_weight and use_rot in train_stfgan.json

Hi , xintao.

1. I have seen that the parameter of pixel_weight is 0 in train_stfgan.json not 1e-2 as in in train_esrgan.json or train_srgan.json.

Is this right ?

2. I think the parameter of use_rot is true not false as in your original train_stfgan.json file.

Best regards .

hello, i am curious about whether do you have not use any activate functions in the generator network?

test with NIQE,MA, PI metrics

Hi , thanks so much for your open source work.

There are Set5, Set14 test PSNR results in supplementary material , but these can not explain the effect of the model. Have you tested the metrics of NIQE, MA, PI?

Best regards.

Open source training code request

Hi Xinntao,
I'm currently working in a computer vision research group focusing on super-resolution. I read your paper, and it brought great inspiration to me. Adding a prior semantics segmentation prob. map is such a great idea to improve the performance for recovering the details of images. My research teammates are also very appreciated for your work, and we are wondering whether you can share your work with us (I mean the training code).

I can send you my email.
Your help would be very appreciated!

the structure of training set

Thanks for sharing your codes with us!

I follow the guidance of the readme to organize the structure of the training set as follows.

It is img and bicseg files in train files .
The structure under the bicseg folder is similar to bicseg/animal/animal_segprob/.pth files.
The structure under the img folder is similar to img/grass/grass_colorimg/.png images and
Img/grass/grass_byteimg/.png images.

However , there is an error when training the code.
It is No such file or directory: '/home/dataset/OST/train/bicseg/plant/plant_colorimg/plant_653.pth'.

I think the reason for the error lies in the structural organization of the training set. Because on line 58 of the LRHR_seg_bg_dataset.py file,
seg = torch.load(HR_path.replace('/img/','/bicseg/').replace('.png', '.pth')),
after replacing OST/train/img with OST/train/bicseg, there is no .png images in the bicseg file.

Should I remove the subsequent replace?

Thanks so much.

Unable to find training code of SFTGAN

Hi on the repo of SFTGAN it is written to find training code at BasicSR, but I am unable to find training code there, could you please help

How to train the segmentation model

Hi, Xinntao.

I didn't find the code to train the segmentation model.

question of code

What is the difference between SFTLayer and SFTLayer_torch in the code?

Hi Xintao, Can you give me your result that you apply your approach on the dataset set5, set14 and BSD 100, I want to quote your paper and results on these dataset in my paper, ok?

Hi Xintao,
Can you give me your result that you apply your approach on the dataset set5, set14 and BSD 100, I want to quote your paper and results on these dataset in my paper, ok?

Training code reqest

Hi @xinntao was this a masters degree thesis?

hi about SFT layer

I am sorry to trouble you, i want to introduce your SFT layer in my image-to-image translation work.If you have already finished SFT layer in pytorch, please send it to me. Very very thanks. My email is [email protected]
thanks
haoze