deokyunkim/progressive-face-super-resolution

Official Pytorch Implementation of Progressive Face Super-Resolution (BMVC 2019 Accepted)

progressive-face-super-resolution's Issues

I write a demo for someone who want to load compressed FAN

First, check there is a checkpoints/compressed_model_011000.pth in your workdir,
then create a empty folder fan_model, and create a new python file there fan_model/__init__.py

# fan_model/__init__.py
import torch.nn as nn

class fan_squeeze(nn.Module):
    def __init__(self,):
        super(fan_squeeze, self).__init__()
        self.conv = nn.Conv2d(3, 32, 5, 1, 2)
        self.layers = nn.Sequential(
            nn.Conv2d(32, 64, 3, 1, 1),
            nn.BatchNorm2d(64),
            nn.LeakyReLU(),
            nn.Conv2d(64, 128, 3, 1, 1),
            nn.BatchNorm2d(128),
            nn.LeakyReLU(),
            nn.Conv2d(128, 128, 3, 1, 1),
            nn.BatchNorm2d(128),
            nn.LeakyReLU(),
            nn.Conv2d(128, 68, 3, 1, 1),
            nn.LeakyReLU(),
            )

    def forward(self, x):
        out = self.conv(x)
        out = self.layers(out)
        return out

Finally, add a new python file just in the workdir

# load_compressed_FAN.py
import torch
from PIL import Image
import torchvision.transforms as transforms

# prepare input image
image = Image.open("./figure/eval_target_image.jpeg").convert('RGB')
totensor = transforms.Compose([
                            transforms.ToTensor(),
                            transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),
                            ])
x = totensor(image).unsqueeze(0)
x = x.cuda()
# load saved model with both structure and weights
# NOTICE: there must be ./fan_model/__init__.py which contains 
#         class fan_squeeze, Otherwise torch.load will fail
#         but no need to import here, I don't know why
model = torch.load('./checkpoints/compressed_model_011000.pth')
model = model.cuda()
model.eval()
# run forward
with torch.no_grad():
    print("x shape:", x.shape)
    heatmap_68 = model(x)
    print("heatmap shape:", heatmap_68.shape)

Now, check the dir structure:

WORKDIR
  |--checkpoints
  |       |__ compressed_model_011000.pth
  |
  |--fan_model
  |       |__ __init__.py
  |
  |__load_compressed_FAN.py

run the load_compressed_FAN.py, it should work

THCudaCheck FAIL

I tried to run the single-image demonstration on one image, using pytorch 1.0.0 and cud 10.0, but it gives me this output:

pre-trained model is loaded step:3, alpha:1 iteration:80000
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1544174967633/work/aten/src/THC/THCGeneral.cpp line=405 error=11 : invalid argument

I mean, I am using pytorch 1.0.0 as you are, so I don't see where the problem could be. Any idea?

Unaligned results

Hi, do your pre-trained models support unaligned inputs? I test it with unaligned inputs and got really bad results. But with aligned inputs, the results look great.

I found you show unaligned results in your paper, can you please provide the pre-trained model on that set? Thank you!

Pretrained model/checkpoints are not available in the repo

Hi Kim,

I have a use case where I want to try your method to upscale generated images without distorting the image quality. However, I am unable to run the demo.py as the pretrained checkpoints are not available in the respected folder. Could you please update the repo or share these with me.
Any help/suggestion would be appreciated.

Thanks,
Anurag Dubey

Could you please upload your training code?

Thank you so much~

It seems your training code is missing, could you please upload it completely?

Distorted image outputs

I have tried running the demo.py script to use the pretrained ckpt file included in your repo, to upscale images from the celeba dataset, but the output images are always highly distorted. Is there some kind of preprocessing that needs to be done on the images before running demo.py
input

output

All images i have tried so far have had similar results

NameError: name 'predict_image' is not defined

pre-trained model is loaded step:3, alpha:1 iteration:80000
traceback (most recent call last):
File "..\eval.py", line 99, in
..
File "..\eval.py", line 29, in test
..
NameError: name 'predict_image' is not defined

Test other LR images

Hi, Thank you for your work.

I wonder, have you test this model on other LR images?
Because when i test my own LR images(LFW), the result was strange.

I convert from HR to LR using your code.

But, unlike CelebA, LFW's resolution is 128x128. So, i remove center crop code.

Result:

Can you advice me? Thank you.

도움주시면 감사하겠습니다! :ㅇ

PSNR

hello,Why don't you compare the performance PSNR of 2x and 4x?

Could you please upload your training code?

Test Batchsize

Thank you for sharing! Recently, I encountered a problem during the test. When the test batchsize changes, the final results will be a little different. Could you give me some advice, thanks. ^_^

Extracting Landamrks Heatmaps ?

Hi
Can we utilize this Network to extract Landmarks Heatmaps ?
By extracting I mean : saving Landmarks Heatmps as images so I can use them for other purposes.
Many Thanks.

##About train

hello,may I ask how to set the train step?I trainded at step1 every time,and after 15000 iterations，the loss was NAN.How to set step = 2 and step = 3 ?

Why do we multiply by 0.5 and adding 0.5 when calculatin PSNR, SSIM and MS-SSIM ?

Hi
Why do we multiply by 0.5 and adding 0.5 when calculatin PSNR, SSIM and MS-SSIM ?
i.e here :
` mse_loss = MSE_Loss**(0.5predicted_image+0.5, 0.5target_image+0.5)
_ssim = ssim(0.5predicted_image+0.5, 0.5target_image+0.5)

ms_ssim = msssim(0.5predicted_image+0.5, 0.5target_image+0.5)`

And if I want to compare for example PSNR for Progressive Face SR and PSNR of FSRNet, should i do the same operation by multiplying by 0.5 and adding 0.5 or it depends on the Network ?
Thanks.

Question about ResBlock in Generator Network

Hi @DeokyunKim
Please can you tell why there is a deffirence between the paper and Model.py in ResBlock for Step =2 ?

In the paper, the architecture shows for step = 2 a ResBlock with 512 channel (second level), but in your Model.py we see that level 1 and 2 in ResBlock have the same channels numbers as output (wich must = 256 i think ==> dim Variable):

self.conv = nn.Sequential(
                            EqualConv2d(dim, dim, kernel_size=3, padding=1, stride=1),
                            nn.BatchNorm2d(dim),
                            nn.ReLU(),
                            EqualConv2d(dim, dim, kernel_size=3, padding=1, stride=1),
                            nn.BatchNorm2d(dim),
                            nn.ReLU()
                            )

FAN heatmap loss problem

Hi, i am curious about the heatmap loss of 64x64 image counterpart, i find the FAN can not directly deal with the 64x64 image to generate the correct 68 point landmark unless the input size is 256x256, did you rescale the 64x64 images to 256x256 resolution? or i take a mistake, thanks

About eval

I tried your eval.py，but predicted_image seems to be a picture of noise. I have confirmed input_image got from the code below,
input_image = input_image.to(device)
if step==1:
target_image = x2_target_image.to(device)
elif step==2:
target_image = x4_target_image.to(device)
else:
target_image = target_image.to(device)

    input_image = input_image.to(device)

and it is a 16*16 image. May I ask what's wrong with my predicted_image? Many thanks for your answer.

About evaluation

It seems that the ./dataset/Anno/list_eval_partition.txt file is missing.

utils.py file seems missing

Hi, thanks for your work!
I found some thing is missing in train.py
where are functions inf_train_gen, requires_grad, get_heat_map, operate_heatmap?

distorted image results

Hi sir,

I am using your model to try to enhance the qualities of some faces that I cropped frame by frame from a video. Before testing it on my own dataset, I tried on some other pictures from celeba dataset, and for some reason the output is very bad. I have tried all different images like 16x16 and 128x128... the result is always distorted. Any idea what can be done here to improve?

or else, Is there any possibility that I can get your training code?

Thanks a lot. This paper is very good and looks very promising.

Input Size

Does the input size have to be 16x16? What if we want to use custom size, or even 32x32?

Could you provide the link of supplementary materials mentioned in the paper?

In the section 4 of the paper, you show the MOS test and mention that more details are explained in the supplementary materials, but I can't find the link of the materials. Could you provide the link of supplementary materials mentioned in the paper? Thanks a lot~

OS Training

Which operating system were these models trained on?

Could you please upload the training code?

How to run this pretrained model on single image ?

Can you please help and share in how i can run this model on a single image ?