chaofengc/psfrgan

PyTorch codes for "Progressive Semantic-Aware Style Transformation for Blind Face Restoration", CVPR2021

PSFR-GAN in PyTorch

Progressive Semantic-Aware Style Transformation for Blind Face Restoration
Chaofeng Chen, Xiaoming Li, Lingbo Yang, Xianhui Lin, Lei Zhang, Kwan-Yee K. Wong

Changelog

• 2021.04.26: Add pytorch vgg19 model to GoogleDrive and remove --distributed option which causes training error.
• 2021.03.22: Update new model at 15 epoch (52.5k iterations).
• 2021.03.19: Add train codes for PSFRGAN and FPN.

Prerequisites and Installation

• Ubuntu 18.04
• CUDA 10.1
• Clone this repository

  git clone https://github.com/chaofengc/PSFR-GAN.git
  cd PSFR-GAN
  

• Python 3.7, install required packages by pip3 install -r requirements.txt

Quick Test

Download Pretrain Models and Dataset

Download the pretrained models from the following link and put them to ./pretrain_models

• Github
• BaiduNetDisk, extract code: gj2r

Test single image

Run the following script to enhance face(s) in single input

python test_enhance_single_unalign.py --test_img_path ./test_dir/test_hzgg.jpg --results_dir test_hzgg_results --gpus 1

This script do the following things:

• Crop and align all the faces from input image, stored at results_dir/LQ_faces
• Parse these faces and then enhance them, results stored at results_dir/ParseMaps and results_dir/HQ
• Paste then enhanced faces back to the original image results_dir/hq_final.jpg
• You can use --gpus to specify how many GPUs to use, <=0 means running on CPU. The program will use GPU with the most available memory. Set CUDA_VISIBLE_DEVICE to specify the GPU if you do not want automatic GPU selection.

Test image folder

To test multiple images, we first crop out all the faces and align them use the following script.

python align_and_crop_dir.py --src_dir test_dir --results_dir test_dir_align_results

For images (e.g. multiface_test.jpg) contain multiple faces, the aligned faces will be stored as multiface_test_{face_index}.jpg
And then parse the aligned faces and enhance them with

python test_enhance_dir_align.py --src_dir test_dir_align_results --results_dir test_dir_enhance_results

Results will be saved to three folders respectively: results_dir/lq, results_dir/parse, results_dir/hq.

Additional test script

For your convenience, we also provide script to test multiple unaligned images and paste the enhance results back. Note the paste back operation could be quite slow for large size images containing many faces (dlib takes time to detect faces in large image).

python test_enhance_dir_unalign.py --src_dir test_dir --results_dir test_unalign_results

This script basically do the same thing as test_enhance_single_unalign.py for each image in src_dir

Train the Model

Data Preparation

• Download FFHQ and put the images to ../datasets/FFHQ/imgs1024
• Download parsing masks (512x512) HERE generated by the pretrained FPN and put them to ../datasets/FFHQ/masks512.

Note: you may change ../datasets/FFHQ to your own path. But images and masks must be stored under your_own_path/imgs1024 and your_own_path/masks512 respectively.

Train Script for PSFRGAN

Here is an example train script for PSFRGAN:

python train.py --gpus 2 --model enhance --name PSFRGAN_v001 \
    --g_lr 0.0001 --d_lr 0.0004 --beta1 0.5 \
    --gan_mode 'hinge' --lambda_pix 10 --lambda_fm 10 --lambda_ss 1000 \
    --Dinput_nc 22 --D_num 3 --n_layers_D 4 \
    --batch_size 2 --dataset ffhq  --dataroot ../datasets/FFHQ \
    --visual_freq 100 --print_freq 10 #--continue_train

• Please change the --name option for different experiments. Tensorboard records with the same name will be moved to check_points/log_archive, and the weight directory will only store weight history of latest experiment with the same name.
• --gpus specify number of GPUs used to train. The script will use GPUs with more available memory first. To specify the GPU index, use export CUDA_VISIBLE_DEVICES=your_gpu_ids before the script.
• Uncomment --continue_train to resume training. Current codes do not resume the optimizer state.
• It needs at least 8GB memory to train with batch_size=1.

Scripts for FPN

You may also train your own FPN and generate masks for the HQ images by yourself with the following steps:

• Download CelebAHQ-Mask dataset. Generate CelebAMask-HQ-mask and CelebAMask-HQ-mask-color with the provided scripts in CelebAMask-HQ/face_parsing/Data_preprocessing/.
• Train FPN with the following commmand

python train.py --gpus 1 --model parse --name FPN_v001 \
    --lr 0.0002 --batch_size 8 \
    --dataset celebahqmask --dataroot ../datasets/CelebAMask-HQ \
    --visual_freq 100 --print_freq 10 #--continue_train

• Generate parsing masks with your own FPN using the following command:

python generate_masks.py --save_masks_dir ../datasets/FFHQ/masks512 --batch_size 8 --parse_net_weight path/to/your/own/FPN 

Citation

@inproceedings{ChenPSFRGAN,
    author = {Chen, Chaofeng and Li, Xiaoming and Lingbo, Yang and Lin, Xianhui and Zhang, Lei and Wong, KKY},
    title = {Progressive Semantic-Aware Style Transformation for Blind Face Restoration},
    Journal = {IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
    year = {2021}
}

License

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

Acknowledgement

This work is inspired by SPADE, and closed related to DFDNet and HiFaceGAN. Our codes largely benefit from CycleGAN.

psfrgan's Issues

about generate_mask.py

Thanks your great work~ When I use the trained model you provided and execute the ‘python generate_mask.py’ command，the output is as follows：


but there is a problem with the color of mask:

hi. where is the pretrained model vgg19-dcbb9e9d.pth

when i train the model ,there is no the pretrained model vgg19-dcbb9e9d.pth

Segmentation fault (core dumped)

Thanks for your great work, I conduct the code and it works well. But after all the code is running over (the results are generated), it occurs "core dumped". This is really weird and I really don't know what happened. Here are the prints:

model [EnhanceModel] was created
Loading pretrained LQ face parsing network from ./pretrain_models/parse_multi_iter_90000.pth
Loading pretrained PSFRGAN from ./pretrain_models/psfrgan_epoch15_net_G.pth
======> Loading image test_dir/Solvay_conference_1927.jpg
======> Saving aligned LQ faces to test_unalign_results/Solvay_conference_1927/LQ_faces
100%|█████████████████████████████████████████████| 29/29 [00:02<00:00, 10.87it/s]
======> Save parsing map and the enhanced faces.
======> Paste the enhanced faces back to the original image.
100%|█████████████████████████████████████████████| 29/29 [00:13<00:00, 2.16it/s]
======> Save final result to test_unalign_results/Solvay_conference_1927/hq_final.jpg
======> Loading image test_dir/test_hzgg.jpg
======> Saving aligned LQ faces to test_unalign_results/test_hzgg/LQ_faces
100%|███████████████████████████████████████████████| 3/3 [00:00<00:00, 11.32it/s]
======> Save parsing map and the enhanced faces.
======> Paste the enhanced faces back to the original image.
100%|███████████████████████████████████████████████| 3/3 [00:00<00:00, 11.76it/s]
======> Save final result to test_unalign_results/test_hzgg/hq_final.jpg
Segmentation fault (core dumped)

About CelebAHQ-Test

Thank you for sharing code of this excellent work. Can you provide the specific test images, such as CelebAHQ-Test or PSFR-RealTest? Thank you very much.

Questions about image artifacts

Based on your PSFRGAN network, I tried to train my own model.
Here are some questions about image artifacts.
During the training process, there are some severe checkboard effects in the early training stage. although the effect will be reduced, it still appears on testing (Please zoon in to see details)

Do you obtain the same results during the early training stage? Whether this Loss_RS increases both image details and the checkboard effect?
Thank you

Testing Images For Folder Giving Error

Simple test image works with this command : python test_enhance_single_unalign.py --test_img_path ./test_dir/test_hzgg.jpg --results_dir test_hzgg_results --gpus 1

For folders : python align_and_crop_dir.py --src_dir test_dir --results_dir test_dir_align_results ////// works as well

but when I run this : python test_enhance_dir_align.py --dataroot test_dir_align_results --results_dir test_dir_enhance_results
I get an error :

(psfrgan) C:\PSFRGAN>python test_enhance_dir_align.py --dataroot test_dir_align_results --results_dir test_dir_enhance_results
dataset [SingleDataset] was created
model [EnhanceModel] was created
Loading pretrained LQ face parsing network from ./pretrain_models/parse_multi_iter_90000.pth
Loading pretrained PSFRGAN from ./pretrain_models/psfrgan_latest_net_G.pth
creating result directory test_dir_enhance_results
0%| | 0/8 [00:00<?, ?it/s]
0%| | 0/4 [00:00<?, ?it/s]
0%| | 0/8 [00:01<?, ?it/s]
Traceback (most recent call last):
File "test_enhance_dir_align.py", line 47, in
save_img.save((save_path))
File "C:\Users\User\anaconda3\envs\psfrgan\lib\site-packages\PIL\Image.py", line 2155, in save
fp = builtins.open(filename, "w+b")
FileNotFoundError: [Errno 2] No such file or directory: 'test_dir_enhance_results\lq\test_dir_align_results\Solvay_conference_1927_0.jpg'

I tried to change to a different version of pillow but non of them worked.
Also if you can fix this one,can we have a final result for the folder images as well? Because it says : Note: This is used to test a large amounts of data, and we do not paste the faces back.

I am on Windows 10 64 bit.

Another question any way to use dfdnet models with your project?

did you train more big input

hi, have you train 1024 input for restore? will 1024 have a better result than 512?

epoch num

hi，in your train code the dauflt total epoch num is 50, but it cost even 20 days on my two 2080ti, and the model you given is 15 epoches ,does 15 is enough? 50 will have a better reslut than 15?

Google collab?

Hi there, can anybody pls make a google collab version?

about the ground truth images

Great work， My understanding ability is limited.
I want to know how ground-truth images are used during training.
thanks！

thanks for excellent work , i want to train your model with my own data, but i have some toruble with discriminitor D

networks.py
class NLayerDiscriminator(nn.Module):
nargs = {'norm_type': norm_type, 'relu_type': relu_type, 'ref_ch': ref_ch}
self.model.append(ConvLayer(cin, cout, scale='down', **nargs))
but the ConvLayer doesnot have any parameters is related to 'ref_ch': ref_ch,
so could you tell what is worng?

lr_policy question

total_epochs is 50 , and lr_decay_iters is 50 , it means the lr is not change in trainning process? all be 0.0002?

if i do not use 'step', --lr_policy', type=str, default='step', help='learning rate policy. [linear | step | plateau | cosine]' which one is better?

Loss_ss is 0.000

Hello chaofeng, thank you for your wonderful work. I would like to ask why the Loss_ss is 0 when I trained with my own dataset? The masks I used are not generated by the net_P, and there are only a part of the images have the label masks, for example, my dataset is about brain, and only the tumor part of the brain has label. So could you please explain what's the Loss_ss equals to 0 means? And also the Loss_pcp is 0.

CUDA out of memory.

are you sure that the model can train on a single GPU with 8G memory.? As it needs to load four(G, D, P, VGG) networks during training. I cannot run train with a 12-G gpu.

hi,when will release the train code?

hi, it is a nice work in cvpr 2021, i have test it.

so will the whole training code relase? ，when about？

wait in hope.

Is there a method to obtain '.mat' semantic segmentation files from this code instead of 'jpg' files

Getting some path errors for Windows 10 for training.Need Help.

I used this code :
python train.py --gpus 1 --model enhance --name PSFRGAN_v001 --g_lr 0.0001 --d_lr 0.0004 --beta1 0.5 --lambda_pix 10 --lambda_fm 10 --lambda_ss 1000 --Dinput_nc 22 --D_num 3 --n_layers_D 4 --batch_size 2 --dataset ffhq --dataroot C:\PSFRGAN\datasets\FFHQ --visual_freq 100 --print_freq 10

Note that I removed the --gan argument from commandline cos it was saying that gan model is not implemented,I tried the other gan options but it was again saying it is not implemented so instead I went into the train_options.py and set the gan from there.
Like so:
parser.add_argument('--gan_mode', type=str, default='wgangp', help='the type of GAN objective. [vanilla| lsgan | wgangp]. vanilla GAN loss is the cross-entropy objective used in the original GAN paper.')

Anyway here is where I got stuck :

C:\PSFRGAN>call C:\Users\Break\anaconda3\Scripts\activate.bat

(base) C:\PSFRGAN>call conda activate PSFRGAN

(PSFRGAN) C:\PSFRGAN>python train.py --gpus 1 --model enhance --name PSFRGAN_v001 --g_lr 0.0001 --d_lr 0.0004 --beta1 0.5 --lambda_pix 10 --lambda_fm 10 --lambda_ss 1000 --Dinput_nc 22 --D_num 3 --n_layers_D 4 --batch_size 2 --dataset ffhq --dataroot C:\PSFRGAN\datasets\FFHQ --visual_freq 100 --print_freq 10
----------------- Options ---------------
D_num: 3
Dinput_nc: 22 [default: 3]
Dnorm: in
Gin_size: 512 [default: 512]
Gnorm: spade
Gout_size: 512 [default: 512]
Pimg_size: 512 [default: 512]
Pnorm: bn
batch_size: 2 [default: 16]
beta1: 0.5
checkpoints_dir: ./check_points
continue_train: False
crop_size: 256
d_lr: 0.0004
data_device: cuda:0 [default: None]
dataroot: C:\PSFRGAN\datasets\FFHQ [default: None]
dataset_name: ffhq [default: single]
debug: False
device: cuda:0 [default: None]
epoch: latest
epoch_count: 1
g_lr: 0.0001
gan_mode: wgangp
gpu_ids: [0] [default: None]
gpus: 1
init_gain: 0.02
init_type: normal
input_nc: 3
isTrain: True [default: None]
lambda_fm: 10.0
lambda_g: 1.0
lambda_pcp: 0.0
lambda_pix: 10.0
lambda_ss: 1000.0
load_iter: 0 [default: 0]
load_size: 512
lr: 0.0002
lr_decay_gamma: 1
lr_decay_iters: 50
lr_policy: step
max_dataset_size: inf
model: enhance
n_epochs: 100
n_epochs_decay: 100
n_layers_D: 4
name: PSFRGAN_v001 [default: experiment_name]
ndf: 64
ngf: 64
niter_decay: 100
no_flip: False
no_strict_load: False
num_threads: 8
output_nc: 3
parse_net_weight: ./pretrain_models/parse_multi_iter_90000.pth
phase: train
preprocess: none
print_freq: 10 [default: 100]
resume_epoch: 0
resume_iter: 0
save_by_iter: False
save_epoch_freq: 5
save_iter_freq: 5000
save_latest_freq: 500
seed: 123
serial_batches: False
suffix:
total_epochs: 50
verbose: False
visual_freq: 100 [default: 400]
----------------- End -------------------
dataset [FFHQDataset] was created
The number of training images = 1000
initialize network with normal
model [EnhanceModel] was created
---------- Networks initialized -------------
[Network G] Total number of parameters : 45.957 M
[Network D] Total number of parameters : 18.872 M

Traceback (most recent call last):
File "train.py", line 78, in
train(opt)
File "train.py", line 22, in train
logger = Logger(opt)
File "C:\PSFRGAN\utils\logger.py", line 30, in init
self.mk_log_file()
File "C:\PSFRGAN\utils\logger.py", line 35, in mk_log_file
mkdirs(self.log_dir)
File "C:\PSFRGAN\utils\utils.py", line 133, in mkdirs
os.makedirs(paths)
File "C:\Users\Break\anaconda3\envs\PSFRGAN\lib\os.py", line 220, in makedirs
mkdir(name, mode)
NotADirectoryError: [WinError 267] Dizin adı geçersiz: './check_points\log_dir\PSFRGAN_v001_2021-04-23_06:18'

---

As you can see it is putting extra \ to the path and some extra errors.Can you please help about this one?

Months ago I have reported a similar path issue and you updated the older version of PSRFGAN to run on windows for unaligned images.I guess this is similar.Hoping for some fix for windows users.

Also I have 3 extra questions

1. if we decide to continue training the base model trained by you,which options shall be set in commandline not to break the model?
2. Which gan options is used for the default model and which gan would you recommend to be used for the best results?
   3)Can we have any extra info for the base pretrained model?How many pictures it has been trained on from ffhq set etc?The reason I ask cos the model is generating some extra wrinkles around the eyes which should not be there,I really want to train more on the model.

Error in training

Hello,

I am trying to train the model from scratch on a custom dataset.

When I run the command:

python train.py --gpus 2 --model enhance --name scratch --g_lr 0.0001 --d_lr 0.0004 --beta1 0.5 --gan_mode 'hinge' --lambda_pix 10 --lambda_fm 10 --lambda_ss 1000 --Dinput_nc 22 --D_num 3 --n_layers_D 4 --batch_size 1 --dataset ffhq --dataroot original_test/ --visual_freq 100 --print_freq 10

I get this error:

`
----------------- Options ---------------
D_num: 3
Dinput_nc: 22 [default: 3]
Dnorm: in
Gin_size: 512 [default: 512]
Gnorm: spade
Gout_size: 512 [default: 512]
Pimg_size: 512 [default: 512]
Pnorm: bn
batch_size: 1 [default: 16]
beta1: 0.5
checkpoints_dir: ./check_points
continue_train: False
crop_size: 256
d_lr: 0.0004
data_device: cuda:1 [default: None]
dataroot: original_test/ [default: None]
dataset_name: ffhq [default: single]
debug: False
device: cuda:0 [default: None]
epoch: latest
epoch_count: 1
g_lr: 0.0001
gan_mode: hinge
gpu_ids: [0, 1] [default: None]
gpus: 2 [default: 1]
init_gain: 0.02
init_type: normal
input_nc: 3
isTrain: True [default: None]
lambda_fm: 10.0
lambda_g: 1.0
lambda_pcp: 0.0
lambda_pix: 10.0
lambda_ss: 1000.0
load_iter: 0 [default: 0]
load_size: 512
lr: 0.0002
lr_decay_gamma: 1
lr_decay_iters: 50
lr_policy: step
max_dataset_size: inf
model: enhance
n_epochs: 100
n_epochs_decay: 100
n_layers_D: 4
name: scratch [default: experiment_name]
ndf: 64
ngf: 64
niter_decay: 100
no_flip: False
no_strict_load: False
num_threads: 8
output_nc: 3
parse_net_weight: ./pretrain_models/parse_multi_iter_90000.pth
phase: train
preprocess: none
print_freq: 10 [default: 100]
resume_epoch: 0
resume_iter: 0
save_by_iter: False
save_epoch_freq: 5
save_iter_freq: 5000
save_latest_freq: 500
seed: 123
serial_batches: False
suffix:
total_epochs: 50
verbose: False
visual_freq: 100 [default: 400]
----------------- End -------------------
dataset [FFHQDataset] was created
The number of training images = 2513
initialize network with normal
model [EnhanceModel] was created
---------- Networks initialized -------------
[Network G] Total number of parameters : 45.957 M
[Network D] Total number of parameters : 18.872 M

Start training from epoch: 00000; iter: 0000000
/usr/bin/nvidia-modprobe: unrecognized option: "-s"

ERROR: Invalid commandline, please run /usr/bin/nvidia-modprobe --help for usage information.

/usr/bin/nvidia-modprobe: unrecognized option: "-s"

ERROR: Invalid commandline, please run /usr/bin/nvidia-modprobe --help for usage information.

/usr/bin/nvidia-modprobe: unrecognized option: "-s"

ERROR: Invalid commandline, please run /usr/bin/nvidia-modprobe --help for usage information.

/usr/bin/nvidia-modprobe: unrecognized option: "-s"

ERROR: Invalid commandline, please run /usr/bin/nvidia-modprobe --help for usage information.

Traceback (most recent call last):
File "train.py", line 78, in
train(opt)
File "train.py", line 39, in train
model.forward(), timer.update_time('Forward')
File "/homes/placeholder/PSFR-GAN/models/enhance_model.py", line 93, in forward
self.real_D_results = self.netD(torch.cat((self.img_HR, self.hr_mask), dim=1), return_feat=True)
File "/homes/placeholder/miniconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 550, in call
result = self.forward(*input, **kwargs)
File "/homes/placeholder/miniconda3/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py", line 155, in forward
outputs = self.parallel_apply(replicas, inputs, kwargs)
File "/homes/placeholder/miniconda3/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py", line 165, in parallel_apply
return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
File "/homes/placeholder/miniconda3/lib/python3.8/site-packages/torch/nn/parallel/parallel_apply.py", line 85, in parallel_apply
output.reraise()
File "/homes/placeholder/miniconda3/lib/python3.8/site-packages/torch/_utils.py", line 395, in reraise
raise self.exc_type(msg)
TypeError: Caught TypeError in replica 1 on device 1.
Original Traceback (most recent call last):
File "/homes/placeholder/miniconda3/lib/python3.8/site-packages/torch/nn/parallel/parallel_apply.py", line 60, in _worker
output = module(*input, **kwargs)
File "/homes/placeholder/miniconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 550, in call
result = self.forward(*input, **kwargs)
TypeError: forward() missing 1 required positional argument: 'input'
`

I am using torch==1.5.1 and torchvision==0.6.1. Could you please help me?

测试FPN时遇到了一个问题

作者您好！您知道这个问题如何解决吗，我的路径好像没有问题阿。期待您的回复。

what is the face parsing label?

hi, the parsing label in your code is?

it seems different from the celebaHQ mask label.

thanks

测试FPN时需要psfrgan_epoch15_net_G.pth吗

作者您好，想请问您psfrgan_epoch15_net_G.pth在测试FPN是起什么作用呢，以及测试时我发现输入的FFHQ是清晰图片，网络是自己生成LR吗？

can you provide me a example of degrade image, I cant get a good result as yours. here is a sample of my degrade image.

here is my degrade code:
ia.Sequential([
ia.OneOf([ia.MotionBlur(k=(5,9)),ia.GaussianBlur((3.0, 5.0)),ia.MedianBlur(k=(5,9)),ia.AverageBlur(k=(5, 9))]),
ia.Lambda(func_images),#降采样用pil放缩到（40，64）在放缩回来，抗锯齿resize
ia.OneOf([ia.AdditiveGaussianNoise(loc=(-20,20),scale=(0,3)),ia.AdditiveLaplaceNoise(loc=(-20,20),scale=(0,3))]),
ia.JpegCompression(70,90)
])
any suggestion about this

about alignment operation

Hello, when training the parse network structure, does the FFHQ dataset used need not be aligned? The FFHQ-mask-512 data provided by Baidu Netdisk seems to be an unaligned

According to https://github.com/chaofengc/PSFRGAN/blob/82a8755116d61e83a302b7e48fb61c53c36e34c9/data/ffhq_dataset.py#L17，there is no alignment in the code

Any news about the training code?

Hi, thanks for the great work. I know you already answered this but I would like to ask again because it has been a few months. Any plans to release the training codes? It would be awesome to try your model on different datasets. Thanks...

image corner checkboard artifacts

have you encountered this problem? The checkboard effect happening on image corners during the training and testing phase both on 512x and 1024x.

a question about the result of Figure 6(b)

Great work！ I have a question about the result of Figure 6(b).
Figure 6(b) shows the progressive restoration process when adding multi-scale inputs progressively. However, since the intermediate feature Fi doesn't goes through the ToRGB convolution layer, it is not in RGB format. How do you get the intermediate image result in the Figure，as follows:

Thanks!

there is a error when translate the psfrgan_latest_net_G.pth to onnx.

hi, thanks for the great work.

when i translate the psfrgan_latest_net_G.pth to onnx, there is a error ocuured, the parsing is ok.

can you give some advices. thanks

try to train 1024 resolution memory out

Hi, this is a great code

I try to train 1024 sizes， Gout_size=1024 and batch_size=1
use RTX 3090 24G，Train one memory out of
Can some configurations be modified to adapt to a sample training

Fine-tuning on custom dataset

Hi, I would like to use your work in a project of mine. I would like to fine-tune your pre-trained model on a custom dataset. To do so, I need the pre-trained discriminator. Could you share the "latest_net_D.pth" file? You can send it to [email protected].

Thanks in advance.

where is the Data_preprocessing for training FPN?

Could you provide pretrained model of discriminator

It's a very great job. I want to reproduce your work and finetune in my dataset. Could you provide pretrained model of discriminator if possible?
Thank you very much!

Cuda ran out of memory.

Hello, I am trying to train the enhancer model from scratch, but for some reason the GPU runs out of memory.
It is a 12GB Nvidia gtx 1080, I have set it to batch size of 1, I have tried using different image sizes (1024,720, and 512).

python train.py --gpus 1 --model enhance --name PSFRGAN_V001 --g_lr 0.0001 --d_lr 0.0004 --beta1 0.5 --gan_mode 'hinge' --lambda_pix 10 --lambda_fm 10 --lambda_ss 1000 --Dinput_nc 22 --D_num 3 --n_layers_D 4 --batch_size 2 --dataset ffhq --dataroot ./datasets/FFHQ --visual_freq 100 --print_freq 10 --n_epochs 25

Thanks for sharing your work.

Great job! Do you have a plan to share the training code?

Waiting...

where can I download the dataset “PSFR-RealTest”?

lookforwad to your reply! Many thanks!

what's F0 in the paper?

There is F0 as network input, may i ask how it is used?

2 Important Requests If Possible

When running the test_enhance_dir_unalign.py I had around 800 images.It gave error on some images due to some pictures are not oriented right,some were very low resolution and for whatever reasons.Anyway it gives error and stops the process,as you can see image 0623 is done,0624 gave error.

My first request is can it just,bypass the error and go to the next file ? 623 fine,624 error(and skip without stopping the process), and do 625 and so on?

My second request is if we have 100 images for example,can the folder processing check "test_unalign_results" folder and auto skip if 45 of the images already exported as into folders(HQ,LQ_faces,ParseMaps) and go from the 46th image if we start the processing back again from start(or if an error happens)?

Edit : Maybe another solution of doing it,cut&paste the files that are giving error like no faces detected,to a directory named "test_dir_troubleshoot" automatically,so they are bypassed and then we may know which are the problematic images,and it keeps processing the files which are fine.

Here, is when it stoped the process...

100%|████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 5.89it/s]
======> Save parsing map and the enhanced faces.
======> Loading image test_dir\0623.jpg
======> Saving aligned LQ faces to test_unalign_results\0623\LQ_faces
100%|████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 5.82it/s]
======> Save parsing map and the enhanced faces.
======> Loading image test_dir\0624.jpg
Traceback (most recent call last):
File "test_enhance_dir_unalign_crop.py", line 34, in
aligned_faces, tform_params = detect_and_align_faces(img, face_detector, lmk_predictor, template_path)
File "C:\PSFRGAN\test_enhance_single_unalign.py", line 24, in detect_and_align_faces
assert len(face_dets) > 0, 'No faces detected'
AssertionError: No faces detected

If you can do it please,it will be excellent!

Could you please provide the training code?

Could you please provide the training code?

Low quality of the output JPEGs

When using PSFRGAN I noticed that there's a lot of JPEG compression artifacts in outputs even, when I use lossless PNGs as inputs. Is it possible to fix this issue in any way? I tried looking for some settings sections in .py files but couldn't find anything related to the quality of JPEGs.

does the result is normal?

when i train my own data at 5000 iter.
the test reslut is:

Нейронів для покращення якості фото

could you please share the supplementary materials

wanna learn more about the degradation model, which are described in the supplementary materials. So could you release them?

about color noise

with Semantic-Aware Style Loss, color noise problem in generated images can be solved ?

Enhanced face not blending properly(?)

I'm testing PSFRGAN on old images and it constantly places hard boxes that are either black or grey around the enhanced subject, is there something I'm missing or doing wrong?

A question about training the FPN

First of all, thank you very much for your work. I met the above problem when I trained FPN. Do you know how to solve it？ Looking forward to your reply.

Thank you for your work. After I retrained netG, there are obvious boundaries on the edges of the face

Thank you for your work. After I retrained netG, there are obvious boundaries on the edges of the face

some question

我有一个疑问，你们这个模型是一个GAN模型，为什么没有判别器呢？

Trying to run the script under Windows 10 - 'could not find the specified file'

Well, I know that in requirements you stated that it would be best choice to test your script in Ubuntu, but I decided to try running it under Windows 10 (64-bit edition). Despite my xtremely poor skills in Python, installation proved successful, however, when I tried to process test images I ran into following errors...

(base) C:\PSFRGAN-master poprawianie twarzy w HQ>python test_enhance_single_unalign.py --test_img_path ./test_dir/test_hzgg.jpg --results_dir test_hzgg_results --gpus 1
Traceback (most recent call last):
  File "test_enhance_single_unalign.py", line 93, in <module>
    opt = TestOptions().parse()
  File "C:\PSFRGAN-master poprawianie twarzy w HQ\options\base_options.py", line 141, in parse
    opt.gpu_ids = utils.get_gpu_memory_map()[1][:opt.gpus]
  File "C:\PSFRGAN-master poprawianie twarzy w HQ\utils\utils.py", line 123, in get_gpu_memory_map
    result = subprocess.check_output(
  File "C:\Users\ivell\anaconda3\lib\subprocess.py", line 411, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "C:\Users\ivell\anaconda3\lib\subprocess.py", line 489, in run
    with Popen(*popenargs, **kwargs) as process:
  File "C:\Users\ivell\anaconda3\lib\subprocess.py", line 854, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "C:\Users\ivell\anaconda3\lib\subprocess.py", line 1307, in _execute_child
    hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
FileNotFoundError: [WinError 2] Nie można odnaleźć określonego pliku

Last line says 'could not find the specified file'.

Non Linearity at the end

Can you please explain why don't you use some kind of non linearity at the end, when want to get RGB images?
I faced the problem that when training it the way it is implemented sometimes I get pretty big numbers at the end and it leads to some artifacts, but when I am trying to train it with something like tanh at the end it goes crazy and just generate picture of 1 color (for ex. full red or white square)

There is displacement of pixel between input and output

Excellent work! But I find that there is some displacement of pixel between input and output. Can this be fixed?

Dlib Import error

Program gives this error, I found a solution on the Internet that you need to install cblas, however, even after installation, nothing has changed

❯ python align_and_crop_dir.py --src_dir test_dir --results_dir test_dir_align_results
Traceback (most recent call last):
  File "align_and_crop_dir.py", line 1, in <module>
    import dlib
  File "/run/media/viktor/Torrent/NeuralNetworks/PSFRGAN/venv/lib/python3.8/site-packages/dlib/__init__.py", line 19, in <module>
    from _dlib_pybind11 import *
ImportError: /run/media/viktor/Torrent/NeuralNetworks/PSFRGAN/venv/lib/python3.8/site-packages/_dlib_pybind11.cpython-38-x86_64-linux-gnu.so: undefined symbol: cblas_dtrsm

❯ pip freeze
absl-py==0.10.0
astunparse==1.6.3
certifi==2020.6.20
cycler==0.10.0
decorator==4.4.2
dlib==19.21.0
future==0.18.2
gast==0.3.3
google-pasta==0.2.0
grpcio==1.32.0
h5py==2.10.0
imageio==2.9.0
imgaug==0.4.0
Keras-Preprocessing==1.1.2
kiwisolver==1.2.0
Markdown==3.2.2
matplotlib==3.3.2
networkx==2.5
numpy==1.19.2
opencv-python==4.4.0.44
opt-einsum==3.3.0
Pillow==7.2.0
protobuf==3.13.0
pyparsing==2.4.7
python-dateutil==2.8.1
PyWavelets==1.1.1
scikit-image==0.17.2
scipy==1.4.1
Shapely==1.7.1
six==1.15.0
tensorboard==1.15.0
tensorboardX==2.1
tensorflow==2.3.1
tensorflow-estimator==2.3.0
termcolor==1.1.0
tifffile==2020.9.22
torch==1.5.1
torchvision==0.6.1
tqdm==4.50.0
Werkzeug==1.0.1
wrapt==1.12.1

❯ pacman -Ss cblas
extra/cblas 3.9.0-3 [Installed]
    C interface to BLAS

❯ neofetch --off
viktor@viktor-p61usb3b3 
----------------------- 
OS: Manjaro Linux x86_64 
Host: P61-USB3-B3 
Kernel: 5.8.6-1-MANJARO 
Uptime: 49 mins 
Packages: 1118 (pacman) 
Shell: zsh 5.8 
Resolution: 1920x1080 
DE: GNOME 3.36.6 
WM: Mutter 
WM Theme: Matcha-dark-sea 
Theme: Nordic-darker [GTK2/3] 
Icons: Papirus-Dark-nordic-blue-folders [GTK2/3] 
Terminal: gnome-terminal 
CPU: Intel i5-2500 (4) @ 3.700GHz 
GPU: NVIDIA GeForce GTX 1080 
Memory: 3115MiB / 15995MiB