xingangpan/deep-generative-prior

Code for deep generative prior (ECCV2020 oral)

Deep Generative Prior (DGP)

Paper

Xingang Pan, Xiaohang Zhan, Bo Dai, Dahua Lin, Chen Change Loy, Ping Luo, "Exploiting Deep Generative Prior for Versatile Image Restoration and Manipulation", ECCV2020 (Oral)

Video: https://youtu.be/p7ToqtwfVko

Demos

DGP exploits the image prior of an off-the-shelf GAN for various image restoration and manipulation.

Image restoration:

Image manipulation:

A learned prior helps internal learning:

Requirements

• python>=3.6

• pytorch>=1.0.1

• others

  pip install -r requirements.txt

Get Started

Before start, please download the pretrained BigGAN at Google drive or Baidu cloud (password: uqtw), and put them to pretrained folder.

Example1: run image colorization example:

sh experiments/examples/run_colorization.sh   

The results will be saved in experiments/examples/images and experiments/examples/image_sheet.

Example2: process images with an image list:

sh experiments/examples/run_inpainting_list.sh   

Example3: evaluate on 1k ImageNet validation images via distributed training based on slurm:

# need to specifiy the root path of imagenet validate set in --root_dir
sh experiments/imagenet1k_128/colorization/train_slurm.sh   

Note:
- BigGAN needs a class condition as input. If no class condition is provided, it would be chosen from a set of random samples.
- The hyperparameters provided may not be optimal, feel free to tune them.

Acknowledgement

The code of BigGAN is borrowed from https://github.com/ajbrock/BigGAN-PyTorch.

Citation

@inproceedings{pan2020dgp,
    author = {Pan, Xingang and Zhan, Xiaohang and Dai, Bo and Lin, Dahua and Loy, Chen Change and Luo, Ping},
    title = {Exploiting Deep Generative Prior for Versatile Image Restoration and Manipulation},
    booktitle = {European Conference on Computer Vision (ECCV)},
    year = {2020}
}

@ARTICLE{pan2020dgp_pami,
    author={Pan, Xingang and Zhan, Xiaohang and Dai, Bo and Lin, Dahua and Loy, Chen Change and Luo, Ping},
    journal={IEEE Transactions on Pattern Analysis and Machine Intelligence}, 
    title={Exploiting Deep Generative Prior for Versatile Image Restoration and Manipulation}, 
    year={2021},
    volume={},
    number={},
    pages={1-1},
    doi={10.1109/TPAMI.2021.3115428}
}

deep-generative-prior's Issues

About G.eval()

Congratulations on your excellent work! In your code, I notice you set generator to eval() mode when you fine-tune the bigGAN, I wonder if it's necessary to set generator to eval() mode when we fine tune it.

What's the purpose of nll loss?

In line 151 of models/dgp.py, you calculated and eventually added to total loss.
However, I cannot find any clue about nll loss in the paper.
Can you tell me why you calculated nll loss?

deep-generative-prior/models/dgp.py

Line 151 in 0a21d98

nll = nll.mean()

pretrained model 'D_256' seems to be corrupted

Thanks for publishing the code and pretrained model for the great paper!

I am not sure whether it is my problem when I download the model, but I got the size checking error for loading the pretrained model 'D_256'. However, when I switch to D_256_ch_64 and G_256_ch_64, the model loading problem disappear.

Thanks for your help!

how to train my datasets?

Hi
can you tell me how to train my datasets?

Pretrained BigGAN 512?

Hey! Thanks for the great work! I was wondering if you have a pretrained model with size 512 or a hint how the models are trained?

How to choose the c vector in the generalization of DGP.

In sec 4.1 of the paper, BigGAN is trained with the c vector.
When generalize the DGP to the images which is not in ImageNet, how to choose a c vector?
Is the c vector fine-tuned in the training stage or fixed ?

Cannot reproduce the image reconstruction results (Table 1 in the paper).

Hi,
Thank you for your great work.
Could you share the training script for the image reconstruction experiment? I use the hyper-parameters of colorization for the image reconstruction experiment. But the results seem to be not good. On the ImageNet 1k val dataset, I only achieve psnr of 25.69 and ssim of 85.12, which are inferior to the results reported in the paper (psnr of 32.89, ssim of 95.95). Could you help me?

Issues about switch off the update G

When I switch off the options --update_G, to perform the reconstruction, which means only optimize the latent code, the result seems far from satisfactory, even for imagenet val datasets.

In my understanding, optimizing just for latent code might not be perfect, but at least it shouldn't be this bad.
Is that consistent with what you tried?

One model per example

Great work!

Just want to clarify one point. Since you choose to fine-tune the parameters of G by using this relaxed reconstruction loss, does it imply that for each example, we need one fine-tuned BigGAN for it? Thanks.

Have you try the FFHQ dataset for human face manipulation?

Great work! The result is very impressive, by the way, have you do experiments on the human face datasets? I

What does class=-1 represent? I'm curious about this.

What does it do to class?

How to train your own dataset for inpainting?

Hi,
How can I use my own dataset for inpainting?

about one image speed ?

good works ! when we infer one image, it will finetune G, so one image it will cost more time ?