Questions about the experimental part

Face-Renovation

HiFaceGAN: Face Renovation via Collaborative Suppression and Replenishment

Lingbo Yang, Chang Liu, Pan Wang, Shanshe Wang, Peiran Ren, Siwei Ma, Wen Gao

Project | arXiv | ACM link| Supplementary Material

Update 20201026: Pretrained checkpoints released to facilitate reproduction.

Update 20200911: Please find video restoration results at this repo!

Update: This paper is accepted at ACM Multimedia 2020.

Usage

Environment

Ubuntu/CentOS
PyTorch 1.0+
CUDA 10.1
python packages: opencv-python, tqdm,
Data augmentation tool: imgaug
Face Recognition Toolkit for evaluation
tqdm to make you less anxious when testing:)

Dataset Preparation

Download FFHQ, resize to 512x512 and split id [65000, 70000) for testing. We only use first 10000 images for training, which takes 2~3 days on a P100 GPU, training with full FFHQ is possible, but could take weeks.

After that, run degrade.py to acquire paired images for training. You need to specify the degradation type and input root in the script first.

Configurations

The configurations is stored in options/config_hifacegan.py, the options should be self-explanatory, but feel free to leave an issue anytime.

Training and Testing

python train.py            # A fool-proof training script
python test.py             # Test on synthetic dataset
python test_nogt.py        # Test on real-world images
python two_source_test.py  # Visualization of Fig 5

Pretrained Models

Download, unzip and put under ./checkpoints. Then change names in configuration file accordingly.

BaiduNetDisk: Extraction code：cxp0

YandexDisk

Note:

These checkpoints works best on synthetic degradation prescribed in degrade.py, don't expect them to handle real-world LQ face images. You can try to fine-tune them with additional collected samples though.
There are two face_renov checkpoints trained under different degradation mixtures. Unfortunately I've forgot which one I used for our paper, so just try both and select the better one. Also, this could give you a hint about how our model behaves under a different degradation setting:)
You may need to set netG=lipspade and ngf=48 inside the configuration file. In case of loading failure, don't hesitate to submit a issue or email me.

Evaluation

Please find in metrics_package folder:

main.py: GPU-based PSNR, SSIM, MS-SSIM, FID
face_dist.py: CPU-based face embedding distance(FED) and landmark localization error (LLE).
PerceptualSimilarity\main.py: GPU-based LPIPS
niqe\niqe.py: NIQE, CPU-based, no reference

Note:

Read the scripts and modify result folder path(s) before testing (do not add / in the end), the results will be displayed on screen and saved in txt.
At least 10GB is required for main.py. If this is too heavy for you, reducebs=250 at line 79
Initializing Inception V3 Model for FID could take several minutes, just be patient. If you find a solution, please submit a PR.
By default face_dist.py script runs with 8 parallel subprocesses, which could cause error on certain environments. In that case, just disable the multiprocessing and replace with a for loop (This would take 2~3 hours for 5k images, you may want to wrap the loop in tqdm to reduce your anxiety).

Benchmark

Please refer to benchmark.md for benchmark experimental settings and performance comparison.

Memory Cost The default model is designed to fit in a P100 card with 16 GB memory. For Titan-X or 1080Ti card with 12 GB memory, you can reduce ngf=48, or further turn batchSize=1 without significant performance drop.

Inference Speed Currently the inference script is single-threaded which runs at 5fps. To further increase the inference speed, possible options are using multi-thread dataloader, batch inference, and combine normalization and convolution operations.

Remarks

Face Renovation is not designed to create a perfect specimen OUT OF you, but to bring out the best WITHIN you.

The Philosophy of Face Renovation | Understanding of HiFaceGAN

License

Citation

Please kindly cite our paper when using this project for your research.

@article{Yang2020HiFaceGANFR,
  title={HiFaceGAN: Face Renovation via Collaborative Suppression and Replenishment},
  author={Lingbo Yang and C. Liu and P. Wang and Shanshe Wang and P. Ren and Siwei Ma and W. Gao},
  journal={Proceedings of the 28th ACM International Conference on Multimedia},
  year={2020}
}

Acknowledgements

The replenishment module borrows the implementation of SPADE.

	class ContentAdaptiveSuppresor(BaseNetwork):
	def __init__(self, opt, sw, sh, n_2xdown,
	norm_layer=nn.InstanceNorm2d):
	super().__init__()
	self.sw = sw
	self.sh = sh
	self.max_ratio = 16
	self.n_2xdown = n_2xdown

	# norm_layer = get_nonspade_norm_layer(opt, opt.norm_E)

	# 20200310: Several Convolution (stride 1) + LIP blocks, 4 fold
	ngf = opt.ngf
	kw = 3
	pw = (kw - 1) // 2

	self.head = nn.Sequential(
	nn.Conv2d(opt.semantic_nc, ngf, kw, stride=1, padding=pw, bias=False),
	norm_layer(ngf),
	nn.ReLU(),
	)
	cur_ratio = 1
	for i in range(n_2xdown):
	next_ratio = min(cur_ratio*2, self.max_ratio)
	model = [
	SimplifiedLIP(ngf*cur_ratio),
	nn.Conv2d(ngfcur_ratio, ngfnext_ratio, kw, stride=1, padding=pw),
	norm_layer(ngf*next_ratio),
	]
	cur_ratio = next_ratio
	if i < n_2xdown - 1:
	model += [nn.ReLU(inplace=True)]
	setattr(self, 'encoder_%d' % i, nn.Sequential(*model))

	def forward(self, x):
	# 20200628: Note the features are arranged from small to large
	x = [self.head(x)]
	for i in range(self.n_2xdown):
	net = getattr(self, 'encoder_%d' % i)
	x = [net(x[0])] + x
	return x

lotayou / face-renovation Goto Github PK

face-renovation's Introduction

Face-Renovation

Project | arXiv | ACM link| Supplementary Material

Update 20201026: Pretrained checkpoints released to facilitate reproduction.

Update 20200911: Please find video restoration results at this repo!

Update: This paper is accepted at ACM Multimedia 2020.

Contents

Usage

Environment

Dataset Preparation

Configurations

Training and Testing

Pretrained Models

Note:

Evaluation

Benchmark

Remarks

Face Renovation is not designed to create a perfect specimen OUT OF you, but to bring out the best WITHIN you.

The Philosophy of Face Renovation | Understanding of HiFaceGAN

License

Citation

Acknowledgements

face-renovation's People

Contributors

Stargazers

Watchers

Forkers

face-renovation's Issues

error: ` Network [HiFaceGANGenerator] was created. Total number of parameters: 130.6 million. To see the architecture, do print(network).

20200218 Critical Bug

Recommend Projects

Recommend Topics

Recommend Org

error:
`
Network [HiFaceGANGenerator] was created. Total number of parameters: 130.6 million. To see the architecture, do print(network).