westlypark/stylerenderer

Implementation for Paper "Inverting Generative Adversarial Renderer for Face Reconstruction"

StyleGAR

TODO: add arxiv link

Implementation of Inverting Generative Adversarial Renderer for Face Reconstruction

TODO: for test

Currently, some models are being modified with open-code resources (3dmm, landmark, segmentation), to get rid of commercial models, will update soon.

Usage

First align all faces according to landmarks:

python uitls_face.py --lmk dlib --bfm BFM.mat --output OUTPUT_PATH DATASET_PATH

This will align according to specified landmark model, you can change to other kinds of landmark detector

Create lmdb datasets:

python prepare_data.py --out LMDB_PATH --n_worker N_WORKER --size SIZE1,SIZE2,SIZE3,... OUTPUT_PATH

This will convert images to jpeg and pre-resizes it. This implementation does not use progressive growing, but you can create multiple resolution datasets using size arguments with comma separated lists, for the cases that you want to try another resolutions later.

Then you can train model in distributed settings

python -m torch.distributed.launch --nproc_per_node=N_GPU --master_port=PORT train.py --batch BATCH_SIZE LMDB_PATH

train.py supports Weights & Biases logging. If you want to use it, add --wandb arguments to the script.

Generate samples

python generate.py --sample N_FACES --pics N_PICS --ckpt PATH_CHECKPOINT

You should change your size (--size 256 for example) if you train with another dimension.

License

Model details and custom CUDA kernel codes are from official repostiories: https://github.com/NVlabs/stylegan2

Codes for Learned Perceptual Image Patch Similarity, LPIPS came from https://github.com/richzhang/PerceptualSimilarity

To match FID scores more closely to tensorflow official implementations, I have used FID Inception V3 implementations in https://github.com/mseitzer/pytorch-fid

The work is built on the implementation of stylegan2 with Pytorch (https://github.com/rosinality/stylegan2-pytorch.git)

stylerenderer's Issues

Regarding the initialization of the class 'GeneratorWithMap'

Hi,

Thank you for sharing the code of the great work!

I have been adopt the code of the generator to my own project, and I am concerned with the following line of code:

StyleRenderer/model.py

Line 216 in a1a093a

self.to_rgbs.append(ToRGB(out_channel, style_dim))

I feel this line is unnecessary as the to_rgb layers are already initialized in

StyleRenderer/model.py

Line 192 in a1a093a

super(GeneratorWithMap, self)._initialize_styled( \

If I understand it corretly, the current version of code will double the amount of to_rgb layers, but only the first half are used in the forward process (the generation results are still good though).

Best regards

Bug: rasterize on 'cuda:1,2,3...'

in /op/rasterize.py
    when i try:

    if use_cuda:
	v = v.cuda(1)
	f = f.cuda(1)
	t = t.cuda(1)
o = rasterize(v, t, f, 5)
   the process keep running, and got nothing from the print() function;
   if try it in torch.distributed, got errors like:
   CUDA error: an illegal memory access was encountered.

When will the pretrained model, test code and BaselFaceModel.mat be available?

['v '], ['tex'], ['tri '] are not found in BFM.mat.

Hi When running th code, the ['v '], ['tex'], ['tri '] attributes of model are not found in BFM.mat. the BFM.mat (01_MorphableModel.mat) is downloaded from the BFM2009 webside: https://faces.dmi.unibas.ch/bfm/main.php?nav=1-0&id=basel_face_model
As we all know， 01_MorphableModel.mat does not have th key of ['v '], ['tex']. Can you share your BFM.mat，or the code to get it?

environment and keys of model

Hello, can you make CUDA and environment public?

In addition, in utils_ In lines 487, 488 and 489 of face. Py, the ['v '], ['tex'], ['tri '] attributes of model are not found.

thank you very much indeed

pretrained checkpoint for testing

Hi Piao, thanks for your awesome works. Currently I find that there is only traning code released. I wonder if it is possible to release the pretrained checkpoint for some testing and evaluation. Or is it possible to send you several images to observe the performances? Looking forward to your reply.

About rasterize

Thank you for your great work. But when I run this commond "python uitls_face.py --lmk dlib --bfm BFM.mat --output OUTPUT_PATH DATASET_PATH", I encounter some issues like below:
Traceback (most recent call last):
File "H:/stylerender/StyleRenderer/utils_face.py", line 522, in
from op import rasterize
File "H:\stylerender\StyleRenderer\op_init_.py", line 3, in
from .rasterize import rasterize
File "H:\stylerender\StyleRenderer\op\rasterize.py", line 15, in
os.path.join(module_path, 'rasterize.cu')
File "C:\Users\Administrator\anaconda3\envs\styleflow\lib\site-packages\torch\utils\cpp_extension.py", line 898, in load
is_python_module)
File "C:\Users\Administrator\anaconda3\envs\styleflow\lib\site-packages\torch\utils\cpp_extension.py", line 1097, in _jit_compile
return _import_module_from_library(name, build_directory, is_python_module)
File "C:\Users\Administrator\anaconda3\envs\styleflow\lib\site-packages\torch\utils\cpp_extension.py", line 1422, in _import_module_from_library
file, path, description = imp.find_module(module_name, [path])
File "C:\Users\Administrator\anaconda3\envs\styleflow\lib\imp.py", line 302, in find_module
raise ImportError(_ERR_MSG.format(name), name=name)
ImportError: No module named 'rasterize'

Whether I need to complie rasterize.cpp and rasterize.cu? Or mayebe the CUDA and pytorch version are wrong?

Version of Bessel Face Model

Hello，could you tell me version of Bessel Face Model ? 2009? 2017 or 2019?

How is latent code generated actually?

“The reconstructed latent code zˆ is estimated based on the concatenation of each layer’ statistic means and standard variances followed by an MLP." this description confuses me, do you mean the latent code is actually generated by means and standard variances?