choyingw/synergynet

3DV 2021: Synergy between 3DMM and 3D Landmarks for Accurate 3D Facial Geometry

SynergyNet

3DV 2021: Synergy between 3DMM and 3D Landmarks for Accurate 3D Facial Geometry

Cho-Ying Wu, Qiangeng Xu, Ulrich Neumann, CGIT Lab at University of Souther California

[paper] [video] [project page]

News [Jul 10, 2022]: Add simplified api for getting 3d landmarks, face mesh, and face pose in only one line. See "Simplified API" It's convenient if you simply want to plug in this method in your work.

News: Add Colab demo

News: Our new work [Cross-Modal Perceptionist] is accepted to CVPR 2022, which is based on this SynergyNet project.

Advantages

👍 SOTA on all 3D facial alignment, face orientation estimation, and 3D face modeling.

👍 Fast inference with 3000fps on a laptop RTX 2080.

👍 Simple implementation with only widely used operations.

(This project is built/tested on Python 3.8 and PyTorch 1.9 on a compatible GPU)

Single Image Inference Demo

1. Clone

   git clone https://github.com/choyingw/SynergyNet

   cd SynergyNet

2. Use conda

   conda create --name SynergyNet

   conda activate SynergyNet

3. Install pre-requisite common packages

   PyTorch 1.9 (should also be compatiable with 1.0+ versions), Torchvision, Opencv, Scipy, Matplotlib, Cython

4. Download data [here] and [here]. Extract these data under the repo root.

These data are processed from [3DDFA] and [FSA-Net].

Download pretrained weights [here]. Put the model under 'pretrained/'

5. Compile Sim3DR and FaceBoxes:

   cd Sim3DR

   ./build_sim3dr.sh

   cd ../FaceBoxes

   ./build_cpu_nms.sh

   cd ..

6. Inference

   python singleImage.py -f img

The default inference requires a compatible GPU to run. If you would like to run on a CPU, please comment the .cuda() and load the pretrained weights into cpu.

Simplified API

We provide a simple API for convenient usage if you want to plug in this method into your work.

import cv2
from synergy3DMM import SynergyNet
model = SynergyNet()
I = cv2.imread(<your image path>)
# get landmark [[y, x, z], 68 (points)], mesh [[y, x, z], 53215 (points)], and face pose (Euler angles [yaw, pitch, roll] and translation [y, x, z])
lmk3d, mesh, pose = model.get_all_outputs(I)

We provide a simple script in singleImage_simple.py

We also provide a setup.py file. Run pip install -e . You can do from synergy3DMM import SynergyNet in other directory. Note that [3dmm_data] and [pretrained weight] (Put the model under 'pretrained/') need to be present.

Benchmark Evaluation

1. Follow Single Image Inference Demo: Step 1-4

2. Benchmarking

   python benchmark.py -w pretrained/best.pth.tar

Print-out results and visualization fo first-50 examples are stored under 'results/' (see 'demo/' for some pre-generated samples as references) are shown.

Updates: Best head pose estimation [pretrained model] (Mean MAE: 3.31) that is better than number reported in paper (3.35). Use -w to load different pretrained models.

Training

1. Follow Single Image Inference Demo: Step 1-4.

2. Download training data from [3DDFA]: train_aug_120x120.zip and extract the zip file under the root folder (Containing about 680K images).

3. bash train_script.sh

4. Please refer to train_script for hyperparameters, such as learning rate, epochs, or GPU device. The default settings take ~19G on a 3090 GPU and about 6 hours for training. If your GPU is less than this size, please decrease the batch size and learning rate proportionally.

Textured Artistic Face Meshes

1. Follow Single Image Inference Demo: Step 1-5.

2. Download artistic faces data [here], which are from [AF-Dataset]. Download our predicted UV maps [here] by UV-texture GAN. Extract them under the root folder.

3. python artistic.py -f art-all --png(whole folder)

   python artistic.py -f art-all/122.png(single image)

Note that this artistic face dataset contains many different level/style face abstration. If a testing image is close to real, the result is much better than those of highly abstract samples.

Textured Real Face Renderings

1. Follow Single Image Inference Demo: Step 1-5.

2. Download our predicted UV maps and real face images for AFLW2000-3D [here] by UV-texture GAN. Extract them under the root folder.

3. python uv_texture_realFaces.py -f texture_data/real --png (whole folder)

   python uv_texture_realFaces.py -f texture_data/real/image00002_real_A.png (single image)

The results (3D meshes and renderings) are stored under 'inference_output'

More Results

We show a comparison with [DECA] using the top-3 largest roll angle samples in AFLW2000-3D.

Facial alignemnt on AFLW2000-3D (NME of facial landmarks):

Face orientation estimation on AFLW2000-3D (MAE of Euler angles):

Results on artistic faces:

Related Project

[Cross-Modal Perceptionist] (analysis on relation for voice and 3D face)

Bibtex

If you find our work useful, please consider to cite our work

@INPROCEEDINGS{wu2021synergy,
  author={Wu, Cho-Ying and Xu, Qiangeng and Neumann, Ulrich},
  booktitle={2021 International Conference on 3D Vision (3DV)}, 
  title={Synergy between 3DMM and 3D Landmarks for Accurate 3D Facial Geometry}, 
  year={2021}
  }

Acknowledgement

The project is developed on [3DDFA] and [FSA-Net]. Thank them for their wonderful work. Thank [3DDFA-V2] for the face detector and rendering codes.

synergynet's Issues

python benchmark.py -w pretrained/best.pth.tar: get RuntimeError: Unexpected key(s) in state_dict: "module.u_tex", "module.w_tex".

Hi, when I try to run python benchmark.py -w pretrained/best.pth.tar, I encounter such an error:

(SynergyNet) heyuan@VIML4:~/Research/3d_face/SynergyNet$ python benchmark.py -w pretrained/best.pth.tar
Traceback (most recent call last):
  File "benchmark.py", line 256, in <module>
    main()
  File "benchmark.py", line 252, in main
    benchmark(args.weights, args)
  File "benchmark.py", line 241, in benchmark
    aflw2000()
  File "benchmark.py", line 235, in aflw2000
    batch_size=128)
  File "benchmark.py", line 113, in extract_param
    model.load_state_dict(checkpoint)
  File "/home/heyuan/Environments/anaconda3/envs/deep3d_pytorch/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1045, in load_state_dict
    self.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for DataParallel:
	Unexpected key(s) in state_dict: "module.u_tex", "module.w_tex". 

Do you have any idea to solve it?

Question about image size

Dear author,

I am YJHong and thanks for sharing your great work!

I would like to use your work for extracting facial landmarks and have 2 questions.

1. Is there any face detection module for extracting facial landmarks ?
   For example 3D-FAN (Face Alignment Network), it detects faces in first and extract landmarks from cropped image.
   While looking "benchmark.py" code, it seems there is no particular face detection module. Am I correct ?

2. If question 1 is correct, how do I crop my own images like AFLW2000-3D_crop dataset you provided ?
   All images in AFLW2000-3D_crop have same (120, 120) size and they looks like all of them have been face-centered cropped with some kind of preprocessing.

3. Image input size is fixed at (120, 120) for SynergyNet ?
   If I want to use SynergyNet, do I should crop images to (120, 120) ?
   Or is it able to just pass raw images to SynergyNet and get landmarks of them ?

Again, thanks for your work!

Best regards,
YJHong.

Training with custom dataset

Hi,
Could you please guide on how to make the target file for only estimating the head pose angles and fine tune the model.
Thank you.

Can not meet the target(1.27 | 1.59 | 1.31) in NoW benchmark using the given pretrained checkpoint

I checked the performance,but do it can not meet the target(1.27 | 1.59 | 1.31) in NoW benchmark using the given pretrained checkpoint. Thanks very much.

Head pose estimation

Hi.

I have a question about the output of head pose estimation.
The head posture output is "Euler angles [yaw, pitch, roll] and translation [y, x, z]". What does this translation [y, x, z] mean?

About the thinking of data preprocess!

Hello, author, long time no contact!

• I have a question to ask you, I see you whiting (standardization) your data (parameters related, which dim=102) as pre-processing. the first 12 dims is the 12 RT matrix elements, right? I just wondering your thinking about why we have to standardize the element of RT matrix, (especially about R matrix), the range of these 9 elements is [-1,1] already (not considering about scale), and after doing standardized, the range of each element is much more incontrollable, so I want to ask about your thinking.
• The reason of this question is that I'm trying to use the element of R matrix as model input, so now I'm considering the step of pre-processing, and below are the statistic about distribution of each element in R-matrix.
  
  

Nms variable dtype mismatch

Hi,

I installed the project without issues, I just had to comment line 49 in FaceBoxes/utils/build.py for Cython to compile but when I run python.exe singleImage.py -f demo\0.png

I get this issue :

Process the image: demo\0.png
Traceback (most recent call last):
File "...\SynergyNet\singleImage.py", line 129, in
main(args)
File "...\SynergyNet\singleImage.py", line 60, in main
rects = face_boxes(img_ori)
File "...\SynergyNet\FaceBoxes\FaceBoxes.py", line 120, in call
keep = nms(dets, nms_threshold)
File "...\SynergyNet\FaceBoxes\utils\nms_wrapper.py", line 18, in nms
return cpu_nms(dets, thresh)
File "nms\cpu_nms.pyx", line 25, in nms.cpu_nms.cpu_nms
ValueError: Buffer dtype mismatch, expected 'int_t' but got 'long long'

Do you know by any chance, how I can fix this ?

Thank you very much

Missing 3dmm_data

Thanks for your work.
Could you upload 3dmm_data dir?
Or the link of 3dmm data for downloading.

how to download ghostnet pretrained model?

how to download ghostnet pretrained model?

Robustness for faces in the wild

I use an image in dataset VoxCeleb1 and render the pncc (projected normalized coordinate code) corresponding to the 3DMM fitted by your pretrained model. I think there is much error in its matching result.

For some other angles, the result is ok

Whether there is an error in param_loss ?

SynergyNet/loss_definition.py

Line 40 in 072c577

loss = self.criterion(input[:,:50], target[:,12:62]).mean(1)

Shaders in rasterizer

Hello,

Does anyone know how the rasterize function (inside SynergyNet\Sim3DR\Sim3DR.py) works under the hood? Do we know what type of shader (eg, Phong) is it using to render the final dense mesh?

Many thanks.

Camera Intrinsic for projecting the mesh and the landmarks

Hi,

I hope you are doing well. I wanted to ask, what is the camera intrinsic that you consider while projecting the mesh and landmarks back to the 2D image?

UnboundLocalError: local variable 'tri' referenced before assignment

Hello, thank your for sharing this amazing work.

When I run the python singleImage.py -f img command, I got the error that:

Process the image:  img/sample_1.jpg
Traceback (most recent call last):
  File "singleImage.py", line 129, in <module>
    main(args)
  File "singleImage.py", line 106, in main
    render(img_ori, vertices_lst, alpha=0.6, wfp=f'inference_output/rendering_overlay/{name}.jpg')
  File "~/SynergyNet/utils/render.py", line 42, in render
    overlap = render_app(ver, tri, overlap, texture=tex)
UnboundLocalError: local variable 'tri' referenced before assignment

I found that it is because the connectivity argument is None by default, but I don't know how to set it to a correct value.

Missing dependencies

Additional python requirements that are needed, but not listed, are: cython and torchvision

Also, and more importantly, this project requires a cuda compatible GPU to run.

Best pose estimation model

Hi @choyingw,
I am trying to use your pose estimation model - the one reproduces the results in the paper (https://drive.google.com/file/d/13LagnHnPvBjWoQwkR3p7egYC6_MVtmG0/view?usp=sharing) - but get fixed pose predicted angles for images with different poses.
When I am using with the regular model you published (https://drive.google.com/file/d/1BVHbiLTfX6iTeJcNbh-jgHjWDoemfrzG/view?usp=sharing) this phenomenon doesn't happen and I get more logical results (but not SOTA for pose estimation).

I was wondering if this happens to you as well and is there a problem in the model you published?

2 version of annotation of benchmark are inaccurate.. if I did any wrong?

• Hello, I'm really appreciate the work you completed, SynergyNet is not only light-weighted but also keep an acceptable accuracy on AFLW2000-3D.

• Although I'm also one of trainer who can't reproduce NME 3.4% (best is 3.674% after fix code problem in here) on original annotation of benchmark, I keep trying to analysis what kind of images model be failed on it and try improve through training process.

• So, I sorted NME of 2000 images, make a grid of 48 worst images and the model alignment on it, and show the ground truth of annotation beside it. (for each pair, left is model output and right is ground truth. and it's reannotated version).
  

• As you can see, some annotation of Ground Truth is not accurate, (index start from 1) pair of (1,1) (1,2) is obvious that its annotation is not worth for reference..., but by other pair (e.g. (8,6)), it shows that reason of large NME is due to model performance instead of inaccurate annotation, namely, it's still have chance to be improved.

• Here is part of my code to post process the file (roi_box, pts68...) you offer in the repo, and visualize the alignment on image. For the inaccurate problem, did I do anything wrong? or is there any opinion you can share for us? I'll be really appreciated for it.

  # put this code in ./aflw2000_data/ and you can run it
  import matplotlib.pyplot as plt
  import numpy as np
  from pathlib import Path
  
  # you can select by image name
  img_name = "image02156.jpg"
  
  img = plt.imread("./AFLW2000-3D_crop/"+img_name)
  
  # choose the version of benchmark annotation (ori or re)
  pts68 = np.load("./eval/AFLW2000-3D.pts68.npy")
  pts68 = np.load("./eval/AFLW2000-3D-Reannotated.pts68.npy")
  
  bbox = np.load("./eval/AFLW2000-3D_crop.roi_box.npy")
  fname_list = Path("./AFLW2000-3D_crop.list").read_text().strip().split('\n')
  
  # coordinate process
  pts68[:,0,:] = (pts68[:,0,:] - bbox[:,[0]]) / (bbox[:,[2]] - bbox[:,[0]]) * 120
  pts68[:,1,:] = (pts68[:,1,:] - bbox[:,[1]]) / (bbox[:,[3]] - bbox[:,[1]]) * 120
  
  fig, ax = plt.subplots()
  
  # plot image
  ax.imshow(img)
  
  # scatter landmarks
  idx = fname_list.index(img_name)
  ax.scatter(pts68[idx,0,:], pts68[idx,1,:])
  
  fig.savefig("alignment.jpg")
  

Issue with cython extension

Hello everyone, I faced this problem when I try to run singleImage.py, although I have built it.
ModuleNotFoundError: No module named 'FaceBoxes.utils.nms.cpu_nms'

How can I solve this?

Label of train_aug_120x120

When I load file './3dmm_data/param_all_norm_v201.pkl' , I found that it has shape ([636252, 102]). I guess that 636252 is the number of image for training but I still dont understand that what 102 is?
Can you explain to me about how you can create your training set?

Different Face Images Have Different Rotations, How To Standardise?

Hello, I'm having issues aligning 1 face from 2 different images into the same view, so that I can run distance-based comparisons with their face features.

I'm aiming to plot the images straight up, as though looking from a camera at eye-level.

However, I can't seem to get the rotation matrix right. The ranges for the x,y,x coordinates for 2 different meshes might also be different. I notice this most especially with the z-axis most frequently.

Do I have to do the rotation first, then normalise using the translation vector or is it the other way around?

Could you kindly take a look and point me in the right direction?

lmk3d, mesh, pose = model.get_all_outputs(I)

# Define the original face pose (Euler angles and translation)
original_yaw = np.deg2rad(pose[0][0][0])   
original_pitch = np.deg2rad(pose[0][0][1])  
original_roll = np.deg2rad(pose[0][0][2])  
original_translation = pose[0][1]  

original_mesh_vertices = mesh[0]

# Function to rotate vertices using Euler angles
def rotate_vertices(vertices, yaw, pitch, roll):
    R_yaw = np.array([[np.cos(yaw), -np.sin(yaw), 0],
                      [np.sin(yaw), np.cos(yaw), 0],
                      [0, 0, 1]])

    R_pitch = np.array([[np.cos(pitch), 0, np.sin(pitch)],
                        [0, 1, 0],
                        [-np.sin(pitch), 0, np.cos(pitch)]])

    R_roll = np.array([[1, 0, 0],
                       [0, np.cos(roll), -np.sin(roll)],
                       [0, np.sin(roll), np.cos(roll)]])

    # Combined rotation matrix
    R = np.dot(R_yaw, np.dot(R_pitch, R_roll))

    # Rotate vertices
    rotated_vertices = np.dot(R, vertices)

    return rotated_vertices

# Function to translate vertices
def translate_vertices(vertices, translation):
    translated_vertices = vertices + np.expand_dims(translation, axis=1)
    return translated_vertices

# Normalize face pose (rotate and translate vertices)
rotated_vertices = rotate_vertices(original_mesh_vertices, -original_yaw, -original_pitch, -original_roll)
normalized_vertices = translate_vertices(rotated_vertices, -original_translation)

About the 3d head position

Hi , can I get the 3d head position

Unable to locate the file keypoints_sim.npy

When I run the command:

python singleImage.py -f img

I receive the following error:

Traceback (most recent call last):
  File "/home/sadat/Desktop/DeltaX_codes/SynergyNet/utils/params.py", line 13, in __init__
    self.keypoints = _load(osp.join(d, 'keypoints_sim.npy'))
  File "/home/sadat/Desktop/DeltaX_codes/SynergyNet/utils/io.py", line 25, in _load
    return np.load(fp)
  File "/home/sadat/miniconda3/envs/SynergyNet/lib/python3.8/site-packages/numpy/lib/npyio.py", line 405, in load
    fid = stack.enter_context(open(os_fspath(file), "rb"))
FileNotFoundError: [Errno 2] No such file or directory: '/home/sadat/Desktop/DeltaX_codes/SynergyNet/utils/../3dmm_data/keypoints_sim.npy'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "singleImage.py", line 6, in <module>
    from model_building import SynergyNet
  File "/home/sadat/Desktop/DeltaX_codes/SynergyNet/model_building.py", line 9, in <module>
    param_pack = ParamsPack()
  File "/home/sadat/Desktop/DeltaX_codes/SynergyNet/utils/params.py", line 37, in __init__
    raise RuntimeError('Missing data')
RuntimeError: Missing data

I am unable to locate the file keypoints_sim.npy

evaluation of AFLW dataset

Face alignment evaluation result of AFLW in the paper and my evaluation result are different.

Could you provide the evaluation code of AFLW dataset?

Training code + How can I render for 3D face output

Hello, amazing work!
Can you please share the training code + 3D face weights and inference code?

Problem compiling FaceBoxes

After trying a couple of times, I find impossible to correctly compile the file ./build_cpu_nms.sh. The error is:

cl : Command line error D8021 : **invalid numeric argument '/Wno-cpp'**
error: command 'C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.34.31933\\bin\\HostX86\\x64\\cl.exe' failed with exit code 2

I am not sure if this could be related to the operating system. I am running the code in Visual Studio Code on Windows 11.

Looking after the error I found this. So I changed in build.py inside FaceBoxes/utils the line of code 47:
from
extra_compile_args=["-Wno-cpp", "-Wno-unused-function"] to
extra_compile_args={'gcc': ['/Qstd=c99']}
After running this, I get the following warnings:

cl : Command line warning D9024 : unrecognized source file type 'gcc', object file assumed
cl : Command line warning D9027 : source file 'gcc' ignored

From where I assume that is basically not compiling the extra modules, but ignoring it. So, it does not show and error, but as expected, the code does not work. The error raises in FaceBoxes\utils\nms_wrapper.py :

  File "nms\cpu_nms.pyx", line 25, in nms.cpu_nms.cpu_nms
ValueError: Buffer dtype mismatch, expected 'int_t' but got 'long long'

I also tried commenting line 47 - in build.py inside FaceBoxes/utils - and uncommenting line 46, which raises the same error.

Any ideas how to solve this?
Many thanks.

training accuracy 3.41？

I followed the training steps, but my final accuracy was only more than 3.8. How can I reproduce the accuracy of 3.41?

This training size is less than 44G (train_aug_120x120.zip)

Hi,
Thanks for sharing your amazing code!

I am trying to run your training code. In Training 2, you mentioned
"Download training data from [3DDFA]: train_aug_120x120.zip and extract the zip file under the root folder (This training size is about 44G)."

However, when I open 3DDFA, I only find a train_aug_120x120.zip with 2.15G. After extracting, it only has 2.8G, which is far less than 44G.

I am wondering, am I missing some information to get the whole dataset?

About hyper-parameters

It seems that some hyper-parameter settings in the code (train_script.sh) are inconsistent with those in the paper, for example, learning rate (0.027 vs. 0.08), loss weight \lambda_{2} (0.05 vs. 0.03), batch size (900 vs. 1024), milestones (48 & 64 vs. 30 & 40), epoch number (50 vs. 80) and lr decay (0.2 vs. 0.1).
Of course these numbers are adjustable but important in the experiments. I want to know how to set these hyper-parameters with backbone mobilenetv2 to get a good performance as yours.

UV-texture GAN code

Thank you so much for sharing your amazing code!

Will you release the code of UV-texture GAN mentioned in the Supplemental Materials of your paper?

support numpy >= 1.20.0 ( np.int --> np.int64)

hi,
there seems to be a minor issue with numpy version.
it is not specified what version you are using.
this pr fixed the below errors to suppoer numpy >= 1.20.0.

thanks

$ python singleImage.py -f img
Traceback (most recent call last):
  File "~/SynergyNet/singleImage.py", line 129, in <module>
    main(args)
  File "~/SynergyNet/singleImage.py", line 28, in main
    model = SynergyNet(args)
  File "~/SynergyNet/model_building.py", line 69, in __init__
    self.triangles = torch.Tensor(self.triangles.astype(np.int)).long().cuda()
  File "~/venvs/env1/lib/python3.10/site-packages/numpy/__init__.py", line 305, in __getattr__
    raise AttributeError(__former_attrs__[attr])
AttributeError: module 'numpy' has no attribute 'int'.
`np.int` was a deprecated alias for the builtin `int`. To avoid this error in existing code, use `int` by itself. Doing this will not modify any behavior and is safe. When replacing `np.int`, you may wish to use e.g. `np.int64` or `np.int32` to specify the precision. If you wish to review your current use, check the release note link for additional information.
The aliases was originally deprecated in NumPy 1.20; for more details and guidance see the original release note at:
    https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations. Did you mean: 'inf'?

# Edit SynergyNet/model_building.py

$ python singleImage.py -f img
Process the image:  img/sample_1.jpg
Traceback (most recent call last):
  File "~/SynergyNet/singleImage.py", line 129, in <module>
    main(args)
  File "~/SynergyNet/singleImage.py", line 60, in main
    rects = face_boxes(img_ori)
  File "~/SynergyNet/FaceBoxes/FaceBoxes.py", line 122, in __call__
    keep = nms(dets, nms_threshold)
  File "~/SynergyNet/FaceBoxes/utils/nms_wrapper.py", line 18, in nms
    return cpu_nms(dets, thresh)
  File "nms/cpu_nms.pyx", line 29, in nms.cpu_nms.cpu_nms
  File "~/venvs/env1/lib/python3.10/site-packages/numpy/__init__.py", line 305, in __getattr__
    raise AttributeError(__former_attrs__[attr])
AttributeError: module 'numpy' has no attribute 'int'.
`np.int` was a deprecated alias for the builtin `int`. To avoid this error in existing code, use `int` by itself. Doing this will not modify any behavior and is safe. When replacing `np.int`, you may wish to use e.g. `np.int64` or `np.int32` to specify the precision. If you wish to review your current use, check the release note link for additional information.
The aliases was originally deprecated in NumPy 1.20; for more details and guidance see the original release note at:
    https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations

# Edit  SynergyNet/FaceBoxes/utils/nms/cpu_nms.pyx 

$ python singleImage.py -f img
Process the image:  img/sample_1.jpg
Save mesh result to inference_output/rendering_overlay/sample_1.jpg
Save landmark result to inference_output/landmarks/sample_1.jpg
Save pose result to inference_output/poses/sample_1.jpg
Process the image:  img/sample_2.jpg
Save mesh result to inference_output/rendering_overlay/sample_2.jpg
Save landmark result to inference_output/landmarks/sample_2.jpg
Save pose result to inference_output/poses/sample_2.jpg
Process the image:  img/sample_3.jpg
Save mesh result to inference_output/rendering_overlay/sample_3.jpg
Save landmark result to inference_output/landmarks/sample_3.jpg
Save pose result to inference_output/poses/sample_3.jpg
Process the image:  img/sample_4.jpg
Save mesh result to inference_output/rendering_overlay/sample_4.jpg
Save landmark result to inference_output/landmarks/sample_4.jpg
Save pose result to inference_output/poses/sample_4.jpg

Normalization, ToTensor, PILToTensor

Hello, I'd been reading and executing your code and noticed that you are using ToTensor transform from torchvision.
I think you need to be aware of it's scaling effects.
Lets break down transform = transforms.Compose([ToTensor(), Normalize(mean=127.5, std=128)])
Assume the intensity of a pixel is 128.
ToTensor converts it to float, BUT ALSO divides it by 255.
So now your intensity is 0.5
Then in Normalize: (0.5 - 127.5) / 128 = -127 / 128 ~ -0.97.
Applying it to other values from range withing [0, 255], we get a range for your intensity in [-0.998, 0.996], which is quiet opposite for what I would consider normalization

The fix is pretty simple, that is to use PILToTensor from the same torchvision, that does not scale your input by 255. Doint that you will get way normal normalization as shown below, in (-1, 1) :)

I will try retraining and benchmarking with better normalization and will submit a pr

Unsatisfied reconstructed results for images in-the-wild

Hi, thank your for sharing this amazing work.

When I ran the demo inference for the in-the-wild images, I got results they are not good.

The input image:

The landmark image:

The blended image:

You can easily find the results have misalignment, I am not sure whether I did something wrong.

BTW, the image size is 512x512.

Face orientation

Hi, I am YJHong and thanks for sharing great work!

I checked code measuring landmark alignment (NME, benchmark_alfw2000.py) though, couldn't find any related code for measuring face orientation (pitch/yaw/roll).

Would you let me know how measure face orientation given 3d landmarks ? (or any related code / repo)

• Have you used cv2.solvePnP function for estimating euler angle ?