Questions about 3D Rendering of Face Image used in Face Specific Augmentation

[iacopomasi]

Questions about 3D Rendering of Face Image used in Face Specific Augmentation

[iacopomasi]

Hi @chamantonio,

for a general overview, you can read this paper:

Masi, I., Hassner, T., Tran, A. T., & Medioni, G. (2017, May). Rapid synthesis of massive face sets for improved face recognition. In Automatic Face & Gesture Recognition (FG 2017), 2017 12th IEEE International Conference on (pp. 604-611). IEEE.

The link is here

Quick reply:

• proj_matrix: is the 3x4 projection matrix from 3D to 2D; from the 3D face to the 2D image. It is M in Eq. (2) of the paper.
• ref_U: is the 224x224 unprojected matrix that contains the 3D coordinates in each channel and serves as a look-up table. For a 3D point in ref_U, say ref_U[i,j]you project it to the image, you sample the texture and then you know how to generate the rendered image since the indices of the matrix will tell you where to write the sampled texture (in [i,j]; This is quickly performed using the 2D interpolation function cv2.remap(). The rendered view is basically ref_U but replaced with the texture content. See Eq. (1) in the paper.
• eyemask and facemask: indices of ref_U that contains the inner part of the faces (without the head) and those of the eyes only. See the config file parameter background to render without the head.

Other very related project you might be interested in:

• FacePoseNet
• MATLAB 3D Model Renderer

I hope this helps.

[chamantonio]

Thank you! Will try to work with this info.