jaedukseo/avatarme

Public repository for the CVPR paper AvatarMe

AvatarMe: Realistically Renderable 3D Facial Reconstruction "in-the-wild":

Public repository for the CVPR 2020 paper AvatarMe, with high resolution results, data and more.

Alexandros Lattas ^1,2, Stylianos Moschoglou ^1,2, Baris Gecer ^1,2, Stylianos Ploumpis ^1,2, Vasileios Triantafyllou ²,
Abhijeet Ghosh ¹, Stefanos Zafeiriou ^1,2
1 Imperial College London
² FaceSoft

[Paper PDF] [CVPR 2020]

Overview

AvatarMe is the first method that is able to reconstruct photorealistic 3D faces from a single ‘in-the-wild” image with an increasing level of detail. To achieve this, we capture a large dataset of facial shape and reflectance and build on a state-of-the 3D texture and shape reconstruction method and successively refine its results in order to generate the high-resolution diffuse and specular components that are required for realistic rendering.

Method

A 3DMM is fitted to an ''in-the-wild'' input image and a completed UV texture is synthesized, while optimizing for the identity match between the rendering and the input. The texture is up-sampled 8 times, to synthesize plausible high-frequency details. We then use an image translation network to de-light the texture and obtain the diffuse albedo with high-frequency details. Then, separate networks are used to infer the specular albedo, diffuse normals and specular normals from the diffuse albedo and the 3DMM shape normals. Moreover, the networks are trained on 512x512 patches and inferences are ran on 1536x1536 patches with a sliding window. Finally, we can transfer the facial shape and the consistently inferred reflectance to a head model. Both face and head can be rendered realistically in any environment.

Patch process for Diffuse and Specular Abedo

Patch process for Diffuse and Specular Normals

Results

For high quality rendered results see the following youtube video.

Data

Information about the public training data will be anounced as soon as possible.

Citation

If you find this work useful, please use the following to cite our paper:

@article{lattas2020avatarme,
  title={AvatarMe: Realistically Renderable 3D Facial Reconstruction "in-the-wild"},
  author={Lattas, Alexandros and Moschoglou, Stylianos and Gecer, Baris and Ploumpis, Stylianos and Triantafyllou, Vasileios and Ghosh, Abhijeet and Zafeiriou, Stefanos},
  journal={arXiv preprint arXiv:2003.13845},
  year={2020}
}