simongiebenhain/nphm

[CVPR'23] Learning Neural Parametric Head Models

Learning Neural Parametric Head Models (NPHM)

Paper | Video | Project Page

This repository contains the implementation of the paper:

Learning Neural Parametric Head Models

Simon Giebenhain, Tobias Kirschstein, Markos Georgopoulos, Martin Rünz, Lourdes Agaptio and Matthias Nießner
CVPR 2023

Update:

We have released an improved version of NPHM that allows for tracking monocular RGB videos: See ProjectPage and Code

If you find our code, dataset or paper useful, please consider citing

@inproceedings{giebenhain2023nphm,
 author={Simon Giebenhain and Tobias Kirschstein and Markos Georgopoulos and  Martin R{\"{u}}nz and Lourdes Agapito and Matthias Nie{\ss}ner},
 title={Learning Neural Parametric Head Models},
 booktitle = {Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)},
 year = {2023}}

Contact Simon Giebenhain for questions, comments and reporting bugs, or open a GitHub Issue.

Installation

First you have to make sure that you have all dependencies in place. The simplest way to do so, is to use anaconda.

You can create an anaconda environment called nphm using

conda create -n "nphm" python=3.9
conda activate nphm

To install this repository and its requirements in development mode run

pip install -e .

If you plan to run a model you will need have a GPU-enabled pytorch version running. For our experiemnts we used pytorch 1.13 and CUDA 11.6.
During training this code uses wandb for logging purposes. wandb needs to be installed additionally, or removed from the code.

Environment Paths

For simplicity we maintain a python file env_paths.py that defines all relevant paths, e.g. for the dataset, model checkpoints, inference results etc.
You will have to modify these values to suit your machine. In case you work with multiple different machines you can remove env_paths.py from the version control of git via

git rm --cached src/NPHM/env_paths.py

which allows to have a differnt version for different machines.

Dataset

We provide detailed information on our dataset here.

Demo

Exploring our Data

For a first look at our please have a look here.

Fitting Point Clouds

After downloading our pretained models, as described here, you can perform reconstruction using

python scripts/fitting/fitting_pointclouds.py -cfg_file scripts/configs/fitting_nphm.yaml -exp_name EXP_NAME -exp_tag EXP_TAG -resolution 256 -demo

Random Sampling

You can generate randomly sampled heads in neutral expression using

python scripts/fitting/fitting_pointclouds.py -cfg_file scripts/configs/fitting_nphm.yaml -exp_name EXP_NAME -exp_tag EXP_TAG -resolution 256 -sample

Data Preparation

Before you can start training you will need to prepare the dataset into a form that can be directly used for supervision.

To train the identity model we prepare samples on the surface for the neutral expression scan for each person, using

python scripts/data_processing/sample_surface.py 

To prepare samples of the forward deformation fields (warping from neutral expression to any other expression of a person) run

python scripts/data_processing/sample_deformation_field.py

Note that this supervision is based on our registered meshes in fixed template topology. It is also restricted to supervise the facial area.

You can state your desired output paths in env_paths.SUPERVISION_IDENTITY and env_paths.SUPERVISION_DEFORMATION_OPEN, respectively.

NOTE: Note that the generated files will take up around ~320GB

Training

NPHM is trained in two stages.

Stage 1 - Learning Geometry in Neutral Expression

First you need to train the identity space using

python scripts/training/train.py -cfg_file scripts/configs/nphm.yaml -local -exp_name SHAPE_EXP_NAME 

To train NPM instead use npm.yaml instead and omit the -local argument.

Stage 2 - Learning Forward Deformations

Afterwards, the expression space can be trained:

python scripts/training/train_corresp.py -cfg_file scripts/configs/nphm_def.yaml -exp_name EXPRESSION_EXP_NAME -mode compress

Make sure that the nphm_def.yaml includes the correct experiment name and epoch in the shape_exp_name and shape_ckpt fields. For NPM use npm_def.yaml and -mode npm.

Pretrained Models

You can download pretrained NPM and NPHM models here. Place the files into env_paths.EXPERIMENT_DIR/

Inference

Test Set

We evaluate our models by fitting back-projected single view point clouds.
More specifically, we propose a test set consisting of 23 identities, with varying number of expressions raning from 7 to 24 expressions. In total it contains 427 scans. Each expression is observed from a randomly, slightly varying frontal view. Additionally each person is observed once from the back. Each observation consists of 2500 3D points.

For inference run

python scripts/fitting/fitting_pointclouds.py -cfg_file scripts/configs/fitting_nphm.yaml -exp_name EXP_NAME -exp_tag EXP_TAG -resolution 400

which optimizes one identity and all expression codes jointly for each person. Results are stored in env_paths.FITTING_DIR/EXP_NAME/EXP_TAG/. The models as well as checkpoints are specified in scripts/configs/fitting_nphm.yaml. For NPM use scripts/configs/fitting_npm.yaml instead.

Evaluation

Metrics against our ground truth scans can be computed using

python scripts/evaluation/eval.py --results_dir FITTING_DIR/forward_EXP_NAME/EXP_TAG/

We compute Chamfer-L1, Chamfer-L2, Normal Consistency and F-Scores @1mm and @5mm. All those metrics are computed in a point-based-fashion, i.e. points (and normals) are sampled on reconstruction and ground truth to compute the metrics. These samples are generated by rendering and back-projecting random views of the meshes, such that models, that reconstruct the mouth interior even if it is closed, are not punished excessively.

Finally, metrics over the full test set are gathered using

python scripts/evaluation/gather.py --results_dir FITTING_DIR/forward_EXP_NAME/EXP_TAG/

which outputs two files FITTING_DIR/forward_EXP_NAME/EXP_TAG/evaluation/total_metrics.csv and FITTING_DIR/forward_EXP_NAME/EXP_TAG/evaluation/total_metrics_face.csv holding average metrics for the complete head and the facial region respecitively.

Offical Metrics for NPM and NPHM

While training and inference happens in a canonical coordiante system (to be specific we use the coordiante system of FLAME and scale it by a factor of 4), we compute metrics in the metric coordiante system in millimeters (mm).

Quantitative results on the complete head

	ChamferL1(mm)	ChamferL2 (mm)	Normal Consitency	F-Score@1mm	F-Score@5mm
NPM	2.657	35.896	0.883	0.478	0.873
NPHM	2.052	13.714	0.874	0.523	0.902

Quantitative results restricted to the facial area

	ChamferL1(mm)	ChamferL2(mm)	Normal Consitency	F-Score@1mm	F-Score@5mm
NPM	0.657	1.129	0.973	0.840	0.994
NPHM	0.531	0.761	0.976	0.891	0.997

NOTE: The number in the paper are not comparable for 2 reasons. 1: the models were trained on only 87 identities, since all remaining scans wer done afterwards. 2: the metrics are not reported in mm.

nphm's Issues

Are you plan to release pretrained models in paper

Great work!

As the title mentioned, the scores of pretrained models in paper are much better than github.
are you plan to release the models that trained in all of identities?

Thanks.

Identity with glasses

Have you tried to test any identity who wears glasses?
Do you have any identity with glasses in your training dataset?

I would like to run such test, but it is not clear how to pre-process it.

The claim in paper that it takes point cloud as input, but there is not any sample about it in documentation or code. Test or dummy data takes the prepared ply meshes: flame, scan and registration. But what is the use of project if it already has such good meshes, how to get mesh from point cloud is still unclear

How you make point clouds using depth map

Hi, after reading your appendix, I want to generate my custom point clouds using depth map that I get using my depth camera, so, I want to know how you convert the depth map to point clouds so that I can test my own data?

Like this:

Can you point out which script or method you used to do this convert? Thanks for your time!

convert a .ply file to an .obj file

Thanks for this amazing work!

While using the NPHM data, I found that we can use the following code to convert .ply files to .obj files. Note that you might also need to copy texture.png into the folder where the target .obj file is located.

from plyfile import PlyData
import numpy as np

def ply_to_obj(ply_file_path, obj_file_path):
    ply_data = PlyData.read(ply_file_path)

    vertex_data = ply_data['vertex'].data
    vertices = np.array([vertex_data['x'], vertex_data['y'], vertex_data['z']]).T

    face_data = ply_data['face'].data
    faces = np.array([face_data['vertex_indices'][i] for i in range(len(face_data))])

    has_uv = True
    uvs = np.array([list(uv) for uv in ply_data['face'].data['texcoord']], dtype=np.float32)

    with open(obj_file_path, 'w') as f:
        f.write(f'mtllib scan.mtl\n')

        for vertex in vertices:
            f.write(f'v {vertex[0]} {vertex[1]} {vertex[2]}\n')

        if has_uv:
            for uv in uvs:
                f.write(f'vt {uv[0]} {uv[1]}\n')
                f.write(f'vt {uv[2]} {uv[3]}\n')
                f.write(f'vt {uv[4]} {uv[5]}\n')

        count = 0
        for face in faces:
            if has_uv:
                face_uv_indices = [i + 1 for i in face]
                f.write('f '+str(face_uv_indices[0])+'/'+str(count*3+1)+' '+str(face_uv_indices[1])+'/'+str(count*3+2)+' '+str(face_uv_indices[2])+'/'+str(count*3+3)+'\n')
                count = count+1
            else:
                face_indices = [i + 1 for i in face]
                f.write(f'f {" ".join(map(str, face_indices))}\n')
    with open(obj_file_path[:-4]+'.mtl', 'w') as f:
        f.write(f'newmtl scan\n')
        f.write(f'map_Kd texture.png\n')

Training on Facescape Dataset

Hi, it's a fantastic work!

I have a question about training NPHM on Facescape Dataset. First, I have prepare the neutral expression data to train the Identity Network, and it can obtain a nice result as the following image shown.

facescape_neutral.zip

And then I use the registered mesh of Facescape dataset and run the scripts/data_processing/sample_deformation_field.py to prepare samples, then train the forward deformation fields. But the results on the training set seems to be wrong. Could you please help me figure out the problem?

facescape_expression.zip

Thanks!

Random Expression Sampling

Hi! Thanks for sharing your nice work!

I am wondering how to sample meaningful expression latent code. For example, those presented on the "Latent Expression Interpolation" section of your project page.

Looking forward to your reply!

how to

linting note: __pycache__

install generated folders __pycache__/ do not need to be checked in. That, build/ and *egg-info/ could be added to .gitignore.
not important... just linting.

Is it possible to get FLAME 51 landmark from the final Head Models?

As title mentioned, are there some methods to get FLAME 51 landmark? Many evaluations need the results of landmark..

RGB Images

Hi,

Thank you very much for the great dataset.

In the paper, you mentioned that, "as part of our dataset we also provide the RGB frames captured during the 3D scanning process". I am wondering where I can find the RGB images per scan.

BR

FLAME model

How to get FLAME model? Which solution was used?

Is it just enough to scale to 4 FLAME model? Without any rotation and moving?

And how did you align point cloud with FLAME model?

Point Cloud - Custom Dataset

Love the work done; great job - could you suggest methodologies for getting a point cloud from monocular video?

How do you fit the model into a 3090 GPU during training?

Hi,

Thank you for your code and dataset. In your supplementary, you mentioned that you did the training on a 3090 GPU. However, when I did the training with my 4090, the memory usage exceeded 24 GB. I wonder how you put the model into a 3090?

Besides, any suggestions on reducing GPU memory usage? I tried to reduce batch size in the nphm.yaml file from 32 to 8, but it didn't work.

problems on facescape data

Hello, dear authors, I've read your paper and downloaded the data, great job!
I run the demo with Facescape publishable data. I fed in the fitting_pointclouds.py with mvs point clould that has been aligned with nphm coordinate.
The neutral expession reuslt looks good yet other expressions seem to be wrong....
I followed the steps described in seciton Fitting Point Clouds , and used the your pretrained data. Could you please help me figure out the problem? thank you very much!

facescpae_344_6_jaw_right.zip

straight longhair has bad results

As title mentioned, I test straight longhair, the output always is curly hair no matter what the parameter is. Face shape and expression are also not good enough.
Input:

NPHM outputs:

Additionally, I find that NPHM cannot deal with long bangs.
Do you have some suggestions to handle this kind of situation?

I upload the test scan ply if you need it to check.
scan.zip

Fitting and model latent code question

Hi,

Thanks for your great work, I have one question that whether I just need to fit Head points cloud to get global id latent code and expression latent code? But the local id latent code is not needed?

how to get registration mesh from scan data？

Hi, it's a fantastic work!
I have a question about how to get registration mesh in the process of preparing data? for example, how to estimate id, exp parameters of flame with input of scan and how to upsample the vertices.

How to make custom pointcloud data as inference input?

Hi, Congratulations. I want to test the inference code with the pointcloud as input. Could you provide some advices? Thanks very much.

Can export 3D obj ?

Hello, thank you for sharing your repo.
I have three questions about your repo. The first question is that can we output obj or ply file ? The second question is that did your model can do texture? And the final question is that can I use my own RGB single image as input?
Thanks.

How would you do the inverse fitting process (from Neural Parametric Head Models to photorealistic portrait)

Super cool implementation, thanks for that! How would you go about the inverse fitting process (from Neural Parametric Head Models to photorealistic portrait)?