deepxmatter/meshtalk

Code for MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglement

meshtalk

This repository contains code to run MeshTalk for face animation from audio. If you use MeshTalk, please cite

@inproceedings{richard2021meshtalk,
    author    = {Richard, Alexander and Zollh\"ofer, Michael and Wen, Yandong and de la Torre, Fernando and Sheikh, Yaser},
    title     = {MeshTalk: 3D Face Animation From Speech Using Cross-Modality Disentanglement},
    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
    month     = {October},
    year      = {2021},
    pages     = {1173-1182}
}

Supplemental Material

Running MeshTalk

Dependencies

ffmpeg
numpy
torch         (tested with v1.10.0)
pytorch3d     (tested with v0.4.0)
torchaudio    (tested with v0.10.0)

Animating a Face Mesh from Audio

Download the pretrained models and unzip them. Make sure your python path contains the root directory (export PYTHONPATH=<your_meshtalk_root_directory>).

Then, run

python animate_face.py --model_dir <your_pretrained_model_dir> --audio_file <your_speech_snippet.wav> --output <your_output_file.mp4>

See a description of command line arguments via python animate_face.py --help. We provide a neutral face template mesh in assets/face_template.obj. Note that the rendered results look slightly different than in the paper and supplemental video because we use a differnt (open source) rendering engine in this repository.

Training your own MeshTalk version

All training code is available in the training directory. The training follows a two-step recipe. First, learn the latent expression code by running train_step1.py, second learn the autoregressive model by running train_step2.py.

Note that we only provide a dataloader that produces dummy data which is always zero. For your training data, implement your own data reader that produces the same assets (template mesh, mesh sequence, audio sequence) as the dummy data reader.

One potential dataset to use for MeshTalk is the Multiface dataset which contains a subset (13 subjects) of the data used in the paper. The dataset includes tracked meshes and audio files.

Note that the geometries in multiface have a slightly different topology than in meshtalk. To convert geometries from multiface to meshtalk, run

python utils/multiface2meshtalk.py <multiface_mesh.bin> <output.obj>

on the .bin files containing the vertex positions of the multiface meshes. Note that the input must be the .bin files from the tracked_mesh directories in multiface, not the .obj files. The output is a .obj file in the same format as assets/face_template.obj.

License

The code and dataset are released under CC-NC 4.0 International license.