To run the complete facial retargeting, you will have to proceed with 8 different steps that will be carried out in your Maya scene and in python. The steps are kept independent to allow faster try and test results without having to re-run all the previous steps. A configuration file is there to help you getting trough all the steps. Therefore, the first thing to do is to set up this config file (config/<your_new_config>.json) with all your different path, names, parameters etc. You will then need to modify the script to refer to your configuration file as to load your set of parameters. I chose to have this config file to allow to tweak some hyper-parameters and try which one gives the best results. Indeed, while retargeting my sequences, I figured that some expressions where looking better when changing a bit the initial parameters. But this could be part of the extremely low amount of markers used in our setup (41 facial landmarks and 4 references).
Other than that, the scripts will take care of finding all the blendshapes of your scene, build the nodes, triangulate your mesh and normalize your sequences with a reference frame set in your configuration file.
To apply facial retargeting from scratch, all you need to do is to follow the scripts numbers from 01 to 08. Following a exhaustive list of what the scripts are doing. As I was working on windows, please note that the script "05_build_LdeltaV.py" has been run in python 2.7 as it uses the pymesh library which is not available for Windows. These following steps have to be carried out in Maya console by copy pasting the scripts within it. The scripts are found under the "maya_script" repository.- 1: Triangulate all your meshes if it's not already done. Simply run the script: 01_triangulate_mesh.
- 2: Attach all the blendshape, this will create the blendshape node where the weights are going to be aplpied to. I would recommend to delete your node if it exists and apply the provided script "02_create_blendshapes_nodes" which will save a mesh_name_list that will be used to ensure the correct indexation of your blendshapes and the final facial retargeting weights.
- 3: Save the vertices' positions that semantically represent your set of markers between your character (avatar) and your actor. To do so, run the script "03_extract_blendshape_pos_and_obj", which will save the positions of all the sparse markers for each blendshape as well as saving the blendshapes into an .obj file which we be use to compute the Laplacian over the full meshes.
- 4: Save your reference frame positions by running "04_save_neutral_pose.py". This will allow the next script to use this positions as a reference for all the delta.
- 5: (Python 2.7 and uses pymesh library!) Compute the Laplacian of all your blendshapes. This step can be carried it only once to save computation time. (05_build_LdeltaV.py).
- 6: Compute your set of "personal actor blendshapes". This script implement the section 4 of the paper.
- 7: Retarget your sequences by computing the blendshape weights. This script implement the section 5 of the paper.
- 8: Finally, run the script "08_apply_weights.py" in your maya console to apply the weights on your scene.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent