This repository is part of the code release for the Siggraph Asia 2023 paper "From Skin to Skeleton: Towards Biomechanically Accurate 3D Digital Humans".
[paper] [project page]
This repo contains the pytorch SKEL loader and the code to align it to SMPL sequences.
SKEL is a parametric body shape and skeleton model. Its pose parameter lets you change the body shape and its pose parameter lets you pose the skeleton in an anatomically plausible way. Given shape and pose parameters, SKEL returns joint locations, a body mesh, and a skeleton mesh. SKEL is differentiable and can be fit to various data like motion capture or SMPL sequences.
For more details, please check our Siggraph 2023 paper: From Skin to Skeleton: Towards Biomechanically Accurate 3D Digital Humans.
For a minimal use, you will have to follow the install instructions in Set up the environment and Downloading SKEL.
To test the SKEL model, you can then run:
python quickstart.py
This runs the forward pass of SKEL and saves the output as separated body and skeleton meshes.
You can then try to change the SKEL parameters knowing the following. SKEL is controled by two parameters:
-
betascontrols the shape of the body (tall, small, skinny, etc...). It is a vector of size 10. The values should remain between -2 and 2. The first value of the vector roughly controls the height, the second value the weight. -
posecontrols the body pose. It is a vector of size 46, most values are angles expressed in radian, and can vary between -3.14 and 3.14. You can see which of the 46 parameters controls which body articulation here .
SKEL(beta, pose) yields:
- A skeleton 3D mesh
- A body mesh
- Anatomical joint locations
The rest of this readme will let you setup our SKEL vizualizer and run some demo code. We hope you enjoy our work!
Clone this repository
git clone https://github.com/MarilynKeller/SKEL
cd SKEL
Create a virtual environment and install the SKEL package
pip install -U pip
python3.8 -m venv skel_venv
source skel_venv/bin/activate
pip install git+https://github.com/mattloper/chumpy
pip install -e .
Create an account on https://skel.is.tue.mpg.de/ (Necessary for the download to work).
Then download the SKEL model from the download page with the "Download Models" button.
Extract the downloaded folder and edit the file SKEL/skel/config.py to specify the folder containing the downloaded SKEL model folder: skel_folder = '/path/to/skel_models_v1.0
Below, we explain how to set up the visualizer for SKEL (only supported on Linux and Mac), and run some example code.
If you want to run the Demos, you will also need our aitviewer fork for visualization:
cd ..
git clone https://github.com/MarilynKeller/aitviewer-skel.git
cd aitviewer-skel
pip install -e .
Edit then the file aitviewer/aitviewer/aitvconfig.yaml to point to the SKEL folder:
skel_models: "/path/to/skel_models_v1.0"
! Note that the MPI mesh package is only supported on Linux systems. If you are not on Linux, you will have to comment out the code depending on the package psbody.mesh and code your own visualization.
If you want to run an alignment to SMPL, you need to download the SMPL model. First, create an account on https://smpl.is.tue.mpg.de/. Then download this file: SMPL_python_v.1.1.0.zip from the download page. And run:
cd ../SKEL
python scripts/setup_smpl.py /path/to/SMPL_python_v.1.1.0.zip
For visualizing the fitting process you need the MPI mesh package, you can install it with the following line:
pip install git+https://github.com/MPI-IS/mesh.git
Note that the mesh package is only supported on Linux, but it is only necessary for the visualization of the fitting processes.
Visualize the effects of the pose parameters of SKEL:
python examples/skel_poses.py --gender male
Vizualize the shape space:
python examples/skel_betas.py --gender female
Visualize the skinning weights of the skin and bones to the SKEL kinematic tree:
python examples/skel_rigging.py --gender female
Visualize the kinematic tree and joint locations of SKEL:
python examples/skel_kintree.py --gender female
You can see a visual of the joint ids here and their names and the list of degrees of freedom here
Visualize a SKEL sequence. You can find a sample SKEL motion in skel_models_v1.0/sample_motion/ and the corresponding SMPL motion.
python examples/skel_sequence.py /path/to/skel_models_v1.x/sample_motion/01_01_poses_skel.pkl -z
To visualize the SMPL sequence alongside :
python examples/skel_sequence.py /path/to/skel_models_v1.0/sample_motion/01_01_poses_skel.pkl -z --smpl_seq /path/to/skel_models_v1.0/sample_motion/01_01_poses.npz
SKEL can be aligned to SMPL sequences. You can download SMPL sequences from the AMASS Download page, and selecting the SMPL+H G sequences.
Here is the command to run the alignment:
python examples/align_to_SMPL.py /path/to/AMASS/CMU/01/01_01_poses.npz -F
If you have another skeleton model, you can define the mapping between your skeleton model joints and SKEL's joint and optimize SKEL.shape and pose in consequence to obtain a skin mesh that fits your skeleton model.
python examples/fit2joints.py
This research was done partly at The Movement Lab in Stanford and Perceiving Systems at the Max Planck Institute for Intelligent Systems.
We thank Neelay Shah for generating all the necessary SMPL fits, A. A. Osman for his expertise on parametric body models, Shashank Tripathi for his help with MOYO, Giorgio Becherini for his expertise on AMASS and mocap fit evaluation, Peter Kultis, Yao Feng and Yuliang Xu for feedbacks on the paper. We also thank the TML lab and the NMBL lab at Stanford for the fruitful discussions, especially Jennifer Maier for her expertise on the shoulder.
If you use this software, please cite the following work and software:
@inproceedings{keller2023skel,
title = {From Skin to Skeleton: Towards Biomechanically Accurate 3D Digital Humans},
author = {Keller, Marilyn and Werling, Keenon and Shin, Soyong and Delp, Scott and
Pujades, Sergi and Liu, C. Karen and Black, Michael J.},
booktitle = {ACM ToG, Proc.~SIGGRAPH Asia},
volume = {42},
number = {6},
month = dec,
year = {2023},
}
This code and model are available for non-commercial scientific research purposes as defined in the LICENSE.txt file.
For any questions about SKEL loading, please contact [email protected].
For commercial licensing, please contact [email protected]
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent
