calciferzh/smpl

NumPy, TensorFlow and PyTorch implementation of human body SMPL model and infant body SMIL model.

SMPL

Numpy, TensorFlow and PyTorch implementation of SMPL model. For C++ implementation (with PyTorch), please see this repo.

Notes:

• If you want to estimate SMPL parameters from a set of sparse keypoint coordinates, please check this repo.
• The model file cannot be provided due to copyright reasons.
• This project cannot fit a SMPL model to a scan - you can check this paper.

Update Feb 2 2019

Now we have a faster PyTorch implementation, and we also support SMIL model. For more details, please check this PR.

Overview

The author-provided implementation was mainly based on chumpy which is kind of unpopular. Meanwhile, the official version cannot run on GPU. This project provides Numpy, TensorFlow and PyTorch implementation of SMPL model.

For more details about SMPL model, see SMPL.

Numpy and Tensorflow Implementation

Contributor: CalciferZh.

The numpy version is faster (since some computations were rewrote in a vectorized manner) and easier to understand (hope so), and the TensorFlow version can run on GPU.

PyTorch Implementation with Batch Input

Contributor: Lotayou and sebftw

The PyTorch version is derived from the Tensorflow version, and in addition supports batch processing and GPU training. The implementation is hosted in smpl_torch.py along with the testing example.

The implementation is tested under Ubuntu 18.04, Python 3.6 and Pytorch 1.0.0 stable. The output is the same as the original Tensorflow implementation, as can be tested with test.py.

SMIL_torch_batch.py can be very fast, but limited by the memory. It also works with sparse tensors. (Saving a lot of said memory)

Usage

1. Download the model file here.

2. Run python preprocess.py /PATH/TO/THE/DOWNLOADED/MODEL to preprocess the official model. preprocess.py will create a new file model.pkl. smpl_np.py and smpl_tf.py both rely on model.pkl. NOTE: the official pickle model contains chumpy object, so prerocess.py requires chumpy to extract official model. You need to modify chumpy's cource code a little bit to make it compatible to preprocess.py (and Python 3). Here is an instruction in Chinese about this.

3. Run python smpl_np.py or python smpl_tf.py or python smpl_torch.py to see the example. Additionally, run python smpl_torch_batch.py for batched support.

smpl's Issues

How to render a 3d scan model into smpl model ?

Now I have a 3d scan human model (.ply), but cant find a way to render it into smpl model. Does smpl model can only be generated from parameters?

Question about dataset

I noticed that there are joints information in the implementation version of the pytorch batch, and these parameters are not available in the model provided by SMPL. Where is this model? Thank you!

Why regress joints after posing?

I noticed we regress the joints twice in the scripts smpl_torch_batch.py and SMIL_torch_batch.py, and I was wondering about the correctness of this.

In the SMPL paper, it seems they define the joint positions as a regression on the rest pose, without pose blend shapes. We regress the final joint positions from the posed vertices with pose blend shapes.

In case this is an error, it can be fixed in SMIL_torch_batch.py by e.g. adding after line 190:
Jtr = G[..., :4, 3]
And then just deleting line 200, where we regress the joints a second time.
The difference is negligible, but it may be significant if you are doing 3D pose estimation. I saw a difference of at most 4 mm for the infant model, so I would estimate a worst-case error of 4 cm for the SMPL dataset.

What does these codes mean in smpl_tf.py line 148 ?

I had thought about it for hours but didn't get it.
Please help me understand it.

results = stacked - \ pack( tf.matmul( stacked, tf.reshape( tf.concat((J, tf.zeros((24, 1), dtype=tf.float64)), axis=1), (24, 4, 1) ) ) )

obj model upside down when pose and beta are given from SMPLify.

Hello,

I used pose and beta from a model file generated by SMPLify. But when the obj file is opened in blender its not face up.

• model file : m1.zip
• obj opened in Blender: m1_obj.zip

I used smpl_np.py:

Question: What mean the parameters in test.py

In test.py

SMPL/test.py

Lines 58 to 59 in 3a572a6

	pose = (np.random.rand(pose_size) - 0.5) * 0.4
	beta = (np.random.rand(beta_size) - 0.5) * 0.06

we see that both shape and pose are being centered (subtract mean = 1/2 = 0.5) and also multiplied by (somehow) arbitrary numbers (scale).

What does 0.4 and 0.06 mean in this context? Are they really arbitrary or is there and intuitive interpretation?
Thanks in advance.

question about implementation

Thanks for your great work.
I have a question about your implementation.Whether this implementation is based on HMR or the original SMPL code ?
Thanks !

Question about the code

Hi, I have a question about this:
lrotmin = tf.squeeze(tf.reshape((R_cube - I_cube), (-1, 1)))
I think that the code may correspond to the eq.9 in the paper. but the eq.9 is to subtract R_n(theta*) instead of an identity matrix. Are they have the same value? thanks a lot. btw, I have read the code offered by the paper writer, but I am still confused with that.
Thanks a lot:)

Regarding The parametrs in smpl pkl file

what does 'J_regressor_prior ' , 'pose' , 'f' , 'J_regressor' , 'betas' , 'kintree_table' , 'J', 'weights_prior' , 'trans' , 'weights' , 'vert_sym_idxs' , 'posedirs' , 'pose_training_info', 'bs_style' , 'v_template' , 'shapedirs' , 'bs_type' in smpl pkl file means ..What values they define .. what is the use of them each..? Thanks in Advance.

About preprocess to generate model.pkl

Run python preprocess.py /PATH/TO/THE/DOWNLOADED/MODEL.
However, the Model downloaded has two items. One for male and one for female. How to combine them? Make the model available for man and women both.

Different result between this project and official implementation

I test this project and official implementation, and the same pose parameter get different result. Is that true? or I make some mistakes?

SMPL joint names and its indexes

Hi,

Could you please specify SMPL joint indexes and its names ?

Why R_cube minus I_cube? What lrotmin means?

I have seen the "lrotmin = torch.reshape(R_cube - I_cube, (-1, 1)).squeeze()" in pytorch version code. So what lrotmin means? Why R_cube minus I_cube?

这东西很冷门嘛

都没人

Error in computing v_shaped

hi,

I ran preprocess.py on the model 'basicModel_m_lbs_10_207_0_v1.1.0.pkl' from the download link as mentioned in README.
But when I ran smpl_np.py it is giving following error:

Error is in calculation of v_shaped, line 88 of smpl_np.py

the meaning of SMPL pose parameters

I don't understand what's the meaning of SMPL pose parameters. There are 72 pose parameters, does every parameter have specific geometrical or physical meaning?

preprocess: KeyError: 'cocoplus_regressor'

Hi, when I run

python preprocess.py ../smpl_python_v.1.0.0/models/basicModel_f_lbs_10_207_0_v1.0.0.pkl

I got the this error. I checked the keys of src_data and found that there is no cocoplus_regressor.

Did I load the wrong pkl file? Thank you in advance.

Conversion between cartesian coordinates and axis-angle representations

Thanks for the great work!
I am trying to understand the code and I have some problems which have confused me for a long time

• How can I get the transformation matrix for specific joints? When we are computing the new location for a specific joint, it seems that if we unroll the matrix multipication here(take pytorch code for example), the rotation matrix for current joint of interest is not included(only rotation matrices from ancestors are included), it that correct?
  These are formulas from original paper:
  
• In SMPL conversion from axis-angle representations to cartesian cooridinates are included implicity(theta -> R -> J), what about the other direction? If we have joints in the format of (x, y, z), how can we get the axis-angle/euler-angle representations?

I am not an english native speaker, so I am sorry for my terrible and abstract expression... appreciate for any help :)

why use axis-angle rather than euler

Hi, I have a question that why smpl choose axis-angle representation? Cause I find I can't rotate the part to a position that I want by giving angles, why don't use euler angle? Euler seems more intuitional. Are there any other considerations?

How can we train the SMPL model itself？

in the paper, they do the pose training using multi-pose dataset. and get the parameters. How can we train like that?

No module named 'file_utils'

when I run SMIL_torch_batch.py, it gave me "No module named 'file_utils"

Also, find_mini_rgbd and MicroRGBD cannot be found as well. I guess they are from file_utils package.

Vertex index of specifc part of human

Hi! Do you know how to get the indices of vertices that belong to a specific part of human? I know there are 24 parts and 6890 vertices but don't know how to match them. Thanks!!!!!!!!!!!!!!

Error: pickle data was truncated

I ran preprocess.py for the male and female pkl seperately, and got two converted models. However, when I ran smpl_torch.py for inference, it said that "_pickle.UnpicklingError: pickle data was truncated" when loading the converted models.

Traceback (most recent call last):
File "/home/yuan/neural_body_fitting/SMPL/smpl_torch.py", line 208, in
test_gpu([1])
File "/home/yuan/neural_body_fitting/SMPL/smpl_torch.py", line 202, in test_gpu
model = SMPLModel(device=device)
File "/home/yuan/neural_body_fitting/SMPL/smpl_torch.py", line 13, in init
params = pickle.load(f)
_pickle.UnpicklingError: pickle data was truncated

Process finished with exit code 1

cuda error

Hi when I run smpl_torch_batch.py
It gets:

File "C:\Users\xjsxu\Anaconda3\lib\site-packages\torch\functional.py", line 755, in norm
return torch._C._VariableFunctions.frobenius_norm(input, dim, keepdim=keepdim)
RuntimeError: Could not run 'aten::conj.out' with arguments from the 'CUDATensorId' backend. 'aten::conj.out' is only available for these backends: [CPUTensorId, VariableTensorId].

CPU version is fine.
Anaconda 3.7.3
torch: 1.4.0

how to get joints from a generated SMPL obj?

if I have a .obj file with SMPL format (6890 veticals and 13766 faces) , is it possible to regress the joints position?

When running the smpl_torch_batch.py in the google colaboratory enviroment, I encounter a problem.

When running the smpl_torch_batch.py in the google colaboratory enviroment, I encounter a problem.
command1:
!python3 smpl_torch_batch.py
error1:
Traceback (most recent call last):
File "smpl_torch_batch.py", line 228, in
test_gpu([1])
File "smpl_torch_batch.py", line 211, in test_gpu
model = SMPLModel(device=device)
File "smpl_torch_batch.py", line 18, in init
np.array(params['joint_regressor'].T.todense())
KeyError: 'joint_regressor'

cmmand2:
!python3 SMIL_torch_batch.py

error2:
Traceback (most recent call last):
File "SMIL_torch_batch.py", line 217, in
from smil_np import SMILModel
ModuleNotFoundError: No module named 'smil_np'

Sorry to bother you. would you please help me solve this error?

(SMPL) huangxinkai@ubuntu:/project/SMPL$ python preprocess.py smpl/models/basic Model_f_lbs_10_207_0_v1.0.0.pkl
(SMPL) huangxinkai@ubuntu:/project/SMPL$ python preprocess.py smpl/models/basic model_m_lbs_10_207_0_v1.0.0.pkl
(SMPL) huangxinkai@ubuntu:/project/SMPL$ python smpl_torch.py
cuda
(SMPL) huangxinkai@ubuntu:/project/SMPL$ python smpl_torch_batch.py
Traceback (most recent call last):
File "smpl_torch_batch.py", line 228, in
test_gpu([1])
File "smpl_torch_batch.py", line 211, in test_gpu
model = SMPLModel(device=device)
File "smpl_torch_batch.py", line 18, in init
np.array(params['joint_regressor'].T.todense())
KeyError: 'joint_regressor'

I find nothing after running python smpl_torch.py and an error occur after running python smpl_torch_batch.py.

here's my package list:
Package Version

---

chumpy 0.69
numpy 1.17.3
pip 19.3.1
scipy 1.3.1
setuptools 41.6.0
six 1.12.0
torch 1.0.0
wheel 0.33.6

my python vision is 3.6

SMPL augmentation: how to rotate a whole SMPL sequence

Hello,
It's a bit out off-topic here, but could you please show me how to rotate a temporal SMPL sequence along the z axis?
Specifically, we have an SMPL animation that consists of N translation and global rotation. Can you suggest me how to rotate the whole animation by an angle along the Z-axis?
Do need to rotate both translation and the global rotation simultaneously with the same rotation matrix?
Thanks

Joints corresponding to smpl vertices

Hello! Thank you for your great work?
Do you know how to get the joints corresponding to the vertices of smpl？
I know that I can get joint points from vertices through ‘J_regressor’ matrix
But how do you know which joint a vertex is bound to? Or the vertex is bound to multiple joint points, which joint point has the most weight？

joint positions after pose

Thank you for this great implementation of SMPL!

Now I have a question. I wonder how to calculate joint positions after applying specific pose. In SMPL paper, we can calculate joint positions of rest pose using J_regressor matrix. And I think posed joint positions should be obtained by applying joint transformation matrix (equation 3 in SMPL paper) to rest pose joint positions, instead of directly using J_regressor to vertices after Linear Blend Skinning. Can you give me some hint on this question?

SMPL : 3D joints positions to pose rotations

Hi,

I am trying to convert 3D joints positions to pose rotations in order to create a SMPL model from these positions.

In the SMPL paper, it is said : "The pose of the body is defined by a standard skeletal rig, where w_k denotes the axis-angle representation of the relative rotation of part k with respect to its parent in the kinematic tree."

I have tried to compute the axis and the angle of each joint with respect to its parents but without success. The method is however simple : if J is the joint position, P the parent position and G the grandparent position, then
axis = GP x PJ
angle = arccos(GP.PJ / (norm(GP).norm(PJ))

To have the angle-axis, I normalise axis and do normalized_axis x angle.

What am I doing wrong ? Should I use a t-pose to find the good poses ?

For batch input

Hi, Thanks for your SMPL_in_TF, which is very useful.
I tried the code and it works very well. but one thing is dimension error when the input is batch files. I used your code in the CNN network and my input is [N, 10] for betas, now I use the tf.while_loops to solve this problem. Maybe later we could consider to improve it for batch inputs. Thanks again for your code

What is the range of normalization?

Thank you for your useful code
In smpl_np.py, I do not know the range of normalization such as pose information
I'm sorry to trouble you, but could I ask you to do this for me?

smpl tensorflow

@CalciferZh im trying to use the tensorflow version as I think it may be faster. However, how do I load the whole model and all its parameters into tensorflow only once, and then keep changing the model by only changing the beta? Is this possible? Otherwise it seems now that it runs slower than CPU version.

Difference between the SMPL's Rodrigues formula and yours

Hi,
In the paper of SMPL, the Rodrigues formula is used as:

However, in your smpl_np.py, I find that,

R = cos * i_cube + (1 - cos) * dot + np.sin(theta) * m (2)
By converting these mathematical notations in (2), (2) can be rewritten as:

And, to my knowledge about Rodrigues formula (e.g., https://en.wikipedia.org/wiki/Rodrigues%27_rotation_formula),
it should be written as the formula of yours, not as the formula used in SMPL paper, right?

writing to .fbx format instead of .obj

Hello. Thank you for the work. Do you think its possible to write to .fbx format using the output of the SMPL model ? I have the following parameters :

pred_cam (n_frames, 3) # weak perspective camera parameters in cropped image space (s,tx,ty)
orig_cam (n_frames, 4) # weak perspective camera parameters in original image space (sx,sy,tx,ty)
verts (n_frames, 6890, 3) # SMPL mesh vertices
pose (n_frames, 72) # SMPL pose parameters
betas (n_frames, 10) # SMPL body shape parameters
joints3d (n_frames, 49, 3) # SMPL 3D joints
joints2d (n_frames, 21, 3) # 2D keypoint detections by STAF if pose tracking enabled otherwise None
bboxes (n_frames, 4) # bbox detections (cx,cy,w,h)
frame_ids (n_frames,) # frame ids in which subject with tracking id #1 appears

JointRegressor in smpl_torch_batch

SMPL/smpl_torch_batch.py

Line 23 in 20787fb

np.array(params['J_regressor'].todense())

Hi! I think this tensor should be transposed, right?

How to get sequence betas and thetas

I want to make my SMPL model move around, so I need the sequence betas and thetas to change the pose of the SMPL model frame by frame.

Wonder how to get the groundtruth betas and thetas, instead of the model predicted ones like VIBE.

Thanks

About rodtigues

Hey, why does the first term have a cosine function?

SMPL/smpl_np.py

Line 154 in b8ace71

R = cos * i_cube + (1 - cos) * dot + np.sin(theta) * m

question about how to convert parms info to numpy

hello, thanks your great job that help me .But when I use the pytorch version SMPL,I find some conflict. FOr example:
self.shapedirs = torch.from_numpy(params['shapedirs']).type(torch.float64)
I got:
TypeError: expected np.ndarray (got Ch)
There still many like this. I want to know is it the version of my environment that cause the problem. And if we can have a conversion offline or by email.
Thx your great job again~

Joint positions after updating verts

Thanks for great work!
I noticed that the joint positions (smpl.J) after smpl.set_params are different from the correct positions.
I think smpl.update may need updating joints at the last line.

If it is not acceptable for a performance reason, implementing it as an option might be good.

Here is code.

import smpl_np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
from mpl_toolkits.mplot3d.art3d import Poly3DCollection

def smpl_plot():
    smpl = smpl_np.SMPLModel(os.path.join(os.environ['HOME'],"smpl/smpl/models/basicModel_neutral_lbs_10_207_0_v1.0.0.pkl"))
    np.random.seed(9608)
    pose = (np.random.rand(*smpl.pose_shape) - 0.5) * 0.4
    beta = (np.random.rand(*smpl.beta_shape) - 0.5) * 0.06
    trans = np.ones(smpl.trans_shape)*0.1
    smpl.set_params(beta=beta, pose=pose, trans=trans)
    fig = plt.figure()
    ax = fig.add_subplot(111, projection='3d')
    mesh = Poly3DCollection(smpl.verts[smpl.faces], alpha=0.05)
    mesh.set_edgecolor((0.3,0.3,0.3))
    mesh.set_facecolor((0.7,0.7,0.7))
    ax.add_collection3d(mesh)
    J = smpl.J
    for j in range(len(J)):
        pos1 = J[j]
        ax.scatter3D([pos1[0]], [pos1[1]], [pos1[2]], label=f"{j}")
    plt.legend()
    plt.show()
smpl_plot()

import smpl_np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
from mpl_toolkits.mplot3d.art3d import Poly3DCollection

def smpl_plot():
    smpl = smpl_np.SMPLModel(os.path.join(os.environ['HOME'],"smpl/smpl/models/basicModel_neutral_lbs_10_207_0_v1.0.0.pkl"))
    np.random.seed(9608)
    pose = (np.random.rand(*smpl.pose_shape) - 0.5) * 0.4
    beta = (np.random.rand(*smpl.beta_shape) - 0.5) * 0.06
    trans = np.ones(smpl.trans_shape)*0.1
    smpl.set_params(beta=beta, pose=pose, trans=trans)
    fig = plt.figure()
    ax = fig.add_subplot(111, projection='3d')
    mesh = Poly3DCollection(smpl.verts[smpl.faces], alpha=0.05)
    mesh.set_edgecolor((0.3,0.3,0.3))
    mesh.set_facecolor((0.7,0.7,0.7))
    ax.add_collection3d(mesh)

    # modification
    J = smpl.J_regressor.dot(smpl.verts)
    for j in range(len(J)):
        pos1 = J[j]
        ax.scatter3D([pos1[0]], [pos1[1]], [pos1[2]], label=f"{j}")
    plt.legend()
    plt.show()
smpl_plot()

Remove transformation of rest pose

Hi,
I can't understand the mathematical reason for the "remove the transformation due to the rest pose". Can you give me some hints? I know that it is the equation (3) in the paper. But I can't explain why it is implemented like this.
Thank you very much.

SMPL/smpl_np.py

Lines 109 to 115 in f7a2eb3

	# remove the transformation due to the rest pose
	G = G - self.pack(
	np.matmul(
	G,
	np.hstack([self.J, np.zeros([24, 1])]).reshape([24, 4, 1])
	)
	)

Pose parameters to joint rotation?

Hi.

Thanks for the great work.

I am trying to export result to Unity humanoid rig.

After running SMPL, I got 24*3=72 pose parameters.

How would I be able to change these parameters to joint rotations of rig?

From my understanding, pose parameters seem to be axis-angle.

Converting axis-angle requires an angle, theta, along with x y z vectors which are pose parameters of SMPL.

Pose parameters I got were mostly in range of [-1,1] which is unlikely to be Euler degree...

Would anyone have good idea to solve this?

Thank you

pose parameter

your job is great!
I read the paper.In paper author just say "72 pose parameters ; 3 for each part plus 3 for the root orientation".I can understand it can calculated by axis-angle, but what's the meaning of root orientation?

How to calculate pose parameter

Thanks for your work about SMPL. I'm a beginner of human modeling, and I have a question about how to get the pose parameter given a set of joints coordinate coefficient . Is it that calculating the angle between to adjacent bones and represent it in terms of axis-angle as the pose parameter? Do I need to substract the rest pose? Could you please provide a detailed calculation steps

AttributeError: module 'torch' has no attribute 'tensordot'

File "smpl_torch.py", line 132, in forward
v_shaped = torch.tensordot(self.shapedirs, betas, dims=([2], [0])) + self.v_template
AttributeError: module 'torch' has no attribute 'tensordot'

Any ideas why and how could this be fixed?

The first three parameters of pose

What do the first three parameters in pose mean? Is it a radian around xyz? The second and third parameters do not conform to this rule. I tried to change the second parameter according to a certain angle. The rotation of the smpl model did not conform to the input rotation angle.

Why is the arm always flat?

@CalciferZh
I run your code but find the arms are always flat.
Is this right?

The specific meaning of shape parameters

Thank you for your job, I am reading the paper now.
I know the pose parameters represent the rotation vectors of joints,
But I didn't find the specific meaning of the shape parameters. There are ten of them, would you please tell me what's the specific meaning of them? (like weight, height?)
Thank you very much.

How to get UV coordinates for the template

I want to represent a SMPL model in 2d form like this which is from zbrush:

How can get the UV coordinates for SMPL model ?