calciferzh/minimal-ik

Solving SMPL/MANO parameters from keypoint coordinates.

Greetings :)

I’m currently a PhD student at ETH Zurich. I received my bachelor's and master's degree at Tsinghua University in 2019 and 2022.

In my daily life, I enjoy cycling and running.

Please check my homepage for more details.

minimal-ik's Issues

update函数只输出keypoints

您好，minimal-IK是一个非常有启发的项目

function
关于update函数我有个问题，我注意到优化的时候只有keypoints被用到，而verts没有被用到，同时要计算verts涉及到几个778size的运算。
请问如果为了速度，只输出keypoints，是否有可能略过verts的部分相关计算

谢谢！

How can we drive the MANO template mesh using FK given xyz data?

Thank you so much for your great work!
But IK is a little time-consuming. Can we do this using FK?

I have a data of 21 joints(xyz), then we can calculate the rotation of the joints. Then we can apply this rotation to template mesh MANO using FK. But the result is weired. Anyone can help?

Thank you very much!

Does the joint name order follows the official SMPL-H, SMPL?

For SMPL-H following https://github.com/vchoutas/smplx/blob/master/smplx/joint_names.py

for SMPL following line 18 to 42?
for SMPL-H following line 18 to 72?
for MANO following line 43 to 72?

Solve error when Input keypoints ?

Hello,CalciferZh.When I input detect 3D points in IK that result is strange.

points = np.asarray([[-0.05845946, -0.03575248, -0.01891485, -0.00118977,  0.02288574, -0.02119502,
                    -0.00542915,  0.01052371,  0.02606209, -0.01342221,  0.01144132,  0.02687377,
                    0.04031715, -0.01040107,  0.01338105 , 0.03538704,  0.05632934, -0.00652865,
                    0.01324268,  0.03186151,  0.05112882],\
                    [ 0.02216181, -0.00476968, -0.02193613, -0.02797434, -0.04152796, -0.02755484,
                    -0.03023094, -0.03193476, -0.03731135, -0.01063601, -0.01484661, -0.01565503,
                    -0.01747507 , 0.01185602,  0.0031302,  -0.00127476, -0.00702907,  0.02705859,
                    0.02097698 , 0.0164664,   0.00978883],\
                    [ 0.86504179,  0.8595106,   0.84762412,  0.83012652,  0.80829155,  0.80431879,
                    0.78179353,  0.78447574,  0.80182135,  0.7954936,   0.78770196,  0.79872048,
                    0.81998879,  0.80215442,  0.79682922 , 0.80361056,  0.8165772,   0.81301314,
                    0.8120501,   0.81251454,  0.81184596]])

Question on the shape of "keypoints"

Hi @CalciferZh
In spmlx , the shape of keypoint is (24,3), as follow:

In

Minimal-IK/example.py

Line 47 in 72bfcbb

params_est = solver.solve(wrapper, keypoints)

the shape of keypoint is (29,3)

Could padding (5,3) to solve this gap?

Keypoint input format for MANO hand model

Hi! After going through the code, I ain't able to understand the following:

If I want to find the pose_abs of the estimated hand using the MANO hand model, what should be the format in which I am required to input the hand key points? By that what I mean is, with respect to which coordinate system should I input the hand key points?

Thanks in advance!

no args in smpl

when i use it in smpl with SMPL_MODEL_PATH, this is no some args in pkl model .
models.py:

Is there any other solution ?

SMPL parameters :
betas(10 x 1) + pose(24 x 3) -------> joint coordinates (and optional other keypoints)

your project:
joint coordinates (and optional other keypoints) -----> betas + pose

   I assume that the betas are fixed , only solve pose paramters。As it's only a "minimal" solution, is there any other solution?

MPI的网址是什么？

看你了的代码，其中提到模型文件到MPI下载 “Download the official model from MPI”， 想知道MPI的地址是什么？谢谢！

想请教一下SMPL参数估计的正确食用方式（扶

首先感谢一下开源（跪）

我们试着从一批FBX动画文件中批量导出关键点信息转成SMPL格式，但在使用这个工具时遇到了点儿问题

左图是example中给出的随机关键点，右图是被我们用于求解的关键点，乍一看问题不大

根据比例估算了一下scale为2.7，以上述关键点为输入，微调了一下example里的代码求解（如下）

def EstSMPLByKeypointAndScale(keypoints, scale):

    np.random.seed(20160923)

    pose_glb = np.zeros([1, 3]) # global rotation


    ########################## smpl settings ##########################
    # note that in smpl and smpl-h no pca for pose is provided
    # therefore in the model we fake an identity matrix as the pca coefficients
    # to make the code compatible

    n_pose = 23 * 3 # degrees of freedom, (n_joints - 1) * 3
    n_shape = 10
    pose_pca = np.random.uniform(-0.2, 0.2, size=n_pose)
    shape = np.random.normal(size=n_shape)
    mesh = KinematicModel(config.SMPL_MODEL_PATH, SMPLArmature, scale=scale)

    ########################## solving params ############################

    wrapper = KinematicPCAWrapper(mesh, n_pose=n_pose)
    solver = Solver(verbose=True)
  
    # 求解
    params_est = solver.solve(wrapper, keypoints)
  
    shape_est, pose_pca_est, pose_glb_est = wrapper.decode(params_est)

    print('----------------------------------------------------------------------')
    print('estimated parameters')
    print('pose pca coefficients:', pose_pca_est)
    print('pose global rotation:', pose_glb_est)
    print('shape: pca coefficients:', shape_est)

    mesh.set_params(pose_pca=pose_pca_est)
    mesh.save_obj('./est.obj')

    print('estimated meshes is saved into est.obj')

但是出来的估计结果mesh输出是这样的（不止一点邪典）

方便请教一下出了这个结果可能是哪一步出了问题吗（泪下），是输入数据分布不太合适、对Scale的微调不足、还是求解代码写错了😭

关于IK的问题

感谢开源！看到您在 #18 提到的基于神经网络的IK方法，请问您有推荐阅读的论文吗？或者应该以什么关键词（中/英）进行搜索呢？

about difference

Minimal-IK/solver.py

Line 53 in 0c18902

d = (res1 - res2) / (2 * self.eps)

How about accuracy of this method? Do you compare it with other methods?

Run my keypoint with poor results

Hi, when I run smpl, the fit is poor when using the keypoints output from my own program as input. I think this may be due to the large rotation of the root node.

Again, we note that this approach cannot handle large global rotations (R0) due to the high non-convexity. For example, when the subject keeps the T pose but faces backwards.

I rotated the keypoint to roughly the same pose as T pose, but it does not solve the problem, do you have any comments?

Make SMPL from COCO Keypoints

Hi! I'm very new to this. I have coco keypoints of a person (18 keypoint format) for multiple frames. I would like to convert my keypoints to the SMPL format so that I can render some animations on Mixamo.
Is this at all possible? And if so, how can I do this?
My keypoint file is of the format [frames,joints,3] -> 300,18,3 (3 refers to positions along the x,y,z axis)

about dependices

Hi, when I run example.py. I got errors as follows:

Traceback (most recent call last): File "/home/haomeng/PycharmProjects/Minimal-IK/example.py", line 20, in <module> mesh = KinematicModel(config.MANO_MODEL_PATH, MANOArmature, scale=1000) File "/home/haomeng/PycharmProjects/Minimal-IK/models.py", line 23, in __init__ params = pickle.load(f) UnicodeDecodeError: 'ascii' codec can't decode byte 0xd5 in position 2: ordinal not in range(128)

It seems that code need to run in python2.X version but another dependices called open3d couldn't run in python2.X.

Could you export your virtual environment and provide the .yaml file.
Thanks!

Performance comparison?

Have you compared with the SMPLify or SMPL-H performance?

Key error

Hi,

Thanks for the great work.
I try to run the example for smpl but there are key errors.
In the code we load these parameters:

` self.J_regressor = params['J_regressor']

  self.skinning_weights = params['skinning_weights']

  self.mesh_pose_basis = params['mesh_pose_basis'] # pose blend shape
  self.mesh_shape_basis = params['mesh_shape_basis']
  self.mesh_template = params['mesh_template']

  self.faces =  params['faces']

  self.parents = params['parents']`

But in the basicModel_f_lbs_10_207_0_v1.0.0.pkl model I only have these parameters:
dict_keys(['J_regressor_prior', 'f', 'J_regressor', 'kintree_table', 'J', 'weights_prior', 'weights', 'vert_sym_idxs', 'posedirs', 'pose_training_info', 'bs_style', 'v_template', 'shapedirs', 'bs_type'])

Loos like only J_regressor is matching. Could you please give me some pointers about it?

How to visualize the .obj file?

Hi,

I successfully ran the example.py and got gt.obj and est.obj. But I didn't see where I can visualize them.

After searching I found that I can visualize it use open3d, but when I load the .obj file it throw error:

import copy
import open3d as o3d

print("Testing mesh in open3d ...")
mesh = o3d.io.read_triangle_mesh("est.obj")
print(mesh)

It gives:
Testing mesh in open3d ... Read geometry::TriangleMesh failed: unknown file extension.

Any suggestions? Thanks!

Issues making this work with anything other than keypoints directly from mano mesh

I've been trying to get this IK solver to work with the FreiHand data but so far haven't been unable to. I wanted to see if there was something that I am doing wrong. Is there any sort of preprocessing needed to keypoints before being fed into the solver?

I have managed to get it working no problem when using the keypoints provided by the mesh

_, keypoints = mesh.set_params(pose_pca=pose_pca, pose_glb=pose_glb, shape=shape)

and using the solver I get a really good result. But when using the XYZ keypoints from the FreiHand dataset I get the following

I have done the following things. First I made sure that I had the joints in the correct order following the MANO convention rather than MPII that FreiHand is in. So

class MANOHandJoints:
    n_joints = 21

    labels = [
        'W',  # 0
        'I0', 'I1', 'I2',  # 3
        'M0', 'M1', 'M2',  # 6
        'L0', 'L1', 'L2',  # 9
        'R0', 'R1', 'R2',  # 12
        'T0', 'T1', 'T2',  # 15
        'I3', 'M3', 'L3', 'R3', 'T3'  # 20, tips are manually added (not in MANO)
    ]

instead of this

class MPIIHandJoints:
    n_joints = 21

    labels = [
        'W',  # 0
        'T0', 'T1', 'T2', 'T3',  # 4
        'I0', 'I1', 'I2', 'I3',  # 8
        'M0', 'M1', 'M2', 'M3',  # 12
        'R0', 'R1', 'R2', 'R3',  # 16
        'L0', 'L1', 'L2', 'L3',  # 20
    ]

Secondly I made sure to scale the coordinates to be in millimeters rather than meters. Do I also need to mean center and normalize similar to how it's done in IKNet? I tried this as well but still was not getting good results. Any help would be greatly appreciated!