hongwenzhang/danet-densepose2smpl

[TPAMI 2020] Learning 3D Human Shape and Pose from Dense Body Parts

✨ Research Highlights ✨

🚩 Human/Hand/Face/Full-body Motion Capture:

👉 DaNet [TPAMI 2020]:

• Learning 3D Human Shape and Pose from Dense Body Parts
  ✨ DensePose to SMPL; part-based regressor; more robust to occlusion

👉 PyMAF [ICCV 2021, Oral] & PyMAF-X [TPAMI 2023]:

• PyMAF: 3D Human Pose and Shape Regression with Pyramidal Mesh Alignment Feedback Loop
  ✨ better mesh-to-image alignment

• PyMAF-X: Towards Well-aligned Full-body Model Regression from Monocular Images
  ✨ an eXpressive version of PyMAF, supporting SMPL-X

👉 JVCR [TIP 2019]:

• Adversarial Learning Semantic Volume for 2D/3D Face Shape Regression in the Wild
  ✨ Joint voxel and coordinate regression

👉 ECT [TIFS 2018]:

• Combining Data-driven and Model-driven Methods for Robust Facial Landmark Detection
  ✨ Robust to occlusions and expressions

👉 More projects related to:

• clothed human reconstruction
• animatable avatar
• behavior understanding

danet-densepose2smpl's Issues

Question about female/neutral model

Really a great work!
I have some issue if I try to use a model different from SMPL male model. If I place female or neutral model, the reconstruction doesn't match with original image. I attach the example of a female pic with different result
here the image:

Here with male model (good):

here with female model:

here with neutral model:

Is it only an issue of visualization of the model shall be trained with different models?

Thanks,

Error of smplx

Hi, thanks for your released codes. When I running the demo.py, it cames some errors:

File "/ssd1/vis/lintianwei/3d/DaNet-3DHumanReconstruction-master/models/smpl.py", line 7, in
from smplx.body_models import ModelOutput
ImportError: cannot import name 'ModelOutput'

Do you know how to fix this?

how to obtain the h36m_mosh_train.npz

Hi!
I would like to retrain this code with the human3.6 dataset. However, I meet an error which h36m_mosh_train.npz can not be found.
Where can I get this file?
thanks!

Error running demo.py

Hi! I tried to run demo.py, however an error occurred as follows:

Traceback (most recent call last):
  File "demo.py", line 184, in <module>
    main()
  File "demo.py", line 149, in main
    global_orient=rotmat_pred[:, 0].unsqueeze(1), pose2rot=False)
  File "/home/yujun/anaconda3/envs/danet/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/yujun/ludai/DaNet-3DHumanReconstruction/models/smpl.py", line 29, in forward
    smpl_output = super(SMPL, self).forward(*args, **kwargs)
  File "/home/yujun/anaconda3/envs/danet/lib/python3.6/site-packages/smplx/body_models.py", line 376, in forward
    self.lbs_weights, pose2rot=pose2rot, dtype=self.dtype)
  File "/home/yujun/anaconda3/envs/danet/lib/python3.6/site-packages/smplx/lbs.py", line 179, in lbs
    v_shaped = v_template + blend_shapes(betas, shapedirs)
  File "/home/yujun/anaconda3/envs/danet/lib/python3.6/site-packages/smplx/lbs.py", line 269, in blend_shapes
    blend_shape = torch.einsum('bl,mkl->bmk', [betas, shape_disps])
  File "/home/yujun/anaconda3/envs/danet/lib/python3.6/site-packages/torch/functional.py", line 211, in einsum
    return torch._C._VariableFunctions.einsum(equation, operands)
RuntimeError: size of dimension does not match previous size, operand 1, dim 2

Can you give me any help regarding this issue? Thanks.

RuntimeError: faces must be contiguous?

Hi, thanks for your code. But when I run python demo.py --checkpoint=data/pretrained_model/danet_model_h36m_itw.pt --img_dir ./examples --use_opendr, this error was reported.

/home/pisceskkk/miniconda3/envs/danet/lib/python3.6/site-packages/torch/nn/functional.py:4004: UserWarning: Default grid_sample and affine_grid behavior has changed to align_corners=False since 1.3.0. Please specify align_corners=True if the old behavior is desired. See the documentation of grid_sample for details.
"Default grid_sample and affine_grid behavior has changed "
/home/pisceskkk/miniconda3/envs/danet/lib/python3.6/site-packages/torch/nn/functional.py:4066: UserWarning: Default grid_sample and affine_grid behavior has changed to align_corners=False since 1.3.0. Please specify align_corners=True if the old behavior is desired. See the documentation of grid_sample for details.
"Default grid_sample and affine_grid behavior has changed "
Traceback (most recent call last):
File "demo.py", line 184, in
main()
File "demo.py", line 151, in main
render_iuv = iuv_renderer.verts2uvimg(verts_pred, camera_pred)
File "/mnt/d/Study/DaNet-3DHumanReconstruction/utils/renderer.py", line 276, in verts2uvimg
dist_coeffs=torch.FloatTensor([[0.] * 5]).to(verts.device))
File "/home/pisceskkk/miniconda3/envs/danet/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/home/pisceskkk/miniconda3/envs/danet/lib/python3.6/site-packages/neural_renderer_pytorch-1.1.3-py3.6-linux-x86_64.egg/neural_renderer/renderer.py", line 74, in forward
return self.render_rgb(vertices, faces, textures, K, R, t, dist_coeffs, orig_size)
File "/home/pisceskkk/miniconda3/envs/danet/lib/python3.6/site-packages/neural_renderer_pytorch-1.1.3-py3.6-linux-x86_64.egg/neural_renderer/renderer.py", line 197, in render_rgb
self.background_color)
File "/home/pisceskkk/miniconda3/envs/danet/lib/python3.6/site-packages/neural_renderer_pytorch-1.1.3-py3.6-linux-x86_64.egg/neural_renderer/rasterize.py", line 394, in rasterize
faces, textures, image_size, anti_aliasing, near, far, eps, background_color, True, False, False)['rgb']
File "/home/pisceskkk/miniconda3/envs/danet/lib/python3.6/site-packages/neural_renderer_pytorch-1.1.3-py3.6-linux-x86_64.egg/neural_renderer/rasterize.py", line 303, in rasterize_rgbad
image_size, near, far, eps, background_color, return_rgb, return_alpha, return_depth)(*inputs)
File "/home/pisceskkk/miniconda3/envs/danet/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/home/pisceskkk/miniconda3/envs/danet/lib/python3.6/site-packages/neural_renderer_pytorch-1.1.3-py3.6-linux-x86_64.egg/neural_renderer/rasterize.py", line 251, in forward
self.return_rgb, self.return_alpha, self.return_depth)
File "/home/pisceskkk/miniconda3/envs/danet/lib/python3.6/site-packages/neural_renderer_pytorch-1.1.3-py3.6-linux-x86_64.egg/neural_renderer/rasterize.py", line 74, in forward
face_inv_map)
File "/home/pisceskkk/miniconda3/envs/danet/lib/python3.6/site-packages/neural_renderer_pytorch-1.1.3-py3.6-linux-x86_64.egg/neural_renderer/rasterize.py", line 166, in forward_face_index_map
ctx.return_depth)
RuntimeError: faces must be contiguous

And my env:
Ubuntu20 in WSL2
python 3.6.13
torch 1.10.0+cu113
neural-renderer-pytorch 1.1.3
opendr 0.78

what's wrong with this?

When would you public the code?

It's an interesting work, I can't wait to read your code. Please release it as soon as possible!

Questions about uv map

Hi, thank you for you awesome work! I am a beginner of computer vision and computer graphics, and I have some confusion about uv map.
1: How to construct the template texture map when you prepare IUV maps?
If I understand correctly, in the texture map of fig2, the horizontal and vertical axis are uv coordiantes, and the value in the pixel is the SMPL vertex coordinate corresponding to uv, so how to construct the texture map? And why the layout of body parts in this texture map is different from the layout of texture map in Densepose paper？Is there an explicit correspondence between uv coordinates and vertex coordinates？Like u,v=f(x,y,z)

Running on different size images?

Hi, thanks for your paper and code. I'm trying to make it run on bigger images, kind of struggling with it. Do you think it will be easy or needs major code and model changes? Thanks

Could I reconstruct my picture to different shape?

What a great work.
I want to reconstruct my person in pictures to 2 different shape( a fat mesh and a thin mesh), how can I make it?
Thanks a lot

How do you prepare and render partial IUV image?

How do you prepare and render both global IUV images and partial IUV ground truth?

For example: how do you use
https://github.com/HongwenZhang/DaNet-3DHumanReconstruction/blob/master/utils/part_utils.py
and how to get the masks for rendering?

Some files cannot be found

Thanks for your excellent work!
I meet an error about some files when running your demo.py.
J_regressor_extra.npy, J_regressor_h36m.npy and smpl_mean_params.npy cannot be found.
I would like to know where I can get these files.
Thanks again!

Issue with dimmensions

Hi
I am working on your model and having issue while running the code as i have all dependencies issue but still i cannot figure out the issue.

I have tried with both the SMPL models like version 1.1.0 and 1.0.0 but it still having dimensions issue

can you please help me in figuring out the issue

ERROR:

using decomposed predictor.
input mode: IUV
/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py:2693: UserWarning: Default grid_sample and affine_grid behavior will be changed to align_corners=False from 1.4.0. See the documentation of grid_sample for details.
warnings.warn("Default grid_sample and affine_grid behavior will be changed "
/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py:2751: UserWarning: Default grid_sample and affine_grid behavior will be changed to align_corners=False from 1.4.0. See the documentation of grid_sample for details.
warnings.warn("Default grid_sample and affine_grid behavior will be changed "
Traceback (most recent call last):
File "demo.py", line 184, in
main()
File "demo.py", line 149, in main
global_orient=rotmat_pred[:, 0].unsqueeze(1), pose2rot=False)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 541, in call
result = self.forward(*input, **kwargs)
File "/content/drive/My Drive/Ayush_Gupta/3d_cloth/DaNet-3DHumanReconstruction/models/smpl.py", line 29, in forward
smpl_output = super(SMPL, self).forward(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/smplx/body_models.py", line 376, in forward
self.lbs_weights, pose2rot=pose2rot, dtype=self.dtype)
File "/usr/local/lib/python3.7/dist-packages/smplx/lbs.py", line 179, in lbs
v_shaped = v_template + blend_shapes(betas, shapedirs)
File "/usr/local/lib/python3.7/dist-packages/smplx/lbs.py", line 265, in blend_shapes
blend_shape = torch.einsum('bl,mkl->bmk', [betas, shape_disps])
File "/usr/local/lib/python3.7/dist-packages/torch/functional.py", line 201, in einsum
return torch._C._VariableFunctions.einsum(equation, operands)
RuntimeError: size of dimension does not match previous size, operand 1, dim 2