zengwang430521/decomr

Repository for the paper " 3D Human Mesh Regression with Dense Correspondence "

decomr's Issues

Data pre-processing about generating UV map

May I ask what is the physical meaning behind line 45-46, is that orthographic projection? or transformation to UV space?
https://github.com/zengwang430521/DecoMR/blob/master/datasets/preprocess/generate_gt_iuv.py#L45

and
https://github.com/zengwang430521/DecoMR/blob/master/datasets/preprocess/generate_gt_iuv.py#L52
why do you have '1 - uv_tmp[:, :, 0]'

Additionally, I agree it is continuous among different semantic body parts in your UV map, which makes the learning easier.
Is the frontal view and back view of the same finger continuous in your defined UV map? as in Fig 3 in original paper.

AttributeError: 'ColoredRenderer' object has no attribute 'vbo_verts_face'

Traceback (most recent call last):
  File "demo.py", line 140, in <module>
    img=img, use_bg=True, body_color='pink')
  File "/data/DecoMR/utils/renderer.py", line 351, in render
    return self.renderer.r
  File "/home/xumingkai/.local/lib/python3.6/site-packages/chumpy/ch.py", line 555, in r
    self._call_on_changed()
  File "/home/xumingkai/.local/lib/python3.6/site-packages/chumpy/ch.py", line 550, in _call_on_changed
    self.on_changed(self._dirty_vars)
  File "/home/xumingkai/.local/lib/python3.6/site-packages/opendr/renderer.py", line 1080, in on_changed
    self.vbo_verts_face.set_array(np.array(self.verts_by_face).astype(np.float32))
AttributeError: 'ColoredRenderer' object has no attribute 'vbo_verts_face'

Want to get the Human3.6m IUV Map

Hi,
Thanks the great work! You propose the Boundary Free IUV Map to replace the DensePose IUV Map. I agree you said the BF IUV Map is more continuous than the DensePose IUV. So I want to use BF IUV Map. But you only provide the 56x56 pred_IUV Map, It is not accurate for me. I want to tranining the generating BF IUV Map model to satisfy my needed. Could you provide the BF IUV Map Datasets?

Thanks very much!

Will the code of constructing new UV map / new obj file be released

Will the code of constructing new UV map / new obj file be released?

how to generate smpl boundry free image?

@zengwang430521 I have seen the smpl boundry free image in the paper and I want to generate it. But I did not find the part that generates smpl boundry free image in the code. Any help would be greatly appreciated.

Some questions about SMPL parameter of Human36m

1、if i use the SMPL parameter generated by SMPLify-X. Should i use the 'trans' matrix genereted by SMPLify-X of SMPL to get the 'sampled_vertices' , or could i use SMPL(pose, betas, trans) to get the 'sampled_vertices' ?
2、And how do we get the camera coordinates from joints calaculated from 'sampled_vertices' generated by SMPL(). use 'self.smpl.get_train_joints(sampled_vertices)' ? but it never use 'J_regressor_h36m.npy'

I've just been in touch with this research direction, and I'm looking forward to your reply. Thank you

THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=383 error=11 : invalid argument

when i try to running the demo ,then it happened,i don't know about this question ,but i install all the environments, please help me,thank you，i really need to running it .

Missing file 'data/basicModel_f_lbs_10_207_0_v1.0.0.pkl'

the file is missing

Missing file 'up_3d_trainval.npz'

Missing file 'up_3d_trainval.npz' when I run
'python preprocess_datasets.py --gt_iuv'

I do not see the 'Bboxes' in original H36M website, can you also provide the 'Bboxes'

Theory behind ' Ensure the neighboring pixels of vt on the UV map are meaningful'

Hi Wang,
I am trying to understand the bary-weight calculation process.
Basically you are trying to mapping the uv space faces to xyz space faces and want to solve the weight between them.
However, I start to get lost from https://github.com/zengwang430521/DecoMR/blob/master/models/uv_generator.py#L179
and fail to understand why and what is the theory/intuition behind that about
the calculation of w, u, v?

I know from https://github.com/zengwang430521/DecoMR/blob/master/models/uv_generator.py#L205 you are trying to construct a mask in UV space, but the lines before that seems confusing to me.

License?

Hi Zeng Wang,

Thanks for this great work! What is the license for this repository? If you could add it, that would be greatly appreciated. Thanks in advance.

FYI - here's what happens if a repository has no license:
"You're under no obligation to choose a license. However, without a license, the default copyright laws apply, meaning that you retain all rights to your source code and no one may reproduce, distribute, or create derivative works from your work."

Best,
Saafke

A potential problem of warp_feature

Hi @zengwang430521 ,
I noticed that in your implementation of the warp_feature, a single element in the output transferred F_{uv} could potentially correspond to multiple elements in the local feature F_{im}.
For example, the u,v coordinate of a element in F_{uv} is [16,16]. While in the IUV image, the u,v coordinates are of float type. Then, rounding is required to create the correspondence, and thus multiple locations in the IUV image (e.g., an element with [u,v] = [16.1, 16.6] and another element with [u,v] = [16.4, 16.7]). Then two locations in F_{im} correspond to the single location in the F_{uv}.

Therefore, I am wondering which feature vectors will be transferred?

Best.

help of predict about camera parameter?

hello,i find we predict the camera parameter[z,tx,ty],and then we get the wold coordinate[tx,ty,2×focal_length/img_size×z],and we get this picture just use the world coordinate in render,

now,i want to use pytorch3d to replace the original renderer,but in pytorch3d,the setting in camera should know the R and T,but we only konw the world coordinate [tx,ty,2×focal_length/img_size×z],can you help me,how can i use pytorch3d to get the same img(There can be no background image)

Bent knee of the reconstructed model

Reconstructed 3d model has bent knees, although the person in the image is standing straight.
Can anyone tell why this is happening???

Network initialization

I search around and only find resnet is initialized, may I ask how is other part like 'ConvBottleNeck' initialized ?

DecoMR/models/layers.py

Line 42 in 69b5986

class ConvBottleNeck(nn.Module):

can i get body25 keypoints (25,3) from smpl mesh (6780,3)

ImportError: /home/ni/anaconda3/lib/python3.7/site-packages/neural_renderer/cuda/load_textures.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZN6caffe26detail37_typeMetaDataInstance_preallocated_32E

when i run the eval.py on the lsp_test dataset.

`/home/ni/anaconda3/bin/python /home/ni/DecoMR/eval.py --checkpoint=data/model/h36m_up3d/checkpoints/h36m_up3d.pt --dataset=lsp --log_freq=20
Checkpoint loaded
data loader finish
Traceback (most recent call last):
File "/home/ni/DecoMR/eval.py", line 318, in
run_evaluation(model, args, options, args.dataset, args.log_freq)
File "/home/ni/DecoMR/eval.py", line 122, in run_evaluation
from utils.part_utils import PartRenderer
File "/home/ni/DecoMR/utils/part_utils.py", line 7, in
import neural_renderer as nr
File "/home/ni/anaconda3/lib/python3.7/site-packages/neural_renderer/init.py", line 3, in
from .load_obj import load_obj
File "/home/ni/anaconda3/lib/python3.7/site-packages/neural_renderer/load_obj.py", line 8, in
import neural_renderer.cuda.load_textures as load_textures_cuda
ImportError: /home/ni/anaconda3/lib/python3.7/site-packages/neural_renderer/cuda/load_textures.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZN6caffe26detail37_typeMetaDataInstance_preallocated_32E

Process finished with exit code 1`

Do you know the cause of this problem

Camera coordinate joint prediction?

May I ask is the pred_vertices in camera coordinate?

DecoMR/demo.py

Line 134 in 726c7a1

pred_vertices = pred_vertices[0].cpu().numpy()

and will the following operations make it in root-relative coordinate and in mm unit? so that we can use the pred_keypoints_3d to calculate MPJPE with the ground truth in root-relative coordinate?

pred_keypoints_3d = smpl.get_train_joints(pred_vertices)[:, joint_mapper]
pred_pelvis = (pred_keypoints_3d[:, [2]] + pred_keypoints_3d[:, [3]]) / 2
pred_keypoints_3d = pred_keypoints_3d - pred_pelvis

How to reconstruct mesh based on the vertex location

May I ask how do you reconstruct SMPL parameters given the vertex location (location map) from your work?
And what is the shape of the location map? (is that a 3 channel xyz denoting the camera coordinate in mm?)

something wrong with resampling uv map

Given smpl shape and pose parameters, I used get_UV_map(verts) and directly resampled the UV map generated from that function. The result is like this:


I have tried 'SMPL' and 'BF'. How to fix this problem and obtain a smooth mesh?

(u,v) coordinates corresponding to mesh vertices

Hi,

Thanks for the great work. Do you have the (u,v) coordinates corresponding to SMPL mesh vertices in your continuous UV map? Or a script to obtain the same?

Thanks

what is the meaning of tv_loss?

definition of tv_loss:

DecoMR/train/base_trainer.py

Line 246 in 69b5986

def tv_loss(self, uv_map):

it is with weight 0.0001 when training on up-3d.
I am not sure about the meaning of the loss.

Will this loss only apply only at uv_res 128 or less? If I have a uv_res at 256, does that mean we can get rid of the loss?

how to generate uv map preoposed in this paper on my own datasets

@zengwang430521 Hi, how to generate uv map preoposed in this paper on my own datasets? Any help would be greatly appreciated.

Why do you assume that all target meshes are female in SURREAL?

I run your evaluation code on SURREAL test set (493 imgs), and found out that the dataloader code only provides female genders.
Why don't you use the groundtruth genders which are provided in the dataset for the correct evaluation?

When I test your pre-trained model on SURREAL test set with groundtruth genders, I get
MPJPE: 67.82
MPVPE (Surface error): 64.59

These numbers are far different from the numbers reported in the paper (MPVPE: 56.5 , Tab.3) and not much good compared to the accuracy of BodyNet (MPVPE (Surface error): 65.8).

When I test your pre-trained model on SURREAL test set with intact code, I get
MPJPE: 54.29
MPVPE (Surface error): 51.65

Do you have any idea to get the numbers you reported in the paper?
And I think assuming all target meshes as female is inappropriate and unfair to compare to BodyNet results.

Question about the coordinate system of location map

Hi, thanks for your great work!

I have a question about the location map. Specifically, I want to know which is the coordinate system w.r.t. the 3 channel(xyz) of the pixel in the location map. It seems be defined in the camera coordinate rather than the world coordinate, am I right? If so, can you tell me why not use the world coordinate?

Thanks!

How does SMPL UV map turn into BF(boundry-free) UV map ？

Hi，

Thank you for such a great job. I am very interested in the new UV map you have proposed. I want to know how SMPL UV map can be converted to this BF UV map.

Thanks.

dp_loss weight meaning?

May I ask what is the meaning of the weight in line 276-280
in https://github.com/zengwang430521/DecoMR/blob/master/train/base_trainer.py#L276?

Is this weighting on whether the u channel should penalize more or v channel should penalize more?
or
it is each pixel and channel weight for activation larger than 0?

Will this weight make overall loss smaller?

and why the UV head needs Sigmoid? only to normalize the activation to 0~1? will this stabilize training and make it converge faster?

DecoMR/models/dense_cnn.py

Line 128 in f96526b

nn.Sigmoid())

Thanks!

How to preprocess SURREAL?

Hi @zengwang430521 , nice work! But there is no script that preprocesses SURREAL. Could you share it?