nicholasli1995/evoskeleton

Official project website for the CVPR 2020 paper (Oral Presentation) "Cascaded deep monocular 3D human pose estimation wth evolutionary training data"

evoskeleton's Issues

Query on Keypoint order for Inference in the wild

Hey, in the inference.py file, it reads the example_annot.npy for annotations. Can you please tell me which indices correspond to which keypoints? Thanks!

Question about the initial population

Hello. I want to ask some questions about initial 3d body-parts
I saw that the original 3d poses are loaded from the file "threeDPose_train.npy", and each sample has 96 dimensions.

I wonder what's the exact physical meaning of these values? Are they the absolute distance (use mm as the unit) in the local coordinate, where the center is the pelvis?

evolved dataset projection issue

Hi, I used your evolved program to evolved my own dataset with the default parameters, my own datset has 161 samples, and the evolved result is 419, but after I project the evolved result(419) to 2D and check its xy position, the valid(if min(xy) <0, invalid) 2D pose is only 47.

My dataset(world coordinate) is not in the meter unit like Human36M, does this affect my evolved result?

My dataset only has 17 joints, the other joints I set the value with 1, does this affect my evolved result?

In my understanding you only check the valid of 3D pose(world coordinate), and use all the evolved data to train, can this evolved strategy guarantee your 2D projection is valid?

Thank you very much

camera params in cameras.npy

how to get cameras.npy, is it provided in the offical website?

Obtain 3D skeleton with 2D key-points as inputs using SMPLify by own data

Thank you so much for providing the code also now I met the problem when I want to obtain a 3D skeleton with 2D keypoint.
The error is like this when I run the code
"python fit_3d.py -dataset_dir anno -model_dir smplify/models/basicModel_neutral_lbs_10_207_0_v1.0.0.pkl"
Traceback (most recent call last):
File "fit_3d.py", line 99, in
main(opt)
File "fit_3d.py", line 62, in main
do_degrees=do_degrees
File "/home/panda/EvoSkeleton/libs/annotator/smplify/fit_3d.py", line 451, in run_single_fit
viz=viz)
File "/home/panda/EvoSkeleton/libs/annotator/smplify/fit_3d.py", line 120, in initialize_camera
center = np.array([img.shape[1] / 2, img.shape[0] / 2])
AttributeError: 'NoneType' object has no attribute 'shape'

It shows that it can not use my 2D input keypoint. Many thanks

Input of the plot_distribution in anglelimits.py

Hi, how can I obtain the H_temp?

About one of the pre-trained models

Hello,

Thank you for sharing this great work, very solid and helpful!

I would like to ask about the pre-trained model provided under 'Inference Example' in: https://github.com/Nicholasli1995/EvoSkeleton/blob/master/docs/TRAINING.md. The G-drive link is: https://drive.google.com/file/d/158oCTK-9Y8Bl9qxidoHcXfqfeeA7qT93/view

May I know which data do you use to train this model, as well as the parameters? I find it works perfectly in your proposed U3DPW dataset, while an EvoSkeleton model that I trained using synthetic H3.6m can only work on some of the example. Therefore, I wonder if this is because of the training data.

Many thanks,
Shiyang

generating other poses from a known pose

Is there a way to use your code to generate all or comment poses from a given image?

AttributeError: 'NoneType' object has no attribute 'keys'

Hello, the following error occurred when I was running the training code. May I ask what the problem is and how to solve it?

Missing "data" & "resource" folder?

I do not see any folder named "data" or "resources" inside the EvoSkeleton folder for the separately downloaded constraints and Human3.6 folders to be in mentioned in https://github.com/Nicholasli1995/EvoSkeleton/blob/master/docs/HHR.md

about training data

The paper said there are 8M 2d-3d pairs, but the total number of the released h36m training data pairs is about 390k before augmentation, is that the correct number?

So if I guess right, that means you only take one h36m camera for each action instead of using 4 cameras for each action.

I don't know whether my guess is correct, so I post this issue to make sure.

Thank you in advance.

Cannot use the annotation tool

After open the 2D annotation tool, the picture will show but cannot be labelled. Only the shortcut command for quit will work, the others won't. The 3D annotation cannot initiate for the missing file "jointAngleModel_v2.npy". Will you please re-upload the full package of this annotation tool, or did I ignore something?

2D Pose Estimation

Hi, Is there a possibility to do 2D pose estimation of an arbitrary image with the current code that is uploaded? the inference code is uploading skeleton from a fixed, predefined dictionary of annotation....
Thanks very much in advance
Best regards,
Leila

2D to 3D with own data

hi
I tried your 'inference.py' module with own 2d data, but the result has some problems：


The reason is that your output need to use human3.6's 3d_mean and std to decode, but my data will produce errors if I use this decoding method, So I want to ask you how can I solve this kind of problem.
thank you very much！

Running annotation_2D in py3 and fit_3d in py2

Hi,

As far as I understand, we first run annotation_2D.py in py3 environment, get the annotation.npy. Then we run the fit_3d.py in py2 environment to get the fitting parameters using smpl tool. However, fit_3d.py under py2 cannot load the annotation.npy since it is generated in py3 environment from annotation_2D.py. The error is shown as followed:

File "/anaconda3/envs/py2/lib/python2.7/site-packages/numpy/lib/format.py", line 701, in read_array
  array = pickle.load(fp, **pickle_kwargs)

ValueError: unsupported pickle protocol: 3

Am I missing something? Thanks in advance!

weird 3D Pose

I ran into a strange problem: lengthen the length of thigh and shorten the length of calf in the 2D Pose, the angle between the thigh and calf is different from normal in the output 3D pose.

eval on 3DHP

nice code, nice paper, nice job!
my question is :
Can you provide the code for evaluating the 3DHP datasets,(e.x. pck3d & AUC calculation) or share other repositories that have the eval. code

run question

hello,I want to ask Where can I see the data I evolved, and what exactly does the evolved_path refer to? (python 2Dto3Dnet.py -train True -num_stages 2 -ws True -ws_name "S1" -twoD_source "HRN/synthetic" -evolved_path "YourDataPath")
Can you give me an example? Thank you very much!

Question about U3DPW and H36M

Hi, thank you very much for your excellent job!

Could you please explain how to use U3DPW as an augmented dataset for H36M? The keypoints on the H36M dataset and the U3DPW dataset are different. The visualization results of the 3d skeleton on two datasets are not totally the same.

Thank you very much for your help!

2D detection results on MPI_INF_3DHP dataset

Hi, thank you for your great work!

Could you please provide the pre-processed 2D detections and 3D poses for MPI_INF_3DHP test set. As such, we can evaluate the model generalizability under cross-dataset setup.

Thank you!

AttributeError: module 'libs.utils' has no attribute 'utils'

Im training the network with 2Dto3Dnet.py with the cmd:
python 2Dto3Dnet.py -train True -num_stages 2 -ws True -ws_name "S1" -twoD_source "HRN/synthetic" -evolved_path "YourDataPath"
How ever it show the error:
Traceback (most recent call last):
File "2Dto3Dnet.py", line 16, in
import libs.utils.utils as utils
File "../libs/utils/utils.py", line 4, in
import libs.dataset.h36m.data_utils as data_utils
File "../libs/dataset/h36m/data_utils.py", line 15, in
import libs.utils.utils as utils
AttributeError: module 'libs.utils' has no attribute 'utils'
Anyone can help me
Many thanks

the input channel order to_spherical calculation

hello, thanks for your outstanding work. I am learning the evolution part and had some questions about the to_spherical calculation.

in EvoSkeleton/libs/skeleton/anglelimits.py line179-192 is the function of to_spherical(xyz), and I have one questions:

1. according to the code, before calling to_spherical(), the skeleton has been re_order to xzy, while the formula of to_spherical use the xyz order, so if the skeleton in xzy order input to to_spherical, can we still get the right result?

Result is a bit strange

Follow your guide, I generated some new data and trained the model with the data.
here is the command:
evolve python evolve.py -generate True -WS True -SS "S1"
train python 2Dto3Dnet.py -train True -num_stages 3 -ws True -ws_name "S1" -twoD_source "synthetic" -evolved_path "/home/hcy/project/3d_human_pose/EvoSkeleton-master/data/human3.6M/evolved/S1/evolved_data/generation_13.npy"
After data enhancement, two files were generated , generation_1.npy and generation_13.npy. I trained with the latter one. The result is a bit strange. After 200 epochs, you test the model with the test data.The picture below shows the test results.

I didn't change any code of your project. But the result is better than 50.5 which you showed on your paper .
I am curious why this happens.

nothing

arrow keyboard doesn't work (Fixed: for matplotlib > 3)

Hi,

Great work!

When I try the 3D Skeleton Interactive Annotation using the pre-trained weights, the keyboard is not working correctly. 0-9 are fine, the figure shows the different selected limb when I click. However, arrow keyboards are not working:

Traceback (most recent call last):
  File "/anaconda3/envs/pytorch/lib/python3.6/site-packages/matplotlib/cbook/__init__.py", line 224, in process
    func(*args, **kwargs)
  File "annotate_3D.py", line 96, in press
    update_skeleton(angle_idx, event.key)
  File "annotate_3D.py", line 337, in update_skeleton
    update_global(angle_idx)
  File "annotate_3D.py", line 282, in update_global
    update_line(line_idx, parent_idx, child_idx)
  File "annotate_3D.py", line 265, in update_line
    lines[line_idx][0].set_3d_properties(z)
  File "/anaconda3/envs/pytorch/lib/python3.6/site-packages/mpl_toolkits/mplot3d/art3d.py", line 143, in set_3d_properties
    zs = np.broadcast_to(zs, xs.shape)
AttributeError: 'list' object has no attribute 'shape'

up, down, right or left all shows the same error. I tried debug thru printing the (event.key) and it shows that the received keyboard is correct (i.e. printing up, down, left, right when I click the corresponding key).

Could you please take a look on this? Thanks a lot!

What is S15678

Thanks for your nice work, the paper is a nice read.

I do not understand this: "We use S15678 as our initial population". What is S15678?

2d poses from 3d pose

hi
how can I extract 2d poses from the resulted 3d pose (2d pose front view, back view, and side view)
thanks

Do you plan to publish part of the HRnet code?

Do you plan to publish part of the HRnet code?

projection from 3d pose to 2d

Hello. Thanks for your excellent work.
I wonder after you obtain the 3d keypoint during evolution, do you randomly assume the camera location, and project the 3d keypoint to target the cameras? Does your project have the demo example?

About the coordinate system

Hi. I want to consult something about the coordinate system.
I saw that before evolution, there is a function called to_local(), which "represents the bone vectors in the local coordinate systems"

EvoSkeleton/libs/skeleton/anglelimits.py

Line 285 in f31e7c2

def to_local(skeleton):

I feel confused about this part. Will the coordinate be converted from camera coordinate to local coordinate? And what's definition of local coordinate and what's the meaning of the conversion?
BTW, it will be my honour if you are willing to provide me with some references.
I am looking forward ton your reply.

How to calculate stats for examples/inference.py? (e.g. mean_3d, std_3d dim_ignore_3d)

In examples/inference.py, how do I calculate stats passed to unNormalizeData() from evo_skeleton.dataset.h36m.data_utils import unNormalizeData, as I am using my own images, and can't create a new stats.npy for my images. I would eventually like to pass video to it too.

Thanks in advance

training input size and inference input size not match

Hi, in your training scirpit h36m2Dpose.py, after loading the dataset, the input shape is (34, ), but in the inference.py, the input size is (32, ), why is that?

The best way to correct 3D pose for multi-person dataset

Hi,

Thanks for the great work.

I have a dataset contains multi person's 3D pose. Some of them are not correct so I wanted to use your tool to correct those pose.

I'm not sure the best way of using the tools. This is what I have in my mind:

1. Crop the single person image using the bounding box from a multi person image.
2. Since I already have the 2D pose annotations, I will skip the 2D annotation, but do the 3D parameter fitting.
3. Then I will do the 3D annotations to correct the pose.
4. Merge every single person's result to the multi-person image.

Something that's not clear in my mind:

In step 1, I have all the 2D and 3D annotations for multiple person. However, they are global. If I crop to a single person image, should I change the pose annotations to local? Also since I crop the image from the bounding box, the size of every image will be different. Will that be a problem? I'm not sure what is the best way of doing it. It seems a lot of works.

Thanks in advance for answering my questions!

AttributeError: module 'libs.dataset.h36m' has no attribute 'data_utils'

Hi

Thx for this model. but i always get this error. Do you know whats the problem?

Regarding the 2D HPE model

Hello,

I am trying to use your pre-trained 2D pose model to estimate 17 joints.
May I know which architecture do you use for it, Stacked Hourglass? Also, what data do you use to train the model, are they 2D joints from Human3.6m?

Thanks,
Shiyang

Location of the hip/pelvis joint?

Does the 2d-to-3d lifting model actually predict a location for the hip/pelvis joint? OR, does the model predict all of the other 16 joints in relation to the hip joint?

It seems like when we run UnNormalizeData in examples/inference.py, the first joint location (which corresponds to the hip) is always [0,0,0] because we are choosing to ignore that joint in stats['dim_ignore_3d']. Is there a way to avoid setting the hip to (0,0,0)?

In my use case, I would like to avoid "fixing" or "pinning" a joint to the origin (or any other arbitrary point) for rendering purposes.

about jointAngle file

not upload jointAngleModel_v2 file

TODO: Model Zoo for pre-trained models under different settings

It seems that the pre-trained model in examples.zip is weakly-supervised trained because it yields mpjpe of 50mm on GT 2D pose.
Where can I find the fully-supervised trained model, to reproduce the results 34.5 in Tab.3 in protocol P1*?

Thanks in advance.

Data Preprocessing when test on 3dhp dataset

After training the model on the H36m dataset, if we need to evaluate on the 3dhp dataset, how to preprocess the 3dhp dataset? Considering that there are differences in the definitions of joint for both datasets (no nose joint in 3dhp), should we delete the nose joint in H36m dataset during the model training process (Input size: 16 * 2 Output size: 15 * 3).
And there are two types of 3D joints coordinates in 3dhp dataset, which one should we use during evaluating, should we delete the nose joint (16 * 3), align the keypoints to the root joint and delete the root joint (15 * 3) ?
Should we first resize the scale of estimated 3dhp 2D coordinates and then normalizing the 2D keypoints using H36m stats? And what about the 3dhp 3D keypoints? Are there any other details that need to attention?

Hello ask a question

could you please let me know whether the code for Heatmap Regression Model(Stage 1) parts already exists with the provided code？

I wonder why your model is "cascaded"

When I think about 'cascaded network', I come up with a network of modules connected in consecutive order, just like cascaded pyramid network(CPN).
In your implementation, every stage gets 2D skeleton inputs, and generates 3D skeleton estimations.
And the estimation loss is just the difference between the target and the sum of their estimations.
Therefore, I think that your 2D-to-3D estimation network is composed of multiple 2D-to-3D estimation models bagged together.
Of course, the gradient flows through every stage.
But the latter stage does not get the output of the former stage, so they are not connected sequentially.

Inconsistency between pretrained HRNet 2D detector and twoDPose_HRN.npy

Hi, appreciate for your excellent work and comprehensively technical detail release! I believe this would be great effort to 3DHPE field.

One question. I encountered some issues when inference HRNet model (I mean the 2D detector) that loads the pretrained weight, given cropped h36m images.

1. Accuracy in the first screenshot shows that the 2D average error (pixel) is around 7, which is inconsistent with reported 4.4.

2. Meanwhile, I print the 2D pose prediction for the frame of (9, 'Directions', 'Directions 1.54138969.h5-sh') from the model inference result, and from released twoDPose_HRN_test.npy.
   The inconsistence appears again as shown in the uploaded 2nd image.

Could you help me get rid of the unexpected situation ?
Did I miss something, or may you release another high-acc pretrained HR model ?

Many thanks !

ImportError: No module named libs.hhr.config

I'm getting this error when I run python ./examples/h36m2Dpose.py

How to make 2d pose estimation output .npy?

Hello.
I want to run in the wild data using inference. but I don't know how to make 2d pose estimation output (.npy file).
According to what I understand, For making npy file, I should use h36m2Dpose.py or annotate_2D.p, right?

Can you explain how?

thank you.

Training with less joints

Hello again,

Sorry for another issue!

I was trying to modify your code to train with 13 joints (instead of 17) in fully supervised manner, so far it seems to run. There are a few things I would like to consult you: (1) should I use lesser number of stages and blocks? (2) should I train with lower learning rate or fewer epochs? (3) Will the removal of head, thorax and spine affect the performance a lot? (4) I can see that you remove nose in your training, may I know why?

I really appreciate your help! Thanks in advance!

Shiyang

how to inference with wild image

hello, I want to use this model to predict some my wild picture . what should I do. How to get these 2d info of my picture.

I can't find the model

Hello, after reading your code, I found that the model file which should under the libs is missing.

Is all of the mpi data used?

happy spring festival,

I wonder about the details when you evaluate the model on the MPI-inf-3DHP dataset.

did you use all of the 8 subjects && 2 sequences && 14 cameras to inference the model?

Resulted 2D pose of the mentioned HR-Net on Human 3.6M

Hi,

Thanks for your great work and code. I noticed that you released the pre-trained model of HR-Net for 2D pose estimator, but we still need the crop step, which may lead to an inconsistent 2D result. Could you please directly provide the estimated 2D pose (it may be in .npy format and shouldn't be too large) from the HR-Net on Human 3.6M? I think it will be very helpful for the community.

Test with a single image

Hi, is it possible to perform inference from a single image chosen by me? I've noticed that, inside the examples folder, there is a file called "inference.py" but it uses the file "example_annot.npy" that contains the pre-calculated 2D key-points of the test images you say to download. Will a version of the inference code be released that will allow a single image (thus different from the ones you propose) to be used to test the network?

Thanks