fantaosha/scope

Reviatalizing Optimization for 3D Human Pose and Shape Estimation: A Sparse Constrained Formulation

Reviatalizing Optimization for 3D Human Pose and Shape Estimation: A Sparse Constrained Formulation

This is the implementation of the approach described in the paper:

Taosha Fan, Kalyan Vasudev Alwala, Donglai Xiang, Weipeng Xu, Todd Murphey, Mustafa Mukadam. Reviatalizing Optimization for 3D Human Pose and Shape Estimation: A Sparse Constrained Formulation. In IEEE/CVF International Conference on Computer Vision (ICCV), 2021.

More demos are available at https://sites.google.com/view/scope-human

Install

git clone https://github.com/fantaosha/SCOPE.git
cd SCOPE
export SCOPE_ROOT=$(pwd)
mkdir build
cd build
cmake ../C++
make -j4

Usage

SMPL Model

Download the SMPL model and extract these.pkl files to $SCOPE_ROOT/model.

Preprocess

cd $SCOPE_ROOT/model
python3 preprocess.py smpl_male.npz YOUR_SMPL_MALE.pkl

Run

cd $SCOPE_ROOT
./build/bin/run --model ./model/smpl_male.npz --prior ./model/joint_prior.json --keypoint ./examples/keypoints.json --result ./examples/results.json

Dataset

2D and 3D Keypoints

The 2D and 3D keypoints estimates from AlphaPose and VideoPose3D can be downloaded from Google Drive.

2D Keypoint Index

0: nose
1: left eye
2: right eye
3: left ear
4: right ear
5: left upper arm
6: right upper arm
7: left elbow
8: right elow
9: left wrist
10: right wrist
11: left hip
12: right hip
13: left knee
14: right knee
15: left ankle
16: right ankle
17: head top
18: thorax
19: middle hip
20: left big toe
21: right big toe
22: left small toe
23: right small toe
24: left heel
25: right heel
26: chest
27: neck

3D Keypoint Index

0: middile hip
1: left hip
2: left knee
3: left ankle
4: right hip
5: right knee
6: right ankle
7: chest
8: thorax
9: neck
10: head top
11: left upper arm
12: left elbow
13: left wrist
14: right upper arm
15: right elbow
16: right wrist

Citation

@article{fan2021revitalizing,
title={Revitalizing Optimization for 3D Human Pose and Shape Estimation: A Sparse Constrained Formulation},
author={Fan, Taosha and Alwala, Kalyan Vasudev and Xiang, Donglai and Xu, Weipeng and Murphey, Todd and Mukadam, Mustafa},
journal={Proceedings of the IEEE/CVF International Conference on Computer Vision},
year={2021}
}

scope's Issues

Render the result using `results.json`

Hi, I just came across the paper and attempted to reproduce the results.

I successfully followed your guide in README.md but am unsure about what to do with result.json to render the 3D model. Could you provide me with some guidance? Thanks in advance.

Questions about 2d keypoints.

1. Did you apply Halpe-FullBody to extract 2d keypoints?
2. How to get keypoints thorax and chest ?

How to do inference using SMPL-H model?

Thank you for your great work!
I've seen that you have showed the results of applying your method on smpl-h. Could you please tell me how to use smpl-h for inference rather than smpl.
THANK YOU VERY MUCH !!!!!

error: ‘Matrix3X’ is not a member of ‘Eigen’

This is a wonderful job!
The following error occurred during compilation:
SCOPE/C++/scope/utils/Setup.cpp:195:36: error: ‘Matrix3X’ is not a member of ‘Eigen’; did you mean ‘Matrix3Xd’?
195 | Measurements = Eigen::Map<Eigen::Matrix3X>(
| ^~~~~~~~
| Matrix3Xd

Excuse me, is there something wrong with the Eigen version I installed?
What is the correct Eigen version?

Could you please release the code of training the joint prior from a scratch?

Thank you for sharing this awesome work!
I was trying to reproduce the normalizing flow used in the joint prior parts, and I noticed that the network structure (e.g. the PReLU implementation) might be different from the RealNVP, or Low-capacity version of normalizing flows mentioned in https://arxiv.org/abs/2003.10350
Could you please release the method or the code for training this network?