menghao666/hand-bmc-pytorch

PyTorch reimplementation of hand-biomechanical-constraints (ECCV2020)

Hand Biomechanical Constraints Pytorch

Unofficial PyTorch reimplementation of Hand-Biomechanical-Constraints (ECCV2020).

This project reimplement following components :

1. 3 kinds of biomechanical soft constraints
2. integrate BMC into training procedure (PyTorch version)

Usage

• Retrieve the code

git clone https://github.com/MengHao666/Hand-BMC-pytorch
cd Hand-BMC-pytorch

• Create and activate the virtual environment with python dependencies

conda env create --file=environment.yml
conda activate bmc

Download data

Download 3D joint location data joints.zip Google Drive or Baidu Pan (2pip), and . These statistics are from following datasets:

• Rendered Handpose Dataset
• GANerated Hands Dataset
• STB Dataset
• Freihand Dataset

Note the data from these datasets under their own licenses.

Calculate BMC

Run the code

python calculate_bmc.py

You will get

• bone_len_max.npy bone_len_min.npy for bone length limits
• curvatures_max.npy curvatures_min.npy for Root bones' curvatures
• PHI_max.npy PHI_min.npy for Root bones' angular distance
• joint_angles.npy for Joint angles

And if u want to check the coordinate system, run the code

cd utils
python calculate_joint_angles.py

• red ,green, blue arrows refer to X,Y,Z of local coordinate system respectively;
• dark arrows refer to bones;
• pink arrows refer to bone projection into X-Z plane of local coordinate system;

One view	Another view

Run the code

python calculate_convex_hull.py

You will get CONVEX_HULLS.npy, i.e. convex hulls to encircle the anatomically plausible joint angles.

And you will also see every convex hull like following figure:

• "Bone PIP" means the bone from MCP joint to PIP joint in thumb
• flexion and abduction is two kinds of angle describing joint rotation
• "ori_convex_hull" means the original convex hull calculated from all joint angle points
• "rdp_convex_hull" means convex hull simplified by the Ramer-Douglas-Peucker algorithm, a polygon simplification algorithm
• "del_convex_hull" means convex hull further simplified by a greedy algorithm
• "rectangle" means the minimal rectangle to surround all joint angle points

Run the code

python plot.py

You will see all the convex hulls

Integrate BMC into training (PyTorch version)

Run the code

python weakloss.py

Experiment results

To check influence of BMC, instead of reimplementing the network of origin paper, I integrate BMC into my own project,

Train and evaluation curve

(AUC means 3D PCK, and ACC_HM means 2D PCK)

3D PCK AUC Diffenence

Dataset	DetNet	DetNet+BMC
RHD	0.9339	0.9364
STB	0.8744	0.8778
DO	0.9378	0.9475
EO	0.9270	0.9182

Note

• Adjusting training parameters carefully, longer training time might further boost accuracy.
• As BMC is a weakly supervised method, it may only make predictions more physically plausible,but cannot boost AUC performance strongly when strong supervision is used.

Limitation

• Due to time limitation, I didn't reimplement the network and experiments of original paper.
• There is a little difference between original paper and my reimplementation. But most of them match.

Citation

This is the unofficial pytorch reimplementation of the paper "Weakly supervised 3d hand pose estimation via biomechanical constraints (ECCV 2020).

If you find the project helpful, please star this project and cite them:

@article{spurr2020weakly,
  title={Weakly supervised 3d hand pose estimation via biomechanical constraints},
  author={Spurr, Adrian and Iqbal, Umar and Molchanov, Pavlo and Hilliges, Otmar and Kautz, Jan},
  journal={arXiv preprint arXiv:2003.09282},
  volume={8},
  year={2020},
  publisher={Springer}
}

hand-bmc-pytorch's Issues

Error when batch size=1

File "/data_sdc/gzh/tmp/neural_render_learn/losses/BMC_loss.py", line 355, in
loss_total, loss_dict = bmc.compute_loss(joints)
File "/data_sdc/gzh/tmp/neural_render_learn/losses/BMC_loss.py", line 326, in compute_loss
TIP_X_axis = torch.matmul(temp_R, DIP_X_axis.unsqueeze(3)).squeeze()
IndexError: Dimension out of range (expected to be in range of [-3, 2], but got 3)

Question about results of using BMC Loss in supervised 3d hand pose estimation

Hi, thanks for your work. But I have some problems about BMC Loss.
I want to add BMC Loss in my experiment which estimates 3d hand pose from a single gray image. But, it's hard to converge. When I add BMC Loss, my original 3d keypoints loss will rise, so I have to decrease BMC's loss_weight util 1e-9.
After training, I compare this experiment with BMC Loss and the baseline without BMC Loss. The baseline without BMC Loss performs better in MPJPE and it has lower BMC Loss in the same test dataset, which means BMC Loss has a negative effect in my experiment.
Is it normal I have this conclusion? Or what maybe wrong in my experiment?

Can't find the rdp algorithm

Hello,thank you for your sharing. When I run the 'python calculate_convex_hull.py', it could't find the 'rdp' module. Thus it will be very greatfull if you can help this.

Normalizing bone length by Reference Bone Length

Hi @MengHao666 , first of all thank you for your implementation.

I have a doubt with relation to the reference bone normalization in the BMCLoss Class. When you compute the min and max values for the bone lengths in the calculate_bmc.py code, all bones are normalized by the reference bone, as shown below:

Hand-BMC-pytorch/calculate_bmc.py

Lines 50 to 57 in d9d1a53

	# root-relative
	joints_root = np.expand_dims(joints[:, cfg.JOINT_ROOT_IDX, :], 1)
	joints = joints - joints_root
	# scale-invariant
	ref_bones = np.linalg.norm(joints[:, cfg.REF_BONE_LINK[0]] - joints[:, cfg.REF_BONE_LINK[1]], axis=1)
	ref_bones = np.expand_dims(ref_bones, [1, 2])
	joints = joints / ref_bones

But in the BMCLoss Class this step is not made, which could generate wrong results if the input data isn't normalized. Is this right?

Hand-BMC-pytorch/weakloss.py

Lines 253 to 278 in d9d1a53

	ALL_bones = [
	(
	joints[:, i, :] -
	joints[:, cfg.SNAP_PARENT[i], :]
	) for i in range(21)
	]
	ALL_bones = torch.stack(ALL_bones[1:], dim=1) # (B,20,3)
	ROOT_bones = ALL_bones[:, cfg.ID_ROOT_bone] # (B,5,3)
	PIP_bones = ALL_bones[:, cfg.ID_PIP_bone]
	DIP_bones = ALL_bones[:, cfg.ID_DIP_bone]
	TIP_bones = ALL_bones[:, cfg.ID_TIP_bone]
	ALL_Z_axis = normalize(ALL_bones)
	PIP_Z_axis = ALL_Z_axis[:, cfg.ID_ROOT_bone]
	DIP_Z_axis = ALL_Z_axis[:, cfg.ID_PIP_bone]
	TIP_Z_axis = ALL_Z_axis[:, cfg.ID_DIP_bone]
	normals = normalize(cross_product(ROOT_bones[:, 1:], ROOT_bones[:, :-1]))
	# compute loss of bone length
	bl_loss = torch.Tensor([0]).cuda()
	if self.lambda_bl:
	bls = two_norm(ALL_bones) # (B,20,1)
	bl_loss = interval_loss(value=bls, min=self.bl_min, max=self.bl_max)
	final_loss += self.lambda_bl * bl_loss
	BMC_losses["bmc_bl"] = bl_loss

Also, did you think what is the easiest way to use the algorithm for the left hand?

Thanks in advance.