jyunlee/interhandgen

InterHandGen: Two-Hand Interaction Generation via Cascaded Reverse Diffusion (CVPR 2024)

🤝 InterHandGen

InterHandGen: Two-Hand Interaction Generation via Cascaded Reverse Diffusion (CVPR 2024)

Jihyun Lee¹, Shunsuke Saito², Giljoo Nam², Minhyuk Sung¹, Tae-Kyun (T-K) Kim^1,3

¹ KAIST, ² Codec Avatars Lab, Meta, ³ Imperial College London

[Project Page] [Paper] [Supplementary Video]

We present 🤝InterHandGen, a novel framework that learns the generative prior of two-hand interaction. Sampling from our model yields plausible and diverse two-hand shapes in close interaction with or without an object. Our prior can be incorporated into any optimization or learning methods to reduce ambiguity in an ill-posed setup. Our key observation is that directly modeling the joint distribution of multiple instances imposes high learning complexity due to its combinatorial nature. Thus, we propose to decompose the modeling of joint distribution into the modeling of factored unconditional and conditional single instance distribution. In particular, we introduce a diffusion model that learns the single-hand distribution unconditional and conditional to another hand via conditioning dropout. For sampling, we combine anti-penetration and classifier-free guidance to enable plausible generation. Furthermore, we establish the rigorous evaluation protocol of two-hand synthesis, where our method significantly outperforms baseline generative models in terms of plausibility and diversity. We also demonstrate that our diffusion prior can boost the performance of two-hand reconstruction from monocular in-the-wild images, achieving new state-of-the-art accuracy.

 

Updates

[Apr 14th 2024] There was a bug in anti-penetration loss guidance weighting, and that part is now fixed. I am sorry for the inconvenience.

 

Environment Setup

1. Clone this repository and install the dependencies specified in requirements.txt.

 $ git clone https://github.com/jyunlee/InterHandGen.git
 $ mv InterHandGen
 $ pip install -r requirements.txt 

2. Install ChamferDistancePytorch.

 $ cd utils
 $ git clone https://github.com/ThibaultGROUEIX/ChamferDistancePytorch.git
 $ mv ChamferDistancePytorch/chamfer3D
 $ python setup.py install 

 

Data Preparation

1. Download InterHand2.6M dataset from its official website.

2. Follow the data pre-processing steps of IntagHand (dataset/interhand.py). Note that you only need the shape annotation files (anno/*.pkl), and you can skip the image preprocessing parts.

3. Download MANO model from its official website. Place the downloaded mano_v1_2 folder under misc directory.

    

Network Training

Train your own two-hand interaction diffusion model using the following command. Note that the pre-trained weights can be downloaded from this Google Drive link.

 $ CUDA_VISIBLE_DEVICES={gpu_num} python interhandgen.py --train 

 

Network Inference

Sample two-hand interactions from the trained model. The number of samples can be controlled by vis_epoch (number of iterations in sampling) and vis_batch (number of samples for each iteration) in the config file (configs/default.yml). For a full evaluation, set vis_epoch = 4 and vis_batch = 2500 to generate 4 * 2500 = 10000 samples.

 $ CUDA_VISIBLE_DEVICES={gpu_num} python interhandgen.py --model_path {trained_model_path} 

 

Evaluation

Compute the evaluation metrics using the sampled two-hand interactions.

 $ cd eval
 $ CUDA_VISIBLE_DEVICES={gpu_num} python evaluate.py --sample_num {number_of_samples} --doc {trained_model_dir} 

 

Citation

If you find this work useful, please consider citing our paper.

@inproceedings{lee2024interhandgen,
    title = {InterHandGen: Two-Hand Interaction Generation via Cascaded Reverse Diffusion},
    author = {Lee, Jihyun and Saito, Shunsuke and Nam, Giljoo and Sung, Minhyuk and Kim, Tae-Kyun},
    booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
    year = {2024}
}

 

Acknowledgements

• Parts of our code are based on DiffPose (forward/reverse diffusion process), Pointnet_Pointnet2_pytorch (feature extraction network for evaluation), MoCapDeform (anti-penetration guidance), and motion-diffusion-model (evaluation metrics). We appreciate the authors for releasing their codes.

interhandgen's Issues

dependency error

What is the version of Python you used? I got a dependency error for Python 3.10, 3.9. 3.8.

Pre-trained weights not compatible with the model

Dear authors, thank you for releasing the code.

Could you elaborate on which checkpoint corresponds to the released model?
I've tried all four released files, however none of them seems to be compatible with the model.
python interhandgen.py --model_path Im2Hands/ref_occ.pt
python interhandgen.py --model_path Im2Hands/init_occ.pt
python interhandgen.py --model_path Im2Hands/intaghand_baseline.pth
python interhandgen.py --model_path Im2Hands/halo_baseline.pt

manolayer about ARCTIC

Hi, @jyunlee! Thank you for sharing your code. I'm trying to use the ARCTIC dataset to visualize hand-object interactions through the differentiable manolayer (https://github.com/hassony2/manopth/blob/master/manopth/manolayer.py), but now I'm encountering some issues. I've noticed that you also use Manolayer, so I'd like to ask for your advice.

I am using the data from arctic_data/data/raw_seqs(https://github.com/zc-alexfan/arctic/blob/master/docs/data/README.md) and extracting hand and object parameters to visualize through manopth, but the hand is always displayed abnormally, and at the same time, the hand and object are intermingling.

I have been troubled by this question for a long time. I would like to know how you use manolayer to visualize data from ARCTIC ( and which part of the dataset) .or Is it feasible to directly use the manolayer you have written(https://github.com/jyunlee/InterHandGen/blob/main/utils/manolayer.py)?

run-time error (in computing)

Thanks for your code, when I run interhand.py I get a dimension mismatch, theoretically, there shouldn't be this error, how to fix it please, thanks!

ERROR - interhandgen.py - 2024-04-02 00:09:05,236 - Traceback (most recent call last):
File "/mnt/e/Lwork/two-hand/Code/InterHandGen/interhandgen.py", line 147, in main
runner.train()
File "/mnt/e/Lwork/two-hand/Code/InterHandGen/runners/diffhand.py", line 367, in train
data, gt_anchor_verts, gt_add_verts, gt_anchor_joints, gt_add_joints, anc_valid, add_valid = self.mano_forward_gt(data)
File "/mnt/e/Lwork/two-hand/Code/InterHandGen/runners/diffhand.py", line 272, in mano_forward_gt
add_handV = add_handV - add_handJ[:, 9, :].unsqueeze(1) + rel_trans.unsqueeze(1)
RuntimeError: The size of tensor a (778) must match the size of tensor b (3) at non-singleton dimension 1

GPU Usage

Thank you for sharing the great work! I have a question regarding the training setup:
What GPUs did you use for training the model? How much GPU memory is required to train the model?
Thank you for your time! Best regards.

IndexError: list index out of range

Thanks for your work, I'm getting the following error with Network Inference.

Penetration volume

It seems that there is an error in the calculation of the penetration volume metric. When directly inferring with the official pre-trained model you provided, the penetration volume metric yields 0. Could you please offer some guidance or suggestions on this matter?

Object-conditional generation

Hi, @jyunlee! Thank you for sharing your code. I'm trying to use the ARCTIC dataset for your code, and have some questions.

• How can you get the object point cloud? (I mean, how can you generate the object point cloud?)
• Do you have any plans to release the code regarding Object-conditional generation?