jyunlee/im2hands

Im2Hands: Learning Attentive Implicit Representation of Interacting Two-Hand Shapes (CVPR 2023)

Im2Hands (Implicit Two Hands)

Im2Hands: Learning Attentive Implicit Representation of Interacting Two-Hand Shapes (CVPR 2023)

Jihyun Lee, Minhyuk Sung, Honggyu Choi, Tae-Kyun (T-K) Kim

[Project Page] [Paper] [Supplementary Video]

---

CVPR 2023 Materials: [Presentation Video] [Poster]

We present Implicit Two Hands (Im2Hands), the first neural implicit representation of two interacting hands. Unlike existing methods on two-hand reconstruction that rely on a parametric hand model and/or low-resolution meshes, Im2Hands can produce fine-grained geometry of two hands with high hand-to-hand and hand-to-image coherency. To handle the shape complexity and interaction context between two hands, Im2Hands models the occupancy volume of two hands - conditioned on an RGB image and coarse 3D keypoints - by two novel attention-based modules responsible for initial occupancy estimation and context-aware occupancy refinement, respectively. Im2Hands first learns per-hand neural articulated occupancy in the canonical space designed for each hand using query-image attention. It then refines the initial two-hand occupancy in the posed space to enhance the coherency between the two hand shapes using query-anchor attention. In addition, we introduce an optional keypoint refinement module to enable robust two-hand shape estimation from predicted hand keypoints in a single-image reconstruction scenario. We experimentally demonstrate the effectiveness of Im2Hands on two-hand reconstruction in comparison to related methods, where ours achieves state-of-the-art results.

 

Environment Setup

Clone this repository and install the dependencies specified in requirements.txt.

 git clone https://github.com/jyunlee/Im2Hands.git
 mv Im2Hands
 pip install -r requirements.txt 

Also, install im2mesh library of Occupancy Networks.

 

Data Preparation

1. Download InterHand2.6M dataset from its official website. Set the value of path variable in your config file as the path of annotation directory of InterHand2.6M.
2. Follow the data pre-processing steps of HALO for sampling query points and the corresponding ground truth occupancy values. For modifying prepare_data_from_mano_param_keypoints.py file of HALO for InterHand2.6M dataset, you can refer to this file. (Sorry that this code is currently a bit messy; I will try to refactor it when I have more time.)
3. Follow the data pre-processing steps of IntagHand (dataset/interhand.py) to obtain normalized two-hand images. Set the value of img_path variable in your config file as the path of the resulting data directory.
4. Place your hand keypoint files - either the ground truth keypoints of InterHand2.6M or predicted keypoints extracted using e.g. IntagHand - under pred_joints sub-directory under your pre-processed data directory.

 

Initial Occupancy Estimation Network

Network Training

Place the pre-trained weights of HALO and IntagHand (halo_baseline.pt and intaghand_baseline.pth) under out/init_occ directory. These files can be also downloaded from this Google Drive link.

Then, Run init_occ_train.py to train your own Initial Occupancy Estimation Network.

 python init_occ_train.py 

Network Inference

Run init_occ_generate.py to generate the initial per-hand shapes.

 python init_occ_generate.py 

 

Two-Hand Occupancy Refinement Network

Network Training

Place the weights of Initial Occupancy Estimation Network (init_occ.pt) under out/ref_occ directory. The pre-trained weights can be also downloaded from this Google Drive link.

Then, run ref_occ_train.py to train your own Refined Occupancy Estimation Network.

 python ref_occ_train.py 

For quantitative evaluation, please refer to eval_meshes.py script of HALO.

Network Inference

Run ref_occ_generate.py to generate the refined two-hand shapes.

 python ref_occ_generate.py 

 

[Optional] Keypoint Refinement Network

Network Training

Place the pre-trained weights of IntagHand (intaghand_baseline.pth) under out/kpts_ref directory. This file can be also downloaded from this Google Drive link.

Also, place (1) your initial hand keypoint files (e.g. predicted using IntagHand) under pred_joints_before_ref sub-directory and (2) the ground truth keypoints of InterHand2.6M under gt_joints sub-directory of your pre-processed data directory.

Then, Run kpts_ref_train.py to train your own Input Keypoint Refinement Network.

 python kpts_ref_train.py 

Network Inference

Run kpts_ref_generate.py to save the refined two-hand keypoints.

 python kpts_ref_generate.py 

For quantitative evaluation, please refer to apps/eval_interhand.py script of IntagHand.

 

Citation

If you find this work useful, please consider citing our paper.

@InProceedings{lee2023im2hands,
    author = {Lee, Jihyun and Sung, Minhyuk and Choi, Honggyu and Kim, Tae-Kyun},
    title = {Im2Hands: Learning Attentive Implicit Representation of Interacting Two-Hand Shapes},
    booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
    year = {2023}
}

 

Acknowledgements

• Our code is based on HALO, IntagHand, and AIR-Nets, and our model parameters are partially initialized from their pre-trained weights. We thank the authors of these three inspiring works.
• We also thank the authors of InterHand2.6M for the useful dataset.

im2hands's Issues

About the use of real scenarios

Hello, if I want to use in a real scene, using the camera to capture real-time video stream, how do I prepare the data? Because I found that the input is not pure picture input, thank you for your excellent work, looking forward to your answer!

Format for `pred_joints` data preparation

Hi,

For Step 4 of the Data Preparation section, could you please let me know the desired format for 3D hand joint positions? I am using GT InterHand2.6M data which is in this format.

If my understanding is correct, I need to save it in the reverse format for each hand, starting with wrist, Pinky1, Pinky2, Pinky3, Pinky4, Ring1, etc. in a .ply file. Could you please confirm?

Thank you.

pose==0.2dev

ERROR: Could not find a version that satisfies the requirement pose==0.2dev (from versions: none)
ERROR: No matching distribution found for pose==0.2dev

GPU Usage

Thank you for sharing your work! I have a question regarding the training setup:

What GPUs did you use to train the model?
How much GPU memory is required to train the model?

Thank you in advance for your assistance! Best regards.

Inference weights

Could you share the trained inference checkpoints from your article?