pengfeiren96/dir

[ICCV 2023 Oral] Decoupled Iterative Refinement Framework for Interacting Hands Reconstruction from a Single RGB Image

dir's Issues

Bug in the code: mano_para_left assigned to right

Is this a bug in the code or am I missing something?

DIR/models/dir.py

Line 292 in 329e381

'pd_mano_para_right': mano_para_left,

Training results are very poor.

Thank you very much for making your training code public.
I used your default config file to train the model, just modify the batch_size to 256. The final results achieved 81+ for MPJPE and 88+ for MPVPE.
I don't know why the results are so bad.

[11/04 02:22:49] Training INFO: [Epoch 49/50][Batch 300/357][lr 0.000000][loss_seg: 0.0541][loss_dense: 0.0002][loss_lovasz: 0.0125][loss_joint_left_uv_0: 0.0055][loss_joint_right_uv_0: 0.0054][loss_mesh_left_uv_0: 0.0071][loss_mesh_right_uv_0: 0.0075][loss_joint_left_xyz_0: 0.0066][loss_joint_right_xyz_0: 0.0064][loss_mesh_left_xyz_0: 0.0080][loss_mesh_right_xyz_0: 0.0082][loss_edge_left_0: 0.0135][loss_edge_right_0: 0.0138][loss_normal_left_0: 0.0294][loss_normal_right_0: 0.0300][loss_offset_0: 0.0033][loss_joint_left_uv_1: 0.0052][loss_joint_right_uv_1: 0.0052][loss_mesh_left_uv_1: 0.0069][loss_mesh_right_uv_1: 0.0068][loss_joint_left_xyz_1: 0.0065][loss_joint_right_xyz_1: 0.0063][loss_mesh_left_xyz_1: 0.0079][loss_mesh_right_xyz_1: 0.0078][loss_edge_left_1: 0.0135][loss_edge_right_1: 0.0137][loss_normal_left_1: 0.0292][loss_normal_right_1: 0.0291][loss_offset_1: 0.0031][loss_joint_left_uv_2: 0.0051][loss_joint_right_uv_2: 0.0050][loss_mesh_left_uv_2: 0.0068][loss_mesh_right_uv_2: 0.0067][loss_joint_left_xyz_2: 0.0065][loss_joint_right_xyz_2: 0.0063][loss_mesh_left_xyz_2: 0.0079][loss_mesh_right_xyz_2: 0.0078][loss_edge_left_2: 0.0135][loss_edge_right_2: 0.0136][loss_normal_left_2: 0.0291][loss_normal_right_2: 0.0291][loss_offset_2: 0.0031]
[11/04 02:24:36] Training INFO: Save checkpoint to ./checkpoints/DIR/checkpoint/latest.pth
[11/04 02:30:25] Training INFO: MPJPE_0: left 80.77075399604499 mm, right 81.86647319326214 mm, AVG 81.31861359465356 mm
[11/04 02:30:25] Training INFO: MPVPE_0: left 87.17533539907605 mm, right 89.05994439241933 mm, AVG 88.11763989574769 mm
[11/04 02:30:25] Training INFO: MPJPE_1: left 80.8181789283659 mm, right 82.71075034258412 mm, AVG 81.76446463547501 mm
[11/04 02:30:25] Training INFO: MPVPE_1: left 87.29970373359382 mm, right 89.37457580776776 mm, AVG 88.33713977068078 mm
[11/04 02:30:25] Training INFO: MPJPE_2: left 81.22921638629016 mm, right 83.13988862084408 mm, AVG 82.18455250356712 mm
[11/04 02:30:25] Training INFO: MPVPE_2: left 87.71276979469785 mm, right 89.84211043399922 mm, AVG 88.77744011434854 mm

code

When to release the complete code

Pretrained Model Weights

Hello, thank you for your inspiring work! :)
Could you also share the model weights used for your experiments in the paper?

Test code on KPT method

Thanks for your excellent job!
In your table 3 , you tested KPT method under MCP alignment with and without scale alignment. Can you share your testing code of this?

Questions about visualizing results

When I visualized with the provided model, result was a list of four elements corresponding to the three stages and others in the original text, but when visualized, I found that for stage1 (result[1]), and stage2(result[2]), no matter what image I input, mesh effect is the same, I would like to ask why this is, in addition, init_out (result[0]) effect is very good, hope to get an answer, thank you!

Coordinate problem

When training a model, don’t we need to subtract the root joint from the model’s output (out)?

Why are GT and OUT inconsistent?

pixel joint mean error and joint mean error

DIR:

joint mean error:
    left: 10.74602734297514 mm, right: 9.60523635149002 mm
    all: 10.17563184723258 mm
vert mean error:
    left: 10.49137581139803 mm, right: 9.40467044711113 mm
    all: 9.94802312925458 mm
pixel joint mean error:
    left: 6.332123279571533 mm, right: 5.808280944824219 mm
    all: 6.070201873779297 mm
pixel vert mean error:
    left: 6.235969543457031 mm, right: 5.725381851196289 mm
    all: 5.98067569732666 mm
root error: 28.983158990740776 mm

IntagHand:

joint mean error:
    left: 8.93425289541483 mm, right: 8.663229644298553 mm
    all: 8.798741269856691 mm
vert mean error:
    left: 9.173248894512653 mm, right: 8.890160359442234 mm
    all: 9.031704626977444 mm

pixel vert mean error and pixel joint mean error is pixel error in 2D space, maybe the MPJPE and MPVPE in the table 2 of DIR paper should are 10.175 and 9.9480.
This is just my personal opinion, please correct me if there are any mistakes.