liuyyy111/egohmr

[ICRA2023] This is the official repository for EgoHMR

EgoHMR

[ICRA 2023] Repo for our paper "EgoHMR: Egocentric Human Mesh Recovery via Hierarchical Latent Diffusion Model" [PAPER]

Abstract

We propose Egocentric Human Mesh Recovery (EgoHMR), a novel hierarchical network based on latent diffusion models. Our method takes a single egocentric frame as the input and it can be trained in an end-to-end manner without supervision of 2D pose. The network is built upon the latent diffusion model by incorporating both global and local features in a hierarchical structure. To train the proposed network, we generate weak labels from synchronized exocentric images. The proposed method can perform human mesh recovery directly from egocentric images and detailed quantitative and qualitative experiments have been conducted to demonstrate the effectiveness of the proposed EgoHMR method.

Contributions

1. We propose an end-to-end hierarchical diffusion probabilistic model for human mesh recovery from an egocentric image captured by a camera looking downwards.
2. Our method skips the inaccurate and tedious process for egocentric 2D pose estimation, and achieves the comparable results with state-of-the-art methods which require 2D cues as supervision.
3. Quantitative and qualitative results on different datasets demonstrate the effectiveness of our proposed hierarchical network for both human mesh recovery and human pose estimation.

Brief Introduction

PreVideo-v1-short.mp4

Dataset

• ECHP Dataset [link]
• EgoPW Dataset [link]

Code

• Part of the code is adapted from [Prohmr] [Stable Diffusion]

Citation

If you find our paper/dataset/code useful, please consider citing:

@INPROCEEDINGS{liuegohmr,
  author={Liu, Yuxuan and Yang, Jianxin and Gu, Xiao and Guo, Yao and Yang, Guang-Zhong},
  booktitle={2023 IEEE International Conference on Robotics and Automation (ICRA)}, 
  title={EgoHMR: Egocentric Human Mesh Recovery via Hierarchical Latent Diffusion Model}, 
  year={2023},
  volume={},
  number={},
  pages={9807-9813},
  doi={10.1109/ICRA48891.2023.10161247}}