toymaker222/humanbehavioranimation

Human Behavior Animation

This project explores the human behavior animation, including gesture, etc., as part of my graduate research at Peking University, supervised by Libin Liu.

---

SIGGRAPH Asia 2022 (Technical Best Paper Award)
Rhythmic Gesticulator: Rhythm-Aware Co-Speech Gesture Synthesis with Hierarchical Neural Embeddings
_{Tenglong Ao, Qingzhe Gao, Yuke Lou, Baoquan Chen, Libin Liu, ACM Trans. Graph. 41, 6, Article 209.}

Automatic synthesis of realistic co-speech gestures is an increasingly important yet challenging task in artificial embodied agent creation. In this work, we present a novel co-speech gesture synthesis method that achieves convincing results both on the rhythm and semantics. For the rhythm, our system contains a robust rhythm-based segmentation pipeline to ensure the temporal coherence between the vocalization and gestures explicitly. For the gesture semantics, we devise a mechanism to effectively disentangle both low- and high-level neural embeddings of speech and motion based on linguistic theory. The high-level embedding corresponds to semantics, while the low-level embedding relates to subtle variations. Lastly, we build correspondence between the hierarchical embeddings of the speech and the motion, resulting in rhythm and semantics-aware gesture synthesis.

- Video (YouTube | Bilibili) - Paper (arXiv) - Code (github) - Dataset (github) -
- Explained (YouTube(English) | 知乎(Chinese)) -

Acknowledgement

The layout of this project is highly inspired by AI4Animation repo.

Copyright Information

This project is only for research or education purposes, and not freely available for commercial use or redistribution.