sibozhang/speech2video

Code for ACCV 2020 "Speech2Video Synthesis with 3D Skeleton Regularization and Expressive Body Poses"

Speech2Video

This is code for "Speech2Video Synthesis with 3D Skeleton Regularization and Expressive Body Poses". ACCV 2020. Project Page

Introduction

We propose a novel approach to convert given speech audio to a photo-realistic speaking video of a specific person, where the output video has synchronized, realistic, and expressive rich body dynamics. We achieve this by first generating 3D skeleton movements from the audio sequence using a recurrent neural network (RNN), and then synthesizing the output video via a conditional generative adversarial network (GAN). To make the skeleton movement realistic and expressive, we embed the knowledge of an articulated 3D human skeleton and a learned dictionary of personal speech iconic gestures into the generation process in both learning and testing pipelines. The former prevents the generation of unreasonable body distortion, while the later helps our model quickly learn meaningful body movement through a few recorded videos. To produce photo-realistic and high-resolution video with motion details, we propose to insert part attention mechanisms in the conditional GAN, where each detailed part, e.g. head and hand, is automatically zoomed in to have their own discriminators.

Data / Preprocessing

pretrained model Download

Citation

Speech2Video Synthesis with 3D Skeleton Regularization and Expressive Body Poses

Miao Liao*, Sibo Zhang*, Peng Wang, Hao Zhu, Xinxin Zuo, Ruigang Yang. PDF Result Video 1 min Spotlight 10 min Presentation

@inproceedings{liao2020speech2video,
  title={Speech2video synthesis with 3D skeleton regularization and expressive body poses},
  author={Liao, Miao and Zhang, Sibo and Wang, Peng and Zhu, Hao and Zuo, Xinxin and Yang, Ruigang},
  booktitle={Proceedings of the Asian Conference on Computer Vision},
  year={2020}
}

Ackowledgements

This code is based on the vid2vid framework.

speech2video's Issues

demo效果

看了demo，感觉人脸的口型和语音没有匹配上。动作比较机械化。

When would the code be released?

Appreciate for your work, when would the code be released?

is the database available for public research？

Dear sibo,

it is a wonderful job firstly.

Could you tell if the database is open for research?

thanks in advance!

the link of project page invalid

the link of project page invalid

When will the code be released and are there plans?

When will the code be released and are there plans?

When will the code be released?

Great Jobs! When will the code be released?

code release

hello, appricate your work!
when will the code and the pretrained models released? i am looking forward to them !

where are the codes ? how to collaborate and test your application locally?

where is the code?

where is the code?

Please share the code

@sibozhang , request you to please share the code. It will help us while working on our research work.