amtsai96/learning-lip-sync-from-audio

Learning Lip Sync of Obama from Speech Audio

Learning-Lip-Sync-from-Audio

• This model is a part (audio to mouth shape) of implementation of 《Synthesizing Obama: Learning Lip Sync from Audio》 (https://grail.cs.washington.edu/projects/AudioToObama/siggraph17_obama.pdf)

• Demo Video: https://youtu.be/CczXKZ7eyX8

learning-lip-sync-from-audio's Issues

您好，已经得到嘴部的landmark，如何合成嘴部到完整的图片呢？

您好，已经得到嘴部的landmark，如何合成嘴部到完整的图片呢？ @amtsai96

How do I train with my own dataset ?

Hi, your work seems amazing. I am looking to train for my own dataset with only one character.
Can you please share detailed instructions for training ? Like preprocessing, video fps required, etc ?

Thank You.

How to generate the pickle piles

in the pca and audio_kp folder

ValueError: Error when checking input: expected lstm_1_input to have shape (100, 26) but got array with shape (30, 26)

I tried to use model = load_model('lb50_ep150_v10\my_model.h5') however, I'm seeing the below error.

Traceback (most recent call last):
File "run.py", line 167, in
y_pred = model.predict(X)
File "C:\Users\Girish\anaconda3\envs\deepfake\lib\site-packages\keras\engine\training.py", line 1441, in predict
x, _, _ = self._standardize_user_data(x)
File "C:\Users\Girish\anaconda3\envs\deepfake\lib\site-packages\keras\engine\training.py", line 579, in _standardize_user_data
exception_prefix='input')
File "C:\Users\Girish\anaconda3\envs\deepfake\lib\site-packages\keras\engine\training_utils.py", line 145, in standardize_input_data
str(data_shape))
ValueError: Error when checking input: expected lstm_1_input to have shape (100, 26) but got array with shape (30, 26)

Can someone please tell me how I can resolve this error?

How can I get good results like yours?

Hi Amada Tsai!
I trained your model with your code but my result is not good as yours. The lip in my result is not correct to the sound.. Could you please tell my how many video you trained? (I see in your code you config the number of video is 20).
Here is my result
https://www.youtube.com/watch?v=lo90EKTDdV0

about your object

hi,sir ，Is your code finished?

How do I obtain the original face with lip sync after the lip points are generated?

I am able to generate the lip sync points for my input. How do I remove the black box and lip animation to obtain the original face with lip sync?