yyheart/lip-sync-wav2lip

Creating an AI model that is proficient in lip-syncing i.e. synchronizing an audio file with a video file.

LIP-SYNC-WAV2LIP

ABOUT

Creating an AI model that is proficient in lip-syncing i.e. synchronizing an audio file with a video file.

Base Repository - Wav2Lip

Pretrained Model Download

Model	Description	Link to the model
Wav2Lip	Highly accurate lip-sync	Link
Wav2Lip + GAN	Slightly inferior lip-sync, but better visual quality	Link

Lip-syncing videos using the pre-trained models (Inference)

You can lip-sync any video to any audio:

python inference.py --checkpoint_path <ckpt> --face <video.mp4> --audio <an-audio-source> 

The result is saved (by default) in results/result_voice.mp4. You can specify it as an argument, similar to several other available options. The audio source can be any file supported by FFMPEG containing audio data: *.wav, *.mp3 or even a video file, from which the code will automatically extract the audio.

Input Files

• Audio Input_Audio

• Video Input_Video

  • Note

    • Make sure all the frames in the video must have a face in it , other wise the model will return No face detected Error

Output File

• Final Video result_video

IMPROVEMENTS

We can bring significant Improvement in the output by adjusting the arguments bringing the change in below variations

• Variations

  • NO SMOOTH ARGUMENT
  • PADDING ARGUMENT
  • RESIZE FACTOR

ISSUES

• The Pretrained model will only work if all the frames in the video must have a face in it

• --box method unknown

  • Raised the issue - Rudrabha/Wav2Lip#42 (comment)