Creating an AI model that is proficient in lip-syncing i.e. synchronizing an audio file with a video file.
Base Repository - Wav2Lip
Model | Description | Link to the model |
---|---|---|
Wav2Lip | Highly accurate lip-sync | Link |
Wav2Lip + GAN | Slightly inferior lip-sync, but better visual quality | Link |
You can lip-sync any video to any audio:
python inference.py --checkpoint_path <ckpt> --face <video.mp4> --audio <an-audio-source>
The result is saved (by default) in results/result_voice.mp4
. You can specify it as an argument, similar to several other available options. The audio source can be any file supported by FFMPEG
containing audio data: *.wav
, *.mp3
or even a video file, from which the code will automatically extract the audio.
-
Audio Input_Audio
-
Video Input_Video
-
- Make sure all the frames in the video must have a face in it , other wise the model will return No face detected Error
-
- Final Video result_video
We can bring significant Improvement in the output by adjusting the arguments bringing the change in below variations
-
- NO SMOOTH ARGUMENT
- PADDING ARGUMENT
- RESIZE FACTOR
- The Pretrained model will only work if all the frames in the video must have a face in it
-
--box method unknown
- Raised the issue - Rudrabha/Wav2Lip#42 (comment)