huangjackson/v2vt

Video to video translation via few shot voice cloning & audio-based lip sync

Warning

Please note that this project is currently under active development and is not yet operational. Features may be incomplete, and functionality is not guaranteed. Development will be slow for a while as I am busy with classes & working on other projects.

v2vt

Video to video translation and dubbing via few shot voice cloning & audio-based lip sync.
See the demo »

Report Bug · Request Feature

Table of Contents

1. Features
2. Getting Started
3. Roadmap
4. Contributing
5. License
6. Contact
7. Acknowledgments

Features

demo.mp4

Currently supports English and Chinese

• Vocal isolation: Isolation of vocals from source video using deep neural networks
• Transcription: Transcription of source video via whisper
• Translation: Translation from source video via CTranslate2 and OPUS-MT
• Few-shot voice cloning: Realistic voice cloning and TTS with as little as 5 seconds of audio from source video
• Audio-based lip sync: Alter faces in source video to match translated audio

Getting Started

Currently only tested in a Windows 11 environment with Python 3.9, PyTorch 2.1.1, CUDA 11.8.

Prerequisites

• Python 3.9
• Anaconda (recommended)

Manual Installation

1. Clone the repo

   git clone https://github.com/huangjackson/v2vt.git
   cd v2vt

2. Create a conda environment (recommended)

   conda create -n v2vt python=3.9
   conda activate v2vt

3. Install ffmpeg

   conda install ffmpeg

4. Install PyTorch and CUDA

   conda install pytorch==2.1.1 torchvision==0.16.1 torchaudio==2.1.1 pytorch-cuda=11.8 -c pytorch -c nvidia

5. Install requirements from requirements.txt

   pip install -r requirements.txt

Usage

1. Navigate to directory

   cd v2vt

2. Run CLI

   python v2vt.py --help

Roadmap

Listed generally in order of priority:

• Vocal isolation
• Transcription
• Translation
• Voice cloning/TTS
  • *Match speed of original video (#3)
  • Multiple GPUs support
  • Support training & using multiple models
• Lip sync
  • *Support lip sync where face isn't always present in video (#1)
  • *Better face detection (#2)
  • Improve inference speed
• Additional languages (currently only en & zh)
• Improve overall speed
• Improve logging (#4)
• Create Colab
• Create live demo on HuggingFace

See the open issues for a full list of proposed features (and known issues).

Contributing

Any contributions are greatly appreciated.

If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement".

1. Fork the Project
2. Create your Feature Branch (git checkout -b feat/AmazingFeature)
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feat/AmazingFeature)
5. Open a Pull Request

License

Distributed under the MIT License. See LICENSE for more information. See individual files and folders for any other licenses credited.

Contact

Jackson Huang - [email protected]

Project Link: https://github.com/huangjackson/v2vt

Acknowledgments

Special thanks to the following people and projects:

• GPT-SoVITS
• video-retalking
• CTranslate2
• ultimatevocalremovergui
• KUIELab & Woosung Choi - For the original MDX-Net music demixing model
• KimberleyJensen - For the Kim Vocal 2 MDX-Net model
• Opus-MT - For translation models
• faster-whisper

v2vt's Issues

Improve logging

Is your feature request related to a problem? Please describe.
Logging style and method is not universal across project (e.g. tts uses Python standard logging module, lipsync uses print).

Describe the solution you'd like
Use same style and method throughout the entire project to improve user experience.

Improve face detection via RetinaFace

Is your feature request related to a problem? Please describe.
When a face isn't detected in a frame by GPEN/face_detect, LipSync inference fails (#1).
Often happens when face's resolution is either too large in the video, or has different proportions.

Describe the solution you'd like
Improve face detection using batched RetinaFace. Train model on larger dataset to improve detection for large faces.

Additional context
GPEN face detection code
OpenTalker/video-retalking#14 (comment)

Match speed of original video during TTS

Is your feature request related to a problem? Please describe.
The translated speech should be similar in speed and timing to the original video's speech so that speech and video content is relevant.

Describe the solution you'd like
During transcription, include timestamps. Use timestamps during TTS to ensure relative similarity in speed and timing.

Describe alternatives you've considered
Group audio by cuts (made in #1) or pauses in audio (end of sentence). Speed up/slow down output audio so that each group fits in the same amount of time as the original video.

LipSync inference fails when video doesn't have face in all frames

Describe the bug
When running lipsync inference with a video that doesn't have a face in all frames (or when GPEN/face_detect is unable to detect a face in all frames), error UnboundLocalError: local variable 'mask_sharp' referenced before assignment is thrown.

To Reproduce
Steps to reproduce the behavior:

1. Download video without a face in all frames (e.g. https://youtu.be/AT1bO_nlxHY?si=tBtVKsmc_N3eEPvC&t=78 at 1:18) as video.mp4
2. Run LipSyncInference with video, using audio from video (saved as audio.wav)

   from lipsync.inference import LipSyncInference
   lsi = LipSyncInference('./video.mp4', './audio.wav')
   lsi.run()

Expected behavior
Frames without a face are skipped (cut out during preprocessing?)

Screenshots/Logs

landmark Det:: 100%|████████████████████████████████████████████████▊| 1937/1943 [00:51<00:00, 149.12it/s]No face detected in this image
No face detected in this image
No face detected in this image
No face detected in this image
No face detected in this image
No face detected in this image
landmark Det:: 100%|██████████████████████████████████████████████████| 1943/1943 [00:51<00:00, 37.96it/s] 
[Step 2] Running 3DMM extraction: 100%|██████████████████████████████| 1943/1943 [00:13<00:00, 146.59it/s] 
Using expression center
Load checkpoint from: C:\Users\Jackson\Projects\v2vt\lipsync\checkpoints\DNet.pt
Load checkpoint from: C:\Users\Jackson\Projects\v2vt\lipsync\checkpoints\LNet.pth
Load checkpoint from: C:\Users\Jackson\Projects\v2vt\lipsync\checkpoints\ENet.pth
[Step 3] Stabilizing expression in video: 100%|███████████████████████| 1943/1943 [02:23<00:00, 13.57it/s] 
[Step 4] Loading audio - 1941 chunks
[Step 5] Enhancing reference frames:  94%|██████████████████████████▎ | 1823/1941 [04:40<00:18,  6.50it/s] 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\Jackson\Projects\v2vt\lipsync\inference.py", line 345, in run
    pred, _, _ = enhancer.process(
  File "C:\Users\Jackson\Projects\v2vt\lipsync\third_part\GPEN\gpen_face_enhancer.py", line 123, in process    mask_sharp, (0, 0), sigmaX=1, sigmaY=1, borderType=cv2.BORDER_DEFAULT)
UnboundLocalError: local variable 'mask_sharp' referenced before assignment

Environment (please complete the following information):

• OS: Windows 11 Pro
• CPU/GPU: NVIDIA GeForce RTX 3070 Ti
• Python: 3.9.19
• Pytorch: 2.1.1
• CUDA: 11.8