Connect the World, Frame by Frame

English｜中文

Overview

VideoLingo is an all-in-one video translation, localization, and dubbing tool aimed at generating Netflix-quality subtitles. It eliminates stiff machine translations and multi-line subtitles while adding high-quality dubbing, enabling global knowledge sharing across language barriers. With an intuitive Streamlit interface, you can transform a video link into a localized video with high-quality bilingual subtitles and dubbing in just a few clicks.

Key features:

YouTube video download via yt-dlp
Word-level subtitle recognition with WhisperX
NLP and GPT-based subtitle segmentation
GPT-generated terminology for coherent translation
3-step direct translation, reflection, and adaptation for professional-level quality
Netflix-standard single-line subtitles only
Dubbing alignment with GPT-SoVITS and other methods
One-click startup and output in Streamlit
Detailed logging with progress resumption
Comprehensive multi-language support

Difference from similar projects: Single-line subtitles only, superior translation quality

Demo

Russian Translation

ru_demo.mp4

GPT-SoVITS

sovits.mp4

OAITTS

OAITTS.mp4

Language Support:

Current input language support and examples:

Input Language	Translation Demo
English	English to Chinese
Russian	Russian to Chinese
French	French to Japanese
German	German to Chinese
Italian	Italian to Chinese
Spanish	Spanish to Chinese
Japanese	Japanese to Chinese
Chinese*	Chinese to English

*Chinese requires separate configuration of the whisperX model, only applicable for local source code installation. See the installation documentation for the configuration process, and be sure to specify the transcription language as zh in the webpage sidebar

Translation language support depends on the capabilities of the large language model used, while dubbing language depends on the chosen TTS method.

Quick Start

Online Experience

Experience VideoLingo quickly in Colab in just 5 minutes:

Local Installation

VideoLingo supports all hardware platforms and operating systems, but performs best with GPU acceleration. For detailed installation instructions , please refer to the documentation: English | 简体中文

Docker Installation

VideoLingo provides a Dockerfile. Please refer to the installation documentation: English | 简体中文

Batch Mode

Usage instructions: English | 简体中文

️Current Limitations

WhisperX performance varies across different devices. Version 1.7 performs demucs voice separation first, but this may result in worse transcription after separation compared to before. This is because whisper itself was trained in environments with background music - before separation it won't transcribe BGM lyrics, but after separation it might transcribe them.
The dubbing feature quality may not be perfect as it's still in testing and development stage, with plans to integrate MascGCT. For best results currently, it's recommended to choose TTS with similar speech rates based on the original video's speed and content characteristics. See the demo for effects.
Multilingual video transcription recognition will only retain the main language. This is because whisperX uses a specialized model for a single language when forcibly aligning word-level subtitles, and will delete unrecognized languages.
Multi-character separate dubbing is under development. While whisperX has VAD potential, specific implementation work is needed, and this feature is not yet supported.

Roadmap

VAD to distinguish speakers, multi-character dubbing
Customizable translation styles
SaaS service
Lip sync for dubbed videos

License

This project is licensed under the Apache 2.0 License. When using this project, please follow these rules:

When publishing works, it is recommended (not mandatory) to credit VideoLingo for subtitle generation.
Follow the terms of the large language models and TTS used for proper attribution.
If you copy the code, please include the full copy of the Apache 2.0 License.

We sincerely thank the following open-source projects for their contributions, which provided important support for the development of VideoLingo: