trent-fellbootman/konnyaku-gpt

GPT-powered subtitle generator that generates accurate, translated subtitles for Japanese animes.

KonnyakuGPT: More Accurate Anime Subtitling with Multimodal GPT

日本語 中文

KonnyakuGPT is an AI-powered high-quality and easy-to-use subtitle generator, primarily targeted for Japanese animes.

Key Features

1. Easy to use: one function call and high quality subtitle file is generated from the video.
2. High quality (depends on your budget): With "medium" quality preset which costs about $4 per 20-minute video, KonnyakuGPT is able to correctly address 80% of video-special terms (character names, etc) and produce subtitles that form a logical story. Subtitles generated by KonnyakuGPT allow audience with no knowledge about Japanese to watch Japanese animes and understand the plots without the need to guess what is going on.
3. Translation support: KonnyakuGPT can create translated subtitles for any language.
4. Pausing and resuming: progress is saved periodically when generating subtitles. You can resume your progress even if on going subtitle generation process is accidentally killed.
5. Configurable: KonnyakuGPT works well even if your video has an unusual setup and contains video-specific special terms (e.g., character names), which is common in Japanese animes. As long as you input background information that depicts the background of the video and include special terms that may occur, KonnyakuGPT will infer the plot and create subtitles that make sense and address special terms correctly.

Here's a screenshot of an episode from the Japanese anime Chimpui, with translated subtitles generated from KonnyakuGPT:

KonnyakuGPT not only addressed and translated the character's name (Japanese: "エリ" / English: "Eri"/ Chinese: "惠理") correctly, but also identified a special object in the scene and inferred its functionality (there is an "imagination gun" in the scene which shoots special rays that inject particular thoughts into creatures, with its effects lasting for 24 hours).

Installation

KonnyakuGPT can be installed via pip:

pip install konnyaku-gpt

Configuration & Usage

Open AI setup

KonnyakuGPT uses Open AI's API services; you must configure the API key before invoking the subtitle generator. The easiest way to do so is to set the environment variable "OPEN_AI_API_KEY" before running the Python script that generates subtitles. E.g.:

# run this line if you are on a Unix-like OS
export OPENAI_API_KEY=<your-api-key>
# run this line if you are on Windows
set OPENAI_API_KEY=<your-api-key>

python <a-script-that-invokes-KonnyakuGPT>

Usage

To create subtitles with predefnied methods, just import the default subtitle generator, input background information, and invoke the generator.

Here is an example script that generates subtitles for an episode in the anime Chimpui by Fujiko F. Fujio:

from pathlib import Path

from konnyaku_gpt.subtitle_generation import DefaultGenerator
from konnyaku_gpt.tricks import simple_split_subtitle_file

# It is recommended to use the 'medium' preset for a balance between quality and cost
generator = DefaultGenerator(quality_preset='medium')

output_path = Path('/output/srt/path')

# Generate the subtitles with AI!
generator.generate_subtitles(video_path=Path('/path/to/Chimpui/episode/mp4'),
                             output_path=output_path,
                             video_background=\
"""The video consists of two episodes from the Japanese anime Chimpui.
The title of the first episode is "レッツゴー銀河レース"; the title of the second episode is "はじめまして、ルルロフです".
There is an introductory screen at the start of each episode, where the title of that episode is shouted out.

Some special terms in Chimpui:

A. Characters:

    1. 春日エリ (Kasuga Eri), often referred to as "Eri" (Chinese: 惠理), "Eri-chan", "Eri-sama" (Chinese: 惠理大人) and "Kasuga" (Chinese: 春日). She is a boyish girl. She has a crush on her classmate Uchiki and does not want to marry the prince from the Mahl planet at all.
    2. Chimpui (チンプイ, Chinese: 芝比). An alien mouse with superpowers.
    3. Wanderyu (ワンダユウ, Chinese: 旺达). An alien dog with superpowers.
    4. Shou Uchiki (内木翔), often referred to as "Uchiki" (Chinese: 内木). A top student. Eri has a crush on him.
    5. Sunemi Koganeyama (小金山スネ美), often referred to as "Sunemi" / "Sunemi-chan" / "Sunemi-san" (Chinese: 诗奈美). A rich girl who likes to brag about things, especially her family's wealth.
    6. Masao Oeyama (大江山政男), often referred to as "Oeyama" (Chinese: 大江山). A strong 6-grader who sometimes bullies his classmates, especially Uchiki.
    7. Shosei Kitsune (木常小政), often referred to as "Shosei" (Chinese: 小政) or "Kitsune" (Chinese: 木常). Oeyama's sidekick.
    8. Hotaru Fujino (藤野ほたる), often referred to as "Hotaru" (Chinese: 小莹). A girlish girl who often dreams about girlish, unrealistic things, like a handsome prince coming to marry her.
    9. Lulealv, often referred to as "Lulealv Denka" (Chinese: 吕诺夫殿下) or simply "Denka" (殿下). The prince of the Mahl planet who wishes to marry Eri. However, Eri doesn't want to marry him at all.

    Chinpui and Wanda are both from the Mahl (Japanese: マール) planet.
    This planet is also referred to as the Mahl planet in English,
    or "玛尔星" in Chinese.

B. Others:
    1. Mahl (マール), the name of the planet that Chinpui and Wanda come from.
    2. Kahou (科法 / かほう). "科法" is the special, advanced and convenient-to-use technologies from the Mahl planet.
    Chinpui and Wanda both uses Kahou.
""",
                             target_language='English',
                             workspace_path=Path('final-episode-workspace'))

# split the subtitles so that each subtitle element is short
simple_split_subtitle_file(output_path)

Troubleshooting

Special Notice For Chinese Users

As there are known issues with certain internet access within China, you will likely need a proxy server to use Open AI services. To set up proxy for KonnyakuGPT, simply specify the proxy server address and port via environment variables in the console which will run KonnyakuGPT, e.g.:

# run this line if you are on a Unix-like OS
export ALL_PROXY=127.0.0.1:<your-proxy-port>
# run this line if you are on Windows
set ALL_PROXY=127.0.0.1:<your-proxy-port>

CUDA Setup

KonnyakuGPT invokes locally deployed deep learning models when generating subtitles. By default, it will try to put the model on GPU if you have a CUDA-compatible graphics card. However, if the VRAM of your GPU is less than 8GB, the models may not fit into your VRAM and an error may be raised.

In this case, you need to tell KonnyakuGPT not to use CUDA, and the easiest way to do so is to mask your GPU via environment variables, i.e.:

# run this line if you are on a Unix-like OS
export CUDA_VISIBLE_DEVICES=''
# run this line if you are on Windows
set CUDA_VISIBLE_DEVICES=''

Homage

This project is developed in memory of the famous Japanese comic writer, Hiroshi Fujimoto (藤本 弘), also known as Fujiko F. Fujio (藤子・F・不二雄). The name "KonnyakoGPT" is inspired by "ほんやくコンニャク" ("Translation Gummy") from Doraemon, his most popular creation.