parisneo/prompt_translator

A stable diffusion extension for translating prompts from 50 languages. The objective is to give users the possibility to use their own language to perform text prompting.

Prompt Translator for Stable Diffusion

Stable diffusion is a text to image generation model. The prompt_translator extension allows people to translate their prompt from any language to English before executing the generation, making it accessible to a wider audience.

Utility

Here is an example of how useful prompt_translator can be:

Let's say that you are a French speaker and want to use Stable Diffusion to generate an image of a beautiful mountain. However, the model requires the prompt to be in English. Without prompt_translator, you would need to either write your prompt in English (if you're comfortable with the language) or use an external translation tool to manually translate your prompt to English.

With prompt_translator, you can simply select "French" as your source language and write your prompt in French. The extension will automatically translate your prompt to English before sending it to the model. This way, you can use Stable Diffusion without having to worry about the language barrier.

Using prompt_translator makes Stable Diffusion more accessible to people who are not comfortable with English or have limited proficiency in the language. It opens up new possibilities for text-to-image generation by allowing users to use their native language to write prompts.

We hope that prompt_translator will make Stable Diffusion more accessible and easier to use for people around the world. Please feel free to contribute to the project or report any issues on our GitHub page.

Installation

To install prompt_translator, clone the repository or extract the zip file to the extensions folder of the sd_webui mother application.

Usage

After installing prompt_translator, a new entry will be added to the Gradio UI. To use the automatic translation tool, click the "Load Translation Model" button to load the translation model. The translation model used in this tool is the mbart-large-50-many-to-one-mmt model developed by Meta (formerly Facebook). You can find more information about the model on its Hugging Face model card.

Once the model is loaded, a dropdown UI will be displayed, where the user can select the source language of their prompt. The user can then write their prompt in the desired language and press the "Generate" button to generate the image. The prompt will automatically be translated to English, and the resulting image will look as described in the text.

Here are some screenshots of the extension in work:

Using the X/Y/Z script, we can test changing words in another language. Here is French:

Supported Languages

The list of supported languages is: Arabic (ar_AR), Czech (cs_CZ), German (de_DE), English (en_XX), Spanish (es_XX), Estonian (et_EE), Finnish (fi_FI), French (fr_XX), Gujarati (gu_IN), Hindi (hi_IN), Italian (it_IT), Japanese (ja_XX), Kazakh (kk_KZ), Korean (ko_KR), Lithuanian (lt_LT), Latvian (lv_LV), Burmese (my_MM), Nepali (ne_NP), Dutch (nl_XX), Romanian (ro_RO), Russian (ru_RU), Sinhala (si_LK), Turkish (tr_TR), Vietnamese (vi_VN), Chinese (zh_CN), Afrikaans (af_ZA), Azerbaijani (az_AZ), Bengali (bn_IN), Persian (fa_IR), Hebrew (he_IL), Croatian (hr_HR), Indonesian (id_ID), Georgian (ka_GE), Khmer (km_KH), Macedonian (mk_MK), Malayalam (ml_IN), Mongolian (mn_MN), Marathi (mr_IN), Polish (pl_PL), Pashto (ps_AF), Portuguese (pt_XX), Swedish (sv_SE), Swahili (sw_KE), Tamil (ta_IN), Telugu (te_IN), Thai (th_TH), Tagalog (tl_XX), Ukrainian (uk_UA), Urdu (ur_PK), Xhosa (xh_ZA), Galician (gl_ES), Slovene (sl_SI).

License

This project is licensed under the MIT license.

Contributing

Contributions to prompt_translator are welcome! If you find a bug or have an idea for a new feature, please create an issue on the project's GitHub page. If you'd like to contribute code, please fork the repository, make your changes, and submit a pull request.

prompt_translator's Issues

Wrong negative prompt

Initial negative prompt:color
Translated negative prompt:The Committee recommends that the State party:

如何在WebAPI中启用这个插件？

实测在WebUI中能够正常翻译中文提示词，但是我希望在API的使用使用，该怎么写参数或者调用哪个接口才可以启用呢？

大家先别用，这个插件有BUG。

用了SD大概率会出现 文生图 图转图功能报错。请谨慎等待作者更新

After downloading 2.4 GB of data and restarting the UI, the language selection did not appear.

Hi,

Building translator
Loading generator
Loading tokenizer
Traceback (most recent call last):
  File "e:\Program Files\Stable_Diffusion\stable-diffusion-webui\venv\lib\site-packages\gradio\routes.py", line 394, in run_predict
    output = await app.get_blocks().process_api(
  File "e:\Program Files\Stable_Diffusion\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 1075, in process_api
    result = await self.call_function(
  File "e:\Program Files\Stable_Diffusion\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 884, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "e:\Program Files\Stable_Diffusion\stable-diffusion-webui\venv\lib\site-packages\anyio\to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "e:\Program Files\Stable_Diffusion\stable-diffusion-webui\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "e:\Program Files\Stable_Diffusion\stable-diffusion-webui\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "E:\Program Files\Stable_Diffusion\stable-diffusion-webui\extensions\prompt_translator\scripts\main.py", line 206, in set_active
    self.translator = MBartTranslator()
  File "E:\Program Files\Stable_Diffusion\stable-diffusion-webui\extensions\prompt_translator\scripts\main.py", line 99, in __init__
    self.tokenizer = MBart50TokenizerFast.from_pretrained(model_name, src_lang=src_lang, tgt_lang=tgt_lang)
  File "e:\Program Files\Stable_Diffusion\stable-diffusion-webui\venv\lib\site-packages\transformers\tokenization_utils_base.py", line 1804, in from_pretrained
    return cls._from_pretrained(
  File "e:\Program Files\Stable_Diffusion\stable-diffusion-webui\venv\lib\site-packages\transformers\tokenization_utils_base.py", line 1959, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "e:\Program Files\Stable_Diffusion\stable-diffusion-webui\venv\lib\site-packages\transformers\models\mbart50\tokenization_mbart50_fast.py", line 135, in __init__
    super().__init__(
  File "e:\Program Files\Stable_Diffusion\stable-diffusion-webui\venv\lib\site-packages\transformers\tokenization_utils_fast.py", line 114, in __init__
    fast_tokenizer = convert_slow_tokenizer(slow_tokenizer)
  File "e:\Program Files\Stable_Diffusion\stable-diffusion-webui\venv\lib\site-packages\transformers\convert_slow_tokenizer.py", line 1162, in convert_slow_tokenizer
    return converter_class(transformer_tokenizer).converted()
  File "e:\Program Files\Stable_Diffusion\stable-diffusion-webui\venv\lib\site-packages\transformers\convert_slow_tokenizer.py", line 438, in __init__
    from .utils import sentencepiece_model_pb2 as model_pb2
  File "e:\Program Files\Stable_Diffusion\stable-diffusion-webui\venv\lib\site-packages\transformers\utils\sentencepiece_model_pb2.py", line 92, in <module>
    _descriptor.EnumValueDescriptor(
  File "e:\Program Files\Stable_Diffusion\stable-diffusion-webui\venv\lib\site-packages\google\protobuf\descriptor.py", line 796, in __new__
    _message.Message._CheckCalledFromGeneratedFile()
TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

Is there Thai option?

Hi, i view in https://huggingface.co/facebook/mbart-large-50-one-to-many-mmt has language, th_TH but i cannot see in option?

Model is [mbart-large-50-many-to-one-mmt], not [mbart-large-50-one-to-many-mmt]

in README.md
usage section, should refer to Hugging Face model card.

many-to-one, not one-to-many.

I followed the instruction and downloaded the model,
same name and size,
but it won't do the translation 😢

Because of network condition, I didn't have a way to get the program to download the model automatically. So I did it manually, then I found the problem, It's not the model.

After I use the right model, the translation is OK,
as below:

Translating prompt to English from 中文
Initial prompt:一位**的公主，骑在独角兽上。
Translated prompt:A Chinese princess, riding on a unicorn.

I think the code might be right, but readme is wrong.

Error installing dictionary

When the dictionary starts to download, an error occurs

in
DESCRIPTOR = _descriptor.FileDescriptor(
File "D:\stable-diffusion-portable-main\venv\lib\site-packages\google\protobuf\descriptor.py", line 1072, in new
return _message.default_pool.AddSerializedFile(serialized_pb)
TypeError: Couldn't build proto file into descriptor pool: duplicate file name sentencepiece_model.proto

API, auto-detect, NLLB-200

Hello everyone,
I love the translation feature. I have some questions:

1. is it possible to call it over API?
2. is it possible to autodetect the source language?
3. is it possible to use NLLB-200 anyhow?

Thank you so much,
Johannes

翻译有问题。。。。。请看图片 提示词给的太多了。。。

无法在sdwebuiapi中使用

感谢作者的付出，我在使用中遇到如下问题：
Error running process: /home/cui/stable-diffusion-webui/extensions/prompt_translator/scripts/main.py
Traceback (most recent call last):
File "/home/cui/stable-diffusion-webui/modules/scripts.py", line 417, in process
script.process(p, *script_args)
File "/home/cui/stable-diffusion-webui/extensions/prompt_translator/scripts/main.py", line 432, in process
print(f"Translating prompt to English from {language_options[language].label}")
TypeError: list indices must be integers or slices, not str

本地中使用是正常的，但在weiuiapi中报这个错误

like this plugin,but have a bug that LORA missing, and Add Chinese comma convert to English symbol function please.

like this plugin,but have a bug that LORA missing,.
thanks share and continual improvement.

挺好的插件，还可以英语与中文混用，就是有点小错误，比如：lora模型会丢失。
谢谢作者的贡献。

Translation from English into English

It would be nice to have a minimal setting, for example, to be able to turn off the translation of a negative prompt, I have a negative prompt in English already fixed in the ui-config.json file. I think you need to remove the translation from English to English, it does not happen correctly. I write a prompt in my native language, but I add specific terms in English, as a result of the translation, the wrong prompt is obtained. How to remove an extension? Where is the Downloading pytorch_model.bin file located: 100%| 2.44G/2.44G ?

Unnecessary spaces in the translation break the prompt

When translating, the markup is damaged. For example, if part of prompt is taken in parentheses and followed by '+' signs for reinforcement, spaces appear before '+' signs after translation, it breaks prompt.

You probably don't affect this directly, but as an option you might consider post-processing the text and removing spaces before control characters.

The Russian language does not work

I checked in German - everything works. I turn on Russian - at first thinks for a long time, then incorrectly issues promt

good work，but remove the rola tag

tag likelora:female_v2:1 be removed in the result.

Is there Turkish option?

Hi! As I can see that turkish is included in here; https://huggingface.co/facebook/mbart-large-50-one-to-many-mmt

but I can't see the turkish option in source language, could you please inform us about this issue?

Thank you for your time and sharing. Have a good day!

Error activating Enable translation

Hi:

I want to try this extension but after following the steps it gives me the following error:

"Building translator
Loading generator (this may take few minutes the first time as I need to download the model)
Loading tokenizer
(...)
TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:

1. Downgrade the protobuf package to 3.20.x or lower.
2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates"

Can you help me? Thank you

Batch issue

When doing a batch of prompts, the extension translate each batch.

Example: I write my prompt and then i want to generate 50 images with this prompt, this should only translate once... but it translate 50 times (for the same prompt)

Doesn't work

Error running process: \Downloads\stable-diffusion-webui-master\stable-diffusion-webui-master\extensions\prompt_translator\scripts\main.py
Traceback (most recent call last):
File "\Downloads\stable-diffusion-webui-master\stable-diffusion-webui-master\modules\scripts.py", line 417, in process
script.process(p, *script_args)
TypeError: Script.process() missing 1 required positional argument: 'language'

Nice idea, but...

...is completely useless for the Hungarian language. Here are three samples:

Initial prompt:egy lány a bozótban áll meztelenül, mögötte piros autó
Translated prompt:You know, Egyptian land, and a sketch, all of that melodrama, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know

Translated by DeepL: a girl standing naked in the bush with a red car behind her

---

Initial prompt:egy béka ül a tó partján, mögötte nádas
Translated prompt:Egyptian Egyptian Egyptian Egyptian Egyptian Egyptian Egyptian Egyptian Egyptian Egyptian Egyptian Egyptian Egyptian

Translated by DeepL: a frog sitting on the shore of a pond with reeds behind him

---

Initial prompt:béka ül a tó partján, mögötte nádas
Translated prompt:You know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know

Translated by DeepL: frog sitting on the shore of the lake, reeds behind him

So it translates completely differently, so that needs to be improved.

ValueError: Couldn't instantiate the backend tokenizer

ValueError: Couldn't instantiate the backend tokenizer from one of:
(1) a tokenizers library serialization file,
(2) a slow tokenizer instance to convert or
(3) an equivalent slow tokenizer class to instantiate and convert.
You need to have sentencepiece installed to convert a slow tokenizer to a fast one.

what can i do

TypeError: 'method' object is not subscriptable

TypeError: 'method' object is not subscriptable
'''
Error calling: /home/xuji/stable-diffusion-webui/extensions/prompt_translator/scripts/main.py/ui
Traceback (most recent call last):
File "/home/xuji/stable-diffusion-webui/modules/scripts.py", line 270, in wrap_call
res = func(*args, **kwargs)
File "/home/xuji/stable-diffusion-webui/extensions/prompt_translator/scripts/main.py", line 403, in ui
self.set_negative_translate_active
TypeError: 'method' object is not subscriptable
'''

A bug has occurred

A bug has occurred, and the Chinese negative prompt word is not available. The software did not translate or load negative prompt words, only positive prompt words were translated.I hope you can solve it