Giter Site home page Giter Site logo

libretranslate / libretranslate Goto Github PK

View Code? Open in Web Editor NEW
7.0K 86.0 655.0 1.96 MB

Free and Open Source Machine Translation API. Self-hosted, offline capable and easy to setup.

Home Page: https://libretranslate.com

License: GNU Affero General Public License v3.0

Python 74.26% HTML 12.88% Dockerfile 3.46% CSS 6.95% Batchfile 0.70% Shell 1.74%
translation translate api machine translator

libretranslate's Introduction

LibreTranslate

Try it online! | API Docs | Community Forum

Python versions Run tests Build and Publish Docker Image Publish package Awesome Humane Tech

Free and Open Source Machine Translation API, entirely self-hosted. Unlike other APIs, it doesn't rely on proprietary providers such as Google or Azure to perform translations. Instead, its translation engine is powered by the open source Argos Translate library.

image

Try it online! | API Docs

API Examples

Simple

Request:

const res = await fetch("https://libretranslate.com/translate", {
  method: "POST",
  body: JSON.stringify({
    q: "Hello!",
    source: "en",
    target: "es"
  }),
  headers: { "Content-Type": "application/json" }
});

console.log(await res.json());

Response:

{
    "translatedText": "¡Hola!"
}

List of language codes: https://libretranslate.com/languages

Auto Detect Language

Request:

const res = await fetch("https://libretranslate.com/translate", {
  method: "POST",
  body: JSON.stringify({
    q: "Ciao!",
    source: "auto",
    target: "en"
  }),
  headers: { "Content-Type": "application/json" }
});

console.log(await res.json());

Response:

{
    "detectedLanguage": {
        "confidence": 83,
        "language": "it"
    },
    "translatedText": "Bye!"
}

HTML (beta)

Request:

const res = await fetch("https://libretranslate.com/translate", {
  method: "POST",
  body: JSON.stringify({
    q: '<p class="green">Hello!</p>',
    source: "en",
    target: "es",
    format: "html"
  }),
  headers: { "Content-Type": "application/json" }
});

console.log(await res.json());

Response:

{
    "translatedText": "<p class=\"green\">¡Hola!</p>"
}

Install and Run

You can run your own API server with just a few lines of setup!

Make sure you have Python installed (3.8 or higher is recommended), then simply run:

pip install libretranslate
libretranslate [args]

Then open a web browser to http://localhost:5000

On Ubuntu 20.04 you can also use the install script available at https://github.com/argosopentech/LibreTranslate-init

Run with Docker

You can also run the application with docker:

Linux/macOS

./run.sh [args]

Windows

run.bat [args]

Build and Run

See CONTIRBUTING.md for information on how to build and run the project yourself.

CUDA

You can use hardware acceleration to speed up translations on a GPU machine with CUDA 11.2 and nvidia-docker installed.

Run this version with:

docker compose -f docker-compose.cuda.yml up -d --build

Arguments

Arguments passed to the process or set via environment variables are split into two kinds.

  • Settings or runtime flags used to toggle specific runmodes or disable parts of the application. These act as toggle when added or removed.

  • Configuration parameters to set various limits and configure the application. These require a parameter to be passed to function, if removed the default parameters are used.

Settings / Flags

Argument Description Default Setting Env. name
--debug Enable debug environment Disabled LT_DEBUG
--ssl Whether to enable SSL Disabled LT_SSL
--api-keys Enable API keys database for per-client rate limits when --req-limit is reached Don't use API keys LT_API_KEYS
--require-api-key-origin Require use of an API key for programmatic access to the API, unless the request origin matches this domain No restrictions on domain origin LT_REQUIRE_API_KEY_ORIGIN
--require-api-key-secret Require use of an API key for programmatic access to the API, unless the client also sends a secret match No secrets required LT_REQUIRE_API_KEY_SECRET
--suggestions Allow user suggestions Disabled LT_SUGGESTIONS
--disable-files-translation Disable files translation File translation allowed LT_DISABLE_FILES_TRANSLATION
--disable-web-ui Disable web ui Web Ui enabled LT_DISABLE_WEB_UI
--update-models Update language models at startup Only on if no models found LT_UPDATE_MODELS
--metrics Enable the /metrics endpoint for exporting Prometheus usage metrics Disabled LT_METRICS

Configuration Parameters

Argument Description Default Parameter Env. name
--host Set host to bind the server to 127.0.0.1 LT_HOST
--port Set port to bind the server to 5000 LT_PORT
--char-limit Set character limit No limit LT_CHAR_LIMIT
--req-limit Set maximum number of requests per minute per client (outside of limits set by api keys) No limit LT_REQ_LIMIT
--req-limit-storage Storage URI to use for request limit data storage. See Flask Limiter memory:// LT_REQ_LIMIT_STORAGE
--req-time-cost Considers a time cost (in seconds) for request limiting purposes. If a request takes 10 seconds and this value is set to 5, the request cost is either 2 or the actual request cost (whichever is greater). No time cost LT_REQ_TIME_COST
--batch-limit Set maximum number of texts to translate in a batch request No limit LT_BATCH_LIMIT
--ga-id Enable Google Analytics on the API client page by providing an ID Empty (no tracking) LT_GA_ID
--frontend-language-source Set frontend default language - source auto LT_FRONTEND_LANGUAGE_SOURCE
--frontend-language-target Set frontend default language - target locale (match site's locale) LT_FRONTEND_LANGUAGE_TARGET
--frontend-timeout Set frontend translation timeout 500 LT_FRONTEND_TIMEOUT
--api-keys-db-path Use a specific path inside the container for the local database. Can be absolute or relative db/api_keys.db LT_API_KEYS_DB_PATH
--api-keys-remote Use this remote endpoint to query for valid API keys instead of using the local database Empty (use local db instead) LT_API_KEYS_REMOTE
--get-api-key-link Show a link in the UI where to direct users to get an API key Empty (no link shown on web ui) LT_GET_API_KEY_LINK
--shared-storage Shared storage URI to use for multi-process data sharing (e.g. when using gunicorn) memory:// LT_SHARED_STORAGE
--load-only Set available languages Empty (use all from argostranslate) LT_LOAD_ONLY
--threads Set number of threads 4 LT_THREADS
--metrics-auth-token Protect the /metrics endpoint by allowing only clients that have a valid Authorization Bearer token Empty (no auth required) LT_METRICS_AUTH_TOKEN
--url-prefix Add prefix to URL: example.com:5000/url-prefix/ / LT_URL_PREFIX

Notes:

  • Each argument has an equivalent environment variable that can be used instead. The env. variables overwrite the default values but have lower priority than the command arguments and are particularly useful if used with Docker. The environment variable names are the upper-snake-case of the equivalent command argument's name with a LT prefix.

  • To configure requirement for api key to use, set --req-limit to 0 and add the --api-keys flag. Requests made without a proper api key will be rejected.

  • Setting --update-models will update models regardless of whether updates are available or not.

Update

Software

If you installed with pip:

pip install -U libretranslate

If you're using docker:

docker pull libretranslate/libretranslate

Language Models

Start the program with the --update-models argument. For example: libretranslate --update-models or ./run.sh --update-models.

Alternatively you can also run the scripts/install_models.py script.

Run with WSGI and Gunicorn

pip install gunicorn
gunicorn --bind 0.0.0.0:5000 'wsgi:app'

You can pass application arguments directly to Gunicorn via:

gunicorn --bind 0.0.0.0:5000 'wsgi:app(api_keys=True)'

Kubernetes Deployment

See Medium article by JM Robles and the improved k8s.yaml by @rasos.

Helm Chart

Based on @rasos work you can now install LibreTranslate on Kubernetes using Helm.

A Helm chart is now available in the helm-chart repository where you can find more details.

You can quickly install LibreTranslate on Kubernetes using Helm with the following command:

helm repo add libretranslate https://libretranslate.github.io/helm-chart/
helm repo update
helm search repo libretranslate

helm install libretranslate libretranslate/libretranslate --namespace libretranslate --create-namespace

Manage API Keys

LibreTranslate supports per-user limit quotas, e.g. you can issue API keys to users so that they can enjoy higher requests limits per minute (if you also set --req-limit). By default all users are rate-limited based on --req-limit, but passing an optional api_key parameter to the REST endpoints allows a user to enjoy higher request limits. You can also specify different character limits that bypass the default --char-limit value on a per-key basis.

To use API keys simply start LibreTranslate with the --api-keys option. If you modified the API keys database path with the option --api-keys-db-path, you must specify the path with the same argument flag when using the ltmanage keys command.

Add New Keys

To issue a new API key with 120 requests per minute limits:

ltmanage keys add 120

To issue a new API key with 120 requests per minute and a maximum of 5,000 characters per request:

ltmanage keys add 120 --char-limit 5000

If you changed the API keys database path:

ltmanage keys --api-keys-db-path path/to/db/dbName.db add 120

Remove Keys

ltmanage keys remove <api-key>

View Keys

ltmanage keys

Prometheus Metrics

LibreTranslate has Prometheus exporter capabilities when you pass the --metrics argument at startup (disabled by default). When metrics are enabled, a /metrics endpoint is mounted on the instance:

http://localhost:5000/metrics

# HELP libretranslate_http_requests_in_flight Multiprocess metric
# TYPE libretranslate_http_requests_in_flight gauge
libretranslate_http_requests_in_flight{api_key="",endpoint="/translate",request_ip="127.0.0.1"} 0.0
# HELP libretranslate_http_request_duration_seconds Multiprocess metric
# TYPE libretranslate_http_request_duration_seconds summary
libretranslate_http_request_duration_seconds_count{api_key="",endpoint="/translate",request_ip="127.0.0.1",status="200"} 0.0
libretranslate_http_request_duration_seconds_sum{api_key="",endpoint="/translate",request_ip="127.0.0.1",status="200"} 0.0

You can then configure prometheus.yml to read the metrics:

scrape_configs:
  - job_name: "libretranslate"

    # Needed only if you use --metrics-auth-token
    #authorization:
      #credentials: "mytoken"

    static_configs:
      - targets: ["localhost:5000"]

To secure the /metrics endpoint you can also use --metrics-auth-token mytoken.

If you use Gunicorn, make sure to create a directory for storing multiprocess data metrics and set PROMETHEUS_MULTIPROC_DIR:

mkdir -p /tmp/prometheus_data
rm /tmp/prometheus_data/*
export PROMETHEUS_MULTIPROC_DIR=/tmp/prometheus_data
gunicorn -c scripts/gunicorn_conf.py --bind 0.0.0.0:5000 'wsgi:app(metrics=True)'

Language Bindings

You can use the LibreTranslate API using the following bindings:

Discourse Plugin

You can use the official discourse translator plugin to translate Discourse topics with LibreTranslate. To install it simply modify /var/discourse/containers/app.yml:

## Plugins go here
## see https://meta.discourse.org/t/19157 for details
hooks:
  after_code:
    - exec:
        cd: $home/plugins
        cmd:
          - git clone https://github.com/discourse/docker_manager.git
          - git clone https://github.com/discourse/discourse-translator
    ...

Then issue ./launcher rebuild app. From the Discourse's admin panel then select "LibreTranslate" as a translation provider and set the relevant endpoint configurations.

See it in action on this page.

Mobile Apps

Web browser

Mirrors

This is a list of public LibreTranslate instances, some require an API key. If you want to add a new URL, please open a pull request.

URL API Key Required Links
libretranslate.com ✔️ [ Get API Key ] [ Service Status ]
translate.terraprint.co -
trans.zillyhuhn.com -
libretranslate.eownerdead.dedyn.io -

TOR/i2p Mirrors

URL
lt.vernccvbvyi5qhfzyqengccj7lkove6bjot2xhh5kajhwvidqafczrad.onion
lt.vern.i2p

Adding New Language Models

You have two options to create new language models:

Most of the training data is from Opus, which is an open source parallel corpus. Check also NLLU

Localization

The LibreTranslate Web UI is available in all the languages for which LibreTranslate can translate to. It can also (roughly) translate itself! Some languages might not appear in the UI since they haven't been reviewed by a human yet. You can enable all languages by turning on --debug mode.

To help improve or review the UI translations:

{
 "name": "<Language>",
 "reviewed": true <-- Change this from false to true
}

UI Languages

Language Reviewed Weblate Link
Arabic Edit
Azerbaijani Edit
Chinese Edit
Chinese (Traditional) Edit
Czech ✔️ Edit
Danish Edit
Dutch Edit
English ✔️ Edit
Esperanto ✔️ Edit
Finnish Edit
French ✔️ Edit
German ✔️ Edit
Greek Edit
Hebrew Edit
Hindi Edit
Hungarian Edit
Indonesian Edit
Irish Edit
Italian ✔️ Edit
Japanese Edit
Kabyle Edit
Korean ✔️ Edit
Occitan Edit
Persian Edit
Polish Edit
Portuguese ✔️ Edit
Russian ✔️ Edit
Slovak Edit
Spanish ✔️ Edit
Swedish Edit
Turkish Edit
Ukranian ✔️ Edit
Vietnamese Edit

Roadmap

Help us by opening a pull request!

Any other idea is welcome also.

FAQ

Can I use your API server at libretranslate.com for my application in production?

In short, yes, but only if you buy an API key. You can always run LibreTranslate for free on your own server of course.

Some translations on libretranslate.com are different than the self-hosted ones. Why?

By default language models are loaded from the argos-index. Sometimes we deploy models on libretranslate.com that haven't been added to the argos-index yet, such as those converted from OPUS (thread)

Where are the language models saved?

In $HOME/.local/share/argos-translate/packages. On Windows that's C:\Users\youruser\.local\share\argos-translate\packages.

Can I use LibreTranslate behind a reverse proxy, like Apache2 or Caddy?

Yes, here are config examples for Apache2 and Caddy that redirect a subdomain (with HTTPS certificate) to LibreTranslate running on a docker at localhost.

sudo docker run -ti --rm -p 127.0.0.1:5000:5000 libretranslate/libretranslate

You can remove 127.0.0.1 on the above command if you want to be able to access it from domain.tld:5000, in addition to subdomain.domain.tld (this can be helpful to determine if there is an issue with Apache2 or the docker container).

Add --restart unless-stopped if you want this docker to start on boot, unless manually stopped.

Apache config

Replace [YOUR_DOMAIN] with your full domain; for example, translate.domain.tld or libretranslate.domain.tld.

Remove # on the ErrorLog and CustomLog lines to log requests.

#Libretranslate

#Redirect http to https
<VirtualHost *:80>
    ServerName http://[YOUR_DOMAIN]
    Redirect / https://[YOUR_DOMAIN]
    # ErrorLog ${APACHE_LOG_DIR}/error.log
    # CustomLog ${APACHE_LOG_DIR}/tr-access.log combined
 </VirtualHost>

#https
<VirtualHost *:443>
    ServerName https://[YOUR_DOMAIN]

    ProxyPass / http://127.0.0.1:5000/
    ProxyPassReverse / http://127.0.0.1:5000/
    ProxyPreserveHost On

    SSLEngine on
    SSLCertificateFile /etc/letsencrypt/live/[YOUR_DOMAIN]/fullchain.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/[YOUR_DOMAIN]/privkey.pem
    SSLCertificateChainFile /etc/letsencrypt/live/[YOUR_DOMAIN]/fullchain.pem

    # ErrorLog ${APACHE_LOG_DIR}/tr-error.log
    # CustomLog ${APACHE_LOG_DIR}/tr-access.log combined
</VirtualHost>

Add this to an existing site config, or a new file in /etc/apache2/sites-available/new-site.conf and run sudo a2ensite new-site.conf.

To get a HTTPS subdomain certificate, install certbot (snap), run sudo certbot certonly --manual --preferred-challenges dns and enter your information (with subdomain.domain.tld as the domain). Add a DNS TXT record with your domain registrar when asked. This will save your certificate and key to /etc/letsencrypt/live/{subdomain.domain.tld}/. Alternatively, comment the SSL lines out if you don't want to use HTTPS.

Caddy config

Replace [YOUR_DOMAIN] with your full domain; for example, translate.domain.tld or libretranslate.domain.tld.

#Libretranslate
[YOUR_DOMAIN] {
  reverse_proxy localhost:5000
}

Add this to an existing Caddyfile or save it as Caddyfile in any directory and run sudo caddy reload in that same directory.

NGINX config

Replace [YOUR_DOMAIN] with your full domain; for example, translate.domain.tld or libretranslate.domain.tld.

Remove # on the access_log and error_log lines to disable logging.

server {
  listen 80;
  server_name [YOUR_DOMAIN];
  return 301 https://$server_name$request_uri;
}

server {
  listen 443 http2 ssl;
  server_name [YOUR_DOMAIN];

  #access_log off;
  #error_log off;

  # SSL Section
  ssl_certificate /etc/letsencrypt/live/[YOUR_DOMAIN]/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/[YOUR_DOMAIN]/privkey.pem;

  ssl_protocols TLSv1.2 TLSv1.3;

  # Using the recommended cipher suite from: https://wiki.mozilla.org/Security/Server_Side_TLS
  ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384';

  ssl_session_timeout 10m;
  ssl_session_cache shared:MozSSL:10m;  # about 40000 sessions
  ssl_session_tickets off;

  # Specifies a curve for ECDHE ciphers.
  ssl_ecdh_curve prime256v1;
  # Server should determine the ciphers, not the client
  ssl_prefer_server_ciphers on;


  # Header section
  add_header Strict-Transport-Security  "max-age=31536000; includeSubDomains; preload" always;
  add_header Referrer-Policy            "strict-origin" always;

  add_header X-Frame-Options            "SAMEORIGIN"    always;
  add_header X-XSS-Protection           "1; mode=block" always;
  add_header X-Content-Type-Options     "nosniff"       always;
  add_header X-Download-Options         "noopen"        always;
  add_header X-Robots-Tag               "none"          always;

  add_header Feature-Policy             "microphone 'none'; camera 'none'; geolocation 'none';"  always;
  # Newer header but not everywhere supported
  add_header Permissions-Policy         "microphone=(), camera=(), geolocation=()" always;

  # Remove X-Powered-By, which is an information leak
  fastcgi_hide_header X-Powered-By;

  # Do not send nginx server header
  server_tokens off;

  # GZIP Section
  gzip on;
  gzip_disable "msie6";

  gzip_vary on;
  gzip_proxied any;
  gzip_comp_level 6;
  gzip_buffers 16 8k;
  gzip_http_version 1.1;
  gzip_min_length 256;
  gzip_types text/xml text/javascript font/ttf font/eot font/otf application/x-javascript application/atom+xml application/javascript application/json application/manifest+json application/rss+xml application/x-web-app-manifest+json application/xhtml+xml application/xml image/svg+xml image/x-icon text/css text/plain;

  location / {
      proxy_pass http://127.0.0.1:5000/;
      proxy_set_header Host $http_host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $scheme;
      client_max_body_size 0;
  }
}

Add this to an existing NGINX config or save it as libretranslate in the /etc/nginx/site-enabled directory and run sudo nginx -s reload.

Can I do batch translations?

Yes, pass an array of strings instead of a string to the q field:

const res = await fetch("https://libretranslate.com/translate", {
  method: "POST",
  body: JSON.stringify({
    q: ["Hello", "world"],
    source: "en",
    target: "es"
  }),
  headers: { "Content-Type": "application/json" }
});

console.log(await res.json());
// {
//     "translatedText": [
//         "Hola",
//         "mundo"
//     ]
// }

Contributing

We welcome contributions! Here's some ideas:

  • Train a new language model using Locomotive. For example, we want to train improved neural networks for German and many other languages.
  • Can you beat the performance of our language models? Train a new one and let's compare it. To submit your model make a post on the community forum with a link to download your .argosmodel file and some sample text that your model has translated.
  • Pick an issue to work on.

Credits

This work is largely possible thanks to Argos Translate, which powers the translation engine.

License

GNU Affero General Public License v3

Trademark

See Trademark Guidelines

libretranslate's People

Contributors

blueberryy avatar boffire avatar catsnote avatar caugner avatar comradekingu avatar dacmot avatar dingedi avatar drivard avatar ecxod avatar euberdeveloper avatar eugene-davis avatar gi-yt avatar idotj avatar jonwiggins avatar jorgesumle avatar mammo0 avatar mufeedali avatar nijel avatar otaldonexus avatar peterdavehello avatar pierotofy avatar pj-finlay avatar reynoldsnlp avatar sethfalco avatar sigaloid avatar symegac avatar vemonet avatar worldworm avatar yogeshwaran01 avatar zerof avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

libretranslate's Issues

broken offline mode

If I try --offline in docker-compose command line, then I will get this error message in container log:

libretranslate_1  | Traceback (most recent call last):
libretranslate_1  |   File "/usr/local/bin/libretranslate", line 8, in <module>
libretranslate_1  |     sys.exit(main())
libretranslate_1  |   File "/usr/local/lib/python3.8/site-packages/app/main.py", line 35, in main
libretranslate_1  |     app = create_app(args)
libretranslate_1  |   File "/usr/local/lib/python3.8/site-packages/app/app.py", line 75, in create_app
libretranslate_1  |     raise AttributeError(f"{args.frontend_language_source} as frontend source language is not supported.")
libretranslate_1  | AttributeError: en as frontend source language is not supported.
libretranslate_1  | Traceback (most recent call last):
libretranslate_1  |   File "/usr/local/bin/libretranslate", line 8, in <module>
libretranslate_1  |     sys.exit(main())
libretranslate_1  |   File "/usr/local/lib/python3.8/site-packages/app/main.py", line 35, in main
libretranslate_1  |     app = create_app(args)
libretranslate_1  |   File "/usr/local/lib/python3.8/site-packages/app/app.py", line 75, in create_app
libretranslate_1  |     raise AttributeError(f"{args.frontend_language_source} as frontend source language is not supported.")
libretranslate_1  | AttributeError: en as frontend source language is not supported.

my docker-compose.yml file:

docker-compose.yml
version: "3"

services:
  libretranslate:
    build: .
    restart: unless-stopped
    ports:
      - 127.0.0.1:5000:5000
    command: --offline

I also get this error message with this command line:

command: --req-limit 100 --char-limit 3000 --host 0.0.0.0 --port 5000 --frontend-language-source en --frontend-language-target de --frontend-timeout 500 --offline

Firefox Addon

Would be nice if you could select text and translate with your libtranslate instance

AttributeError: es as frontend target language is not supported.

matt@matt:~/clean/LibreTranslate$ libretranslate 
Traceback (most recent call last):
  File "/home/matt/.local/bin/libretranslate", line 8, in <module>
    sys.exit(main())
  File "/home/matt/.local/lib/python3.8/site-packages/app/main.py", line 35, in main
    app = create_app(args)
  File "/home/matt/.local/lib/python3.8/site-packages/app/app.py", line 77, in create_app
    raise AttributeError(f"{args.frontend_language_target} as frontend target language is not supported.")

Only happened once I started editing source to try to add --load-only. I can't get this to go away even after I pip3 remove libretranslate.

/detect endpoint not found - 404

Hello, I am getting a 404 error when trying to send a detect request

POST http://localhost:5000/detect

{
    "q": "hola mundo"
}

Response:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>404 Not Found</title>
<h1>Not Found</h1>
<p>The requested URL was not found on the server. If you entered the URL manually please check your spelling and try
	again.</p>

Package sentencepiece was not found in the pkg-config search path.

Hello

I setup a python (Python 3.9.2) virtualenv and executed pip install:

(libretranslate) [user@tower libretranslate]$ pip install libretranslate
Collecting libretranslate
  Using cached libretranslate-1.1.0-py3-none-any.whl (25 kB)
Collecting flask-swagger-ui==3.36.0
  Using cached flask-swagger-ui-3.36.0.tar.gz (4.9 MB)
Collecting flask-swagger==0.2.14
  Using cached flask-swagger-0.2.14.tar.gz (8.6 kB)
Collecting Flask-Limiter==1.4
  Using cached Flask_Limiter-1.4-py3-none-any.whl (15 kB)
Collecting waitress==1.4.4
  Using cached waitress-1.4.4-py2.py3-none-any.whl (58 kB)
Collecting langdetect==1.0.8
  Using cached langdetect-1.0.8.tar.gz (981 kB)
Collecting argostranslate==1.1.2
  Using cached argostranslate-1.1.2-py3-none-any.whl (15 kB)
Collecting Flask==1.1.2
  Using cached Flask-1.1.2-py2.py3-none-any.whl (94 kB)
Collecting ctranslate2==1.17.1
  Using cached ctranslate2-1.17.1-cp39-cp39-manylinux2014_x86_64.whl (75.4 MB)
Collecting sentencepiece==0.1.91
  Using cached sentencepiece-0.1.91.tar.gz (500 kB)
    ERROR: Command errored out with exit status 1:
     command: /home/user/Development/Python/libretranslate/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-mmn8dm7_/sentencepiece_d8e2afc20c7b4dfaa497d02952fb73f7/setup.py'"'"'; __file__='"'"'/tmp/pip-install-mmn8dm7_/sentencepiece_d8e2afc20c7b4dfaa497d02952fb73f7/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-2ej8edip
         cwd: /tmp/pip-install-mmn8dm7_/sentencepiece_d8e2afc20c7b4dfaa497d02952fb73f7/
    Complete output (5 lines):
    **Package sentencepiece was not found in the pkg-config search path.**
    Perhaps you should add the directory containing `sentencepiece.pc'
    to the PKG_CONFIG_PATH environment variable
    Package 'sentencepiece', required by 'virtual:world', not found
    Failed to find sentencepiece pkgconfig
    ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/ba/f6/520b56e5977f62aee48833da8b4ff2fdc2b10ebfa0dd78556b1d707d4086/sentencepiece-0.1.91.tar.gz#sha256=f9700cf607ea064d9fad34c751fbf49953dcc56fe68c54b277481aa0aec5c18f (from https://pypi.org/simple/sentencepiece/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
INFO: pip is looking at multiple versions of <Python from Requires-Python> to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of libretranslate to determine which version is compatible with other requirements. This could take a while.
ERROR: Could not find a version that satisfies the requirement sentencepiece==0.1.91 (from argostranslate)
ERROR: No matching distribution found for sentencepiece==0.1.91

Machine:

$ uname -a
Linux tower 5.10.19-1-MANJARO #1 SMP PREEMPT Fri Feb 26 11:38:34 UTC 2021 x86_64 GNU/Linux

Regards

Missing a bracket in API request example

The API example is missing a } after headers

From:

const res = await fetch("https://libretranslate.com/translate", {
	method: "POST",
	body: JSON.stringify({
		q: "",
		source: "en",
		target: "es"
	}),
	headers: {
		"Content-Type": "application/json"
	});

console.log(await res.json());

To:

const res = await fetch("https://libretranslate.com/translate", {
	method: "POST",
	body: JSON.stringify({
		q: "",
		source: "en",
		target: "es"
	}),
	headers: {
		"Content-Type": "application/json"}
	});

console.log(await res.json());

Translating markdown documents

While some markdown works well other markdown content is getting messed up.

Would be nice to be able to directly translate markdown documents that keep all the markdown.

contribution

Hey, this looks like a really cool project, how am I able to contribute to the translation aspect, (I am very much not a programmer)?

Option to install language models

I installed libretranslate using pip and after running libretranslate command, libretranslate started downloading models but because my internet connection was lost now after running libretranslate the server starts and other models are not downloaded.

also when i try to translate with available options i get this error message : "Error while calling /translate"

Help with accessing on local network

Very sorry if this is obvious stuff, but I have no experience with web APIs and I could use some help.

I have LibreTranslate running successfully in an LXC container on my home server, and from within the lxc it responds to curl -X GET "localhost:5000/frontend/settings" -H "accept: application/json" fine, but when I try to send a request to it from another computer on the network I get :

root@Ubuntu:~$ curl -X GET "10.0.0.33:5000/frontend/settings" -H  "accept: application/json"
curl: (7) Failed to connect to 10.0.0.33 port 5000: Connection refused

How can I make LibreTranslate available to my local network (but not the internet)?

I'm sure this isn't a LibreTranslate specific issue, but I wasn't sure where else to ask for help. If anyone could point me towards a guide (or even just what to google to find a guide) I would be super appreciative.

Installation docs are not quite updated

Hey together,

I just tried installing by these commands as you described:

git clone https://github.com/uav4geo/LibreTranslate --recurse-submodules
cd LibreTranslate
pip install -r requirements.txt
python main.py [args]

Seems like I have to install the models by my own. Would you update the documents or do create an auto-install?

(venv) [me@somehostLibreTranslate]$ python3 install_models.py
Removing old /home/me/webservices/LibreTranslate/installed_models
Creating /home/me/webservices/LibreTranslate/installed_models
Installing models/ar_en.argosmodel...
Installing models/de_en.argosmodel...
Installing models/en_ar.argosmodel...
...
Installed 9 language models!

Your new fan. :)

Front-end interface is not 100% self-hosted

As includes remote dependencies, for example:

<link href="https://fonts.googleapis.com/icon?family=Material+Icons" rel="stylesheet">
<script src="https://cdn.jsdelivr.net/npm/vue@2"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/prism/1.22.0/prism.min.js"></script>

Packaging for pip-based distribution

Hello, I was wondering if we could expect the project to have, as goal, pip-based releases (e.g. wheel builds or tgz archives) ?

This would greatly simplify setup and updates, including dependency management.

Control panel to manage api Keys in a simpler way

It would be good if they added a control panel to control the access of the api Keys of the api, since doing it by commands is not very practical I think, in my opinion it would be a good idea

What can that control panel have?

  1. See the api keys that are currently enabled
  2. Disable or enable api key
  3. Enable or disable api key system
  4. And among other more options for the operation of the website or api

That would be a suggestion that sounds like a good idea to me

pip install not possible due to OOM

Your Build instructions require this step:

pip install -r requirements.txt

which results in :

...
Collecting numpy
Using cached https://files.pythonhosted.org/packages/c7/e6/dccac76b7e825915ffb906beeba5a953597b6cfe1fe686b5276e122cb07c/numpy-1.20.1-cp38-cp38-manylinux2010_x86_64.whl
Collecting torch>=1.3.0
Downloading https://files.pythonhosted.org/packages/1d/a9/f349273a0327fdf20a73188c9c3aa7dbce68f86fad422eadd366fd2ed7a0/torch-1.7.1-cp38-cp38-manylinux1_x86_64.whl (776.8MB)
|███████████ | 269.3MB 76.0MB/s eta 0:00:07ERROR: Could not install packages due to an EnvironmentError: [Errno 28] Auf dem Gerät ist kein Speicherplatz mehr verfügbar

Which has more than enough space left:

/dev/xvda1 78G 29G 46G 39% /

Which lead to the only limiting factor on this server: /tmp/ which runs out of space, because pip downloads stuff there, instead of "./"

After freeing up some space there, the same step failed with a "killed":

Collecting numpy
Using cached https://files.pythonhosted.org/packages/c7/e6/dccac76b7e825915ffb906beeba5a953597b6cfe1fe686b5276e122cb07c/numpy-1.20.1-cp38-cp38-manylinux2010_x86_64.whl
Collecting torch>=1.3.0
Downloading https://files.pythonhosted.org/packages/1d/a9/f349273a0327fdf20a73188c9c3aa7dbce68f86fad422eadd366fd2ed7a0/torch-1.7.1-cp38-cp38-manylinux1_x86_64.whl (776.8MB)
|████████████████████████████████| 776.8MB 67.4MB/s eta 0:00:01Getötet

("getötet" German => "killed" )

because it gets killed, it doe snot remove it's tempfiles, which brings us back to step 1 in this Bugreport.

You really need to check things BEFORE the actions starts :)

[Docker] UnboundLocalError: local variable 'limiter' referenced before assignment

ubuntu@dns:~$ sudo docker run --restart unless-stopped -ti -p 127.0.0.1:5000:5000 libretranslate/libretranslate
Unable to find image 'libretranslate/libretranslate:latest' locally
latest: Pulling from libretranslate/libretranslate
0ecb575e629c: Pull complete 
7467d1831b69: Pull complete 
feab2c490a3c: Pull complete 
f15a0f46f8c3: Pull complete 
937782447ff6: Pull complete 
e78b7aaaab2c: Pull complete 
d6174e4a4e34: Pull complete 
45440a62f00d: Pull complete 
308e5e6db8bd: Pull complete 
59c4688e1459: Pull complete 
d57b72e02cb3: Pull complete 
c06705b84d78: Pull complete 
7f92627d08b8: Pull complete 
Digest: sha256:519a627fef1bdca6c61cfd39b5aeab424f4cbf11289a50f37fd45f7ec5645df9
Status: Downloaded newer image for libretranslate/libretranslate:latest
Updating language models
Found 16 models
Downloading Arabic -> English (1.0) ...
Downloading German -> English (1.0) ...
Downloading English -> Arabic (1.0) ...
Downloading English -> German (1.0) ...
Downloading English -> Spanish (1.0) ...
Downloading English -> French (1.0) ...
Downloading English -> Italian (1.0) ...
Downloading English -> Portuguese (1.0) ...
Downloading English -> Russian (1.0) ...
Downloading English -> Chinese (1.0) ...
Downloading Spanish -> English (1.0) ...
Downloading French -> English (1.0) ...
Downloading Italian -> English (1.0) ...
Downloading Portuguese -> English (1.0) ...
Downloading Russian -> English (1.0) ...
Downloading Chinese -> English (1.0) ...
Loaded support for 9 languages (16 models total)!
Traceback (most recent call last):
  File "/usr/local/bin/libretranslate", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/site-packages/app/main.py", line 35, in main
    app = create_app(args)
  File "/usr/local/lib/python3.8/site-packages/app/app.py", line 100, in create_app
    @limiter.exempt
UnboundLocalError: local variable 'limiter' referenced before assignment

Note the restart (I want it to auto start) and the 127.0.0.1, should I instead have changed it with the arguments? Could this cause the issue?

Dockerfile issues

The current Dockerfile has a couple of issues:

  1. It copies the whole models directory, without .dockerignoreing the containing .git directory (see below).
  2. It copies all together, which prevents a (reusable) layer for the model.
  3. It copies everything in the / (root) directory.
$ docker run -it --rm --entrypoint ls libretranslate/libretranslate -la models
total 1353400
drwxrwxrwx 2 root root     4096 Jan 18 15:07 .
drwxr-xr-x 1 root root     4096 Jan 19 17:43 ..
-rw-rw-rw- 1 root root       37 Jan 18 15:07 .git
-rw-rw-rw- 1 root root     1064 Jan 18 15:07 LICENSE
-rw-rw-rw- 1 root root      135 Jan 18 15:07 README.md
-rw-rw-rw- 1 root root 81869670 Jan 18 15:07 ar_en.argosmodel
-rw-rw-rw- 1 root root 86704510 Jan 18 15:07 de_en.argosmodel
-rw-rw-rw- 1 root root 88502631 Jan 18 15:07 en_ar.argosmodel
-rw-rw-rw- 1 root root 87106499 Jan 18 15:07 en_de.argosmodel
-rw-rw-rw- 1 root root 87503191 Jan 18 15:07 en_es.argosmodel
-rw-rw-rw- 1 root root 87359317 Jan 18 15:07 en_fr.argosmodel
-rw-rw-rw- 1 root root 87660780 Jan 18 15:07 en_it.argosmodel
-rw-rw-rw- 1 root root 87825848 Jan 18 15:07 en_pt.argosmodel
-rw-rw-rw- 1 root root 88386622 Jan 18 15:07 en_ru.argosmodel
-rw-rw-rw- 1 root root 88906802 Jan 18 15:07 en_zh.argosmodel
-rw-rw-rw- 1 root root 87381097 Jan 18 15:07 es_en.argosmodel
-rw-rw-rw- 1 root root 87171251 Jan 18 15:07 fr_en.argosmodel
-rw-rw-rw- 1 root root 87190224 Jan 18 15:07 it_en.argosmodel
-rw-rw-rw- 1 root root 86831489 Jan 18 15:07 pt_en.argosmodel
-rw-rw-rw- 1 root root 83506747 Jan 18 15:07 ru_en.argosmodel
-rw-rw-rw- 1 root root 81917366 Jan 18 15:07 zh_en.argosmodel
$ docker run -it --rm --entrypoint du libretranslate/libretranslate -h models
1.3G    models

Emojis

Hello! I enjoy using this service very much! Congrats and thanks to the community :D

I have noticed that emoji are parsed into the text and was wondering if there was a way to escape \uXXXX (unicode for emoji), treating them as proper nouns as to preserve them?
I would like to do any type of contribution but wanted to make sure if it is currently possible before messing around with the code.

Thanks in advance

salad salad salad salad salad salad salad salad

salad salad salad salad salad salad salad salad salad salad salad salad salad salad salad salad salad salad salad salad salad salad salad salad salad salad salad salad salad salad salad salad salad salad salad salad salad salad salad salad salad salad salad salad salad salad salad salad sala salad salad salad salad salad sala salad salad salad salad salad salad salad salad salad sala salad salad salad salad salad salad salad salad salad salad salad salad sala salad salad salad sala salad salad salad salad salad salad salad salad salad salad salad salad salad salad salad salad salad salad salad salad salad salad salad salad salad salad salad salad salad salad salad salad sala salad salad salad salad salad salad salad salad salad salad salad sala sala sala sala

Captura de pantalla de 2021-02-18 18-45-47

Uppercase in word lets translation fail

I used https://translate.astian.org/
I tried a simple translation from german to spanish.
I used the german wird "Hallo" which should be translated to "Hola"
I entered the german word as "HAllo" which is translated to "Halibut"
Also "HallO" fails and many other typos
As a user I would expect that it works case insensitiv.

Feature request: Offline mode

Hi, I work a lot with ESL Students and sometimes at their school the internet connection is not up to speed or availability.

Is there a way to have a translation app (English / Spanish) pre-downloaded into their device or hybrid app and being able to display the results to the user.

Thanks

This is great

This looks really cool! Let me know when the web demo is stable enough and I'll link it from the Argos Translate README.

Privacy policy on site

Could we get a privacy policy added to libretranslate.com? It would be useful to understand how text supplied for translation is handled.

How to install on Debian stable "ctranslate2==1.17.1"

Hi there! Cool project. I'm trying both direct install, or building via source. This is on Debian 10:

 $ pip3 install libretranslate
Collecting libretranslate
  Downloading https://files.pythonhosted.org/packages/de/2d/1c13e912914ddb0e2823810163e5c4a790a3275e980f67822e333b3860b8/libretranslate-1.1.0-py3-none-any.whl
...
Collecting ctranslate2==1.17.1 (from argostranslate==1.1.2->libretranslate)
  Could not find a version that satisfies the requirement ctranslate2==1.17.1 (from argostranslate==1.1.2->libretranslate) (from versions: )
No matching distribution found for ctranslate2==1.17.1 (from argostranslate==1.1.2->libretranslate)
$ git clone https://github.com/uav4geo/LibreTranslate
Cloning into 'LibreTranslate'...
...
$ cd LibreTranslate/
$ pip3 install -e .
Obtaining file:///home/x/Documents/devel/LibreTranslate
Collecting Flask-Limiter==1.4 (from libretranslate==1.1.0)
  Using cached https://files.pythonhosted.org/packages/1c/66/1bc848a3d37bed2a4c6cea7b7e39b830b2cd848dc7dde759926bb896f8e8/Flask_Limiter-1.4-py3-none-any.whl
...
Collecting ctranslate2==1.17.1 (from argostranslate==1.1.2->libretranslate==1.1.0)
  Could not find a version that satisfies the requirement ctranslate2==1.17.1 (from argostranslate==1.1.2->libretranslate==1.1.0) (from versions: )
No matching distribution found for ctranslate2==1.17.1 (from argostranslate==1.1.2->libretranslate==1.1.0)

or

$ python3 main.py
Traceback (most recent call last):
  File "main.py", line 1, in <module>
    from app import main
  File "/home/x/Documents/devel/LibreTranslate/app/__init__.py", line 1, in <module>
    from .main import main
  File "/home/x/Documents/devel/LibreTranslate/app/main.py", line 2, in <module>
    from app.app import create_app
  File "/home/x/Documents/devel/LibreTranslate/app/app.py", line 1, in <module>
    from flask import Flask, render_template, jsonify, request, abort, send_from_directory
ModuleNotFoundError: No module named 'flask'

Python3 is 3.7.3.

Any hints?

Translation speed is to slow on HDD

Up to 8 sec on trying translate A4
I think, it would be nice to cache database indexes in the RAM as the argosmodels collection has about 1Gb summary.

Issue with docker-compose

Hey,

Thanks for the good job ! But I have an issue with dockeer-compose, indeed when I use docker-compose up -d --build the container launches perfectly :

alt text

but when I go to http://localhost:5000 there is nothing.
However, with docker run -ti --rm -p 5000:5000 libretranslate/libretranslate it works.
I would like to know if I made something wrong knowing that I have the latest version. Docker-compose is very important for my deployment configuration.

Thank you in advance !

[FR] Limit to two languages

It crashed my 1GB vm when I scrolled through the dropdown languages, I assume it had to load all the language models and process them at once. The API already supports /languages but I was unable to find anywhere to limit to a single model. Thanks!

Translating a text with arteristics to its sides

I don't understand why the English translation (I don't know if it happens also with the other languages) of the phrase I put in Spanish appears with the sign and not with the asterisks that I had put in the Spanish phrase.

image

Add frontend language default parameter

Hello there,
first of all: really great work!

I would love to see a parameter/config/function to set the default language for the frontend. So that you do not always have to set them by hand.

Here I would like to take over the implementation. But first I would like to clarify a few questions with you:

  • does this feature make sense at all?
  • Implementation via cookie in the frontend so it remembers for each user?
    OR
  • Add two cli arguments and integrate them into the frontend via template variables?
    OR
  • cli arguments and fetch in frontend via api. e.g. GET /languages/defaults -> { "source": "en", "target": "es"}
    OR
  • cli arguments and fetch in frontend via api. e.g. GET /frontend/settings -> { "source": "en", "target": "es", "darkmode": true}
    -> here already in view for EsmailELBoBDev2@68ce70c

I prefer one of the last two options. But I am not sure if this is a bit overloaded, because the main feature of this project is the translation api and the frontend is more of a showcase? (So this feature should rather be included in the project which uses this API...)?

Batch translation

It would be very nice if the translate endpoint would accept multiple texts (or qs) at once, like DeepL does.

This should significantly improve the translation performance.

  • If q is a string, translate it, and return the translation as a string.
  • If q is an array (list of strings), translate each item, and return the translations as an array (list of strings).

Note that DeepL has a limit of 50 texts per requests, which could be an argument like --char-limit (e.g. --text-limit or --q-limit).

grafik
(Source: https://www.deepl.com/docs-api/translating-text/request/)

[bug] Chinese Language - "hello how are you" phrase

Used Online version to test quality of all language; surprisingly I ran into this bug;

Issue: This phrase is not translated in Chinese

Reproduce: Input English text as per below;

English input: hello how are you

Chinese output:  hello how are you

But Chinese language itself does seem to work though, if you write something else aside from those 4 words for example:

English input: hello how are you
how are you
test
test

You will get this:

英文投入: hello how are you
How are you
测试
测试

Image

PS: Thank you for making LibreTranslate. If model training isn't too CPU/GPU intensive (got old PC) and easy to train, I would love to contribute.

Possibility of Offline Usage?

I've been looking for an offline translator that could be used as an desktop & mobile browser (chromium & Firefox) extension. Current web page translation tools 1) send your data away 2) don't work offline 3) won't be as performant on slow connections...

Any thoughts on the feasibility of this idea?
Thanks!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.