Adding translation model in the pipeline for the back-translation task

[fani-lab]

Adding translation model in the pipeline for the back-translation task

[farinamhz]

Based on my research on available choices for the translation model, we have the following options:

• Google Translate API

  • It is not free
  • For the Translation API Basic, the First 500,000 characters per month are free with $10 credit every month, and it also provides $ 300 after creating google cloud - and 500,000 to 1 billion characters per month are $20 per million characters.
  • There are some Pypi libraries such as googletrans 3.0.0 and googletrans-py 4.0.0 which are wrappers as a free Google Translate API for Python. However, none of them works now, as developers are not updating them anymore.

• Translate Library of Pypi (translate 3.6.1): It is free but not accurate translation

• Pretrained models, which are popular, open-source, and available in Hugging Face:

  • Flan-T5 model from Google:

    • An enhanced version of T5

  • No Language Left Behind (NLLB) from Meta:

    • New model
    • Translations directly between 200 languages, including low-resource languages
    • With the following choices
      - facebook/nllb-200-1.3B
      - facebook/nllb-200-3.3B
      - facebook/nllb-200-distilled-1.3B
      - facebook/nllb-200-distilled-600M

  • MT5 from Google:

    • A multilingual variant of T5
    • Older than others

For the early experiment, I have chosen the NLLB, nllb-200-distilled-600M version, and it is working properly on the back-translation task for text-to-text.
However, this needs some other work for the allignments to be added to the pipeline.

@hosseinfani

[farinamhz]

Hi @hosseinfani ,
We have the module of translation now, which is working on a dataset of texts, and results will be the translation and back translation of those texts, each of them a csv file containing one column of sentences.
You can see early results in this commit: link

Now, I will go to work on the word alignment module.

[hosseinfani]

The available languages in nllb:

https://github.com/facebookresearch/flores/tree/main/flores200#languages-in-flores-200

[hosseinfani]

@farinamhz
I think we can safely close this issue. Let me know otherwise.