maohbao/error-detection-mt

Critical Error Detection in Machine Translation - Project for Computational Semantics for NLP, ETH Zurich, 2021.

Critical Error Detection in Machine Translation with BERT, XLM-RoBERTa and HuggingFace

Introduction

In this repository, we tackle the problem of critical-error detection in machine translation. To that end, we fine-tune different transformer based language models, including BERT and the multilingual XLM-RoBERTa model, and facilitate extensive evaluation. This PyTorch implementation leverages the Hugging face transformers to download pre-trained models, enable quick research experiments, access datasets and evaluation metrics.

All our code is accessible through Jupyter Notebooks that can be run on Google Colab, allowing full reproducibility of results.

This task is part of the WMT'21 shared task on quality estimation.

---

Task and Dataset

The goal of this task is to predict sentence-level binary scores indicating whether or not a translation contains (at least one) critical error. Translations with such errors are defined as translations that deviate in meaning as compared to the source sentence in such a way that they are misleading and may carry health, safety, legal, reputation, religious or financial implications.

The data consists of Wikipedia comments in English extracted from two sources: the Jigsaw Toxic Comment Classification Challenge and the Wikipedia Comments Corpus, with translations generated by the ML50 multilingual translation model by FAIR. It contains instances in the following languages:

• English-Czech
• English-Japanese
• English-Chinese
• English-German

The dataset is provided by WMT'21 and we prepared it as a serialized pickle file for the purpose of this task. See all the details here.

---

We include:

• End-to-end pipeline implementation (training, validation, prediction, evaluation)
• Easy adaptability, possible to be extended
• Facilitation of quick experiments, even with limited computational resources, using Google Colab
• Threshold choice for the classification decision (not necessarily 0.5)
• Reproducible results with seed settings

Parts of this code have been taken and adapted from NadirEM and we thank the author for providing such a nice template.

---

Approaches

As mentioned above, we implement different approaches, most of which can be found in our general notebook.

These are:

• Monolingual BERT based on source and backtranslations. The translations have been backtranslated using the ML50 translation model, which is the same as used for the original translations. See how this was done here.
• Fine-tuning XLM-RoBERTa with various adaptations. Specifically, we also perform transfer-learning using weights from TransQuest.
• A Siamese approach, combining two monolingual models for English-(Japanese/Chinese). See siamese_ja_zh.ipynb.

---

Credits: Marine Hoche & Alex Hägele