nlp2ct/meta-curriculum

Meta-Curriculum Learning for Domain Adaptation in Neural Machine Translation (AAAI 2021)

Meta-Curriculum

Meta-Curriculum Learning for Domain Adaptation in Neural Machine Translation (AAAI 2021)

Update: There are some problems with the OPUS corpus in terms of data quality. To make a fair comparison, we would like to suggest taking the results reported in a subsequent work (Table 2) as the reference. They reproduced the experiments on a cleaner benchmark and further improved the performance. We appreciate their effort in checking the results.

Citation

Please cite as:

@inproceedings{zhan2021metacl,
  title={Meta-Curriculum Learning for Domain Adaptation in Neural Machine Translation},
  author={Zhan, Runzhe and Liu, Xuebo and Wong, Derek F. and Chao, Lidia S.},
  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
  volume={35},
  number={16},
  month={May}, 
  year={2021},
  pages={14310-14318}
}

Requirements and Installation

This implementation is based on fairseq(v0.6.2) and partial code from Sharaf, Hassan, and Daume III (2020).

• PyTorch version >= 1.2.0
• Python version >= 3.6
• CUDA & cudatoolkit >= 9.0

git clone https://github.com/NLP2CT/Meta-Curriculum
cd Meta-Curriculum
pip install --editable .

Pipeline

1. Train a baseline model following the SOP in examples/translation/README.md. See our script general_train.sh (also utilize it for baseline finetuning).
2. Use the scripts containing in the folder lm_score/general_domain_script to train a general domain NLM.
3. Finetune the domain-specific NLM following the script lm_score/finetune_lm/continue_lm_domain.sh.
4. Score the adaptation divergence for domain corpus:

CUDA_VISIBLE_DEVICES=0 python lm_score/finetune_lm/score.py --general-lm GENERAL DOMAIN NLM PATH --domain-lms DOMAIN NLMs PATH --bpe-code BPE CODE --data-path DOMAIN CORPUS PATH --domains [DOMAIN1, DOMAIN2, ...]

Please note that you may separately run the LM training/scoring with higher version fairseq (>=0.9.0) due to the API changes.

5. Prepare meta-learning data set using meta_data_prep.py.

python meta_data_prep.py --data-path DOMAIN_DATA_PATH --split-dir META_SPLIT_SAVE_DIR
                              --spm-model SPM_MODEL_PATH --k-support N --k-query N
                              --meta_train_task N --meta_test_task N
                              --unseen-domains [UNSEEN_DOMAINS ...] 
                              --seen-domains [SEEN_DOMAINS ...]

6. (Meta-Train) Train meta-learning model with curriculum using the script meta_train_ccl.sh.
7. Score the unseen domains using the script score_unseens.sh.
8. (Meta-Test) Finetune meta-trained model with curriculum using the script cl_finetune.sh.

🌟 COVID-19 English-German Small-Scale Parallel Corpus

See covid19-ende/covid_de.txt and covid19-ende/covid_en.txt (Raw data without preprocessing).

meta-curriculum's Issues

When should sentencepieces be used?

Hi, according to the paper, "the corpus is encoded with sentencepieces (Kudo and Richardson 2018) using 40k joint dictionary."
But it seems that a spm-model is not needed before step 5 preparing meta-learning data. So I wonder when should I train this model?
I only realize that you used subword-nmt in the pre-process of lm_train, but still not knowing when to use sentencepieces.

Some questions about paper reproduction

Hi, I am a rookie in NMT, nowI want to try to combine meta-learning and NMT, and then I saw your paper and found your open source code so far, but your open source project readme file is not detailed enough on how to reproduce, I don’t know how to run。 So, could you write more detailed explanation? For example, I downloaded nine fields of data, but I don't know how to put it into your project, and what code is running to process the data? Or put it directly into the folder? I'm not sure how to proceed, if you can see my issue, please help me, thank you!

Datasets are different

I am testing a vanilla T5-small model recently but get very low Bleus on almost every domain data.
I found that we have different amounts of sentences for each domain.
I guess you have filtered the data so I wonder if you can share me your datasets.

Adapter Modules

I am trying to find the code of the adapter modules that Meta-NMT added to the model.
They don't mention the details in the paper and the code is hard to understand.
Since Meta-Curriculum uses most of their code, so I hope this question can be answered here.

Thank you!