No language model pretraining in these results?

[facebookresearch]

No language model pretraining in these results?

[glample]

Hi,

For everything we report in the paper, we never explicitly train a language model (we never use --lm_before or --lm_after). What we meant in the paper is that by using the denoising auto-encoder loss, we are "sort of" training the language model part of the decoder.

For instance, if an input sentence is "this restaurant is really good", and that the noise (word dropout / word shuffling) transforms it to: "this restaurant really good". Then the decoder will learn to reconstruct "this restaurant is really good" given the missing word in the noisy sentence. To do so, the decoder has to learn something about the target language, and to learn to output fluent sentences. This is what we meant by training the language model.

[pranaymanocha]

Hi,
Thanks for the prompt response. I get the point that the denoising autoencoder acts like an LM. But shouldn't the denoising autoencoder be pre-trained first before being used in back-translation?

[glample]

You can do that. At the beginning, the back-translation is kind of useless because the generated translations are very bad. We tried to have a schedule to disable the back-translation initially, but this didn't make any difference on the final performance.

[pranaymanocha]

Thanks for the reply. Appreciate it.

[Julisa-test]

Hi @glample ,

My task is mn-zh translation，and when I run the task on unsupervised NMT, the result just is bleu_mn_zh_test -> 0.270000 and bleu_zh_mn_test -> 1.850000 after 20 epochs, I think maybe the Language model is not good, if I pre-trained the LM, how should I use it to the unsupervised NMT?

Thanks for your time.

[glample]

Hi,

I would recommend having a look at https://github.com/facebookresearch/XLM/ , XLM provides efficient framework for language model pretraining / unsupervised MT, and works much better than the code in this repo.

[Julisa-test]

Thanks for the reply. @glample