cannot reproduce the results of unsupervised NMT

[facebookresearch]

cannot reproduce the results of unsupervised NMT

[tobyyouup]

@glample I have already reproduced the BLEU score of en-fr, nearly 25 BLEU score, thanks for your help!

[glample]

Thank you for confirming. I just reran the command above, and after 4 days I get:

Epoch - 127

PPL  - en -> fr (valid)       - 35.51
PPL  - fr -> en (valid)       - 46.04
BLEU - en -> fr (valid)       - 22.41
BLEU - fr -> en (valid)       - 20.92

PPL  - en -> fr (test)        - 19.46
PPL  - fr -> en (test)        - 25.52
BLEU - en -> fr (test)        - 25.99
BLEU - fr -> en (test)        - 25.10

PPL  - en -> fr -> en (valid) -  1.79
PPL  - fr -> en -> fr (valid) -  1.90
BLEU - en -> fr -> en (valid) - 62.95
BLEU - fr -> en -> fr (valid) - 60.39

PPL  - en -> fr -> en (test)  -  1.81
PPL  - fr -> en -> fr (test)  -  1.75
BLEU - en -> fr -> en (test)  - 62.08
BLEU - fr -> en -> fr (test)  - 61.50

[glample]

Hi, can you send me your full training log?

[tobyyouup]

Thanks for your reply, I put the log in the gist: https://gist.github.com/tobyyouup/1b093ddec2155a053df3171df463bf35

Look forward to your further reply :)

[glample]

The parameters in your log slightly differ from the ones you posted above, in particular:
--n_enc_layers '1' --n_dec_layers '1' --share_enc '1' --share_dec '1'
as opposed to:
--n_enc_layers '4' --n_dec_layers '4' --share_enc '3' --share_dec '3'

But the date/time seems different so I assume you pasted the wrong logfile?

[tobyyouup]

oops, I just use one layer encoder and decoder in the shell scripts. No wonder cannot get the high BLEU score. I will fix and run again. Will report to you if I reproduce the score. Thanks!

[chang-xu]

@glample Have you tried to reproduce the results on en<->de? How about the performance? I just get a very low bleu after 70 epochs following the settings of en<->fr:

pl_de_en_valid -> 53.356531
bleu_de_en_valid -> 15.140000
ppl_en_de_valid -> 59.143495
bleu_en_de_valid -> 11.740000
ppl_de_en_test -> 40.975900
bleu_de_en_test -> 17.100000
ppl_en_de_test -> 43.684113
bleu_en_de_test -> 13.150000
ppl_de_en_de_valid -> 3.449342
bleu_de_en_de_valid -> 34.960000
ppl_de_en_de_test -> 3.369246
bleu_de_en_de_test -> 35.320000
ppl_en_de_en_valid -> 3.278750
bleu_en_de_en_valid -> 42.210000
ppl_en_de_en_test -> 3.132072
bleu_en_de_en_test -> 42.580000

[glample]

@chang-xu were you able to reproduce the en-fr results? Or you only tried en-de?

[chang-xu]

@glample I tried to reproduce the en-fr results:

below is my reproduced results in en-fr after 83 epochs:

ppl_en_fr_valid -> 36.674233
bleu_en_fr_valid -> 20.800000
ppl_fr_en_valid -> 49.298057
bleu_fr_en_valid -> 20.070000
ppl_en_fr_test -> 20.068776
bleu_en_fr_test -> 24.230000
ppl_fr_en_test -> 27.563477
bleu_fr_en_test -> 23.720000
ppl_fr_en_fr_valid -> 1.966098
bleu_fr_en_fr_valid -> 58.630000
ppl_fr_en_fr_test -> 1.806122
bleu_fr_en_fr_test -> 59.120000
ppl_en_fr_en_valid -> 1.889292
bleu_en_fr_en_valid -> 61.100000
ppl_en_fr_en_test -> 1.892268
bleu_en_fr_en_test -> 60.780000