Reproduction of Ja-En

[neulab]

Reproduction of Ja-En

[twadada]

I see, but I think it should be clear that you used the option for zh-en to allow others to compare the scores in a fair condition. (apologies if it's been already mentioned somewhere). Thanks a lot for your quick responses!

[zdou0830]

Hi, could you set max_steps to 40000 and see if you can reproduce the results? If still not, could you send me your datasets ([email protected]) and I'll try to figure out the reason.

Also, just to confirm, could you reproduce the results of multilingual BERT without fine-tuning?

[twadada]

Also, just to confirm, could you reproduce the results of multilingual BERT without fine-tuning?

Yes, I did in Ja-En.

However, actually I got a slightly different result in Zh-En; I got 17.9 but the paper says it is 18.1. I think this has something to do with how to handle possible alignments, which do not exist in Ja-En. In fact, using the ignorePossible option at tools/aer.py produced 18.1, and 17.9 without it. So I'm assuming you used the ignorePossible option to calculate the AER, is that correct?

[zdou0830]

Thanks! Yes, for Zh-En that is correct.

[twadada]

So did you use the --ignorePossible option for all the language pairs? If so, I think that should be noted at least in README, as it is not the standard way of calculating AER.

[zdou0830]

Nope, for all the other language pairs I didn't use that.