Non-deterministic results when -ncpus != 1 (mgiza bin)

[moses-smt]

Non-deterministic results when -ncpus != 1 (mgiza bin)

[hieuhoang]

I doubt anyone will look into it. Why is it a problem? In fact, I'm surprised cpu=1 is deterministic

[cgr71ii]

Well... since there is not a proper documentation where I could look into it, I thought it was not the expected. Since you are not surprised about this, am I wrong thinking that to be non-deterministic is the expected?

[hieuhoang]

you're right that the results should be determinstric or non-deterministic regardless of how many threads are used.

I don't know the code that well so don't take my word for it. In my mind, it should be non-determistic during training due to randomness in word clustering. However, you seem to find the it non-deter. even during inference. That could be an issue.

I'm not sure who can come to your rescue, mgiza is abadonware these days. Perhaps @edwardgao, the original author has some time

Btw, running the command with your data crashes for me. I'm not sure if that has anything to do with it

[cgr71ii]

I have run the commands again and they work for me. Have you run

cat corpus.fr-en.cooc.1 corpus.fr-en.cooc.2 > corpus.fr-en.cooc

? I had to split the file to be able to upload it to the issue.

If you share the log perhaps I could find if something is wrong in my installation.