Raise EOFError

[facebookresearch]

Raise EOFError

[jplu]

I did solved the issue by using Python 3.5.3, apparently Python 3.6 does not likes concurencies when multiple heavy processes already run on the machine :(

[glample]

Could you share the full log file? When you say "the NMT runs for a while" is it after a few epochs or something? Usually the issues on the CPU threads happen immediately or not at all.

[liujiqiang999]

Sorry for late reply. This is the full log file, nohup.out.txt . This error did occur during training.

[jplu]

Up! I get the exact same error at the difference that the training crashs at the beginning of the epoch 0. Find here my log.txt file.

OS: Debian unstable (sid)
GPU: K80
Python version: 3.6.7
Pytorch version: 0.4.1

[jplu]

I did update my Pytorch version to 1.0.0.dev20181022 and still the same exception :(

[glample]

Did you have to change something in the code? Or just using another version of Python solved the issue?

[jplu]

Well, to be more specific, firstly I noticed that adding a timeout to the poll() function at line 72 partially solved the problem, because apparently there was a sync issue between the read/write in the Pipe. Unfortunatelly, this solution was slowing the process a lot and sometimes, depending of the usage of the machine, was changing. Then I wanted to see if the Python 3.5 version was working or not, I did install it, run the training and it worked perfectly without changing any line of code.

I'm not good enough in Python to be able to explain what is the difference between 3.5 and 3.6 in multiprocessing to know really what was the issue, sorry :(

[frankang]

In my case, this might be caused by a mismatch between the pytorch version (compiled on cuda9.0) and the cudatoolkit version (10.0). I reinstalled the correct pytorch, cudatoolkit, and upgraded to python 3.7.7, issue gone.

[vincentg]

FYI, this issue was solved in Python 3.7 by this python issue (see the same stack trace there):
https://bugs.python.org/issue31069

It can likely be closed