Multiprocessing Hanging in Python 3.6+

[anttttti]

Multiprocessing Hanging in Python 3.6+

[canivel]

Well i found ... it does not work inside jupyter because it need to run directly , In short, the subprocesses do not know they are subprocesses and are attempting to run the main script recursively.
So don't use jupyter for multiprocessing ...

[nelsonperry]

Hi.
I would just like to add, as a fairly novice programmer, that I experience exactly the same problem on Linux Ubuntu and Mint
when using the python wrapper Idle. Programmes using the multiprocessing module just hang for ever - even CTRL-C will not interrupt the endless loop.
However, running Python from the command line gives flawless execution.
I have had exactly the same frustrations when writing bits of code using the Curses module. They work fine from the command line but will not run in IDLE.
The reason, I think, is although it looks as though the IDLE window (and I guess Jupyter too -though I've not used that ) is some kind of terminal window, in fact it is no such thing and the overheads in having this so-called user-friendly environment are stopping Python from its usual memory management tasks.

Moral of the story: Stick to a basic editor and the command line.

[anttttti]

Jupyter shouldn't have problems with multiprocessing, since this is basic Python functionality. This could still be a Windows issue, or issue with Windows & Python3. The library was initially developed under Windows & Python2, and there were no issues with multiprocessing earlier.

Windows uses spawn, linux uses fork to create the subprocesses. This introduced some differences in behaviour under Python2.

Most often multiprocessing with Wordbatch hangs due to not enough memory to start a new worker. Unfortunately there's no signal from a hanging subprocess to multiprocessing, so it's not possible to know when a subprocess has hanged. Modifying the multiprocessing library to catch hanging subprocesses could help debug and fix issues like this.

[canivel]

The strange is if I execute from the command line it works flawless with multiprocessing... I've 128gb ram locally, and the max consumption that i tested was a matrix of (14689280, 8388608) that took 25gb of ram and 240 secs to execute with 8 procs from the command line...

From jupyter even if I try the (14345, 8388608) matrix it goes no where with multiprocessing, but works with serial...

[canivel]

maybe is something like this
jupyter/notebook#1703

[M-Harrington]

@anttttti I've reproduced the error using a very simple function. It does seem to be a problem with Windows and Jupyter. It's not clear to me as to why it does not work. I've posted my short example in a stackoverflow post here.

[anttttti]

@anttttti I've reproduced the error using a very simple function. It does seem to be a problem with Windows and Jupyter. It's not clear to me as to why it does not work. I've posted my short example in a stackoverflow post here.

Thanks for reporting this. Strange that Jupyter does that. With Wordbatch 1.4 you can try Dask, Ray, Spark or Loky as an alternative backend, to see if those better with Jupyter.