Cannot create pebble.ProcessPool() multiple times within a single run, all task got stuck and time out

[jcierocki]

Cannot create pebble.ProcessPool() multiple times within a single run, all task got stuck and time out

[jcierocki]

Additional INFO:

I switched to futureproof and then, while trying to spawn the 2nd pool I finally got a meaningful error instead of timeout. There was a problem with relative import. On the first run, it worked always perfectly (as long as I used ipython) but then every next pool was silently generating errors related to pickle being unable to serialize due to unresolved imports. Switching to top-lvl package imports fixed the problem, at least together with futureproof. I will later check whether it works now with pebble.

Anyway, this issue might not be the problem of pebble itself but inherited from multiprocessing or pickle. But still, those silent errors are extremely annoying which prevents me from using pebble in the near future.

[noxdafox]

Hello,

without a minimum reproducible example, I cannot validate the issue. Pebble does not silence the issue, a pickle.PicklingError would be raised by future.result() in your example.

A pool of workers is designed to abstract the execution logic while still maintaining good control over your resources. If you spawn multiple pools at the same time and assign lots of workers to them, you will starve of resources as your Operating System is mostly busing managing all the processes. This might be the reason why you see the timeouts, the functions don't get to do their job in time.

[noxdafox]

Closing due to inactivity. Please re-open if the issue occurs again.