Corrupted Queue in Python's multiprocessing Pool implementation

[noxdafox]

Corrupted Queue in Python's multiprocessing Pool implementation

[noxdafox]

Hello,

this Pool implementation was added because there was none which would support disaster recovery and hanging workers while still offering a clean interface.

Billiard was the closest one but its undocumented and confusing API was causing issues in our systems. Moreover, it was not properly handling timeouts: celery/billiard#104. It might have improved by now.

The Pebble implementation is quite stable and we use it in some high load systems.

Among the issues/features it handles/offers:

• timing out operations
• task cancellation (with worker termination)
• crashing workers (python interpreter disasters or C libraries segfaults)
• allows to transfer large data between server and workers
• iteration (map) over faulty results

The only known issue I'm investigating (apart from issue #10) is the pool hanging in rare occasions where very large data is being transferred from the workers back to the server at once (in a single result). I'm not 100% sure is due to Pebble though.

The lines you linked deal with hard-killing an unresponsive worker (hanging in a C loop for example) without corrupting the Queue. But there are plenty of other small corner cases to deal with.

[andresriancho]

Matteo thanks for the detailed and quick response!

Seems that the code is pretty stable. When you say very large data is being transferred from the workers back to the server that happens when the workers are remote (over the network)? How large is very large?

I'll start writing some unittests for my current implementation and then migrate to Pebble. Hopefully it will all work well 👍

[noxdafox]

By server I mean the Pool process. Pebble is not capable to deal with remote processes. For that, I would recommend to take a look at Celery or Luigi.

As I said, I'm not sure about such issue as I've been dragged over other things a while ago. I'll try to resume the investigation in that regards and, if there's seems to be a real issue, I'll open a report myself.

Let me know if you encounter any problem integrating Pebble.

I will close this issue for now.

[andresriancho]

Integration with my code went really well andresriancho/w3af@27c6e25

I now have less code to maintain, and the whole thing seems to be working as expected.

Thanks for making pebble open source!

[noxdafox]

Np, glad it helps somebody.

I took a look at my notes regarding the "large data issue". It was a test which was hanging due to a mistake of mine in the test itself. I will fix the test in the following days (not really urgent).

On Windows, it might be problematic when transferring large amount of data through the Pool. I need to research a bit on how to improve that.

Nevertheless it's not a good idea to transfer large chunks of data via IPC. Better to rely on the filesystem for such use cases.