Comments (9)
Goal of a process pool is to abstract the management of the worker processes from the main application/service.
Hence, there is not a proper moment when running a callback as the nature of the problem is asynchronous. The main reason why callbacks were ran before terminating the processes was to execute the post-processing as fast as possible with the idea of handling the process termination in a later phase.
How is the main loop supposed to know what resources to clean up?
from pebble.
from pebble.
This is not the first time something similar comes out and I now see a valid Use Case for it.
I cannot see issues where the current behaviour is expected (running a callback while the timing out/cancelled process still runs) so I think it would be safe changing the behaviour.
I ran some tests over the WE and did not identify issues. I will make a new release soon with this enhancement.
from pebble.
@noxdafox: thank you very much! Glad to hear!
from pebble.
Issue resolved in 5.0.1
. Thanks for reporting.
from pebble.
Thanks super much! However, i find this is not working as i expected (and i think this is possibly an issue with my expectations), because cancellation of a running task is not notified to the user callback through task_done (the calling of which we just reordered in 5.0.1
). What happens when you cancel a running task scheduled on a process pool:
-
pebble/common.py: 77
set cancelled state, and invokes callbacks! -
pebble/pool/process.py
in separate thread,
a. pool manager loop updates pool's status (line 187),
b. which updates task status (line 240),
c. which for every cancelled task (line 251)
d. stops the worker and calls task_done, which (line 303)
e. calls set_running_or_notify_cancel() on the task's future (line 311).
All of 2 will happen after 1, since it runs in a different thread that will be assigned a processing slice after 1 is done (if i understand Python correctly, or at least this ordering may occur). This is not easy to fix. Fixing it to behave like a finished or failed task (callback invoked after this status is actually reached) would involve adding an extra state to the PebbleFuture (e.g. BEING_CANCELLED
), the task manager picking up on this state and cancelling the worker, and then task_done calling a new member function of PebbleFuture set_cancelled()
(analogous to set_result()
and set_exception()
). cancel()
would skip all this machinery if State is not running (could defer to super class implementation). set_cancelled()
would presumably call add_cancelled()
on any waiters, making calling set_running_or_notify_cancel()
superfluous in this case(?). On top of that, what should cancel()
return when invoked on a task that is already running? Probably False since it isn't cancelled yet.
All rather complicated in any case i guess. Perhaps i would have more success installing my own waiter (implementation detail as that is) which would get invoked at the right time for cancellation and normal or exceptional finishes now in 5.0.1! I'll give that a try. I'm happy to think along and try out the above however, should you wish to pursue it. Thanks in any case!
from pebble.
Yes, registering a waiter and (ab)using it as a way to get my callback run at the right time does the trick! So my problem is solved, if a bit dirtyly
from pebble.
I actually forgot that callbacks are executed on Future.cancel()
. This is why I was mentioning that it's hard to provide guarantees given the asynchronous nature of the problem.
The callback should work on TimeoutError
as expected but I won't change the Future
behavior as it would diverge too much from the original concurrent.futures
API design.
Glad you found a workaround for your need.
from pebble.
@noxdafox, fully agree, you shouldn't change future's behavior. It may however be good to document that Future.cancel()
may lead to callback execution before the worker process is actually cancelled.
Thanks again for the 5.0.1
enhancement, which made my workaround possible!
from pebble.
Related Issues (20)
- Pebble wheels pushed to PyPI are incorrectly tagged HOT 3
- The timeout argument of ProcessPool().submit() is inconsistent with ThreadPool().submit() HOT 3
- Bug: new `submit` function makes it impossible to call a function that has a `timeout` argument HOT 6
- Logging process name inside concurrent.process HOT 2
- How to handle errors when using pool.schedule HOT 1
- shutdown of main program HOT 2
- @concurrent.process returned future blocks/hangs on running(), cancelled(), done() calls HOT 2
- Get information about broken process HOT 4
- Documentation for Pebble indicates threads created with a ThreadPool are cancellable HOT 1
- Channel mutex timeout HOT 6
- How can I use a multiprocessing.manager alongside with pebble to avoid re-importing the function everytime? HOT 1
- Type hint error of wrapped function HOT 7
- ISSUE with with ProcessPool when scheduled function return exception (not raise it) HOT 1
- BUG with ProcessPool when scheduled function raise BaseException HOT 1
- issue with handling frozen exceptions in worker thread or process HOT 1
- Cannot create pebble.ProcessPool() multiple times within a single run, all task got stuck and time out HOT 3
- error while running @concurrent.process repetitively HOT 4
- using multiprocess context (for dill support) no longer working HOT 2
- Workers using 100% CPU not getting killed after timeout HOT 11
- Behavior of Process.map with chunksize > 1 and a timeout HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pebble.