Comments (5)
Thank you for responding)
In fact, I would like to expand the task. I found this in the process.py code:
def update_tasks(self):
"""Handles timing out Tasks."""
for task in self.task_manager.timeout_tasks():
self.task_manager.task_done(
task.id, TimeoutError("Task timeout", task.timeout))
self.worker_manager.stop_worker(task.worker_id)
for task in self.task_manager.cancelled_tasks():
self.task_manager.task_done(
task.id, CancelledError())
self.worker_manager.stop_worker(task.worker_id)
You write the task.timeout
variable to the exception, but it doesn't seem to make much sense to me. Since if this exception occurred and I am the person who ran the code, I know what timeout I specified. Why not do it like this?
def update_tasks(self):
"""Handles timing out Tasks."""
for task in self.task_manager.timeout_tasks():
self.task_manager.task_done(
task.id, TimeoutError("Task timeout", task.timeout, task.worker_id, task.id))
self.worker_manager.stop_worker(task.worker_id)
for task in self.task_manager.cancelled_tasks():
self.task_manager.task_done(
task.id, CancelledError())
self.worker_manager.stop_worker(task.worker_id)
thus, intercepting an exception, it will not be difficult to find out the pid of the process and the number of the task for which this exception occurred.
I propose to do the same in the case of raise Exception in task.
Thanks!
from pebble.
Hello,
it is not possible to add the PID to all exceptions. CancelledError
is handled by the futures objects themselves and the exceptions raised within the user provided future are not to be touched in order to avoid overwriting information from the exceptions themselves.
Only error we can add the PID to is TimeoutError
.
What is the use case you have in mind? Why do you need the worker PID? This is internal information and should not be of concern to the user.
from pebble.
I agree to the worker_id account. Perhaps this information will be superfluous. But task_id is useful, for example, to understand which tasks have not worked. I think this is a frequent request
from pebble.
The task_id
is an internal information the pool uses to track the workload. It has no value for the user as it's an incremental counter.
If you want to understand which submission worked and which didn't, you can use the ID of the Future
object itself as it's the same object throughout the whole lifecycle.
In [4]: future = p.schedule(example)
In [5]: id(future)
Out[5]: 140154882138608
The future is always the same object even when passed to callbacks.
In [6]: def done_callback(future):
...: print(id(future))
In [7]: future.add_done_callback(done_callback)
140154882138608
Any attribute you will add to the Future
object will persist allowing you to understand which submission failed. For example, you can append to the Future
object the function and the arguments which were passed.
In [1]: import pebble
In [2]: p = pebble.ProcessPool()
In [3]: def done_callback(future):
...: print(future.function)
...: print(future.args)
In [4]: def my_function(arg1, arg2):
...: return 1
In [5]: future = p.schedule(my_function, args=[1, 2])
In [7]: future.function = my_function
In [8]: future.args = [1, 2]
In [9]: future.add_done_callback(done_callback)
<function my_function at 0x7fd7f3bf2820>
[1, 2]
from pebble.
Closing this issue due to lack of response. Please re-open if you require further clarifications.
from pebble.
Related Issues (20)
- Bug: new `submit` function makes it impossible to call a function that has a `timeout` argument HOT 6
- Logging process name inside concurrent.process HOT 2
- How to handle errors when using pool.schedule HOT 1
- shutdown of main program HOT 2
- @concurrent.process returned future blocks/hangs on running(), cancelled(), done() calls HOT 2
- Get information about broken process HOT 4
- Documentation for Pebble indicates threads created with a ThreadPool are cancellable HOT 1
- Channel mutex timeout HOT 6
- How can I use a multiprocessing.manager alongside with pebble to avoid re-importing the function everytime? HOT 1
- Type hint error of wrapped function HOT 7
- ISSUE with with ProcessPool when scheduled function return exception (not raise it) HOT 1
- BUG with ProcessPool when scheduled function raise BaseException HOT 1
- issue with handling frozen exceptions in worker thread or process HOT 1
- Cannot create pebble.ProcessPool() multiple times within a single run, all task got stuck and time out HOT 3
- error while running @concurrent.process repetitively HOT 4
- using multiprocess context (for dill support) no longer working HOT 2
- Workers using 100% CPU not getting killed after timeout HOT 11
- Behavior of Process.map with chunksize > 1 and a timeout HOT 5
- map function is extremely slow, seems to be executing sequentially HOT 1
- Cancel future returned from run_in_executor caused SIGTERM signal to main process HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pebble.