Comments (4)
Async mode works by limiting profiling to one async context. For example, you could start a profiler in the context of a web request, and then when you await on a database call, thats detected as 'out-of-context', and so the time spent there is attributed to the await
.
But when starting from the command line, at the start of any program there is no async context because the program hasn't started any async coroutines yet.
This comment means that setting the async_mode would have no effect, because what that does is 'tag' the contextlib context with the profiler. But this context is duplicated into all subsequent async tasks, so the entire program is considered 'in-context' from the perspective of the profiler. As a result, we can't track awaits.
I can kinda see what you mean about wanting to profile a program from the perspective of many async tasks. I'm not sure if that would be possible. I'd have to hook into the creation of coroutines.
I also haven't figured out how to do multiple threads yet, which might be the first port of call :)
What's your program, out of interest?
from pyinstrument.
hi, thanks for your detailed response, my program just transform and check files, but it is a very long process and it has a couple of services that are ran concurrently, fetching files and gathering async request, usually using asyincio.to_thread() (some libraries i use are not properly built for async operations, so i need to put them in another thread) which may also causes issues, so i got both of them at once.
The rest of the code is synchronous, but I'm experimenting putting more parts of the code into async functions, with the drawback they are obsucured in the pyinstruments output, of course i can analyze them one by one, but i thought it may be nice to have everything at once.
In any case many thanks for the fantastic profiler, I love it :) if there is anything i can help, tell me.
from pyinstrument.
Yeah i think best bet for you would be to figure out which of those services is the slowest (i.e. the rate-determining step), and run pyinstrument from inside the async function/method.
From the perspective of Pyinstrument, aside from figuring out how to get notified of new async contexts, I don't think it's easy to decide which of those contexts you care about (e.g. your service) versus the frames which are stuff you don't (e.g. waiting on file I/O). I might be wrong there, maybe there's a way, but it's not obvious to me.
from pyinstrument.
i think everything is interesting, I would love to see displayed each co-routine in a color when working and another when waiting, but I need to take a look in the code here to understand a bit more all of the things you mention, because I have no idea yet on how feasible it is.
In any way feel free to close the issue.
from pyinstrument.
Related Issues (20)
- Significant performance overhead when timing requires a syscall on Linux HOT 18
- Pyinstrument hiding useful frames HOT 8
- Automatic Web Deployment HOT 1
- Interpolation bug in jupyter magic HOT 3
- replace deprecated appdirs with platformdirs HOT 1
- HTML Renderer show-all HOT 2
- `pytinstrument` does not track the call stack of `__aenter__` and `__aexit__`
- Console renderer has much more detail than speedscope renderer HOT 1
- pstats renderer total time HOT 4
- nevergrad import fails when profiler is active HOT 3
- Profiler().write_html() does not accept "show_all" argument, in contrast with API docs HOT 3
- Asyncio background tasks are added to total wall clock time
- recursive functions HOT 2
- Memory consumption increases when profiling in an iteration
- [bug] pyinstruments not working TypeError: 'NoneType' object is not callable
- Example on how to disable profiling in FastAPI tests
- RuntimeError: cannot release un-acquired lock w/ gevent
- Pyinstrument ProfilerState object is not callable HOT 1
- profiler.output_text: float division by zero HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pyinstrument.