sumerc / yappi Goto Github PK
View Code? Open in Web Editor NEWYet Another Python Profiler, but this time multithreading, asyncio and gevent aware.
License: MIT License
Yet Another Python Profiler, but this time multithreading, asyncio and gevent aware.
License: MIT License
On Python 2.7 when constantly calling start/stop/get_func_stats I repeatedly see the following error:
SystemError: NULL object passed to Py_BuildValue
Any idea what could be causing this?
Currently YThreadStat does not provide OS thread id. We only provide ctx_id
which is specific to yappi. It would be nice to have this information as users might be able to more introspection with this information from the Thread
object.
I noticed this when trying to use yappi with tornado. Here is a simple repro:
import yappi
yappi.start()
def test_mem():
buf = bytearray()
buf += b't' * 200
view = memoryview(buf)[10:].tobytes()
del buf[:10]
return view
if __name__ == "__main__":
test_mem()
Without yappi, the code completes successfully. With yappi I get the following error:
Traceback (most recent call last):
File "yappi_repro.py", line 14, in <module>
test_mem()
File "yappi_repro.py", line 9, in test_mem
del buf[:10]
BufferError: Existing exports of data: object cannot be re-sized
Reproducible on python 3.6, 3.7 and 3.8. I have not tried python 2.7
First of all, thanks a lot for this project, yappi is really the best Python profiler !
In my projects I use gevent extensively. I was full of hope when I found this blog article:
https://emptysqua.re/blog/greenletprofiler/
Someone made a greenlet profiler a few years ago on top of yappi...
But the project is Python 2, and it was coming with a modified, bundled version of yappi.
Now that yappi supports coroutines (with asyncio, unfortunately), could you please give me
some guidance how to redo what was done with greenletprofiler ?
I would be happy to contribute to yappi with gevent support, but I need some help - my C++
skills are rusty and I don't know the code.
I saw in this discussion #21 that it was something you thought about once... But I am afraid set_ctx_backend()
was finally not implemented ?
Thanks a lot
I tried to install yappi and got this error. I use Windows 10 1909 and Python 3.7. I cloned the repository from the github and removed the slash symbol from https://github.com/sumerc/yappi/blob/master/setup.py#L82. Then I installed yappi via python setup.py install
Collecting yappi
Using cached https://files.pythonhosted.org/packages/90/8e/ac718e8ffaffb7c92c09eb2cec33557f8f1b3ec32af5b7599a993a7700c8/yappi-1.2.1.tar.gz
Complete output from command python setup.py egg_info:
running egg_info
creating pip-egg-info\yappi.egg-info
writing pip-egg-info\yappi.egg-info\PKG-INFO
writing dependency_links to pip-egg-info\yappi.egg-info\dependency_links.txt
writing entry points to pip-egg-info\yappi.egg-info\entry_points.txt
writing top-level names to pip-egg-info\yappi.egg-info\top_level.txt
writing manifest file 'pip-egg-info\yappi.egg-info\SOURCES.txt'
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\dimam\AppData\Local\Temp\pip-install-b6ze0_h7\yappi\setup.py", line 95, in <module>
url=HOMEPAGE,
File "C:\Users\dimam\Desktop\Programming\exchange_bot\venv\lib\site-packages\setuptools\__init__.py", line 145, in setup
return distutils.core.setup(**attrs)
File "C:\Users\dimam\AppData\Local\Programs\Python\Python37\lib\distutils\core.py", line 148, in setup
dist.run_commands()
File "C:\Users\dimam\AppData\Local\Programs\Python\Python37\lib\distutils\dist.py", line 966, in run_commands
self.run_command(cmd)
File "C:\Users\dimam\AppData\Local\Programs\Python\Python37\lib\distutils\dist.py", line 985, in run_command
cmd_obj.run()
File "C:\Users\dimam\Desktop\Programming\exchange_bot\venv\lib\site-packages\setuptools\command\egg_info.py", line 296, in run
self.find_sources()
File "C:\Users\dimam\Desktop\Programming\exchange_bot\venv\lib\site-packages\setuptools\command\egg_info.py", line 303, in find_sources
mm.run()
File "C:\Users\dimam\Desktop\Programming\exchange_bot\venv\lib\site-packages\setuptools\command\egg_info.py", line 534, in run
self.add_defaults()
File "C:\Users\dimam\Desktop\Programming\exchange_bot\venv\lib\site-packages\setuptools\command\egg_info.py", line 570, in add_defaults
sdist.add_defaults(self)
File "C:\Users\dimam\AppData\Local\Programs\Python\Python37\lib\distutils\command\sdist.py", line 226, in add_defaults
self._add_defaults_python()
File "C:\Users\dimam\Desktop\Programming\exchange_bot\venv\lib\site-packages\setuptools\command\sdist.py", line 127, in _add_defaults_python
build_py = self.get_finalized_command('build_py')
File "C:\Users\dimam\AppData\Local\Programs\Python\Python37\lib\distutils\cmd.py", line 299, in get_finalized_command
cmd_obj.ensure_finalized()
File "C:\Users\dimam\AppData\Local\Programs\Python\Python37\lib\distutils\cmd.py", line 107, in ensure_finalized
self.finalize_options()
File "C:\Users\dimam\Desktop\Programming\exchange_bot\venv\lib\site-packages\setuptools\command\build_py.py", line 34, in finalize_options
orig.build_py.finalize_options(self)
File "C:\Users\dimam\AppData\Local\Programs\Python\Python37\lib\distutils\command\build_py.py", line 55, in finalize_options
self.package_dir[name] = convert_path(path)
File "C:\Users\dimam\AppData\Local\Programs\Python\Python37\lib\distutils\util.py", line 112, in convert_path
raise ValueError("path '%s' cannot end with '/'" % pathname)
ValueError: path 'yappi/' cannot end with '/'
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in C:\Users\dimam\AppData\Local\Temp\pip-install-b6ze0_h7\yappi\
Hi!
I noticed the changelog for 1.3.0 was missing.
Just thought I would bring this up so it can be added.
Thanks!
ytid = PyLong_FromLong(ycurthreadindex++);
never releases the reference.
This is a rather cosmetic issue, yet I'm running a logger.debug(yappi.get_func_stats({"tag": ctx_id}).print_all())
and find it difficult to read the output for the names (see below), is it possible to get a "nicer" output
I thought using a fmt
kwarg and passingsomething like in https://docs.python.org/3/library/logging.html#logrecord-attributes would be awesome, at least it would feel natural
I currently have to use debug_print()
to get a sense of those names but it would be awesome to be able to do a print_all(fmt=''%(filename)s %(funcName)s %(lineno)d")
for instance
Clock type: WALL
Ordered by: totaltime, desc
name ncall tsub ttot tavg
..ase.py:19 BenchMiddleware.__call__ 1 0.000016 0.009153 0.009153
..arlette/middleware/base.py:36 coro 2 0.000000 0.009081 0.004540
./main.py:116 middleware 1 0.000017 0.009018 0.009018
..se.py:28 BenchMiddleware.call_next 2 0.006288 0.006426 0.003213
..py:47 ExceptionMiddleware.__call__ 1 0.000009 0.006233 0.006233
../routing.py:578 APIRouter.__call__ 1 0.000029 0.006224 0.006224
..e/routing.py:200 APIRoute.__call__ 1 0.000015 0.006133 0.006133
..ckages/starlette/routing.py:38 app 1 0.000029 0.006118 0.006118
..ackages/fastapi/routing.py:100 app 1 0.000024 0.006025 0.006025
..ection.py:1400 Connection._execute 2 0.000035 0.004728 0.002364
..ction.py:1406 Connection.__execute 2 0.000027 0.004687 0.002344
..ion.py:1421 Connection._do_execute 2 0.002150 0.004648 0.002324
..py:137 circuits_json_in_postgresql 1 0.000034 0.003894 0.003894
..nection.py:443 Connection.fetchrow 1 0.000005 0.003417 0.003417`
If the callback passed to set_context_id_callback
returns a number that is too large for a C long
, yappi will throw an error along the lines of
Traceback (most recent call last):
...
...
File "./test_bench.py", line 37, in dispatch
tracked_stats[name] = yappi.get_func_stats({"name": call_to_track, "ctx_id": ctx_id})
File "C:\Users\royrenn\Documents\prog\fastapi-test\.pyenv\lib\site-packages\yappi.py", line 1001, in get_func_stats
stats = YFuncStats().get(filter=filter)
File "C:\Users\royrenn\Documents\prog\fastapi-test\.pyenv\lib\site-packages\yappi.py", line 660, in get
_yappi.enum_func_stats(self._enumerator, filter)
SystemError: <built-in function enum_func_stats> returned a result with an error set
with the following error sometimes(?) being logged:
[*] [yappi-err] context id callback returned non-integer
Having looked into it, this seems to be related to the use of PyLong_AsLong
here:
Lines 363 to 376 in 7b3f08f
I ended up running into this issue when running code similar to this on Windows. As it happens, id(my_object)
is a 64 bits unsigned integer on 64-bits CPython, but a long
on Windows is only 32 bits; only long long
s are 64 bits. This could turn out to be an issue in other use cases, so something should probably be done to adress the potential mismatch between the range of Python int
s and C types.
Hi @Suhail-MOHD,
Not sure if you have been following, but the issue you opened gevent/gevent#1670 seems to fixed/merged.
Do you have any idea on how we should move forward with this? Any change required on our side, maybe update documentation?
Hi
When attempting to build for Python 3.4 (with mingw32), I get the following error:
_yappi.c: In function '_yapp_callback':
_yappi.c:680:36: error: 'PyFrameObject {aka struct _frame}' has no member named 'f_tstate'
current_ctx = _thread2ctx(frame->f_tstate);
I opened up C:\Python34\include\frameobject.h
on my PC, and indeed no such member is present in the 3.4 version of frameobject.h
, despite it being present in earlier versions of Python.
What would be a way to work around this?
There is no CLI option and/or decorator to enable Greenlet profiling for now.
delete
When yappi is installed with pip, it crashes with the following error:
TEXT VERSION:
Collecting yappi
Using cached https://files.pythonhosted.org/packages/d2/92/7cd637a19fa2a10c0e55a44f8b36bcb83f0e1943ba8f1fb5edb15c819f2e/yappi-1.0.tar.gz
Installing collected packages: yappi
Running setup.py install for yappi ... error
ERROR: Command errored out with exit status 1:
command: 'c:\users\58394\appdata\local\programs\python\python37\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\Users\58394\AppData\Local\Temp\pip-install-zsdo72o2\yappi\setup.py'"'"'; file='"'"'C:\Users\58394\AppData\Local\Temp\pip-install-zsdo72o2\yappi\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record 'C:\Users\58394\AppData\Local\Temp\pip-record-1uijfgsm\install-record.txt' --single-version-externally-managed --compile
cwd: C:\Users\58394\AppData\Local\Temp\pip-install-zsdo72o2\yappi
Complete output (9 lines):
running install
running build
running build_py
creating build
creating build\lib.win-amd64-3.7
copying yappi.py -> build\lib.win-amd64-3.7
running build_ext
building '_yappi' extension
error: Microsoft Visual C++ 14.0 is required. Get it with "Microsoft Visual C++ Build Tools": https://visualstudio.microsoft.com/downloads/
----------------------------------------
ERROR: Command errored out with exit status 1: 'c:\users\58394\appdata\local\programs\python\python37\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\Users\58394\AppData\Local\Temp\pip-install-zsdo72o2\yappi\setup.py'"'"'; file='"'"'C:\Users\58394\AppData\Local\Temp\pip-install-zsdo72o2\yappi\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record 'C:\Users\58394\AppData\Local\Temp\pip-record-1uijfgsm\install-record.txt' --single-version-externally-managed --compile Check the logs for full command output.
Wondering when a function is treated as a "builtin". Are numpy, opencv, scipy, pandas, etc which are wrappers for compilied implementations "builtins"?
There is a note to run a manual test using scipy/numpy, but I cannot find a tests script or further details:
https://github.com/sumerc/yappi/blob/master/tests/manual_tests.txt
This test uses "yappi.start(builtins=True)" when using testing "old style co-routines" with asyncio, but not for all asyncio tests:
https://github.com/sumerc/yappi/blob/master/tests/test_asyncio.py
Hi,
This project is really awesome and useful for people looking for thread related information.
I was doing some PoC for a aiohttp project, here i found some unclear data in thread stats as follows:
I have created 2 aiohttp session :
Also from stats i m not able to get which thread is assigned to particluar function,it would be great if we can link somehow which thread belongs to which async code block, so it would help to find and improvise long running thread's code/function.
Most important it would be big advantage to have event loop stats also, as it will solve many problems faced in frameworks like aiohttp, where you would definitely like to know which threads are taking long time on eventloop.
Poc code :
import aiohttp
import asyncio
import time
import yappi
async def first():
url = "http://www.mocky.io/v2/5e11f1513100005a37593fa8?mocky-delay=2000ms"
async with aiohttp.ClientSession().get(url) as resp:
print(resp.status)
print(await resp.text())
async def second():
url = "http://www.mocky.io/v2/5e11f339310000c956593fa9?mocky-delay=4000ms"
session = aiohttp.ClientSession()
async with session.get(url) as resp:
print(resp.status)
print(await resp.text())
url = "http://www.mocky.io/v2/5e11f1513100005a37593fa8?mocky-delay=2000ms"
async with session.get(url) as resp:
print(resp.status)
print(await resp.text())
async def third():
for a in range(1000):
print(a)
def newf():
print("in newf")
async def fire():
s = time.time()
await third()
newf()
await second()
await first()
print(time.time() - s)
yappi.start()
loop = asyncio.get_event_loop()
loop.run_until_complete(fire())
yappi.stop()
yappi.get_thread_stats().print_all()
ThreadStat :
name id tid ttot scnt
Thread 1 139813237880576 0.186779 2
Thread 2 139813220611840 0.186705 2
_MainThread 0 139813331167040 0.041664 5
Filters appear to behave as negative filters, removing data from the output. Is this correct/intended?
This is a test harness I was using to determine filter behavior (called genr.py
):
import time
import random
import asyncio
import yappi
basename = 'genr'
debug = ''
if __debug__:
debug = '_debug'
statsname = f'./{basename}{debug}.pstats'
N = 4
SLEEP = 0.5
times = [random.random() for i in range(N)]
async def genr():
for t in times:
await asyncio.sleep(t)
yield t
async def sleeploop(i):
time.sleep(i)
# await asyncio.sleep(i)
async def run():
values = []
async for t in genr():
values.append(t)
await sleeploop(SLEEP)
return values
async def main():
print('start')
yappi.set_clock_type('wall')
tick = time.time()
yappi.clear_stats()
yappi.start(builtins=False)
values = await run()
yappi.stop()
tock = time.time()
filter = {}
##########################
# DEFINE FILTER HERE
# filter['module'] = 'genr.py'
##########################
stats = yappi.get_func_stats(filter=filter) # type: yappi.YFuncStats
for stat in stats:
print("%s %s %0.6f" % (stat.name, stat.module, stat.ttot))
# stats.print_all()
# stas.print_all()
# obj = yappi.convert2pstats(stats) # type: profile.Stats
# obj.dump_stats(statsname)
print(f'took: {tock - tick}')
assert values == times
print(times)
print(f'Total random time: {sum(times)}')
print(f'Total run time: {sum(times) + SLEEP}')
print('fin')
if __name__ == '__main__':
asyncio.run(main())
If this is intended behavior, would you accept a PR updating the filter docs to mention this ?
Deprecation warnings are raised due to invalid escape sequences. This can be fixed by using raw strings or escaping the literals.
find . -iname '*.py' | grep -v example | xargs -P 4 -I{} python3.8 -Wall -m py_compile {} ./tests/test_functionality.py:299: DeprecationWarning: invalid escape sequence \s
b'name\s+id\s+tid\s+ttot\s+scnt\s*\n', out
https://pypi.org/project/yappi/
There's some trick to pass content type along with long_description
through setup.py
to fix that.
It seems yappi
gets confused when a coroutine is scheduled as a task and awaited from the same function and the task returns first. If the task returns first the timings from the explicit await on the coroutine get lost (still contributes to the call count):
from asyncio import create_task, run, sleep
import yappi
yappi.set_clock_type('wall')
async def test(d=0):
t = create_task(sleep(5+d))
await sleep(5)
await t
for d in [-2, 0, 2]:
with yappi.run():
run(test(d))
print(f'#### d={d} (expect tsub ~ ttot ~ 2*5+d={2*5+d}):') # but we get 5-d if d < 0
yappi.get_func_stats(filter_callback=lambda x: yappi.func_matches(x, [sleep])).print_all()
yappi.clear_stats()
The current values (tsub, ttot, tavg) do not provide sufficient profiling data when dealing with temporary speed-ups or slowdowns.
My request is to add:
This would provide valuable data for tracking which sections of code are most affected by drops/spikes in performance (cpu & memory consumption).
Any ideas how to fix this?
I first got an error saying to install Microsoft Visual Studio C++ 14.0 or later build tools. I installed them from https://visualstudio.microsoft.com/visual-cpp-build-tools/
Here is the checkbox I ticked for the build tools. Which version of MSVC does yappi require?
❯ pip install yappi --no-cache-dir
Collecting yappi
Downloading yappi-1.3.2.tar.gz (58 kB)
|████████████████████████████████| 58 kB 2.0 MB/s
Building wheels for collected packages: yappi
Building wheel for yappi (setup.py) ... error
ERROR: Command errored out with exit status 1:
command: 'c:\tools\miniconda3\python.exe' -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Use
rs\\jaan\\AppData\\Local\\Temp\\pip-install-930ozymw\\yappi_e32707e233e74c018e9d5f8a6d367185\\setup.py'"'"'; __file__='"
'"'C:\\Users\\jaan\\AppData\\Local\\Temp\\pip-install-930ozymw\\yappi_e32707e233e74c018e9d5f8a6d367185\\setup.py'"'"';f
= getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools im
port setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '
"'"'exec'"'"'))' bdist_wheel -d 'C:\Users\jaan\AppData\Local\Temp\pip-wheel-6lwb2tps'
cwd: C:\Users\jaan\AppData\Local\Temp\pip-install-930ozymw\yappi_e32707e233e74c018e9d5f8a6d367185
Complete output (15 lines):
running bdist_wheel
running build
running build_py
creating build
creating build\lib.win32-3.8
copying yappi\yappi.py -> build\lib.win32-3.8
running build_ext
building '_yappi' extension
creating build\temp.win32-3.8
creating build\temp.win32-3.8\Release
creating build\temp.win32-3.8\Release\yappi
C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.28.29910\bin\HostX86\x86\cl.exe /c /no
logo /Ox /W3 /GL /DNDEBUG /MD -Ic:\tools\miniconda3\include -Ic:\tools\miniconda3\include "-IC:\Program Files (x86)\Micr
osoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.28.29910\include" /Tcyappi/_yappi.c /Fobuild\temp.win32-3.8\Release
\yappi/_yappi.obj
_yappi.c
c:\tools\miniconda3\include\pyconfig.h(59): fatal error C1083: Cannot open include file: 'io.h': No such file or direc
tory
error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools\\VC\\Tools\\MSVC\\14.28.29910\\bin\
\HostX86\\x86\\cl.exe' failed with exit status 2
----------------------------------------
ERROR: Failed building wheel for yappi
Running setup.py clean for yappi
Failed to build yappi
Installing collected packages: yappi
Running setup.py install for yappi ... error
ERROR: Command errored out with exit status 1:
command: 'c:\tools\miniconda3\python.exe' -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\U
sers\\jaan\\AppData\\Local\\Temp\\pip-install-930ozymw\\yappi_e32707e233e74c018e9d5f8a6d367185\\setup.py'"'"'; __file__=
'"'"'C:\\Users\\jaan\\AppData\\Local\\Temp\\pip-install-930ozymw\\yappi_e32707e233e74c018e9d5f8a6d367185\\setup.py'"'"';
f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools
import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__,
'"'"'exec'"'"'))' install --record 'C:\Users\jaan\AppData\Local\Temp\pip-record-wf2xdwwh\install-record.txt' --single-v
ersion-externally-managed --compile --install-headers 'c:\tools\miniconda3\Include\yappi'
cwd: C:\Users\jaan\AppData\Local\Temp\pip-install-930ozymw\yappi_e32707e233e74c018e9d5f8a6d367185\
Complete output (15 lines):
running install
running build
running build_py
creating build
creating build\lib.win32-3.8
copying yappi\yappi.py -> build\lib.win32-3.8
running build_ext
building '_yappi' extension
creating build\temp.win32-3.8
creating build\temp.win32-3.8\Release
creating build\temp.win32-3.8\Release\yappi
C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.28.29910\bin\HostX86\x86\cl.exe /c /
nologo /Ox /W3 /GL /DNDEBUG /MD -Ic:\tools\miniconda3\include -Ic:\tools\miniconda3\include "-IC:\Program Files (x86)\Mi
crosoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.28.29910\include" /Tcyappi/_yappi.c /Fobuild\temp.win32-3.8\Relea
se\yappi/_yappi.obj
_yappi.c
c:\tools\miniconda3\include\pyconfig.h(59): fatal error C1083: Cannot open include file: 'io.h': No such file or dir
ectory
error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools\\VC\\Tools\\MSVC\\14.28.29910\\bi
n\\HostX86\\x86\\cl.exe' failed with exit status 2
----------------------------------------
ERROR: Command errored out with exit status 1: 'c:\tools\miniconda3\python.exe' -u -c 'import io, os, sys, setuptools, t
okenize; sys.argv[0] = '"'"'C:\\Users\\jaan\\AppData\\Local\\Temp\\pip-install-930ozymw\\yappi_e32707e233e74c018e9d5f8a6
d367185\\setup.py'"'"'; __file__='"'"'C:\\Users\\jaan\\AppData\\Local\\Temp\\pip-install-930ozymw\\yappi_e32707e233e74c0
18e9d5f8a6d367185\\setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else
io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.clo
se();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\jaan\AppData\Local\Temp\pip-record-wf2xdw
wh\install-record.txt' --single-version-externally-managed --compile --install-headers 'c:\tools\miniconda3\Include\yapp
i' Check the logs for full command output.
I'm wondering if there is a good way to convert a callable into its full_name
representation, for passing to a filter. This would make it easier to extract targeted results from profiler output.
It seems to me this is, in general, the most reliable way to ensure you are getting the results of precisely the function you want to track, but I don't see an easy way to get the full_name
of a function given just a reference to the function (it looks like this is computed inside the C code?).
Perhaps I should just filter on the combination of name
and modname
instead?
Due to the changes in python/cpython#20803 (made the code a lot simpler), yappi isn't compiling on 3.10;
$ python -m pip install yappi
Collecting yappi
Downloading yappi-1.3.0.tar.gz (58 kB)
|████████████████████████████████| 58 kB 407 kB/s
Building wheels for collected packages: yappi
Building wheel for yappi (setup.py) ... error
ERROR: Command errored out with exit status 1:
command: /home/isidentical/.venv/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-nwzsms_w/yappi/setup.py'"'"'; __file__='"'"'/tmp/pip-install-nwzsms_w/yappi/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-77oifsdj
cwd: /tmp/pip-install-nwzsms_w/yappi/
Complete output (23 lines):
/tmp/timer_create17izrlg5.c: In function ‘main’:
/tmp/timer_create17izrlg5.c:2:5: warning: implicit declaration of function ‘timer_create’ [-Wimplicit-function-declaration]
2 | timer_create();
| ^~~~~~~~~~~~
running bdist_wheel
running build
running build_py
/home/isidentical/.venv/lib/python3.10/site-packages/setuptools/lib2to3_ex.py:13: PendingDeprecationWarning: lib2to3 package is deprecated and may not be able to parse Python 3.10+
from lib2to3.refactor import RefactoringTool, get_fixers_from_package
creating build
creating build/lib.linux-x86_64-3.10-pydebug
copying yappi/yappi.py -> build/lib.linux-x86_64-3.10-pydebug
running build_ext
building '_yappi' extension
creating build/temp.linux-x86_64-3.10-pydebug
creating build/temp.linux-x86_64-3.10-pydebug/yappi
gcc -pthread -Wno-unused-result -Wsign-compare -g -Og -Wall -fPIC -DLIB_RT_AVAILABLE=1 -I/home/isidentical/.venv/include -I/usr/local/include/python3.10d -c yappi/_yappi.c -o build/temp.linux-x86_64-3.10-pydebug/yappi/_yappi.o
yappi/_yappi.c: In function ‘_call_leave’:
yappi/_yappi.c:1008:20: error: ‘PyFrameObject’ {aka ‘struct _frame’} has no member named ‘f_stacktop’; did you mean ‘f_stackdepth’?
1008 | if (frame->f_stacktop) {
| ^~~~~~~~~~
| f_stackdepth
error: command '/usr/lib/ccache/gcc' failed with exit code 1
----------------------------------------
ERROR: Failed building wheel for yappi
> pip install yappi
Collecting yappi
Using cached https://files.pythonhosted.org/packages/37/dc/86bbe1822cdc6dbf46c644061bd24217f6a0f056f00162a3697c9bea7575/yappi-1.2.3.tar.gz
Installing collected packages: yappi
Running setup.py install for yappi ... error
ERROR: Command errored out with exit status 1:
command: 'c:\users\rabbott\appdata\local\programs\python\python38-32\python.exe\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\Users\rabbott\AppData\Local\Temp\pip-install-q4kk7o84\yappi\setup.py'"'"'; file='"'"'C:\Users\rabbott\AppData\Local\Temp\pip-install-q4kk7o84\yappi\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record 'C:\Users\rabbott\AppData\Local\Temp\pip-record-yukv2gvu\install-record.txt' --single-version-externally-managed --compile
cwd: C:\Users\rabbott\AppData\Local\Temp\pip-install-q4kk7o84\yappi
Complete output (9 lines):
running install
running build
running build_py
creating build
creating build\lib.win32-3.8
copying yappi\yappi.py -> build\lib.win32-3.8
running build_ext
building '_yappi' extension
error: Microsoft Visual C++ 14.0 is required. Get it with "Microsoft Visual C++ Build Tools": https://visualstudio.microsoft.com/downloads/
----------------------------------------
ERROR: Command errored out with exit status 1: 'c:\users\rabbott\appdata\local\programs\python\python38-32\python.exe\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\Users\rabbott\AppData\Local\Temp\pip-install-q4kk7o84\yappi\setup.py'"'"'; file='"'"'C:\Users\rabbott\AppData\Local\Temp\pip-install-q4kk7o84\yappi\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record 'C:\Users\rabbott\AppData\Local\Temp\pip-record-yukv2gvu\install-record.txt' --single-version-externally-managed --compile Check the logs for full command output.
Hi there,
It would be lovely to filter stats by package. Because you're using PyObject_RichCompareBool
as part of the filter it's already possible to achieve this with a helper object. Would you be interested in integrating this as a feature with a stable API?
My current implementation is:
import dataclasses
import importlib
import os
@dataclasses.dataclass
class PackageModule:
package: str
def __post_init__(self):
mod = importlib.import_module(self.package)
self.fn = mod.__file__
if self.fn.endswith("__init__.py"):
self.fn = os.path.dirname(self.fn)
def __eq__(self, other):
return other.startswith(self.fn)
yappi.get_func_stats(filter={"modname": PackageModule("apd.aggregation")}).print_all()
There are caveats to this, mainly that it requires the module to be importable, and not undesirable to import (such as potential import-time side-effects), but I think it improves the usability a fair bit.
What do you think? If you're interested I'm happy to put together a PR.
Matt
Could you please tag the source? This allows distributions to get the complete source from GitHub without cloning.
Thanks
Is there a way to show time.sleep() in wall time? In the following example, sleep doesn't show up:
import yappi
import time
def foo():
time.sleep(1)
yappi.set_clock_type("wall")
yappi.start()
foo()
yappi.stop()
yappi.get_func_stats().debug_print()
yappi.get_func_stats().print_all()
Latest version in pypi is 1.0 but latest version in setup.py in master is 0.99.
This is inconsistent and makes difficult to track what is actually in pypi and should be fixed. BTW, where is the tarball in pypi coming from?
I tried using yappi
on some code that is executed through Bazel and it was difficult to read the stats because the paths all begin with an identical, deep prefix. Something like this:
/home/user/.cache/bazel/_bazel_user/a1901621c7221d386cf4d8fa1b80a433/sandbox/sandboxfs/5/code/bazel-out/k8-opt/bin/path/in/project/target.runfiles/<the bit I actually care about starts here>
A --prefix
or --strip-prefix
option that would allow me to drop the predictable and repeated parts of file paths would be a help to be able to focus on the more interesting part at the end.
Python 3.10 will be included in Fedora 35. To make that update smoother, we're building Fedora packages with early pre-releases of Python 3.10 and yappi 1.3.0 is failing to build.
https://bugzilla.redhat.com/show_bug.cgi?id=1899557
I see some patches in master branch to support python 3.10 but it's not included in any release. May you create a new release which includes the required patches?
Coroutines being run via asyncio.gather
do not show-up in the callgraph for the calling function.
asyncio.gather
returns a Future
gathering the results from the provided coroutines. Timings are correct (up to the caveats in #21) but the callgraph only shows the creation of the gathering future. The caller for the coroutines run via gather
is the event loop.
Is there any way to provide hints to yappi to change the caller for the coroutines?
Example:
from asyncio import gather, run, sleep
import yappi
async def aio_worker(id):
await sleep(1.)
return id
async def doit():
task1 = aio_worker(1)
task2 = aio_worker(2)
return await gather(task1, task2)
if __name__ == '__main__':
yappi.set_clock_type('wall')
with yappi.run(builtins=True):
run(doit())
stats = yappi.get_func_stats()
print("\n\nCallers of 'aio_worker'")
ps = yappi.convert2pstats(stats)
ps.print_callees('doit')
ps.print_callees('gather') # <- this schedules a future collecting results, only see its __init__ in callees
ps.print_callers('aio_worker') # <- only caller is the event loop: "__builtin__:0(<method 'run' of 'Context' objects>)"
For me it would be ok if gather
would not show-up in the callgraph at all and aio_worker
looked like a direct callee of doit
.
Hello, thanks for taking care of this beauty all these years!
Can we have a way to increase the depth of data yappi captures? I want to be able to see more functions in my call graph than the default.
I've noticed that the call graph stops at a function, that is implemented in python, so there are definitely missing callees after that which I would like to inspect.
I'm using kcachegrind, maybe its a limitation of that tool - I apologize if that is the case!
Hello with the release of 3.10 in just a few days would be important to cut (at least a sdist level) release of the main branch, as the current version fails to build/install without https://github.com/sumerc/yappi/pull/66/files
For the sake of profiling ASGI apps, it would be really helpful if there was a way to make use of python 3.7's contextvars while filtering via yappi.get_func_stats
. If yappi supported this, you could get per-request-response-cycle profiling through an ASGI middleware.
I'm not sure if the necessary functionality already exists or not, but wanted to open an issue in case anyone else has thought about this, or in case someone is aware of an existing way to accomplish this.
See fastapi/fastapi#701 for more context on how we'd like to use this.
Yappi is currently poorly suited to long-running applications, since the only way to free a set of profiling stats once they're no longer needed is to clear the all of the profiler's memory, effectively resetting it. In a multi-threaded or async system, this is almost certain to interfere with the other threads, and even if losing profiler data isn't critical, it could lead to some frustration when attempting to isolate some of the longer code paths applications that are meant to run for extended periods of time (such as any kind of server).
There should be an alternative to clear_stats()
that allows the user to pass the same kind of filter passed to get_func_stats()
to cleanup select parts of the profiler stats. Alternatively, this could be implementad as a method to YFuncStat
, which would free all memory related to the function calls contained within.
I'm getting this error when attempting to install 0.99 on Python 3.7 + Windows 7
In SORT_TYPES_FUNCSTATS
, the sort types tavg
and avgtime
point to the incorrect YFuncStat key (the ctx_id
key). Discovered when trying to use stats.sort('tavg', 'desc')
.
self._filter = filter
_yappi.enum_func_stats(self._enumerator)
self._filter = None
enumeration is done based on filter and then when we try to make the YChildFuncs we cannot get them because filter excluded them.
Since the docs aren't in git, I can't make a PR to update them.
$ cat profile_asyncio.py
import time
import asyncio
def burn_cpu(secs):
t0 = time.process_time()
elapsed = 0
while (elapsed <= secs):
for _ in range(1000):
pass
elapsed = time.process_time() - t0
async def burn_async_io(secs):
await asyncio.sleep(secs)
def burn_io(secs):
time.sleep(secs)
async def foo():
burn_cpu(1.0)
await burn_async_io(1.0)
burn_io(1.0)
await burn_async_io(1.0)
asyncio.run(foo())
# Confusing numbers
$ python -m yappi profile_asyncio.py | grep profile_asyncio
profile_asyncio.py:1 <module> 1 0.000047 1.083084 1.083084
profile_asyncio.py:21 foo 1 0.000097 1.000622 1.000622
profile_asyncio.py:5 burn_cpu 1 0.936388 1.000017 1.000017
profile_asyncio.py:13 burn_async_io 2 0.000053 0.000433 0.000216
profile_asyncio.py:17 burn_io 1 0.000020 0.000075 0.000075
# Consistent numbers with the documentation
$ python -m yappi -c wall profile_asyncio.py | grep profile_asyncio
profile_asyncio.py:1 <module> 1 0.000044 4.057246 4.057246
profile_asyncio.py:21 foo 1 0.000095 4.006760 4.006760
profile_asyncio.py:13 burn_async_io 2 0.000038 2.003446 1.001723
profile_asyncio.py:5 burn_cpu 1 0.955977 1.002077 1.002077
profile_asyncio.py:17 burn_io 1 0.000017 1.001142 1.001142
edit:
Environment: Ubuntu 18.04, python 3.7.3, yappi 1.2.3
Good day,
I'm author of pptop, which has a plugin for yappi profiler. After updating to 1.2.1, Yappi prints a lot of "Internal Error 15" messages when working. I believe the issue is because pptop plugin starts yappi from the own thread, however everything seems to work fine except this message.
p.s. it would be also nice to have an ability obtaining data fields from get_func_stats() by keywords, as I see some of their indexes are changed from version to version.
hello, I am trying to install yappi from pypy, but unfortunately got the following error logs,
:~/src/pypy/pypy3.7-v7.3.4-linux64$ ./bin/pypy -mpip install yappi
Collecting yappi
Using cached yappi-1.3.2.tar.gz (58 kB)
Using legacy setup.py install for yappi, since package 'wheel' is not installed.
Installing collected packages: yappi
Running setup.py install for yappi ... error
ERROR: Command errored out with exit status 1:
command: /home/sjt/src/pypy/pypy3.7-v7.3.4-linux64/bin/pypy -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-fh92ib9i/yappi/setup.py'"'"'; __file__='"'"'/tmp/pip-install-fh92ib9i/yappi/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-6hi_7gfn/install-record.txt --single-version-externally-managed --compile --install-headers /home/sjt/src/pypy/pypy3.7-v7.3.4-linux64/include/yappi
cwd: /tmp/pip-install-fh92ib9i/yappi/
Complete output (132 lines):
/tmp/timer_createq262fzd3.c: In function ‘main’:
/tmp/timer_createq262fzd3.c:2:5: warning: implicit declaration of function ‘timer_create’ [-Wimplicit-function-declaration]
2 | timer_create();
| ^~~~~~~~~~~~
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.7
copying yappi/yappi.py -> build/lib.linux-x86_64-3.7
running build_ext
building '_yappi' extension
creating build/temp.linux-x86_64-3.7
creating build/temp.linux-x86_64-3.7/yappi
gcc -pthread -DNDEBUG -O2 -fPIC -DLIB_RT_AVAILABLE=1 -I/home/sjt/src/pypy/pypy3.7-v7.3.4-linux64/include -c yappi/_yappi.c -o build/temp.linux-x86_64-3.7/yappi/_yappi.o
yappi/_yappi.c:193: warning: "PyLong_AsVoidPtr" redefined
193 | #define PyLong_AsVoidPtr (uintptr_t)PyLong_AsVoidPtr
|
In file included from /home/sjt/src/pypy/pypy3.7-v7.3.4-linux64/include/Python.h:141,
from yappi/config.h:4,
from yappi/_yappi.c:10:
/home/sjt/src/pypy/pypy3.7-v7.3.4-linux64/include/pypy_decl.h:432: note: this is the location of the previous definition
432 | #define PyLong_AsVoidPtr PyPyLong_AsVoidPtr
|
yappi/_yappi.c: In function ‘IS_SUSPENDED’:
yappi/_yappi.c:222:18: error: ‘PyFrameObject’ {aka ‘struct _frame’} has no member named ‘f_stacktop’
222 | return (frame->f_stacktop != NULL);
| ^~
yappi/_yappi.c: In function ‘IS_ASYNC’:
yappi/_yappi.c:236:50: error: ‘CO_ASYNC_GENERATOR’ undeclared (first use in this function); did you mean ‘CO_GENERATOR’?
236 | result = result || frame->f_code->co_flags & CO_ASYNC_GENERATOR;
| ^~~~~~~~~~~~~~~~~~
| CO_GENERATOR
yappi/_yappi.c:236:50: note: each undeclared identifier is reported only once for each function it appears in
yappi/_yappi.c: In function ‘_current_tag’:
yappi/_yappi.c:193:37: warning: implicit declaration of function ‘PyLong_AsVoidPtr’; did you mean ‘PyPyLong_AsVoidPtr’? [-Wimplicit-function-declaration]
193 | #define PyLong_AsVoidPtr (uintptr_t)PyLong_AsVoidPtr
| ^~~~~~~~~~~~~~~~
yappi/_yappi.c:389:14: note: in expansion of macro ‘PyLong_AsVoidPtr’
389 | result = PyLong_AsVoidPtr(r);
| ^~~~~~~~~~~~~~~~
yappi/_yappi.c: In function ‘_code2pit’:
yappi/_yappi.c:657:23: error: ‘PyCodeObject’ {aka ‘struct <anonymous>’} has no member named ‘co_firstlineno’
657 | pit->lineno = cobj->co_firstlineno;
| ^~
yappi/_yappi.c:661:5: warning: implicit declaration of function ‘PyFrame_FastToLocals’ [-Wimplicit-function-declaration]
661 | PyFrame_FastToLocals(fobj);
| ^~~~~~~~~~~~~~~~~~~~
yappi/_yappi.c:663:72: error: ‘PyCodeObject’ {aka ‘struct <anonymous>’} has no member named ‘co_varnames’; did you mean ‘co_name’?
663 | const char *firstarg = PyStr_AS_CSTRING(PyTuple_GET_ITEM(cobj->co_varnames, 0));
| ^~~~~~~~~~~
yappi/_yappi.c:182:46: note: in definition of macro ‘PyStr_AS_CSTRING’
182 | #define PyStr_AS_CSTRING(s) PyUnicode_AsUTF8(s)
| ^
yappi/_yappi.c:663:49: note: in expansion of macro ‘PyTuple_GET_ITEM’
663 | const char *firstarg = PyStr_AS_CSTRING(PyTuple_GET_ITEM(cobj->co_varnames, 0));
| ^~~~~~~~~~~~~~~~
yappi/_yappi.c:688:5: warning: implicit declaration of function ‘PyFrame_LocalsToFast’ [-Wimplicit-function-declaration]
688 | PyFrame_LocalsToFast(fobj, 0);
| ^~~~~~~~~~~~~~~~~~~~
yappi/_yappi.c: In function ‘_yapp_callback’:
yappi/_yappi.c:1181:10: error: ‘PyTrace_CALL’ undeclared (first use in this function)
1181 | case PyTrace_CALL:
| ^~~~~~~~~~~~
yappi/_yappi.c:1184:10: error: ‘PyTrace_RETURN’ undeclared (first use in this function)
1184 | case PyTrace_RETURN: // either normally or with an exception
| ^~~~~~~~~~~~~~
yappi/_yappi.c:1192:10: error: ‘PyTrace_C_CALL’ undeclared (first use in this function)
1192 | case PyTrace_C_CALL:
| ^~~~~~~~~~~~~~
yappi/_yappi.c:1197:10: error: ‘PyTrace_C_RETURN’ undeclared (first use in this function)
1197 | case PyTrace_C_RETURN:
| ^~~~~~~~~~~~~~~~
yappi/_yappi.c:1198:10: error: ‘PyTrace_C_EXCEPTION’ undeclared (first use in this function)
1198 | case PyTrace_C_EXCEPTION:
| ^~~~~~~~~~~~~~~~~~~
yappi/_yappi.c: In function ‘_bootstrap_thread’:
yappi/_yappi.c:1261:7: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘use_tracing’
1261 | ts->use_tracing = 1;
| ^~
yappi/_yappi.c:1262:7: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘c_profilefunc’
1262 | ts->c_profilefunc = _yapp_callback;
| ^~
yappi/_yappi.c: In function ‘_profile_thread’:
yappi/_yappi.c:1292:7: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘use_tracing’
1292 | ts->use_tracing = 1;
| ^~
yappi/_yappi.c:1293:7: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘c_profilefunc’
1293 | ts->c_profilefunc = _yapp_callback;
| ^~
yappi/_yappi.c:1295:18: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘thread_id’
1295 | ctx->tid = ts->thread_id;
| ^~
yappi/_yappi.c: In function ‘_unprofile_thread’:
yappi/_yappi.c:1309:7: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘use_tracing’
1309 | ts->use_tracing = 0;
| ^~
yappi/_yappi.c:1310:7: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘c_profilefunc’
1310 | ts->c_profilefunc = NULL;
| ^~
yappi/_yappi.c: In function ‘_ensure_thread_profiled’:
yappi/_yappi.c:1318:11: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘c_profilefunc’
1318 | if (ts->c_profilefunc != _yapp_callback)
| ^~
yappi/_yappi.c: In function ‘_enum_threads’:
yappi/_yappi.c:1330:17: warning: implicit declaration of function ‘PyInterpreterState_ThreadHead’; did you mean ‘PyInterpreterState_Head’? [-Wimplicit-function-declaration]
1330 | for (ts=PyInterpreterState_ThreadHead(is) ; ts != NULL; ts = ts->next) {
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| PyInterpreterState_Head
yappi/_yappi.c:1330:16: warning: assignment to ‘PyThreadState *’ {aka ‘struct _ts *’} from ‘int’ makes pointer from integer without a cast [-Wint-conversion]
1330 | for (ts=PyInterpreterState_ThreadHead(is) ; ts != NULL; ts = ts->next) {
| ^
yappi/_yappi.c:1330:72: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘next’
1330 | for (ts=PyInterpreterState_ThreadHead(is) ; ts != NULL; ts = ts->next) {
| ^~
yappi/_yappi.c: In function ‘profile_event’:
yappi/_yappi.c:1396:37: error: ‘PyTrace_CALL’ undeclared (first use in this function)
1396 | _yapp_callback(self, frame, PyTrace_CALL, arg);
| ^~~~~~~~~~~~
yappi/_yappi.c:1398:37: error: ‘PyTrace_RETURN’ undeclared (first use in this function)
1398 | _yapp_callback(self, frame, PyTrace_RETURN, arg);
| ^~~~~~~~~~~~~~
yappi/_yappi.c:1400:37: error: ‘PyTrace_C_CALL’ undeclared (first use in this function)
1400 | _yapp_callback(self, frame, PyTrace_C_CALL, arg);
| ^~~~~~~~~~~~~~
yappi/_yappi.c:1402:37: error: ‘PyTrace_C_RETURN’ undeclared (first use in this function)
1402 | _yapp_callback(self, frame, PyTrace_C_RETURN, arg);
| ^~~~~~~~~~~~~~~~
yappi/_yappi.c:1404:37: error: ‘PyTrace_C_EXCEPTION’ undeclared (first use in this function)
1404 | _yapp_callback(self, frame, PyTrace_C_EXCEPTION, arg);
| ^~~~~~~~~~~~~~~~~~~
error: command 'gcc' failed with exit status 1
----------------------------------------
ERROR: Command errored out with exit status 1: /home/sjt/src/pypy/pypy3.7-v7.3.4-linux64/bin/pypy -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-fh92ib9i/yappi/setup.py'"'"'; __file__='"'"'/tmp/pip-install-fh92ib9i/yappi/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-6hi_7gfn/install-record.txt --single-version-externally-managed --compile --install-headers /home/sjt/src/pypy/pypy3.7-v7.3.4-linux64/include/yappi Check the logs for full command output.
any idea how to fix it? thank you
Is it possible to get thread stats in pstats format?
The documentation says that to filter function call stats by module name to use {"module": "..."}
Lines 90 to 94 in f638a12
However, the C code has the following:
Lines 1594 to 1618 in f638a12
This can be demonstrated by working with the actual code:
(Pdb) all_results = yappi.get_func_stats()
(Pdb) len(all_results)
2558
(Pdb) len(yappi.get_func_stats(filter={"module": all_results[0].module}))
2558
(Pdb) len(yappi.get_func_stats(filter={"modname": all_results[0].module}))
55
In addition, there are no automated tests for filtering by module name.
As part of an application profiler plugin project for the ASGI protocol, where users can just pass a list of functions to profile, I was using a simple yappi.get_func_stats({"name": somefunction.__qualname__, "tag": ctx_tag})
filter, thinking it would be enough to unambigously select those functions. I later found out that __qualname__
is only unambiguous within the scope of a module, so I should in all likelyhood specify somefunction.__module__
as part of the filter as well, but that got me to look at the code yappi uses to identify the functions being called:
Lines 546 to 567 in 6c97f55
It uses ml_name
as the function name, which apparently corresponds to __name__
, not __qualname__
like I initially figured yappi would use, and has a few edge-cases for both function and module name that could make it difficult/unreliable to build the correct filter automatically.
Shouldn't yappi have some kind of filter_from_function_descriptor
function to generate a detailled filter from a function descriptor automatically, instead of leaving users guessing as to whether they're accidentally profiling other functions with the same name as the one they want?
======================================================================
FAIL: test_async_wall (test_asyncio_context_vars.AsyncUsage)
----------------------------------------------------------------------
Traceback (most recent call last):
File "tests\test_asyncio_context_vars.py", line 50, in test_async_wall
self.assertGreaterEqual(ttot, self.duration)
AssertionError: 0.059142719999999996 not greater than or equal to 0.1
======================================================================
FAIL: test_async_wall (test_asyncio_context_vars.AsyncUsageWithContextvars)
----------------------------------------------------------------------
Traceback (most recent call last):
File "tests\test_asyncio_context_vars.py", line 50, in test_async_wall
self.assertGreaterEqual(ttot, self.duration)
AssertionError: 0.090103 not greater than or equal to 0.1
----------------------------------------------------------------------
Ran 77 tests in 17.157s
FAILED (failures=2, skipped=2)
Command exited with code 1
Thanks for this great tool.
Are there plans to support additional load formats, specifically pstat?
The use case is to be able to load existing pstat profiles, merge with a new stat and visualize the merged profile using a tool like SnakeViz.
If this can't be supported, is there a way to visualize ystat?
Thanks.
If we could have a .whl of yappi available in pypi.org, it would have multiple benefits:
The latest release of yappi has begun to phase out get_func_stats(filter={})
in favor of tag
, ctx_id
and filter_callback
. However, I ended up noticing while using tag
to profile coroutines, it seems that tag
and ctx_id
are currently being ignored.
Lines 1094 to 1104 in b0e424f
(Notice how both variables aren't passed or stored anywhere)
It can currently be worked around by writing this:
yappi.get_func_stats(
filter=dict(tag=ctx_tag),
#...
)
rather than this
yappi.get_func_stats(tag=ctx_tag)
The v1.0 behaviour for coroutines was actually useful to detect incorrect use of non-asyncio I/O calls which block the loop but not CPU. It might be convenient to add an option to switch to the old behaviour (if it is not a big deal). Of course, using the old version of yappi is still an option for users but not as elegant as would be a switch.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.