Giter Site home page Giter Site logo

what-studio / profiling Goto Github PK

View Code? Open in Web Editor NEW
3.0K 74.0 115.0 633 KB

Was an interactive continuous Python profiler.

License: BSD 3-Clause "New" or "Revised" License

Python 98.75% C 1.25%
python live-profiling profiling debug statistical-profiling

profiling's Introduction

☠ This project is not maintained anymore. We highly recommend switching to py-spy which provides better performance and usability.


Profiling

The profiling package is an interactive continuous Python profiler. It is inspired from Unity 3D profiler. This package provides these features:

  • Profiling statistics keep the frame stack.
  • An interactive TUI profiling statistics viewer.
  • Provides both of statistical and deterministic profiling.
  • Utilities for remote profiling.
  • Thread or greenlet aware CPU timer.
  • Supports Python 2.7, 3.3, 3.4 and 3.5.
  • Currently supports only Linux.

Build Status Coverage Status

Installation

Install the latest release via PyPI:

$ pip install profiling

Profiling

To profile a single program, simply run the profiling command:

$ profiling your-program.py

Then an interactive viewer will be executed:

If your program uses greenlets, choose greenlet timer:

$ profiling --timer=greenlet your-program.py

With --dump option, it saves the profiling result to a file. You can browse the saved result by using the view subcommand:

$ profiling --dump=your-program.prf your-program.py
$ profiling view your-program.prf

If your script reads sys.argv, append your arguments after --. It isolates your arguments from the profiling command:

$ profiling your-program.py -- --your-flag --your-param=42

Live Profiling

If your program has a long life time like a web server, a profiling result at the end of program is not helpful enough. Probably you need a continuous profiler. It can be achived by the live-profile subcommand:

$ profiling live-profile webserver.py

See a demo:

asciicast

There's a live-profiling server also. The server doesn't profile the program at ordinary times. But when a client connects to the server, it starts to profile and reports the results to the all connected clients.

Start a profling server by the remote-profile subcommand:

$ profiling remote-profile webserver.py --bind 127.0.0.1:8912

And also run a client for the server by the view subcommand:

$ profiling view 127.0.0.1:8912

Statistical Profiling

TracingProfiler, the default profiler, implements a deterministic profiler for deep call graph. Of course, it has heavy overhead. The overhead can pollute your profiling result or can make your application to be slow.

In contrast, SamplingProfiler implements a statistical profiler. Like other statistical profilers, it also has only very cheap overhead. When you profile you can choose it by just --sampling (shortly -S) option:

$ profiling live-profile -S webserver.py
                         ^^

Timeit then Profiling

Do you use timeit to check the performance of your code?

$ python -m timeit -s 'from trueskill import *' 'rate_1vs1(Rating(), Rating())'
1000 loops, best of 3: 722 usec per loop

If you want to profile the checked code, simply use the timeit subcommand:

$ profiling timeit -s 'from trueskill import *' 'rate_1vs1(Rating(), Rating())'
  ^^^^^^^^^

Profiling from Code

You can also profile your program by profiling.tracing.TracingProfiler or profiling.sampling.SamplingProfiler directly:

from profiling.tracing import TracingProfiler

# profile your program.
profiler = TracingProfiler()
profiler.start()
...  # run your program.
profiler.stop()

# or using context manager.
with profiler:
    ...  # run your program.

# view and interact with the result.
profiler.run_viewer()
# or save profile data to file
profiler.dump('path/to/file')

Viewer Key Bindings

  • q - Quit.
  • space - Pause/Resume.
  • \ - Toggle layout between NESTED and FLAT.
  • and - Navigate frames.
  • - Expand the frame.
  • - Fold the frame.
  • > - Go to the hotspot.
  • esc - Defocus.
  • [ and ] - Change sorting column.

Columns

Common

  • FUNCTION
    1. The function name with the code location. (e.g. my_func (my_code.py:42), my_func (my_module:42))
    2. Only the location without line number. (e.g. my_code.py, my_module)

Tracing Profiler

  • CALLS - Total call count of the function.
  • OWN (Exclusive Time) - Total spent time in the function excluding sub calls.
  • /CALL after OWN - Exclusive time per call.
  • % after OWN - Exclusive time per total spent time.
  • DEEP (Inclusive Time) - Total spent time in the function.
  • /CALL after DEEP - Inclusive time per call.
  • % after DEEP - Inclusive time per total spent time.

Sampling Profiler

  • OWN (Exclusive Samples) - Number of samples which are collected during the direct execution of the function.
  • % after OWN - Exclusive samples per number of the total samples.
  • DEEP (Inclusive Samples) - Number of samples which are collected during the excution of the function.
  • % after DEEP - Inclusive samples per number of the total samples.

Testing

There are some additional requirements to run the test code, which can be installed by running the following command.

$ pip install $(python test/fit_requirements.py test/requirements.txt)

Then you should be able to run pytest.

$ pytest -v

Thanks to

Licensing

Written by Heungsub Lee at What! Studio in Nexon, and distributed under the BSD 3-Clause license.

profiling's People

Contributors

akatrevorjay avatar dahlia avatar falsetru avatar ivuk avatar justintarthur avatar ku1ik avatar kxepal avatar lqez avatar mcdoker18 avatar mingrammer avatar mj111 avatar parkayun avatar ravipudi avatar sublee avatar suminb avatar timgates42 avatar tomviner avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

profiling's Issues

Flat mode

By default, a profiling result is a tree of frame stacks. If we can see the result as a flat list, it is also helpful to find a hotspot. We already have profiling.stats.FlatStatistics but we can't see this look on the viewer. This feature should be mapped with a some human-friendly key command.

Recursion RuntimeError on importing Requests lib

I'm running into a weird issue while trying to profile a simple project. I isolated the problem to the Requests library.

Just by importing requests (I tested 2.6.2 and 2.7.0), profiling will throw a recursion runtime error.

I'm using OSX 10.10.5 and python 3.4. Can anyone replicate this behaviour?

Here is the isolated script:

import requests

print('Hello World')

This is the console output after it does nothing for a while:
(NOTE: It does execute the Hello World print)

(testenv)Mac:active-monitoring-plugin carlo$ python -m profiling profile ~/Desktop/weird.py
Hello World
Traceback (most recent call last):
  File "/usr/local/Cellar/python3/3.4.3_2/Frameworks/Python.framework/Versions/3.4/lib/python3.4/runpy.py", line 170, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/Cellar/python3/3.4.3_2/Frameworks/Python.framework/Versions/3.4/lib/python3.4/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/carlo/.virtualenvs/testenv/lib/python3.4/site-packages/profiling/__main__.py", line 674, in <module>
    cli(prog_name='python -m profiling')
  File "/Users/carlo/.virtualenvs/testenv/lib/python3.4/site-packages/click/core.py", line 700, in __call__
    return self.main(*args, **kwargs)
  File "/Users/carlo/.virtualenvs/testenv/lib/python3.4/site-packages/click/core.py", line 680, in main
    rv = self.invoke(ctx)
  File "/Users/carlo/.virtualenvs/testenv/lib/python3.4/site-packages/click/core.py", line 1027, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/carlo/.virtualenvs/testenv/lib/python3.4/site-packages/click/core.py", line 873, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/carlo/.virtualenvs/testenv/lib/python3.4/site-packages/click/core.py", line 508, in invoke
    return callback(*args, **kwargs)
  File "/Users/carlo/.virtualenvs/testenv/lib/python3.4/site-packages/profiling/__main__.py", line 399, in wrapped
    return f(**kwargs)
  File "/Users/carlo/.virtualenvs/testenv/lib/python3.4/site-packages/profiling/__main__.py", line 376, in wrapped
    return f(profiler_factory=profiler_factory, **kwargs)
  File "/Users/carlo/.virtualenvs/testenv/lib/python3.4/site-packages/profiling/__main__.py", line 488, in profile
    mono=mono)
  File "/Users/carlo/.virtualenvs/testenv/lib/python3.4/site-packages/profiling/__main__.py", line 447, in __profile__
    stats, cpu_time, wall_time = profiler.result()
  File "/Users/carlo/.virtualenvs/testenv/lib/python3.4/site-packages/profiling/tracing/__init__.py", line 123, in result
    frozen_stats, cpu_time, wall_time = base.result()
  File "/Users/carlo/.virtualenvs/testenv/lib/python3.4/site-packages/profiling/profiler.py", line 63, in result
    frozen_stats = FrozenStatistics(self.stats)
  File "/Users/carlo/.virtualenvs/testenv/lib/python3.4/site-packages/profiling/stats.py", line 50, in __call__
    obj = super(StatisticsMeta, cls).__call__(*args, **kwargs)
  File "/Users/carlo/.virtualenvs/testenv/lib/python3.4/site-packages/profiling/stats.py", line 257, in __init__
    self._children = self._freeze_children(stats)
  File "/Users/carlo/.virtualenvs/testenv/lib/python3.4/site-packages/profiling/stats.py", line 262, in _freeze_children
    return [cls(s) for s in children]
...
...
...
    return [cls(s) for s in children]
  File "/Users/carlo/.virtualenvs/testenv/lib/python3.4/site-packages/profiling/stats.py", line 262, in <listcomp>
    return [cls(s) for s in children]
  File "/Users/carlo/.virtualenvs/testenv/lib/python3.4/site-packages/profiling/stats.py", line 50, in __call__
    obj = super(StatisticsMeta, cls).__call__(*args, **kwargs)
  File "/Users/carlo/.virtualenvs/testenv/lib/python3.4/site-packages/profiling/stats.py", line 257, in __init__
    self._children = self._freeze_children(stats)
  File "/Users/carlo/.virtualenvs/testenv/lib/python3.4/site-packages/profiling/stats.py", line 262, in _freeze_children
    return [cls(s) for s in children]
  File "/Users/carlo/.virtualenvs/testenv/lib/python3.4/site-packages/profiling/stats.py", line 262, in <listcomp>
    return [cls(s) for s in children]
  File "/Users/carlo/.virtualenvs/testenv/lib/python3.4/site-packages/profiling/stats.py", line 50, in __call__
    obj = super(StatisticsMeta, cls).__call__(*args, **kwargs)
  File "/Users/carlo/.virtualenvs/testenv/lib/python3.4/site-packages/profiling/stats.py", line 252, in __init__
    value = getattr(stats, attr)
  File "/Users/carlo/.virtualenvs/testenv/lib/python3.4/site-packages/profiling/stats.py", line 186, in module
    module = inspect.getmodule(self.code)
  File "/usr/local/Cellar/python3/3.4.3_2/Frameworks/Python.framework/Versions/3.4/lib/python3.4/inspect.py", line 612, in getmodule
    file = getabsfile(object, _filename)
  File "/usr/local/Cellar/python3/3.4.3_2/Frameworks/Python.framework/Versions/3.4/lib/python3.4/inspect.py", line 595, in getabsfile
    _filename = getsourcefile(object) or getfile(object)
  File "/usr/local/Cellar/python3/3.4.3_2/Frameworks/Python.framework/Versions/3.4/lib/python3.4/inspect.py", line 583, in getsourcefile
    if getattr(getmodule(object, filename), '__loader__', None) is not None:
  File "/usr/local/Cellar/python3/3.4.3_2/Frameworks/Python.framework/Versions/3.4/lib/python3.4/inspect.py", line 612, in getmodule
    file = getabsfile(object, _filename)
  File "/usr/local/Cellar/python3/3.4.3_2/Frameworks/Python.framework/Versions/3.4/lib/python3.4/inspect.py", line 596, in getabsfile
    return os.path.normcase(os.path.abspath(_filename))
  File "/Users/carlo/.virtualenvs/testenv/bin/../lib/python3.4/posixpath.py", line 357, in abspath
    if not isabs(path):
  File "/Users/carlo/.virtualenvs/testenv/bin/../lib/python3.4/posixpath.py", line 62, in isabs
    sep = _get_sep(s)
RuntimeError: maximum recursion depth exceeded

timeit-profile

We use timeit to compare performance of similar code:

$ python -m timeit -s 'import foo' 'foo.egg()'
10000 loops, best of 3: 25.2 usec per loop
$ python -m timeit -s 'import foo' 'foo.spam()'
10000 loops, best of 3: 40.3 usec per loop

Now we know foo.spam() is slower. But why?

profiling should provide timeit as a command. It will be helpful to inquire into the cause of the performance loss.

$ python -m profiling timeit -s 'import foo' 'foo.spam()'
            ^^^^^^^^^
           Just add it.

Run twisted concurently

Is there anyway I can run profile with twisted?
I tried running like python -m profiling profile =twistd -y server.py but it doesn't seem to work. Any idea?

Thanks.

setup.py test fails with newest pip

e0961da

OS X El Capitan

❯ python --version
Python 2.7.10

~/work/python/profiling .venv master*
❯ pip --version
pip 8.1.2 from /Users/me/work/python/profiling/.venv/lib/python2.7/site-packages (python 2.7)
~/work/python/profiling master*
❯ virtualenv .venv
New python executable in /Users/me/work/python/profiling/.venv/bin/python
Installing setuptools, pip, wheel.... .venv/bin/actdone.

~/work/python/profiling master*
❯ . .venv/bin/activate

~/work/python/profiling .venv master*
❯ pip install -e .
❯ python setup.py test
running test
Searching for yappi>=0.92
Best match: yappi 0.94
Processing yappi-0.94-py2.7-macosx-10.11-intel.egg

Using /Users/me/work/python/profiling/.eggs/yappi-0.94-py2.7-macosx-10.11-intel.egg
Searching for greenlet>=0.4.4
Reading https://pypi.python.org/simple/greenlet/
Best match: greenlet 0.4.9
Downloading https://pypi.python.org/packages/ba/19/7ae57aa8b66f918859206532b1afd7f876582e3c87434ff33261da1cf50c/greenlet-0.4.9.tar.gz#md5=00bb1822d8511cc85f052e89d1fd919b
Processing greenlet-0.4.9.tar.gz
Writing /var/folders/kx/gmc5nl8x2hg370c3nktpc7vm0000gn/T/easy_install-lVzsh8/greenlet-0.4.9/setup.cfg
Running greenlet-0.4.9/setup.py -q bdist_egg --dist-dir /var/folders/kx/gmc5nl8x2hg370c3nktpc7vm0000gn/T/easy_install-lVzsh8/greenlet-0.4.9/egg-dist-tmp-gCLetw
creating /Users/me/work/python/profiling/.eggs/greenlet-0.4.9-py2.7-macosx-10.11-intel.egg
Extracting greenlet-0.4.9-py2.7-macosx-10.11-intel.egg to /Users/me/work/python/profiling/.eggs

Installed /Users/me/work/python/profiling/.eggs/greenlet-0.4.9-py2.7-macosx-10.11-intel.egg
Searching for gevent>=1.1rc2
Best match: gevent 1.1.1
Processing gevent-1.1.1-py2.7-macosx-10.11-intel.egg

Using /Users/me/work/python/profiling/.eggs/gevent-1.1.1-py2.7-macosx-10.11-intel.egg
Traceback (most recent call last):
  File "setup.py", line 135, in <module>
    test_suite='...',
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/Users/me/work/python/profiling/.venv/lib/python2.7/site-packages/setuptools/command/test.py", line 152, in run
    self.distribution.fetch_build_eggs(self.distribution.tests_require)
  File "/Users/me/work/python/profiling/.venv/lib/python2.7/site-packages/setuptools/dist.py", line 313, in fetch_build_eggs
    replace_conflicting=True,
  File "/Users/me/work/python/profiling/.venv/lib/python2.7/site-packages/pkg_resources/__init__.py", line 834, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.VersionConflict: (gevent 1.1.1 (/Users/me/work/python/profiling/.eggs/gevent-1.1.1-py2.7-macosx-10.11-intel.egg), Requirement.parse('gevent==1.1rc1'))

Invalid Python 2 syntax

In remote/asyncio.py on line 69 the new Python 3 only syntax yield from was used, making it not anymore compatible with Python 2 as stated in the README. It raises a SyntaxError.

Profil flask app

Hello, is there anyway that I can profile a flask app with profiling?

setup.py failure on Windows VC 9.0

On profiling/speedup.c:15-21

    const PySetObject* ignored_codes;
    if (!PyArg_ParseTuple(args, "OOOOO", &frame, &base_frame, &base_code,
                                         &ignored_frames, &ignored_codes))
    {
        return NULL;
    }
    PyObject* frame_stack = PyList_New(0);

However Visual C++ for Python 2.7 (VC 9.0) raises C2275 error:

profiling/speedup.c(21) : error C2275: 'PyObject' : illegal use of this type as an expression

Workaround (for VC 9.0):

    const PySetObject* ignored_codes;
    PyObject* frame_stack;
    if (!PyArg_ParseTuple(args, "OOOOO", &frame, &base_frame, &base_code,
                                         &ignored_frames, &ignored_codes))
    {
        return NULL;
    }
    frame_stack = PyList_New(0);

`NoneType` when using live-profile

When I run $ profiling live-profile manage.py runserver on a tweaked manage.py file that only prints the current path, I get

Exception TypeError: "'NoneType' object is not callable" in <function _remove at 0x7f75f23457d0> ignored

The contents of manage.py are:

#!/usr/bin/env python
import os
import sys

if __name__ == "__main__":
    from pprint import pprint as p
    p(sys.path)

    # os.environ.setdefault("DJANGO_SETTINGS_MODULE", "test_app.settings")
    # from django.core.management import execute_from_command_line
    # execute_from_command_line(sys.argv)

For replication (in case you are not familiar with Django): you can start a brand new Django 1.7 project with $ django-admin startproject test_app and then tweak manage.py to only figure what the path is.

Can't profile with eventlet

Can't profile a server with eventlet. I tried remote and live profile, but didn't work.

cmdline: python -m profiling remote-profile --timer=greenlet --bind 127.0.0.1:1234 ....

Traceback is displayed on server when I execute view.
Traceback (most recent call last):
File "/home/...../lib/python2.7/site-packages/eventlet/hubs/hub.py", line 346, in fire_timers
timer()
File "/home/...../lib/python2.7/site-packages/eventlet/hubs/timer.py", line 56, in call
cb(_args, *_kw)
File "/home/...../lib/python2.7/site-packages/eventlet/semaphore.py", line 121, in _do_acquire
waiter.switch()
error: cannot switch to a different thread

cmdline: python -m profiling live-profile --timer=greenlet ...

Only displays not available and server doesn't start..

Live profiling stops updating at some point + Crashes on repeated viewer reconnects

In my use case, there is an algorithm that ends up stalling after running for a long time. I would like to use live-profiling to find the exact cause when the stall happens. It would be inconvenient to dump and post analyze since the stall happens sporadically, and restarting the run is relatively expensive.

The issue I'm facing with this tool is that the profiling updates stop after a while, even though the main program is running as normal. One work around I found was to run in remote-profile mode, where even if the profiling updates stop, I can restart it again by reconnecting the viewer. However, repeating doing this eventually causes the below exception.

Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
File "/usr/lib/python3.5/threading.py", line 862, in run
self._target(*self._args, **self._kwargs)
File "/home/wzli/.local/lib/python3.5/site-packages/profiling/remote/select.py", line 35, in serve_forever
self.dispatch_sockets()
File "/home/wzli/.local/lib/python3.5/site-packages/profiling/remote/select.py", line 105, in dispatch_sockets
self.connected(sock)
File "/home/wzli/.local/lib/python3.5/site-packages/profiling/remote/init.py", line 228, in connected
self._start_profiling()
File "/home/wzli/.local/lib/python3.5/site-packages/profiling/remote/select.py", line 53, in _start_profiling
self.profile_periodically()
File "/home/wzli/.local/lib/python3.5/site-packages/profiling/remote/select.py", line 57, in profile_periodically
self.dispatch_sockets(self.interval)
File "/home/wzli/.local/lib/python3.5/site-packages/profiling/remote/select.py", line 105, in dispatch_sockets
self.connected(sock)
File "/home/wzli/.local/lib/python3.5/site-packages/profiling/remote/init.py", line 228, in connected
self._start_profiling()
File "/home/wzli/.local/lib/python3.5/site-packages/profiling/remote/select.py", line 53, in _start_profiling
self.profile_periodically()
File "/home/wzli/.local/lib/python3.5/site-packages/profiling/remote/select.py", line 57, in profile_periodically
self.dispatch_sockets(self.interval)
File "/home/wzli/.local/lib/python3.5/site-packages/profiling/remote/select.py", line 105, in dispatch_sockets
self.connected(sock)
File "/home/wzli/.local/lib/python3.5/site-packages/profiling/remote/init.py", line 228, in connected
self._start_profiling()
File "/home/wzli/.local/lib/python3.5/site-packages/profiling/remote/select.py", line 53, in _start_profiling
self.profile_periodically()
File "/home/wzli/.local/lib/python3.5/site-packages/profiling/remote/select.py", line 57, in profile_periodically
self.dispatch_sockets(self.interval)
File "/home/wzli/.local/lib/python3.5/site-packages/profiling/remote/select.py", line 105, in dispatch_sockets
self.connected(sock)
File "/home/wzli/.local/lib/python3.5/site-packages/profiling/remote/init.py", line 228, in connected
self._start_profiling()
File "/home/wzli/.local/lib/python3.5/site-packages/profiling/remote/select.py", line 53, in _start_profiling
self.profile_periodically()
File "/home/wzli/.local/lib/python3.5/site-packages/profiling/remote/select.py", line 57, in profile_periodically
self.dispatch_sockets(self.interval)
File "/home/wzli/.local/lib/python3.5/site-packages/profiling/remote/select.py", line 105, in dispatch_sockets
self.connected(sock)
File "/home/wzli/.local/lib/python3.5/site-packages/profiling/remote/init.py", line 228, in connected
self._start_profiling()
File "/home/wzli/.local/lib/python3.5/site-packages/profiling/remote/select.py", line 53, in _start_profiling
self.profile_periodically()
File "/home/wzli/.local/lib/python3.5/site-packages/profiling/remote/select.py", line 57, in profile_periodically
self.dispatch_sockets(self.interval)
File "/home/wzli/.local/lib/python3.5/site-packages/profiling/remote/select.py", line 105, in dispatch_sockets
self.connected(sock)
File "/home/wzli/.local/lib/python3.5/site-packages/profiling/remote/init.py", line 228, in connected
self._start_profiling()
File "/home/wzli/.local/lib/python3.5/site-packages/profiling/remote/select.py", line 53, in _start_profiling
self.profile_periodically()
File "/home/wzli/.local/lib/python3.5/site-packages/profiling/remote/select.py", line 57, in profile_periodically
self.dispatch_sockets(self.interval)
File "/home/wzli/.local/lib/python3.5/site-packages/profiling/remote/select.py", line 105, in dispatch_sockets
self.connected(sock)
File "/home/wzli/.local/lib/python3.5/site-packages/profiling/remote/init.py", line 228, in connected
self._start_profiling()
File "/home/wzli/.local/lib/python3.5/site-packages/profiling/remote/select.py", line 53, in _start_profiling
self.profile_periodically()
File "/home/wzli/.local/lib/python3.5/site-packages/profiling/remote/select.py", line 56, in profile_periodically
for __ in self.profiling():
File "/home/wzli/.local/lib/python3.5/site-packages/profiling/remote/init.py", line 180, in profiling
self.profiler.stop()
File "/home/wzli/.local/lib/python3.5/site-packages/profiling/utils.py", line 63, in stop
raise RuntimeError('Not started')
RuntimeError: Not started

Profiler doesn't start

Traceback (most recent call last):
  File "/apps/python/2.7.6/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/apps/python/2.7.6/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/apps/python/2.7.6/lib/python2.7/site-packages/profiling/__main__.py", line 592, in <module>
    cli(prog_name='python -m profiling')
  File "/apps/python/2.7.6/lib/python2.7/site-packages/click/core.py", line 664, in __call__
    return self.main(*args, **kwargs)
  File "/apps/python/2.7.6/lib/python2.7/site-packages/click/core.py", line 644, in main
    rv = self.invoke(ctx)
  File "/apps/python/2.7.6/lib/python2.7/site-packages/click/core.py", line 991, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/apps/python/2.7.6/lib/python2.7/site-packages/click/core.py", line 837, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/apps/python/2.7.6/lib/python2.7/site-packages/click/core.py", line 464, in invoke
    return callback(*args, **kwargs)
  File "/apps/python/2.7.6/lib/python2.7/site-packages/profiling/__main__.py", line 347, in profile
    dump_filename=dump_filename, mono=mono)
  File "/apps/python/2.7.6/lib/python2.7/site-packages/profiling/__main__.py", line 314, in __profile__
    loop.run()
  File "/apps/python/2.7.6/lib/python2.7/site-packages/urwid/main_loop.py", line 278, in run
    self._run()
  File "/apps/python/2.7.6/lib/python2.7/site-packages/urwid/main_loop.py", line 375, in _run
    self.event_loop.run()
  File "/apps/python/2.7.6/lib/python2.7/site-packages/urwid/main_loop.py", line 678, in run
    self._loop()
  File "/apps/python/2.7.6/lib/python2.7/site-packages/urwid/main_loop.py", line 699, in _loop
    ready, w, err = select.select(fds, [], fds, timeout)
select.error: (9, 'Bad file descriptor')

windows live-profiling exception

getting an exception directly at starting:

Traceback (most recent call last):
File "C:\Program Files\Python27\lib\runpy.py", line 162, in run_module_as_main
"main", fname, loader, pkg_name)
File "C:\Program Files\Python27\lib\runpy.py", line 72, in run_code
exec code in run_globals
File "C:\Program Files\Python27\lib\site-packages\profiling__main
.py", line 33, in
from .remote.background import START_SIGNO, STOP_SIGNO, BackgroundProfiler
File "C:\Program Files\Python27\lib\site-packages\profiling\remote\background.py", line 20, in
START_SIGNO = signal.SIGUSR1
AttributeError: 'module' object has no attribute 'SIGUSR1'

Does profiling work well with asyncio?

I noticed that the README mentions support for greenlets. How about Python 3.4+'s asyncio?

Should we just use --timer=greenlet for all the varieties of asynchronous programming that Python supports (greenlets, threading, multiprocessing, asyncio, etc.)?

Can't see profiling, got a error message

Under
centos 6.3
Python 2.6.6

# pip install git+https://github.com/what-studio/profiling.git
# python -m profiling live-profile api.py 9000

Above cmd give me

/usr/bin/python: profiling is a package and cannot be directly executed

Profile Flask app running uwsgi

I have a docker container that runs uwsgi to lunch my flask app, how do I profile thing? I'd like to see performance of one of my routes that's slow.

I tried:

        environment:
            PYTHONPATH: ".:/code:/code/libs"
            FLASK_APP: "pinqueue"
            FLASK_DEBUG: "true"
            PINQUEUE_ENV: "development"
            USER: ${USER}
            TLS_ENABLED: "true"
        command: uwsgi --gevent 10 --gevent-monkey-patch --http-socket 0.0.0.0:10001 --http-websockets --module pinqueue:app --master --processes 4 --enable-threads --honour-stdin --py-autoreload=3 --buffer-size=65535

Sampling Profiler

A sampling profiler is more efficient for live-profile or remote-profile. Profiling should implement that also.

Getting a usage error when I try and profile my testharness

Getting a usage error when I try and profile my tests:

python -m profiling TestTransfer.py
Usage: main.py [OPTIONS] COMMAND [ARGS]...

Obviously just running without profile works. The test harness does the usual:

if __name__ == '__main__':
    unittest.main()

time.clock_gettime is not work in Mac

time.clock_gettime(clk_id)
Return the time of the specified clock clk_id.

Availability: Unix.

New in version 3.3.

time.clock_gettime is only used in Unix,can you have some method to add support to MacOS?

Add profiling to PyPI

I realize the README says this: "This project is still under development, so you should install it via GitHub instead of PyPI:" but IMO, profiling is definitely in a good enough position where a stable release can be thrown onto PyPI.

Profiling of asyncio application failing

I'm trying to profile a (rather complex) asyncio application (https://github.com/kharidiron/StarryPy3k) and am running into the issue that profiling will not import from other files in the same directory.

$ profiling live-profile --eventloop-aware=asyncio server.py

Traceback (most recent call last):
File "/Users/kharidiron/Workspace/PyCharm/VEnvs/StarryPy3k/bin/profiling", line 9, in
load_entry_point('profiling==0.0.0.dev0', 'console_scripts', 'profiling')()
File "/Users/kharidiron/Workspace/PyCharm/VEnvs/StarryPy3k/lib/python3.5/site-packages/click/core.py", line 716, in call
return self.main(_args, *_kwargs)
File "/Users/kharidiron/Workspace/PyCharm/VEnvs/StarryPy3k/lib/python3.5/site-packages/click/core.py", line 696, in main
rv = self.invoke(ctx)
File "/Users/kharidiron/Workspace/PyCharm/VEnvs/StarryPy3k/lib/python3.5/site-packages/click/core.py", line 1060, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/kharidiron/Workspace/PyCharm/VEnvs/StarryPy3k/lib/python3.5/site-packages/click/core.py", line 889, in invoke
return ctx.invoke(self.callback, _ctx.params)
File "/Users/kharidiron/Workspace/PyCharm/VEnvs/StarryPy3k/lib/python3.5/site-packages/click/core.py", line 534, in invoke
return callback(_args, **kwargs)
File "/Users/kharidiron/Workspace/PyCharm/VEnvs/StarryPy3k/lib/python3.5/site-packages/profiling/main.py", line 525, in wrapped
return f(
kwargs)
File "/Users/kharidiron/Workspace/PyCharm/VEnvs/StarryPy3k/lib/python3.5/site-packages/profiling/main.py", line 502, in wrapped
return f(profiler_factory=profiler_factory, *kwargs)
File "/Users/kharidiron/Workspace/PyCharm/VEnvs/StarryPy3k/lib/python3.5/site-packages/profiling/main.py", line 679, in live_profile
exec
(code, globals
)
File "server.py", line 5, in
from configuration_manager import ConfigurationManager
ImportError: No module named 'configuration_manager'

Am I doing something wrong?

(edit formatting)

error in pip3 install

sudo pip3 install profiling
t always shows same error

[Installing collected packages: profiling
  Running setup.py install for profiling ... error
    Complete output from command /usr/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-emh1dfcl/profiling/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-x7q4tkex/install-record.txt --single-version-externally-managed --compile:]

Upload wheel to pypi

Wheels are very convenient especially when one wants to install with missing compilation requirements.

analysis got some problem!

my code run with only 2 second , and then it used at least 180s by profiling -.- is something wrong with it ??

More test cases

The test coverage is too low. Now it's only 49%. I should make more test cases.

pip install fails on Windows

Attempt to install profiling on Windows 7, Python 2.7 (32bit):

building 'profiling.speedup' extension
creating build\temp.win32-2.7
creating build\temp.win32-2.7\Release
creating build\temp.win32-2.7\Release\profiling
C:\Program Files\TDM-GCC\bin\gcc.exe -mno-cygwin -mdll -O -Wall "-IC:\Program Files (x86)\Python\include" "-IC:\Program Files (x86)\Python\PC" -c profiling/speedup.c -o build\temp.win32-2.7\Release\profiling\speedup.o
gcc: error: unrecognized command line option '-mno-cygwin'

Tried with MinGW-gcc 4.7.2, and TDM-gcc 4.8 and 5.1.

Cannot profile programs with arguments

python -m profiling live-profile -i 5 ./runtests.py '-k test_full_stack_benchmarks.py -v --bench --bench-only'
Usage: main.py live-profile [OPTIONS] SCRIPT [ARGV]...

Error: no such option: -k

Profiling django's dev server

Hello,

I tried to profile django's dev server with profiling live-profile manage.py runserver, but without success. The viewer seems that is showing only calls for bootstrapping django. Nothing happens as requests are made to the server. Am I missing something? Or is what I am trying to do currently not supported?

profiler doesn't cd to current directory

profiling script should cd to current directory because currently all relative path of profiled app are broken

e.g. if I do this:

import sys, os

PROJECT_PATH = os.path.normpath(os.path.join(sys.modules[__name__].__file__,'../'))

I will get wrong path and won't be able to add needed modules

currently __file__ reference points to: python2.7/site-packages/profiling/__main__.py

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.