jiffyclub/snakeviz

An in-browser Python profile viewer

SnakeViz

About

SnakeViz is a viewer for Python profiling data that runs as a web application in your browser. It is inspired by the wxPython profile viewer RunSnakeRun.

View the docs at https://jiffyclub.github.io/snakeviz/.

snakeviz's Issues

IPython magic extension

Make an IPython magic extension that launches WebSnakeViz. Could work much like the built-in %prun (%%prun) magic (and be built on top of that):

%load_ext wsv
%wsv myfunc(args)
%%wsv func1(arg1)
func2(arg2)

Add square map visualisation

I had never seen the Sunburst visualisation before, and while it is very pretty, I don't find it very intuitive to browse. How much work would it be to add e.g. a square map visualisation?

Improve node highlighting

The current highlighting darkens nodes, which is not always effective. A quick short term solution is to use a distinct, bright color for highlighting. Eventually it would be cool to have animated, pulsed node highlighting to really draw the eye.

Revise help text of -b option

As noted in #46 (comment), the help text of the -b option should note that browsers can be specified by the names described on this list: https://docs.python.org/3.4/library/webbrowser.html.

IPython: magic function %snakeviz not found

I recently upgraded my computer to OS X 10.10 Yosemite and re-installed Macports and snakeviz.
Now snakeviz works on the command line, but %snakeviz in IPython doesn't work:

ERROR: Line magic function `%snakeviz` not found.

The snakeviz install instructions don't mention that one has to register %snakeviz with IPython somehow and I think last time this just worked out of the box for me !?

Any idea what is wrong with my setup or what I have to do?

Browser does not open on Mac

Environment: Macbook Air 2012, OS X 10.8.2, Safari 8.0.3., bash-3.2.57, python 2.7.5 (Mac default), snakeviz-0.2.1

Repro

1. Created a run.prof from cProfile against a test script.
2. Open Terminal on Mac.
3. Type "snakeviz run.prof".

Problem:
See this message hanging forever: "snakeviz web server started on 127.0.0.1:8080; enter Ctrl-C to exit", and nothing happens beyond this.

Expected: Browser (Safari by default) should open and show some eye candies.

Using SnakeViz on a Notebook that's hosted elsewhere

Firstly, SnakeViz is awesome. Thanks for putting it together.

We have machines deployed on AWS Ubuntu machines, and use IPython Notebooks on those. When I run SnakeViz in those, I get a response *** Profile stats marshalled to file u'/tmp/tmp8aZzx1'., but no new window / tab opens up.
When I try the exact same code on my local MacOS X machine (the glob example), a new tab pops up with every run, just at it should.

Is this a problem inherent in the notebook? Or is this something that could be solved within SnakeViz?

Thanks,
Max

Table/figure integration

Selecting a row in table should highlight that function in the visualization, possibly in a different color than the mouse-over highlighting. What should be done when the user mouses over a node highlighted by a table selection? The Google table implementation currently in use allows for setting up selection event handlers. The table allows for multiple rows to be selected.

Some documentation on the table spec is here: https://developers.google.com/chart/interactive/docs/gallery/table

Error Loading Profile

I'm getting this generic error. Is my script just too big?

Error Loading Profile

Sorry, we were not able to load this profile! You can try profiling a smaller portion of your code. The statistics table below might help you narrow down which portion of your code to focus on.

Windows abspath is different

>>> os.path.abspath('/c:/myfile')
'C:\\c:\\myfile'

This bug in class VizHandler(handler.Handler) causes snakeviz to just not work on windows. Also, it seems the tab in the browser opened is http://127.0.0.1:8080/viz/fileU:/test.pyprofile i.e. notice there's no slash after the file but I haven't debugged that.

Vanishing/Flashing tooltip in right half of circle.

Using version 0.1.1 on Mac OS X Mavericks, I am noticing some issues with the tool tips when on the right-side of the circle. In particular, in Safari the tool tip flashes then vanishes. In Chrome, it stays visible, but flashes repeatedly whenever the mouse is moved (occasionally it vanishes). Firefox has a behaviour that averages the two - less flicker than Chrome, but it vanishes more frequently.

Tool-tips on the left half of the graph are fine in all three browsers. This is most-likely related to issue #4.

Firefox: 24.0
Safari: 7.0
Chrome: 31.0.1650.57

Michael

P.S. It might be useful to include a --version option to make it easier for people to report the snakeviz version for bug reports.

P.P.S. (did I mention how fantastic snakeviz is?!?! especially given how difficult it is to get RunSnakeRun to work on a 64bit Mac! thanks!)

Multiple reports displaying support

Hi!
I'm not sure that this is the best place for it. I wanted to make an improvement proposal, it would be really great to be able to view several reports at a time, maybe with some search functionality across the reports. I've bumped into the problem when I'm working with 66 files of profiling data, but maybe I miss some obvious points?

Improve tooltip positioning

The tooltip is currently always placed above and to the right of the mouse. It would be nice if it was smart and could change positions based on its position in the page so that, for example, it didn't run into the right hand edge of the browser window.

Running snakeviz with Django

I've had a go using runprofileserver: https://github.com/django-extensions/django-extensions/blob/master/django_extensions/management/commands/runprofileserver.py

but I get:

File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pstats.py", line 110, in load_stats
self.stats = marshal.load(f)
ValueError: bad marshal data (unknown type code)

both with and without the --kcachegrind switch. I'm guessing that the output isn't in the right format but I don't know much about this topic so I'm hoping you can shed some light.

Print complete URL when starting Snakeviz

If you start SnakeViz and it fails to open a browser the user has no way to know what URL to use to manually point a browser at the server. SnakeViz should print a complete URL to the terminal that a user can copy and paste into a browser to see the application. (Spun off from #46.)

Option to display percentages of total runtime

One useful feature of runsnakerun that might be a good addition to snakeviz would be an option for switching the displayed timings from the current absolute scale (seconds) to percentages of the total runtime.

This might also provide a good discriminator for dropping branches of the call tree with a small cumulative runtime (i.e. by default drop branches with a cumulative time of < 0.1% of the total time, for instance).

/static/bootstrap/img/ missing

I pip installed snakeviz and the resulting install didn't have bootstrap's img directory.

Scale figure size to window size

The figure size is currently hard-coded in wheel.js. We should probably make it dependent on the browser's window size.

Add handling when JSONHandler errors

If there is an error in the JSON handler the user is presented with a big empty SVG and the stats table. There should be an error message to indicate something has gone wrong.

When JSONHandler fails to return the d3.json callback function is called with null so that function should be expanded to handle the error case.

Improve table data and representation

Currently the stats table data is generated by capturing the output of pstats.print_stats(), which limits the numerical data. Any time smaller than 0.001 is shown as 0. This should be improved, especially so that the table sorting can use the full precision of the data.

Table cells can be be specified with both a numeric value and a string representation. For display purposes it would probably be best to calculate an explicit string representation of numbers (e.g. '10.4g', which automatically switches to exponential notation for large/small numbers) instead of having the table display all of the decimal points.

Give a "first try" example in the docs?

Could you add a super-simple example to the docs that first-time snakeviz (or maybe even Python profiler) users can run and then play with the GUI?

I first tried this which timed out when trying to load the profile into snakeviz and second this which does load, but doesn't contain any useful profile info (clicking on JSON gives this).

Maybe there could even be an examples folder with 3 to 5 interesting profiling examples of increasing complexity that the user can run to learn how it works?

ZeroDivisionError: float division by zero

I recently updated snakeviz from 0.1.4 to latest version using pip upgrade. The reason of upgrading it is that the chart was not getting displayed. Although the table with calls and other data was there. After reading this article (http://penandpants.com/2014/12/01/snakeviz-0-2/) , I thought it will work after upgrade. But after upgrade it even failed to show the table and gave following error:

Traceback (most recent call last):
  File "C:\Python27\lib\site-packages\tornado\web.py", line 1332, in _execute
    result = method(*self.path_args, **self.path_kwargs)
  File "C:\Python27\lib\site-packages\snakeviz\main.py", line 30, in get
    table_rows=table_rows(s), callees=json_stats(s))
  File "C:\Python27\lib\site-packages\snakeviz\stats.py", line 33, in table_rows
    tot_time_per = fmt(v[2] / v[0])
ZeroDivisionError: float division by zero

From the error, it looks that error handling for 'Division by zero' is missing.
My Env:
Windows 7
Python2.7.8

I also tried running it on Python 3.4, but it also failed with following error:

Traceback (most recent call last):
  File "C:\Python34\lib\site-packages\tornado\web.py", line 1332, in _execute
    result = method(*self.path_args, **self.path_kwargs)
  File "C:\Python34\lib\site-packages\snakeviz\main.py", line 27, in get
    s = Stats(profile_name)
  File "C:\Python34\lib\pstats.py", line 69, in __init__
    self.init(arg)
  File "C:\Python34\lib\pstats.py", line 83, in init
    self.load_stats(arg)
  File "C:\Python34\lib\pstats.py", line 97, in load_stats
    self.stats = marshal.load(f)
ValueError: bad marshal data (unknown type code)

Note: I can't share cprofile output as it has customer data.

Thanks,
Naveen

Add loading graphic and timeout behavior

Users could potentially load profiles that take a couple seconds to load or completely overwhelm the profile handling. We should add a loading graphic that displays while the profile is being processed, and the processing should timeout after ~10 seconds and the web page should display an error and suggestion like:

"There was an error processing this profile. You could try profiling a smaller piece of your program and displaying that. You can use the table below to inspect which functions are taking the most time."

Ignores cmdline argument for port

I try setting the port for snakeviz and it always runs on 8080.

Monospaced table text

I'd find the data in the table much easier to read if it would be in a monospaced font and slightly more dense.

'int' object is not iterable

Using snakeviz 0.4:

Traceback (most recent call last):
  File ".../site-packages/tornado/web.py", line 1413, in _execute
    result = method(*self.path_args, **self.path_kwargs)
  File ".../site-packages/snakeviz/main.py", line 30, in get
    table_rows=table_rows(s), callees=json_stats(s))
  File ".../site-packages/snakeviz/stats.py", line 65, in json_stats
    (keyfmt(*ck), list(cv)) for ck, cv in stats.stats[k][-1].items())
  File ".../site-packages/snakeviz/stats.py", line 65, in <genexpr>
    (keyfmt(*ck), list(cv)) for ck, cv in stats.stats[k][-1].items())
TypeError: 'int' object is not iterable

Add a button to collaps the visualization

Make it possible to collaps the div containing the figure to make it easier to work with the stats table, which is pretty useful in and of itself.

can't run on a machine with no local browser

I'm trying to run this on my VM so I can look at the results from the host machine which doesn't have Python. However I can't find a way around this error:

snakeviz: error: no web browser found: could not locate runnable browser

Improve the profile data to JSON conversion

The function stats_to_tree_dict in upload.py currently fails for some profiles for unknown reasons. The current known profile which causes it to fail is large, deep, and contains complex recursion.

Improve large profile handling

This issue is a replacement for #2 and eventually even this issue may need to be broken out into some more specific issues.

At the moment "large" profiles cause a number of problems for websnakeviz:

• They can overwhelm the stats_to_tree_dict function by creating so many dictionaries that memory usage becomes unsustainable.
• If stats_to_tree_dict does return the dictionary the tree may be so large that it swamps the json.dumps function.
• And then even if that returns the call tree might just have more nodes/complexity than can be elegantly displayed by the current sunburst visualization while staying responsive.

This issue should be a good place to discuss how to tackle the various aspects of large profile handling and any specific plans can spawn new issues.

Add in-browser filesystem browser for single_user_mode

I added a single_user_mode in 91413da that's specifically for a single user running the webserver on their local desktop, and it has very different security assumptions, for example allowing arbitrary files on the filesystem to be opened. This being the case, we can also build in a file browser to make it easy to switch between profile files in a directory.

Obviously this is not high priority, but it could be nice.

Memory usage

Hi,

Just a quick question because I'm really interested by snakeviz. Is it possible to profile memory usage with snakeviz ? Runsnakerun could do it with meliae.

Thanks !

HTTP 404 error due to bad URI on Win32

I recently installed Snakeviz for the first time using pip install snakeviz on my work laptop (thanks for making it pip-installable) which runs 64-bit Windows 7. After the speedy install, I was ready to test it on a profile file I had laying around. I fired up the server with snakeviz test.prof and was promptly greeted by this in Chrome:

Traceback (most recent call last):
File "C:\Python27\lib\site-packages\tornado\web.py", line 1037, in _execute
self.prepare()
File "C:\Python27\lib\site-packages\tornado\web.py", line 1487, in prepare
raise HTTPError(self._status_code)
HTTPError: HTTP 404: Not Found

I noticed the URL looked funny:
http://127.0.0.1:8080/viz/filec:/Users/mscruggs/Desktop/test.prof

I was able to get the page to load and use the tool by removing the drive portion of the URL (I removed "c:"):
http://127.0.0.1:8080/viz/file/Users/mscruggs/Desktop/test.prof

I placed print self.request on a new line in tornado/web.py just prior to self.prepare() on line 1037:

The output:

HTTPRequest(protocol='http', host='127.0.0.1:8080', method='GET', uri='/viz/filec:/Users/mscruggs/Desktop/test.prof', version='HTTP/1.1', remote_ip='127.0.0.1', body='', headers={'Connection': 'keep-alive', 'Accept-Language': 'en-US,en;q=0.8', 'Accept-Encoding': 'gzip,deflate,sdch', 'Host': '127.0.0.1:8080', 'Accept':
'text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8', 'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.66 Safari/537.36'})

I'm running Tornado 2.4 and Jinja2 2.7.1 on Python 2.7.5.

Please let me know if you have any questions or requests for additional debugging output/info.

Embed diagram in static HTML

I would like to include the central snakeviz diagram in a blogpost. Is there a convenient way to make this possible?

Add option to start SnakeViz as server-only

In some situations where a browser is not available it may still be useful to start the SnakeViz server and connect to it by other means. SnakeViz should have an option that causes the CLI to start the server, but not try to open it in a browser. (Spun off from #30.)

Broken on Python 2.6

  File "snakeviz/stats.py", line 51
    return {keyfmt(*k): v for k, v in d.items()}
                            ^
SyntaxError: invalid syntax

Dict comprehensions are new in 2.7.

Fix node size calculation

There are still some occasional errors calculating the correct sizes of nodes, see for example the norm_fit.prof example profile. Some nodes end up with a much larger angular extent than their parents.

Probably related to this there are cases when the summed sizes of a node's children is larger than the node's size. The code currently glosses over this but it's actually an indication that something might come out incorrectly in the display.

This all seems connected to the naive handling of calculated sizes when there are recursive calls, in that case the sum of calling parent.child_cumulative_time on all of a node's children will result in a value greater than 1. There should be some normalization or other appropriate handling to account for that.

Not working through SSH port forwards

Hi! Thanks for this wonderful tool! I'm using it a lot and many times I don't have direct access to where my files and soft are. So I use port forwarding a lot. However, this doesn't work with snakeviz.

Could it be due to absolute vs relative paths in the web application?

Thanks in advance to check this out and let me know if I can help

Using snakeviz with Ipython

I installed snakeviz, but cant get it to work with my ipython notebook. I running the follwoing code:

%snakeviz
ms = [5,10,15,20]
ns = range(5,10,1)
thd = 2
for m in ms:
mlabel = "losing prob: m=%s" % m
prob = []
for n in ns:
prob.append(estimate_losing_prob(m,n,thd))

and I'm getting:

*** Profile stats marshalled to file u'c:\users\idoda\appdata\local\temp\tmp03iiiv'.

What am I doing wrong?

How to run SnakeViz?

I'm a relative noobie to python so I'm missing something simple because I can't seem to run SnakeViz on my Mac (10.8.5, using fink python).

The website says to simply type snakeviz name_of_profile on the command line, but that doesn't seem to work, with the error command not found. I must be misunderstanding something basic because usually I run python scripts with something like python snakeviz.py name_of_profile, but there doesn't seem to be a snakeviz.py or anything similar that I would be able to call.

I'm pretty sure it's installed correctly because when I enter the python interpreter and type import snakeviz it appears to be successful, I just don't know how to actually invoke the program.

Can you help?

Tom

Fix automatic port selection

As described at https://github.com/jiffyclub/snakeviz/blob/master/snakeviz/cli.py#L20 the automatic port selection is currently not a thing.

Click functionality not working

Maybe a missing JS issue? Nothing happens when I click on different parts of the graph. Except the Reset button become enabled. Ubuntu 14.04, Chrome 36.0.1985.125.

Interactivity not working

Cool idea guys.

I can click on rings of the graph and the graph updates. However, hovering does not display any information about what function is associated with the ring. Clicking on rows in the table appears to not update anything either.

I've generated a simple profile with this script:

import time

def other_function():
    time.sleep(0.01)

def function():
    time.sleep(0.1)
    for i in range(10):
        other_function()

def main():
    for i in range(15):
        function()


if __name__ == '__main__':
    main()

and then generate profile data with

python -m cProfile -o output.prf profile.py

and then start snakeviz with

snakeviz output.prf

Visualization problems at leaf nodes

The sunburst visualization shows bizarre, incorrect behavior when zooming to the time.sleep leaf nodes in the sleep_recurse2.prof sample profile. This may or may not be related to issue #8.

Re-implement pstatsloader in JavaScript

One of the bottlenecks described in #12 is that it's not sensible to construct a JSON string of the entire call tree for a lot of profiles. It just takes too long and takes up too much space. We can get around this by only JSON-ifying the pstats.Stats.stats dictionary and constructing the call tree as a JavaScript structure in much the same way that the runsnake.pstatsloader module does in Python.

Alternate root directory

First off, I wanted to say thanks. I just started using this today, and the visualization is fantastic.

It would be nice to have an option to set the root directory for snakeviz, so the url doesn't need to include the absolute path. For example, if I generate a lot of profiles into a single directory, this would make it much faster to manually change the url to view them

The file browser you have discussed in some other tickets would be a nice feature also.

Create links to code?

Would it be possible to turn filename:lineno(function) like glob.py:20(iglob) into links that open the file at that line when clicked?

This would allow quickly browsing and reading the pieces of code that appear of interest due to their profile characteristics.

I don't know if the profiler stores the full file path, but it seems this should be possible for the common case where someone just created a profile and then runs snakeviz on it with the same Python interpreter on the same machine with the same site-packages?

Python3 support

Looks like there's some syntax issues with trying to run this in python3. I.e.:

$ snakeviz ./Framework/Settings/Classes/tests/String.profileTraceback (most recent call last):
  File "/usr/local/bin/snakeviz", line 9, in <module>
    load_entry_point('snakeviz==0.1.1', 'console_scripts', 'snakeviz')()
  File "/usr/lib/python3/dist-packages/pkg_resources.py", line 351, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/lib/python3/dist-packages/pkg_resources.py", line 2363, in load_entry_point
    return ep.load()
  File "/usr/lib/python3/dist-packages/pkg_resources.py", line 2088, in load
    entry = __import__(self.module_name, globals(),globals(), ['__name__'])
  File "/usr/local/lib/python3.4/dist-packages/snakeviz-0.1.1-py3.4.egg/snakeviz/cli.py", line 44
    except IOError, e:
                  ^
SyntaxError: invalid syntax

Curious if the scope of python2.x type stuff is know, and if there's some thoughts of making it python3 runable.

One can install this to the python2.x environment, but doesn't help you when trying to read python3 profiles:

File "/usr/lib/python2.7/pstats.py", line 110, in load_stats
self.stats = marshal.load(f)
ValueError: bad marshal data (unknown type code)

Bundle JS libraries?

I'm on a super-slow internet connection and get this error:

Error Loading Profile

Sorry, we were not able to load this profile! You can try profiling a smaller portion of your code. The statistics table below might help you narrow down which portion of your code to focus on.

I think the problem is that loading jquery.js and d3.js times out, but the error message doesn't say so.
The error message is incorrect in that there is not statistics table displayed below.

How about bundling the JS libraries?
(No-one really cares about an extra 1 MB these days, but slow internet connection or offline use happen.)

Add zooming to the sunburst visualization

Give give users the ability to zoom the sunburst by clicking on a node. This will make the node's parent the new root node. After zooming users will be able to zoom out a level by clicking on the sunburst center. A "Home" or "Reset" button should return to the most zoomed out (default) view.

An example: http://www.jasondavies.com/coffee-wheel/