jandy-team/jandy

A historical performance trace tool/service

Jandy

This project is fork from vondom/sky

Jandy is a dynamic analysis project for opensource developer using on github and travis-ci

Abstract

This project is made for opensource developers. The purpose of this project is analysis of performance and finds a tunning point of your applications. To open source developer who want to improve performance of your application, he and she should be used to find performance improvement on this service by application profiling. Hence, it support for many languages(java, python, ruby, etc…), then our service focused language specific performance(multi threaded, bad syntax uses, …). Then, we show the profile result how calling methods through intuitive, intelligent and efficiently data visualization.

Key Features

• Be tested and profiled application automatically through Travis-CI and Github integration
• Support for major languages such as java, python
• Visualize of application profiling results intuitively, effectively and detailed
• Provides to report by E-Mail, Twitter

Architecture & Introduction Paper

• Korean: Slideshare

Contributing & How to build

Read contributing document: CONTRIBUTING.md

License

This project is licensed by GPLv3

ThirdParty Libraries

• Spring Boot - http://projects.spring.io/spring-boot/
• Hibernate - http://hibernate.org/
• HSQLDB - http://hsqldb.org/
• MySQL Connector Java - http://dev.mysql.com/doc/connector-j/en/
• Jaal - https://github.com/jcooky/jaal
• QueryDsl - http://www.querydsl.com/
• EGit - https://github.com/eclipse/egit-github
• Guava - https://github.com/google/guava
• Apache Commons - https://commons.apache.org/
• Apache Thrift - https://thrift.apache.org
• jQuery - http://jquery.com/
• Bootstrap - http://getbootstrap.com/
• d3js - http://d3js.org/
• RaphaelJS - http://raphaeljs.com/
• lodash - https://lodash.com/docs
• font-awesome - https://fortawesome.github.io/Font-Awesome/
• gson - https://github.com/google/gson
• prettytime - http://www.ocpsoft.org/prettytime/
• slf4j - http://slf4j.org/
• PyYAML - http://pyyaml.org/
• requests - http://docs.python-requests.org/

Maintainer

• @JCooky

Thanks for contributing

• @edanprof - Director & Designer
• @syjsmk - Developer

jandy's Issues

Redesign for build page with 'Build Feeds'

We need a tutorial module for jandy contributor at first

Do that we not only support the master but also add to support the other branches

Add a web page for user who don't register any projects

Jandy has a bug related Github and logins

Python profiler can't find some global/local scope variable

When outside code is excuted in Jandy profiler we make sub_global and sub_local dictionary and set them as scope variables.
https://github.com/jcooky/jandy/blob/master/jandy-python/jandy/main.py#L28

But it fails as following when I tried to test request module.

Traceback (most recent call last):
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/runpy.py", line 170, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/nuko/.virtualenvs/test_import/lib/python3.4/site-packages/jandy/__main__.py", line 4, in <module>
    main()
  File "/Users/nuko/.virtualenvs/test_import/lib/python3.4/site-packages/jandy/main.py", line 39, in main
    six.exec_(code, globals(), sub_locals)
  File "setup.py", line 96, in <module>
    'socks': ['PySocks>=1.5.6'],
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/distutils/dist.py", line 955, in run_commands
    self.run_command(cmd)
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/distutils/dist.py", line 972, in run_command
    cmd_obj = self.get_command_obj(command)
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/distutils/dist.py", line 847, in get_command_obj
    cmd_obj = self.command_obj[command] = klass(self)
  File "/Users/nuko/.virtualenvs/test_import/lib/python3.4/site-packages/setuptools/__init__.py", line 125, in __init__
    _Command.__init__(self,dist)
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/distutils/cmd.py", line 62, in __init__
    self.initialize_options()
  File "setup.py", line 17, in initialize_options
    TestCommand.initialize_options(self)
NameError: name 'TestCommand' is not defined

Then I modified the global() like this

globals().update({
            '__file__': progname,
            '__name__': '__main__',
            '__package__': None
        })

I replaced sub_global and sub_local with this modified global() to save the profiling data.
What I don't understand is that it still raise NameError: name 'TestCommand' is not defined error when I use sub_local for local scope variable. I should use modified global() for both global and local scope parameter.

No salinity reading

Why is there no salinity reading when the Jandy system indicates when there is an issue with salt. This would mean that the salt level is known. Can this salt level be included in the API so that it can be reported in the integration?

Access token changes because of time out

Get 401 error in openapi request

at io.jandy.service.GitHubService.lambda$getOrgRepos$7(GitHubService.java:113) ~[classes/:na]
at io.jandy.service.GitHubService.executeWithCache0(GitHubService.java:136) ~[classes/:na]
at io.jandy.service.GitHubService.executeWithCache(GitHubService.java:130) ~[classes/:na]
at io.jandy.service.GitHubService.getOrgRepos(GitHubService.java:113) ~[classes/:na]
at io.jandy.web.ProjectController.getProjects(ProjectController.java:147) ~[classes/:na]

Hsql exception in project importing

org.hsqldb.HsqlException: integrity constraint violation: unique constraint or index violation; UK_2TF70CANVAWTJU721WCIO5JIB table: PROJECT

at io.jandy.web.ProfileController.importProject(ProfileController.java:115) ~[classes/:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_91]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_91]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_91]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_91]

Cache of Jandy about sign in is not refreshed

Should show build and profiling status for a project

Raise a SizeLimitExceededException in jandy-server

Uploading file size of profiling result thourgh thrift is too large

Jandy-python profiler is slower than other profilers

Today I tested the performance of Jandy-python profiler with my sample Django project.
Profiling time was measured which starts and ends with running profiler.
The result time(second) is as below.

--	jandy-python	cProfile	pyinstrument	vmprof	plop
python2	34.050	1.038	0.887	0.685	0.752
python3	11.682	1.030	1.157	0.955	0.901

Jandy-python profiler spent a lot more time than others. And I could figure out that compiled code execution time was almost same with the profiler exit time.

Until now I think "settrace" function inside profiling code makes some problems.
I found some info about this function.
http://stackoverflow.com/questions/1692866/what-cool-hacks-can-be-done-using-sys-settrace/1693108#1693108

And it looks like other profilers don't use "settrace" function.(at least as a default option.)
Python "signal" is used instead and it has lower overhead.

But "signal" doesn't support inter-thread communication. So we need other approach if we use this.
yappi <- This profiler supports multithread profiling.
Below is the Github page of other profilers that I used this time.

• pyinstrument - https://github.com/joerick/pyinstrument
• vmprof - https://github.com/vmprof/vmprof-python
• plop - https://github.com/bdarnell/plop

Change project name to "Coue"

"Every Day in Every Way I'm Getting Better"

How about to make a link in the build flow graph to search build-ticket quickly in the build-feed

Attached Snap shot.

Thanks 👍

Write documents of Jandy usage

Service occurs 500 error if committer name is not exists

Have the rate limit problem that accessing page logged in github user is depend on API

How to decrease github api dependencies.
We think that fix the problem is that jandy application has cache github api response in database

Does command option conflict with pytest?

When I profile Flask module with Jandy profiler, below error is occurred.
Flask is using pytest module for their testing and I guess "-m" command conflicts when Jandy adds this to script part of .jandy.yml file.

https://pytest.org/latest/usage.html

(test_import)gimmyeongjuui-MacBook-Pro:tests nuko$ jandy
===================================================================== test session starts =====================================================================
platform darwin -- Python 3.4.3, pytest-2.8.7, py-1.4.31, pluggy-0.3.1
rootdir: /Users/nuko/Git/flask, inifile: setup.cfg
plugins: cov-2.2.1, httpbin-0.2.0, mock-0.11.0
collected 292 items 

============================================================ 292 tests deselected by "-m 'jandy'" =============================================================
=============================================================== 292 deselected in 0.15 seconds ================================================================
Traceback (most recent call last):
  File "/Users/nuko/.virtualenvs/test_import/lib/python3.4/shutil.py", line 523, in move
    os.rename(src, real_dst)
FileNotFoundError: [Errno 2] No such file or directory: 'python-profiler-result.jandy' -> 'Mytest.jandy'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/nuko/.virtualenvs/test_import/bin/jandy", line 75, in <module>
    do_profile(yml['language'].lower(), sampleId, argv)
  File "/Users/nuko/.virtualenvs/test_import/bin/jandy", line 55, in do_profile
    shutil.move('python-profiler-result.jandy', id + '.jandy'),
  File "/Users/nuko/.virtualenvs/test_import/lib/python3.4/shutil.py", line 535, in move
    copy2(src, real_dst)
  File "/Users/nuko/.virtualenvs/test_import/lib/python3.4/shutil.py", line 245, in copy2
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/Users/nuko/.virtualenvs/test_import/lib/python3.4/shutil.py", line 108, in copyfile
    with open(src, 'rb') as fsrc:
FileNotFoundError: [Errno 2] No such file or directory: 'python-profiler-result.jandy'

We need to make overview page at the enterance

• profiling result for now
• what kind of profiling do you have in your project ?

Make a new test script for python profiler

The 'SimpleAsyncTaskExecutor-1' error raised and data can't be saved properly

The other day I solved this error with @jcooky by making new MySQL database correctly.
Command we used was create database jandy default collate=utf8_bin default charset=utf8;

Database is created like this.

mysql> show create database jandy;
+----------+----------------------------------------------------------------+
| Database | Create Database                                                |
+----------+----------------------------------------------------------------+
| jandy    | CREATE DATABASE `jandy` /*!40100 DEFAULT CHARACTER SET utf8 */ |
+----------+----------------------------------------------------------------+
1 row in set (0.00 sec)

And this error is still raised. Could this be related with using multiple terminal window at the same time?
I doubt the change of environment variables like TRAVIS_BUILD_ID or TRAVIS_BUILD_NUMBER.

Exception in thread "SimpleAsyncTaskExecutor-1" java.lang.RuntimeException: com.mysema.query.NonUniqueResultException: Only one result is allowed for uniqueResult calls
    at io.jandy.web.api.TravisRestController.lambda$putResults$13(TravisRestController.java:137)
    at org.springframework.util.concurrent.ListenableFutureCallbackRegistry.failure(ListenableFutureCallbackRegistry.java:133)
    at org.springframework.util.concurrent.ListenableFutureTask.done(ListenableFutureTask.java:87)
    at java.util.concurrent.FutureTask.finishCompletion(FutureTask.java:384)
    at java.util.concurrent.FutureTask.set(FutureTask.java:233)
    at java.util.concurrent.FutureTask.run(FutureTask.java:274)
    at java.lang.Thread.run(Thread.java:745)
Caused by: com.mysema.query.NonUniqueResultException: Only one result is allowed for uniqueResult calls
    at com.mysema.query.jpa.impl.AbstractJPAQuery.uniqueResult(AbstractJPAQuery.java:317)
    at com.mysema.query.jpa.impl.AbstractJPAQuery.uniqueResult(AbstractJPAQuery.java:304)
    at io.jandy.domain.ProfContextDumpRepositoryImpl.findLastProfile(ProfContextDumpRepositoryImpl.java:38)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.executeMethodOn(RepositoryFactorySupport.java:483)
    at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.doInvoke(RepositoryFactorySupport.java:458)
    at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.invoke(RepositoryFactorySupport.java:440)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.data.projection.DefaultMethodInvokingMethodInterceptor.invoke(DefaultMethodInvokingMethodInterceptor.java:61)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
    at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:281)
    at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:136)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.data.jpa.repository.support.CrudMethodMetadataPostProcessor$CrudMethodMetadataPopulatingMethodInterceptor.invoke(CrudMethodMetadataPostProcessor.java:131)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208)
    at com.sun.proxy.$Proxy118.findLastProfile(Unknown Source)
    at io.jandy.web.api.TravisRestController.lambda$null$11(TravisRestController.java:99)
    at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:133)
    at io.jandy.web.api.TravisRestController.lambda$putResults$12(TravisRestController.java:85)
    at org.springframework.util.concurrent.ListenableFutureCallbackRegistry.success(ListenableFutureCallbackRegistry.java:118)
    at org.springframework.util.concurrent.ListenableFutureTask.done(ListenableFutureTask.java:71)
    ... 4 more

Add features for multiple commands to profiling of Jandy

We support multiple commands.
The commands is named "Sample". Then we add feature for "Sample"s

Change permission about github account

Change personal user data permission

Improve performance that graph rendering is very slow in page "/repos/{user}/{repo}"

Profiled data should be saved by own library version and Python(Java) verison.

How can jandy show thread information?

Divide this repository into modules that it manages a language

• Each module has a profiler
• Each module has a sample
• Each module manage issues of itself

Fix a rate limit problem of the Travis-CI and GitHub API

This repository has a strong dependency of OpenAPIs
Then, we cache information for output of the APIs in our database

What can jandy show additional information of program profiling

Start new queue project

Add filter to python profiler

Write a new document

Use docker for deploment of binary distribution of jandy-server

Add a function to migrate about each version of a DB for service management

Github user repository count limitation

In user profile page some repositories are not shown. Currently the count is limited to 30.

When a project is not imported, the Jandy doesn't have to show the project

Currently, Jandy is showing disabled projects
We fix it for next version

Show "call by" with function parents of the function in graph page

Currently, jandy shows "call by" of a function parent. But, next version of Jandy shows "call by" of all parents of function

Cannot support embedded DB for developing

Occured error for docker login in build of travis-ci

Integrate github with the OAuth2 protocol

This issue is that sign in github through its login process with the OAuth2 protocol.

• use the module named for spring-security-oauth2

Support Django Framework of python language

Dependency of thrift in Jandy profilers occurs to do not run for django project about python3

The 'profile' page is not shown whole repositories which are owned by user

• No loading GitHub repositories without scroll in page
• Load imported projects into top of the page

Fix badge image caching problem on the github

On the github, it must be caching image.
Therefore, we need to change the badge image in README.md at build finished

How to fix this problem?