jvm-profiling-tools/honest-profiler

A sampling JVM profiler without the safepoint sample bias

honest-profiler

A usable and honest profiler for the JVM. For documentation please refer to The Wiki

• Download the Binary
• How to Build

honest-profiler's Issues

Support Mac OS

I don't have a mac to test on, need to find a helpful person.

Correct compilation under clang 3.5

When investigating #24 I realised that the build doesn't compile under clang 3.5. It should do.

Evaluate profiling interval

Profiling interval seems broken. Currently set usec to 1000, but it doesn't interrupt anywhere near that frequently.

sigsegv crash

no core dump but in

Stack: [0x00007ff8384fc000,0x00007ff8385fd000],  sp=0x00007ff8385fb840,  free space=1022k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x82e284]  Method::checked_resolve_jmethod_id(_jmethodID*)+0x34
V  [libjvm.so+0x6cc126]  jvmti_GetMethodName+0x1f6
C  [liblagent.so+0x8523]  Profiler::lookupFrameInformation(JVMPI_CallFrame const&, _jvmtiEnv*, MethodListener&)+0x43
C  [liblagent.so+0xb0ba]  LogWriter::record(JVMPI_CallTrace const&)+0x6a
C  [liblagent.so+0xa90a]  CircularQueue::pop()+0x2a
C  [liblagent.so+0xa959]  Processor::run()+0x19
V  [libjvm.so+0x736402]  JvmtiAgentThread::call_start_function()+0xc2
V  [libjvm.so+0x9e303f]  JavaThread::thread_main_inner()+0xdf
V  [libjvm.so+0x9e3205]  JavaThread::run()+0x1b5
V  [libjvm.so+0x8a00c8]  java_start(Thread*)+0x108

flat profile could show "self" and "tree" %age

currently the flat profile shows the %age of samples overall (I believe).
one enhancement would be to show that number as "self" - i.e. the time that method took on its own,

and in another column the "tree" time, the time for that method including all the time of the methods it called.

then we can sort / filter / analyse that list on both metrics fairly quickly.

Support Windows

I don't use windows or know the toolchain, need to find a helpful person.

Asgct::GetAsgct() returns NULL causing sigsegv on OSX

On OSX java 8u11, profiler crashes on startup every time. Seems to be that the pointer to asyncgetcalltrace is never set?

(lldb) up
frame #1: 0x0000000100360eac liblagent.so`Profiler::handle(this=0x0000000100400940, signum=27, info=0x00000001006fbed8, context=0x00000001006fbf40) + 252 at profiler.cpp:128
125 trace.env_id = env;
126
127 ASGCTType asgct = Asgct::GetAsgct();
-> 128 (*asgct)(&trace, kMaxFramesToCapture, context);
129
130 if (trace.num_frames < 0) {
131 int idx = -trace.num_frames;
(lldb) p asgct
(ASGCTType) $0 = 0x0000000000000000

Allow for a different unittest path

I've had to make the following change to the Makefile on OpenSUSE 13.1 to convince make-test to run:

diff --git a/Makefile b/Makefile
index 384cfe1..37afa37 100644
--- a/Makefile
+++ b/Makefile
@@ -66,7 +66,7 @@ COPTS:=$(PLATFORM_COPTS) $(CXX_COPTS) $(GLOBAL_COPTS) $(PLATFORM_WARNINGS) \
 # TODO: consider re-adding in production -fno-exceptions

 INCLUDES+=-I /usr/include/i386-linux-gnu/c++/4.8/ -I$(JAVA_HOME)/$(HEADERS) -I$(JAVA_HOME)/$(HEADERS)/$(UNAME) 
-TEST_INCLUDES=$(INCLUDES) -I/usr/include/unittest++
+TEST_INCLUDES=$(INCLUDES) -I/usr/include/unittest-cpp

 # LDFLAGS+=-Wl,--export-dynamic-symbol=Agent_OnLoad

hot attach support

It would be useful to be able to connect to a running process using the Attach API, and start collecting data.

Have the option to Collapse different trees into one tree across different threads.

the tree roots are all 1 each, meaning we can't see which tree in the forest is more costly than another.

the first line below is the flat profile, followed by the tree profile roots

$ grep java.lang.Thread.run /tmp/prof.out
0.21 java.lang.Thread.run

1.00 java.lang.Thread.run
1.00 java.lang.Thread.run
1.00 java.lang.Thread.run
1.00 java.lang.Thread.run
1.00 java.lang.Thread.run
1.00 java.lang.Thread.run
1.00 java.lang.Thread.run

Have the agent be able to send back profile information to remote daemons

use web-sockets as the transport medium.

Text and %age bars disappear frequently in the UI

Expanding or collapsing a tree item seems to cause this.
As new threads/methods are added to the tree it also seems to happen.

Scrolling the window will cause them to return.

notes on installing on centos7

• there's a requirement for java 8 and the oracle jdk rather than openjdk
• the c++ agent requires google protocol buffers, and the protoc binary needs copying to /usr/bin from /usr/local/bin

I had a couple of test failures also:

Tests in error:
recognise a removed machine(Web Socket Machine Finder): (..)
send data to the log consumer(Web Socket Machine Finder): (..)

Tests run: 20, Failures: 0, Errors: 2, Skipped: 1

Unify the build system

Maybe get maven to delegate to make for the c++ code.

tag samples by thread state

to be able to filter/report flat and tree profiles by one or more thread states.

this would be useful if you are looking for cpu consumers and don't want to see any samples that are waiting.

would be interested to see if we can distinguish between code like epoll that java will report as runnable but is waiting not consuming cpu.

Profile Filter methods

by
< N% of samples
and/or method name regex

to filter the ConsoleEntry view to a manageable size, add the ability to specify a threshold for printing methods, rationale being that typically those methods with fewer samples are less likely to provide optimisation opportunities.

I can do this easily enough with text processing for now so low priority,
what it would help with is reducing the amount of indentation for deep stacks.

The profiles are all the wrong way up

facepalm.

If two agents share the current directory the second will overwrite the first's log

Fix: Give log files a unique name.

Finish ProfileAdapter

Add opening UI window for opening files and listing local, connectable, JVMs

build errors in circular_queue.h

Maybe I'm doing something dumb, but I get these build errors:

root@bgregg-test-i-27eff009:~/honest-profiler# make
g++ -I/usr/include/i386-linux-gnu/c++/4.8/ -I/usr/jdk1.8.0_05/include -I/usr/jdk1.8.0_05/include/linux -I /usr/include/i386-linux-gnu/c++/4.8/ -I/usr/jdk1.8.0_05/include -I/usr/jdk1.8.0_05/include/linux   -mfpmath=sse -std=gnu++0x -fdiagnostics-show-option -fno-omit-frame-pointer -fno-strict-aliasing -funsigned-char -fno-asynchronous-unwind-tables -msse2 -g -D__STDC_FORMAT_MACROS  -Wframe-larger-than=16384 -Wno-unused-but-set-variable -Wunused-but-set-parameter -Wvla -Wno-conversion-null -Wno-unknown-pragmas -Wno-builtin-macro-redefined -Wall -Werror -Wformat-security -Wno-char-subscripts -Wno-sign-compare -Wno-strict-overflow -Wwrite-strings -Wnon-virtual-dtor -Woverloaded-virtual -O2 -Fvisibility=hidden -fPIC -c /root/honest-profiler/src/main/cpp/agent.cpp -o build/agent.pic.o
In file included from /root/honest-profiler/src/main/cpp/log_writer.h:6:0,
                 from /root/honest-profiler/src/main/cpp/processor.h:2,
                 from /root/honest-profiler/src/main/cpp/profiler.h:6,
                 from /root/honest-profiler/src/main/cpp/agent.cpp:9:
/root/honest-profiler/src/main/cpp/circular_queue.h:36:21: error: variable 'CircularQueue final' has initializer but incomplete type
/root/honest-profiler/src/main/cpp/circular_queue.h:37:1: error: expected primary-expression before 'public'
/root/honest-profiler/src/main/cpp/circular_queue.h:37:1: error: expected '}' before 'public'
/root/honest-profiler/src/main/cpp/circular_queue.h:37:1: error: expected ',' or ';' before 'public'
/root/honest-profiler/src/main/cpp/circular_queue.h:43:18: error: declaration of '~CircularQueue' as non-member
/root/honest-profiler/src/main/cpp/circular_queue.h:49:1: error: expected unqualified-id before 'private'
/root/honest-profiler/src/main/cpp/circular_queue.h:59:32: error: non-member function 'size_t advance(size_t)' cannot have cv-qualifier
/root/honest-profiler/src/main/cpp/circular_queue.h:62:1: error: expected declaration before '}' token
make: *** [build/agent.pic.o] Error 1
root@bgregg-test-i-27eff009:~/honest-profiler# g++ --version
g++ (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3
Copyright (C) 2011 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

maven hits the same errors.

Compilation errors with gcc 4.8.1

Hi,

I just tried to compile the native code on openSUSE 13.1 and get the following errors

[INFO] --- exec-maven-plugin:1.1:exec (make-test) @ honest-profiler ---
[INFO] g++ -I/usr/include/i386-linux-gnu/c++/4.8/ -I /usr/include/i386-linux-gnu/c++/4.8/ -I/opt/jdk1.8/include -I/opt/jdk1.8/include/linux -I/usr/include/unittest-cpp -mfpmath=sse -std=gnu++0x -fdiagnostics-show-option -fno-omit-frame-pointer -fno-strict-aliasing -funsigned-char -fno-asynchronous-unwind-tables -msse2 -g -D__STDC_FORMAT_MACROS -Wframe-larger-than=16384 -Wno-unused-but-set-variable -Wunused-but-set-parameter -Wvla -Wno-conversion-null -Wno-unknown-pragmas -Wno-builtin-macro-redefined -Wall -Werror -Wformat-security -Wno-char-subscripts -Wno-sign-compare -Wno-strict-overflow -Wwrite-strings -Wnon-virtual-dtor -Woverloaded-virtual -O2 -Fvisibility=hidden -fPIC -c /home/ADOBENET/rmuntean/w/git/honest-profiler/src/test/cpp/test_circular_queue.cpp -o build-test/test_circular_queue.pic.o
[INFO] In file included from /usr/include/unittest-cpp/UnitTest++.h:14:0,
[INFO] from /home/ADOBENET/rmuntean/w/git/honest-profiler/src/test/cpp/test_circular_queue.cpp:1:
[INFO] /home/ADOBENET/rmuntean/w/git/honest-profiler/src/test/cpp/test_circular_queue.cpp: In function ‘void checkStackTrace(const JVMPI_CallTrace&, int)’:
[INFO] /home/ADOBENET/rmuntean/w/git/honest-profiler/src/test/cpp/test_circular_queue.cpp:27:25: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
[INFO] CHECK_EQUAL((JNIEnv )envId, trace.env_id);
[INFO] ^
[INFO] /usr/include/unittest-cpp/CheckMacros.h:51:69: note: in definition of macro ‘CHECK_EQUAL’
[INFO] UnitTest::CheckEqual(UnitTest::CurrentTest::Results(), expected, actual, UnitTest::TestDetails(UnitTest::CurrentTest::Details(), LINE));
[INFO] ^
[INFO] /home/ADOBENET/rmuntean/w/git/honest-profiler/src/test/cpp/test_circular_queue.cpp: In function ‘void pushLocalTraceOnto(CircularQueue&, int)’:
[INFO] /home/ADOBENET/rmuntean/w/git/honest-profiler/src/test/cpp/test_circular_queue.cpp:53:19: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
[INFO] givenStackTrace(envId);
[INFO] ^
[INFO] /home/ADOBENET/rmuntean/w/git/honest-profiler/src/test/cpp/test_circular_queue.cpp:21:28: note: in definition of macro ‘givenStackTrace’
[INFO] trace.env_id = (JNIEnv *)envId;
[INFO] ^
[INFO] /home/ADOBENET/rmuntean/w/git/honest-profiler/src/test/cpp/test_circular_queue.cpp: In function ‘void runnable(int, CircularQueue)’:
[INFO] /home/ADOBENET/rmuntean/w/git/honest-profiler/src/test/cpp/test_circular_queue.cpp:105:19: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
[INFO] givenStackTrace(start);
[INFO] ^
[INFO] /home/ADOBENET/rmuntean/w/git/honest-profiler/src/test/cpp/test_circular_queue.cpp:21:28: note: in definition of macro ‘givenStackTrace’
[INFO] trace.env_id = (JNIEnv )envId;
[INFO] ^
[INFO] /home/ADOBENET/rmuntean/w/git/honest-profiler/src/test/cpp/test_circular_queue.cpp:108:31: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
[INFO] trace.env_id = (JNIEnv *) i;
[INFO] ^
[INFO] /home/ADOBENET/rmuntean/w/git/honest-profiler/src/test/cpp/test_circular_queue.cpp: In member function ‘void GivenQueueMultiThreadedRunHelper::RunImpl()’:
[INFO] /home/ADOBENET/rmuntean/w/git/honest-profiler/src/test/cpp/test_circular_queue.cpp:129:29: error: cast from ‘JNIEnv {aka JNIEnv}’ to ‘int’ loses precision [-fpermissive]
[INFO] int counter = (int) out.env_id;
[INFO] ^
[INFO] cc1plus: all warnings being treated as errors
[INFO] make: *_* [build-test/test_circular_queue.pic.o] Error 1

$ gcc --version
gcc (SUSE Linux) 4.8.1 20130909 [gcc-4_8-branch revision 202388]
Copyright (C) 2013 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

why does it only run under java 8?

when i compile the code, i found it require java 8, do you want to support java 7 and lower?

Unreported BufferUnderflowException

While this (or possibly any?) log file is being written, both the console and javafx UIs print this to the console, but continue as if it hasn't happened.

I can't share the log file.

java.nio.BufferUnderflowException
    at java.nio.Buffer.nextGetIndex(Buffer.java:506)
    at java.nio.DirectByteBuffer.getInt(DirectByteBuffer.java:677)
    at com.insightfullogic.honest_profiler.core.parser.LogParser.readStackFrame(LogParser.java:72)
    at com.insightfullogic.honest_profiler.core.parser.LogParser.readRecord(LogParser.java:42)
    at com.insightfullogic.honest_profiler.core.DataConsumer.accept(DataConsumer.java:31)
    at com.insightfullogic.honest_profiler.core.LogConsumer.run(LogConsumer.java:29)
    at com.insightfullogic.honest_profiler.core.Conductor.consumeFile(Conductor.java:31)
    at com.insightfullogic.honest_profiler.delivery.javafx.landing.LandingViewModel.open(LandingViewModel.java:61)

Both UIs seem also to not be processing the whole log, but aren't getting useful information out of it: (The code isn't using ZipFile or nanoTime or Permissions or files or..., I suspect these entries are from the nanoseconds after the thing started, before the "error" in the log?):

Number of stack traces: 123397

Flat Profile:
 0.00 java.util.zip.ZipFile.open
 0.00 java.lang.System.nanoTime
 0.00 java.security.Permissions.add
 0.00 java.io.UnixFileSystem.getBooleanAttributes0
 0.00 java.util.zip.ZipFile.<init>
 0.00 java.util.jar.JarVerifier.<init>

Default to using tmpfs for the log files

Only use a persisted file if people provide a -log option.

disambiguate "duplicate" methods

sometimes we get "duplicated" method names:

...
0.07 java.net.URLClassLoader$1.run
0.07 java.net.URLClassLoader$1.run
...

this could happen for a number of reasons, it would be useful to have the option to collapse/aggregate these into one line item in the reporting or to disambiguate them with some additional information to indicate what one or other relates to or how it came to be different from the other.

else this could be a bug where we are double counting a method in some way.

Decouple flat profiles from trees

Ability to interactively start and stop the profiler

Add configuration for things that should be configurable

For example:

• Sampling frequency.
• Number of frames to be captured.
• Name of file to dump the log to.

Remove the dependency on boost iostreams

Makes it easier for people using Macs to build the system and cuts down on complexity. Can just straight mmap as we're not supporting Windows at the moment. @mikeb01 was looking into this.

Add clang-analyser support to the makefile

Proper 64 bit testing

Currently only tested on a 32bit install.

Log rotation

Request to have logs that get rotated after some time period, to make it easier to manage and upload them.

Stop NIHing encoding

Replace with an existing external protocol, eg SBE or protocol buffers.

FileNotFoundException: log.hpl when CWD != app's CWD

java.io.FileNotFoundException: log.hpl (No such file or directory)
    at java.io.RandomAccessFile.open(Native Method)
    at java.io.RandomAccessFile.<init>(RandomAccessFile.java:236)
    at com.insightfullogic.honest_profiler.core.conductor.LogConsumer.<init>(LogConsumer.java:23)
    at com.insightfullogic.honest_profiler.core.conductor.Conductor.pipeFile(Conductor.java:25)
    at com.insightfullogic.honest_profiler.delivery.javafx.landing.LandingViewModel.monitor(LandingViewModel.java:73)

• LocalMachineSource#attach's agentProperties never seems to have a "user.dir" in it,
• userDir ends up all null
• VirtualMachine#getLogFile's new File(null, "log.hpl") kindly treats this as a path relative to the honest-profiler starting directory.

Ability to build java code without having to compile c++ code

Maybe a maven profile.

Flamegraphs

CPU flame graphs are a useful way of visualizing a lot of stacktraces. It would be nice to have an output mode which is compatible with the input of the flame graph tools at https://github.com/brendangregg/FlameGraph or alternatively a renderer output in the UI which draws flame graphs. If it is implemented in the UI, it would be important to be able to generate a flame graph from a previously recorded log file as well as some live snapshot mode.

Historical log storage and lookup

siblings in tree view are not sorted

for ease of viewing and priority to the most costly methods it would be useful to sort siblings in the tree so that the highest %age was printed first.

process hang

the profiler starts but hangs before the application starts doing any useful work.
this does not happen all the time.

here is gdb output attached to the running process:

(gdb) info threads
  Id   Target Id         Frame 
  14   Thread 0x7f1c9691c700 (LWP 6429) "java" 0x00007f1c95e18d3c in __lll_lock_wait_private () from /lib64/libc.so.6
  13   Thread 0x7f1c7e8cd700 (LWP 6430) "java" 0x00007f1c964fc705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  12   Thread 0x7f1c7e7cc700 (LWP 6431) "java" 0x00007f1c964fc705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  11   Thread 0x7f1c7db55700 (LWP 6432) "java" 0x00007f1c964fcab2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  10   Thread 0x7f1c7cfba700 (LWP 6433) "java" 0x00007f1c964fc705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  9    Thread 0x7f1c7ceb9700 (LWP 6434) "java" 0x00007f1c964fc705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  8    Thread 0x7f1c7cdb8700 (LWP 6435) "java" 0x00007f1c964fc705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  7    Thread 0x7f1c7ccb7700 (LWP 6436) "java" 0x00007f1c964fc705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  6    Thread 0x7f1c7cbb6700 (LWP 6437) "java" 0x00007f1c964fc705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  5    Thread 0x7f1c7cab5700 (LWP 6438) "java" 0x00007f1c964fc705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  4    Thread 0x7f1c7c9b4700 (LWP 6439) "java" 0x00007f1c964fc705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  3    Thread 0x7f1c7c8b3700 (LWP 6440) "java" 0x00007f1c964fc705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  2    Thread 0x7f1c7c7b2700 (LWP 6441) "java" 0x00007f1c964fc705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
* 1    Thread 0x7f1c9691d740 (LWP 6428) "java" 0x00007f1c964f9f37 in pthread_join () from /lib64/libpthread.so.0
(gdb) thread 14
[Switching to thread 14 (Thread 0x7f1c9691c700 (LWP 6429))]
#0  0x00007f1c95e18d3c in __lll_lock_wait_private () from /lib64/libc.so.6
(gdb) where
#0  0x00007f1c95e18d3c in __lll_lock_wait_private () from /lib64/libc.so.6
#1  0x00007f1c95d97f66 in _L_lock_12192 () from /lib64/libc.so.6
#2  0x00007f1c95d954e1 in malloc () from /lib64/libc.so.6
#3  0x00007f1c9670e0a6 in tls_get_addr_tail () from /lib64/ld-linux-x86-64.so.2
#4  0x00007f1c942a6279 in CurrentJniEnv () at /home/vagrant/git/RichardWarburton/honest-profiler/src/main/cpp/globals.h:134
#5  Profiler::handle (this=0x7f1c900086a0, signum=<optimized out>, info=<optimized out>, context=0x7f1c969199c0) at /home/vagrant/git/RichardWarburton/honest-profiler/src/main/cpp/profiler.cpp:113
#6  <signal handler called>
#7  0x00007f1c95e05d87 in mprotect () from /lib64/libc.so.6
#8  0x00007f1c95d9414d in _int_malloc () from /lib64/libc.so.6
#9  0x00007f1c95d954ec in malloc () from /lib64/libc.so.6
#10 0x00007f1c95689448 in os::malloc(unsigned long, unsigned short, unsigned char*) () from /usr/lib/jvm/jdk1.8.0_05/jre/lib/amd64/server/libjvm.so
#11 0x00007f1c9509f9d3 in ChunkPool::allocate(unsigned long, AllocFailStrategy::AllocFailEnum) () from /usr/lib/jvm/jdk1.8.0_05/jre/lib/amd64/server/libjvm.so
#12 0x00007f1c9509f0a1 in Arena::grow(unsigned long, AllocFailStrategy::AllocFailEnum) () from /usr/lib/jvm/jdk1.8.0_05/jre/lib/amd64/server/libjvm.so
#13 0x00007f1c950d4a47 in HandleArea::allocate_handle(oopDesc*) () from /usr/lib/jvm/jdk1.8.0_05/jre/lib/amd64/server/libjvm.so
#14 0x00007f1c954232cd in initialize_static_field(fieldDescriptor*, Thread*) () from /usr/lib/jvm/jdk1.8.0_05/jre/lib/amd64/server/libjvm.so
#15 0x00007f1c953cf897 in InstanceKlass::do_local_static_fields_impl(instanceKlassHandle, void (*)(fieldDescriptor*, Thread*), Thread*) () from /usr/lib/jvm/jdk1.8.0_05/jre/lib/amd64/server/libjvm.so
#16 0x00007f1c95425548 in java_lang_Class::create_mirror(KlassHandle, Handle, Thread*) () from /usr/lib/jvm/jdk1.8.0_05/jre/lib/amd64/server/libjvm.so
#17 0x00007f1c95213c03 in ClassFileParser::parseClassFile(Symbol*, ClassLoaderData*, Handle, KlassHandle, GrowableArray<Handle>*, TempNewSymbol&, bool, Thread*) ()
   from /usr/lib/jvm/jdk1.8.0_05/jre/lib/amd64/server/libjvm.so
#18 0x00007f1c952177fd in ClassLoader::load_classfile(Symbol*, Thread*) () from /usr/lib/jvm/jdk1.8.0_05/jre/lib/amd64/server/libjvm.so
#19 0x00007f1c957a02f0 in SystemDictionary::load_instance_class(Symbol*, Handle, Thread*) () from /usr/lib/jvm/jdk1.8.0_05/jre/lib/amd64/server/libjvm.so
#20 0x00007f1c9579f3c6 in SystemDictionary::resolve_instance_class_or_null(Symbol*, Handle, Handle, Thread*) () from /usr/lib/jvm/jdk1.8.0_05/jre/lib/amd64/server/libjvm.so
#21 0x00007f1c957a0783 in SystemDictionary::resolve_or_fail(Symbol*, Handle, Handle, bool, Thread*) () from /usr/lib/jvm/jdk1.8.0_05/jre/lib/amd64/server/libjvm.so
#22 0x00007f1c9529c078 in ConstantPool::klass_at_impl(constantPoolHandle, int, Thread*) () from /usr/lib/jvm/jdk1.8.0_05/jre/lib/amd64/server/libjvm.so
#23 0x00007f1c952a1670 in ConstantPool::klass_ref_at(int, Thread*) () from /usr/lib/jvm/jdk1.8.0_05/jre/lib/amd64/server/libjvm.so
#24 0x00007f1c9556d891 in LinkResolver::resolve_pool(KlassHandle&, Symbol*&, Symbol*&, KlassHandle&, constantPoolHandle, int, Thread*) () from /usr/lib/jvm/jdk1.8.0_05/jre/lib/amd64/server/libjvm.so
#25 0x00007f1c955742e5 in LinkResolver::resolve_invokestatic(CallInfo&, constantPoolHandle, int, Thread*) () from /usr/lib/jvm/jdk1.8.0_05/jre/lib/amd64/server/libjvm.so
#26 0x00007f1c955771d5 in LinkResolver::resolve_invoke(CallInfo&, Handle, constantPoolHandle, int, Bytecodes::Code, Thread*) () from /usr/lib/jvm/jdk1.8.0_05/jre/lib/amd64/server/libjvm.so
#27 0x00007f1c95411aa2 in InterpreterRuntime::resolve_invoke(JavaThread*, Bytecodes::Code) () from /usr/lib/jvm/jdk1.8.0_05/jre/lib/amd64/server/libjvm.so
#28 0x00007f1c7ecac6c8 in ?? ()
#29 0x00007f1c7ecac683 in ?? ()
#30 0x00007f1c9691b778 in ?? ()
#31 0x00007f1c7d2a9368 in ?? ()
#32 0x00007f1c9691b7c8 in ?? ()
#33 0x00007f1c7d419a68 in ?? ()
#34 0x0000000000000000 in ?? ()




(gdb) thread 1
[Switching to thread 1 (Thread 0x7f1c9691d740 (LWP 6428))]
#0  0x00007f1c964f9f37 in pthread_join () from /lib64/libpthread.so.0
(gdb) where
#0  0x00007f1c964f9f37 in pthread_join () from /lib64/libpthread.so.0
#1  0x00007f1c962e4d15 in ContinueInNewThread0 () from /usr/lib/jvm/jdk1.8.0_05/bin/../lib/amd64/jli/libjli.so
#2  0x00007f1c962e063a in ContinueInNewThread () from /usr/lib/jvm/jdk1.8.0_05/bin/../lib/amd64/jli/libjli.so
#3  0x00007f1c962e3250 in JLI_Launch () from /usr/lib/jvm/jdk1.8.0_05/bin/../lib/amd64/jli/libjli.so
#4  0x0000000000400696 in main ()
(gdb) thread 11
[Switching to thread 11 (Thread 0x7f1c7db55700 (LWP 6432))]
#0  0x00007f1c964fcab2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
(gdb) where
#0  0x00007f1c964fcab2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f1c9569102f in os::PlatformEvent::park(long) () from /usr/lib/jvm/jdk1.8.0_05/jre/lib/amd64/server/libjvm.so
#2  0x00007f1c9565044e in Monitor::IWait(Thread*, long) () from /usr/lib/jvm/jdk1.8.0_05/jre/lib/amd64/server/libjvm.so
#3  0x00007f1c95650a16 in Monitor::wait(bool, long, bool) () from /usr/lib/jvm/jdk1.8.0_05/jre/lib/amd64/server/libjvm.so
#4  0x00007f1c952926c7 in ConcurrentMarkSweepThread::wait_on_cms_lock_for_scavenge(long) () from /usr/lib/jvm/jdk1.8.0_05/jre/lib/amd64/server/libjvm.so
#5  0x00007f1c95292d30 in ConcurrentMarkSweepThread::run() () from /usr/lib/jvm/jdk1.8.0_05/jre/lib/amd64/server/libjvm.so
#6  0x00007f1c956920c8 in java_start(Thread*) () from /usr/lib/jvm/jdk1.8.0_05/jre/lib/amd64/server/libjvm.so
#7  0x00007f1c964f8df3 in start_thread () from /lib64/libpthread.so.0
#8  0x00007f1c95e0b3dd in clone () from /lib64/libc.so.6
(gdb) thread 7
[Switching to thread 7 (Thread 0x7f1c7ccb7700 (LWP 6436))]
#0  0x00007f1c964fc705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
(gdb) where
#0  0x00007f1c964fc705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f1c9568bef3 in os::PlatformEvent::park() () from /usr/lib/jvm/jdk1.8.0_05/jre/lib/amd64/server/libjvm.so
#2  0x00007f1c956500f7 in Monitor::IWait(Thread*, long) () from /usr/lib/jvm/jdk1.8.0_05/jre/lib/amd64/server/libjvm.so
#3  0x00007f1c95650a96 in Monitor::wait(bool, long, bool) () from /usr/lib/jvm/jdk1.8.0_05/jre/lib/amd64/server/libjvm.so
#4  0x00007f1c952684ce in SurrogateLockerThread::loop() () from /usr/lib/jvm/jdk1.8.0_05/jre/lib/amd64/server/libjvm.so
#5  0x00007f1c957d503f in JavaThread::thread_main_inner() () from /usr/lib/jvm/jdk1.8.0_05/jre/lib/amd64/server/libjvm.so
#6  0x00007f1c957d5205 in JavaThread::run() () from /usr/lib/jvm/jdk1.8.0_05/jre/lib/amd64/server/libjvm.so
#7  0x00007f1c956920c8 in java_start(Thread*) () from /usr/lib/jvm/jdk1.8.0_05/jre/lib/amd64/server/libjvm.so
#8  0x00007f1c964f8df3 in start_thread () from /lib64/libpthread.so.0
#9  0x00007f1c95e0b3dd in clone () from /lib64/libc.so.6

Clean the file when restarting the agent

Implement a GUI

Currently the UI only performs command line printing. Would be nice to implement a richer UI in JavaFX or Swing.

Build fails due to dependency on lambda-behave

Fixed using:

git clone [email protected]:RichardWarburton/lambda-behave.git
cd lambda-behave/
git checkout lambda-behave-parent-0.2
mvn clean install

Might need a docs update until lambda-behave makes it to central.

Truncate log file when application completes

Can this be done in a way that doesn't disrupt a monitoring program?

Online profiling

Currently the UI just reads a log. In order to profile while the application is running means being able to read the profile log while writing it.

Makefile assumes building for 32-bit machine

When trying to build under linux (debian/ubuntu), the Makefile defaults to BITS=32, which causes -m32 to be passed to $(CXX) (which is called $(CC) to annoy me), giving confusing errors like:

In file included from <command-line>:0:0:
/usr/include/stdc-predef.h:30:26: fatal error: bits/predefs.h: No such file or directory
 #include <bits/predefs.h>

In file included from /usr/include/features.h:341:0,
                 from /usr/include/stdio.h:27,
                 from /var/tmp/faux140221.1/honest-profiler/src/main/cpp/agent.cc:1:
/usr/include/stdc-predef.h:30:26: fatal error: bits/predefs.h: No such file or directory

Hook up the reading of local logs in the web ui

Including decoupling non-parser concerns from the parser and into the conductor.

Proper error logging in the agent

Have a nice, text-based, log format with human-readable error messages.

console client hangs when piped through head(1) (?)

faux@alohura:~/ins/honest-profiler% time ./console -log ~/code/glue/foo/gui/log.hpl | head     
Number of stack traces: 23684

Flat Profile:
    0.97 java.util.DualPivotQuicksort.sort
    0.11 com.gs.collections.impl.bag.mutable.primitive.DoubleHashBag.add
    0.04 java.util.Arrays.copyOf
./console -log ~/code/glue/foo/gui/log.hpl  13.75s user 0.76s system 125% cpu 11.537 total
head  0.00s user 0.00s system 0% cpu 0.532 total


faux@alohura:~/ins/honest-profiler% time ./console -log ~/code/glue/foo/gui/log.hpl > /dev/null
./console -log ~/code/glue/foo/gui/log.hpl > /dev/null  3.58s user 0.22s system 345% cpu 1.100 total

Can't explain that.

Maybe a built-in limit option would be nice?

Move to cmake

For windows portability (see #2). The cmake file should also be called as part of the maven build.