jvm-profiling-tools/perf-map-agent

A java agent to generate method mappings to use with the linux `perf` tool

perf-map-agent

A java agent to generate /tmp/perf-<pid>.map files for just-in-time(JIT)-compiled methods for use with the Linux perf tools.

Build

Make sure JAVA_HOME is configured to point to a JDK. You need cmake >= 2.8.6 (see #30). Then run the following on the command line:

cmake .
make

# will create links to run scripts in <somedir>
bin/create-links-in <somedir>

Architecture

Linux perf tools will expect symbols for code executed from unknown memory regions at /tmp/perf-<pid>.map. This allows runtimes that generate code on the fly to supply dynamic symbol mappings to be used with the perf suite of tools.

perf-map-agent is an agent that will generate such a mapping file for Java applications. It consists of a Java agent written C and a small Java bootstrap application which attaches the agent to a running Java process.

When the agent is attached it instructs the JVM to report code blobs generated by the JVM at runtime for various purposes. Most importantly, this includes JIT-compiled methods but also various dynamically-generated infrastructure parts like the dynamically created interpreter, adaptors, and jump tables for virtual dispatch (see vtable and itable entries). The agent creates a /tmp/perf-<pid>.map file which it fills with one line per code blob that maps a memory location to a code blob name.

The Java application takes the PID of a Java process as an argument and an arbitrary number of additional arguments which it passes to the agent. It then attaches to the target process and instructs it to load the agent library.

Command line scripts

The bin directory contains a set of shell scripts to combine common perf / dtrace perations with creating the map file.

• create-java-perf-map.sh <pid> <options*> takes a PID and options. It knows where to find libraries relative to the bin directory.
• perf-java-top <pid> <perf-top-options> takes a PID and additional options to pass to perf top. Uses the agent to create a new /tmp/perf-<pid>.map and then calls perf top with the given options.
• perf-java-record-stack <pid> <perf-record-options> takes a PID and additional options to pass to perf record. Runs perf record -g -p <pid> <perf-record-options> to collect performance data including stack traces. Afterwards it uses the agent to create a new /tmp/perf-<pid>.map file.
• perf-java-report-stack <pid> <perf-record-options> calls first perf-java-record-stack <pid> <perf-record-options> and then runs perf report to directly analyze the captured data. You can call perf report -i /tmp/perf-<pid>.data again with any options after the script has exited to further analyze the data from the previous run.
• perf-java-flames <pid> <perf-record-options> collects data with perf-java-record-stack and then creates a visualization using @brendangregg's FlameGraph tools. To get meaningful stacktraces spanning several JIT-compiled methods, you need to run your JVM with -XX:+PreserveFramePointer (which is available starting from JDK8 update 60 build 19) as detailed in ag netflix blog entry.
• create-links-in <targetdir> will install symbolic links to the above scripts into <targetdir>.
• dtrace-java-record-stack <pid> takes a PID. Runsdtrace to collect performance data including stack traces. Afterwards it uses the agent to create a new /tmp/perf-<pid>.map file.
• dtrace-java-flames <pid> collects data with dtrace-java-record-stack and then creates a visualization using @brendangregg's FlameGraph tools. To get meaningful stacktraces spanning several JIT-compiled methods, you need to run your JVM with -XX:+PreserveFramePointer (which is available starting from JDK8 update 60 build 19) as detailed in ag netflix blog entry.

Environment variables:

• PERF_MAP_OPTIONS: a string of additional options to pass to the agent as described below.
• PERF_RECORD_SECONDS: the number of seconds, perf-java-report-stack and similar tools will record performance data
• PERF_RECORD_FREQ: the sampling frequence as passed to perf record -F
• FLAMEGRAPH_DIR: the directory into which @brendangregg's FlameGraph has been checked out
• PERF_JAVA_TMP: the directory to put temporary files in, the default is /tmp
• PERF_DATA_FILE: the file name where perf-java-record-stack will output performance data into, the default is $PERF_JAVA_TMP/perf-<pid>.data
• PERF_COLLAPSE_OPTS: a string of additional flags to pass to stackcollapse-perf.pl (found in FLAMEGRAPH_DIR), (add --inline with unfoldall perfmap)
• PERF_FLAME_OUTPUT: the file name to which the flamegraph SVG will be written, the default is flamegraph-<pid>.svg
• PERF_FLAME_OPTS: options to pass to flamegraph.pl (found in FLAMEGRAPH_DIR), the default is --color java
• DTRACE_SECONDS: the number of seconds, dtrace and similar tools will record performance data
• DTRACE_FREQ: the sampling frequence as passed to dtrace
• DTRACE_JAVA_TMP: the directory to put temporary files in, the default is /tmp
• DTRACE_DATA_FILE: the file name where dtrace-java-record-stack will output performance data into, the default is $DTRACE_JAVA_TMP/dtrace-<pid>.data

Options

You can add a comma separated list of options to perf-java (or the AttachOnce runner). These options are currently supported:

• unfold: Create extra entries for every codeblock inside a method that was inlined from elsewhere (named <inlined_method> in <root_method>). Be aware of the effects of 'skid' in relation with unfolding. See the section below. Also, see the below section about inaccurate inlining information.
• unfoldall: Similar to unfold but will include the complete inlined stack at a code location in the form root_method->inlined method 1->inlined method 2->...->inlined method on top.
• unfoldsimple: similar to unfold, however, the extra entries do not include the " in <root_method>" part
• msig: include full method signature in the name string
• dottedclass: convert class signature (Ljava/lang/Class;) to the usual class names with segments separated by dots (java.lang.Class). NOTE: this currently breaks coloring when used in combination with flamegraphs.
• sourcepos: Adds the name of the source file and the line number on which it is declared for each method. Useful when profiling Scala applications that crate a lot of synthetic classes and methods. Does not work with native methods.

Known Issues

Skid

You should be aware that instruction level profiling is not absolutely accurate but suffers from 'skid'. 'skid' means that the actual instruction pointer may already have moved a bit further when a sample is recorded. In that case, (possibly hot) code is reported at an address shortly after the actual hot instruction. See this sample from one of Brendan's presentations demonstrating this issue.

If using unfold, perf-map-agent will report sections that contain code inlined from other methods as separate entries. Unfolded entries can be quite short, e.g. an inlined getter may only consist of a few instructions that now lives inside of another method's JITed code. The next few instructions may then already belong to another entry. In such a case, it is more likely that skid will not only affect the instruction pointer inside of a method entry but may affect which entry is chosen in the first place.

Skid that occurs inside a method is only visible when analyzing the actual assembler code (as with perf annotate). Skid that affects the actual symbol resolution to choose a wrong entry will be much more visible as wrong entries will be reported with tools that operate on the symbol level like the standard views of perf report, perf top, or in flame graphs.

So, while it is tempting to enable unfolded entries for the perceived extra resolution, this extra information is sometimes just noise which will not only clutter the overall view but may also be misleading or wrong.

Inaccurate mappings using the unfold* options

Hotspot does not retain line number and other debug information for inlined code at other places than safepoints. This makes sense because you don't usually observe code running between safepoints from the JVM's perspective. This is different when observing a process from the outside like with perf. For observed code locations outside of safepoints, the JVM will not report any inlining information and perf-map-agent will assign those areas to the host method of the inlining.

For more fidelity, Hotspot can be instructed to include debug information for non-safepoints as well. Use -XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints when running the target process. Note, however, that this will produce a lot more information with the generated perf-<pid>.map file potentially growing to MBs of size.

Agent Library Unloading

Unloading or reloading of a changed agent library is not supported by the JVM (but re-attaching is). Therefore, if you make changes to the agent and recompile it you need to restart a target process that has an older version loaded to use the newer version.

Missing symbols for libjvm.so

libjvm.so is the runtime component of the JVM. It is not covered by perf-map-agent but perf will use debug symbols as provided by the distribution. If symbols for libjvm.so are missing see instructions for your Linux distribution to install debug symbols for the JVM. See also issue #39 which contains a few pointers about how to install these.

Disclaimer

I'm not a professional C code writer. The code is very "experimental", and it is e.g. missing checks for error conditions etc.. Use it at your own risk. You have been warned!

License

This library is licensed under GPLv2. See the LICENSE file.

perf-map-agent's Issues

Research using uprobes on JIT'd methods

Challenges:

• uprobes only works with VMA's backed by files while the code cache of the (Hotspot) JVM uses an anonymous region
• uprobes apparently doesn't allow the code memory region to be writable
• dynamic unloading of JIT code requires some process to unregister the probe in time
• it would be nice to access variables in a probe but then you need to know the mapping between Java / Bytecode variables and native registers / stack, the JVM tracks this information for various purposes, but it's probably not exposed in JVMTI

cannot print method defined by user program

Really a newbie question

When I use the agent on a perf record command line and then run perf report, I just see some built-in methods ，like System.out.prinln, but it does not print other methods ,such as , "main, print_rectangle（a user-defining method which just prints some "*"）". I follow the steps on your your blog Java in Flames ( 1. install perf-map-agent , 2. profiling and flame graph generation:).

When I run perf record -F 49 -a -g -- sleep 30; ./FlameGraph/jmaps , I see the perf-5084.map file.
The perf report shows some unknown info:
Samples: 331 of event 'cycles', Event count (approx.): 3811883239
Children Self Command Shared Object Symbol ◆
-- 69.94% 0.00% java [unknown] [.] 0xb3f180e200026c80 ▒
-- 0xb3f180e200026c80 ▒
81.85% 0 ▒
18.15% 0x8000268d8 ▒
-- 69.13% 1.37% java perf-7341.map [.] Ljava/io/FileOutputSt▒
-- Ljava/io/FileOutputStream;::writeBytes ▒
++ 81.75% Ljava/io/PrintStream;::print ▒
++ 18.25% Lsun/nio/cs/StreamEncoder;::flushBuffer ▒
++ 67.76% 0.00% java libjava.so [.] Java_java_io_FileOutp▒
++ 66.79% 0.00% java libpthread-2.17.so [.] 0xffff80d8d849543d ▒
++ 63.03% 6.52% java perf-7341.map [.] Ljava/io/PrintStream;▒
++ 62.09% 0.00% java libjava.so [.] 0xffff80d8dabfe1f0 ▒
++ 56.92% 0.00% java [unknown] [.] 0000000000000000 ▒
++ 56.66% 0.69% java [kernel.kallsyms] [k] sys_write ▒
++ 56.30% 0.00% java [kernel.kallsyms] [k] system_call_fastpath ▒
++ 51.99% 1.85% java [kernel.kallsyms] [k] vfs_write

My java version is 1.8.0.131. And up to date perf-map-agent repo~

Missing method name from flamegraph

I am trying to build flamegraph, flamegraph is getting created successfully, but it is having a lot of methods without name / Generic symbol.

Out put which i get is below.

[aemauthor@local-aem62-test bin]$ ./perf-java-flames 17715
Recording events for 15 seconds (adapt by setting PERF_RECORD_SECONDS)
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.696 MB /tmp/perf-17715.data (618 samples) ]
Failed to open 3E, continuing without symbols
Flame graph SVG written to PERF_FLAME_OUTPUT='/home/aemauthor/perf-map-agent/bin/flamegraph-17715.svg'.

Perf data, method map and Flamegraph are attached.

Archive.zip

Clarify license of shell scripts

The repo as a whole is marked as GPLv2, though the C and java files have an explicit header indicating they're GPL2, but the shell scripts don't.

Are the shell scripts also GPL2?

I'm looking at using these for another JIT (i.e. not using the JIT, just the dtrace + perf map stuff), and considering packaging an 'easy profiling' dev-dependency - but we'd want to use that with projects licensed under licenses such as MIT (e.g. https://github.com/hhvm/hhast/blob/master/LICENSE) or more obscure combinations such as PHP+Zend (e.g. https://github.com/facebook/hhvm/ - parts are taken from PHP itself so we can not use other licenses there)

If they are GPL, we'll likely document how to use them, but avoid distributing them or adding any dependencies on them.

bump up to new version for new release?

The released pre-built binary got slightly old. https://github.com/jvm-profiling-tools/perf-map-agent/releases
For automated usecases, pre-built binary from official repository is helpful (especially in non-tech meaning).

Build is passing, so making new release should be easy.

MacOS problem with bin/create-links-in

My old version of MacOS does not support the readlink -f option. I do not know if newer versions of MacOS support this option.

$ bash -xv bin/create-links-in /usr/local/bin
#!/bin/sh
set -e
+ set -e
#set -x

TARGET_DIR=$1
+ TARGET_DIR=/usr/local/bin
BIN_DIR=`dirname $0`
dirname $0
++ dirname bin/create-links-in
+ BIN_DIR=bin

if [ -z "$TARGET_DIR" ]; then
  echo "Please specify a target directory."
  exit
fi
+ '[' -z /usr/local/bin ']'

ln -s $(readlink -f $BIN_DIR/perf-java-top) $TARGET_DIR
readlink -f $BIN_DIR/perf-java-top
++ readlink -f bin/perf-java-top
readlink: illegal option -- f
usage: readlink [-n] [file ...]
+ ln -s /usr/local/bin
ln: ./bin: File exists

Perf not listing more than 1 line in JVM stack

In one of my cluster, I can attach perf-map agent and get the symbol dump without issue.

I can record "perf record -F 99 -ag" and generate script via "perf script". However, it lists only one line for the JVM stack (the top frame in the stack). JVM is started with "-XX:-OmitStackTraceInFastThrow".

Has anyone observed something like this?

Problem building in Centos 6

When running make I receive the following error:

src/java/AttachOnce.java:25: cannot access com.sun.tools.attach.VirtualMachine
bad class file: com/sun/tools/attach/VirtualMachine.class(com/sun/tools/attach:VirtualMachine.class)
class file has wrong version 52.0, should be 50.0
Please remove or make sure it appears in the correct subdirectory of the classpath.
import com.sun.tools.attach.VirtualMachine;
                           ^

Full output:

$ JAVA_HOME=/usr/java/jdk1.8.0_60/ cmake .
-- JNI_INCLUDE_DIRS=/usr/java/jdk1.8/include;/usr/java/jdk1.8/include/linux;/usr/java/jdk1.8/include
-- JNI_LIBRARIES=/usr/java/jdk1.8/jre/lib/amd64/libjawt.so;/usr/java/jdk1.8/jre/lib/amd64/server/libjvm.so
-- JAVA_INCLUDE_PATH=/usr/java/jdk1.8/include
-- JAVA_INCLUDE_PATH2=/usr/java/jdk1.8/include/linux
-- Configuring done
-- Generating done
-- Build files have been written to: /root/perf-map-agent
$ make
[ 20%] Building Java objects for attach-main.jar
src/java/AttachOnce.java:25: cannot access com.sun.tools.attach.VirtualMachine
bad class file: com/sun/tools/attach/VirtualMachine.class(com/sun/tools/attach:VirtualMachine.class)
class file has wrong version 52.0, should be 50.0
Please remove or make sure it appears in the correct subdirectory of the classpath.
import com.sun.tools.attach.VirtualMachine;
                           ^
make[2]: *** [out/CMakeFiles/attach-main.dir/java_compiled_attach-main] Error 1
make[1]: *** [CMakeFiles/attach-main.dir/all] Error 2
make: *** [all] Error 2
$

I run a strace -fe file make and it is finding tools.jar file:

[pid  2128] open("/usr/java/jdk1.8/include/../lib/tools.jar", O_RDONLY) = 5
src/java/AttachOnce.java:25: cannot access com.sun.tools.attach.VirtualMachine
bad class file: com/sun/tools/attach/VirtualMachine.class(com/sun/tools/attach:VirtualMachine.class)
class file has wrong version 52.0, should be 50.0
Please remove or make sure it appears in the correct subdirectory of the classpath.
import com.sun.tools.attach.VirtualMachine;
                           ^
[pid  2154] +++ exited with 0 +++

Unable to load agent sence Premain-Class manifest attribute is missing

When I try to launch my jar with the agentpath set to attach-main.jar, I get

Failed to find Premain-Class manifest attribute in /perf-map/out/attach-main.jar

Any ideas?

JDK inline info not accurate in all cases

For some methods there can be some chunks of code for which proper inline information seems to be missing (compared to PrintAssembly output).

I don't think there can be done anything about that realistically inside of perf-map-agent but if we find some clues why that would be the case it could be reported to openjdk (maybe it is but I couldn't find it).

For reference, here are the code locations where this info is gathered:

Through the agent/JVMTI interface:

https://github.com/AdoptOpenJDK/openjdk-jdk11/blob/19fb8f93c59dfd791f62d41f332db9e306bc1422/src/hotspot/share/prims/jvmtiExport.cpp#L2093

In contrast, for PrintAssembly:

From here:

https://github.com/AdoptOpenJDK/openjdk-jdk11/blob/3e78ed84f13da94b98f6365935b2192cf0ac2ee0/src/hotspot/share/compiler/disassembler.cpp#L359

to finally:

https://github.com/AdoptOpenJDK/openjdk-jdk11/blob/3e78ed84f13da94b98f6365935b2192cf0ac2ee0/src/hotspot/share/code/nmethod.cpp#L2652

From my quick investigation, the issue might be the way that scopes_pcs_begin reports interesting (= scoped) code regions, in contrast to what scopes are reported if you consider every single PC of an nmethod.

Not able to resolve symbols

Hi

I'm trying to use your project on a fedora 24 using openjdk 1.8.0.101 . The problem can be seen using perf-java-top and flame graphs as well . The create-java-perf-map seems to work well but it did saw how could get its output with perf to plot with frame. Jstack works as well (but as you mention it would be a terrible poor performance and quality).

I thought it would be something with preserving the framepointer but the jdk version is compatible and I'm using the following JVM options:

-XX:+PreserveFramePointer -XX:+StartAttachListener -XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints -XX:-OmitStackTraceInFastThrow -XX:+ShowHiddenFrames

Do you have any suggestions about how to solve this problem?

Thanks in advance
Luan

java 9 support

http://openjdk.java.net/jeps/220

some changes needed not limited to:

the tools.jar has gone. change create-java-perf-map.sh to find the classes via add-modules

(cd perf-map-agent/out/ && java --add-modules jdk.attach -jar ./attach-main.jar $(pidof java ) )

Unable to build with java 8

posix4e@posix4e-B85H3-M:~/projects/perf-map-agent$ cmake .
-- The C compiler identification is GNU 4.8.1
-- The CXX compiler identification is GNU 4.8.1
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Found JNI: /usr/lib/jvm/java-8-oracle/jre/lib/amd64/libjawt.so
-- JNI_INCLUDE_DIRS=/usr/lib/jvm/java-8-oracle/include;/usr/lib/jvm/java-8-oracle/include/linux;/usr/lib/jvm/java-8-oracle/include
-- JNI_LIBRARIES=/usr/lib/jvm/java-8-oracle/jre/lib/amd64/libjawt.so;/usr/lib/jvm/java-8-oracle/jre/lib/amd64/server/libjvm.so
-- Configuring done
-- Generating done
-- Build files have been written to: /home/posix4e/projects/perf-map-agent
posix4e@posix4e-B85H3-M:~/projects/perf-map-agent$ echo $JAVA_HOME
/usr/lib/jvm/java-8-oracle
posix4e@posix4e-B85H3-M:~/projects/perf-map-agent$ export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64/
posix4e@posix4e-B85H3-M:~/projects/perf-map-agent$ set -o vi
posix4e@posix4e-B85H3-M:~/projects/perf-map-agent$ rm -rf cmake_install.cmake  CMakeFiles/ CMakeCache.txt
posix4e@posix4e-B85H3-M:~/projects/perf-map-agent$ cmake .
-- The C compiler identification is GNU 4.8.1
-- The CXX compiler identification is GNU 4.8.1
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Found JNI: /usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/libjawt.so
-- JNI_INCLUDE_DIRS=/usr/lib/jvm/java-7-openjdk-amd64/include;/usr/lib/jvm/java-7-openjdk-amd64/include;/usr/lib/jvm/java-7-openjdk-amd64/include
-- JNI_LIBRARIES=/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/libjawt.so;/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server/libjvm.so
-- Configuring done
-- Generating done
-- Build files have been written to: /home/posix4e/projects/perf-map-agent
posix4e@posix4e-B85H3-M:~/projects/perf-map-agent$ make
Scanning dependencies of target perfmap
[100%] Building C object CMakeFiles/perfmap.dir/perf-map-agent.c.o
/home/posix4e/projects/perf-map-agent/perf-map-agent.c:14:6: warning: conflicting types for ‘open_file’ [enabled by default]
 void open_file() {
      ^
/home/posix4e/projects/perf-map-agent/perf-map-agent.c:11:9: note: previous implicit declaration of ‘open_file’ was here
         open_file();
         ^
/home/posix4e/projects/perf-map-agent/perf-map-agent.c: In function ‘cbCompiledMethodLoad’:
/home/posix4e/projects/perf-map-agent/perf-map-agent.c:46:5: warning: format ‘%lx’ expects argument of type ‘long unsigned int’, but argument 3 has type ‘const void *’ [-Wformat=]
     fprintf(method_file, "%lx %x %s.%s%s\n", code_addr, code_size, csig, name, msig);
     ^
/home/posix4e/projects/perf-map-agent/perf-map-agent.c: In function ‘cbDynamicCodeGenerated’:
/home/posix4e/projects/perf-map-agent/perf-map-agent.c:65:5: warning: format ‘%lx’ expects argument of type ‘long unsigned int’, but argument 3 has type ‘const void *’ [-Wformat=]
     fprintf(method_file, "%lx %x %s\n", address, length, name);
     ^
/home/posix4e/projects/perf-map-agent/perf-map-agent.c: In function ‘Agent_OnLoad’:
/home/posix4e/projects/perf-map-agent/perf-map-agent.c:93:11: warning: incompatible implicit declaration of built-in function ‘memset’ [enabled by default]
     (void)memset(&capabilities,0, sizeof(capabilities));
           ^
Linking C shared library libperfmap.so
[100%] Built target perfmap
posix4e@posix4e-B85H3-M:~/projects/perf-map-agent$ ls
CMakeCache.txt  cmake_install.cmake  libperfmap.so  perf-map-agent.c
CMakeFiles      CMakeLists.txt       Makefile       README.md
posix4e@posix4e-B85H3-M:~/projects/perf-map-agent$ rm -rf cmake_install.cmake  CMakeFiles/ CMakeCache.txt
posix4e@posix4e-B85H3-M:~/projects/perf-map-agent$ cmake .                      -- The C compiler identification is GNU 4.8.1
-- The CXX compiler identification is GNU 4.8.1
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Found JNI: /usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/libjawt.so
-- JNI_INCLUDE_DIRS=/usr/lib/jvm/java-7-openjdk-amd64/include;/usr/lib/jvm/java-7-openjdk-amd64/include;/usr/lib/jvm/java-7-openjdk-amd64/include
-- JNI_LIBRARIES=/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/libjawt.so;/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server/libjvm.so
-- Configuring done
-- Generating done
-- Build files have been written to: /home/posix4e/projects/perf-map-agent
posix4e@posix4e-B85H3-M:~/projects/perf-map-agent$ make                         Scanning dependencies of target perfmap
[100%] Building C object CMakeFiles/perfmap.dir/perf-map-agent.c.o
/home/posix4e/projects/perf-map-agent/perf-map-agent.c:14:6: warning: conflicting types for ‘open_file’ [enabled by default]
 void open_file() {
      ^
/home/posix4e/projects/perf-map-agent/perf-map-agent.c:11:9: note: previous implicit declaration of ‘open_file’ was here
         open_file();
         ^
/home/posix4e/projects/perf-map-agent/perf-map-agent.c: In function ‘cbCompiledMethodLoad’:
/home/posix4e/projects/perf-map-agent/perf-map-agent.c:46:5: warning: format ‘%lx’ expects argument of type ‘long unsigned int’, but argument 3 has type ‘const void *’ [-Wformat=]
     fprintf(method_file, "%lx %x %s.%s%s\n", code_addr, code_size, csig, name, msig);
     ^
/home/posix4e/projects/perf-map-agent/perf-map-agent.c: In function ‘cbDynamicCodeGenerated’:
/home/posix4e/projects/perf-map-agent/perf-map-agent.c:65:5: warning: format ‘%lx’ expects argument of type ‘long unsigned int’, but argument 3 has type ‘const void *’ [-Wformat=]
     fprintf(method_file, "%lx %x %s\n", address, length, name);
     ^
/home/posix4e/projects/perf-map-agent/perf-map-agent.c: In function ‘Agent_OnLoad’:
/home/posix4e/projects/perf-map-agent/perf-map-agent.c:93:11: warning: incompatible implicit declaration of built-in function ‘memset’ [enabled by default]
     (void)memset(&capabilities,0, sizeof(capabilities));
           ^
Linking C shared library libperfmap.so
[100%] Built target perfmap
posix4e@posix4e-B85H3-M:~/projects/perf-map-agent$ export JAVA_HOME=/usr/lib/jvm/java-8-oracle/
posix4e@posix4e-B85H3-M:~/projects/perf-map-agent$ rm -rf cmake_install.cmake  CMakeFiles/ CMakeCache.txt
posix4e@posix4e-B85H3-M:~/projects/perf-map-agent$ cmake .                      -- The C compiler identification is GNU 4.8.1
-- The CXX compiler identification is GNU 4.8.1
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Found JNI: /usr/lib/jvm/java-8-oracle/jre/lib/amd64/libjawt.so
-- JNI_INCLUDE_DIRS=/usr/lib/jvm/java-8-oracle/include;/usr/lib/jvm/java-8-oracle/include/linux;/usr/lib/jvm/java-8-oracle/include
-- JNI_LIBRARIES=/usr/lib/jvm/java-8-oracle/jre/lib/amd64/libjawt.so;/usr/lib/jvm/java-8-oracle/jre/lib/amd64/server/libjvm.so
-- Configuring done
-- Generating done
-- Build files have been written to: /home/posix4e/projects/perf-map-agent
posix4e@posix4e-B85H3-M:~/projects/perf-map-agent$ make                         Scanning dependencies of target perfmap
[100%] Building C object CMakeFiles/perfmap.dir/perf-map-agent.c.o
In file included from /home/posix4e/projects/perf-map-agent/perf-map-agent.c:1:0:
/usr/lib/jvm/java-8-oracle/include/jni.h:45:20: fatal error: jni_md.h: No such file or directory
 #include "jni_md.h"
                    ^
compilation terminated.
make[2]: *** [CMakeFiles/perfmap.dir/perf-map-agent.c.o] Error 1
make[1]: *** [CMakeFiles/perfmap.dir/all] Error 2
make: *** [all] Error 2
posix4e@posix4e-B85H3-M:~/projects/perf-map-agent$ ls9390

unfold reports wrong last method (or may even crash)

As discovered by @nitsanw. Proposed fix in #21.

Can make in CentOS

Hi
I have ran cmake .
but show me as below
do you know what it is happened?
finished or unfinished?

   -- The C compiler identification is GNU
-- The CXX compiler identification is GNU
-- Check for working C compiler: /usr/bin/gcc
-- Check for working C compiler: /usr/bin/gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
CMake Error at CMakeLists.txt:24 (include):
  include could not find load file:

    UseJava


CMake Error at CMakeLists.txt:30 (add_jar):
  Unknown CMake command "add_jar".


-- Configuring incomplete, errors occurred!

Failed to load agent library

Hi guys,

I used Java(TM) SE Runtime Environment (build 1.8.0_60-ea-b25)
and run ./create-java-perf-map.sh $pid

got:

Exception in thread "main" com.sun.tools.attach.AgentLoadException: Failed to load agent library
at sun.tools.attach.LinuxVirtualMachine.execute(LinuxVirtualMachine.java:224)
at sun.tools.attach.HotSpotVirtualMachine.loadAgentLibrary(HotSpotVirtualMachine.java:58)
at sun.tools.attach.HotSpotVirtualMachine.loadAgentPath(HotSpotVirtualMachine.java:88)
at net.virtualvoid.perf.AttachOnce.loadAgent(AttachOnce.java:39)
at net.virtualvoid.perf.AttachOnce.main(AttachOnce.java:33)

Any idea?

Thanks so much.

Some C stack frames with Java ancestor not correctly shown

HI guys!

I have produced a flame graph with:

PERF_COLLAPSE_OPTS="--kernel --tid" PERF_RECORD_FREQ=99 PERF_RECORD_SECONDS=10 PERF_MAP_OPTIONS=unfoldall ~/perf-map-agent/bin/perf-java-flames <pid>

But seems that (some) C stack frames are not correctly shown as children of the related Java calls.
Am I missing anything in the configuration?

Thanks,
Franz

Resolve `perf-map-agent` directory better

Thanks for your small but very useful tool. I've been working with the latest commits from master on the tool and I hit some issues mainly related to how the paths are solved and the main JAR file is executed. Here is a suggest to improve a bit for simpler usage:

diff --git a/bin/create-java-perf-map.sh b/bin/create-java-perf-map.sh
index 604fa44..b83f342 100755
--- a/bin/create-java-perf-map.sh
+++ b/bin/create-java-perf-map.sh
@@ -6,8 +6,7 @@ CUR_DIR=`pwd`
 PID=$1
 OPTIONS=$2
 ATTACH_JAR=attach-main.jar
-PERF_MAP_DIR=$(dirname $(readlink -f $0))/..
-ATTACH_JAR_PATH=$PERF_MAP_DIR/out/$ATTACH_JAR
+PERF_MAP_DIR=$(realpath $(dirname $(readlink -f $0))/..)
 PERF_MAP_FILE=/tmp/perf-$PID.map

 if [ -z $JAVA_HOME ]; then
@@ -18,5 +17,5 @@ fi
 [ -d $JAVA_HOME ] || (echo "JAVA_HOME directory at '$JAVA_HOME' does not exist." && false)

 sudo rm $PERF_MAP_FILE -f
-(cd $PERF_MAP_DIR && java -cp $ATTACH_JAR_PATH:$JAVA_HOME/lib/tools.jar net.virtualvoid.perf.AttachOnce $PID $OPTIONS)
+(cd $PERF_MAP_DIR/out && java -cp $ATTACH_JAR:$JAVA_HOME/lib/tools.jar net.virtualvoid.perf.AttachOnce $PID $OPTIONS)
 sudo chown root:root $PERF_MAP_FILE
diff --git a/bin/perf-java-flames b/bin/perf-java-flames
index eb7c235..4a3d4ee 100755
--- a/bin/perf-java-flames
+++ b/bin/perf-java-flames
@@ -26,7 +26,7 @@ if [ -z $PERF_DATA_FILE ]; then
 fi

 if [ -z $PERF_FLAME_OUTPUT ]; then
-  PERF_FLAME_OUTPUT=flamegraph-$PID.svg
+  PERF_FLAME_OUTPUT=$PERF_JAVA_TMP/flamegraph-$PID.svg
 fi

 $PERF_MAP_DIR/bin/perf-java-record-stack $*

Summary:

• Remove ATTACH_JAR_PATH.
• Use realpath for more correctly resolve the base directory of perf-map-agent.
• Default PERF_FLAME_OUTPUT to $PERF_JAVA_TMP/flamegraph-$PID.svg if not already set.

include build info in out directory

I've found that perf-map-agent compiled for OracleJDK can crash OpenJDK (usually on the second invocation). I'm changing my tooling so that we have builds for each. This ticket is to suggest that build info is included in the out directory, so that other tooling can double check what it was built against. Eg, just a:

echo JAVA_HOME=$JAVA_HOME > out/build.txt

From the build process would be handy.

For reference, here's stack trace from one of the crashes:

Core was generated by `java -Xms24952m -Xmx24952m -XX:MaxMetaspaceSize=512m -XX:+HeapDumpOnOutOfMemory'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007fe7a6d0c428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
54	../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
[Current thread is 1 (Thread 0x7fe7253fb700 (LWP 5078))]
Installing openjdk unwinder
(gdb) bt
#0  0x00007fe7a6d0c428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
#1  0x00007fe7a6d0e02a in __GI_abort () at abort.c:89
#2  0x00007fe7a6012db9 in os::abort(bool) (dump_core=<optimized out>) at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/os/linux/vm/os_linux.cpp:1509
#3  0x00007fe7a61c6897 in VMError::report_and_die() (this=this@entry=0x7fe7253f7e20)
    at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/share/vm/utilities/vmError.cpp:1060
#4  0x00007fe7a61c6fb7 in crash_handler(int, siginfo_t*, void*) (sig=11, info=0x7fe7253f8070, ucVoid=0x7fe7253f7f40)
    at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/os/linux/vm/vmError_linux.cpp:106
#5  0x00007fe7a6d0c4b0 in <signal handler called> () at /lib/x86_64-linux-gnu/libc.so.6
#6  0x00007fe7a6e1a287 in __GI__dl_addr (symbolp=0x0, mapp=0x0, info=0x7fe7253f83f0, match=0x7fe6f0001800, addr=140630735942332) at dl-addr.c:61
#7  0x00007fe7a6e1a287 in __GI__dl_addr (address=0x7fe7250f72bc, info=0x7fe7253f83f0, mapp=0x0, symbolp=0x0) at dl-addr.c:137
#8  0x00007fe7a6018b49 in os::find(unsigned char*, outputStream*) (addr=addr@entry=0x7fe7250f72bc "\205W\255Ox\361\030\233\067O\200\262\266\067\071_\321\357\335+\204\360\344_\223_", st=st@entry=0x7fe7253f8b90) at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/os/linux/vm/os_linux.cpp:5166
#9  0x00007fe7a600d61d in os::print_location(outputStream*, long, bool) (st=0x7fe7253f8b90, x=140630735942332, verbose=<optimized out>)
    at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/share/vm/runtime/os.cpp:1030
#10 0x00007fe7a601cfbb in os::print_register_info(outputStream*, void*) (st=0x7fe7253f8b90, context=0x7fe7253f8e00)
    at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/os_cpu/linux_x86/vm/os_linux_x86.cpp:852
#11 0x00007fe7a61c557a in VMError::report(outputStream*) (this=0x7fe7253f8c90, st=st@entry=0x7fe7253f8b90)
    at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/share/vm/utilities/vmError.cpp:542
#12 0x00007fe7a61c63eb in VMError::report_and_die() (this=this@entry=0x7fe7253f8c90)
    at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/share/vm/utilities/vmError.cpp:971
#13 0x00007fe7a601c29f in JVM_handle_linux_signal(int, siginfo_t*, void*, int) (sig=sig@entry=11, info=info@entry=0x7fe7253f8f30, ucVoid=ucVoid@entry=0x7fe7253f8e00, abort_if_unrecognized=abort_if_unrecognized@entry=1) at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/os_cpu/linux_x86/vm/os_linux_x86.cpp:541
#14 0x00007fe7a600fef8 in signalHandler(int, siginfo_t*, void*) (sig=11, info=0x7fe7253f8f30, uc=0x7fe7253f8e00)
    at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/os/linux/vm/os_linux.cpp:4346
#15 0x00007fe7a6d0c4b0 in <signal handler called> () at /lib/x86_64-linux-gnu/libc.so.6
#16 0x00007fe7a72b9cda in check_match (undef_name=undef_name@entry=0x7fe6f00076b0 "Agent_OnAttach", ref=ref@entry=0x0, version=version@entry=0x0, flags=flags@entry=2, type_class=type_class@entry=0, sym=sym@entry=0x6f8, symidx=44, strtab=0x7a0 <error: Cannot access memory at address 0x7a0>, map=0x7fe6f0001800, versioned_sym=0x7fe7253f9338, num_versions=0x7fe7253f9334) at dl-lookup.c:92
#17 0x00007fe7a72ba64b in do_lookup_x (undef_name=undef_name@entry=0x7fe6f00076b0 "Agent_OnAttach", new_hash=new_hash@entry=1336760197, old_hash=old_hash@entry=0x7fe7253f9400, ref=0x0, result=result@entry=0x7fe7253f9410, scope=<optimized out>, i=0, version=0x0, flags=2, skip=0x0, type_class=0, undef_map=0x7fe6f0001800) at dl-lookup.c:423
#18 0x00007fe7a72bab1f in _dl_lookup_symbol_x (undef_name=0x7fe6f00076b0 "Agent_OnAttach", undef_map=0x7fe6f0001800, ref=0x7fe7253f94e8, symbol_scope=0x7fe6f0001b88, version=0x0, type_class=0, flags=2, skip_map=0x0) at dl-lookup.c:829
#19 0x00007fe7a6e1a873 in do_sym (handle=0x7fe6f0001800, name=0x7fe6f00076b0 "Agent_OnAttach", who=<optimized out>, vers=vers@entry=0x0, flags=flags@entry=2) at dl-sym.c:168
#20 0x00007fe7a6e1ad1d in _dl_sym (handle=<optimized out>, name=<optimized out>, who=<optimized out>) at dl-sym.c:273
#21 0x00007fe7a68ba034 in dlsym_doit (a=a@entry=0x7fe7253f96f0) at dlsym.c:50
#22 0x00007fe7a72c0564 in _dl_catch_error (objname=0x7fe6f00017c0, errstring=0x7fe6f00017c8, mallocedp=0x7fe6f00017b8, operate=0x7fe7a68ba020 <dlsym_doit>, args=0x7fe7253f96f0) at dl-error.c:187
#23 0x00007fe7a68ba571 in _dlerror_run (operate=operate@entry=0x7fe7a68ba020 <dlsym_doit>, args=args@entry=0x7fe7253f96f0) at dlerror.c:163
#24 0x00007fe7a68ba088 in __dlsym (handle=handle@entry=0x7fe6f0001800, name=name@entry=0x7fe6f00076b0 "Agent_OnAttach") at dlsym.c:70
#25 0x00007fe7a60138f4 in os::dll_lookup(void*, char const*) (handle=handle@entry=0x7fe6f0001800, name=name@entry=0x7fe6f00076b0 "Agent_OnAttach")
    at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/os/linux/vm/os_linux.cpp:2052
#26 0x00007fe7a600c34c in os::find_agent_function(AgentLibrary*, bool, char const**, unsigned long) (agent_lib=agent_lib@entry=0x7fe6f00079a0, check_lib=check_lib@entry=false,---Type <return> to continue, or q <return> to quit--- 
 syms=syms@entry=0x7fe7253f9860, syms_len=syms_len@entry=1) at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/share/vm/runtime/os.cpp:490
#27 0x00007fe7a5e8f5f3 in JvmtiExport::load_agent_library(AttachOperation*, outputStream*) (op=0x7fe6f0000a30, st=0x7fe7253fad20)
    at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/share/vm/prims/jvmtiExport.cpp:2252
#28 0x00007fe7a5a43db1 in attach_listener_thread_entry(JavaThread*, Thread*) (thread=<optimized out>, __the_thread__=<optimized out>)
    at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/share/vm/services/attachListener.cpp:453
#29 0x00007fe7a616aa07 in JavaThread::thread_main_inner() (this=this@entry=0x7fe708001000)
    at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/share/vm/runtime/thread.cpp:1699
#30 0x00007fe7a616aecc in JavaThread::run() (this=0x7fe708001000) at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/share/vm/runtime/thread.cpp:1679
#31 0x00007fe7a6011d82 in java_start(Thread*) (thread=0x7fe708001000) at /build/openjdk-8-VTMhfL/openjdk-8-8u131-b11/src/hotspot/src/os/linux/vm/os_linux.cpp:790
#32 0x00007fe7a66a36ba in start_thread (arg=0x7fe7253fb700) at pthread_create.c:333
#33 0x00007fe7a6dde3dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

option to disable full method signatures

Would like an option to disable full method signatures. Eg, "nomsig". So this:

7feca143ba60 164c0 Lio/netty/handler/codec/http/DefaultHttpHeaders;.set(Ljava/lang/CharSequence;Ljava/lang/Object;)Lio

would become:

7feca143ba60 164c0 Lio/netty/handler/codec/http/DefaultHttpHeaders;.set

annotate a JITted method

Possible newbie question...

When I use the agent on a perf record command line and then run perf report, I do indeed see the JITted methods listed with event percentages, etc. But when I hit RETURN and try to drill down into a JITted method, no code is shown.

Lack of perf PMU event

Hi,

It's impossible to record plain PMU events like mispredicted branches with perf-java-recors-stack. This script supports profiling with specified frequency only (-F argument).

I'd like to propose either add one more script (like perf-java-record-stacks) which is able to record PMU events or modify existing perf-java-record-stack script in order to let user to specify PMU event to profile. If you are ok for these changes, I would implement it.

--Sergey

Allow attaching to running JVMs

• properly handle attaching
• add user program to control attaching
• (maybe fix the permission issue this way: if the program controlling the attachment is run as root it may be possible to create the file and share a descriptor for that file with enough permissions with the agent)

unfold reports inlined methods off-by-one

As discovered by @nitsanw. Proposed fix in #21.

incorrect method resolution?

Hi,

I'm seeing some methods being unexpectedly listed high on perf top, which are called very infrequently. I have cross-checked with a JMC flight recording (which I assume to be quite accurate), and these paths were not showing up there at all.

Is there a chance that the map is somehow inaccurate, perhaps over time with hotspot recompiling / deoptimizing / moving stuff around?

Btw, this agent is quite useful, keep up the good work :)

Regards,
Viktor

Document how to run perf-map-agent against Docker containers

Hi team,

We have our application running inside a container. If thats the case the map-agent should be installed outside or inside the container. I presume it should be done inside the container because the jdk we use is referenced inside. Could you please review and let us know the ideal approach.

Sattish.

Document or improve scripts to make perf-java-flames work with docker

I'm trying to create a Java process FlameGraph with perf-java-flames. It seems to run successfully, but I can't find the resulting svg file.

$ ./perf-java-flames 161991 -F 99 -a -g -- sleep 30
Recording events for 15 seconds (adapt by setting PERF_RECORD_SECONDS)
Warning:
PID/TID switch overriding SYSTEM
$

CentOS 7
3.10.0-327.36.3.el7.x86_64
cmake version 2.8.12.2

Up-to-date versions of perf-map-agent and the FlameGraph repo.

Interpreter frames

I often see "Interpreter" frames. I think I was expecting more symbols.

   100.00%     java  perf-20844.map  [.] Interpreter
               |
               --- Interpreter
                   Interpreter
                   Interpreter
                   Interpreter
                   call_stub
                   JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)
                   jni_invoke_static(JNIEnv_*, JavaValue*, _jobject*, JNICallType, _jmethodID*, JNI_ArgumentPusher*, Thread*)
                   jni_CallStaticVoidMethod
                   JavaMain
                   start_thread

CMake issue

Hi
I have ran cmake. Some error happened. Do you have any idea?

-- JNI_INCLUDE_DIRS=/usr/lib/jvm/java-7-openjdk-amd64/include;/usr/lib/jvm/java-7-openjdk-amd64/include;/usr/lib/jvm/java-7-openjdk-amd64/include
-- JNI_LIBRARIES=/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/libjawt.so;/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server/libjvm.so
-- JAVA_INCLUDE_PATH=/usr/lib/jvm/java-7-openjdk-amd64/include
-- JAVA_INCLUDE_PATH2=/usr/lib/jvm/java-7-openjdk-amd64/include
CMake Error at /usr/share/cmake-2.8/Modules/UseJava.cmake:269 (string):
string sub-command LENGTH requires two arguments.
Call Stack (most recent call first):
CMakeLists.txt:30 (add_jar)

-- Configuring incomplete, errors occurred!

when do i have to create the map

The perf map should be created before or after the per record/top commands?

'jvmticmlr.h' file not found

➜  perf-map-agent git:(master) ✗ make
-- JNI_INCLUDE_DIRS=/System/Library/Frameworks/JavaVM.framework/Headers;/System/Library/Frameworks/JavaVM.framework/Headers;/System/Library/Frameworks/JavaVM.framework/Headers
-- JNI_LIBRARIES=/System/Library/Frameworks/JavaVM.framework;/System/Library/Frameworks/JavaVM.framework
-- JAVA_INCLUDE_PATH=/System/Library/Frameworks/JavaVM.framework/Headers
-- JAVA_INCLUDE_PATH2=/System/Library/Frameworks/JavaVM.framework/Headers
-- Configuring done
CMake Warning (dev):
  Policy CMP0042 is not set: MACOSX_RPATH is enabled by default.  Run "cmake
  --help-policy CMP0042" for policy details.  Use the cmake_policy command to
  set the policy and suppress this warning.

  MACOSX_RPATH is not specified for the following targets:

   perfmap

This warning is for project developers.  Use -Wno-dev to suppress it.

-- Generating done
-- Build files have been written to: /Users/xiezhaodong/Downloads/perf-profile/perf-map-agent
[ 16%] Building Java objects for attach-main.jar
[ 33%] Generating out/CMakeFiles/attach-main.dir/java_class_filelist
[ 50%] Creating Java archive attach-main.jar
[ 50%] Built target attach-main
[ 66%] Building C object CMakeFiles/perfmap.dir/src/c/perf-map-agent.c.o
/Users/xiezhaodong/Downloads/perf-profile/perf-map-agent/src/c/perf-map-agent.c:30:10: fatal error: 'jvmticmlr.h' file not found
#include <jvmticmlr.h>
         ^
1 error generated.
make[2]: *** [CMakeFiles/perfmap.dir/src/c/perf-map-agent.c.o] Error 1
make[1]: *** [CMakeFiles/perfmap.dir/all] Error 2
make: *** [all] Error 2

I change CMakelist.text line set(CMAKE_JAVA_INCLUDE_PATH /Library/Java/JavaVirtualMachines/jdk1.7.0_79.jdk/Contents/Home/lib/tools.jar)

Could not find or load main class net.virtualvoid.perf.AttachOnce

# ./bin/perf-java-top 19835
Error: Could not find or load main class net.virtualvoid.perf.AttachOnce

scripts expects attach-main.jar to be found in out dir, while it is in the project root dir.

Report missing tools.jar in build and run script instead of failing with NoClassDefFoundError

Hello,

After succesfully creating flamegraphs a few days ago I tried it again today and only get this error:

Error: A JNI error has occurred, please check your installation and try again Exception in thread "main" java.lang.NoClassDefFoundError: com/sun/tools/attach/AgentInitializationException at java.lang.Class.getDeclaredMethods0(Native Method) at java.lang.Class.privateGetDeclaredMethods(Class.java:2701) at java.lang.Class.privateGetMethodRecursive(Class.java:3048) at java.lang.Class.getMethod0(Class.java:3018) at java.lang.Class.getMethod(Class.java:1784) at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544) at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526) Caused by: java.lang.ClassNotFoundException: com.sun.tools.attach.AgentInitializationException at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 7 more

I haven't changed anything on the machine. Is there a solution to this or a way to get more error information?

High CPU time in vtable chunks

During our code profiling tests on 16-core systems running 8-16 worker threads, we are observing a significant CPU time (25%) in "vtable chunks" based on the generated symbol map. I think this symbol is listed multiple times as well in the symbol map. This overhead also increases as the number of running threads is gradually raised from 8 to 16.

Is this an overhead of perf-map-agent or something else ? Would like to also confirm if its related to virtual methods in our code.

Another unrelated issue is that perf top always displays "unable to open vsyscalls". Is there a way to resolve it. Does this mean that perf is unable to display system calls and their CPU time ? All processes are being executed by the "root" user.

Generating mappings for transient java processes / agentlib mode

Hi all,
I'm trying to profile systems that consist of lots of java processes being started and stopped over time. Does anyone know of a way to ensure that I can get a symbol mapping of each java process that gets run on my system? Just running once won't capture these transient processes, so I'd ideally like something that automatically makes a symbol dump, maybe on process termination? Any thoughts would be appreciate.

Not sure if this is the right place to ask, feel free to point me somewhere else!
Thanks a lot for this tool.

cmake error(cmake version is 2.8.12.2)

i installed cmake, perf, and cmake version is 2.8.12.2, but failed

[root@test perf-map-agent]# cmake .

-- Could NOT find JNI (missing:  JAVA_AWT_LIBRARY JAVA_JVM_LIBRARY JAVA_INCLUDE_PATH JAVA_INCLUDE_PATH2 JAVA_AWT_INCLUDE_PATH)
CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
JAVA_INCLUDE_PATH (ADVANCED)
   used as include directory in directory /root/perf-map-agent
   used as include directory in directory /root/perf-map-agent
JAVA_INCLUDE_PATH2 (ADVANCED)
   used as include directory in directory /root/perf-map-agent
   used as include directory in directory /root/perf-map-agent

-- Configuring incomplete, errors occurred!
See also "/root/perf-map-agent/CMakeFiles/CMakeOutput.log".
See also "/root/perf-map-agent/CMakeFiles/CMakeError.log".

CMakeError.log

Compiling the CXX compiler identification source file "CMakeCXXCompilerId.cpp" failed.
Compiler: CMAKE_CXX_COMPILER-NOTFOUND
Build flags:
Id flags:

The output was:
No such file or directory


Compiling the CXX compiler identification source file "CMakeCXXCompilerId.cpp" failed.
Compiler: CMAKE_CXX_COMPILER-NOTFOUND
Build flags:
Id flags: -c

The output was:
No such file or directory


Checking whether the CXX compiler is IAR using "" did not match "IAR .+ Compiler":
Compiling the CXX compiler identification source file "CMakeCXXCompilerId.cpp" failed.
Compiler: CMAKE_CXX_COMPILER-NOTFOUND
Build flags:
Id flags:

The output was:
No such file or directory


Compiling the CXX compiler identification source file "CMakeCXXCompilerId.cpp" failed.
Compiler: CMAKE_CXX_COMPILER-NOTFOUND
Build flags:
Id flags: -c

The output was:
No such file or directory


Checking whether the CXX compiler is IAR using "" did not match "IAR .+ Compiler":

run perf script fail

hello!
i have a question about perf
i ran the command as below

perf script|./stackcollapse-perf.pl |./flamegraph.pl --color=java --hash > flamegraph1.svg

system show me

Failed to open /usr/lib64/libc-2.17.so;56dd2155, continuing without symbols
Failed to open /usr/lib64/libpthread-2.17.so;56dd2155, continuing without symbols

os:CentOS Linux release 7.2.1511 (Core)
cmake:cmake version 2.8.11

Partial Stacks

This is probably a JVM issue, but I'll file it here anyway as it's something anyone may run into when using perf-map-agent.

Many stacks that perf reports look partial/incomplete, showing the top frame only. Eg, the following output is from "perf report --stdio":

     0.78%     java  perf-22919.map      [.] Lio/netty/handler/codec/http/DefaultHttpHeaders;.add(Ljava/lang/CharSequence;Ljava/lang/Object;)Lio/netty/handler/codec/
               |
               --- Lio/netty/handler/codec/http/DefaultHttpHeaders;.add(Ljava/lang/CharSequence;Ljava/lang/Object;)Lio/netty/handler/codec/http/HttpHeaders;

     0.78%     java  perf-22919.map      [.] Lrx/subjects/PublishSubject;.onNext(Ljava/lang/Object;)V
               |
               --- Lrx/subjects/PublishSubject;.onNext(Ljava/lang/Object;)V

     0.78%     java  perf-22919.map      [.] Lio/netty/channel/AbstractChannel$AbstractUnsafe;.write(Ljava/lang/Object;Lio/netty/channel/ChannelPromise;)V
               |
               --- Lio/netty/channel/AbstractChannel$AbstractUnsafe;.write(Ljava/lang/Object;Lio/netty/channel/ChannelPromise;)V

It looks a lot like omit-frame-pointer optimization, however, I believe this is from hotspot. If there is a hotspot equivalent option for -fno-omit-frame-pointer, I haven't found it yet.

I am using:

-XX:-OmitStackTraceInFastThrow -XX:+UnlockDiagnosticVMOptions -XX:+ShowHiddenFrames 

Improve reporting of inlined frames

I think PMA is in a unique position to break new grounds in Java profiling by reporting the Java stack while highlighting the separation between real frames and virtual frames(inlined methods). To do this we will need:

1. Report all frames when reporting inlined data: This can easily lead to VERY long strings. I'm not sure what the maximum 'method' name length supported by perf is.
2. Support from flame graphs in colouring and displaying the frames.

make will not build perf-map-agent: fatal error: 'error.h' file not found

$ make

[ 50%] Built target attach-main
[ 66%] Building C object CMakeFiles/perfmap.dir/src/c/perf-map-file.c.o
/Users/justinthomas/tools/perf-map-agent/src/c/perf-map-file.c:24:10: fatal error: 'error.h' file not found
#include <error.h>
         ^
1 error generated.
make[2]: *** [CMakeFiles/perfmap.dir/src/c/perf-map-file.c.o] Error 1
make[1]: *** [CMakeFiles/perfmap.dir/all] Error 2
make: *** [all] Error 2

add troubleshooting section

There is a known issues section, but it is reasonably easy to not generate the symbols for a number of reasons.
These problems should be fairly easy to deal with if documented in troubleshooting steps/procedure.
Sure they will be documented in the issues but not everyone reads the issues and then if needed wastes a roundtrip as we ask for diagnostic information.

perf script using perf-java-report-stack output will get "Misordered timestamps: 0"

jdk: OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)
cmake: version 2.8.12.2
perf: version 3.10.0-693.11.1.el7.x86_64.debug
OS: centOS 7

After using perf-java-record-stack <pid> get output in \tmp\perf-<pid>.data, but using perf script -i /tmp/perf-<pid>.data -d will result in Misordered timestamps: 0. If perf script not run in debug mod, nothing happens and got an empty stack file.

perf-java-top <pid> and perf-java-report-stack <pid> works as expected, but sometimes fail with empty sample, I guess that is normal.

I try this in several machine I've got, all the same.

Attached my .data and .map file.
perf-11484.zip

(still) unable to resolve JIT symbols

Hi Johannes,

I'm trying to profile a Java program (SPECjbb2015) using linux perf + your perf-map-agent.

• java version "1.8.0_111"
  Java(TM) SE Runtime Environment (build 1.8.0_111-b14)
  Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)
• perf-map-agent is compiled with JAVA_HOME set to the directory of
  the above java (i.e. java is ${JAVA_HOME}/bin/java )
• Lunux version is 4.4.0-45 (Ubuntu)
• -XX:+PreserveFramePointer is used with java invocation
• /opt/perf-map-agent-master/bin/create-java-perf-map.sh $pid is invoked right
  after Java invocation where $pid is the pid of java
• the symbols from Java runtime seem to be OK, as I can see lines like follows:
  6.44% 201751 java libjvm.so [.] PSPromotionManager::copy_to_survivor_space
• Most unresolved symbols are from perf-$pid-map, but still few came from libXXX (e.g. libc)
  or vdso
• Profiling 'cycles' events, the aggregated number of samples for unresolved symbols
  take about 55% to 60% of the total samples.
• Since JRE symbols are resolved, unresolved symbols seem to come from SPECjbb' jar file.
  I recompiled it with the following options to pom.xml

-verbose -g -Xlint:all,-options,-path which inflated the jar size from 1.8MB to 9.1MB However, the results looked the same ( was already true in the original pom.xml).

Do you think of any (possible) mistake I'm making ?
Or any options I should try ?
Thank you very much
Hitoshi

time to generate map file

Hello, Authors,
Thank you to open-source this projects. The agent and handy scripts are extremely helpful for me.

I have a problem about JIT-ed code. I end up with a lot of PCs which point to nowhere but fall in the range of map files. I interpret it in this way: the symbols in the map file are not accurate.

I have a question about this line. It seems that two statements happen in sequence.

perf-map-agent/bin/perf-java-record-stack

Line 25 in d9843a0

sudo perf record -F $PERF_RECORD_FREQ -o $PERF_DATA_FILE -g -p $* -- sleep $PERF_RECORD_SECONDS

That is to say, the script generates map file right after 'perf record' has done. Why not make them run in parallel? In current model, will agent has a skewed view? or I have to make sure my profiling program enter a stable state ?

Improve support for JSVC

Update scripts to detect jsvc process in addition to java. I may propose a patch but this will take me a few days.

perf-map-agent as jvmti agent on start

Hi,

I tried to use perf-map-agent as pure JVMTI agent via -agentlib argument. But the current implementation doesn't have Agent_OnLoad method.

So, does it worth to implement Agen_OnLoad function (I'm ready to do it)? If so, should I handle a situation when compiled method was abandoned (is it possible to handle these situations?)?

--Sergey

Cannot build on OS X El Capitan

Hi,

I can't seem to build most of the targets on El Capitan.

$ cmake --version
cmake version 3.5.20160411-g9ac11

CMake suite maintained and supported by Kitware (kitware.com/cmake)

$ cmake .
-- JNI_INCLUDE_DIRS=/System/Library/Frameworks/JavaVM.framework/Headers;/System/Library/Frameworks/JavaVM.framework/Headers;/System/Library/Frameworks/JavaVM.framework/Headers
-- JNI_LIBRARIES=/System/Library/Frameworks/JavaVM.framework;/System/Library/Frameworks/JavaVM.framework
-- JAVA_INCLUDE_PATH=/System/Library/Frameworks/JavaVM.framework/Headers
-- JAVA_INCLUDE_PATH2=/System/Library/Frameworks/JavaVM.framework/Headers
-- LIBS: /Library/Java/JavaVirtualMachines/jdk1.8.0_77.jdk/Contents/Home/bin/jar  incl: /System/Library/Frameworks/JavaVM.framework/Headers/../lib/tools.jar output_dir: /Users/username/code/perf-map-agent/out
-- Configuring done
CMake Warning (dev):
  Policy CMP0042 is not set: MACOSX_RPATH is enabled by default.  Run "cmake
  --help-policy CMP0042" for policy details.  Use the cmake_policy command to
  set the policy and suppress this warning.

  MACOSX_RPATH is not specified for the following targets:

   perfmap

This warning is for project developers.  Use -Wno-dev to suppress it.

-- Generating done
-- Build files have been written to: /Users/username/code/perf-map-agent

$ make --debug
GNU Make 3.81
Copyright (C) 2006  Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.

This program built for i386-apple-darwin11.3.0
Reading makefiles...
Updating goal targets....
 File `default_target' does not exist.
   File `all' does not exist.
     File `cmake_check_build_system' does not exist.
    Must remake target `cmake_check_build_system'.
    Successfully remade target file `cmake_check_build_system'.
  Must remake target `all'.
GNU Make 3.81
Copyright (C) 2006  Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.

This program built for i386-apple-darwin11.3.0
Reading makefiles...
Updating goal targets....
 File `all' does not exist.
   File `CMakeFiles/attach-main.dir/all' does not exist.
  Must remake target `CMakeFiles/attach-main.dir/all'.
GNU Make 3.81
Copyright (C) 2006  Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.

This program built for i386-apple-darwin11.3.0
Reading makefiles...
Updating goal targets....
 File `CMakeFiles/attach-main.dir/depend' does not exist.
Must remake target `CMakeFiles/attach-main.dir/depend'.
Successfully remade target file `CMakeFiles/attach-main.dir/depend'.
GNU Make 3.81
Copyright (C) 2006  Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.

This program built for i386-apple-darwin11.3.0
Reading makefiles...
Updating goal targets....
 File `CMakeFiles/attach-main.dir/build' does not exist.
   File `attach-main' does not exist.
     File `CMakeFiles/attach-main' does not exist.
       File `out/attach-main.jar' does not exist.
           File `out/CMakeFiles/attach-main.dir/java_compiled_attach-main' does not exist.
          Must remake target `out/CMakeFiles/attach-main.dir/java_compiled_attach-main'.
[ 16%] Building Java objects for attach-main.jar
src/java/AttachOnce.java:25: error: package com.sun.tools.attach does not exist
import com.sun.tools.attach.VirtualMachine;
                           ^
src/java/AttachOnce.java:37: error: cannot find symbol
        VirtualMachine vm = VirtualMachine.attach(pid);
        ^
  symbol:   class VirtualMachine
  location: class AttachOnce
src/java/AttachOnce.java:37: error: cannot find symbol
        VirtualMachine vm = VirtualMachine.attach(pid);
                            ^
  symbol:   variable VirtualMachine
  location: class AttachOnce
src/java/AttachOnce.java:46: error: package com.sun.tools.attach does not exist
        } catch(com.sun.tools.attach.AgentInitializationException e) {
                                    ^
4 errors
make[2]: *** [out/CMakeFiles/attach-main.dir/java_compiled_attach-main] Error 1
make[1]: *** [CMakeFiles/attach-main.dir/all] Error 2
make: *** [all] Error 2

Annotate methods in flamegraphs with info from JIT compiler

I'm filing it here in lack of a better place so that I don't forget about it.

E.g. if a frame couldn't be inlined because it hit MaxInlineDepth, that info could be somehow annotated graphically in the flame graph. Similar for other inlining decisions.

The tool needed would take the output of -XX:+LogCompilation to produce annotations. Flamegraph would need to be able to merge those annotations into the flamegraph.

./bin/dtrace-java-record-stack is not working

Ran ./bin/dtrace-java-record-stack but the command does not work. I think the dtrace command inside the ./bin/dtrace-java-record-stack is not used properly.

cat /tmp/dtrace-25840.data
Usage /usr/bin/dtrace [--help] [-h | -G] [-C [-I]] -s File.d [-o ]

Add a verbose mode for easier troubleshooting and log error messages

Currently PMA fails or succeeds silently. When things are good this is a non-issue, when something doesn't work you probably want to see a bit more output.
I've played around with the idea and would like to log:

1. Any JVMTI failure
2. Agent method processing time
3. At debug level add a per method printout of some form?

Ideas and feedback before I turn this into a PR?