Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/jvm-profiling-tools/perf-map-agent

A java agent to generate method mappings to use with the linux `perf` tool
https://github.com/jvm-profiling-tools/perf-map-agent

Last synced: 6 days ago
JSON representation

A java agent to generate method mappings to use with the linux `perf` tool

Awesome Lists containing this project

README 

        
# perf-map-agent



[![Join the chat at https://gitter.im/jrudolph/perf-map-agent](https://badges.gitter.im/jrudolph/perf-map-agent.svg)](https://gitter.im/jrudolph/perf-map-agent?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)

[![Build Status](https://travis-ci.org/jvm-profiling-tools/perf-map-agent.svg?branch=master)](https://travis-ci.org/jvm-profiling-tools/perf-map-agent)



A java agent to generate `/tmp/perf-.map` files for just-in-time(JIT)-compiled methods for use with the [Linux `perf` tools](https://perf.wiki.kernel.org/index.php/Main_Page).



## Build



Make sure `JAVA_HOME` is configured to point to a JDK. You need cmake >= 2.8.6 (see [#30](https://github.com/jrudolph/perf-map-agent/issues/30)). Then run the following on the command line:



    cmake .

    make



    # will create links to run scripts in 

    bin/create-links-in 



## Architecture



Linux `perf` tools will expect symbols for code executed from unknown memory regions at `/tmp/perf-.map`. This allows runtimes that

generate code on the fly to supply dynamic symbol mappings to be used with the `perf` suite of tools.



perf-map-agent is an agent that will generate such a mapping file for Java applications. It consists of a Java agent written C and a small

Java bootstrap application which attaches the agent to a running Java process.



When the agent is attached it instructs the JVM to report code blobs generated by the JVM at runtime for various purposes. Most importantly,

this includes JIT-compiled methods but also various dynamically-generated infrastructure parts like the dynamically created interpreter, 

adaptors, and jump tables for virtual dispatch (see `vtable` and `itable` entries). The agent creates a `/tmp/perf-.map` file which

it fills with one line per code blob that maps a memory location to a code blob name.



The Java application takes the PID of a Java process as an argument and an arbitrary number of additional arguments which it passes to the agent.

It then attaches to the target process and instructs it to load the agent library.



## Command line scripts



The `bin` directory contains a set of shell scripts to combine common `perf` / `dtrace` perations with creating the map file.



 - `create-java-perf-map.sh  ` takes a PID and options. It knows where to find libraries relative to the `bin` directory.

 - `perf-java-top  ` takes a PID and additional options to pass to `perf top`. Uses the agent to create a new

    `/tmp/perf-.map` and then calls `perf top` with the given options.

 - `perf-java-record-stack  ` takes a PID and additional options to pass to `perf record`. Runs

   `perf record -g -p  ` to collect performance data including stack traces. Afterwards it uses the agent to create a

   new `/tmp/perf-.map` file.

 - `perf-java-report-stack  ` calls first `perf-java-record-stack  ` and then runs

   `perf report` to directly analyze the captured data. You can call `perf report -i /tmp/perf-.data` again with any options after the

   script has exited to further analyze the data from the previous run.

 - `perf-java-flames  ` collects data with `perf-java-record-stack` and then creates a visualization

   using [@brendangregg's FlameGraph](https://github.com/brendangregg/FlameGraph) tools. To get meaningful stacktraces spanning several JIT-compiled methods,

   you need to run your JVM with `-XX:+PreserveFramePointer` (which is available starting from JDK8 update 60 build 19) as detailed

   in [ag netflix blog entry](http://techblog.netflix.com/2015/07/java-in-flames.html).

 - `create-links-in ` will install symbolic links to the above scripts into ``.

 - `dtrace-java-record-stack ` takes a PID. Runs`dtrace` to collect performance data including stack traces. Afterwards it uses the agent to create a

   new `/tmp/perf-.map` file.

 - `dtrace-java-flames ` collects data with `dtrace-java-record-stack` and then creates a visualization

   using [@brendangregg's FlameGraph](https://github.com/brendangregg/FlameGraph) tools. To get meaningful stacktraces spanning several JIT-compiled methods,

   you need to run your JVM with `-XX:+PreserveFramePointer` (which is available starting from JDK8 update 60 build 19) as detailed

   in [ag netflix blog entry](http://techblog.netflix.com/2015/07/java-in-flames.html).



Environment variables:



 - `PERF_MAP_OPTIONS`: a string of additional options to pass to the agent as described below.

 - `PERF_RECORD_SECONDS`: the number of seconds, `perf-java-report-stack` and similar tools will record performance data

 - `PERF_RECORD_FREQ`: the sampling frequence as passed to `perf record -F`

 - `FLAMEGRAPH_DIR`: the directory into which [@brendangregg's FlameGraph](https://github.com/brendangregg/FlameGraph) has been checked out

 - `PERF_JAVA_TMP`: the directory to put temporary files in, the default is `/tmp`

 - `PERF_DATA_FILE`: the file name where `perf-java-record-stack` will output performance data into, the default is `$PERF_JAVA_TMP/perf-.data`

 - `PERF_COLLAPSE_OPTS`: a string of additional flags to pass to stackcollapse-perf.pl (found in FLAMEGRAPH_DIR), (add --inline with unfoldall perfmap) 

 - `PERF_FLAME_OUTPUT`: the file name to which the flamegraph SVG will be written, the default is `flamegraph-.svg`

 - `PERF_FLAME_OPTS`: options to pass to flamegraph.pl (found in FLAMEGRAPH_DIR), the default is  `--color java`

 - `DTRACE_SECONDS`: the number of seconds, `dtrace` and similar tools will record performance data

 - `DTRACE_FREQ`: the sampling frequence as passed to `dtrace`

 - `DTRACE_JAVA_TMP`: the directory to put temporary files in, the default is `/tmp`

 - `DTRACE_DATA_FILE`: the file name where `dtrace-java-record-stack` will output performance data into, the default is `$DTRACE_JAVA_TMP/dtrace-.data`



## Options



You can add a comma separated list of options to `perf-java` (or the `AttachOnce` runner). These options are currently supported:



 - `unfold`: Create extra entries for every codeblock inside a method that was inlined from elsewhere

    (named <inlined_method> in <root_method>). Be aware of the effects of 'skid' in relation with unfolding.

    See the section below. Also, see the below section about inaccurate inlining information.

 - `unfoldall`: Similar to `unfold` but will include the complete inlined stack at a code location in the form

    `root_method->inlined method 1->inlined method 2->...->inlined method on top`.

 - `unfoldsimple`: similar to `unfold`, however, the extra entries do not include the " in <root_method>" part

 - `msig`: include full method signature in the name string

 - `dottedclass`: convert class signature (`Ljava/lang/Class;`) to the usual class names with segments separated by dots

   (`java.lang.Class`). NOTE: this currently breaks coloring when used in combination with [flamegraphs](https://github.com/brendangregg/FlameGraph).

 - `sourcepos`: Adds the name of the source file and the line number on which it is declared for each method. Useful

   when profiling Scala applications that crate a lot of synthetic classes and methods. Does not work with native methods.



## Known Issues



### Skid



You should be aware that instruction level profiling is not absolutely accurate but suffers from

'[skid](http://www.spinics.net/lists/linux-perf-users/msg02157.html)'. 'skid' means that the actual instruction

pointer may already have moved a bit further when a sample is recorded. In that case, (possibly hot) code is reported at

an address shortly after the actual hot instruction. See this [sample from one of Brendan's presentations](http://www.slideshare.net/brendangregg/scale2015-linux-perfprofiling/65) demonstrating this issue.



If using `unfold`, perf-map-agent will report sections that contain code inlined from other methods as separate entries.

Unfolded entries can be quite short, e.g. an inlined getter may only consist of a few instructions that now lives inside of another

method's JITed code. The next few instructions may then already belong to another entry. In such a case, it is more likely that skid

will not only affect the instruction pointer inside of a method entry but may affect which entry is chosen in the first place.



Skid that occurs inside a method is only visible when analyzing the actual assembler code (as with `perf annotate`). Skid that

affects the actual symbol resolution to choose a wrong entry will be much more visible as wrong entries will be reported with

tools that operate on the symbol level like the standard views of `perf report`, `perf top`, or in flame graphs.



So, while it is tempting to enable unfolded entries for the perceived extra resolution, this extra information is sometimes just noise

which will not only clutter the overall view but may also be misleading or wrong.



### Inaccurate mappings using the `unfold*` options



Hotspot does not retain line number and other debug information for inlined code at other places than safepoints. This

makes sense because you don't usually observe code running between safepoints from the JVM's perspective. This is different

when observing a process from the outside like with `perf`. For observed code locations outside of safepoints, the JVM will

not report any inlining information and perf-map-agent will assign those areas to the host method of the inlining.



For more fidelity, Hotspot can be instructed to include debug information for non-safepoints as well. Use

`-XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints` when running the target process. Note, however, that this will

produce a lot more information with the generated `perf-.map` file potentially growing to MBs of size.



### Agent Library Unloading



Unloading or reloading of a changed agent library is not supported by the JVM (but re-attaching is). Therefore, if you make changes to the

agent and recompile it you need to restart a target process that has an older version loaded to use the newer version.



### Missing symbols for libjvm.so



libjvm.so is the runtime component of the JVM. It is not covered by perf-map-agent but perf will use debug symbols as

provided by the distribution. If symbols for libjvm.so are missing see instructions for your Linux distribution to

install debug symbols for the JVM. See also [issue #39](https://github.com/jrudolph/perf-map-agent/issues/39) which

contains a few pointers about how to install these.



## Disclaimer



I'm not a professional C code writer. The code is very "experimental", and it is e.g. missing checks for error conditions etc.. Use it at your own risk. You have been warned!



## License



This library is licensed under GPLv2. See the LICENSE file.