Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/codingcat/kittenwhisker

debugging performance issues for Spark applications
https://github.com/codingcat/kittenwhisker

apache-spark debugging flamegraph jvm jvm-performance performance spark

Last synced: 2 months ago
JSON representation

debugging performance issues for Spark applications

Awesome Lists containing this project

README 

          
# KittenWhisker



`KittenWhisker` is a tool to facilitate the performance debugging of Spark applications.

 It generates symbol files for JIT compiled methods of Drivers and Executors which can be further

  used by Linux [`perf`](https://perf.wiki.kernel.org/index.php/Main_Page) tools.



The below is an example of FlameGraph generated by KittenWhisker (Sorry for missing the original SVG

 file, just do it intentionally to protect some credential infomation)



![example flame graph](./example.png)



I will show how to [configure KittenWhisker](#configure) and [use KittenWhisker](#usage).

 

## Use KittenWhisker

 

The workflow of KittenWhisker is as following:

 

1. By providing java options through `spark-submit`, KittenWhisker is attached to driver and executor

processes as a java agent



2. In the java agent, it will (1) fork a new process triggering linux perf tool to sample the CPU usage 

specific to the Driver/Executor process; and (2) generate symbol files for JIT methods with

 [perf-map-agent by Johannes Rudolph](https://github.com/jvm-profiling-tools/perf-map-agent) 

 

3. Eventually, the generated files are uploaded to the HDFS-compatible system 



4. Users can collect the uploaded data and do further analysis, e.g. generating FlameGraph.



## Run Spark Application with KittenWhisker



1. Compose your perf command



    The result from KittenWhisker is based on Linux's perf-event, so you have to specify the `perf`

    command for KittenWhisker to run. For example, you can save the following command in a file named

    `perf.conf`: (more usage of perf can be found in [Brendan's blog post](http://www.brendangregg.com/perf.html) )

    

    `sudo perf record -F 99 -g sleep 120`

    

    Note that, `sleep 120` will make perf to sample the usage for 2 mins, and KittenWhisker will insert

    `-p` and `-o` right before sleep command, which specifies the PID to be monitored and the name of output

     file respectively.



2. compose spark-submit command in start_app_cmd.sh. The below is an example of the command



```

spark-submit --master yarn-client --class perf.RunBenchmark --files

 /home/zhunan/code/kittenwhisker/perf.conf --driver-class-path $JAVA_HOME/lib/tools.jar --jars

  /home/zhunan/code/kittenwhisker/target/kittenwhisker-0.1-SNAPSHOT-jar-with-dependencies.jar,

  $JAVA_HOME/lib/tools.jar --conf spark.driver.extraJavaOptions=-javaagent:/home/zhunan/code/

  kittenwhisker/target/kittenwhisker-0.1-SNAPSHOT-jar-with-dependencies.jar=waitingLength=200000,

  targetDirectory=/flameperf/ --conf "spark.executor.extraJavaOptions=-javaagent:

  ./happysparking-0.1-SNAPSHOT-jar-with-dependencies.jar=waitingLength=200000,

  targetDirectory=/flameperf/" --conf "spark.executor.extraClassPath=./tools.jar" --driver-memory

   16g --executor-memory 20g --executor-cores 8 --num-executors 4

    /home/zhunan/code/spark-benchmark/target/scala-2.11/spark-benchmark.jar

     sql --benchmark perf.TPCDS --database db1

      --path /tpcds/ --executionMode parquet -i 1 --outputDir /outputresults/ --reportFormat parquet

```



  In this command, we did the following things 



* --files

   /home/zhunan/code/kittenwhisker/perf.conf: upload the file containing perf command perf.conf

    to every executors/driver;

     

* --driver-class-path $JAVA_HOME/lib/tools.jar: specify the driver class path to make Sun's tool jar

to be accessible by driver process;



* --jars

       /home/zhunan/code/kittenwhisker/target/kittenwhisker-0.1-SNAPSHOT-jar-with-dependencies.jar,

       $JAVA_HOME/lib/tools.jar: upload KittenWhisker's jar and Sun's tools.jar to the executors'

       class paths;



* --conf spark.driver.extraJavaOptions=-javaagent:/home/zhunan/code/

       kittenwhisker/target/kittenwhisker-0.1-SNAPSHOT-jar-with-dependencies.jar=waitingLength=200000,

       targetDirectory=/flameperf/: we specify java options for the driver to make KittenWhisker attached

       to the Driver process as a java agent; In this option, we specified two parameters for KittenWhisker,

       waitingLength and targetDirectory. `waitingLength` will make KittenWhisker wait for a period with a

       specified length (in ms), which is useful to try to measure only after your JVM is warmed up.

       `targetDirectory` is the directory in a HDFS-compatible system which is to store the files generated by

       KittenWhisker.

         

* --conf "spark.executor.extraJavaOptions=-javaagent:

       ./happysparking-0.1-SNAPSHOT-jar-with-dependencies.jar=waitingLength=200000,

       targetDirectory=/flameperf/": similar to the last one, we specify java options for the executors

              

* --conf "spark.executor.extraClassPath=./tools.jar": specifies tools.jar should be included in

   the executors' class path

     

* Others: the others are Spark's resource-related and application specific parameters.

     

     

3. Run Spark application with KittenWhisker



    now you can run your spark application with KittenWhisker.

    

    `./start_test.sh zhunan /flameperf ./`



    The first parameter of the script is the name to login to the nodes in your cluster; the second

    one is the directory in the HDFS-compatible system to save the generated files; the third parameter

    specifies the local location to download data from HDFS.

    

    This script will grant `yarn` user the sudo permission to run perf and chmod commands, since Linux's

    perf tool requires that. After the application is finished, the permission would be revoked.

    

4. Produce Flamegraph with KittenWhisker



    With one line command, you can generate Flame graphs for all processes in your spark applications.

     

    `./generate_flamegraph.sh ./flameperf`

     

     the only parameter is the local directory path saving the downloaded data. 



##  Configure KittenWhisker 



### Prepare Slaves File



Before you install KittenWhisker, you have to prepare a `slaves` files containing IP address of all nodes

 in your cluster (or all nodes which can be used to run Spark Drivers/Executors)

 

### Clone KittenWhisker repository and install it



1. First you have to clone the repo from github



`git clone https://github.com/CodingCat/KittenWhisker.git`



2. Then you can compile it with maven (ensure that you installed `cmake` beforehand) 



`cd KittenWhisker; mvn package`



### Install Depended Tools 



Then you need to install several tools depended by `KittenWhisker` 



1. Install Linux's Perf tool in every node of the cluster



    Depends on your OS, you have various ways to do it, e.g. in Ubuntu, you can do it by 

    

    ```

    sudo apt-get install linux-tools-common linux-tools-generic linux-tools-`uname -r`

    ```

    

    You can also install via a helper script provided by KittenWhisker, run the following command in

     the root path of KittenWhisker

    

    ```

    ./install_packages.sh username_to_login_to_machines 

    ```

    

    This script will read slaves file, login to each machine specified in the file with the

     username you provided and install perf tool. (Assuming the machine is with Ubuntu OS)  

    

2. Install JDK debugging symbols (if you are using OpenJDK)



   If you are using OpenJDK, JDK debugging symbols are used to interpret the JVM runtime function names.

    To install it, you can run 

    

    ```

    sudo apt-get install openjdk-8-dbg

    ```

    

    Or with the helper script in KittenWhisker, you can simply do

    

    ```

    ./install_packages.sh username_to_login_to_machines 

    ```

    

    with the assumption that you are using Ubuntu. 

    

3. Install Flamegraph tool (just need to be done in the machine you currently login in )



   Install Flamegraph tool is as simple as clone Brendan's repo and setup environment variable for

    KittenWhisker. Do the following

    

    ```

    git clone https://github.com/brendangregg/FlameGraph.git

    

    cd FlameGraph; echo "export FLAMEGRAPH_DIR=`pwd`" >> ~/.bashrc

    ```



## Acknowledgement



This work cannot be done without the existing tools from other developers. It uses [perf-map-agent by

 Johannes Rudolph](https://github.com/jvm-profiling-tools/perf-map-agent), and it also grabs some code

 from [Min Zhou's perfj](https://github.com/coderplay/perfj). The flame graph generation part is, of course,

 based on [Brendan Gregg's FlameGraph Tool](https://github.com/brendangregg/FlameGraph)



## Disclaimer



It is *NOT* a software released by Microsoft and the code is very experimental!

 You will use it at your own risk. You have been warned!