{"id":16940404,"url":"https://github.com/codingcat/kittenwhisker","last_synced_at":"2025-09-02T10:34:29.387Z","repository":{"id":146766660,"uuid":"92545215","full_name":"CodingCat/KittenWhisker","owner":"CodingCat","description":"debugging performance issues for Spark applications","archived":false,"fork":false,"pushed_at":"2018-09-14T02:00:40.000Z","size":152,"stargazers_count":8,"open_issues_count":2,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-25T15:15:28.349Z","etag":null,"topics":["apache-spark","debugging","flamegraph","jvm","jvm-performance","performance","spark"],"latest_commit_sha":null,"homepage":"","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CodingCat.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-05-26T20:13:21.000Z","updated_at":"2023-05-28T15:55:24.000Z","dependencies_parsed_at":null,"dependency_job_id":"2704065f-466a-4951-8c33-35dfcf19f0f6","html_url":"https://github.com/CodingCat/KittenWhisker","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CodingCat%2FKittenWhisker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CodingCat%2FKittenWhisker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CodingCat%2FKittenWhisker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CodingCat%2FKittenWhisker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CodingCat","download_url":"https://codeload.github.com/CodingCat/KittenWhisker/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248465351,"owners_count":21108244,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-spark","debugging","flamegraph","jvm","jvm-performance","performance","spark"],"created_at":"2024-10-13T21:06:59.493Z","updated_at":"2025-04-11T19:21:59.141Z","avatar_url":"https://github.com/CodingCat.png","language":"C","readme":"# KittenWhisker\n\n`KittenWhisker` is a tool to facilitate the performance debugging of Spark applications.\n It generates symbol files for JIT compiled methods of Drivers and Executors which can be further\n  used by Linux [`perf`](https://perf.wiki.kernel.org/index.php/Main_Page) tools.\n\nThe below is an example of FlameGraph generated by KittenWhisker (Sorry for missing the original SVG\n file, just do it intentionally to protect some credential infomation)\n\n![example flame graph](./example.png)\n\n\nI will show how to [configure KittenWhisker](#configure) and [use KittenWhisker](#usage).\n \n## \u003ca name=\"usage\"\u003eUse KittenWhisker\u003c/a\u003e\n \nThe workflow of KittenWhisker is as following:\n \n1. By providing java options through `spark-submit`, KittenWhisker is attached to driver and executor\nprocesses as a java agent\n\n2. In the java agent, it will (1) fork a new process triggering linux perf tool to sample the CPU usage \nspecific to the Driver/Executor process; and (2) generate symbol files for JIT methods with\n [perf-map-agent by Johannes Rudolph](https://github.com/jvm-profiling-tools/perf-map-agent) \n \n3. Eventually, the generated files are uploaded to the HDFS-compatible system \n\n4. Users can collect the uploaded data and do further analysis, e.g. generating FlameGraph.\n\n## Run Spark Application with KittenWhisker\n\n1. Compose your perf command\n\n    The result from KittenWhisker is based on Linux's perf-event, so you have to specify the `perf`\n    command for KittenWhisker to run. For example, you can save the following command in a file named\n    `perf.conf`: (more usage of perf can be found in [Brendan's blog post](http://www.brendangregg.com/perf.html) )\n    \n    `sudo perf record -F 99 -g sleep 120`\n    \n    Note that, `sleep 120` will make perf to sample the usage for 2 mins, and KittenWhisker will insert\n    `-p` and `-o` right before sleep command, which specifies the PID to be monitored and the name of output\n     file respectively.\n\n2. compose spark-submit command in start_app_cmd.sh. The below is an example of the command\n\n```\nspark-submit --master yarn-client --class perf.RunBenchmark --files\n /home/zhunan/code/kittenwhisker/perf.conf --driver-class-path $JAVA_HOME/lib/tools.jar --jars\n  /home/zhunan/code/kittenwhisker/target/kittenwhisker-0.1-SNAPSHOT-jar-with-dependencies.jar,\n  $JAVA_HOME/lib/tools.jar --conf spark.driver.extraJavaOptions=-javaagent:/home/zhunan/code/\n  kittenwhisker/target/kittenwhisker-0.1-SNAPSHOT-jar-with-dependencies.jar=waitingLength=200000,\n  targetDirectory=/flameperf/ --conf \"spark.executor.extraJavaOptions=-javaagent:\n  ./happysparking-0.1-SNAPSHOT-jar-with-dependencies.jar=waitingLength=200000,\n  targetDirectory=/flameperf/\" --conf \"spark.executor.extraClassPath=./tools.jar\" --driver-memory\n   16g --executor-memory 20g --executor-cores 8 --num-executors 4\n    /home/zhunan/code/spark-benchmark/target/scala-2.11/spark-benchmark.jar\n     sql --benchmark perf.TPCDS --database db1\n      --path /tpcds/ --executionMode parquet -i 1 --outputDir /outputresults/ --reportFormat parquet\n```\n\n  In this command, we did the following things \n\n* \u003cb\u003e--files\n   /home/zhunan/code/kittenwhisker/perf.conf\u003c/b\u003e: upload the file containing perf command perf.conf\n    to every executors/driver;\n     \n* \u003cb\u003e--driver-class-path $JAVA_HOME/lib/tools.jar\u003c/b\u003e: specify the driver class path to make Sun's tool jar\nto be accessible by driver process;\n\n* \u003cb\u003e--jars\n       /home/zhunan/code/kittenwhisker/target/kittenwhisker-0.1-SNAPSHOT-jar-with-dependencies.jar,\n       $JAVA_HOME/lib/tools.jar\u003c/b\u003e: upload KittenWhisker's jar and Sun's tools.jar to the executors'\n       class paths;\n\n* \u003cb\u003e--conf spark.driver.extraJavaOptions=-javaagent:/home/zhunan/code/\n       kittenwhisker/target/kittenwhisker-0.1-SNAPSHOT-jar-with-dependencies.jar=waitingLength=200000,\n       targetDirectory=/flameperf/\u003c/b\u003e: we specify java options for the driver to make KittenWhisker attached\n       to the Driver process as a java agent; In this option, we specified two parameters for KittenWhisker,\n       waitingLength and targetDirectory. `waitingLength` will make KittenWhisker wait for a period with a\n       specified length (in ms), which is useful to try to measure only after your JVM is warmed up.\n       `targetDirectory` is the directory in a HDFS-compatible system which is to store the files generated by\n       KittenWhisker.\n         \n* \u003cb\u003e--conf \"spark.executor.extraJavaOptions=-javaagent:\n       ./happysparking-0.1-SNAPSHOT-jar-with-dependencies.jar=waitingLength=200000,\n       targetDirectory=/flameperf/\"\u003c/b\u003e: similar to the last one, we specify java options for the executors\n              \n* \u003cb\u003e--conf \"spark.executor.extraClassPath=./tools.jar\"\u003c/b\u003e: specifies tools.jar should be included in\n   the executors' class path\n     \n* \u003cb\u003eOthers\u003c/b\u003e: the others are Spark's resource-related and application specific parameters.\n     \n     \n3. Run Spark application with KittenWhisker\n\n    now you can run your spark application with KittenWhisker.\n    \n    `./start_test.sh zhunan /flameperf ./`\n\n    The first parameter of the script is the name to login to the nodes in your cluster; the second\n    one is the directory in the HDFS-compatible system to save the generated files; the third parameter\n    specifies the local location to download data from HDFS.\n    \n    This script will grant `yarn` user the sudo permission to run perf and chmod commands, since Linux's\n    perf tool requires that. After the application is finished, the permission would be revoked.\n    \n4. Produce Flamegraph with KittenWhisker\n\n    With one line command, you can generate Flame graphs for all processes in your spark applications.\n     \n    `./generate_flamegraph.sh ./flameperf`\n     \n     the only parameter is the local directory path saving the downloaded data. \n\n\n##  \u003ca name=\"configure\"\u003eConfigure KittenWhisker\u003c/a\u003e \n\n### Prepare Slaves File\n\nBefore you install KittenWhisker, you have to prepare a `slaves` files containing IP address of all nodes\n in your cluster (or all nodes which can be used to run Spark Drivers/Executors)\n \n### Clone KittenWhisker repository and install it\n\n1. First you have to clone the repo from github\n\n`git clone https://github.com/CodingCat/KittenWhisker.git`\n\n2. Then you can compile it with maven (ensure that you installed `cmake` beforehand) \n\n`cd KittenWhisker; mvn package`\n\n### Install Depended Tools \n\nThen you need to install several tools depended by `KittenWhisker` \n\n1. Install Linux's Perf tool in every node of the cluster\n\n    Depends on your OS, you have various ways to do it, e.g. in Ubuntu, you can do it by \n    \n    ```\n    sudo apt-get install linux-tools-common linux-tools-generic linux-tools-`uname -r`\n    ```\n    \n    You can also install via a helper script provided by KittenWhisker, run the following command in\n     the root path of KittenWhisker\n    \n    ```\n    ./install_packages.sh username_to_login_to_machines \n    ```\n    \n    This script will read slaves file, login to each machine specified in the file with the\n     username you provided and install perf tool. (Assuming the machine is with Ubuntu OS)  \n    \n2. Install JDK debugging symbols (if you are using OpenJDK)\n\n   If you are using OpenJDK, JDK debugging symbols are used to interpret the JVM runtime function names.\n    To install it, you can run \n    \n    ```\n    sudo apt-get install openjdk-8-dbg\n    ```\n    \n    Or with the helper script in KittenWhisker, you can simply do\n    \n    ```\n    ./install_packages.sh username_to_login_to_machines \n    ```\n    \n    with the assumption that you are using Ubuntu. \n    \n3. Install Flamegraph tool (just need to be done in the machine you currently login in )\n\n   Install Flamegraph tool is as simple as clone Brendan's repo and setup environment variable for\n    KittenWhisker. Do the following\n    \n    ```\n    git clone https://github.com/brendangregg/FlameGraph.git\n    \n    cd FlameGraph; echo \"export FLAMEGRAPH_DIR=`pwd`\" \u003e\u003e ~/.bashrc\n    ```\n\n## Acknowledgement\n\nThis work cannot be done without the existing tools from other developers. It uses [perf-map-agent by\n Johannes Rudolph](https://github.com/jvm-profiling-tools/perf-map-agent), and it also grabs some code\n from [Min Zhou's perfj](https://github.com/coderplay/perfj). The flame graph generation part is, of course,\n based on [Brendan Gregg's FlameGraph Tool](https://github.com/brendangregg/FlameGraph)\n\n## Disclaimer\n\nIt is *NOT* a software released by Microsoft and the code is very experimental!\n You will use it at your own risk. You have been warned!\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcodingcat%2Fkittenwhisker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcodingcat%2Fkittenwhisker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcodingcat%2Fkittenwhisker/lists"}