{"id":23536487,"url":"https://github.com/hgn/kernel-magnifier","last_synced_at":"2025-04-23T11:56:38.807Z","repository":{"id":208473665,"uuid":"721722896","full_name":"hgn/kernel-magnifier","owner":"hgn","description":"A Kernel Ftrace Recorder and Visualizer to Analyse Executed Kernel Callchains","archived":false,"fork":false,"pushed_at":"2023-12-05T21:13:59.000Z","size":5664,"stargazers_count":12,"open_issues_count":0,"forks_count":2,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-04-23T11:56:27.944Z","etag":null,"topics":["analysis","kernel","linux","linux-kernel"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hgn.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-11-21T16:34:58.000Z","updated_at":"2025-03-21T06:20:00.000Z","dependencies_parsed_at":"2023-12-05T21:42:07.457Z","dependency_job_id":null,"html_url":"https://github.com/hgn/kernel-magnifier","commit_stats":null,"previous_names":["hgn/ftrace-callgrapher"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hgn%2Fkernel-magnifier","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hgn%2Fkernel-magnifier/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hgn%2Fkernel-magnifier/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hgn%2Fkernel-magnifier/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hgn","download_url":"https://codeload.github.com/hgn/kernel-magnifier/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250430589,"owners_count":21429323,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["analysis","kernel","linux","linux-kernel"],"created_at":"2024-12-26T02:29:06.854Z","updated_at":"2025-04-23T11:56:38.785Z","avatar_url":"https://github.com/hgn.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/kernel-magnifier-readme.png\" alt=\"Kernel Magnifier\"\u003e\u003cbr\u003e\n\u003c/p\u003e\n\n*A Linux Kernel Execution Flow Research Tool for Upcomming Kernel Hackers (and Veterans)*\n\n## Background - Problem Statement\n\nYou are new to Linux kernel development and want to develop a driver,\ncontribute new network stack functionality, better understand the complex\nprocess scheduler or just chase a kernel bug - then the Kernel Magnifier could\nprovide some support.\n\nMany developers new to the Linux kernel development find it difficult to\nunderstand the kernel. The kernel is a beast of its own. It is not simple\ncode, on the contrary: even if you have mastered programming languages such as\nC/C++ with a black belt, it is incredibly tedious to understand the kernel.\nThis is due to the following reasons, among others\n\n- The Linux kernel has its very own runtime, which is completely different from\n  userspace. There are many execution contexts which are complex even for\n  experienced kernel developers\n- Many things are processed asynchronously! Top and bottom halves from the\n  Informatics lecture are still familiar to many. But the kernel is much more\n  complex here. There are softwirqs, workers, tasklets and other context and\n  subsystem add custom implementations - like NAPI for the network stack\n  (imagine this as an on-demand bulk processing mechanism) -\n  on top of it. None of this makes the kernel any simpler.\n- The kernel is highly optimized, often every instruction in the processing hot\n  path is optimized to elicit the last percent of performance\n- Many indirect functions via function pointers are included in the kernel,\n  e.g. fileops structure.\n- The kernel has grown over decades - technical debts have also accumulated\n  here, which do not make the whole thing any easier\n\n## Kernel Magnifier\n\nThe Kernel Magnifier helps by recording the complete\u003csup\u003e$\u003c/sup\u003e function call\nchain of executed kernel functions (ftrace) and then allow to visualize these\nin an graph form. The graph form provides a relatively clear representation of\ncaller/calllee and call chains. It also shows which functions are called\nfrequently and which are called less frequently. This information is useful for\ngaining an overview (hot path, slow path).\n\n\u003csup\u003e$\u003c/sup\u003e: Becuase of the high frequency tracing an overflow can occur and\nis not rare. Furthermore there are a lot of functions which are inlined in the\nbuild process of the Linux kernel, also these functions are not traceable. And\nlast but not least: some functions of the ftrace subsystem are also not\ntraceable for recursive reasons.\n\n## Usage\n\n## Recording Data\n\nWithout arguments, ftrace-callgrapher will by default capture trace data on all\nCPUs for 10 seconds:\n\n```\n$ sudo kernel-magnifier.py record\nRecord mode - now starting recording traces for 10.0 seconds\nWrote data to kernel-magnifier.data\nRecord filesize: 657.75 MiB\n```\n\nThis will generate huge amount of data, even for the later post processing.\nAdditionally, it burdens the CPU cores and you risk data loss. If data can be\nfiltered in the recoring phase: perfect. Two options allow filtering for now:\nthe recorded time and a filter on what CPUs recording should be done. The later\nis really important especially on 16+ multi-core systems.\n\n```\n$ sudo kernel-magnifier.py record --record-time 10 --cpumask 1\nRecord mode - now starting recording traces for 10.0 seconds\nLimit recording to CPU mask 1\nWrote data to kernel-magnifier.data\nRecorded filesize: 199.38 MiB\n```\n\n## Visualizing Recorded Data\n\nVisualization is quite ease, just call with visualize as an argument:\n\n```\n$ kernel-magnifier.py visualize\nVisualization mode - now generating visualization...\nparsing completed, found 2316184 events\nfunction-calls.png generated\nkernel-magnifier.pdf generated\n```\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/graph-full.png\" alt=\"Full without Filtering\"\u003e\u003cbr\u003e\n\u003c/p\u003e\n\n[Original PDF (filesize 2MiB)](docs/kernel-magnifier.pdf)\n\nThe previous illustration show a full graph, of all callchains recording within\n10 seconds of kernel high live an rather idle system.\n\n# Symbol Filtering\n\nThe kernel magnifier becomes particularly useful if you limit the visualization\nto the relevant functions.\n\nFor symbol path filtering a mapping table `function name` to `filename` must be generated.\nThis map file contains a pure line based mapping between kernel function name\nand source code line within the Linux kernel source tree. For this you need a\ntool named `dwarfdump` as well as debug package of the kernel, see Installation\nnotes.\n\n```\n$ kernel-magnifier.py generate-symbol-map -k /usr/lib/debug/boot/vmlinux-$(uname -r)\n```\n\nNow filter just for *net*work related files, filtering for `drivers/net/`,\n`net/` and some other files named somehow `net`.\n\n```\n$ kernel-magnifier.py visualize --filter-filepath net\nVisualization mode - now generating visualization...\nparsing completed, found 2316184 events\nfunction-calls.png generated\nkernel-magnifier.pdf generated\n```\n\nNow the graph is limited to *net*work related functions and become really\nuseful. You can start zooming with you PDF viewer and looking at interesting\naspects.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/graph-net.png\" alt=\"Full without net Filtering\"\u003e\u003cbr\u003e\n\u003c/p\u003e\n\nThe following picture shows the right section enlarged. These are routines for\nincoming packet processing via `recvmsg()`\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/graph-net-sockrec.png\" alt=\"Full without net Filtering, zoomed\"\u003e\u003cbr\u003e\n\u003c/p\u003e\n\nAnother use case is to analyze the Linux Process/Task scheduler, this can be accomplished via\n\n```\n$ kernel-magnifier.py visualize --filter-filepath kernel/sched/fair.c,/kernel/sched/sched.h\n```\n\nThe illustration show one third of all scheduler related function. What is also\nvisible in the image: functions called often a more highlighted in red. The\n\"reddisher\", the hotter the function.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/graph-sched.png\" alt=\"Full with sched Filtering\"\u003e\u003cbr\u003e\n\u003c/p\u003e\n\n# Function Call Count\n\nBecause the function call is just available, the visualizer always plot the\nlist of called functions, sorted by highest. Here for the *net* use case\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/kernel-function-calls-sorted.png\" alt=\"kernel function calls\"\u003e\u003cbr\u003e\n\u003c/p\u003e\n\n\n# Installation\n\nJust clone/download the repository and execute the main script:\nhttps://github.com/hgn/kernel-magnifier.git\n\nThe script requires python3-pygraphviz, python3-matplotlib and python3-numpy.\n\nkernel-magnifier requires optionally debug symbols to map symbols to source\ncode files. For the actual mapping we use the dwarf information, to get the\ndata the tool use dwarfdump, so just install the packages.\n\nFor Debian Trixie:\n\n```\n# Mandatory\n$ apt-get install python3-pygraphviz python3-matplotlib python3-numpy\n\n# Optional, for symbol filtering required\n$ apt-get install dwarfdump \n$ apt-get install linux-image-amd64-dbg\n```\n\n\u003e NOTE: linux-image-amd64-dbg will consume roughly 600MiB of harddisk\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhgn%2Fkernel-magnifier","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhgn%2Fkernel-magnifier","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhgn%2Fkernel-magnifier/lists"}