{"id":19248834,"url":"https://github.com/tjfontaine/vm_profile_guest","last_synced_at":"2025-06-15T20:32:49.381Z","repository":{"id":66088970,"uuid":"50807899","full_name":"tjfontaine/vm_profile_guest","owner":"tjfontaine","description":"Profile Arbitrary VM Guests with DTrace in KVM or Hypervisor.framework","archived":false,"fork":false,"pushed_at":"2020-05-18T17:33:20.000Z","size":15,"stargazers_count":5,"open_issues_count":0,"forks_count":0,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-02-23T16:16:14.605Z","etag":null,"topics":["dtrace","hypervisor-framework","kvm","linux","macos","profiling"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tjfontaine.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-02-01T02:21:19.000Z","updated_at":"2025-01-27T10:04:33.000Z","dependencies_parsed_at":"2023-02-20T20:45:14.720Z","dependency_job_id":null,"html_url":"https://github.com/tjfontaine/vm_profile_guest","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/tjfontaine/vm_profile_guest","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tjfontaine%2Fvm_profile_guest","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tjfontaine%2Fvm_profile_guest/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tjfontaine%2Fvm_profile_guest/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tjfontaine%2Fvm_profile_guest/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tjfontaine","download_url":"https://codeload.github.com/tjfontaine/vm_profile_guest/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tjfontaine%2Fvm_profile_guest/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":260046048,"owners_count":22950797,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dtrace","hypervisor-framework","kvm","linux","macos","profiling"],"created_at":"2024-11-09T18:11:04.813Z","updated_at":"2025-06-15T20:32:49.336Z","avatar_url":"https://github.com/tjfontaine.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Profiling VM Guests with DTrace\n\nGiven a Linux or macOS environment that has DTrace and `KVM` or\n`Hypervisor.framework` respectively, you can print stack traces for their\nvCPU's.  Tested with QEMU on macOS Mojave and Catalina (`-accel hvf`), as well\nas Oracle Linux 7 with UEK5 (`-enable-kvm`). If you supply a symbol file in the\nformat used by `nm`, addresses (if found) will be resolved into their\nappropriate symbol name. The output can be used with tools for producing Flame\nGraphs.\n\n## Usage\n\n```\nusage: vm_profile_guest.py [-h] --pid PID [--symbol_file SYMBOL_FILE]\n                           [--frame_count FRAME_COUNT] [--duration DURATION]\n                           [--profile_interval PROFILE_INTERVAL]\n                           [--script_only] [--base_address BASE_ADDRESS]\n\nProfile VM Guests\n\noptional arguments:\n  -h, --help            show this help message and exit\n  --pid PID             PID of QEMU process to profile\n  --symbol_file SYMBOL_FILE\n                        Optional symbol file for name resolution\n  --frame_count FRAME_COUNT\n                        Number of frames to walk\n  --duration DURATION   Duration to profile\n  --profile_interval PROFILE_INTERVAL\n                        Interval to query VCPUs\n  --script_only         Only print DTrace script\n  --base_address BASE_ADDRESS\n                        base address value\n```\n\n## Output\n\nHere is the output of a [rumpkernel](https://github.com/rumpkernel/rumprun)\nrunning the [nginx package](https://github.com/rumpkernel/rumprun-packages):\n\n```\ncpu_intr_ack+a\nipintr+88f\nsithread+77\nthreadbouncer+50\nbmk_cpu_sched_bouncer+b\n1\n\nbmk_platform_cpu_block+5a\nschedule+89\nbmk_sched_block+23\nwait.isra.0+61\nrumpuser_cv_wait+3c\nsithread+88\nthreadbouncer+50\nbmk_cpu_sched_bouncer+b\n1\n\ncpu_intr_ack+a\nthreadbouncer+50\nbmk_cpu_sched_bouncer+b\n1\n\nbmk_platform_cpu_block+5a\nschedule+89\nbmk_sched_block+23\nwait.isra.0+61\nrumpuser_cv_wait+3c\ndocvwait+7c\nrumpns_cv_timedwait_sig+45\nrumpns_kevent1+318\nrumpns_sys___kevent50+33\nrump_syscall+78\n_sys___kevent50+4e\n__kevent50+30\nngx_kqueue_process_events+93\nngx_process_events_and_timers+9d\nngx_single_process_cycle+74\nrumprun_main1+a4d\nmainbouncer+4c\npthread__create_tramp+6e\nbmk_cpu_sched_bouncer+b\n1\n```\n\nThe format is ready to be used with tools for producing [Flame\nGraphs](http://www.brendangregg.com/flamegraphs.html) such as can be found in\nBrendan Gregg's [tools](https://github.com/brendangregg/FlameGraph).\n\n## Caveats and Assumptions\n\n### Frame Pointers\n\nIn order to walk the stack this tool assumes the guest has preserved frame\npointers. That is to say it's been compiled with `-fno-omit-frame-pointer`.\nThis is crucial, if `RBP` isn't in fact the base pointer, you will not get\nsensical information from this tool. Suffice it to say, compilers no longer\npreserve frame pointers by default, so you'll need to make sure you have this\ncovered.\n\n### Address Spaces\n\nNormally for DTrace one only has to think about `kernel` and `user` address\nspaces, and to remember that DTrace is operating in the `kernel` and so you may\nneed to `copyin` to the kernel addresse space from `user` space.\n\nBut when profiling a Virtual Machine one must also be aware of the addressing\nwithin that guest machine.\n\nThe demo above demonstrates using the tool against a Unikernel running inside\nthe VM. This greatly simplifies the effort needed to produce a stack trace\nbecause we can assume the unikernel has a single unified address layout. That\nis to say, the guest doesn't have its own notion of `kernel` and `user` address\nspaces.\n\nKeep in mind that DTrace has strict requirements regarding loops and branching,\nso if you want to be able to profile a full stack from a guest running a more\ncomplicated (useful?) operating system, you'll need to be able to resolve\n\"easily\" `guest-user` space addresses into `host-user` addresses such that they\ncan be `copyin`'d and the stack can be walked.\n\nAlso, DTrace requires the memory for the guest to be paged in for the `copyin`\nto work, it's possible that given your `VMM` this may not be the case. If the\nscript is failing with multiple errors about not being able to `copyin` an\naddress, and you're using `qemu` try using the flag `-mem-prealloc` (though\nthis could also be representative of missing frame pointers).\n\n### macOS\n\nFor this tool to work you must have SIP configured to allow _both_ DTrace *and*\nDebugging (because we want DTrace to _attach_ to the process). As such, that\nmeans you need to boot into restore mode and from a terminal issue the\nfollowing command: `csrutil enable --without dtrace --without debug`\n\nThe ability to trace a guest running on `Hypervisor.framework` is (putting it\nlightly) very fragile, perhaps even more fragile than the notion of walking a\nstack of a running virtual machine. While there is enough `CTF` in Apple's\nkernel to identify `hv_thread_target` and `hv_task_target`, they are merely\nmapped to `void*` instead of their more concrete types. This is to be expected,\nas they're largely an implementation detail and no rational person should be\ndepending on the layout of any of the related structures. But yet, here we are.\n\nTo that end, through a combination of reverse engineering `AppleHV.kext`,\nDTrace's `tracemem`, and a version of [Philipp Oppermann's Rust\nOS](https://github.com/phil-opp/blog_os/tree/post-02) we were able to deduce\nenough about the structures to make this tool work.\n\nThat being said, Apple can (and will?) change any of these layouts at any time,\ncertainly between major releases but also within a patch (though the latter is\nless likely).\n\nRegarding `Hypervisor.framework`'s `hv_vcpu_read_register()`, first given\nDTrace's design center it can't call a user space function (which is a good\nthing).  Second, while it's possible to implement the same logic from\n`Hypervisor.framework`, you're dealing with `user` addresses, which makes it\ndifficult to reason about the addresses in question, and unnecessary since\nthere is a `kernel` address sitting in `curthread` and `curtask` just waiting\nfor your usage.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftjfontaine%2Fvm_profile_guest","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftjfontaine%2Fvm_profile_guest","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftjfontaine%2Fvm_profile_guest/lists"}