{"id":13464478,"url":"https://github.com/cirosantilli/linux-kernel-module-cheat","last_synced_at":"2026-03-05T13:02:03.752Z","repository":{"id":37381790,"uuid":"64534859","full_name":"cirosantilli/linux-kernel-module-cheat","owner":"cirosantilli","description":"The perfect emulation setup to study and develop the Linux kernel, kernel modules, QEMU, gem5 and x86_64, ARMv7 and ARMv8 userland and baremetal assembly, ANSI C, C++ and POSIX. GDB step debug and KGDB just work. Powered by Buildroot and crosstool-NG. Highly automated. Thoroughly documented. Automated tests. \"Tested\" in an Ubuntu 24.04 host.","archived":false,"fork":false,"pushed_at":"2025-05-13T09:21:32.000Z","size":11666,"stargazers_count":4419,"open_issues_count":71,"forks_count":619,"subscribers_count":144,"default_branch":"master","last_synced_at":"2025-12-06T20:59:31.166Z","etag":null,"topics":["buildroot","gdb","kgdb","linux-kernel","linux-kernel-module","qemu"],"latest_commit_sha":null,"homepage":"https://cirosantilli.com/linux-kernel-module-cheat","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cirosantilli.png","metadata":{"files":{"readme":"README.adoc","changelog":null,"contributing":"CONTRIBUTING.adoc","funding":".github/FUNDING.yml","license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null},"funding":{"github":"cirosantilli"}},"created_at":"2016-07-30T08:38:26.000Z","updated_at":"2025-12-06T07:12:32.000Z","dependencies_parsed_at":"2022-07-08T07:39:53.718Z","dependency_job_id":"2e2474db-3a20-4757-b4b9-361b1097a3f9","html_url":"https://github.com/cirosantilli/linux-kernel-module-cheat","commit_stats":{"total_commits":1774,"total_committers":9,"mean_commits":"197.11111111111111","dds":0.005073280721533235,"last_synced_commit":"0195cc01dfac7a720332250b4d7352add39f19b0"},"previous_names":[],"tags_count":15,"template":false,"template_full_name":null,"purl":"pkg:github/cirosantilli/linux-kernel-module-cheat","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cirosantilli%2Flinux-kernel-module-cheat","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cirosantilli%2Flinux-kernel-module-cheat/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cirosantilli%2Flinux-kernel-module-cheat/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cirosantilli%2Flinux-kernel-module-cheat/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cirosantilli","download_url":"https://codeload.github.com/cirosantilli/linux-kernel-module-cheat/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cirosantilli%2Flinux-kernel-module-cheat/sbom","scorecard":{"id":283276,"data":{"date":"2025-08-11","repo":{"name":"github.com/cirosantilli/linux-kernel-module-cheat","commit":"fb39ab4611b5465def3c4af4735f38aa977b3550"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":1.4,"checks":[{"name":"Code-Review","score":0,"reason":"Found 0/30 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"SAST","score":0,"reason":"no SAST tool detected","details":["Warn: no pull requests merged into dev branch"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE.txt:0","Info: FSF or OSI recognized license: GNU General Public License v3.0: LICENSE.txt:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"Signed-Releases","score":0,"reason":"Project has not signed or included provenance with any releases.","details":["Warn: release artifact v3.0 not signed: https://api.github.com/repos/cirosantilli/linux-kernel-module-cheat/releases/15105578","Warn: release artifact v3.0-rc1 not signed: https://api.github.com/repos/cirosantilli/linux-kernel-module-cheat/releases/15074929","Warn: release artifact sha-c53ccb02782e6b5ba94c38c72597101cde86c4ff not signed: https://api.github.com/repos/cirosantilli/linux-kernel-module-cheat/releases/13045857","Warn: release artifact v3.0 does not have provenance: https://api.github.com/repos/cirosantilli/linux-kernel-module-cheat/releases/15105578","Warn: release artifact v3.0-rc1 does not have provenance: https://api.github.com/repos/cirosantilli/linux-kernel-module-cheat/releases/15074929","Warn: release artifact sha-c53ccb02782e6b5ba94c38c72597101cde86c4ff does not have provenance: https://api.github.com/repos/cirosantilli/linux-kernel-module-cheat/releases/13045857"],"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: containerImage not pinned by hash: Dockerfile:2: pin your Docker image by updating ubuntu:20.04 to ubuntu:20.04@sha256:8feb4d8ca5354def3d8fce243717141ce31e2c428701f6682bd2fafe15388214","Warn: pipCommand not pinned by hash: setup:34","Info: 0 out of 1 containerImage dependencies pinned","Info: 0 out of 1 pipCommand dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Vulnerabilities","score":0,"reason":"54 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: GHSA-cr5q-6q9f-rq6q","Warn: Project is vulnerable to: GHSA-7vh7-fw88-wj87","Warn: Project is vulnerable to: GHSA-353f-x4gh-cqq8","Warn: Project is vulnerable to: GHSA-5w6v-399v-w3cc","Warn: Project is vulnerable to: GHSA-mrxw-mxhj-p664","Warn: Project is vulnerable to: GHSA-r95h-9x8f-r3f7","Warn: Project is vulnerable to: GHSA-vvfq-8hwr-qm4m","Warn: Project is vulnerable to: GHSA-xc9x-jj77-9p9j","Warn: Project is vulnerable to: GHSA-2rxp-v6pw-ch6m","Warn: Project is vulnerable to: GHSA-4xqq-m2hx-25v8","Warn: Project is vulnerable to: GHSA-5866-49gr-22v4","Warn: Project is vulnerable to: GHSA-r55c-59qm-vjw6","Warn: Project is vulnerable to: GHSA-vg3r-rm7w-2xgh","Warn: Project is vulnerable to: GHSA-vmwr-mc7x-5vc3","Warn: Project is vulnerable to: PYSEC-2025-49 / GHSA-5rjg-fvgr-3xxf","Warn: Project is vulnerable to: GHSA-93q8-gq69-wqmw","Warn: Project is vulnerable to: GHSA-qwcr-r2fm-qrc7","Warn: Project is vulnerable to: GHSA-v6h2-p8h4-qcjw","Warn: Project is vulnerable to: GHSA-grv7-fg5c-xmjg","Warn: Project is vulnerable to: GHSA-pxg6-pf52-xh8x","Warn: Project is vulnerable to: GHSA-4gxf-g5gf-22h4","Warn: Project is vulnerable to: GHSA-rv95-896h-c2vc","Warn: Project is vulnerable to: GHSA-qw6h-vgh9-j6wx","Warn: Project is vulnerable to: GHSA-fjxv-7rqg-78g4","Warn: Project is vulnerable to: GHSA-896r-f27r-55mw","Warn: Project is vulnerable to: GHSA-f8q6-p94x-37v3","Warn: Project is vulnerable to: GHSA-xvch-5gv4-984h","Warn: Project is vulnerable to: GHSA-8hfj-j24r-96c4","Warn: Project is vulnerable to: GHSA-wc69-rhjr-hc9g","Warn: Project is vulnerable to: GHSA-56x4-j7p9-fcf9","Warn: Project is vulnerable to: GHSA-v78c-4p63-2j6c","Warn: Project is vulnerable to: GHSA-qrpm-p2h7-hrv2","Warn: Project is vulnerable to: GHSA-mwcw-c2x4-8c55","Warn: Project is vulnerable to: GHSA-9wv6-86v2-598j","Warn: Project is vulnerable to: GHSA-rhx6-c78j-4q9w","Warn: Project is vulnerable to: GHSA-hrpp-h998-j3pp","Warn: Project is vulnerable to: GHSA-p8p7-x288-28g6","Warn: Project is vulnerable to: GHSA-c2qf-rxjj-qqgw","Warn: Project is vulnerable to: GHSA-m6fv-jmcg-4jfg","Warn: Project is vulnerable to: GHSA-wrh9-cjv3-2hpw","Warn: Project is vulnerable to: GHSA-8c25-f3mj-v6h8","Warn: Project is vulnerable to: GHSA-vqfx-gj96-3w95","Warn: Project is vulnerable to: GHSA-f598-mfpv-gmfx","Warn: Project is vulnerable to: GHSA-cm22-4g7w-348p","Warn: Project is vulnerable to: GHSA-9qrh-qjmc-5w2p","Warn: Project is vulnerable to: GHSA-jqv5-7xpx-qj74","Warn: Project is vulnerable to: GHSA-3jfq-g458-7qm9","Warn: Project is vulnerable to: GHSA-5955-9wpr-37jh","Warn: Project is vulnerable to: GHSA-f5x3-32g6-xq36","Warn: Project is vulnerable to: GHSA-r628-mhmh-qjhw","Warn: Project is vulnerable to: GHSA-9r2w-394v-53qc","Warn: Project is vulnerable to: GHSA-qq89-hq3f-393p","Warn: Project is vulnerable to: GHSA-72xf-g2v4-qvf3","Warn: Project is vulnerable to: GHSA-qgmg-gppg-76g5"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}}]},"last_synced_at":"2025-08-17T16:30:58.235Z","repository_id":37381790,"created_at":"2025-08-17T16:30:58.235Z","updated_at":"2025-08-17T16:30:58.235Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30127210,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-05T12:40:50.676Z","status":"ssl_error","status_checked_at":"2026-03-05T12:39:32.209Z","response_time":93,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["buildroot","gdb","kgdb","linux-kernel","linux-kernel-module","qemu"],"created_at":"2024-07-31T14:00:44.062Z","updated_at":"2026-03-05T13:02:03.649Z","avatar_url":"https://github.com/cirosantilli.png","language":"Python","funding_links":["https://github.com/sponsors/cirosantilli"],"categories":["Linux kernel and device driver development","Python","HarmonyOS","others"],"sub_categories":["ESP8266","Windows Manager"],"readme":"= Linux Kernel Module Cheat\n:cirosantilli-media-base: https://raw.githubusercontent.com/cirosantilli/media/master/\n:description: The perfect emulation setup to study and develop the \u003c\u003clinux-kernel\u003e\u003e v5.9.2, kernel modules, \u003c\u003cqemu-buildroot-setup,QEMU\u003e\u003e, \u003c\u003cgem5-buildroot-setup,gem5\u003e\u003e and x86_64, ARMv7 and ARMv8 \u003c\u003cuserland-assembly,userland\u003e\u003e and \u003c\u003cbaremetal-setup,baremetal\u003e\u003e assembly, \u003c\u003cc,ANSI C\u003e\u003e, \u003c\u003ccpp,C++\u003e\u003e and \u003c\u003cposix,POSIX\u003e\u003e. EVERYTHING is built from source. \u003c\u003cgdb\u003e\u003e and \u003c\u003ckgdb\u003e\u003e just work. Powered by \u003c\u003cabout-the-qemu-buildroot-setup,Buildroot\u003e\u003e and \u003c\u003cabout-the-baremetal-setup,crosstool-NG\u003e\u003e. Highly automated. Thoroughly documented. Automated \u003c\u003ctest-this-repo,tests\u003e\u003e. \"Tested\" in an Ubuntu 20.04 host.\n:idprefix:\n:idseparator: -\n:nofooter:\n:sectanchors:\n:sectlinks:\n:sectnumlevels: 6\n:sectnums:\n:toc-title:\n:toc: macro\n:toclevels: 6\n\nhttps://zenodo.org/badge/latestdoi/64534859[image:https://zenodo.org/badge/64534859.svg[]]\n\n{description}\n\nhttps://twitter.com/dakami/status/1344853681749934080[Dan Kaminski-approved]™ https://en.wikipedia.org/wiki/Dan_Kaminsky[RIP].\n\nTL;DR: xref:qemu-buildroot-setup-getting-started[xrefstyle=full] tested on Ubuntu 24.04:\n\n....\ngit clone https://github.com/cirosantilli/linux-kernel-module-cheat\ncd linux-kernel-module-cheat\nsudo apt install docker\npython3 -m venv .venv\n. .venv/bin/activate\n./setup\n./run-docker create\n./run-docker sh\n....\n\nThis leaves you inside a Docker shell. Then inside Docker:\n\n....\n./build --download-dependencies qemu-buildroot\n./run\n....\n\nand you are now in a Linux userland shell running on QEMU with everything built fully from source.\n\nThe source code for this page is located at: https://github.com/cirosantilli/linux-kernel-module-cheat[]. Due to https://github.com/isaacs/github/issues/1610[a GitHub limitation], this README is too long and not fully rendered on github.com, so either use:\n\n* https://cirosantilli.com/linux-kernel-module-cheat\n* https://cirosantilli.com/linux-kernel-module-cheat/index-split[]: split header version\n* \u003c\u003cbuild-the-documentation,build the docs yourself\u003e\u003e\n\n**https://www.youtube.com/watch?v=HDJFyCma32U[Project presentation on YouTube]**:\n\nimage::https://github.com/cirosantilli/media/blob/master/Linux_kernel_module_cheat_presentation.png?raw=true[demo,link=https://www.youtube.com/watch?v=HDJFyCma32U]\n\n**https://www.youtube.com/watch?v=fgDhe1tN50o[Project demo on YouTube]**:\n\nimage::https://github.com/cirosantilli/media/blob/master/Linux_kernel_module_cheat_demo.png?raw=true[demo,link=https://www.youtube.com/watch?v=fgDhe1tN50o]\n\nhttps://github.com/cirosantilli/china-dictatorship | https://cirosantilli.com/china-dictatorship/xinjiang\n\nimage::https://raw.githubusercontent.com/cirosantilli/china-dictatorship-media/master/Xinjiang_prisoners_sitting_identified.jpeg[width=800,link=https://github.com/cirosantilli/china-dictatorship]\n\ntoc::[]\n\n== `--china`\n\nThe most important functionality of this repository is the `--china` option, sample usage:\n\n....\npython3 -m venv .venv\n. .venv/bin/activate\n./setup\n./run --china \u003e index.html\nfirefox index.html\n....\n\nsee also: https://cirosantilli.com/china-dictatorship/mirrors\n\nThe secondary systems programming functionality is described on the sections below starting from \u003c\u003cgetting-started\u003e\u003e.\n\nimage::https://raw.githubusercontent.com/cirosantilli/china-dictatorship-media/master/Tiananmen_cute_girls.jpg[width=800]\n\n== Getting started\n\nEach child section describes a possible different setup for this repo.\n\nIf you don't know which one to go for, start with \u003c\u003cqemu-buildroot-setup-getting-started\u003e\u003e.\n\nDesign goals of this project are documented at: xref:design-goals[xrefstyle=full].\n\n=== Should you waste your life with systems programming?\n\nBeing the hardcore person who fully understands an important complex system such as a computer, it does have a nice ring to it doesn't it?\n\nBut before you dedicate your life to this nonsense, do consider the following points:\n\n* almost all contributions to the kernel are done by large companies, and if you are not an employee in one of them, you are likely not going to be able to do much.\n+\nThis can be inferred by the fact that the `devices/` directory is by far the largest in the kernel.\n+\nThe kernel is of course just an interface to hardware, and the hardware developers start developing their kernel stuff even before specs are publicly released, both to help with hardware development and to have things working when the announcement is made.\n+\nFurthermore, I believe that there are in-tree devices which have never been properly publicly documented. Linus is of course fine with this, since code == documentation for him, but it is not as easy for mere mortals.\n+\nThere are some less hardware bound higher level layers in the kernel which might not require being in a hardware company, and a few people must be living off it.\n+\nBut of course, those are heavily motivated by the underlying hardware characteristics, and it is very likely that most of the people working there were previously at a hardware company.\n+\nIn that sense, therefore, the kernel is not as open as one might want to believe.\n+\nOf course, if there is some https://stackoverflow.com/questions/1697842/do-graphic-cards-have-instruction-sets-of-their-own/1697883[super useful and undocumented hardware that is just waiting there to be reverse engineered], then that's a much juicier target :-)\n* it is impossible to become rich with this knowledge.\n+\nThis is partly implied by the fact that you need to be in a big company to make useful low level things, and therefore you will only be a tiny cog in the engine.\n+\nThe key problem is that the entry cost of hardware design is just too insanely high for startups in general.\n* Is learning this the most useful thing that you think can do for society?\n+\nOr are you just learning it for job security and having a nice sounding title?\n+\nI'm not a huge fan of the person, but I think Jobs said it right: https://www.youtube.com/watch?v=FF-tKLISfPE\n+\nFirst determine the useful goal, and then backtrack down to the most efficient thing you can do to reach it.\n* there are two things that sadden me compared to physics-based engineering:\n+\n--\n** you will never become eternally famous. All tech disappears sooner or later, while laws of nature, at least as useful approximations, stay unchanged.\n** every problem that you face is caused by imperfections introduced by other humans.\n+\nIt is much easier to accept limitations of physics, and even natural selection in biology, which are not produced by a sentient being (?).\n--\n+\nPhysics-based engineering, just like low level hardware, is of course completely closed source however, since wrestling against the laws of physics is about the most expensive thing humans can do, so there's also a downside to it.\n\nAre you fine with those points, and ready to continue wasting your life with this crap?\n\nGood. In that case, read on, and let's have some fun together ;-)\n\nRelated: \u003c\u003csoft-topics\u003e\u003e.\n\n=== QEMU Buildroot setup\n\n==== QEMU Buildroot setup getting started\n\nThis setup has been tested on Ubuntu 20.04.\n\nThe Buildroot build is already broken on Ubuntu 21.04 onwards: https://github.com/cirosantilli/linux-kernel-module-cheat/issues/155[], so just do this from inside a 20.04 Docker instead as shown in the \u003c\u003cdocker\u003e\u003e setup. We could fix the build on Ubuntu 21.04, but it will break again inevitably later on.\n\nFor other host operating systems see: xref:supported-hosts[xrefstyle=full].\n\nReserve 12Gb of disk and run:\n\n....\ngit clone https://github.com/cirosantilli/linux-kernel-module-cheat\ncd linux-kernel-module-cheat\npython3 -m venv .venv\n. .venv/bin/activate\n./setup\n./build --download-dependencies qemu-buildroot\n./run\n....\n\nYou don't need to clone recursively even though we have `.git` submodules: `download-dependencies` fetches just the submodules that you need for this build to save time.\n\nIf something goes wrong, see: xref:common-build-issues[xrefstyle=full] and use our issue tracker: https://github.com/cirosantilli/linux-kernel-module-cheat/issues\n\nThe initial build will take a while (30 minutes to 2 hours) to clone and build, see \u003c\u003cbenchmark-builds\u003e\u003e for more details.\n\nIf you don't want to wait, you could also try the following faster but much more limited methods:\n\n* \u003c\u003cprebuilt\u003e\u003e\n* \u003c\u003chost\u003e\u003e\n\nbut you will soon find that they are simply not enough if you anywhere near serious about systems programming.\n\nAfter `./run`, QEMU opens up leaving you in the \u003c\u003clkmc-home,`/lkmc/` directory\u003e\u003e, and you can start playing with the kernel modules inside the simulated system:\n\n....\ninsmod hello.ko\ninsmod hello2.ko\nrmmod hello\nrmmod hello2\n....\n\nThis should print to the screen:\n\n....\nhello init\nhello2 init\nhello cleanup\nhello2 cleanup\n....\n\nwhich are `printk` messages from `init` and `cleanup` methods of those modules.\n\nSources:\n\n* link:kernel_modules/hello.c[]\n* link:kernel_modules/hello2.c[]\n\nQuit QEMU with:\n\n....\nCtrl-A X\n....\n\nSee also: xref:quit-qemu-from-text-mode[xrefstyle=full].\n\nAll available modules can be found in the link:kernel_modules[] directory.\n\nIt is super easy to build for different \u003c\u003ccpu-architecture,CPU architectures\u003e\u003e, just use the `--arch` option:\n\n....\npython3 -m venv .venv\n. .venv/bin/activate\n./setup\n./build --arch aarch64 --download-dependencies qemu-buildroot\n./run --arch aarch64\n....\n\nTo avoid typing `--arch aarch64` many times, you can set the default arch as explained at: xref:default-command-line-arguments[xrefstyle=full]\n\nI now urge you to read the following sections which contain widely applicable information:\n\n* \u003c\u003crun-command-after-boot\u003e\u003e\n* \u003c\u003cclean-the-build\u003e\u003e\n* \u003c\u003cbuild-the-documentation\u003e\u003e\n* Linux kernel\n** \u003c\u003cprintk\u003e\u003e\n** \u003c\u003ckernel-command-line-parameters\u003e\u003e\n\nOnce you use \u003c\u003cgdb\u003e\u003e and \u003c\u003ctmux\u003e\u003e, your terminal will look a bit like this:\n\n....\n[ 1.451857] input: AT Translated Set 2 keyboard as /devices/platform/i8042/s1│loading @0xffffffffc0000000: ../kernel_modules-1.0//timer.ko\n[ 1.454310] ledtrig-cpu: registered to indicate activity on CPUs │(gdb) b lkmc_timer_callback\n[ 1.455621] usbcore: registered new interface driver usbhid │Breakpoint 1 at 0xffffffffc0000000: file /home/ciro/bak/git/linux-kernel-module\n[ 1.455811] usbhid: USB HID core driver │-cheat/out/x86_64/buildroot/build/kernel_modules-1.0/./timer.c, line 28.\n[ 1.462044] NET: Registered protocol family 10 │(gdb) c\n[ 1.467911] Segment Routing with IPv6 │Continuing.\n[ 1.468407] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver │\n[ 1.470859] NET: Registered protocol family 17 │Breakpoint 1, lkmc_timer_callback (data=0xffffffffc0002000 \u003cmytimer\u003e)\n[ 1.472017] 9pnet: Installing 9P2000 support │ at /linux-kernel-module-cheat//out/x86_64/buildroot/build/\n[ 1.475461] sched_clock: Marking stable (1473574872, 0)-\u003e(1554017593, -80442)│kernel_modules-1.0/./timer.c:28\n[ 1.479419] ALSA device list: │28 {\n[ 1.479567] No soundcards found. │(gdb) c\n[ 1.619187] ata2.00: ATAPI: QEMU DVD-ROM, 2.5+, max UDMA/100 │Continuing.\n[ 1.622954] ata2.00: configured for MWDMA2 │\n[ 1.644048] scsi 1:0:0:0: CD-ROM QEMU QEMU DVD-ROM 2.5+ P5│Breakpoint 1, lkmc_timer_callback (data=0xffffffffc0002000 \u003cmytimer\u003e)\n[ 1.741966] tsc: Refined TSC clocksource calibration: 2904.010 MHz │ at /linux-kernel-module-cheat//out/x86_64/buildroot/build/\n[ 1.742796] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x29dc0f4s│kernel_modules-1.0/./timer.c:28\n[ 1.743648] clocksource: Switched to clocksource tsc │28 {\n[ 2.072945] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8043│(gdb) bt\n[ 2.078641] EXT4-fs (vda): couldn't mount as ext3 due to feature incompatibis│#0 lkmc_timer_callback (data=0xffffffffc0002000 \u003cmytimer\u003e)\n[ 2.080350] EXT4-fs (vda): mounting ext2 file system using the ext4 subsystem│ at /linux-kernel-module-cheat//out/x86_64/buildroot/build/\n[ 2.088978] EXT4-fs (vda): mounted filesystem without journal. Opts: (null) │kernel_modules-1.0/./timer.c:28\n[ 2.089872] VFS: Mounted root (ext2 filesystem) readonly on device 254:0. │#1 0xffffffff810ab494 in call_timer_fn (timer=0xffffffffc0002000 \u003cmytimer\u003e,\n[ 2.097168] devtmpfs: mounted │ fn=0xffffffffc0000000 \u003clkmc_timer_callback\u003e) at kernel/time/timer.c:1326\n[ 2.126472] Freeing unused kernel memory: 1264K │#2 0xffffffff810ab71f in expire_timers (head=\u003coptimized out\u003e,\n[ 2.126706] Write protecting the kernel read-only data: 16384k │ base=\u003coptimized out\u003e) at kernel/time/timer.c:1363\n[ 2.129388] Freeing unused kernel memory: 2024K │#3 __run_timers (base=\u003coptimized out\u003e) at kernel/time/timer.c:1666\n[ 2.139370] Freeing unused kernel memory: 1284K │#4 run_timer_softirq (h=\u003coptimized out\u003e) at kernel/time/timer.c:1692\n[ 2.246231] EXT4-fs (vda): warning: mounting unchecked fs, running e2fsck isd│#5 0xffffffff81a000cc in __do_softirq () at kernel/softirq.c:285\n[ 2.259574] EXT4-fs (vda): re-mounted. Opts: block_validity,barrier,user_xatr│#6 0xffffffff810577cc in invoke_softirq () at kernel/softirq.c:365\nhello S98 │#7 irq_exit () at kernel/softirq.c:405\n │#8 0xffffffff818021ba in exiting_irq () at ./arch/x86/include/asm/apic.h:541\nApr 15 23:59:23 login[49]: root login on 'console' │#9 smp_apic_timer_interrupt (regs=\u003coptimized out\u003e)\nhello /root/.profile │ at arch/x86/kernel/apic/apic.c:1052\n# insmod /timer.ko │#10 0xffffffff8180190f in apic_timer_interrupt ()\n[ 6.791945] timer: loading out-of-tree module taints kernel. │ at arch/x86/entry/entry_64.S:857\n# [ 7.821621] 4294894248 │#11 0xffffffff82003df8 in init_thread_union ()\n[ 8.851385] 4294894504 │#12 0x0000000000000000 in ?? ()\n │(gdb)\n....\n\n==== How to hack stuff\n\nBesides a seamless \u003c\u003cqemu-buildroot-setup-getting-started,initial build\u003e\u003e, this project also aims to make it effortless to modify and rebuild several major components of the system, to serve as an awesome development setup.\n\n===== Your first Linux kernel hack\n\nLet's hack up the \u003c\u003clinux-kernel-entry-point, Linux kernel entry point\u003e\u003e, which is an easy place to start.\n\nOpen the file:\n\n....\nvim submodules/linux/init/main.c\n....\n\nand find the `start_kernel` function, then add there a:\n\n....\npr_info(\"I'VE HACKED THE LINUX KERNEL!!!\");\n....\n\nThen rebuild the Linux kernel, quit QEMU and reboot the modified kernel:\n\n....\n./build-linux\n./run\n....\n\nand, surely enough, your message has appeared at the beginning of the boot:\n\n....\n\u003c6\u003e[ 0.000000] I'VE HACKED THE LINUX KERNEL!!!\n....\n\nSo you are now officially a Linux kernel hacker, way to go!\n\nWe could have used just link:build[] to rebuild the kernel as in the \u003c\u003cqemu-buildroot-setup-getting-started,initial build\u003e\u003e instead of link:build-linux[], but building just the required individual components is preferred during development:\n\n* saves a few seconds from parsing Make scripts and reading timestamps\n* makes it easier to understand what is being done in more detail\n* allows passing more specific options to customize the build\n\nThe link:build[] script is just a lightweight wrapper that calls the smaller build scripts, and you can see what `./build` does with:\n\n....\n./build --dry-run\n....\n\nsee also: \u003c\u003cdry-run\u003e\u003e.\n\nWhen you reach difficulties, QEMU makes it possible to easily GDB step debug the Linux kernel source code, see: xref:gdb[xrefstyle=full].\n\n===== Your first kernel module hack\n\nEdit link:kernel_modules/hello.c[] to contain:\n\n....\npr_info(\"hello init hacked\\n\");\n....\n\nand rebuild with:\n\n....\n./build-modules\n....\n\nNow there are two ways to test it out: the fast way, and the safe way.\n\nThe fast way is, without quitting or rebooting QEMU, just directly re-insert the module with:\n\n....\ninsmod /mnt/9p/out_rootfs_overlay/lkmc/hello.ko\n....\n\nand the new `pr_info` message should now show on the terminal at the end of the boot.\n\nIf you are simultaneously developing the test script and the kernel module, some smart test scripts should take the kernel module as first argument so you can directly run:\n\n....\n/mnt/9p/rootfs_overlay/lkmc/scull.sh /mnt/9p/out_rootfs_overlay/lkmc/scull.ko\n....\n\nand it will pick up both the test script and the kernel module from host.\n\nThis works because we have a \u003c\u003c9p\u003e\u003e mount there setup by default, which mounts the host directory that contains the build outputs on the guest:\n\n....\nls \"$(./getvar out_rootfs_overlay_dir)\"\n....\n\nThe fast method is slightly risky because your previously insmodded buggy kernel module attempt might have corrupted the kernel memory, which could affect future runs.\n\nSuch failures are however unlikely, and you should be fine if you don't see anything weird happening.\n\nThe safe way is to fist \u003c\u003crebuild-buildroot-while-running,quit QEMU\u003e\u003e, rebuild the modules put them somewhere QEMU can see and then reboot. So you could either place it in the root filesystem:\n\n....\n./build-modules\n./build-buildroot\n./run --eval-after 'insmod hello.ko'\n....\n\nwhere `./build-buildroot` is required after `./build-modules` because it re-generates the root filesystem with the modules that we compiled at `./build-modules`.\n\nAlternatively, for a slightly faster turnaround just leave it on 9p and use it from there:\n\n....\n./build-modules\n./run --eval-after 'insmod /mnt/9p/out_rootfs_overlay/lkmc/hello.ko'\n....\n\nYou can see that `./build` does that as well, by running:\n\n....\n./build --dry-run\n....\n\nSee also: \u003c\u003cdry-run\u003e\u003e.\n\n`--eval-after` is optional: you could just type `insmod hello.ko` in the terminal, but this makes it run automatically at the end of boot, and then drops you into a shell.\n\nIf the guest and host are the same arch, typically x86_64, you can speed up boot further with \u003c\u003ckvm\u003e\u003e:\n\n....\n./run --kvm\n....\n\nAll of this put together makes the safe procedure acceptably fast for regular development as well.\n\nIt is also easy to GDB step debug kernel modules with our setup, see: xref:gdb-step-debug-kernel-module[xrefstyle=full].\n\n===== Your first glibc hack\n\nWe use \u003c\u003clibc-choice,glibc as our default libc now\u003e\u003e, and it is tracked as an unmodified submodule at link:submodules/glibc[], at the exact same version that Buildroot has it, which can be found at: https://github.com/buildroot/buildroot/blob/2018.05/package/glibc/glibc.mk#L13[package/glibc/glibc.mk]. Buildroot 2018.05 applies no patches.\n\nLet's hack up the `puts` function:\n\n....\n./build-buildroot -- glibc-reconfigure\n....\n\nwith the patch:\n\n....\ndiff --git a/libio/ioputs.c b/libio/ioputs.c\nindex 706b20b492..23185948f3 100644\n--- a/libio/ioputs.c\n+++ b/libio/ioputs.c\n@@ -38,8 +38,9 @@ _IO_puts (const char *str)\n if ((_IO_vtable_offset (_IO_stdout) != 0\n || _IO_fwide (_IO_stdout, -1) == -1)\n \u0026\u0026 _IO_sputn (_IO_stdout, str, len) == len\n+ \u0026\u0026 _IO_sputn (_IO_stdout, \" hacked\", 7) == 7\n \u0026\u0026 _IO_putc_unlocked ('\\n', _IO_stdout) != EOF)\n- result = MIN (INT_MAX, len + 1);\n+ result = MIN (INT_MAX, len + 1 + 7);\n\n _IO_release_lock (_IO_stdout);\n return result;\n....\n\nAnd then:\n\n....\n./run --eval-after './c/hello.out'\n....\n\noutputs:\n\n....\nhello hacked\n....\n\nLol!\n\nWe can also test our hacked glibc on \u003c\u003cuser-mode-simulation\u003e\u003e with:\n\n....\n./run --userland userland/c/hello.c\n....\n\nI just noticed that this is actually a good way to develop glibc for other archs.\n\nIn this example, we got away without recompiling the userland program because we made a change that did not affect the glibc ABI, see this answer for an introduction to ABI stability: https://stackoverflow.com/questions/2171177/what-is-an-application-binary-interface-abi/54967743#54967743\n\nNote that for arch agnostic features that don't rely on bleeding kernel changes that you host doesn't yet have, you can develop glibc natively as explained at:\n\n* https://stackoverflow.com/questions/10412684/how-to-compile-my-own-glibc-c-standard-library-from-source-and-use-it/52454710#52454710\n* https://stackoverflow.com/questions/847179/multiple-glibc-libraries-on-a-single-host/52454603#52454603\n* https://stackoverflow.com/questions/2856438/how-can-i-link-to-a-specific-glibc-version/52550158#52550158 more focus on symbol versioning, but no one knows how to do it, so I answered\n\nTested on a30ed0f047523ff2368d421ee2cce0800682c44e + 1.\n\n===== Your first Binutils hack\n\nHave you ever felt that a single `inc` instruction was not enough? Really? Me too!\n\nSo let's hack the \u003c\u003cgnu-gas-assembler\u003e\u003e, which is part of https://en.wikipedia.org/wiki/GNU_Binutils[GNU Binutils], to add a new shiny version of `inc` called... `myinc`!\n\nGCC uses GNU GAS as its backend, so we will test out new mnemonic with an \u003c\u003cgcc-inline-assembly\u003e\u003e test program: link:userland/arch/x86_64/binutils_hack.c[], which is just a copy of link:userland/arch/x86_64/binutils_nohack.c[] but with `myinc` instead of `inc`.\n\nThe inline assembly is disabled with an `#ifdef`, so first modify the source to enable that.\n\nThen, try to build userland:\n\n....\n./build-userland\n....\n\nand watch it fail with:\n\n....\nbinutils_hack.c:8: Error: no such instruction: `myinc %rax'\n....\n\nNow, edit the file\n\n....\nvim submodules/binutils-gdb/opcodes/i386-tbl.h\n....\n\nand add a copy of the `\"inc\"` instruction just next to it, but with the new name `\"myinc\"`:\n\n....\ndiff --git a/opcodes/i386-tbl.h b/opcodes/i386-tbl.h\nindex af583ce578..3cc341f303 100644\n--- a/opcodes/i386-tbl.h\n+++ b/opcodes/i386-tbl.h\n@@ -1502,6 +1502,19 @@ const insn_template i386_optab[] =\n { { { 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n \t 0, 0, 1, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0,\n \t 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0 } } } },\n+ { \"myinc\", 1, 0xfe, 0x0, 1,\n+ { { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n+ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n+ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n+ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n+ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 } },\n+ { 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n+ 0, 1, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0,\n+ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n+ 0, 0, 0, 0, 0, 0 },\n+ { { { 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n+\t 0, 0, 1, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0,\n+\t 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0 } } } },\n { \"sub\", 2, 0x28, None, 1,\n { { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n....\n\nFinally, rebuild Binutils, userland and test our program with \u003c\u003cuser-mode-simulation\u003e\u003e:\n\n....\n./build-buildroot -- host-binutils-rebuild\n./build-userland --static\n./run --static --userland userland/arch/x86_64/binutils_hack.c\n....\n\nand we se that `myinc` worked since the assert did not fail!\n\nTested on b60784d59bee993bf0de5cde6c6380dd69420dda + 1.\n\n===== Your first GCC hack\n\nOK, now time to hack GCC.\n\nFor convenience, let's use the \u003c\u003cuser-mode-simulation\u003e\u003e.\n\nIf we run the program link:userland/c/gcc_hack.c[]:\n\n....\n./build-userland --static\n./run --static --userland userland/c/gcc_hack.c\n....\n\nit produces the normal boring output:\n\n....\ni = 2\nj = 0\n....\n\nSo how about we swap `++` and `--` to make things more fun?\n\nOpen the file:\n\n....\nvim submodules/gcc/gcc/c/c-parser.c\n....\n\nand find the function `c_parser_postfix_expression_after_primary`.\n\nIn that function, swap `case CPP_PLUS_PLUS` and `case CPP_MINUS_MINUS`:\n\n....\ndiff --git a/gcc/c/c-parser.c b/gcc/c/c-parser.c\nindex 101afb8e35f..89535d1759a 100644\n--- a/gcc/c/c-parser.c\n+++ b/gcc/c/c-parser.c\n@@ -8529,7 +8529,7 @@ c_parser_postfix_expression_after_primary (c_parser *parser,\n \t\texpr.original_type = DECL_BIT_FIELD_TYPE (field);\n \t }\n \t break;\n-\tcase CPP_PLUS_PLUS:\n+\tcase CPP_MINUS_MINUS:\n \t /* Postincrement. */\n \t start = expr.get_start ();\n \t finish = c_parser_peek_token (parser)-\u003eget_finish ();\n@@ -8548,7 +8548,7 @@ c_parser_postfix_expression_after_primary (c_parser *parser,\n \t expr.original_code = ERROR_MARK;\n \t expr.original_type = NULL;\n \t break;\n-\tcase CPP_MINUS_MINUS:\n+\tcase CPP_PLUS_PLUS:\n \t /* Postdecrement. */\n \t start = expr.get_start ();\n \t finish = c_parser_peek_token (parser)-\u003eget_finish ();\n....\n\nNow rebuild GCC, the program and re-run it:\n\n....\n./build-buildroot -- host-gcc-final-rebuild\n./build-userland --static\n./run --static --userland userland/c/gcc_hack.c\n....\n\nand the new ouptut is now:\n\n....\ni = 2\nj = 0\n....\n\nWe need to use the ugly `-final` thing because GCC has to packages in Buildroot, `-initial` and `-final`: https://stackoverflow.com/questions/54992977/how-to-select-an-override-srcdir-source-for-gcc-when-building-buildroot No one is able to example precisely with a minimal example why this is required:\n\n* https://stackoverflow.com/questions/39883865/why-multiple-passes-for-building-linux-from-scratch-lfs\n* https://stackoverflow.com/questions/27457835/why-do-cross-compilers-have-a-two-stage-compilation\n\n==== About the QEMU Buildroot setup\n\nWhat QEMU and Buildroot are:\n\n* \u003c\u003cintroduction-to-buildroot\u003e\u003e\n* \u003c\u003cintroduction-to-qemu\u003e\u003e\n\nThis is our reference setup, and the best supported one, use it unless you have good reason not to.\n\nIt was historically the first one we did, and all sections have been tested with this setup unless explicitly noted.\n\nRead the following sections for further introductory material:\n\n* \u003c\u003cintroduction-to-qemu\u003e\u003e\n* \u003c\u003cintroduction-to-buildroot\u003e\u003e\n\n[[dry-run]]\n=== Dry run to get commands for your project\n\nOne of the major features of this repository is that we try to support the `--dry-run` option really well for all scripts.\n\nThis option, as the name suggests, outputs the external commands that would be run (or more precisely: equivalent commands), without actually running them.\n\nThis allows you to just clone this repository and get full working commands to integrate into your project, without having to build or use this setup further!\n\nFor example, we can obtain a QEMU run for the file link:userland/c/hello.c[] in \u003c\u003cuser-mode-simulation\u003e\u003e by adding `--dry-run` to the normal command:\n\n....\n./run --dry-run --userland userland/c/hello.c\n....\n\nwhich as of LKMC a18f28e263c91362519ef550150b5c9d75fa3679 + 1 outputs:\n\n....\n+ /path/to/linux-kernel-module-cheat/out/qemu/default/opt/x86_64-linux-user/qemu-x86_64 \\\n -L /path/to/linux-kernel-module-cheat/out/buildroot/build/default/x86_64/target \\\n -r 5.2.1 \\\n -seed 0 \\\n -trace enable=load_file,file=/path/to/linux-kernel-module-cheat/out/run/qemu/x86_64/0/trace.bin \\\n -cpu max \\\n /path/to/linux-kernel-module-cheat/out/userland/default/x86_64/c/hello.out \\\n;\n....\n\nSo observe that the command contains:\n\n* `+`: sign to differentiate it from program stdout, much like bash `-x` output. This is not a valid part of the generated Bash command however.\n* the actual command nicely, indented and with arguments broken one per line, but with continuing backslashes so you can just copy paste into a terminal\n+\nFor setups that don't support the newline e.g. \u003c\u003cgem5-eclipse-configuration,Eclipse debugging\u003e\u003e, you can turn them off with `--print-cmd-oneline`\n* `;`: both a valid part of the Bash command, and a visual mark the end of the command\n\nFor the specific case of running emulators such as QEMU, the last command is also automatically placed in a file for your convenience and later inspection:\n\n....\ncat \"$(./getvar run_dir)/run.sh\"\n....\n\nSince we need this so often, the last run command is also stored for convenience at:\n\n....\ncat out/run.sh\n....\n\nalthough this won't of course work well for \u003c\u003csimultaneous-runs\u003e\u003e.\n\nFurthermore, `--dry-run` also automatically specifies, in valid Bash shell syntax:\n\n* environment variables used to run the command with syntax `+ ENV_VAR_1=abc ENV_VAR_2=def ./some/command`\n* change in working directory with `+ cd /some/new/path \u0026\u0026 ./some/command`\n\n=== gem5 Buildroot setup\n\n==== About the gem5 Buildroot setup\n\nThis setup is like the \u003c\u003cqemu-buildroot-setup\u003e\u003e, but it uses http://gem5.org/[gem5] instead of QEMU as a system simulator.\n\nQEMU tries to run as fast as possible and give correct results at the end, but it does not tell us how many CPU cycles it takes to do something, just the number of instructions it ran. This kind of simulation is known as functional simulation.\n\nThe number of instructions executed is a very poor estimator of performance because in modern computers, a lot of time is spent waiting for memory requests rather than the instructions themselves.\n\ngem5 on the other hand, can simulate the system in more detail than QEMU, including:\n\n* simplified CPU pipeline\n* caches\n* DRAM timing\n\nand can therefore be used to estimate system performance, see: xref:gem5-run-benchmark[xrefstyle=full] for an example.\n\nThe downside of gem5 much slower than QEMU because of the greater simulation detail.\n\nSee \u003c\u003cgem5-vs-qemu\u003e\u003e for a more thorough comparison.\n\n==== gem5 Buildroot setup getting started\n\nFor the most part, if you just add the `--emulator gem5` option or `*-gem5` suffix to all commands and everything should magically work.\n\nIf you haven't built Buildroot yet for \u003c\u003cqemu-buildroot-setup\u003e\u003e, you can build from the beginning with:\n\n....\npython3 -m venv .venv\n. .venv/bin/activate\n./setup\n./build --download-dependencies gem5-buildroot\n./run --emulator gem5\n....\n\nIf you have already built previously, don't be afraid: gem5 and QEMU use almost the same root filesystem and kernel, so `./build` will be fast.\n\nRemember that the gem5 boot is \u003c\u003cbenchmark-linux-kernel-boot,considerably slower\u003e\u003e than QEMU since the simulation is more detailed.\n\nIf you have a relatively new GCC version and the gem5 build fails on your machine, see: \u003c\u003cgem5-build-broken-on-recent-compiler-version\u003e\u003e.\n\nTo get a terminal, either open a new shell and run:\n\n....\n./gem5-shell\n....\n\nYou can quit the shell without killing gem5 by typing tilde followed by a period:\n\n....\n~.\n....\n\nIf you are inside \u003c\u003ctmux\u003e\u003e, which I highly recommend, you can both run gem5 stdout and open the guest terminal on a split window with:\n\n....\n./run --emulator gem5 --tmux\n....\n\nSee also: xref:tmux-gem5[xrefstyle=full].\n\nAt the end of boot, it might not be very clear that you have the shell since some \u003c\u003cprintk\u003e\u003e messages may appear in front of the prompt like this:\n\n....\n# \u003c6\u003e[ 1.215329] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x1cd486fa865, max_idle_ns: 440795259574 ns\n\u003c6\u003e[ 1.215351] clocksource: Switched to clocksource tsc\n....\n\nbut if you look closely, the `PS1` prompt marker `#` is there already, just hit enter and a clear prompt line will appear.\n\nIf you forgot to open the shell and gem5 exit, you can inspect the terminal output post-mortem at:\n\n....\nless \"$(./getvar --emulator gem5 m5out_dir)/system.pc.com_1.device\"\n....\n\nMore gem5 information is present at: xref:gem5[xrefstyle=full]\n\nGood next steps are:\n\n* \u003c\u003cgem5-run-benchmark\u003e\u003e: how to run a benchmark in gem5 full system, including how to boot Linux, checkpoint and restore to skip the boot on a fast CPU\n* \u003c\u003cm5out-directory\u003e\u003e: understand the output files that gem5 produces, which contain information about your run\n* \u003c\u003cm5ops\u003e\u003e: magic guest instructions used to control gem5\n* \u003c\u003cadd-new-files-to-the-buildroot-image\u003e\u003e: how to add your own files to the image if you have a benchmark that we don't already support out of the box (also send a pull request!)\n\n[[docker]]\n=== Docker host setup\n\nThis repository has been tested inside clean https://en.wikipedia.org/wiki/Docker_(software)[Docker] containers.\n\nThis is a good option if you are on a Linux host, but the native setup failed due to your weird host distribution, and you have better things to do with your life than to debug it. See also: xref:supported-hosts[xrefstyle=full].\n\nFor example, to do a \u003c\u003cqemu-buildroot-setup\u003e\u003e inside Docker, run:\n\n....\nsudo apt-get install docker\npython3 -m venv .venv\n. .venv/bin/activate\n./setup\n./run-docker create \u0026\u0026 \\\n./run-docker sh -- ./build --download-dependencies qemu-buildroot\n./run-docker\n....\n\nYou are now left inside a shell in the Docker! From there, just run as usual:\n\n....\n./run\n....\n\nThe host git top level directory is mounted inside the guest with a https://stackoverflow.com/questions/23439126/how-to-mount-a-host-directory-in-a-docker-container[Docker volume], which means for example that you can use your host's GUI text editor directly on the files. Just don't forget that if you nuke that directory on the guest, then it gets nuked on the host as well!\n\nCommand breakdown:\n\n* `./run-docker create`: create the image and container.\n+\nNeeded only the very first time you use Docker, or if you run `./run-docker DESTROY` to restart for scratch, or save some disk space.\n+\nThe image and container name is `lkmc`. The container shows under:\n+\n....\ndocker ps -a\n....\n+\nand the image shows under:\n+\n....\ndocker images\n....\n* `./run-docker`: open a shell on the container.\n+\nIf it has not been started previously, start it. This can also be done explicitly with:\n+\n....\n./run-docker start\n....\n+\nQuit the shell as usual with `Ctrl-D`\n+\nThis can be called multiple times from different host terminals to open multiple shells.\n* `./run-docker stop`: stop the container.\n+\nThis might save a bit of CPU and RAM once you stop working on this project, but it should not be a lot.\n* `./run-docker DESTROY`: delete the container and image.\n+\nThis doesn't really clean the build, since we mount the guest's working directory on the host git top-level, so you basically just got rid of the `apt-get` installs.\n+\nTo actually delete the Docker build, run on host:\n+\n....\n# sudo rm -rf out.docker\n....\n\nTo use \u003c\u003cgdb\u003e\u003e from inside Docker, you need a second shell inside the container. You can either do that from another shell with:\n\n....\n./run-docker\n....\n\nor even better, by starting a \u003c\u003ctmux\u003e\u003e session inside the container. We install `tmux` by default in the container.\n\nYou can also start a second shell and run a command in it at the same time with:\n\n....\n./run-docker sh -- ./run-gdb start_kernel\n....\n\nTo use \u003c\u003cqemu-graphic-mode\u003e\u003e from Docker, run:\n\n....\n./run --graphic --vnc\n....\n\nand then on host:\n\n....\nsudo apt-get install vinagre\n./vnc\n....\n\nTODO make files created inside Docker be owned by the current user in host instead of `root`:\n\n* https://stackoverflow.com/questions/33681396/how-do-i-write-to-a-volume-container-as-non-root-in-docker\n* https://stackoverflow.com/questions/23544282/what-is-the-best-way-to-manage-permissions-for-docker-shared-volumes\n* https://stackoverflow.com/questions/31779802/shared-volume-file-permissions-ownership-docker\n\n[[prebuilt]]\n=== Prebuilt setup\n\n==== About the prebuilt setup\n\nThis setup uses prebuilt binaries that we upload to GitHub from time to time.\n\nWe don't currently provide a full prebuilt because it would be too big to host freely, notably because of the cross toolchain.\n\nOur prebuilts currently include:\n\n* \u003c\u003cqemu-buildroot-setup\u003e\u003e binaries\n** Linux kernel\n** root filesystem\n* \u003c\u003cbaremetal-setup\u003e\u003e binaries for QEMU\n\nFor more details, see our our \u003c\u003crelease,release procedure\u003e\u003e.\n\nAdvantage of this setup: saves time and disk space on the initial install, which is expensive in largely due to building the toolchain.\n\nThe limitations are severe however:\n\n* can't \u003c\u003cgdb,GDB step debug the kernel\u003e\u003e, since the source and cross toolchain with GDB are not available. Buildroot cannot easily use a host toolchain: xref:prebuilt-toolchain[xrefstyle=full].\n+\nMaybe we could work around this by just downloading the kernel source somehow, and using a host prebuilt GDB, but we felt that it would be too messy and unreliable.\n* you won't get the latest version of this repository. Our \u003c\u003ctravis\u003e\u003e attempt to automate builds failed, and storing a release for every commit would likely make GitHub mad at us anyway.\n* \u003c\u003cgem5\u003e\u003e is not currently supported. The major blocking point is how to avoid distributing the kernel images twice: once for gem5 which uses `vmlinux`, and once for QEMU which uses `arch/*` images, see also:\n** https://github.com/cirosantilli/linux-kernel-module-cheat/issues/79\n** \u003c\u003cvmlinux-vs-bzimage-vs-zimage-vs-image\u003e\u003e.\n\nThis setup might be good enough for those developing simulators, as that requires less image modification. But once again, if you are serious about this, why not just let your computer build the \u003c\u003cqemu-buildroot-setup,full featured setup\u003e\u003e while you take a coffee or a nap? :-)\n\n==== Prebuilt setup getting started\n\nCheckout to the latest tag and use the Ubuntu packaged QEMU to boot Linux:\n\n....\nsudo apt-get install qemu-system-x86\ngit clone https://github.com/cirosantilli/linux-kernel-module-cheat\ncd linux-kernel-module-cheat\ngit checkout \"$(git rev-list --tags --max-count=1)\"\n./release-download-latest\nunzip lkmc-*.zip\n./run --qemu-which host\n....\n\nYou have to checkout to the latest tag to ensure that the scripts match the release format: https://stackoverflow.com/questions/1404796/how-to-get-the-latest-tag-name-in-current-branch-in-git\n\nThis is known not to work for aarch64 on an Ubuntu 16.04 host with QEMU 2.5.0, presumably because QEMU is too old, the terminal does not show any output. I haven't investigated why.\n\nOr to run a baremetal example instead:\n\n....\n./run \\\n --arch aarch64 \\\n --baremetal userland/c/hello.c \\\n --qemu-which host \\\n;\n....\n\nBe saner and use our custom built QEMU instead:\n\n....\npython3 -m venv .venv\n. .venv/bin/activate\n./setup\n./build --download-dependencies qemu\n./run\n....\n\nTo build the kernel modules as in \u003c\u003cyour-first-kernel-module-hack\u003e\u003e do:\n\n....\ngit submodule update --depth 1 --init --recursive \"$(./getvar linux_source_dir)\"\n./build-linux --no-modules-install -- modules_prepare\n./build-modules --gcc-which host\n./run\n....\n\nTODO: for now the only way to test those modules out without \u003c\u003cqemu-buildroot-setup-getting-started,building Buildroot\u003e\u003e is with 9p, since we currently rely on Buildroot to manipulate the root filesystem.\n\nCommand explanation:\n\n* `modules_prepare` does the minimal build procedure required on the kernel for us to be able to compile the kernel modules, and is way faster than doing a full kernel build. A full kernel build would also work however.\n* `--gcc-which host` selects your host Ubuntu packaged GCC, since you don't have the Buildroot toolchain\n* `--no-modules-install` is required otherwise the `make modules_install` target we run by default fails, since the kernel wasn't built\n\nTo modify the Linux kernel, build and use it as usual:\n\n....\ngit submodule update --depth 1 --init --recursive \"$(./getvar linux_source_dir)\"\n./build-linux\n./run\n....\n\n////\nFor gem5, do:\n\n....\ngit submodule update --init --depth 1 \"$(./getvar linux_source_dir)\"\nsudo apt-get install qemu-utils\n./build-gem5\n./run --emulator gem5 --qemu-which host\n....\n\n`qemu-utils` is required because we currently distribute `.qcow2` files which \u003c\u003cgem5-qcow2,gem5 can't handle\u003e\u003e, so we need `qemu-img` to extract them first.\n\nThe Linux kernel is required for `extract-vmlinux` to convert the compressed kernel image which QEMU understands into the raw vmlinux that gem5 understands: https://superuser.com/questions/298826/how-do-i-uncompress-vmlinuz-to-vmlinux\n////\n\n////\n[[ubuntu]]\n=== Ubuntu guest setup\n\n==== About the Ubuntu guest setup\n\nThis setup is similar to \u003c\u003cprebuilt\u003e\u003e, but instead of using Buildroot for the root filesystem, it downloads an Ubuntu image with Docker, and uses that as the root filesystem.\n\nThe rationale for choice of Ubuntu as a second distribution in addition to Buildroot can be found at: xref:linux-distro-choice[xrefstyle=full]\n\nAdvantages over Buildroot:\n\n* saves build time\n* you get to play with a huge selection of Debian packages out of the box\n* more representative of most non-embedded production systems than BusyBox\n\nDisadvantages:\n\n* less visibility: https://askubuntu.com/questions/82302/how-to-compile-ubuntu-from-source-code The fact that that question has no answer makes me cringe\n* less compatibility, e.g. no one knows what the officially supported cross compilers are: https://askubuntu.com/questions/1046294/what-are-the-officially-supported-cross-compilers-for-ubuntu-server-alternative\n\nDocker is used here just as an image download provider since it has a wide variety of images. Why we don't just download the regular Ubuntu disk image:\n\n* that image is not ready to boot, but rather goes into an interactive installer: https://askubuntu.com/questions/884534/how-to-run-ubuntu-16-04-desktop-on-qemu/1046792#1046792\n* the default Ubuntu image has a large collection of software, and is large. The docker version is much more minimal.\n\nOne alternative would be to use https://wiki.ubuntu.com/Base[Ubuntu base] which can be downloaded from: http://cdimage.ubuntu.com/ubuntu-base That provides a `.tgz` and comes very close to what we obtain with Docker, but without the need for `sudo`.\n\n==== Ubuntu guest setup getting started\n\nTODO\n\n....\nsudo ./build-docker\n./run --docker\n....\n\n`sudo` is required for Docker operations: https://askubuntu.com/questions/477551/how-can-i-use-docker-without-sudo\n////\n\n[[host]]\n=== Host kernel module setup\n\n**THIS IS DANGEROUS (AND FUN), YOU HAVE BEEN WARNED**\n\nThis method runs the kernel modules directly on your host computer without a VM, and saves you the compilation time and disk usage of the virtual machine method.\n\nIt has however severe limitations:\n\n* can't control which kernel version and build options to use. So some of the modules will likely not compile because of kernel API changes, since https://stackoverflow.com/questions/37098482/how-to-build-a-linux-kernel-module-so-that-it-is-compatible-with-all-kernel-rele/45429681#45429681[the Linux kernel does not have a stable kernel module API].\n* bugs can easily break you system. E.g.:\n** segfaults can trivially lead to a kernel crash, and require a reboot\n** your disk could get erased. Yes, this can also happen with `sudo` from userland. But you should not use `sudo` when developing newbie programs. And for the kernel you don't have the choice not to use `sudo`.\n** even more subtle system corruption such as https://unix.stackexchange.com/questions/78858/cannot-remove-or-reinsert-kernel-module-after-error-while-inserting-it-without-r[not being able to rmmod]\n* can't control which hardware is used, notably the CPU architecture\n* can't step debug it with \u003c\u003cgdb,GDB\u003e\u003e easily. The alternatives are https://en.wikipedia.org/wiki/JTAG[JTAG] or \u003c\u003ckgdb\u003e\u003e, but those are less reliable, and require extra hardware.\n\nStill interested?\n\n....\n./build-modules --host\n....\n\nCompilation will likely fail for some modules because of kernel or toolchain differences that we can't control on the host.\n\nThe best workaround is to compile just your modules with:\n\n....\n./build-modules --host -- hello hello2\n....\n\nwhich is equivalent to:\n\n....\n./build-modules \\\n --gcc-which host \\\n --host \\\n -- \\\n kernel_modules/hello.c \\\n kernel_modules/hello2.c \\\n;\n....\n\nOr just remove the `.c` extension from the failing files and try again:\n\n....\ncd \"$(./getvar kernel_modules_source_dir)\"\nmv broken.c broken.c~\n....\n\nOnce you manage to compile, and have come to terms with the fact that this may blow up your host, try it out with:\n\n....\ncd \"$(./getvar kernel_modules_build_host_subdir)\"\nsudo insmod hello.ko\n\n# Our module is there.\nsudo lsmod | grep hello\n\n# Last message should be: hello init\ndmesg -T\n\nsudo rmmod hello\n\n# Last message should be: hello exit\ndmesg -T\n\n# Not present anymore\nsudo lsmod | grep hello\n....\n\n==== Hello host\n\nMinimal host build system example:\n\n....\ncd hello_host_kernel_module\nmake\nsudo insmod hello.ko\ndmesg\nsudo rmmod hello.ko\ndmesg\n....\n\n=== Userland setup\n\n==== About the userland setup\n\nIn order to test the kernel and emulators, userland content in the form of executables and scripts is of course required, and we store it mostly under:\n\n* link:userland/[]\n* \u003c\u003crootfs-overlay\u003e\u003e\n* \u003c\u003cadd-new-buildroot-packages\u003e\u003e\n\nWhen we started this repository, it only contained content that interacted very closely with the kernel, or that had required performance analysis.\n\nHowever, we soon started to notice that this had an increasing overlap with other userland test repositories: we were duplicating build and test infrastructure and even some examples.\n\nTherefore, we decided to consolidate other userland tutorials that we had scattered around into this repository.\n\nNotable userland content included / moving into this repository includes:\n\n* \u003c\u003cuserland-assembly\u003e\u003e\n* \u003c\u003cc\u003e\u003e\n* \u003c\u003ccpp\u003e\u003e\n* \u003c\u003cposix\u003e\u003e\n* \u003c\u003calgorithms\u003e\u003e\n\n==== Userland setup getting started\n\nThere are several ways to run our \u003c\u003cuserland-content\u003e\u003e, notably:\n\n* natively on the host as shown at: xref:userland-setup-getting-started-natively[xrefstyle=full]\n+\nCan only run examples compatible with your host CPU architecture and OS, but has the fastest setup and runtimes.\n* from user mode simulation with:\n+\n--\n** the host prebuilt toolchain: xref:userland-setup-getting-started-with-prebuilt-toolchain-and-qemu-user-mode[xrefstyle=full]\n** the Buildroot toolchain you built yourself: xref:qemu-user-mode-getting-started[xrefstyle=full]\n--\n+\nThis setup:\n+\n--\n** can run most examples, including those for other CPU architectures, with the notable exception of examples that rely on kernel modules\n** can run reproducible approximate performance experiments with gem5, see e.g. \u003c\u003cbst-vs-heap-vs-hashmap\u003e\u003e\n--\n* from full system simulation as shown at: xref:qemu-buildroot-setup-getting-started[xrefstyle=full].\n+\nThis is the most reproducible and controlled environment, and all examples work there. But also the slower one to setup.\n\n===== Userland setup getting started natively\n\nWith this setup, we will use the host toolchain and execute executables directly on the host.\n\nNo toolchain build is required, so you can just download your distro toolchain and jump straight into it.\n\nBuild, run and example, and clean it in-tree with:\n\n....\nsudo apt-get install gcc\ncd userland\n./build c/hello\n./c/hello.out\n./build --clean\n....\n\nSource: link:userland/c/hello.c[].\n\nBuild an entire directory and test it:\n\n....\ncd userland\n./build c\n./test c\n....\n\nBuild the current directory and test it:\n\n....\ncd userland/c\n./build\n./test\n....\n\nAs mentioned at \u003c\u003cuserland-libs-directory\u003e\u003e, tests under link:userland/libs[] require certain optional libraries to be installed, and are not built or tested by default.\n\nYou can install those libraries with:\n\n....\ncd linux-kernel-module-cheat\npython3 -m venv .venv\n. .venv/bin/activate\n./setup\n./build --download-dependencies userland-host\n....\n\nand then build the examples and test with:\n\n....\n./build --package-all\n./test --package-all\n....\n\nPass custom compiler options:\n\n....\n./build --ccflags='-foptimize-sibling-calls -foptimize-strlen' --force-rebuild\n....\n\nHere we used `--force-rebuild` to force rebuild since the sources weren't modified since the last build.\n\nSome CLI options have more specialized flags, e.g. `-O` for the \u003c\u003coptimization-level-of-a-build\u003e\u003e:\n\n....\n./build --optimization-level 3 --force-rebuild\n....\n\nSee also \u003c\u003cuser-mode-static-executables\u003e\u003e for `--static`.\n\nThe `build` scripts inside link:userland/[] are just symlinks to link:build-userland-in-tree[] which you can also use from toplevel as:\n\n....\n./build-userland-in-tree\n./build-userland-in-tree userland/c\n./build-userland-in-tree userland/c/hello.c\n....\n\n`build-userland-in-tree` is in turn just a thin wrapper around link:build-userland[]:\n\n....\n./build-userland --gcc-which host --in-tree userland/c\n....\n\nSo you can use any option supported by `build-userland` script freely with `build-userland-in-tree` and `build`.\n\nThe situation is analogous for link:userland/test[], link:test-executables-in-tree[] and link:test-executables[], which are further documented at: xref:user-mode-tests[xrefstyle=full].\n\nDo a more clean out-of-tree build instead and run the program:\n\n....\n./build-userland --gcc-which host --userland-build-id host\n./run --emulator native --userland userland/c/hello.c --userland-build-id host\n....\n\nHere we:\n\n* put the host executables in a separate \u003c\u003cbuild-variants,build variant\u003e\u003e to avoid conflict with Buildroot builds.\n* ran with the `--emulator native` option to run the program natively\n\nIn this case you can debub the program with:\n\n....\n./run --debug-vm --emulator native --userland userland/c/hello.c --userland-build-id host\n....\n\nas shown at: xref:debug-the-emulator[xrefstyle=full], although direct GDB host usage works as well of course.\n\n===== Userland setup getting started with prebuilt toolchain and QEMU user mode\n\nIf you are lazy to built the Buildroot toolchain and QEMU, but want to run e.g. ARM \u003c\u003cuserland-assembly\u003e\u003e in \u003c\u003cuser-mode-simulation\u003e\u003e, you can get away on Ubuntu 18.04 with just:\n\n....\nsudo apt-get install gcc-aarch64-linux-gnu qemu-system-aarch64\n./build-userland \\\n --arch aarch64 \\\n --gcc-which host \\\n --userland-build-id host \\\n;\n./run \\\n --arch aarch64 \\\n --qemu-which host \\\n --userland-build-id host \\\n --userland userland/c/command_line_arguments.c \\\n --cli-args 'asdf \"qw er\"' \\\n;\n....\n\nwhere:\n\n* `--gcc-which host`: use the host toolchain.\n+\nWe must pass this to `./run` as well because QEMU must know which dynamic libraries to use. See also: xref:user-mode-static-executables[xrefstyle=full].\n* `--userland-build-id host`: put the host built into a \u003c\u003cbuild-variants\u003e\u003e\n\nThis present the usual trade-offs of using prebuilts as mentioned at: xref:prebuilt[xrefstyle=full].\n\nOther functionality are analogous, e.g. testing:\n\n....\n./test-executables \\\n --arch aarch64 \\\n --gcc-which host \\\n --qemu-which host \\\n --userland-build-id host \\\n;\n....\n\nand \u003c\u003cuser-mode-gdb\u003e\u003e:\n\n....\n./run \\\n --arch aarch64 \\\n --gdb \\\n --gcc-which host \\\n --qemu-which host \\\n --userland-build-id host \\\n --userland userland/c/command_line_arguments.c \\\n --cli-args 'asdf \"qw er\"' \\\n;\n....\n\n===== Userland setup getting started full system\n\nFirst ensure that \u003c\u003cqemu-buildroot-setup\u003e\u003e is working.\n\nAfter doing that setup, you can already execute your userland programs from inside QEMU: the only missing step is how to rebuild executables and run them.\n\nAnd the answer is exactly analogous to what is shown at: xref:your-first-kernel-module-hack[xrefstyle=full]\n\nFor example, if we modify link:userland/c/hello.c[] to print out something different, we can just rebuild it with:\n\n....\n./build-userland\n....\n\nSource: link:build-userland[]. `./build` calls that script automatically for us when doing the initial full build.\n\nNow, run the program either without rebooting use the \u003c\u003c9p\u003e\u003e mount:\n\n....\n/mnt/9p/out_rootfs_overlay/c/hello.out\n....\n\nor shutdown QEMU, add the executable to the root filesystem:\n\n....\n./build-buildroot\n....\n\nreboot and use the root filesystem as usual:\n\n....\n./hello.out\n....\n\n=== Baremetal setup\n\n==== About the baremetal setup\n\nThis setup does not use the Linux kernel nor Buildroot at all: it just runs your very own minimal OS.\n\n`x86_64` is not currently supported, only `arm` and `aarch64`: I had made some x86 bare metal examples at: https://github.com/cirosantilli/x86-bare-metal-examples but I'm lazy to port them here now. Pull requests are welcome.\n\nThe main reason this setup is included in this project, despite the word \"Linux\" being on the project name, is that a lot of the emulator boilerplate can be reused for both use cases.\n\nThis setup allows you to make a tiny OS and that runs just a few instructions, use it to fully control the CPU to better understand the simulators for example, or develop your own OS if you are into that.\n\nYou can also use C and a subset of the C standard library because we enable https://en.wikipedia.org/wiki/Newlib[Newlib] by default. See also:\n\n* https://electronics.stackexchange.com/questions/223929/c-standard-libraries-on-bare-metal/400077#400077\n* https://stackoverflow.com/questions/13063055/does-a-libc-os-exist/59771531#59771531\n\nOur C bare-metal compiler is built with https://github.com/crosstool-ng/crosstool-ng[crosstool-NG]. If you have already built \u003c\u003cqemu-buildroot-setup,Buildroot\u003e\u003e previously, you will end up with two GCCs installed. Unfortunately I don't see a solution for this, since we need separate toolchains for Newlib on baremetal and glibc on Linux: https://stackoverflow.com/questions/38956680/difference-between-arm-none-eabi-and-arm-linux-gnueabi/38989869#38989869\n\n==== Baremetal setup getting started\n\nEvery `.c` file inside link:baremetal/[] and `.S` file inside `baremetal/arch/\u003carch\u003e/` generates a separate baremetal image.\n\nFor example, to run link:baremetal/arch/aarch64/dump_regs.c[] in QEMU do:\n\n....\npython3 -m venv .venv\n. .venv/bin/activate\n./setup\n./build --arch aarch64 --download-dependencies qemu-baremetal\n./run --arch aarch64 --baremetal baremetal/arch/aarch64/dump_regs.c\n....\n\nAnd the terminal prints the values of certain system registers. This example prints registers that are only accessible from \u003c\u003carm-exception-levels,EL1\u003e\u003e or higher, and thus could not be run in userland.\n\nIn addition to the examples under link:baremetal/[], several of the \u003c\u003cuserland-content,userland examples\u003e\u003e can also be run in baremetal! This is largely due to the \u003c\u003cabout-the-baremetal-setup,awesomeness of Newlib\u003e\u003e.\n\nThe examples that work include most \u003c\u003cc,C examples\u003e\u003e that don't rely on complicated syscalls such as threads, and almost all the \u003c\u003cuserland-assembly\u003e\u003e examples.\n\nThe exact list of userland programs that work in baremetal is specified in \u003c\u003cpath-properties\u003e\u003e with the `baremetal` property, but you can also easily find it out with a \u003c\u003cbaremetal-tests,baremetal test dry run\u003e\u003e:\n\n....\n./test-executables --arch aarch64 --dry-run --mode baremetal\n....\n\nFor example, we can run the C hello world link:userland/c/hello.c[] simply as:\n\n....\n./run --arch aarch64 --baremetal userland/c/hello.c\n....\n\nand that outputs to the serial port the string:\n\n....\nhello\n....\n\nwhich QEMU shows on the host terminal.\n\nTo modify a baremetal program, simply edit the file, e.g.\n\n....\nvim userland/c/hello.c\n....\n\nand rebuild:\n\n....\n./build-baremetal --arch aarch64\n./run --arch aarch64 --baremetal userland/c/hello.c\n....\n\n`./build qemu-baremetal` that we run previously is only needed for the initial build. That script calls link:build-baremetal[] for us, in addition to building prerequisites such as QEMU and crosstool-NG.\n\n`./build-baremetal` uses crosstool-NG, and so it must be preceded by link:build-crosstool-ng[], which `./build qemu-baremetal` also calls.\n\nNow let's run link:userland/arch/aarch64/add.S[]:\n\n....\n./run --arch aarch64 --baremetal userland/arch/aarch64/add.S\n....\n\nThis time, the terminal does not print anything, which indicates success: if you look into the source, you will see that we just have an assertion there.\n\nYou can see a sample assertion fail in link:userland/c/assert_fail.c[]:\n\n....\n./run --arch aarch64 --baremetal userland/c/assert_fail.c\n....\n\nand the terminal contains:\n\n....\nlkmc_exit_status_134\nerror: simulation error detected by parsing logs\n....\n\nand the exit status of our script is 1:\n\n....\necho $?\n....\n\nYou can run all the baremetal examples in one go and check that all assertions passed with:\n\n....\n./test-executables --arch aarch64 --mode baremetal\n....\n\nTo use gem5 instead of QEMU do:\n\n....\npython3 -m venv .venv\n. .venv/bin/activate\n./setup\n./build --download-dependencies gem5-baremetal\n./run --arch aarch64 --baremetal userland/c/hello.c --emulator gem5\n....\n\nand then \u003c\u003cqemu-buildroot-setup,as usual\u003e\u003e open a shell with:\n\n....\n./gem5-shell\n....\n\nOr as usual, \u003c\u003ctmux\u003e\u003e users can do both in one go with:\n\n....\n./run --arch aarch64 --baremetal userland/c/hello.c --emulator gem5 --tmux\n....\n\nTODO: the carriage returns are a bit different than in QEMU, see: xref:gem5-baremetal-carriage-return[xrefstyle=full].\n\nNote that `./build-baremetal` requires the `--emulator gem5` option, and generates separate executable images for both, as can be seen from:\n\n....\necho \"$(./getvar --arch aarch64 --baremetal userland/c/hello.c --emulator qemu image)\"\necho \"$(./getvar --arch aarch64 --baremetal userland/c/hello.c --emulator gem5 image)\"\n....\n\nThis is unlike the Linux kernel that has a single image for both QEMU and gem5:\n\n....\necho \"$(./getvar --arch aarch64 --emulator qemu image)\"\necho \"$(./getvar --arch aarch64 --emulator gem5 image)\"\n....\n\nThe reason for that is that on baremetal we don't parse the \u003c\u003cdevice-tree,device tress\u003e\u003e from memory like the Linux kernel does, which tells the kernel for example the UART address, and many other system parameters.\n\n`gem5` also supports the `RealViewPBX` machine, which represents an older hardware compared to the default `VExpress_GEM5_V1`:\n\n....\n./build-baremetal --arch aarch64 --emulator gem5 --machine RealViewPBX\n./run --arch aarch64 --baremetal userland/c/hello.c --emulator gem5 --machine RealViewPBX\n....\n\nsee also: xref:gem5-arm-platforms[xrefstyle=full].\n\nThis generates yet new separate images with new magic constants:\n\n....\necho \"$(./getvar --arch aarch64 --baremetal userland/c/hello.c --emulator gem5 --machine VExpress_GEM5_V1 image)\"\necho \"$(./getvar --arch aarch64 --baremetal userland/c/hello.c --emulator gem5 --machine RealViewPBX image)\"\n....\n\nBut just stick to newer and better `VExpress_GEM5_V1` unless you have a good reason to use `RealViewPBX`.\n\nWhen doing baremetal programming, it is likely that you will want to learn userland assembly first, see: xref:userland-assembly[xrefstyle=full].\n\nFor more information on baremetal, see the section: xref:baremetal[xrefstyle=full].\n\nThe following subjects are particularly important:\n\n* \u003c\u003ctracing\u003e\u003e\n* \u003c\u003cbaremetal-gdb-step-debug\u003e\u003e\n\n=== Build the documentation\n\nYou don't need to depend on GitHub.\n\nFor a quick and dirty build, install https://asciidoctor.org/[Asciidoctor] however you like and build:\n\n....\nasciidoctor README.adoc\nxdg-open README.html\n....\n\nFor development, you will want to do a more controlled build with extra error checking as follows.\n\nTODO: get this working seamlessly on Docker. For now some quick instructions for host building. For the initial build, first install RVM and Ruby as per https://www.rvm.io/rvm/install[]:\n\n....\n\\curl -sSL https://get.rvm.io | bash\nrvm install 3.2.3\n....\n\nThe TODO Docker instructions which are not yet working should look simply something like this:\n\n....\n./run-docker\n./build --download-dependencies doc\n....\n\nwhich also downloads build dependencies.\n\nThen the following times just to the faster:\n\n....\n./build-doc\n....\n\nSource: link:build-doc[]\n\nThe HTML output is located at:\n\n....\nxdg-open out/README.html\n....\n\nMore information about our documentation internals can be found at: xref:documentation[xrefstyle=full]\n\n[[gdb]]\n== GDB step debug\n\n=== GDB step debug kernel boot\n\n`--gdb-wait` makes QEMU and gem5 wait for a GDB connection, otherwise we could accidentally go past the point we want to break at:\n\n....\n./run --gdb-wait\n....\n\nSay you want to break at `start_kernel`. So on another shell:\n\n....\n./run-gdb start_kernel\n....\n\nor at a given line:\n\n....\n./run-gdb init/main.c:1088\n....\n\nNow QEMU will stop there, and you can use the normal GDB commands:\n\n....\nlist\nnext\ncontinue\n....\n\nSee also:\n\n* https://stackoverflow.com/questions/11408041/how-to-debug-the-linux-kernel-with-gdb-and-qemu/33203642#33203642\n* https://stackoverflow.com/questions/4943857/linux-kernel-live-debugging-how-its-done-and-what-tools-are-used/42316607#42316607\n\n==== GDB step debug kernel boot other archs\n\nJust don't forget to pass `--arch` to `./run-gdb`, e.g.:\n\n....\n./run --arch aarch64 --gdb-wait\n....\n\nand:\n\n....\n./run-gdb --arch aarch64 start_kernel\n....\n\n[[kernel-o0]]\n==== Disable kernel compiler optimizations\n\nhttps://stackoverflow.com/questions/29151235/how-to-de-optimize-the-linux-kernel-to-and-compile-it-with-o0\n\n`O=0` is an impossible dream, `O=2` being the default.\n\nSo get ready for some weird jumps, and `\u003cvalue optimized out\u003e` fun. Why, Linux, why.\n\nThe `-O` level of some other userland content can be controlled as explained at: \u003c\u003coptimization-level-of-a-build\u003e\u003e.\n\n=== GDB step debug kernel post-boot\n\nLet's observe the kernel `write` system call as it reacts to some userland actions.\n\nStart QEMU with just:\n\n....\n./run\n....\n\nand after boot inside a shell run:\n\n....\n./count.sh\n....\n\nwhich counts to infinity to stdout. Source: link:rootfs_overlay/lkmc/count.sh[].\n\nThen in another shell, run:\n\n....\n./run-gdb\n....\n\nand then hit:\n\n....\nCtrl-C\nbreak __x64_sys_write\ncontinue\ncontinue\ncontinue\n....\n\nAnd you now control the counting on the first shell from GDB!\n\nBefore v4.17, the symbol name was just `sys_write`, the change happened at https://github.com/torvalds/linux/commit/d5a00528b58cdb2c71206e18bd021e34c4eab878[d5a00528b58cdb2c71206e18bd021e34c4eab878]. As of Linux v 4.19, the function is called `sys_write` in `arm`, and `__arm64_sys_write` in `aarch64`. One good way to find it if the name changes again is to try:\n\n....\nrbreak .*sys_write\n....\n\nor just have a quick look at the sources!\n\nWhen you hit `Ctrl-C`, if we happen to be inside kernel code at that point, which is very likely if there are no heavy background tasks waiting, and we are just waiting on a `sleep` type system call of the command prompt, we can already see the source for the random place inside the kernel where we stopped.\n\n=== tmux\n\ntmux just makes things even more fun by allowing us to see both the terminal for:\n\n* emulator stdout\n* \u003c\u003cgdb\u003e\u003e\n\nat once without dragging windows around!\n\nFirst start `tmux` with:\n\n....\ntmux\n....\n\nNow that you are inside a shell inside tmux, you can start GDB simply with:\n\n....\n./run --gdb\n....\n\nwhich is just a convenient shortcut for:\n\n....\n./run --gdb-wait --tmux --tmux-args start_kernel\n....\n\nThis splits the terminal into two panes:\n\n* left: usual QEMU with terminal\n* right: GDB\n\nand focuses on the GDB pane.\n\nNow you can navigate with the usual tmux shortcuts:\n\n* switch between the two panes with: `Ctrl-B O`\n* close either pane by killing its terminal with `Ctrl-D` as usual\n\nSee the tmux manual for further details:\n\n....\nman tmux\n....\n\nTo start again, switch back to the QEMU pane with `Ctrl-O`, kill the emulator, and re-run:\n\n....\n./run --gdb\n....\n\nThis automatically clears the GDB pane, and starts a new one.\n\nThe option `--tmux-args` determines which options will be passed to the program running on the second tmux pane, and is equivalent to:\n\nThis is equivalent to:\n\n....\n./run --gdb-wait\n./run-gdb start_kernel\n....\n\nDue to Python's CLI parsing quicks, if the link:run-gdb[] arguments start with a dash `-`, you have to use the `=` sign, e.g. to \u003c\u003cgdb-step-debug-early-boot\u003e\u003e:\n\n....\n./run --gdb --tmux-args=--no-continue\n....\n\nBibliography: https://unix.stackexchange.com/questions/152738/how-to-split-a-new-window-and-run-a-command-in-this-new-window-using-tmux/432111#432111\n\n==== tmux gem5\n\nIf you are using gem5 instead of QEMU, `--tmux` has a different effect by default: it opens the gem5 terminal instead of the debugger:\n\n....\n./run --emulator gem5 --tmux\n....\n\nTo open a new pane with GDB instead of the terminal, use:\n\n....\n./run --gdb\n....\n\nwhich is equivalent to:\n\n....\n./run --emulator gem5 --gdb-wait --tmux --tmux-args start_kernel --tmux-program gdb\n....\n\n`--tmux-program` implies `--tmux`, so we can just write:\n\n....\n./run --emulator gem5 --gdb-wait --tmux-program gdb\n....\n\nIf you also want to see both GDB and the terminal with gem5, then you will need to open a separate shell manually as usual with `./gem5-shell`.\n\nFrom inside tmux, you can create new terminals on a new window with `Ctrl-B C` split a pane yet again vertically with `Ctrl-B %` or horizontally with `Ctrl-B \"`.\n\n=== GDB step debug kernel module\n\nhttps://stackoverflow.com/questions/28607538/how-to-debug-linux-kernel-modules-with-qemu/44095831#44095831\n\nLoadable kernel modules are a bit trickier since the kernel can place them at different memory locations depending on load order.\n\nSo we cannot set the breakpoints before `insmod`.\n\nHowever, the Linux kernel GDB scripts offer the `lx-symbols` command, which takes care of that beautifully for us.\n\nShell 1:\n\n....\n./run\n....\n\nWait for the boot to end and run:\n\n....\ninsmod timer.ko\n....\n\nSource: link:kernel_modules/timer.c[].\n\nThis prints a message to dmesg every second.\n\nShell 2:\n\n....\n./run-gdb\n....\n\nIn GDB, hit `Ctrl-C`, and note how it says:\n\n....\nscanning for modules in /root/linux-kernel-module-cheat/out/kernel_modules/x86_64/kernel_modules\nloading @0xffffffffc0000000: /root/linux-kernel-module-cheat/out/kernel_modules/x86_64/kernel_modules/timer.ko\n....\n\nThat's `lx-symbols` working! Now simply:\n\n....\nbreak lkmc_timer_callback\ncontinue\ncontinue\ncontinue\n....\n\nand we now control the callback from GDB!\n\nJust don't forget to remove your breakpoints after `rmmod`, or they will point to stale memory locations.\n\nTODO: why does `break work_func` for `insmod kthread.ko` not very well? Sometimes it breaks but not others.\n\n[[gdb-step-debug-kernel-module-arm]]\n==== GDB step debug kernel module insmodded by init on ARM\n\nTODO on `arm` 51e31cdc2933a774c2a0dc62664ad8acec1d2dbe it does not always work, and `lx-symbols` fails with the message:\n\n....\nloading vmlinux\nTraceback (most recent call last):\n File \"/linux-kernel-module-cheat//out/arm/buildroot/build/linux-custom/scripts/gdb/linux/symbols.py\", line 163, in invoke\n self.load_all_symbols()\n File \"/linux-kernel-module-cheat//out/arm/buildroot/build/linux-custom/scripts/gdb/linux/symbols.py\", line 150, in load_all_symbols\n [self.load_module_symbols(module) for module in module_list]\n File \"/linux-kernel-module-cheat//out/arm/buildroot/build/linux-custom/scripts/gdb/linux/symbols.py\", line 110, in load_module_symbols\n module_name = module['name'].string()\ngdb.MemoryError: Cannot access memory at address 0xbf0000cc\nError occurred in Python command: Cannot access memory at address 0xbf0000cc\n....\n\nCan't reproduce on `x86_64` and `aarch64` are fine.\n\nIt is kind of random: if you just `insmod` manually and then immediately `./run-gdb --arch arm`, then it usually works.\n\nBut this fails most of the time: shell 1:\n\n....\n./run --arch arm --eval-after 'insmod hello.ko'\n....\n\nshell 2:\n\n....\n./run-gdb --arch arm\n....\n\nthen hit `Ctrl-C` on shell 2, and voila.\n\nThen:\n\n....\ncat /proc/modules\n....\n\nsays that the load address is:\n\n....\n0xbf000000\n....\n\nso it is close to the failing `0xbf0000cc`.\n\n`readelf`:\n\n....\n./run-toolchain readelf -- -s \"$(./getvar kernel_modules_build_subdir)/hello.ko\"\n....\n\ndoes not give any interesting hits at `cc`, no symbol was placed that far.\n\n[[gdb-module-init]]\n==== GDB module_init\n\nTODO find a more convenient method. We have working methods, but they are not ideal.\n\nThis is not very easy, since by the time the module finishes loading, and `lx-symbols` can work properly, `module_init` has already finished running!\n\nPossibly asked at:\n\n* https://stackoverflow.com/questions/37059320/debug-a-kernel-module-being-loaded\n* https://stackoverflow.com/questions/11888412/debug-the-init-module-call-of-a-linux-kernel-module\n\n[[gdb-module-init-step-into-it]]\n===== GDB module_init step into it\n\nThis is the best method we've found so far.\n\nThe kernel calls `module_init` synchronously, therefore it is not hard to step into that call.\n\nAs of 4.16, the call happens in `do_one_initcall`, so we can do in shell 1:\n\n....\n./run\n....\n\nshell 2 after boot finishes (because there are other calls to `do_init_module` at boot, presumably for the built-in modules):\n\n....\n./run-gdb do_one_initcall\n....\n\nthen step until the line:\n\n....\n833 ret = fn();\n....\n\nwhich does the actual call, and then step into it.\n\nFor the next time, you can also put a breakpoint there directly:\n\n....\n./run-gdb init/main.c:833\n....\n\nHow we found this out: first we got \u003c\u003cgdb-module-init-calculate-entry-address\u003e\u003e working, and then we did a `bt`. AKA cheating :-)\n\n[[gdb-module-init-calculate-entry-address]]\n===== GDB module_init calculate entry address\n\nThis works, but is a bit annoying.\n\nThe key observation is that the load address of kernel modules is deterministic: there is a pre allocated memory region https://www.kernel.org/doc/Documentation/x86/x86_64/mm.txt \"module mapping space\" filled from bottom up.\n\nSo once we find the address the first time, we can just reuse it afterwards, as long as we don't modify the module.\n\nDo a fresh boot and get the module:\n\n....\n./run --eval-after './pr_debug.sh;insmod fops.ko;./linux/poweroff.out'\n....\n\nThe boot must be fresh, because the load address changes every time we insert, even after removing previous modules.\n\nThe base address shows on terminal:\n\n....\n0xffffffffc0000000 .text\n....\n\nNow let's find the offset of `myinit`:\n\n....\n./run-toolchain readelf -- \\\n -s \"$(./getvar kernel_modules_build_subdir)/fops.ko\" | \\\n grep myinit\n....\n\nwhich gives:\n\n....\n 30: 0000000000000240 43 FUNC LOCAL DEFAULT 2 myinit\n....\n\nso the offset address is `0x240` and we deduce that the function will be placed at:\n\n....\n0xffffffffc0000000 + 0x240 = 0xffffffffc0000240\n....\n\nNow we can just do a fresh boot on shell 1:\n\n....\n./run --eval 'insmod fops.ko;./linux/poweroff.out' --gdb-wait\n....\n\nand on shell 2:\n\n....\n./run-gdb '*0xffffffffc0000240'\n....\n\nGDB then breaks, and `lx-symbols` works.\n\n[[gdb-module-init-break-at-the-end-of-sys-init-module]]\n===== GDB module_init break at the end of sys_init_module\n\nTODO not working. This could be potentially very convenient.\n\nThe idea here is to break at a point late enough inside `sys_init_module`, at which point `lx-symbols` can be called and do its magic.\n\nBeware that there are both `sys_init_module` and `sys_finit_module` syscalls, and `insmod` uses `fmodule_init` by default.\n\nBoth call `do_module_init` however, which is what `lx-symbols` hooks to.\n\nIf we try:\n\n....\nb sys_finit_module\n....\n\nthen hitting:\n\n....\nn\n....\n\ndoes not break, and insertion happens, likely because of optimizations? \u003c\u003ckernel-o0\u003e\u003e\n\nThen we try:\n\n....\nb do_init_module\n....\n\nA naive:\n\n....\nfin\n....\n\nalso fails to break!\n\nFinally, in despair we notice that \u003c\u003cpr-debug\u003e\u003e prints the kernel load address as explained at \u003c\u003cbypass-lx-symbols\u003e\u003e.\n\nSo, if we set a breakpoint just after that message is printed by searching where that happens on the Linux source code, we must be able to get the correct load address before `init_module` happens.\n\n[[gdb-module-init-add-trap-instruction]]\n===== GDB module_init add trap instruction\n\nThis is another possibility: we could modify the module source by adding a trap instruction of some kind.\n\nThis appears to be described at: https://www.linuxjournal.com/article/4525\n\nBut it refers to a `gdbstart` script which is not in the tree anymore and beyond my `git log` capabilities.\n\nAnd just adding:\n\n....\nasm( \" int $3\");\n....\n\ndirectly gives an \u003c\u003coops,oops\u003e\u003e as I'd expect.\n\n==== Bypass lx-symbols\n\nUseless, but a good way to show how hardcore you are. Disable `lx-symbols` with:\n\n....\n./run-gdb --no-lxsymbols\n....\n\nFrom inside guest:\n\n....\ninsmod timer.ko\ncat /proc/modules\n....\n\nas mentioned at:\n\n* https://stackoverflow.com/questions/6384605/how-to-get-address-of-a-kernel-module-loaded-using-insmod/6385818\n* https://unix.stackexchange.com/questions/194405/get-base-address-and-size-of-a-loaded-kernel-module\n\nThis will give a line of form:\n\n....\nfops 2327 0 - Live 0xfffffffa00000000\n....\n\nAnd then tell GDB where the module was loaded with:\n\n....\nCtrl-C\nadd-symbol-file ../../../rootfs_overlay/x86_64/timer.ko 0xffffffffc0000000\n0xffffffffc0000000\n....\n\nAlternatively, if the module panics before you can read `/proc/modules`, there is a \u003c\u003cpr-debug\u003e\u003e which shows the load address:\n\n....\necho 8 \u003e /proc/sys/kernel/printk\necho 'file kernel/module.c +p' \u003e /sys/kernel/debug/dynamic_debug/control\n./linux/myinsmod.out hello.ko\n....\n\nAnd then search for a line of type:\n\n....\n[ 84.877482] 0xfffffffa00000000 .text\n....\n\nTested on 4f4749148273c282e80b58c59db1b47049e190bf + 1.\n\n=== GDB step debug early boot\n\nTODO successfully debug the very first instruction that the Linux kernel runs, before `start_kernel`!\n\nBreak at the very first instruction executed by QEMU:\n\n....\n./run-gdb --no-continue\n....\n\nNote however that early boot parts appear to be relocated in memory somehow, and therefore:\n\n* you won't see the source location in GDB, only assembly\n* you won't be able to break by symbol in those early locations\n\nFurther discussion at: \u003c\u003clinux-kernel-entry-point\u003e\u003e.\n\nIn the specific case of gem5 aarch64 at least:\n\n* gem5 relocates the kernel in memory to a fixed location, see e.g. https://gem5.atlassian.net/browse/GEM5-787\n* `--param 'system.workload.early_kernel_symbols=True` should in theory duplicate the symbols to the correct physical location, but it was broken at one point: https://gem5.atlassian.net/browse/GEM5-785\n* gem5 executes directly from vmlinux, so there is no decompression code involved, so you actually immediately start running the \"true\" first instruction from `head.S` as described at: https://stackoverflow.com/questions/18266063/does-linux-kernel-have-main-function/33422401#33422401\n* once the MMU gets turned on at kernel symbol `__primary_switched`, the virtual address matches the ELF symbols, and you start seeing correct symbols without the need for `early_kernel_symbols`. This can be observed clearly with `function_trace = True`: https://stackoverflow.com/questions/64049487/how-to-trace-executed-guest-function-symbol-names-with-their-timestamp-in-gem5/64049488#64049488 which produces:\n+\n....\n0: _kernel_flags_le_lo32 (12500)\n12500: __crc_tcp_add_backlog (1000)\n13500: __crc_crypto_alg_tested (6500)\n20000: __crc_tcp_add_backlog (10000)\n30000: __crc_crypto_alg_tested (500)\n30500: __crc_scsi_is_host_device (5000)\n35500: __crc_crypto_alg_tested (1500)\n37000: __crc_scsi_is_host_device (4000)\n41000: __crc_crypto_alg_tested (3000)\n44000: __crc_tcp_add_backlog (263500)\n307500: __crc_crypto_alg_tested (975500)\n1283000: __crc_tcp_add_backlog (77191500)\n78474500: __crc_crypto_alg_tested (1000)\n78475500: __crc_scsi_is_host_device (19500)\n78495000: __crc_crypto_alg_tested (500)\n78495500: __crc_scsi_is_host_device (13500)\n78509000: __primary_switched (14000)\n78523000: memset (21118000)\n99641000: __primary_switched (2500)\n99643500: start_kernel (11000)\n....\n+\nso we see that `__primary_switched` is the first non-trash symbol (non-`__crc_*` and non-`_kernel_flags_*`, which are just informative symbols, not actual executable code)\n\n==== Linux kernel entry point\n\nTODO https://stackoverflow.com/questions/2589845/what-are-the-first-operations-that-the-linux-kernel-executes-on-boot\n\nAs mentioned at: \u003c\u003cgdb-step-debug-early-boot\u003e\u003e, the very first kernel instructions executed appear to be placed into memory at a different location than that of the kernel ELF section.\n\nAs a result, we are unable to break on early symbols such as:\n\n....\n./run-gdb extract_kernel\n./run-gdb main\n....\n\n\u003c\u003cgem5-execall-trace-format\u003e\u003e\u003e\u003e however does show the right symbols however! This could be because \u003c\u003cvmlinux-vs-bzimage-vs-zimage-vs-image,gem5 uses vmlinux to boot\u003e\u003e, which QEMU uses the compressed version, and as mentioned on the Stack Overflow answer, the entry point is actually a tiny decompresser routine.\n\nI also tried to hack `run-gdb` with:\n\n....\n@@ -81,7 +81,7 @@ else\n ${gdb} \\\n -q \\\\\n -ex 'add-auto-load-safe-path $(pwd)' \\\\\n--ex 'file vmlinux' \\\\\n+-ex 'file arch/arm/boot/compressed/vmlinux' \\\\\n -ex 'target remote localhost:${port}' \\\\\n ${brk} \\\n -ex 'continue' \\\\\n....\n\nand no I do have the symbols from `arch/arm/boot/compressed/vmlinux'`, but the breaks still don't work.\n\nv4.19 also added a `CONFIG_HAVE_KERNEL_UNCOMPRESSED=y` option for having the kernel uncompressed which could make following the startup easier, but it is only available on s390. `aarch64` however is already uncompressed by default, so might be the easiest one. See also: xref:vmlinux-vs-bzimage-vs-zimage-vs-image[xrefstyle=full].\n\nYou then need the associated `KERNEL_UNCOMPRESSED` to enable it if available:\n\n....\nconfig KERNEL_UNCOMPRESSED\n bool \"None\"\n depends on HAVE_KERNEL_UNCOMPRESSED\n....\n\n===== arm64 secondary CPU entry point\n\nIn gem5 aarch64 Linux v4.18, experimentally the entry point of secondary CPUs seems to be `secondary_holding_pen` as shown at https://gist.github.com/cirosantilli2/34a7bc450fcb6c1c1a910369be1fdd90\n\nWhat happens is that:\n\n* the bootloader goes in in WFE\n* the kernel writes the entry point to the secondary CPU (the address of `secondary_holding_pen`) with CPU0 at the address given to the kernel in the `cpu-release-addr` of the DTB\n* the kernel wakes up the bootloader with a SEV, and the bootloader boots to the address the kernel told it\n\nThe CPU0 action happens at: https://github.com/cirosantilli/linux/blob/v5.7/arch/arm64/kernel/smp_spin_table.c[]:\n\nHere's the code that writes the address and does SEV:\n\n....\nstatic int smp_spin_table_cpu_prepare(unsigned int cpu)\n{\n\t__le64 __iomem *release_addr;\n\n\tif (!cpu_release_addr[cpu])\n\t\treturn -ENODEV;\n\n\t/*\n\t * The cpu-release-addr may or may not be inside the linear mapping.\n\t * As ioremap_cache will either give us a new mapping or reuse the\n\t * existing linear mapping, we can use it to cover both cases. In\n\t * either case the memory will be MT_NORMAL.\n\t */\n\trelease_addr = ioremap_cache(cpu_release_addr[cpu],\n\t\t\t\t sizeof(*release_addr));\n\tif (!release_addr)\n\t\treturn -ENOMEM;\n\n\t/*\n\t * We write the release address as LE regardless of the native\n\t * endianess of the kernel. Therefore, any boot-loaders that\n\t * read this address need to convert this address to the\n\t * boot-loader's endianess before jumping. This is mandated by\n\t * the boot protocol.\n\t */\n\twriteq_relaxed(__pa_symbol(secondary_holding_pen), release_addr);\n\t__flush_dcache_area((__force void *)release_addr,\n\t\t\t sizeof(*release_addr));\n\n\t/*\n\t * Send an event to wake up the secondary CPU.\n\t */\n\tsev();\n....\n\nand here's the code that reads the value from the DTB:\n\n....\nstatic int smp_spin_table_cpu_init(unsigned int cpu)\n{\n\tstruct device_node *dn;\n\tint ret;\n\n\tdn = of_get_cpu_node(cpu, NULL);\n\tif (!dn)\n\t\treturn -ENODEV;\n\n\t/*\n\t * Determine the address from which the CPU is polling.\n\t */\n\tret = of_property_read_u64(dn, \"cpu-release-addr\",\n\t\t\t\t \u0026cpu_release_addr[cpu]);\n....\n\n==== Linux kernel arch-agnostic entry point\n\n`start_kernel` is the first C function to be executed basically: https://stackoverflow.com/questions/18266063/does-kernel-have-main-function/33422401#33422401\n\nFor the earlier arch-specific entry point, see: \u003c\u003clinux-kernel-entry-point\u003e\u003e.\n\n==== Linux kernel early boot messages\n\nWhen booting Linux on a slow emulator like \u003c\u003cgem5\u003e\u003e, what you observe is that:\n\n* first nothing shows for a while\n* then at once, a bunch of message lines show at once followed on aarch64 Linux 5.4.3 by:\n+\n....\n[ 0.081311] printk: console [ttyAMA0] enabled\n....\n\nThis means of course that all the previous messages had been generated earlier and stored, but were only printed to the terminal once the terminal itself was enabled.\n\nNotably for example the very first message:\n\n....\n[ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd070]\n....\n\nhappens very early in the boot process.\n\nIf you get a failure before that, it will be hard to see the print messages.\n\nOne possible solution is to parse the dmesg buffer, gem5 actually implements that: \u003c\u003cgem5-m5out-system-dmesg-file\u003e\u003e.\n\n=== GDB step debug userland processes\n\nQEMU's `-gdb` GDB breakpoints are set on virtual addresses, so you can in theory debug userland processes as well.\n\n* https://stackoverflow.com/questions/26271901/is-it-possible-to-use-gdb-and-qemu-to-debug-linux-user-space-programs-and-kernel\n* https://stackoverflow.com/questions/16273614/debug-init-on-qemu-using-gdb\n\nYou will generally want to use \u003c\u003cgdbserver\u003e\u003e for this as it is more reliable, but this method can overcome the following limitations of `gdbserver`:\n\n* the emulator does not support host to guest networking. This seems to be the case for gem5 as explained at: xref:gem5-host-to-guest-networking[xrefstyle=full]\n* cannot see the start of the `init` process easily\n* `gdbserver` alters the working of the kernel, and makes your run less representative\n\nKnown limitations of direct userland debugging:\n\n* the kernel might switch context to another process or to the kernel itself e.g. on a system call, and then TODO confirm the PIC would go to weird places and source code would be missing.\n+\nSolutions to this are being researched at: xref:lx-ps[xrefstyle=full].\n* TODO step into shared libraries. If I attempt to load them explicitly:\n+\n....\n(gdb) sharedlibrary ../../staging/lib/libc.so.0\nNo loaded shared libraries match the pattern `../../staging/lib/libc.so.0'.\n....\n+\nsince GDB does not know that libc is loaded.\n\n==== GDB step debug userland custom init\n\nThis is the userland debug setup most likely to work, since at init time there is only one userland executable running.\n\nFor executables from the link:userland/[] directory such as link:userland/posix/count.c[]:\n\n* Shell 1:\n+\n....\n./run --gdb-wait --kernel-cli 'init=/lkmc/posix/count.out'\n....\n* Shell 2:\n+\n....\n./run-gdb --userland userland/posix/count.c main\n....\n+\nAlternatively, we could also pass the full path to the executable:\n+\n....\n./run-gdb --userland \"$(./getvar userland_build_dir)/posix/count.out\" main\n....\n+\nPath resolution is analogous to \u003c\u003cbaremetal-setup-getting-started,that of `./run --baremetal`\u003e\u003e.\n\nThen, as soon as boot ends, we are left inside a debug session that looks just like what `gdbserver` would produce.\n\n==== GDB step debug userland BusyBox init\n\nBusyBox custom init process:\n\n* Shell 1:\n+\n....\n./run --gdb-wait --kernel-cli 'init=/bin/ls'\n....\n* Shell 2:\n+\n....\n./run-gdb --userland \"$(./getvar buildroot_build_build_dir)\"/busybox-*/busybox ls_main\n....\n\nThis follows BusyBox' convention of calling the main for each executable as `\u003cexec\u003e_main` since the `busybox` executable has many \"mains\".\n\nBusyBox default init process:\n\n* Shell 1:\n+\n....\n./run --gdb-wait\n....\n* Shell 2:\n+\n....\n./run-gdb --userland \"$(./getvar buildroot_build_build_dir)\"/busybox-*/busybox init_main\n....\n\n`init` cannot be debugged with \u003c\u003cgdbserver\u003e\u003e without modifying the source, or else `/sbin/init` exits early with:\n\n....\n\"must be run as PID 1\"\n....\n\n==== GDB step debug userland non-init\n\nNon-init process:\n\n* Shell 1:\n+\n....\n./run --gdb-wait\n....\n* Shell 2:\n+\n....\n./run-gdb --userland userland/linux/rand_check.c main\n....\n* Shell 1 after the boot finishes:\n+\n....\n./linux/rand_check.out\n....\n\nThis is the least reliable setup as there might be other processes that use the given virtual address.\n\n[[gdb-step-debug-userland-non-init-without-gdb-wait]]\n===== GDB step debug userland non-init without --gdb-wait\n\nTODO: if I try \u003c\u003cgdb-step-debug-userland-non-init\u003e\u003e without `--gdb-wait` and the `break main` that we do inside `./run-gdb` says:\n\n....\nCannot access memory at address 0x10604\n....\n\nand then GDB never breaks. Tested at ac8663a44a450c3eadafe14031186813f90c21e4 + 1.\n\nThe exact behaviour seems to depend on the architecture:\n\n* `arm`: happens always\n* `x86_64`: appears to happen only if you try to connect GDB as fast as possible, before init has been reached.\n* `aarch64`: could not observe the problem\n\nWe have also double checked the address with:\n\n....\n./run-toolchain --arch arm readelf -- \\\n -s \"$(./getvar --arch arm userland_build_dir)/linux/myinsmod.out\" | \\\n grep main\n....\n\nand from GDB:\n\n....\ninfo line main\n....\n\nand both give:\n\n....\n000105fc\n....\n\nwhich is just 8 bytes before `0x10604`.\n\n`gdbserver` also says `0x10604`.\n\nHowever, if do a `Ctrl-C` in GDB, and then a direct:\n\n....\nb *0x000105fc\n....\n\nit works. Why?!\n\nOn GEM5, x86 can also give the `Cannot access memory at address`, so maybe it is also unreliable on QEMU, and works just by coincidence.\n\n=== GDB call\n\nGDB can call functions as explained at: https://stackoverflow.com/questions/1354731/how-to-evaluate-functions-in-gdb\n\nHowever this is failing for us:\n\n* some symbols are not visible to `call` even though `b` sees them\n* for those that are, `call` fails with an E14 error\n\nE.g.: if we break on `__x64_sys_write` on `count.sh`:\n\n....\n\u003e\u003e\u003e call printk(0, \"asdf\")\nCould not fetch register \"orig_rax\"; remote failure reply 'E14'\n\u003e\u003e\u003e b printk\nBreakpoint 2 at 0xffffffff81091bca: file kernel/printk/printk.c, line 1824.\n\u003e\u003e\u003e call fdget_pos(fd)\nNo symbol \"fdget_pos\" in current context.\n\u003e\u003e\u003e b fdget_pos\nBreakpoint 3 at 0xffffffff811615e3: fdget_pos. (9 locations)\n\u003e\u003e\u003e\n....\n\neven though `fdget_pos` is the first thing `__x64_sys_write` does:\n\n....\n581 SYSCALL_DEFINE3(write, unsigned int, fd, const char __user *, buf,\n582 size_t, count)\n583 {\n584 struct fd f = fdget_pos(fd);\n....\n\nI also noticed that I get the same error:\n\n....\nCould not fetch register \"orig_rax\"; remote failure reply 'E14'\n....\n\nwhen trying to use:\n\n....\nfin\n....\n\non many (all?) functions.\n\nSee also: https://github.com/cirosantilli/linux-kernel-module-cheat/issues/19\n\n=== GDB view ARM system registers\n\n`info all-registers` shows some of them.\n\nThe implementation is described at: https://stackoverflow.com/questions/46415059/how-to-observe-aarch64-system-registers-in-qemu/53043044#53043044\n\n=== GDB step debug multicore userland\n\nFor a more minimal baremetal multicore setup, see: xref:arm-baremetal-multicore[xrefstyle=full].\n\nWe can set and get which cores the Linux kernel allows a program to run on with `sched_getaffinity` and `sched_setaffinity`:\n\n....\n./run --cpus 2 --eval-after './linux/sched_getaffinity.out'\n....\n\nSource: link:userland/linux/sched_getaffinity.c[]\n\nSample output:\n\n....\nsched_getaffinity = 1 1\nsched_getcpu = 1\nsched_getaffinity = 1 0\nsched_getcpu = 0\n....\n\nWhich shows us that:\n\n* initially:\n** all 2 cores were enabled as shown by `sched_getaffinity = 1 1`\n** the process was randomly assigned to run on core 1 (the second one) as shown by `sched_getcpu = 1`. If we run this several times, it will also run on core 0 sometimes.\n* then we restrict the affinity to just core 0, and we see that the program was actually moved to core 0\n\nThe number of cores is modified as explained at: xref:number-of-cores[xrefstyle=full]\n\n`taskset` from the util-linux package sets the initial core affinity of a program:\n\n....\n./build-buildroot \\\n --config 'BR2_PACKAGE_UTIL_LINUX=y' \\\n --config 'BR2_PACKAGE_UTIL_LINUX_SCHEDUTILS=y' \\\n;\n./run --eval-after 'taskset -c 1,1 ./linux/sched_getaffinity.out'\n....\n\noutput:\n\n....\nsched_getaffinity = 0 1\nsched_getcpu = 1\nsched_getaffinity = 1 0\nsched_getcpu = 0\n....\n\nso we see that the affinity was restricted to the second core from the start.\n\nLet's do a QEMU observation to justify this example being in the repository with \u003c\u003cgdb-step-debug-userland-non-init,userland breakpoints\u003e\u003e.\n\nWe will run our `./linux/sched_getaffinity.out` infinitely many times, on core 0 and core 1 alternatively:\n\n....\n./run \\\n --cpus 2 \\\n --eval-after 'i=0; while true; do taskset -c $i,$i ./linux/sched_getaffinity.out; i=$((! $i)); done' \\\n --gdb-wait \\\n;\n....\n\non another shell:\n\n....\n./run-gdb --userland \"$(./getvar userland_build_dir)/linux/sched_getaffinity.out\" main\n....\n\nThen, inside GDB:\n\n....\n(gdb) info threads\n Id Target Id Frame\n* 1 Thread 1 (CPU#0 [running]) main () at sched_getaffinity.c:30\n 2 Thread 2 (CPU#1 [halted ]) native_safe_halt () at ./arch/x86/include/asm/irqflags.h:55\n(gdb) c\n(gdb) info threads\n Id Target Id Frame\n 1 Thread 1 (CPU#0 [halted ]) native_safe_halt () at ./arch/x86/include/asm/irqflags.h:55\n* 2 Thread 2 (CPU#1 [running]) main () at sched_getaffinity.c:30\n(gdb) c\n....\n\nand we observe that `info threads` shows the actual correct core on which the process was restricted to run by `taskset`!\n\nWe should also try it out with kernel modules: https://stackoverflow.com/questions/28347876/set-cpu-affinity-on-a-loadable-linux-kernel-module\n\nTODO we then tried:\n\n....\n./run --cpus 2 --eval-after './linux/sched_getaffinity_threads.out'\n....\n\nand:\n\n....\n./run-gdb --userland \"$(./getvar userland_build_dir)/linux/sched_getaffinity_threads.out\"\n....\n\nto switch between two simultaneous live threads with different affinities, it just didn't break on our threads:\n\n....\nb main_thread_0\n....\n\nNote that secondary cores in gem5 are kind of broken however: \u003c\u003cgem5-gdb-step-debug-secondary-cores\u003e\u003e.\n\nBibliography:\n\n* https://stackoverflow.com/questions/10490756/how-to-use-sched-getaffinity-and-sched-setaffinity-in-linux-from-c/50117787#50117787\n** https://stackoverflow.com/questions/663958/how-to-control-which-core-a-process-runs-on/50210009#50210009\n** https://stackoverflow.com/questions/280909/cpu-affinity/54478296#54478296\n** https://unix.stackexchange.com/questions/73/how-can-i-set-the-processor-affinity-of-a-process-on-linux/441098#441098 (summary only)\n* https://stackoverflow.com/questions/42800801/how-to-use-gdb-to-debug-qemu-with-smp-symmetric-multiple-processors\n\n=== Linux kernel GDB scripts\n\nWe source the Linux kernel GDB scripts by default for `lx-symbols`, but they also contains some other goodies worth looking into.\n\nThose scripts basically parse some in-kernel data structures to offer greater visibility with GDB.\n\nAll defined commands are prefixed by `lx-`, so to get a full list just try to tab complete that.\n\nThere aren't as many as I'd like, and the ones that do exist are pretty self explanatory, but let's give a few examples.\n\nShow dmesg:\n\n....\nlx-dmesg\n....\n\nShow the \u003c\u003ckernel-command-line-parameters\u003e\u003e:\n\n....\nlx-cmdline\n....\n\nDump the device tree to a `fdtdump.dtb` file in the current directory:\n\n....\nlx-fdtdump\npwd\n....\n\nList inserted kernel modules:\n\n....\nlx-lsmod\n....\n\nSample output:\n\n....\nAddress Module Size Used by\n0xffffff80006d0000 hello 16384 0\n....\n\nBibliography:\n\n* https://events.static.linuxfound.org/sites/events/files/slides/Debugging%20the%20Linux%20Kernel%20with%20GDB.pdf\n* https://wiki.linaro.org/LandingTeams/ST/GDB\n\n==== lx-ps\n\nList all processes:\n\n....\nlx-ps\n....\n\nSample output:\n\n....\n0xffff88000ed08000 1 init\n0xffff88000ed08ac0 2 kthreadd\n....\n\nThe second and third fields are obviously PID and process name.\n\nThe first one is more interesting, and contains the address of the `task_struct` in memory.\n\nThis can be confirmed with:\n\n....\np ((struct task_struct)*0xffff88000ed08000\n....\n\nwhich contains the correct PID for all threads I've tried:\n\n....\npid = 1,\n....\n\nTODO get the PC of the kthreads: https://stackoverflow.com/questions/26030910/find-program-counter-of-process-in-kernel Then we would be able to see where the threads are stopped in the code!\n\nOn ARM, I tried:\n\n....\ntask_pt_regs((struct thread_info *)((struct task_struct)*0xffffffc00e8f8000))-\u003euregs[ARM_pc]\n....\n\nbut `task_pt_regs` is a `#define` and GDB cannot see defines without `-ggdb3`: https://stackoverflow.com/questions/2934006/how-do-i-print-a-defined-constant-in-gdb which are apparently not set?\n\nBibliography:\n\n* https://stackoverflow.com/questions/9561546/thread-aware-gdb-for-kernel\n* https://wiki.linaro.org/LandingTeams/ST/GDB\n* https://events.static.linuxfound.org/sites/events/files/slides/Debugging%20the%20Linux%20Kernel%20with%20GDB.pdf presentation: https://www.youtube.com/watch?v=pqn5hIrz3A8\n\n[[config-pid-in-contextidr]]\n===== CONFIG_PID_IN_CONTEXTIDR\n\nhttps://stackoverflow.com/questions/54133479/accessing-logical-software-thread-id-in-gem5 on ARM the kernel can store an indication of PID in the CONTEXTIDR_EL1 register, making that much easier to observe from simulators.\n\nIn particular, gem5 prints that number out by default on `ExecAll` messages!\n\nLet's test it out with \u003c\u003clinux-kernel-build-variants\u003e\u003e + \u003c\u003cgem5-restore-new-script\u003e\u003e:\n\n....\n./build-linux --arch aarch64 --linux-build-id CONFIG_PID_IN_CONTEXTIDR --config 'CONFIG_PID_IN_CONTEXTIDR=y'\n# Checkpoint run.\n./run --arch aarch64 --emulator gem5 --linux-build-id CONFIG_PID_IN_CONTEXTIDR --eval './gem5.sh'\n# Trace run.\n./run \\\n --arch aarch64 \\\n --emulator gem5 \\\n --gem5-readfile 'posix/getpid.out; posix/getpid.out' \\\n --gem5-restore 1 \\\n --linux-build-id CONFIG_PID_IN_CONTEXTIDR \\\n --trace FmtFlag,ExecAll,-ExecSymbol \\\n;\n....\n\nThe terminal runs both programs which output their PID to stdout:\n\n....\npid=44\npid=45\n....\n\nBy quickly inspecting the `trace.txt` file, we immediately notice that the `system.cpu: A\u003cn\u003e` part of the logs, which used to always be `system.cpu: A0`, now has a few different values! Nice!\n\nWe can briefly summarize those values by removing repetitions:\n\n....\ncut -d' ' -f4 \"$(./getvar --arch aarch64 --emulator gem5 trace_txt_file)\" | uniq -c\n....\n\ngives:\n\n....\n 97227 A39\n 147476 A38\n 222052 A40\n 1 terminal\n1117724 A40\n 27529 A31\n 43868 A40\n 27487 A31\n 138349 A40\n 13781 A38\n 231246 A40\n 25536 A38\n 28337 A40\n 214799 A38\n 963561 A41\n 92603 A38\n 27511 A31\n 224384 A38\n 564949 A42\n 182360 A38\n 729009 A43\n 8398 A23\n 20200 A10\n 636848 A43\n 187995 A44\n 27529 A31\n 70071 A44\n 16981 A0\n 623806 A44\n 16981 A0\n 139319 A44\n 24487 A0\n 174986 A44\n 25420 A0\n 89611 A44\n 16981 A0\n 183184 A44\n 24728 A0\n 89608 A44\n 17226 A0\n 899075 A44\n 24974 A0\n 250608 A44\n 137700 A43\n1497997 A45\n 227485 A43\n 138147 A38\n 482646 A46\n....\n\nI'm not smart enough to be able to deduce all of those IDs, but we can at least see that:\n\n* A44 and A45 are there as expected from stdout!\n* A39 must be the end of the execution of `m5 checkpoint`\n* so we guess that A38 is the shell as it comes next\n* the weird \"terminal\" line is `336969745500: system.terminal: attach terminal 0`\n* which is the shell PID? I should have printed that as well :-)\n* why are there so many other PIDs? This was supposed to be a silent system without daemons!\n* A0 is presumably the kernel. However we see process switches without going into A0, so I'm not sure how, it appears to count kernel instructions as part of processes\n* A46 has to be the `m5 exit` call\n\nOr if you want to have some real fun, try: link:baremetal/arch/aarch64/contextidr_el1.c[]:\n\n....\n./run --arch aarch64 --emulator gem5 --baremetal baremetal/arch/aarch64/contextidr_el1.c --trace-insts-stdout\n....\n\nin which we directly set the register ourselves! Output excerpt:\n\n....\n 31500: system.cpu: A0 T0 : @main+12 : ldr x0, [sp, #12] : MemRead : D=0x0000000000000001 A=0x82fffffc flags=(IsInteger|IsMemRef|IsLoad)\n 32000: system.cpu: A1 T0 : @main+16 : msr contextidr_el1, x0 : IntAlu : D=0x0000000000000001 flags=(IsInteger|IsSerializeAfter|IsNonSpeculative)\n 32500: system.cpu: A1 T0 : @main+20 : ldr x0, [sp, #12] : MemRead : D=0x0000000000000001 A=0x82fffffc flags=(IsInteger|IsMemRef|IsLoad)\n 33000: system.cpu: A1 T0 : @main+24 : add w0, w0, #1 : IntAlu : D=0x0000000000000002 flags=(IsInteger)\n 33500: system.cpu: A1 T0 : @main+28 : str x0, [sp, #12] : MemWrite : D=0x0000000000000002 A=0x82fffffc flags=(IsInteger|IsMemRef|IsStore)\n 34000: system.cpu: A1 T0 : @main+32 : ldr x0, [sp, #12] : MemRead : D=0x0000000000000002 A=0x82fffffc flags=(IsInteger|IsMemRef|IsLoad)\n 34500: system.cpu: A1 T0 : @main+36 : subs w0, #9 : IntAlu : D=0x0000000000000000 flags=(IsInteger)\n 35000: system.cpu: A1 T0 : @main+40 : b.le \u003cmain+12\u003e : IntAlu : flags=(IsControl|IsDirectControl|IsCondControl)\n 35500: system.cpu: A1 T0 : @main+12 : ldr x0, [sp, #12] : MemRead : D=0x0000000000000002 A=0x82fffffc flags=(IsInteger|IsMemRef|IsLoad)\n 36000: system.cpu: A2 T0 : @main+16 : msr contextidr_el1, x0 : IntAlu : D=0x0000000000000002 flags=(IsInteger|IsSerializeAfter|IsNonSpeculative)\n 36500: system.cpu: A2 T0 : @main+20 : ldr x0, [sp, #12] : MemRead : D=0x0000000000000002 A=0x82fffffc flags=(IsInteger|IsMemRef|IsLoad)\n 37000: system.cpu: A2 T0 : @main+24 : add w0, w0, #1 : IntAlu : D=0x0000000000000003 flags=(IsInteger)\n 37500: system.cpu: A2 T0 : @main+28 : str x0, [sp, #12] : MemWrite : D=0x0000000000000003 A=0x82fffffc flags=(IsInteger|IsMemRef|IsStore)\n 38000: system.cpu: A2 T0 : @main+32 : ldr x0, [sp, #12] : MemRead : D=0x0000000000000003 A=0x82fffffc flags=(IsInteger|IsMemRef|IsLoad)\n 38500: system.cpu: A2 T0 : @main+36 : subs w0, #9 : IntAlu : D=0x0000000000000000 flags=(IsInteger)\n 39000: system.cpu: A2 T0 : @main+40 : b.le \u003cmain+12\u003e : IntAlu : flags=(IsControl|IsDirectControl|IsCondControl)\n 39500: system.cpu: A2 T0 : @main+12 : ldr x0, [sp, #12] : MemRead : D=0x0000000000000003 A=0x82fffffc flags=(IsInteger|IsMemRef|IsLoad)\n 40000: system.cpu: A3 T0 : @main+16 : msr contextidr_el1, x0 : IntAlu : D=0x0000000000000003 flags=(IsInteger|IsSerializeAfter|IsNonSpeculative)\n....\n\n\u003c\u003carmarm8-fa\u003e\u003e D13.2.27 \"CONTEXTIDR_EL1, Context ID Register (EL1)\" documents `CONTEXTIDR_EL1` as:\n\n____\nIdentifies the current Process Identifier.\n\nThe value of the whole of this register is called the Context ID and is used by:\n\n* The debug logic, for Linked and Unlinked Context ID matching.\n* The trace logic, to identify the current process.\n\nThe significance of this register is for debug and trace use only.\n____\n\nTested on 145769fc387dc5ee63ec82e55e6b131d9c968538 + 1.\n\n=== Debug the GDB remote protocol\n\nFor when it breaks again, or you want to add a new feature!\n\n....\n./run --debug\n./run-gdb --before '-ex \"set remotetimeout 99999\" -ex \"set debug remote 1\"' start_kernel\n....\n\nSee also: https://stackoverflow.com/questions/13496389/gdb-remote-protocol-how-to-analyse-packets\n\n[[remote-g-packet]]\n==== Remote 'g' packet reply is too long\n\nThis error means that the GDB server, e.g. in QEMU, sent more registers than the GDB client expected.\n\nThis can happen for the following reasons:\n\n* you set the architecture of the client wrong, often 32 vs 64 bit as mentioned at: https://stackoverflow.com/questions/4896316/gdb-remote-cross-debugging-fails-with-remote-g-packet-reply-is-too-long\n* there is a bug in the GDB server and the XML description does not match the number of registers actually sent\n* the GDB server does not send XML target descriptions and your GDB expects a different number of registers by default. E.g., gem5 d4b3e064adeeace3c3e7d106801f95c14637c12f does not send the XML files\n\nThe XML target description format is described a bit further at: https://stackoverflow.com/questions/46415059/how-to-observe-aarch64-system-registers-in-qemu/53043044#53043044\n\n== KGDB\n\nKGDB is kernel dark magic that allows you to GDB the kernel on real hardware without any extra hardware support.\n\nIt is useless with QEMU since we already have full system visibility with `-gdb`. So the goal of this setup is just to prepare you for what to expect when you will be in the treches of real hardware.\n\nKGDB is cheaper than JTAG (free) and easier to setup (all you need is serial), but with less visibility as it depends on the kernel working, so e.g.: dies on panic, does not see boot sequence.\n\nFirst run the kernel with:\n\n....\n./run --kgdb\n....\n\nthis passes the following options on the kernel CLI:\n\n....\nkgdbwait kgdboc=ttyS1,115200\n....\n\n`kgdbwait` tells the kernel to wait for KGDB to connect.\n\nSo the kernel sets things up enough for KGDB to start working, and then boot pauses waiting for connection:\n\n....\n\u003c6\u003e[ 4.866050] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled\n\u003c6\u003e","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcirosantilli%2Flinux-kernel-module-cheat","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcirosantilli%2Flinux-kernel-module-cheat","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcirosantilli%2Flinux-kernel-module-cheat/lists"}