{"id":15650170,"url":"https://github.com/64kramsystem/qemu-pinning","last_synced_at":"2025-09-11T17:44:35.059Z","repository":{"id":43938600,"uuid":"88334814","full_name":"64kramsystem/qemu-pinning","owner":"64kramsystem","description":"My QEMU fork with pinning (affinity) support and a few tweaks.","archived":false,"fork":false,"pushed_at":"2024-12-11T13:29:57.000Z","size":547218,"stargazers_count":41,"open_issues_count":1,"forks_count":13,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-01-17T04:03:14.291Z","etag":null,"topics":["hypervisor","qemu","risc-v","riscv","virtualization"],"latest_commit_sha":null,"homepage":"","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/64kramsystem.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"COPYING","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-04-15T08:37:28.000Z","updated_at":"2025-01-09T15:38:40.000Z","dependencies_parsed_at":"2024-10-03T12:34:20.365Z","dependency_job_id":"e162e0af-6f7d-4263-afe2-0648cc7a55ce","html_url":"https://github.com/64kramsystem/qemu-pinning","commit_stats":null,"previous_names":[],"tags_count":379,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/64kramsystem%2Fqemu-pinning","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/64kramsystem%2Fqemu-pinning/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/64kramsystem%2Fqemu-pinning/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/64kramsystem%2Fqemu-pinning/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/64kramsystem","download_url":"https://codeload.github.com/64kramsystem/qemu-pinning/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":235199278,"owners_count":18951516,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["hypervisor","qemu","risc-v","riscv","virtualization"],"created_at":"2024-10-03T12:33:39.801Z","updated_at":"2025-01-22T22:57:18.012Z","avatar_url":"https://github.com/64kramsystem.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"# QEMU-Pinning: QEMU fork with pinning (affinity) support\n\nFork of QEMU, with support for pinning virtual cpus/cores/threads to the physical counterparts.\n\n- [QEMU-Pinning: QEMU fork with pinning (affinity) support](#qemu-pinning-qemu-fork-with-pinning-affinity-support)\n  - [Status](#status)\n  - [Patch notes](#patch-notes)\n  - [Building the project](#building-the-project)\n  - [Pinning comamndline options](#pinning-comamndline-options)\n  - [Common pinning scenarios/scripts, and Windows mapping](#common-pinning-scenariosscripts-and-windows-mapping)\n    - [One vCPU per core](#one-vcpu-per-core)\n    - [One vCPUs per core/thread, except one core](#one-vcpus-per-corethread-except-one-core)\n    - [Arbitrary number of vCPUs](#arbitrary-number-of-vcpus)\n  - [Verifying the pinning](#verifying-the-pinning)\n  - [Multi-socket CPUs](#multi-socket-cpus)\n  - [Performance](#performance)\n  - [Repository (git) structure](#repository-git-structure)\n  - [Why not libvirt?](#why-not-libvirt)\n\n## Status\n\nThis project is \"semi-discontinued\", due to lack of resources (QEMU developers often refactor the codebase, which sometimes makes updating the patch non-trivial for a dev without domain knowledge).\n\nI will sync `master` with new QEMU versions, unless cases where the patch has conflicts that I can't easily solve; in such situations, I will open an issue tagged with `help wanted`. Users can open issues for general problems, but I can't guarantee solving them (I'll also label such issues with `help wanted`).\n\nI maintain branches with all the previous patched versions, so if at any point in time, the latest QEMU hasn't been patched, an older version can be used.\n\nThanks to the user [arcnx](https://github.com/arcnmx), who made the patch compatible with v6.2.\n\nThe user [igrekster](https://github.com/igrekster) has kindly [contributed a Python script](https://github.com/64kramsystem/qemu-pinning/wiki/Python-pinning-script) for pinning the QEMU processes without patching QEMU. Users wishing to improve it can open a (phony) PR or contact me directly (find the contacts in my [profile page](https://github.com/64kramsystem)).\n\n## Patch notes\n\nThe code was originally a QEMU 2.4.1 patch [published on the QEMU mailing list](https://www.mail-archive.com/qemu-discuss%40nongnu.org/msg02253.html); I've made significant fixes/cleanups/improvements, and I periodically rebase/update/extend it on top of the release QEMU versions.\n\nPinning is accomplished using the Linux interfaces `cpu_set_t` (and related macros) and `pthread_setaffinity_np`.\n\nThere is one addition (separated from pinning) in this fork:\n\n- allowing Samba symlink access; use $SUDO_USER, when present\n\nAnd an option is provided to enable easy(er) to debug builds.\n\nSee the build script help (`build_pinning_qemu_binary.sh --help`) for the details.\n\n## Building the project\n\nA script is provided in order to build the project on common Linux distros, and to enable additional features.\n\nFrom the project root, run:\n\n```sh\n# Add `--help` to display the help.\n#\n./build_pinning_qemu_binary.sh\n```\n\nThe project will build, and display the locations of the binaries (system/user emulators).\n\nA few important notes:\n\n- this is a rather minimal build configuration, and it won't have any secondary feature aside the mentioned ones (GTK VGA, Pulseaudio, USB, AIO); if you need more features, you can easily alter the `./configure` command in the build script;\n- if the binary is moved to another location, BIOS files may need to be moved as well (e.g. the `pc-bios` directory for the x86-64 architecture).\n\nIt's trivial to adjust the script to run it on other distributions.\n\n## Pinning comamndline options\n\nPinning is exposed through the `-vcpu` commandline option; this is the generic structure of a pinned execution:\n\n```sh\nqemu-system-x86_64 \\\n  -smp $total_vcpus,cores=$vcores,sockets=$vsockets,threads=$vthreads \\\n  -vcpu vcpunum=$vcpu_number,affinity=$host_processor_number \\\n  -vcpu vcpunum=$vcpu_number,affinity=$host_processor_number \\\n  ...\n```\n\nThe following section will show common pinning scenarios, and convenient scripts that prepare them.\n\n## Common pinning scenarios/scripts, and Windows mapping\n\nThe scenarios in this section (except where specified) apply to a processor with the following topology (a Ryzen 7 3800x):\n\n```\n$ lstopo --of console --no-io --no-caches\n\nMachine (31GB total) + Package L#0\n  NUMANode L#0 (P#0 31GB)\n  Core L#0\n    PU L#0 (P#0)\n    PU L#1 (P#8)\n  Core L#1\n    PU L#2 (P#1)\n    PU L#3 (P#9)\n  Core L#2\n    PU L#4 (P#2)\n    PU L#5 (P#10)\n  Core L#3\n    PU L#6 (P#3)\n    PU L#7 (P#11)\n  Core L#4\n    PU L#8 (P#4)\n    PU L#9 (P#12)\n  Core L#5\n    PU L#10 (P#5)\n    PU L#11 (P#13)\n  Core L#6\n    PU L#12 (P#6)\n    PU L#13 (P#14)\n  Core L#7\n    PU L#14 (P#7)\n    PU L#15 (P#15)\n```\n\nEmpyrically, based on the scripts presented here, QEMU exposes the threads (vcpus) sequentially, and Windows interprets physical processors as contiguous blocks.\n\nTherefore, if one wants to, say, pin core 0 and its two threads, they need to pass:\n\n```\n-smp 2,cores=1,sockets=1,threads=2 -vcpu vcpunum=0,affinity=0 -vcpu vcpunum=1,affinity=8\n```\n\n### One vCPU per core\n\nAssign one virtual CPU per host core (*not per thread*):\n\n```sh\nCORES_NUMBER=$(lscpu --all -p=CORE | grep -v '^#' | sort | uniq | wc -l)\n\nSMP_PARAMS=\"  -smp $CORES_NUMBER,cores=$CORES_NUMBER,sockets=1,threads=1\"\n\nfor core_number in $(seq 1 $CORES_NUMBER); do\n  SMP_PARAMS+=\"  \\\\\n  -vcpu vcpunum=$((core_number - 1)),affinity=$((core_number - 1))\"\ndone\n\necho \"$SMP_PARAMS\"\n#  -smp 8,cores=8,sockets=1,threads=1  \\\n#  -vcpu vcpunum=0,affinity=0  \\\n#  -vcpu vcpunum=1,affinity=1  \\\n#  -vcpu vcpunum=2,affinity=2  \\\n#  -vcpu vcpunum=3,affinity=3  \\\n#  -vcpu vcpunum=4,affinity=4  \\\n#  -vcpu vcpunum=5,affinity=5  \\\n#  -vcpu vcpunum=6,affinity=6  \\\n#  -vcpu vcpunum=7,affinity=7\n```\n\n### One vCPUs per core/thread, except one core\n\nA typical configuration is to give all hardware cores/threads to the guest, with the exception of one core.\n\nFirst, one needs to obtain the host cpu layout; a simple way is:\n\n```sh\n$ lscpu --extended\n\nCPU NODE SOCKET CORE L1d:L1i:L2:L3 ONLINE MAXMHZ    MINMHZ\n0   0    0      0    0:0:0:0       yes    4200,0000 800,0000\n1   0    0      1    1:1:1:0       yes    4200,0000 800,0000\n2   0    0      2    2:2:2:0       yes    4200,0000 800,0000\n3   0    0      3    3:3:3:0       yes    4200,0000 800,0000\n4   0    0      0    0:0:0:0       yes    4200,0000 800,0000\n5   0    0      1    1:1:1:0       yes    4200,0000 800,0000\n6   0    0      2    2:2:2:0       yes    4200,0000 800,0000\n7   0    0      3    3:3:3:0       yes    4200,0000 800,0000\n```\n\n`CPU` represents a CPU from a Linux perspective, therefore, if the CPU supports SMT (\"Hyper-threading\" on Intel), each `CPU` is a thread.  \nIn this case, for example, `CORE 0` will have two threads, represented by `CPU 0` and `CPU 4`.\n\nWith the configuration above, and the objective of passing all except one core, the pinning parameters are:\n\n```\n-smp 6,cores=3,sockets=1,threads=2\n-vcpu vcpunum=0,affinity=1 -vcpu vcpunum=1,affinity=5\n-vcpu vcpunum=2,affinity=2 -vcpu vcpunum=3,affinity=6\n-vcpu vcpunum=4,affinity=3 -vcpu vcpunum=5,affinity=7\n```\n\nThis will yield, in a Windows guest, 3 physical processors with 2 logical processors each, mapped to the host `CPU`s (1,5), (2,6) and (3,7).\n\nThe configuration above can be be automated with:\n\n```sh\n# Exclude the core 0, and cluster the threads, sorted by (socket,core)\nCPUS_DATA=$(lscpu --all --parse=SOCKET,CORE,CPU | grep -vP '^(#|0,0)' | sort -t ',' -n)\n\nTHREADS=$(echo \"$CPUS_DATA\" | wc -l)\nCORES=$(echo \"$CPUS_DATA\" | cut -d ',' -f 2 | sort | uniq | wc -l)\nSOCKETS=$(echo \"$CPUS_DATA\" | cut -d ',' -f 1 | sort | uniq | wc -l)\n\nSMP_PARAMS=\"  -smp $THREADS,cores=$CORES,threads=$(($THREADS / $CORES))\"\n\nvcpu=0; while read cpu_entry; do\n  affinity=$(echo $cpu_entry | cut -d ',' -f 3)\n  SMP_PARAMS+=\"  \\\\\n  -vcpu vcpunum=$vcpu,affinity=$affinity\"\n  vcpu=$(($vcpu + 1))\ndone \u003c\u003c\u003c \"$CPUS_DATA\"\n\necho \"$SMP_PARAMS\"\n#  -smp 14,cores=7,threads=2  \\\n#  -vcpu vcpunum=0,affinity=1  \\\n#  -vcpu vcpunum=1,affinity=9  \\\n#  -vcpu vcpunum=2,affinity=10  \\\n#  -vcpu vcpunum=3,affinity=2  \\\n#  -vcpu vcpunum=4,affinity=11  \\\n#  -vcpu vcpunum=5,affinity=3  \\\n#  -vcpu vcpunum=6,affinity=12  \\\n#  -vcpu vcpunum=7,affinity=4  \\\n#  -vcpu vcpunum=8,affinity=13  \\\n#  -vcpu vcpunum=9,affinity=5  \\\n#  -vcpu vcpunum=10,affinity=14  \\\n#  -vcpu vcpunum=11,affinity=6  \\\n#  -vcpu vcpunum=12,affinity=15  \\\n#  -vcpu vcpunum=13,affinity=7\n```\n\n### Arbitrary number of vCPUs\n\nThe following script, courtesy of Frédéric Pétrot (University of Grenoble), prepare a pinning for an arbitrary number of vCPUs, passed to the script:\n\n```sh\n#!/bin/bash\n\nif test -z \"$1\"; then\n  echo \"$0: Please provide the number of virtual cpus\"\n  exit\nfi\n\nnvcpus=$1\n\n# Chosing to sort in such a way that it is easier to see if there is a bug in the program :)\nCPUS_DATA=$(lscpu --all --parse=SOCKET,CORE,CPU | grep -vP '^(#)' | sort -t ',' -k 1,1n -k 2,2n -k 3,3n)\n\ndeclare CPUS_ENTRY\ni=0; while read cpu_entry; do\n  CPUS_ENTRY[$i]=$cpu_entry\n  i=$(($i + 1))\ndone \u003c\u003c\u003c \"$CPUS_DATA\"\ncpus=$(nproc)\n\nTHREADS=$(echo \"$CPUS_DATA\" | wc -l)\nCORES=$(echo \"$CPUS_DATA\" | cut -d ',' -f 2 | sort | uniq | wc -l)\nSOCKETS=$(echo \"$CPUS_DATA\" | cut -d ',' -f 1 | sort | uniq | wc -l)\n\n# A bit of a wild guess, ...\nthreads=$(($THREADS/$CORES))\ncores=$(($(($nvcpus + 1))/$threads))\n\nQEMU_SMP=\"  -smp $nvcpus,cores=$cores,threads=$threads\"\n\nfor vcpu in $(seq 0 $(($nvcpus - 1))); do\n  affinity=$(echo ${CPUS_ENTRY[$(($vcpu%$cpus))]} | cut -d ',' -f 3)\n  QEMU_AFFINITIES=\"$QEMU_AFFINITIES  \\\\\n  -vcpu vcpunum=$vcpu,affinity=$affinity\"\ndone\n\necho \"$QEMU_SMP $QEMU_AFFINITIES\"\n```\n\nResult of `vcpu-assign.sh 10`:\n\n```sh\n  -smp 10,cores=5,threads=2   \\\n  -vcpu vcpunum=0,affinity=0  \\\n  -vcpu vcpunum=1,affinity=8  \\\n  -vcpu vcpunum=2,affinity=1  \\\n  -vcpu vcpunum=3,affinity=9  \\\n  -vcpu vcpunum=4,affinity=2  \\\n  -vcpu vcpunum=5,affinity=10  \\\n  -vcpu vcpunum=6,affinity=3  \\\n  -vcpu vcpunum=7,affinity=11  \\\n  -vcpu vcpunum=8,affinity=4  \\\n  -vcpu vcpunum=9,affinity=12\n```\n\n## Verifying the pinning\n\nPinning can be verified in many ways.\n\nAn easy one is to use `htop` on the host, and the Microsoft-advised [Cpu Stress tool](https://blogs.msdn.microsoft.com/vijaysk/2012/10/26/tools-to-simulate-cpu-memory-disk-load) on the guest, then rotate the affinity via Task Manager.\n\nThe procedure is:\n\n- run `htop` on the host\n- start the guest\n- run the CPI Stress tool, using 1 thread active, with `Maximum` activity\n- open the Task manager, `Details` tab\n- right click on `CPUSTRES.EXE`\n- now set one CPU at a time, and check which CPU will have 100% occupation on the host\n\nDon't forget that the Windows CPU (thread) indexes won't match the Linux ones!\n\n## Multi-socket CPUs\n\nThis patch should also support multi-socket CPUs.\n\n## Performance\n\nPinning does not necessarily improve the performance; in some cases, it may actually worsen, therefore, when enabling it, users should carefully test the impact on the given use case.\n\nIn general terms, the pinned processors should be excluded from the kernel scheduling (`isolcpus` boot option). Under normal conditions, at least in terms of throughput, the Linux kernel does a good job at scheduling.\n\nPerformance testing with pinning has been the subject of the research paper \"To Pin or Not to Pin: Asserting the Scalability of QEMU Parallel Implementation\".\n\n## Repository (git) structure\n\nSince this fork needs to strictly adhere to the upstream git structure, a system of branches has been established, in order to keep the changes compartmentalized and easy to inspect/track.\n\nThe `master` branch is always the latest QEMU stable version (generally, with a lag of a few days, unless there are specific issues), with the latest versions of the patches and the build script.\n\nThe fork commits are (in order from HEAD):\n\n1. pinning patch\n2. building script; `README.md`; gitignore update\n3. other patches, e.g. Samba symlinks access\n\nThe repository contains the line of historical branches `vX.Y.Z-pinning`, which are snapshots of previous stable versions, and the corresponding `vX.Y.Z-pinning-changes` branches.\n\n## Why not libvirt?\n\nI've found libvirt to be a very interesting idea, but ultimately, a leaky abstraction:\n\n1. the compatibility with all the QEMU versions is not guaranteed (for example, v2.10 wasn't supported for some time after release);\n2. the typical GUI (`virt-manager`) is poor (many entries must be edited via `virsh edit`);\n3. since the ultimate reference is QEMU, one ends up thinking how to make things work with QEMU, then finding the libvirt configuration counterpart.\n\nPoint 3 may be caused by my poor libvirt knowledge, but the fact that libvirt's functionality is built on top of QEMU always stands, and complex QEMU configurations are bound to have translation challenges.\n\nI'm sure of course, that for simple setups, `libvirt` + `virt-manager` may work very well.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F64kramsystem%2Fqemu-pinning","html_url":"https://awesome.ecosyste.ms/projects/github.com%2F64kramsystem%2Fqemu-pinning","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F64kramsystem%2Fqemu-pinning/lists"}