{"id":15148746,"url":"https://github.com/voutilad/virtio_vmmci","last_synced_at":"2025-10-16T10:24:27.280Z","repository":{"id":87607410,"uuid":"171144493","full_name":"voutilad/virtio_vmmci","owner":"voutilad","description":"My 3/4-hearted attempt at making a Linux virtio driver for OpenBSD VMM Control Interface","archived":false,"fork":false,"pushed_at":"2024-12-25T17:37:11.000Z","size":534,"stargazers_count":39,"open_issues_count":3,"forks_count":4,"subscribers_count":9,"default_branch":"master","last_synced_at":"2025-06-08T19:42:51.030Z","etag":null,"topics":["linux","openbsd"],"latest_commit_sha":null,"homepage":"","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/voutilad.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-02-17T16:29:31.000Z","updated_at":"2024-12-05T18:49:08.000Z","dependencies_parsed_at":null,"dependency_job_id":"56bd29d7-344b-4f8b-a282-739061b0e842","html_url":"https://github.com/voutilad/virtio_vmmci","commit_stats":{"total_commits":79,"total_committers":1,"mean_commits":79.0,"dds":0.0,"last_synced_commit":"f56985fb080b6cb799028bf01c9644ac28b947c3"},"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"purl":"pkg:github/voutilad/virtio_vmmci","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/voutilad%2Fvirtio_vmmci","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/voutilad%2Fvirtio_vmmci/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/voutilad%2Fvirtio_vmmci/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/voutilad%2Fvirtio_vmmci/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/voutilad","download_url":"https://codeload.github.com/voutilad/virtio_vmmci/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/voutilad%2Fvirtio_vmmci/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273322092,"owners_count":25085019,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-02T02:00:09.530Z","response_time":77,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["linux","openbsd"],"created_at":"2024-09-26T13:22:29.328Z","updated_at":"2025-10-16T10:24:27.138Z","avatar_url":"https://github.com/voutilad.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"# An OpenBSD VMM Control Interface (vmmci) for Linux\n_...or \"How I learned to shut my x270 laptop and not worry about my VMs.\"_\n\n[![builds.sr.ht status](https://builds.sr.ht/~voutilad/virtio_vmmci.svg)](https://builds.sr.ht/~voutilad/virtio_vmmci?)\n\nThis is an implementation of [vmmci(4)](https://man.openbsd.org/vmmci) for\nLinux using a customized version of the `virtio_pci` driver from the\nmainline kernel. It currently supports the following:\n\n1. **Clean Shutdowns on Request**\n   When requested by `vmctl(8)`...you can safely use\n   `vmctl stop \u003cyou linux guest\u003e` and it'll nicely stop services and\n   sync disks!\n\n2. **System Time Synchronization**\n   When the host `vmd(8)` emulation of the hardware clock detects a\n   clock drift (most likely due to the host being suspended/resumed),\n   it fires a `SYNCRTC` message that the Linux `vmmci` driver responds\n   to by synchronizing system time to the hardware clock time. (This\n   currently only happens during certain host events like resuming\n   from a suspended state.)\n\n3. **Tracking Clock Drift**\n   At regular intervals (currently 20s), `vmmci` will measure current\n   clock drift, recording the current drift amount in seconds and\n   nanoseconds parts readable via `sysctl vmmci`\n\n\u003e **NOTE:** if you're here to deal with constant, excessive clock\n\u003e drift, see the [FAQ](#wait-why-isnt-this-fixing-my-clock-drift-issues)!\n\n## Example with Linux Guests\n![vmd(8) and 3 Linux guests](/example.png?raw=true \"VMD(8) and 3 Linux Guests\")\n\nAbove is a screenshot of the clock sync in practice. Tmux pane `0` is\nmy instance of `vmd(8)` running in the foreground with verbose\nlogging. The other panes:\n\n1. **Alpine 3.8.4** (virt) with kernel 4.14.104-0-virt\n2. **Debian Buster** (9.8) with kernel 4.9.0-8-amd64 (yeah, something\n   is jacked up with dmesg's time...but it IS correct in\n   `journalctl(1)` and when checking `timedatectl(1)`)\n3. **Ubuntu 18.04** with my custom kernel 4.20.13-obsd+\n\nTake note of the `rtc_fire1` log events from `vmd(8)`. That's where my\nlaptop comes out of hibernation and the virtual rtc detects a drift\nand sends sync requests to the guests. Each Linux guest receives the\nrequest, performs the clock step, and ack's.\n\n## Known Issues or Caveats\nBefore you dive in, a few things to note:\n\n1. I test and develop using OpenBSD snapshots, so relatively in sync\n   with _-current_. (This should work with OpenBSD 6.7 and later.)\n\n2. I lean heavily on the simplification that OpenBSD virtualization\n   guests are single CPU currently.\n\n3. This currenly won't solve larger clock issues, such as major drift.\n\n4. I primarily focus on supporting the newest long-term support\n   kernels picked up by major distros, which means Linux 5.4 at the\n   moment.\n\n5. I focus my testing on **Alpine Linux** guests using their `-virt`\n   releases since it's simple to install and manage without a lot of\n   ancillary stuff. Plus, _I personally like Alpine_.\n\n## Installation \u0026 Usage\nThis Linux VMMCI currently comes in **two parts:**\n\n1. `virtio_pci_obsd.ko` -- handles the quirks of getting Linux's\n   virtio pci framework to properly work with the VMM Control\n   Interface device from `vmd(8)`\n2. `virtio_vmmci.ko` -- virtio device driver that replicates the\n   behavior of OpenBSD's `vmmci(4)` driver\n\n_You will need both modules installed!_\n\nAssuming you've got a recent Linux distro running as a guest already\nunder OpenBSD, it shouldn't be more than a few minutes to get things\nup and running.\n\n### 1. Prerequisites\nInstall the tools required to build kernel modules using your package\nmanager or whatever you normally use to install stuff.\n\nFor Alpine systems running the `-virt` flavored kernel:\n\n```sh\n# apk add gcc make linux-virt-dev\n```\n\nBasically you need your kernel headers and some GCC tooling.\n\n### 2. Compiling\nThis should be easy and expose issues with your lack of prerequisites\nor an incompatability with your kernel version:\n\n```sh\n$ make\n```\n\n\u003e A common source of compiler warnings are from variations in kernel\n\u003e versions. Please share your kernel version and the compiler output\n\u003e if you have issues!\n\n### 3. Installation\nAs root, simply run:\n\n```sh\n# make install\n```\n\nYou'll probably see some SSL errors and complaints about missing key\nfiles. _This is expected as you're building an out-of-tree kernel\nmodule that isn't being signed._ If you'd like to sign the module,\nyou're on your own at the moment, but maybe read the Linux kernel\ndocumentation on it here:\n\nAt this point, you'll have 2 new kernel modules. You should see them\nif you run:\n\n```sh\n$ ls -l /lib/modules/$(uname -r)/extra\ntotal 36\n-rw-r--r--    1 root     root         15272 May  9 20:42 virtio_pci_obsd.ko\n-rw-r--r--    1 root     root         19872 May  9 20:42 virtio_vmmci.ko\n```\n\n### 4. Loading the modules\nThis also should be easy now since their properly installed. Simply\nrun:\n\n```sh\n# modprobe virtio_vmmci\n```\n\nIt should load both the `virtio_vmmci.ko` and `virtio_pci_obsd.ko`\nmodules. They'll be visible when running `lsmod(8)`, but you won't see\na \"depends on\" entry due to it being a \"soft\" dependency.\n\n### 5. Checking it's Loaded\nAfter you load `virtio_pci_obsd.ko` you should see your system match\nand enable the vmmci PCI device. Check `dmesg(1)` and you should see\nsomething like:\n\n```\n[  825.819945] virtio_pci_obsd: loading out-of-tree module taints kernel.\n[  825.819945] virtio_pci_obsd: module verification failed: signature and/or required key missing - tainting kernel\n[  825.819945] virtio-pci-obsd 0000:00:05.0: runtime IRQ mapping not provided by arch\n[  825.819945] virtio_pci_obsd_match: matching 0x0777\n[  825.819945] virtio_pci_obsd_match: found OpenBSD device\n[  825.819945] virtio-pci-obsd 0000:00:05.0: enabling bus mastering\n```\n\nIf you check with `lspci(8)` in verbose mode (`lspci -v`) you should\nsee the device and the fact it's using our `virti_pci_obsd` driver:\n\n```\n00:05.0 Communication controller: Device 0b5d:0777\n        Subsystem: Device 0b5d:ffff\n        Flags: bus master, fast devsel, latency 0, IRQ 9\n        I/O ports at 5000 [size=4K]\n        Kernel driver in use: virtio-pci-obsd\n```\n\nWhen you load `virtio_vmmci.ko`, you should see a confirmation the\nmodule is loaded:\n\n```\n[  256.030878] virtio_vmmci: started VMM Control Interface driver\n```\n\nYou can enable debug mode either by passing a `debug=1` argument when\nloading the `virtio_vmmci.ko` module or toggle it afterwards by\nwriting either a `0` (off) or `1`/any positive integer (on) to\n`/sys/modules/virtio_vmmci/parameters/debug` as the root user. When\ndebug mode is on, you'll get extra dmesg noise like:\n\n```\n[17769.012388] virtio_vmmci: [vmmci_validate] not implemented\n[17769.012388] virtio_vmmci: [vmmci_probe] initializing vmmci device\n[17769.012388] virtio_vmmci: [vmmci_probe] ...found feature TIMESYNC\n[17769.012388] virtio_vmmci: [vmmci_probe] ...found feature ACK\n[17769.012388] virtio_vmmci: [vmmci_probe] ...found feature SYNCRTC\n[17769.012388] virtio_vmmci: started VMM Control Interface driver\n[17769.034540] virtio_vmmci: [clock_work_func] starting clock synchronization\n[17769.034864] virtio_vmmci: [clock_work_func] guest clock: 1550959642.629898000, host clock: 1550959642.638556712\n[17769.034867] virtio_vmmci: [clock_work_func] current time delta: -1.991341288\n[17769.034870] virtio_vmmci: [clock_work_func] clock synchronization routine finished\n```\n\nLastly, check the sysctl tables. The driver registers 2 particular\nvalues that contain the seconds and nanoseconds portion of the last\nmeasured drift amount:\n\n```\nyou@guest:~/virtio_vmmci$ sudo sysctl vmmci\nvmmci.drift_nsec = 199647574\nvmmci.drift_sec = 1\n```\n\nIn the above example, the total drift is `1.199647574 seconds`.\n\n\u003e In the future I may expose the last measured time as well\n\n### 5. Configuring autoloading at boot time\nThis is pretty simple in modern distros that use\n`/etc/modules-load.d`. As root, create a file\n`/etc/modules-load.d/virtio_vmmci.conf` with the contents:\n\n```\nvirtio_vmmci\n```\n\nAt boot, you should see the modules loaded automatically.\n\n## Testing and Confirming Module Installation\nThere are a few things you can do to validate your installation.\n\n### Clock Sync\nYou can easily test the clock synchronization by suspending your\nOpenBSD host by triggering `zzz` manually or by something like closing\nyour laptop lid. Wait at least 10 seconds or so and resume your\nOpenBSD system. In the Linux guest, your `dmesg(1)` output will tell\nyou (in less than 30 seconds) that it's detected a clock drift and\nit's sync'ing the clock:\n\n```\n[15670.027879] virtio_vmmci: [clock_work_func] current time delta: 91.482370612\n[15670.027879] virtio_vmmci: detected drift greater than 5 seconds, synchronizing clock\n[15670.027879] virtio_vmmci: [clock_work_func] clock synchronization routine finished\n```\n\nIf you check `date` or `timedatectl` on the Linux guest you should see\nthe system time is very close to our host time.\n\n### Clean Shutdown\nHow can we test a clean shutdown? It's not too hard, but it might not\nwork the same between distros and versions. Here's what I've done on\nAlpine 3.11.6.\n\nAssuming your vm is up and running:\n\n1. Use `tmux(1)` or another means of getting 2 terminal sessions going\n   at once.\n2. In one session, `vmctl console \u003cvm name or id\u003e` to connect to the\n   VM over the serial console. (This obviously assumes your guest is\n   configured to work that way.)\n3. In another session, issue `vmctl stop \u003cvm name or id\u003e`.\n4. Back in the serial console session, you should see your init\n   system...probably `systemd`...start running through the shutdown\n   process.\n\nThere _may_ be some variations. The Linux vmmci driver calls a kernel\nhelper function that handles orchestrating the shutdown via\nuserspace. (The question of how to shutdown a Linux system from\nkernelspace is quite fascinating to explore.)\n\n# Seldomly Asked Questions\nSome questions that people...mainly myself...have had...\n\n## Wait, why isn't this fixing my clock drift issues?\nMy initial release would constantly adjust the guest clock when\ndetecting drift. I since removed the functionality and will not re-add\nit no matter how much it's requested.\n\nSome reasons I removed it:\n\n- It's a bandaid on a bigger issue, not a real solution.\n- You can apply a bandaid already using something like `hwclock -us`,\n  but since it uses `settimeofday(2)` it may not trigger pending\n  timers properly!\n\nConstant, excessive drift shouldn't be the norm. Using refined-jiffies\nwill cause this.\n\nIf you or a loved one experience excessive clock drift in your Linux\nguests under OpenBSD's vmm(4)/vmd(8) hypervisor framework, please try\nthe following:\n\n- Build and install my other Linux kernel:\n  [vmm-clock](https://github.com/voutilad/vmm_clock)\n- Use OpenBSD-current as of 1 July 2020 or so when my vmd(8) patch[6]\n  was merged into the tree\n\n\u003e You will need BOTH...vmm-clock will crash your guest if you don't\n\u003e have a vmd(8) instance with the stability improvements.\n\n## _Isn't just using settimeofday(2) dangerous?_\nThis isn't using the userland `settimeofday(2)` system call and\ninstead using a particular kernel function (`do_settimeofday64`[3])\nthat appears to be pretty analagous to OpenBSD's kernel's\n`tc_setclock` function[4] in that it steps the system clock while\ntriggering any alarms or timeouts that would fire.\n\nLooking at how VirtualBox handles this with their userland guest\nadditions services, they look for large clock drifts where \"large\"\nis currently \u003e 30 minutes. If it's large, it just uses\n`settimeofday(2)`. Otherwise, it tries to use something like\n`adjtimex(2)` to accelerate the clock up to the correct time. (This is\nsomething I may consider for vmmci after some more usage/testing.)\n\nSee their source for `VBoxServiceTimeSync.cpp`[5].\n\n## _Can't you just use OpenNTPD or some other NTP daemon?_\nMaybe for small clock disturbances/drifts, but it's not ideal for\nmajor stepping and only solves the clock problem.\n\nThere are two reasons I'd consider using `virtio_vmmci` either in\naddition to or in place of relying on an NTP daemon:\n\n1. **Not every guest has network access.** This precludes NTP as an\n   option. Even if the guest has limited network access, it still\n   needs access to an NTP server, ideally multiple. This isn't always\n   the case.\n\n2. **Large clock drifts like when you suspend your laptop for an\n   evening make most NTP daemons sad.** I've never seen an NTP daemon\n   that is cool with just jumping the system time ahead\n   (i.e. _stepping_) like that. Some require special config to even\n   do. Yes, `ntpd(8)` supports a `-s` flag to do an actual set of the time\n   and not just an adjustment, but even as the man page says it's for\n   startup. (Useful for embedded, clock-less systems like a Raspberry\n   Pi.)\n\nA lot of modern Linux distros install and enable an NTP daemon by\ndefault these days. That's fine. But don't forget vmmci gives you\n**clean shutdowns** as well as properly stepping the clock after a\nlong suspend/hibernation!\n\n## Why all the nasty Virtio PCI glue code?\nFew reasons, but for more background see my email to\n_misc@openbsd.org:_ https://marc.info/?t=155102953000002\n\nIn short:\n\n1. OpenBSD purposely uses self-asigned PCI and Virtio device\n   identifiers to \"hide\" the VMM Control Interface device\n2. Linux's virtio pci code is a LOT more complex and is trying to\n   handle a variety of virtio devices...but can't handle a particular\n   quirk with how the VMM Control Interface deals with config register i/o.\n\n# Future Work\n\nSee the [issues](https://github.com/voutilad/virtio_vmmci/issues/)\npage for my ideas on future enhancements. Feel free to add some\nyourself, but keep in mind this is:\n\n1. Not my job...it's a hobby\n2. It's for my personal use first and foremost\n3. My current job is in software but has nothing to do with kernels,\n   virtualization, etc. so this is truly an after-hours thing.\n\n# Acknowledgements!\n1. Thanks to the OpenBSD `vmm(4)`/`vmd(8)` hackers...especially those that put\n   together OpenBSD's `vmmci(4)` driver which acted as my reference point.\n\n2. The [bootlin cross-referencer](https://elixir.bootlin.com/linux/latest/source)\n   because holy hell is that thing 10x more useful than poking around Torvald's\n   mirror of the official Linux Git repo.\n\n3. This page from \"The kernel development community\" was very helpful in\n   figuring out how to schedule \"deferred work\" in the kernel:\n   https://linux-kernel-labs.github.io/master/labs/deferred_work.html\n\n4. The `virtio_balloon.c` driver in the Linux kernel tree is a relatively\n   simple virtio example to understand Linux virtio drivers.\n\n5. The wireguard kernel module source tree for showing how to properly\n   build out of tree modules:\n   https://git.zx2c4.com/wireguard-linux-compat/tree/src\n\n6. Folks that have helped test on different distros with different\n   kernel versions :-)\n\n# Footnotes\nGitHub might not render these...but believe me they're here :-)\n\n[1] Linux Kernel documentation on generating a private key for signing\nkernel modules:\nhttps://www.kernel.org/doc/html/v4.15/admin-guide/module-signing.html#generating-signing-keys\n\n[2] See this write-up on time-sync in vm's:\nhttp://archive.is/ndiy3\n\n[3] See the `time/timekeeping.c` source file:\nhttps://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/kernel/time/timekeeping.c?h=v4.20#n1220\n\n[4] OpenBSD's `sys/kern/kern_tc.c`:\nhttps://github.com/openbsd/src/blob/e12a049bd4bbd1e8315c373a739e08972ed6dd1d/sys/kern/kern_tc.c#L382\n\n[5] VirtualBox's `VBoxServiceTimeSync.cpp`:\nhttps://www.virtualbox.org/browser/vbox/trunk/src/VBox/Additions/common/VBoxService/VBoxServiceTimeSync.cpp?rev=76553#L683\n\n[6] My vmd(8) stability fixes:\nhttps://github.com/openbsd/src/commit/08fd0ce3179b426bc00beaee67fffdfa71997830\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvoutilad%2Fvirtio_vmmci","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvoutilad%2Fvirtio_vmmci","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvoutilad%2Fvirtio_vmmci/lists"}