https://github.com/pmem/run_qemu
A script to create bootable OS images, and run qemu with a locally built kernel.
https://github.com/pmem/run_qemu
linux linux-kernel mkosi qemu
Last synced: 7 months ago
JSON representation
A script to create bootable OS images, and run qemu with a locally built kernel.
- Host: GitHub
- URL: https://github.com/pmem/run_qemu
- Owner: pmem
- License: cc0-1.0
- Created: 2021-03-08T18:22:31.000Z (about 5 years ago)
- Default Branch: main
- Last Pushed: 2025-06-13T14:30:30.000Z (10 months ago)
- Last Synced: 2025-09-01T11:25:33.767Z (7 months ago)
- Topics: linux, linux-kernel, mkosi, qemu
- Language: Shell
- Homepage:
- Size: 232 KB
- Stars: 69
- Watchers: 14
- Forks: 28
- Open Issues: 12
-
Metadata Files:
- Readme: README.md
- License: LICENSE
- Codeowners: .github/CODEOWNERS
Awesome Lists containing this project
README
# Requirements
- mkosi
- e.g. `dnf install mkosi`
- `qemu-system-x86_64`
- rsync
- dracut
- enabled virtualization (KVM)
- nopasswd sudo preferred, or run as root, or enter passwords into the prompt
several times
- `argbash` to generate the argument parser lib (using `parser_generator.m4`)
## mkosi
Finding an mkosi version compatible with your operating system version and
particular test configuration can sometimes be challenging. First, try the
version packaged with your operating system. This will indirectly install
mkosi dependencies (which have surprisingly not changed that much across
mkosi versions). If that OS-provided mkosi version does not work for you,
check the long mkosi section below.
# Installation
- symlink the `run_qemu.sh` script into somewhere in your `PATH`
- e.g. `ln -s ~/git/run_qemu/run_qemu.sh ~/bin/run_qemu.sh`
- **Note:** Supporting files in this repo are required to be in the same
location as the script, after any symlink resolution. Copying just the
script itself will not work.
- **Bash Completion**
- Copy or symlink the `run_qemu` file into the default bash completions dir
- The completions directory can be found using:
`pkg-config --variable=completionsdir bash-completion`
# Usage Notes
- Run this from the top level of a Linux kernel tree
- e.g. `run_qemu.sh --cxl --git-qemu`
- The script can/will:
- Build the kernel with whatever .config is present
(It is up to the user to manage the .config)
- Create a rootfs image with the chosen distro using `mkosi`
- Perform some basic setup on the rootfs, including installing the kernel,
utilities (such as `ndctl`), and other convenience operations such as
copying `.ssh/*.pub` keys for easy access, and your `~/.bashrc`
etc.
- Boot qemu with the newly compiled kernel provided on the qemu command line,
and using the rootfs image above
- Various options influence the qemu command line generated - there are
options to select NUMA config, NVDIMMs, NVME devices, CXL devices etc.
- More detailed CLI help is available with `run_qemu.sh --help`
- Once qemu starts, in nographic mode, the Linux console 'takes over' the
terminal. To interact with it, the following are useful:
- `Ctrl-a c` : switch between the qemu monitor prompt `(qemu)` and console
- `Ctrl-a x` : kill qemu and exit
- `mkosi` creates a package cache in `mkosi.cache/` If a cache is present,
it will always use only that, and never go over the network even if newer
packages are available. To force re-fetching everything, remove this
directory, or --rebuild=wipe which removes the `builddir` entirely.
- Which `qemu` to use can be overridden from the environment:
`qemu=/path/to/qemu/build/qemu-system-x86_64 ./run_qemu.sh [options]`
- List of variables that have overrides via `env`:
- `qemu`
- `gdb`
- `distro`
- `rev`
- `builddir`
- `ndctl`
- To use the 'hostfwd' network, put this in your `.ssh/config`:
Host rq
Hostname localhost
User root
Port 10022
StrictHostKeyChecking no
UserKnownHostsFile /dev/null
And then `ssh rq`. You may need to open port 10022 on any local firewalls.
- The root password for the guest VM is `root` by default but note many Linux
distributions restrict remote root access in various ways. The serial console
automatically logs in, and a password isn't required.
## CXL Usage
The script enables generating a sane QEMU commandline for instantiating a basic CXL topology. Since QEMU support for CXL isn't yet upstream, `--git-qemu` is additionally required. The CXL related options are:
- `--cxl`: Enables a simple CXL topology with:
- single host bridge
- 512M window size at 0x4c00000000
- Bus #52
- single root port
- single Type 3 device
- Persistent 256M
- simple label storage area
- --cxl-debug: Add any and all flags for extra debug (kernel and QEMU)
- --cxl-hb: Turn q35 into a CXL capable Host bridge. Don't use this option unless you're working on support for this.
- --cxl-test-run: Attempt to do a sanity test of the kernel and QEMU configuration.
## DAX Usage
- --dax-debug: Add any and all flags for extra debug of dax modules (kernel)
### Kernel config
- Make sure to Turn on CXL related options in the kernel's .config:
```
$ grep -i cxl .config
CONFIG_CXL_BUS=y
CONFIG_CXL_PCI=m
CONFIG_CXL_MEM_RAW_COMMANDS=y
CONFIG_CXL_ACPI=m
CONFIG_CXL_PMEM=m
CONFIG_CXL_MEM=m
CONFIG_CXL_PORT=y
CONFIG_CXL_SUSPEND=y
```
The following is a way to check basic sanity within the QEMU guest:
```shell
lspci | grep '3[45]:00'
34:00.0 PCI bridge: Intel Corporation Device 7075
35:00.0 Memory controller [0502]: Intel Corporation Device 0d93 (rev 01)
readlink -f /sys/bus/cxl/devices/mem0
/sys/devices/pci0000:34/0000:34:00.0/0000:35:00.0/mem0
```
# mkosi
## run mkosi from source
There are many variables (OS distro and version for host and build
machine, mkosi version, python dependencies,...) and no solution works
in every situation. `pipx` offers a relatively good compatibility and
simplicity trade-off, so it is explained first and in most detail but
if this does not work then there are alternatives mentioned below.
```
sudo dnf install mkosi
sudo dnf remove --noautoremove mkosi # keep mkosi dependencies installed
git clone https://github.com/systemd/mkosi
cd mkosi
# Make sure everything is clean when re-using an existing clone
git status --ignored
pipx install --system-site-packages --editable .
pipx list
mkosi --version
export mkosi_bin=$(type -ap mkosi)
```
Defining `mkosi_bin` is required because `run_qemu.sh` runs mkosi as root
(`sudo $mkosi_bin`) but `pipx` installs in a subdirectory of your
`$HOME` which is not in root's `PATH`. Installing mkosi as root is not
recommended and does not even work for various, complex reasons.
If you get a "module mkosi not found" error with some mkosi versions, define
`$mkosi_bin` like this instead:
```
export mkosi_bin="$HOME"/.local/share/pipx/venvs/mkosi/bin/mkosi
"$mkosi_bin" --version
```
To select a specific `mkosi` release:
```
$ cd mkosi
$ git checkout vNN # where vNN is the release desired
```
Thanks to the `--editable` option, there's no need to re-run `pipx` when
using git to switch between different mkosi versions... **unless** some
mkosi dependencies change! If you experience any mkosi error or have any
doubt then use `pipx` to `uninstall` and re-install.
Alternatively, can try to add `mkosi/bin/mkosi` to your PATH or a create a
symlink to it instead of using `pipx`. This seems to work with mkosi
releases v15 and above.
`mkosi/README.md` describes more installation alternatives. Note they can
differ across mkosi versions.
## mkosi v15+ (Fedora 39+)
mkosi v15 made a lot of backwards-incompatible changes. run_qemu has
configuration files for both mkosi v14- and mkosi v15+ now but please
report any bug.
To mitigate this, Fedora 40 packages both mkosi 14 and 22. So, it is not
required to clone and and run mkosi from source to switch between these
two versions. Instead:
```
# dnf remove --noautoremove mkosi
# dnf install mkosi14
```
Fedora 41 has stopped packaging mkosi14.
### mkosi configuration files in run_qemu
Fortunately, the location of mkosi configuration files changed at the same
v15 time. So `run_qemu.sh` creates a different mkosi configuration folder depending
on which mkosi version is detected: `qbuild/mkosi.default.d/*.conf` for
version 14 and before versus `qbuild/mkosi.conf.d/*.conf` for version 15
and above.
While no such major break of backwards compatibility has happened after v15
(yet?), mkosi development is very fast-paced. Various Linux distributions come
with various mkosi versions. So try to keep mkosi configuration(s) as simple as
possible to avoid accidentally breaking someone else using a different mkosi
version. Rely on mkosi default values as much as possible.
Fortunately, most mkosi versions are thoroughly documented and you can
easily check the documentation of any version without installing anything.
- For versions 14 and before, use this syntax:
https://github.com/systemd/mkosi/blob/v14/man/mkosi.1
- For versions between 15 and 24 go to:
https://github.com/systemd/mkosi/blob/v15/mkosi/resources/mkosi.md
- For versions 25 and above:
https://github.com/systemd/mkosi/blob/v25/mkosi/resources/man/mkosi.1.md