Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/cloud-hypervisor/cloud-hypervisor

A Virtual Machine Monitor for modern Cloud workloads. Features include CPU, memory and device hotplug, support for running Windows and Linux guests, device offload with vhost-user and a minimal compact footprint. Written in Rust with a strong focus on security.
https://github.com/cloud-hypervisor/cloud-hypervisor

cloud-workloads kvm rust-vmm virtualization

Last synced: 1 day ago
JSON representation

A Virtual Machine Monitor for modern Cloud workloads. Features include CPU, memory and device hotplug, support for running Windows and Linux guests, device offload with vhost-user and a minimal compact footprint. Written in Rust with a strong focus on security.

Awesome Lists containing this project

README 

        
- [1. What is Cloud Hypervisor?](#1-what-is-cloud-hypervisor)

  - [Objectives](#objectives)

    - [High Level](#high-level)

    - [Architectures](#architectures)

    - [Guest OS](#guest-os)

- [2. Getting Started](#2-getting-started)

  - [Host OS](#host-os)

  - [Use Pre-built Binaries](#use-pre-built-binaries)

  - [Packages](#packages)

  - [Building from Source](#building-from-source)

  - [Booting Linux](#booting-linux)

    - [Firmware Booting](#firmware-booting)

    - [Custom Kernel and Disk Image](#custom-kernel-and-disk-image)

      - [Building your Kernel](#building-your-kernel)

      - [Disk image](#disk-image)

      - [Booting the guest VM](#booting-the-guest-vm)

- [3. Status](#3-status)

  - [Hot Plug](#hot-plug)

  - [Device Model](#device-model)

  - [Roadmap](#roadmap)

- [4. Relationship with _Rust VMM_ Project](#4-relationship-with-rust-vmm-project)

  - [Differences with Firecracker and crosvm](#differences-with-firecracker-and-crosvm)

- [5. Community](#5-community)

  - [Contribute](#contribute)

  - [Slack](#slack)

  - [Mailing list](#mailing-list)

  - [Security issues](#security-issues)



# 1. What is Cloud Hypervisor?



Cloud Hypervisor is an open source Virtual Machine Monitor (VMM) that runs on

top of the [KVM](https://www.kernel.org/doc/Documentation/virtual/kvm/api.txt)

hypervisor and the Microsoft Hypervisor (MSHV).



The project focuses on running modern, _Cloud Workloads_, on specific, common,

hardware architectures. In this case _Cloud Workloads_ refers to those that are

run by customers inside a Cloud Service Provider. This means modern operating

systems with most I/O handled by

paravirtualised devices (e.g. _virtio_), no requirement for legacy devices, and

64-bit CPUs.



Cloud Hypervisor is implemented in [Rust](https://www.rust-lang.org/) and is

based on the [Rust VMM](https://github.com/rust-vmm) crates.



## Objectives



### High Level



- Runs on KVM or MSHV

- Minimal emulation

- Low latency

- Low memory footprint

- Low complexity

- High performance

- Small attack surface

- 64-bit support only

- CPU, memory, PCI hotplug

- Machine to machine migration



### Architectures



Cloud Hypervisor supports the `x86-64` and `AArch64` architectures. There are

minor differences in functionality between the two architectures

(see [#1125](https://github.com/cloud-hypervisor/cloud-hypervisor/issues/1125)).



### Guest OS



Cloud Hypervisor supports `64-bit Linux` and Windows 10/Windows Server 2019.



# 2. Getting Started



The following sections describe how to build and run Cloud Hypervisor.



## Prerequisites for AArch64



- AArch64 servers (recommended) or development boards equipped with the GICv3

  interrupt controller.



## Host OS



For required KVM functionality and adequate performance the recommended host

kernel version is 5.13. The majority of the CI currently tests with kernel

version 5.15.



## Use Pre-built Binaries



The recommended approach to getting started with Cloud Hypervisor is by using a

pre-built binary. Binaries are available for the [latest

release](https://github.com/cloud-hypervisor/cloud-hypervisor/releases/latest).

Use `cloud-hypervisor-static` for `x86-64` or `cloud-hypervisor-static-aarch64`

for `AArch64` platform.



## Packages



For convenience, packages are also available targeting some popular Linux

distributions. This is thanks to the [Open Build

Service](https://build.opensuse.org). The [OBS

README](https://github.com/cloud-hypervisor/obs-packaging) explains how to

enable the repository in a supported Linux distribution and install Cloud Hypervisor

and accompanying packages. Please report any packaging issues in the

[obs-packaging](https://github.com/cloud-hypervisor/obs-packaging) repository.



## Building from Source



Please see the [instructions for building from source](docs/building.md) if you

do not wish to use the pre-built binaries.



## Booting Linux



Cloud Hypervisor supports direct kernel boot (the x86-64 kernel requires the kernel

built with PVH support or a bzImage) or booting via a firmware (either [Rust Hypervisor

Firmware](https://github.com/cloud-hypervisor/rust-hypervisor-firmware) or an

edk2 UEFI firmware called `CLOUDHV` / `CLOUDHV_EFI`.)



Binary builds of the firmware files are available for the latest release of

[Rust Hypervisor

Firmware](https://github.com/cloud-hypervisor/rust-hypervisor-firmware/releases/latest)

and [our edk2

repository](https://github.com/cloud-hypervisor/edk2/releases/latest)



The choice of firmware depends on your guest OS choice; some experimentation

may be required.



### Firmware Booting



Cloud Hypervisor supports booting disk images containing all needed components

to run cloud workloads, a.k.a. cloud images.



The following sample commands will download an Ubuntu Cloud image, converting

it into a format that Cloud Hypervisor can use and a firmware to boot the image

with.



```shell

$ wget https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64.img

$ qemu-img convert -p -f qcow2 -O raw focal-server-cloudimg-amd64.img focal-server-cloudimg-amd64.raw

$ wget https://github.com/cloud-hypervisor/rust-hypervisor-firmware/releases/download/0.4.2/hypervisor-fw

```



The Ubuntu cloud images do not ship with a default password so it necessary to

use a `cloud-init` disk image to customise the image on the first boot. A basic

`cloud-init` image is generated by this [script](scripts/create-cloud-init.sh).

This seeds the image with a default username/password of `cloud/cloud123`. It

is only necessary to add this disk image on the first boot. Script also assigns

default IP address using `test_data/cloud-init/ubuntu/local/network-config` details

with `--net "mac=12:34:56:78:90:ab,tap="` option. Then the matching mac address

interface will be enabled as per `network-config` details.



```shell

$ sudo setcap cap_net_admin+ep ./cloud-hypervisor

$ ./create-cloud-init.sh

$ ./cloud-hypervisor \

	--kernel ./hypervisor-fw \

	--disk path=focal-server-cloudimg-amd64.raw path=/tmp/ubuntu-cloudinit.img \

	--cpus boot=4 \

	--memory size=1024M \

	--net "tap=,mac=,ip=,mask="

```



If access to the firmware messages or interaction with the boot loader (e.g.

GRUB) is required then it necessary to switch to the serial console instead of

`virtio-console`.



```shell

$ ./cloud-hypervisor \

	--kernel ./hypervisor-fw \

	--disk path=focal-server-cloudimg-amd64.raw path=/tmp/ubuntu-cloudinit.img \

	--cpus boot=4 \

	--memory size=1024M \

	--net "tap=,mac=,ip=,mask=" \

	--serial tty \

	--console off

```



### Custom Kernel and Disk Image



#### Building your Kernel



Cloud Hypervisor also supports direct kernel boot. For x86-64, a `vmlinux` ELF kernel (compiled with PVH support) or a regular bzImage are supported. In order to support development there is a custom branch; however provided the required options are enabled any recent kernel will suffice.



To build the kernel:



```shell

# Clone the Cloud Hypervisor Linux branch

$ git clone --depth 1 https://github.com/cloud-hypervisor/linux.git -b ch-6.2 linux-cloud-hypervisor

$ pushd linux-cloud-hypervisor

# Use the x86-64 cloud-hypervisor kernel config to build your kernel for x86-64

$ wget https://raw.githubusercontent.com/cloud-hypervisor/cloud-hypervisor/main/resources/linux-config-x86_64

# Use the AArch64 cloud-hypervisor kernel config to build your kernel for AArch64

$ wget https://raw.githubusercontent.com/cloud-hypervisor/cloud-hypervisor/main/resources/linux-config-aarch64

$ cp linux-config-x86_64 .config  # x86-64

$ cp linux-config-aarch64 .config # AArch64

# Do native build of the x86-64 kernel

$ KCFLAGS="-Wa,-mx86-used-note=no" make bzImage -j `nproc`

# Do native build of the AArch64 kernel

$ make -j `nproc`

$ popd

```



For x86-64, the `vmlinux` kernel image will then be located at

`linux-cloud-hypervisor/arch/x86/boot/compressed/vmlinux.bin`.

For AArch64, the `Image` kernel image will then be located at

`linux-cloud-hypervisor/arch/arm64/boot/Image`.



#### Disk image



For the disk image the same Ubuntu image as before can be used. This contains

an `ext4` root filesystem.



```shell

$ wget https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64.img # x86-64

$ wget https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-arm64.img # AArch64

$ qemu-img convert -p -f qcow2 -O raw focal-server-cloudimg-amd64.img focal-server-cloudimg-amd64.raw # x86-64

$ qemu-img convert -p -f qcow2 -O raw focal-server-cloudimg-arm64.img focal-server-cloudimg-arm64.raw # AArch64

```



#### Booting the guest VM



These sample commands boot the disk image using the custom kernel whilst also

supplying the desired kernel command line.



- x86-64



```shell

$ sudo setcap cap_net_admin+ep ./cloud-hypervisor

$ ./create-cloud-init.sh

$ ./cloud-hypervisor \

	--kernel ./linux-cloud-hypervisor/arch/x86/boot/compressed/vmlinux.bin \

	--disk path=focal-server-cloudimg-amd64.raw path=/tmp/ubuntu-cloudinit.img \

	--cmdline "console=hvc0 root=/dev/vda1 rw" \

	--cpus boot=4 \

	--memory size=1024M \

	--net "tap=,mac=,ip=,mask="

```



- AArch64



```shell

$ sudo setcap cap_net_admin+ep ./cloud-hypervisor

$ ./create-cloud-init.sh

$ ./cloud-hypervisor \

	--kernel ./linux-cloud-hypervisor/arch/arm64/boot/Image \

	--disk path=focal-server-cloudimg-arm64.raw path=/tmp/ubuntu-cloudinit.img \

	--cmdline "console=hvc0 root=/dev/vda1 rw" \

	--cpus boot=4 \

	--memory size=1024M \

	--net "tap=,mac=,ip=,mask="

```



If earlier kernel messages are required the serial console should be used instead of `virtio-console`.



- x86-64



```shell

$ ./cloud-hypervisor \

	--kernel ./linux-cloud-hypervisor/arch/x86/boot/compressed/vmlinux.bin \

	--console off \

	--serial tty \

	--disk path=focal-server-cloudimg-amd64.raw \

	--cmdline "console=ttyS0 root=/dev/vda1 rw" \

	--cpus boot=4 \

	--memory size=1024M \

	--net "tap=,mac=,ip=,mask="

```



- AArch64



```shell

$ ./cloud-hypervisor \

	--kernel ./linux-cloud-hypervisor/arch/arm64/boot/Image \

	--console off \

	--serial tty \

	--disk path=focal-server-cloudimg-arm64.raw \

	--cmdline "console=ttyAMA0 root=/dev/vda1 rw" \

	--cpus boot=4 \

	--memory size=1024M \

	--net "tap=,mac=,ip=,mask="

```



# 3. Status



Cloud Hypervisor is under active development. The following stability

guarantees are currently made:



* The API (including command line options) will not be removed or changed in a

  breaking way without a minimum of 2 major releases notice. Where possible

  warnings will be given about the use of deprecated functionality and the

  deprecations will be documented in the release notes.



* Point releases will be made between individual releases where there are

  substantial bug fixes or security issues that need to be fixed. These point

  releases will only include bug fixes.



Currently the following items are **not** guaranteed across updates:



* Snapshot/restore is not supported across different versions

* Live migration is not supported across different versions

* The following features are considered experimental and may change

  substantially between releases: TDX, vfio-user, vDPA.



Further details can be found in the [release documentation](docs/releases.md).



As of 2023-01-03, the following cloud images are supported:



- [Ubuntu Focal](https://cloud-images.ubuntu.com/focal/current/) (focal-server-cloudimg-{amd64,arm64}.img)

- [Ubuntu Jammy](https://cloud-images.ubuntu.com/jammy/current/) (jammy-server-cloudimg-{amd64,arm64}.img )

- [Fedora 36](https://fedora.mirrorservice.org/fedora/linux/releases/36/Cloud/) ([Fedora-Cloud-Base-36-1.5.x86_64.raw.xz](https://fedora.mirrorservice.org/fedora/linux/releases/36/Cloud/x86_64/images/) / [Fedora-Cloud-Base-36-1.5.aarch64.raw.xz](https://fedora.mirrorservice.org/fedora/linux/releases/36/Cloud/aarch64/images/))



Direct kernel boot to userspace should work with a rootfs from most

distributions although you may need to enable exotic filesystem types in the

reference kernel configuration (e.g. XFS or btrfs.)



## Hot Plug



Cloud Hypervisor supports hotplug of CPUs, passthrough devices (VFIO),

`virtio-{net,block,pmem,fs,vsock}` and memory resizing. This

[document](docs/hotplug.md) details how to add devices to a running VM.



## Device Model



Details of the device model can be found in this

[documentation](docs/device_model.md).



## Roadmap



The project roadmap is tracked through a [GitHub

project](https://github.com/orgs/cloud-hypervisor/projects/6).



# 4. Relationship with _Rust VMM_ Project



In order to satisfy the design goal of having a high-performance,

security-focused hypervisor the decision was made to use the

[Rust](https://www.rust-lang.org/) programming language. The language's strong

focus on memory and thread safety makes it an ideal candidate for implementing

VMMs.



Instead of implementing the VMM components from scratch, Cloud Hypervisor is

importing the [Rust VMM](https://github.com/rust-vmm) crates, and sharing code

and architecture together with other VMMs like e.g. Amazon's

[Firecracker](https://firecracker-microvm.github.io/) and Google's

[crosvm](https://chromium.googlesource.com/chromiumos/platform/crosvm/).



Cloud Hypervisor embraces the _Rust VMM_ project's goals, which is to be able

to share and re-use as many virtualization crates as possible.



## Differences with Firecracker and crosvm



A large part of the Cloud Hypervisor code is based on either the Firecracker or

the crosvm project's implementations. Both of these are VMMs written in Rust

with a focus on safety and security, like Cloud Hypervisor.



The goal of the Cloud Hypervisor project differs from the aforementioned

projects in that it aims to be a general purpose VMM for _Cloud Workloads_ and

not limited to container/serverless or client workloads.



The Cloud Hypervisor community thanks the communities of both the Firecracker

and crosvm projects for their excellent work.



# 5. Community



The Cloud Hypervisor project follows the governance, and community guidelines

described in the [Community](https://github.com/cloud-hypervisor/community)

repository.



## Contribute



The project strongly believes in building a global, diverse and collaborative

community around the Cloud Hypervisor project. Anyone who is interested in

[contributing](CONTRIBUTING.md) to the project is welcome to participate.



Contributing to a open source project like Cloud Hypervisor covers a lot more

than just sending code. Testing, documentation, pull request

reviews, bug reports, feature requests, project improvement suggestions, etc,

are all equal and welcome means of contribution. See the

[CONTRIBUTING](CONTRIBUTING.md) document for more details.



## Slack



Get an [invite to our Slack channel](https://join.slack.com/t/cloud-hypervisor/shared_invite/enQtNjY3MTE3MDkwNDQ4LWQ1MTA1ZDVmODkwMWQ1MTRhYzk4ZGNlN2UwNTI3ZmFlODU0OTcwOWZjMTkwZDExYWE3YjFmNzgzY2FmNDAyMjI),

 [join us on Slack](https://cloud-hypervisor.slack.com/), and [participate in our community activities](https://cloud-hypervisor.slack.com/archives/C04R5DUQVBN).



## Mailing list



Please report bugs using the [GitHub issue

tracker](https://github.com/cloud-hypervisor/cloud-hypervisor/issues) but for

broader community discussions you may use our [mailing

list](https://lists.cloudhypervisor.org/g/dev/).



## Security issues



Please contact the maintainers listed in the MAINTAINERS.md file with security issues.