{"id":7558147,"url":"https://github.com/containers/virtcontainers","last_synced_at":"2025-07-08T16:30:40.462Z","repository":{"id":66883965,"uuid":"71125072","full_name":"containers/virtcontainers","owner":"containers","description":"A Go package for building hardware virtualized container runtimes","archived":true,"fork":false,"pushed_at":"2018-03-19T21:25:13.000Z","size":11137,"stargazers_count":139,"open_issues_count":13,"forks_count":43,"subscribers_count":26,"default_branch":"master","last_synced_at":"2024-08-05T08:33:00.887Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/containers.png","metadata":{"files":{"readme":"README-original.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2016-10-17T10:08:28.000Z","updated_at":"2024-04-18T13:10:23.000Z","dependencies_parsed_at":null,"dependency_job_id":"1c5b3939-828c-4061-9e5d-a7d091b6eeba","html_url":"https://github.com/containers/virtcontainers","commit_stats":null,"previous_names":[],"tags_count":24,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/containers%2Fvirtcontainers","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/containers%2Fvirtcontainers/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/containers%2Fvirtcontainers/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/containers%2Fvirtcontainers/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/containers","download_url":"https://codeload.github.com/containers/virtcontainers/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225449265,"owners_count":17476066,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-04-08T01:50:20.615Z","updated_at":"2024-11-20T00:30:46.263Z","avatar_url":"https://github.com/containers.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"[![Build Status](https://travis-ci.org/containers/virtcontainers.svg?branch=master)](https://travis-ci.org/containers/virtcontainers)\n[![Build Status](http://cc-jenkins-ci.westus2.cloudapp.azure.com/job/virtcontainers-ubuntu-16-04-master/badge/icon)](http://cc-jenkins-ci.westus2.cloudapp.azure.com/job/virtcontainers-ubuntu-16-04-master)\n[![Build Status](http://cc-jenkins-ci.westus2.cloudapp.azure.com/job/virtcontainers-ubuntu-17-04-master/badge/icon)](http://cc-jenkins-ci.westus2.cloudapp.azure.com/job/virtcontainers-ubuntu-17-04-master)\n[![Build Status](http://cc-jenkins-ci.westus2.cloudapp.azure.com/job/virtcontainers-fedora-26-master/badge/icon)](http://cc-jenkins-ci.westus2.cloudapp.azure.com/job/virtcontainers-fedora-26-master)\n[![Go Report Card](https://goreportcard.com/badge/github.com/containers/virtcontainers)](https://goreportcard.com/report/github.com/containers/virtcontainers)\n[![Coverage Status](https://coveralls.io/repos/github/containers/virtcontainers/badge.svg?branch=master)](https://coveralls.io/github/containers/virtcontainers?branch=master)\n[![GoDoc](https://godoc.org/github.com/containers/virtcontainers?status.svg)](https://godoc.org/github.com/containers/virtcontainers)\n\nTable of Contents\n=================\n\n   * [What is it ?](#what-is-it-)\n   * [Background](#background)\n   * [Out of scope](#out-of-scope)\n      * [virtcontainers and Kubernetes CRI](#virtcontainers-and-kubernetes-cri)\n   * [Design](#design)\n      * [Pods](#pods)\n      * [Hypervisors](#hypervisors)\n      * [Agents](#agents)\n      * [Shim](#shim)\n      * [Proxy](#proxy)\n   * [API](#api)\n      * [Pod API](#pod-api)\n      * [Container API](#container-api)\n   * [Networking](#networking)\n      * [CNM](#cnm)\n      * [CNI](#cni)\n   * [Storage](#storage)\n      * [How to check if container uses devicemapper block device as its rootfs](#how-to-check-if-container-uses-devicemapper-block-device-as-its-rootfs)\n   * [Devices](#devices)\n      * [How to pass a device using VFIO-passthrough](#how-to-pass-a-device-using-vfio-passthrough)\n   * [Developers](#developers)\n\n# What is it ?\n\n`virtcontainers` is a Go library that can be used to build hardware-virtualized container\nruntimes.\n\n# Background\n\nThe few existing VM-based container runtimes (Clear Containers, runv, rkt's\nkvm stage 1) all share the same hardware virtualization semantics but use different\ncode bases to implement them. `virtcontainers`'s goal is to factorize this code into\na common Go library.\n\nIdeally, VM-based container runtime implementations would become translation\nlayers from the runtime specification they implement (e.g. the [OCI runtime-spec][oci]\nor the [Kubernetes CRI][cri]) to the `virtcontainers` API.\n\n`virtcontainers` is [Clear Containers][cc]'s runtime foundational package for their\n[runtime][cc-runtime] implementation\n\n[oci]: https://github.com/opencontainers/runtime-spec\n[cri]: https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/container-runtime-interface-v1.md\n[cc]: https://github.com/clearcontainers/\n[cc-runtime]: https://github.com/clearcontainers/runtime/\n\n# Out of scope\n\nImplementing a container runtime is out of scope for this project. Any\ntools or executables in this repository are only provided for demonstration or\ntesting purposes.\n\n## virtcontainers and Kubernetes CRI\n\n`virtcontainers`'s API is loosely inspired by the Kubernetes [CRI][cri] because\nwe believe it provides the right level of abstractions for containerized pods.\nHowever, despite the API similarities between the two projects, the goal of\n`virtcontainers` is _not_ to build a CRI implementation, but instead to provide a\ngeneric, runtime-specification agnostic, hardware-virtualized containers\nlibrary that other projects could leverage to implement CRI themselves.\n\n# Design\n\n## Pods\n\nThe `virtcontainers` execution unit is a _pod_, i.e. `virtcontainers` users start pods where\ncontainers will be running.\n\n`virtcontainers` creates a pod by starting a virtual machine and setting the pod\nup within that environment. Starting a pod means launching all containers with\nthe VM pod runtime environment.\n\n## Hypervisors\n\nThe `virtcontainers` package relies on hypervisors to start and stop virtual machine where\npods will be running. An hypervisor is defined by an Hypervisor interface implementation,\nand the default implementation is the QEMU one.\n\n## Agents\n\nDuring the lifecycle of a container, the runtime running on the host needs to interact with\nthe virtual machine guest OS in order to start new commands to be executed as part of a given\ncontainer workload, set new networking routes or interfaces, fetch a container standard or\nerror output, and so on.\nThere are many existing and potential solutions to resolve that problem and `virtcontainers` abstracts\nthis through the Agent interface.\n\n## Shim\n\nIn some cases the runtime will need a translation shim between the higher level container\nstack (e.g. Docker) and the virtual machine holding the container workload. This is needed\nfor container stacks that make strong assumptions on the nature of the container they're\nmonitoring. In cases where they assume containers are simply regular host processes, a shim\nlayer is needed to translate host specific semantics into e.g. agent controlled virtual\nmachine ones.\n\n## Proxy\n\nWhen hardware virtualized containers have limited I/O multiplexing capabilities,\nruntimes may decide to rely on an external host proxy to support cases where several\nruntime instances are talking to the same container.\n\n# API\n\nThe high level `virtcontainers` API is the following one:\n\n## Pod API\n\n* `CreatePod(podConfig PodConfig)` creates a Pod.\nThe virtual machine is started and the Pod is prepared.\n\n* `DeletePod(podID string)` deletes a Pod.\nThe virtual machine is shut down and all information related to the Pod are removed.\nThe function will fail if the Pod is running. In that case `StopPod()` has to be called first.\n\n* `StartPod(podID string)` starts an already created Pod.\nThe Pod and all its containers are started.\n\n* `RunPod(podConfig PodConfig)` creates and starts a Pod.\nThis performs `CreatePod()` + `StartPod()`.\n\n* `StopPod(podID string)` stops an already running Pod.\nThe Pod and all its containers are stopped.\n\n* `PausePod(podID string)` pauses an existing Pod.\n\n* `ResumePod(podID string)` resume a paused Pod.\n\n* `StatusPod(podID string)` returns a detailed Pod status.\n\n* `ListPod()` lists all Pods on the host.\nIt returns a detailed status for every Pod.\n\n## Container API\n\n* `CreateContainer(podID string, containerConfig ContainerConfig)` creates a Container on an existing Pod.\n\n* `DeleteContainer(podID, containerID string)` deletes a Container from a Pod.\nIf the Container is running it has to be stopped first.\n\n* `StartContainer(podID, containerID string)` starts an already created Container.\nThe Pod has to be running.\n\n* `StopContainer(podID, containerID string)` stops an already running Container.\n\n* `EnterContainer(podID, containerID string, cmd Cmd)` enters an already running Container and runs a given command.\n\n* `StatusContainer(podID, containerID string)` returns a detailed Container status.\n\n* `KillContainer(podID, containerID string, signal syscall.Signal, all bool)` sends a signal to all or one container inside a Pod.\n\nAn example tool using the `virtcontainers` API is provided in the `hack/virtc` package.\n\n# Networking\n\n`virtcontainers` supports the 2 major container networking models: the [Container Network Model (CNM)][cnm] and the [Container Network Interface (CNI)][cni].\n\nTypically the former is the Docker default networking model while the later is used on Kubernetes deployments.\n\n`virtcontainers` callers can select one or the other, on a per pod basis, by setting their `PodConfig`'s `NetworkModel` field properly.\n\n[cnm]: https://github.com/docker/libnetwork/blob/master/docs/design.md\n[cni]: https://github.com/containernetworking/cni/\n\n## CNM\n\n![High-level CNM Diagram](documentation/network/CNM_overall_diagram.png)\n\n__CNM lifecycle__\n\n1.  RequestPool\n\n2.  CreateNetwork\n\n3.  RequestAddress\n\n4.  CreateEndPoint\n\n5.  CreateContainer\n\n6.  Create config.json\n\n7.  Create PID and network namespace\n\n8.  ProcessExternalKey\n\n9.  JoinEndPoint\n\n10. LaunchContainer\n\n11. Launch\n\n12. Run container\n\n![Detailed CNM Diagram](documentation/network/CNM_detailed_diagram.png)\n\n__Runtime network setup with CNM__\n\n1. Read config.json\n\n2. Create the network namespace ([code](https://github.com/containers/virtcontainers/blob/0.5.0/cnm.go#L108-L120))\n\n3. Call the prestart hook (from inside the netns) ([code](https://github.com/containers/virtcontainers/blob/0.5.0/api.go#L46-L49))\n\n4. Scan network interfaces inside netns and get the name of the interface created by prestart hook ([code](https://github.com/containers/virtcontainers/blob/0.5.0/cnm.go#L70-L106))\n\n5. Create bridge, TAP, and link all together with network interface previously created ([code](https://github.com/containers/virtcontainers/blob/0.5.0/network.go#L123-L205))\n\n6. Start VM inside the netns and start the container ([code](https://github.com/containers/virtcontainers/blob/0.5.0/api.go#L66-L70))\n\n__Drawbacks of CNM__\n\nThere are three drawbacks about using CNM instead of CNI:\n* The way we call into it is not very explicit: Have to re-exec dockerd binary so that it can accept parameters and execute the prestart hook related to network setup.\n* Implicit way to designate the network namespace: Instead of explicitely giving the netns to dockerd, we give it the PID of our runtime so that it can find the netns from this PID. This means we have to make sure being in the right netns while calling the hook, otherwise the veth pair will be created with the wrong netns.\n* No results are back from the hook: We have to scan the network interfaces to discover which one has been created inside the netns. This introduces more latency in the code because it forces us to scan the network in the CreatePod path, which is critical for starting the VM as quick as possible.\n\n\n## CNI\n\n![CNI Diagram](documentation/network/CNI_diagram.png)\n\n__Runtime network setup with CNI__\n\n1. Create the network namespace ([code](https://github.com/containers/virtcontainers/blob/0.5.0/cni.go#L64-L76))\n\n2. Get CNI plugin information ([code](https://github.com/containers/virtcontainers/blob/0.5.0/cni.go#L29-L32))\n\n3. Start the plugin (providing previously created netns) to add a network described into /etc/cni/net.d/ directory. At that time, the CNI plugin will create the cni0 network interface and a veth pair between the host and the created netns. It links cni0 to the veth pair before to exit. ([code](https://github.com/containers/virtcontainers/blob/0.5.0/cni.go#L34-L45))\n\n4. Create bridge, TAP, and link all together with network interface previously created ([code](https://github.com/containers/virtcontainers/blob/0.5.0/network.go#L123-L205))\n\n5. Start VM inside the netns and start the container ([code](https://github.com/containers/virtcontainers/blob/0.5.0/api.go#L66-L70))\n\n# Storage\n\nContainer workloads are shared with the virtualized environment through 9pfs.\nThe devicemapper storage driver is a special case. The driver uses dedicated block devices rather than formatted filesystems, and operates at the block level rather than the file level. This knowledge has been used to directly use the underlying block device instead of the overlay file system for the container root file system. The block device maps to the top read-write layer for the overlay. This approach gives much better I/O performance compared to using 9pfs to share the container file system.\n\nThe approach above does introduce a limitation in terms of dynamic file copy in/out of the container via `docker cp` operations.\nThe copy operation from host to container accesses the mounted file system on the host side. This is not expected to work and may lead to inconsistencies as the block device will be simultaneously written to, from two different mounts.\nThe copy operation from container to host will work, provided the user calls `sync(1)` from within the container prior to the copy to make sure any outstanding cached data is written to the block device.\n\n```\ndocker cp [OPTIONS] CONTAINER:SRC_PATH HOST:DEST_PATH\ndocker cp [OPTIONS] HOST:SRC_PATH CONTAINER:DEST_PATH\n```\n\nAbility to hotplug block devices has been added, which makes it possible to use block devices for containers started after the VM has been launched.\n\n## How to check if container uses devicemapper block device as its rootfs\n\nStart a container. Call mount(8) within the container. You should see '/' mounted on /dev/vda device.\n\n# Devices\n\nSupport has been added to pass [VFIO](https://www.kernel.org/doc/Documentation/vfio.txt) \nassigned devices on the docker command line with --device.\nSupport for passing other devices including block devices with --device has\nnot been added added yet.\n\n## How to pass a device using VFIO-passthrough\n\n1. Requirements\n\nIOMMU group represents the smallest set of devices for which the IOMMU has\nvisibility and which is isolated from other groups.  VFIO uses this information\nto enforce safe ownership of devices for userspace. \n\nYou will need Intel VT-d capable hardware. Check if IOMMU is enabled in your host\nkernel by verifying `CONFIG_VFIO_NOIOMMU` is not in the kernel config. If it is set,\nyou will need to rebuild your kernel.\n\nThe following kernel configs need to be enabled:\n```\nCONFIG_VFIO_IOMMU_TYPE1=m \nCONFIG_VFIO=m\nCONFIG_VFIO_PCI=m\n```\n\nIn addition, you need to pass `intel_iommu=on` on the kernel command line.\n\n2. Identify BDF(Bus-Device-Function) of the PCI device to be assigned.\n\n\n```\n$ lspci -D | grep -e Ethernet -e Network\n0000:01:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)\n\n$ BDF=0000:01:00.0\n```\n\n3. Find vendor and device id.\n\n```\n$ lspci -n -s $BDF\n01:00.0 0200: 8086:1528 (rev 01)\n```\n\n4. Find IOMMU group.\n\n```\n$ readlink /sys/bus/pci/devices/$BDF/iommu_group\n../../../../kernel/iommu_groups/16\n```\n\n5. Unbind the device from host driver.\n\n```\n$ echo $BDF | sudo tee /sys/bus/pci/devices/$BDF/driver/unbind\n```\n\n6. Bind the device to vfio-pci.\n\n```\n$ sudo modprobe vfio-pci\n$ echo 8086 1528 | sudo tee /sys/bus/pci/drivers/vfio-pci/new_id\n$ echo $BDF | sudo tee --append /sys/bus/pci/drivers/vfio-pci/bind\n```\n\n7. Check /dev/vfio\n\n```\n$ ls /dev/vfio\n16 vfio\n```\n\n8. Start a Clear Containers container passing the VFIO group on the docker command line.\n\n```\ndocker run -it --device=/dev/vfio/16 centos/tools bash\n```\n\n9. Running `lspci` within the container should show the device among the \nPCI devices. The driver for the device needs to be present within the\nClear Containers kernel. If the driver is missing,  you can add it to your\ncustom container kernel using the [osbuilder](https://github.com/clearcontainers/osbuilder)\ntooling.\n\n# Developers\n\nFor information on how to build, develop and test `virtcontainers`, see the\n[developer documentation](documentation/Developers.md).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcontainers%2Fvirtcontainers","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcontainers%2Fvirtcontainers","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcontainers%2Fvirtcontainers/lists"}