{"id":18652949,"url":"https://github.com/chromeos/virtio-media","last_synced_at":"2025-10-19T02:25:51.549Z","repository":{"id":221275661,"uuid":"731414168","full_name":"chromeos/virtio-media","owner":"chromeos","description":null,"archived":false,"fork":false,"pushed_at":"2024-10-16T07:36:21.000Z","size":267,"stargazers_count":5,"open_issues_count":5,"forks_count":1,"subscribers_count":6,"default_branch":"main","last_synced_at":"2024-10-17T21:59:22.458Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/chromeos.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-12-14T03:04:55.000Z","updated_at":"2024-10-16T07:36:25.000Z","dependencies_parsed_at":"2024-07-12T08:43:28.349Z","dependency_job_id":"7ab476bb-214a-40d7-b1a3-7a0a2e52e6bf","html_url":"https://github.com/chromeos/virtio-media","commit_stats":null,"previous_names":["chromeos/virtio-media"],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chromeos%2Fvirtio-media","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chromeos%2Fvirtio-media/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chromeos%2Fvirtio-media/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chromeos%2Fvirtio-media/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/chromeos","download_url":"https://codeload.github.com/chromeos/virtio-media/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223472780,"owners_count":17150745,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-07T07:09:31.861Z","updated_at":"2025-10-19T02:25:51.468Z","avatar_url":"https://github.com/chromeos.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Virtio-media\n\nThis is a virtio protocol definition, companion Linux guest kernel driver, and\nset of host-side devices for virtualizing media devices using virtio, following\nthe same model (and structures) as V4L2. It can be used to virtualize cameras,\ncodec devices, or any other device supported by V4L2.\n\nWant to try it? See the [TRY_IT_OUT document](/TRY_IT_OUT.md).\n\nV4L2 is a UAPI that allows a less privileged entity (user-space) to use video\nhardware exposed by a more privileged entity (the kernel). Virtio-media is an\nencapsulation of this API into virtio, turning it into a virtualization API for\nall classes of video devices supported by V4L2, where the host plays the role of\nthe kernel and the guest the role of user-space.\n\nThe host is therefore responsible for presenting a virtual device that behaves\nlike an actual V4L2 device, which the guest can control.\n\nThis repository includes a simple guest Linux kernel module supporting this\nprotocol. On the host side, devices can be implemented in several ways:\n\n1. By forwarding a V4L2 device from the host into a guest. This works if the\n   device is already supported by V4L2 on the host.\n2. By emulating a V4L2 device on the host, from the actual interface that the\n   device provides.\n\nNote that virtio-media does not require the use of a V4L2 device driver or of\nLinux on the host or guest side - V4L2 is only used as a host-guest protocol,\nand both sides are free to convert it from/to any model that they wish to use.\n\nThe complete definition of V4L2 structures and ioctls can be found under the\n[V4L2 UAPI documentation](https://www.kernel.org/doc/html/latest/userspace-api/media/index.html),\nwhich should be referred to alongside this document.\n\n## Driver status\n\nThe driver (in the `driver/` directory) should be working and supporting most\nV4L2 features, with the exception of the following:\n\n- [Read/Write API](https://www.kernel.org/doc/html/v4.8/media/uapi/v4l/rw.html),\n  which is obsolete and inefficient.\n- Overlay interface and associated ioctls, i.e.\n  - `VIDIOC_OVERLAY`\n  - `VIDIOC_G/S_FBUF`\n- `DMABUF` buffers (this will be supported at least for virtio objects, other\n  kinds of DMABUFs may or may not be usable)\n- `VIDIOC_EXPBUF` (to be implemented)\n- `VIDIOC_G/S_EDID` (to be implemented if it makes sense in a virtual context)\n- Media API and requests. This will probably be supported in the future behind a\n  feature flag.\n\n## Devices status\n\nThe `devices/` directory contains a Rust crate implementing helper functions to\nparse the protocol, interfaces to easily implement devices, and a couple of\ndevice implementations. It is written to be easily pluggable on any VMM. See the\nrustdoc in the `devices/` directory for more information.\n\nImplemented devices are:\n\n- A simple video capture device generating a pattern (`simple_device.rs`),\n  purely software-based and thus not requiring any kind of hardware. This is\n  here for reference and testing purposes.\n- A proxy device for host V4L2 devices, i.e. a device allowing to expose a host\n  V4L2 device to the guest almost as-is (`v4l2_device_proxy.rs`).\n- A FFmpeg-based video decoder device as a separate crate in\n  `extras/ffmpeg-decoder`.\n\n## Virtio device ID\n\nVirtio-media uses device ID `48`.\n\n## Virtqueues\n\nThere are two queues in use:\n\n0 : commandq - queue for driver commands and device responses to these commands.\nThe device MUST return the descriptor chains it receives as soon as possible,\nand must never hold to them for indefinite periods of time.\n\n1 : eventq - queue for events sent by the device to the driver. The driver MUST\nre-queue the descriptor chains returned by the device as soon as possible, and\nmust never hold on them for indefinite periods of time.\n\n## Configuration area\n\nThe configuration area contains the following information:\n\n```c\nstruct virtio_v4l2_config {\n    /// The device_caps field of struct video_device.\n    u32 device_caps;\n    /// The vfl_devnode_type of the device.\n    u32 device_type;\n    /// The `card` field of v4l2_capability.\n    u8 card[32];\n}\n```\n\n## Shared memory regions\n\nShared memory region `0` is used to map `MMAP` buffers into the guest using the\n`VIRTIO_MEDIA_CMD_MMAP` command. If the host does not provide it, then `MMAP`\nbuffers cannot be mapped into the guest.\n\n## Protocol\n\nAll structures managing the virtio protocol are defined and documented in\n`protocol.h`. Please refer to this file whenever a `virtio_media_cmd_*` or\n`virtio_media_resp_*` structure is mentioned.\n\nCommands are queued on the `commandq` by the driver for the device to process.\nThey all start by an instance of `struct virtio_media_cmd_header` and include\ndevice-writable descriptors for the device to write the result of the command in\na `struct virtio_media_resp_header`.\n\nThe errors returned by each command are standard Linux kernel error codes. For\ninstance, a command that contains invalid options will return `EINVAL`.\n\nEvents are sent on the `eventq` by the device for the driver to handle. They all\nstart by an instance of `struct virtio_media_event_header`.\n\n## Session management\n\nIn order to use the device, the driver needs to open a session. This act is\nequivalent to opening the `/dev/videoX` device file of the V4L2 device.\nDepending on the type of device, it may be possible to open several sessions\nconcurrently.\n\nA session is opened by queueing a `struct virtio_media_cmd_open` along with a\ndescriptor to receive a `struct virtio_media_resp_open` to the commandq. An open\nsession can be closed with `struct virtio_media_cmd_close`.\n\nWhile the session is opened, its ID can be used to perform actions on it, most\ncommonly V4L2 ioctls.\n\n## Ioctls\n\nIoctls are the main way to interact with V4L2 devices, and therefore\nvirtio-media features a command to perform an ioctl on an open session.\n\nIn order to perform an ioctl, the driver queues a\n`struct virtio_media_cmd_ioctl` along with a descriptor to receive a\n`struct virtio_media_resp_ioctl` on the commandq. The code of the ioctl can be\nextracted from the\n[videodev2.h](https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/videodev.html)\nheader file, which defines the ioctls' codes, type of payload, and direction.\nFor instance, the `VIDIOC_G_FMT` ioctl is defined as follows:\n\n```c\n#define VIDIOC_G_FMT _IOWR('V',  4, struct v4l2_format)\n```\n\nThis tells us that its ioctl code is `4`, that its payload is a\n`struct v4l2_format`, and that its direction is `WR`, i.e. the payload is\nwritten by both the driver and the device.\n\nThe payload layout is always a 64-bit representation of the corresponding V4L2\nstructure, irrespective of the host and guest architecture.\n\n### Ioctls payload\n\nThe payload of an ioctl in the descriptor chain follows the command structure,\nthe reponse structure, or both depending on the direction:\n\n- An `_IOR` ioctl is read-only for the driver, meaning the payload follows the\n  response in the device-writable section of the descriptor chain.\n- An `_IOW` ioctl is read-only for the device, meaning the payload follows the\n  command in the driver-writable section of the descriptor chain.\n- An `_IORW` ioctl is writable by both the device and driver, meaning the\n  payload must follow both the command in the driver-writable section of the\n  descriptor chain, and the response in the device-writable section.\n\nFor instance, the `VIDIOC_G_STD` ioctl is defined as follows:\n\n```c\n#define VIDIOC_G_STD _IOR('V', 23, v4l2_std_id)\n```\n\nThus, its layout on the commandq will be:\n\n```text\n+-------------------------------------+\n| struct virtio_media_cmd_ioctl       |\n+=====================================+\n| struct virtio_media_resp_ioctl      |\n+-------------------------------------+\n| v4l2_std_id                         |\n+-------------------------------------+\n```\n\n(in these diagrams, the `====` line signals the delimitation between\ndevice-readable and device-writable descriptors).\n\n`VIDIOC_SUBSCRIBE_EVENT` is defined as follows:\n\n```c\n#define VIDIOC_SUBSCRIBE_EVENT _IOW('V', 90, struct v4l2_event_subscription)\n```\n\nMeaning its layout on the commandq will be:\n\n```text\n+-------------------------------------+\n| struct virtio_media_cmd_ioctl       |\n+-------------------------------------+\n| struct v4l2_event_subscription      |\n+=====================================+\n| struct virtio_media_resp_ioctl      |\n+-------------------------------------+\n```\n\nFinally, `VIDIOC_G_FMT` is a `WR` ioctl:\n\n```c\n#define VIDIOC_G_FMT _IOWR('V',  4, struct v4l2_format)\n```\n\nIts layout on the commandq will thus be:\n\n```text\n+-------------------------------------+\n| struct virtio_media_cmd_ioctl       |\n+-------------------------------------+\n| struct v4l2_format                  |\n+=====================================+\n| struct virtio_media_resp_ioctl      |\n+-------------------------------------+\n| struct v4l2_format                  |\n+-------------------------------------+\n```\n\nA common optimization for `WR` ioctls is to provide the payload using\ndescriptors that both point to the same buffer. This mimics the behavior of V4L2\nioctls where the data is only passed once and used as both input and output by\nthe kernel.\n\nIn case of success, the device MUST always write the payload in the\ndevice-writable part of the descriptor chain.\n\nIn case of failure, the device is free to write the payload in the\ndevice-writable part of the descriptor chain or not. Some errors may still\nresult in the payload being updated, and in this case the device is expected to\nwrite the updated payload (for instance, `G_EXT_CTRLS` may return `EINVAL` but\nupdate the `size` member of the requested controls if the provided size was not\nenough). If the device has not written the payload after an error, the driver\nMUST assume that the payload has not been modified.\n\n### Handling of pointers in ioctl payload\n\nA few structures used as ioctl payloads contain pointers the link to further\ndata needed for the ioctl. There are notably:\n\n- The `planes` pointer of `struct v4l2_buffer`, which size is determined by the\n  `length` member,\n- The `controls` pointer of `struct v4l2_ext_controls`, which size is determined\n  by the `count` member.\n\nIf the size of the pointed area is determined to be non-zero, then the main\npayload is immediately followed by the pointed data in their order of appearance\nin the structure, and the pointer value itself is ignored by the device, which\nmust also return the value initially passed by the driver. For instance, for a\n`struct v4l2_ext_controls` which `count` is `16`:\n\n```text\n+--------------------------------------+\n| struct v4l2_ext_controls             |\n+--------------------------------------+\n| struct v4l2_ext_control for plane 0  |\n| struct v4l2_ext_control for plane 1  |\n| ...                                  |\n| struct v4l2_ext_control for plane 15 |\n+--------------------------------------+\n```\n\nSimilarly, a multiplanar `struct v4l2_buffer` with its `length` member set to 3\nwill be laid out as follows:\n\n```text\n+-------------------------------------+\n| struct v4l2_buffer                  |\n+-------------------------------------+\n| struct v4l2_plane for plane 0       |\n| struct v4l2_plane for plane 1       |\n| struct v4l2_plane for plane 2       |\n+-------------------------------------+\n```\n\n### Handling of pointers to userspace memory\n\nA few pointers are special in that they point to userspace memory. They are:\n\n- The `m.userptr` member of `struct v4l2_buffer` and `struct v4l2_plane`\n  (technically an `unsigned long`, but designated a userspace address),\n- The `ptr` member of `struct v4l2_ext_ctrl`.\n\nThese pointers can cover large areas of scattered memory, which has the\npotential to require more descriptors than the virtio queue can provide. For\nthese particular pointers only, a list of `struct virtio_media_sg_entry` that\ncovers the needed amount of memory for the pointer is used instead of using\ndescriptors to map the pointed memory directly.\n\nFor each such pointer to read, the device reads as many SG entries as needed to\ncover the length of the pointed buffer, as described by its parent structure\n(`length` member of `struct v4l2_buffer` or `struct v4l2_plane` for buffer\nmemory, and `size` member of `struct v4l2_ext_control` for control data).\n\nSince the device never needs to modify the list of SG entries, it is only\nprovided by the driver in the device-readable section of the descriptor chain,\nand not repeated in the device-writable section, even for `WR` ioctls.\n\nTo illustrate the data layout, here is what the descriptor chain of a\n`VIDIOC_QBUF` ioctl queueing a 3-planar `USERPTR` buffer would look like:\n\n```text\n+---------------------------------------------------+\n| struct virtio_media_cmd_ioctl                     |\n+---------------------------------------------------+\n| struct v4l2_buffer                                |\n+---------------------------------------------------+\n| struct v4l2_plane for plane 0                     |\n| struct v4l2_plane for plane 1                     |\n| struct v4l2_plane for plane 2                     |\n+---------------------------------------------------+\n| array of struct virtio_media_sg_entry for plane 0 |\n+---------------------------------------------------+\n| array of struct virtio_media_sg_entry for plane 1 |\n+---------------------------------------------------+\n| array of struct virtio_media_sg_entry for plane 2 |\n+===================================================+\n| struct virtio_media_resp_ioctl                    |\n+---------------------------------------------------+\n| struct v4l2_buffer                                |\n+---------------------------------------------------+\n| struct v4l2_plane for plane 0                     |\n| struct v4l2_plane for plane 1                     |\n| struct v4l2_plane for plane 2                     |\n+---------------------------------------------------+\n```\n\nSince the ioctl is `RW`, the payload is repeated in both the device-readable and\ndevice-writable sections of the descriptor chain, but the device-writable\nsection does not include the SG lists to guest memory.\n\n### Unsupported ioctls\n\nA few ioctls are replaced by other, more suitable mechanisms. If being requested\nthese ioctls, the device must return the same response as it would for an\nunknown ioctl, i.e. `ENOTTY`.\n\n- `VIDIOC_QUERYCAP` is replaced by reading the configuration area.\n- `VIDIOC_DQBUF` is replaced by a dedicated event.\n- `VIDIOC_DQEVENT` is replaced by a dedicated event.\n- `VIDIOC_G_JPEGCOMP` and `VIDIOC_S_JPEGCOMP` are deprecated and replaced by the\n  controls of the JPEG class.\n- `VIDIOC_LOG_STATUS` is a guest-only operation and shall not be implemented by\n  the host.\n\n## Events\n\nEvents are a way for the device to inform the driver about asynchronous events\nthat it should know about. In virtio-media, they are used as a replacement for\nthe `VIDIOC_DQBUF` and `VIDIOC_DQEVENT` ioctls and the polling mechanism, which\nwould be impractical to implement on top of virtio.\n\n### Dequeued buffer events\n\nA `struct virtio_media_event_dqbuf` event is queued on the eventq by the device\nevery time a buffer previously queued using the `VIDIOC_QBUF` ioctl is done\nbeing processed and can be used by the driver again. This is like an implicit\n`VIDIOC_DQBUF` ioctl.\n\nPointer values in the `struct v4l2_buffer` and `struct v4l2_plane` are\nmeaningless and must be ignored by the driver. It is recommended that the device\nsets them to `NULL` in order to avoid leaking potential host addresses.\n\nNote that in the case of a `USERPTR` buffer, the `struct v4l2_buffer` used as\nevent payload is not followed by the buffer memory: since that memory is the\nsame that the driver submitted with the `VIDIOC_QBUF`, it would be redundant to\nhave it here.\n\n### Dequeued V4L2 event event\n\nA `struct virtio_media_event_event` event is queued on the eventq by the device\nevery time an event the driver previously subscribed to using the\n`VIDIOC_SUBSCRIBE_EVENT` ioctl has been signaled. This is like an implicit\n`VIDIOC_DQEVENT` ioctl.\n\n## Memory types\n\nThe semantics of the three V4L2 memory types (`MMAP`, `USERPTR` and `DMABUF`)\ncan easily be mapped to a guest/host context.\n\n### MMAP\n\nIn virtio-media, `MMAP` buffers are provisioned by the host, just like they are\nby the kernel in regular V4L2. Similarly to how userspace can map a `MMAP`\nbuffer into its address space using `mmap` and `munmap`, the virtio-media driver\ncan map host buffers into the guest space by queueing the\n`struct virtio_media_cmd_mmap` and `struct virtio_media_cmd_munmap` commands to\nthe commandq.\n\n### USERPTR\n\nIn virtio-media, `USERPTR` buffers and provisioned by the guest, just like they\nare by userspace in regular V4L2. Instances of `struct v4l2_buffer` and\n`struct v4l2_plane` of this type are followed by a series of descriptors mapping\nthe buffer backing memory in guest space.\n\nFor the host convenience, the backing memory must start with a new descriptor -\nthis allows the host to easily map the buffer memory to render into it instead\nof having to do a copy.\n\nThe host must not alter the pointer values provided by the guest, i.e. the\n`m.userptr` member of `struct v4l2_buffer` and `struct v4l2_plane` must be\nreturned to the guest with the same value as it was provided.\n\n### DMABUF\n\nIn virtio-media, `DMABUF` buffers are provisioned by a virtio object, just like\nthey are by a DMABUF in regular V4L2. Virtio objects are 16-bytes UUIDs and do\nnot fit in the placeholders for file descriptors, so they follow their embedding\ndata structure as needed and the device must leave the V4L2 structure\nplaceholder unchanged. For instance, a 3-planar `struct v4l2_buffer` with the\n`V4L2_MEMORY_DMABUF` memory type will have the following layout:\n\n```text\n+-------------------------------------+\n| struct v4l2_buffer                  |\n+-------------------------------------+\n| struct v4l2_plane for plane 0       |\n| struct v4l2_plane for plane 1       |\n| struct v4l2_plane for plane 2       |\n+-------------------------------------+\n| 16 byte UUID for plane 0            |\n+-------------------------------------+\n| 16 byte UUID for plane 1            |\n+-------------------------------------+\n| 16 byte UUID for plane 2            |\n+-------------------------------------+\n```\n\nContrary to `USERPTR` buffers, virtio objects UUIDs need to be added in both the\ndevice-readable and device-writable section of the descriptor chain.\n\nHost-allocated buffers with the `V4L2_MEMORY_MMAP` memory type can also be\nexported as virtio objects for use with another virtio device using the\n`VIDIOC_EXPBUF` ioctl. The `fd` placefolder of `v4l2_exportbuffer` means that\nspace for the UUID needs to be reserved right after that structure, so the ioctl\nlayout will looks as follows:\n\n```text\n+-------------------------------------+\n| struct virtio_media_cmd_ioctl       |\n+-------------------------------------+\n| struct v4l2_exportbuffer            |\n+-------------------------------------+\n| 16 bytes UUID for exported buffer   |\n+=====================================+\n| struct virtio_media_resp_ioctl      |\n+-------------------------------------+\n| struct v4l2_exportbuffer            |\n+-------------------------------------+\n| 16 bytes UUID for exported buffer   |\n+-------------------------------------+\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchromeos%2Fvirtio-media","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchromeos%2Fvirtio-media","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchromeos%2Fvirtio-media/lists"}