Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.
Awesome Lists | Featured Topics | Projects
https://github.com/chromeos/virtio-media

Last synced: 7 days ago
JSON representation
Host: GitHub
URL: https://github.com/chromeos/virtio-media
Owner: chromeos
License: bsd-3-clause
Created: 2023-12-14T03:04:55.000Z (11 months ago)
Default Branch: main
Last Pushed: 2024-10-16T07:36:21.000Z (29 days ago)
Last Synced: 2024-10-17T21:59:22.458Z (27 days ago)
Language: Rust
Size: 261 KB
Stars: 5
Watchers: 6
Forks: 1
Open Issues: 5
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
Awesome Lists containing this project

README

        # Virtio-media

This is a virtio protocol definition, companion Linux guest kernel driver, and

set of host-side devices for virtualizing media devices using virtio, following

the same model (and structures) as V4L2. It can be used to virtualize cameras,

codec devices, or any other device supported by V4L2.

Want to try it? See the [TRY_IT_OUT document](/TRY_IT_OUT.md).

V4L2 is a UAPI that allows a less privileged entity (user-space) to use video

hardware exposed by a more privileged entity (the kernel). Virtio-media is an

encapsulation of this API into virtio, turning it into a virtualization API for

all classes of video devices supported by V4L2, where the host plays the role of

the kernel and the guest the role of user-space.

The host is therefore responsible for presenting a virtual device that behaves

like an actual V4L2 device, which the guest can control.

This repository includes a simple guest Linux kernel module supporting this

protocol. On the host side, devices can be implemented in several ways:

1. By forwarding a V4L2 device from the host into a guest. This works if the

   device is already supported by V4L2 on the host.

2. By emulating a V4L2 device on the host, from the actual interface that the

   device provides.

Note that virtio-media does not require the use of a V4L2 device driver or of

Linux on the host or guest side - V4L2 is only used as a host-guest protocol,

and both sides are free to convert it from/to any model that they wish to use.

The complete definition of V4L2 structures and ioctls can be found under the

[V4L2 UAPI documentation](https://www.kernel.org/doc/html/latest/userspace-api/media/index.html),

which should be referred to alongside this document.

## Driver status

The driver (in the `driver/` directory) should be working and supporting most

V4L2 features, with the exception of the following:

- [Read/Write API](https://www.kernel.org/doc/html/v4.8/media/uapi/v4l/rw.html),

  which is obsolete and inefficient.

- Overlay interface and associated ioctls, i.e.

  - `VIDIOC_OVERLAY`

  - `VIDIOC_G/S_FBUF`

- `DMABUF` buffers (this will be supported at least for virtio objects, other

  kinds of DMABUFs may or may not be usable)

- `VIDIOC_EXPBUF` (to be implemented)

- `VIDIOC_G/S_EDID` (to be implemented if it makes sense in a virtual context)

- Media API and requests. This will probably be supported in the future behind a

  feature flag.

## Devices status

The `devices/` directory contains a Rust crate implementing helper functions to

parse the protocol, interfaces to easily implement devices, and a couple of

device implementations. It is written to be easily pluggable on any VMM. See the

rustdoc in the `devices/` directory for more information.

Implemented devices are:

- A simple video capture device generating a pattern (`simple_device.rs`),

  purely software-based and thus not requiring any kind of hardware. This is

  here for reference and testing purposes.

- A proxy device for host V4L2 devices, i.e. a device allowing to expose a host

  V4L2 device to the guest almost as-is (`v4l2_device_proxy.rs`).

* A FFmpeg-based video decoder device as a separate crate in

  `extras/ffmpeg-decoder`.

## Virtio device ID

Virtio-media uses device ID `49`.

## Virtqueues

There are two queues in use:

0 : commandq - queue for driver commands and device responses to these commands.

The device MUST return the descriptor chains it receives as soon as possible,

and must never hold to them for indefinite periods of time.

1 : eventq - queue for events sent by the device to the driver. The driver MUST

re-queue the descriptor chains returned by the device as soon as possible, and

must never hold on them for indefinite periods of time.

## Configuration area

The configuration area contains the following information:

```c

struct virtio_v4l2_config {

    /// The device_caps field of struct video_device.

    u32 device_caps;

    /// The vfl_devnode_type of the device.

    u32 device_type;

    /// The `card` field of v4l2_capability.

    u8 card[32];

}

```

## Shared memory regions

Shared memory region `0` is used to map `MMAP` buffers into the guest using the

`VIRTIO_MEDIA_CMD_MMAP` command. If the host does not provide it, then `MMAP`

buffers cannot be mapped into the guest.

## Protocol

All structures managing the virtio protocol are defined and documented in

`protocol.h`. Please refer to this file whenever a `virtio_media_cmd_*` or

`virtio_media_resp_*` structure is mentioned.

Commands are queued on the `commandq` by the driver for the device to process.

They all start by an instance of `struct virtio_media_cmd_header` and include

device-writable descriptors for the device to write the result of the command in

a `struct virtio_media_resp_header`.

The errors returned by each command are standard Linux kernel error codes. For

instance, a command that contains invalid options will return `EINVAL`.

Events are sent on the `eventq` by the device for the driver to handle. They all

start by an instance of `struct virtio_media_event_header`.

## Session management

In order to use the device, the driver needs to open a session. This act is

equivalent to opening the `/dev/videoX` device file of the V4L2 device.

Depending on the type of device, it may be possible to open several sessions

concurrently.

A session is opened by queueing a `struct virtio_media_cmd_open` along with a

descriptor to receive a `struct virtio_media_resp_open` to the commandq. An open

session can be closed with `struct virtio_media_cmd_close`.

While the session is opened, its ID can be used to perform actions on it, most

commonly V4L2 ioctls.

## Ioctls

Ioctls are the main way to interact with V4L2 devices, and therefore

virtio-media features a command to perform an ioctl on an open session.

In order to perform an ioctl, the driver queues a

`struct virtio_media_cmd_ioctl` along with a descriptor to receive a

`struct virtio_media_resp_ioctl` on the commandq. The code of the ioctl can be

extracted from the

[videodev2.h](https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/videodev.html)

header file, which defines the ioctls' codes, type of payload, and direction.

For instance, the `VIDIOC_G_FMT` ioctl is defined as follows:

```c

#define VIDIOC_G_FMT _IOWR('V',  4, struct v4l2_format)

```

This tells us that its ioctl code is `4`, that its payload is a

`struct v4l2_format`, and that its direction is `WR`, i.e. the payload is

written by both the driver and the device.

The payload layout is always a 64-bit representation of the corresponding V4L2

structure, irrespective of the host and guest architecture.

### Ioctls payload

The payload of an ioctl in the descriptor chain follows the command structure,

the reponse structure, or both depending on the direction:

- An `_IOR` ioctl is read-only for the driver, meaning the payload follows the

  response in the device-writable section of the descriptor chain.

- An `_IOW` ioctl is read-only for the device, meaning the payload follows the

  command in the driver-writable section of the descriptor chain.

- An `_IORW` ioctl is writable by both the device and driver, meaning the

  payload must follow both the command in the driver-writable section of the

  descriptor chain, and the response in the device-writable section.

For instance, the `VIDIOC_G_STD` ioctl is defined as follows:

```c

#define VIDIOC_G_STD _IOR('V', 23, v4l2_std_id)

```

Thus, its layout on the commandq will be:

```text

+-------------------------------------+

| struct virtio_media_cmd_ioctl       |

+=====================================+

| struct virtio_media_resp_ioctl      |

+-------------------------------------+

| v4l2_std_id                         |

+-------------------------------------+

```

(in these diagrams, the `====` line signals the delimitation between

device-readable and device-writable descriptors).

`VIDIOC_SUBSCRIBE_EVENT` is defined as follows:

```c

#define VIDIOC_SUBSCRIBE_EVENT _IOW('V', 90, struct v4l2_event_subscription)

```

Meaning its layout on the commandq will be:

```text

+-------------------------------------+

| struct virtio_media_cmd_ioctl       |

+-------------------------------------+

| struct v4l2_event_subscription      |

+=====================================+

| struct virtio_media_resp_ioctl      |

+-------------------------------------+

```

Finally, `VIDIOC_G_FMT` is a `WR` ioctl:

```c

#define VIDIOC_G_FMT _IOWR('V',  4, struct v4l2_format)

```

Its layout on the commandq will thus be:

```text

+-------------------------------------+

| struct virtio_media_cmd_ioctl       |

+-------------------------------------+

| struct v4l2_format                  |

+=====================================+

| struct virtio_media_resp_ioctl      |

+-------------------------------------+

| struct v4l2_format                  |

+-------------------------------------+

```

A common optimization for `WR` ioctls is to provide the payload using

descriptors that both point to the same buffer. This mimics the behavior of V4L2

ioctls where the data is only passed once and used as both input and output by

the kernel.

In case of success, the device MUST always write the payload in the

device-writable part of the descriptor chain.

In case of failure, the device is free to write the payload in the

device-writable part of the descriptor chain or not. Some errors may still

result in the payload being updated, and in this case the device is expected to

write the updated payload (for instance, `G_EXT_CTRLS` may return `EINVAL` but

update the `size` member of the requested controls if the provided size was not

enough). If the device has not written the payload after an error, the driver

MUST assume that the payload has not been modified.

### Handling of pointers in ioctl payload

A few structures used as ioctl payloads contain pointers the link to further

data needed for the ioctl. There are notably:

- The `planes` pointer of `struct v4l2_buffer`, which size is determined by the

  `length` member,

- The `controls` pointer of `struct v4l2_ext_controls`, which size is determined

  by the `count` member.

If the size of the pointed area is determined to be non-zero, then the main

payload is immediately followed by the pointed data in their order of appearance

in the structure, and the pointer value itself is ignored by the device, which

must also return the value initially passed by the driver. For instance, for a

`struct v4l2_ext_controls` which `count` is `16`:

```text

+--------------------------------------+

| struct v4l2_ext_controls             |

+--------------------------------------+

| struct v4l2_ext_control for plane 0  |

| struct v4l2_ext_control for plane 1  |

| ...                                  |

| struct v4l2_ext_control for plane 15 |

+--------------------------------------+

```

Similarly, a multiplanar `struct v4l2_buffer` with its `length` member set to 3

will be laid out as follows:

```text

+-------------------------------------+

| struct v4l2_buffer                  |

+-------------------------------------+

| struct v4l2_plane for plane 0       |

| struct v4l2_plane for plane 1       |

| struct v4l2_plane for plane 2       |

+-------------------------------------+

```

### Handling of pointers to userspace memory

A few pointers are special in that they point to userspace memory. They are:

- The `m.userptr` member of `struct v4l2_buffer` and `struct v4l2_plane`

  (technically an `unsigned long`, but designated a userspace address),

- The `ptr` member of `struct v4l2_ext_ctrl`.

These pointers can cover large areas of scattered memory, which has the

potential to require more descriptors than the virtio queue can provide. For

these particular pointers only, a list of `struct virtio_media_sg_entry` that

covers the needed amount of memory for the pointer is used instead of using

descriptors to map the pointed memory directly.

For each such pointer to read, the device reads as many SG entries as needed to

cover the length of the pointed buffer, as described by its parent structure

(`length` member of `struct v4l2_buffer` or `struct v4l2_plane` for buffer

memory, and `size` member of `struct v4l2_ext_control` for control data).

Since the device never needs to modify the list of SG entries, it is only

provided by the driver in the device-readable section of the descriptor chain,

and not repeated in the device-writable section, even for `WR` ioctls.

To illustrate the data layout, here is what the descriptor chain of a

`VIDIOC_QBUF` ioctl queueing a 3-planar `USERPTR` buffer would look like:

```text

+---------------------------------------------------+

| struct virtio_media_cmd_ioctl                     |

+---------------------------------------------------+

| struct v4l2_buffer                                |

+---------------------------------------------------+

| struct v4l2_plane for plane 0                     |

| struct v4l2_plane for plane 1                     |

| struct v4l2_plane for plane 2                     |

+---------------------------------------------------+

| array of struct virtio_media_sg_entry for plane 0 |

+---------------------------------------------------+

| array of struct virtio_media_sg_entry for plane 1 |

+---------------------------------------------------+

| array of struct virtio_media_sg_entry for plane 2 |

+===================================================+

| struct virtio_media_resp_ioctl                    |

+---------------------------------------------------+

| struct v4l2_buffer                                |

+---------------------------------------------------+

| struct v4l2_plane for plane 0                     |

| struct v4l2_plane for plane 1                     |

| struct v4l2_plane for plane 2                     |

+---------------------------------------------------+

```

Since the ioctl is `RW`, the payload is repeated in both the device-readable and

device-writable sections of the descriptor chain, but the device-writable

section does not include the SG lists to guest memory.

### Unsupported ioctls

A few ioctls are replaced by other, more suitable mechanisms. If being requested

these ioctls, the device must return the same response as it would for an

unknown ioctl, i.e. `ENOTTY`.

- `VIDIOC_QUERYCAP` is replaced by reading the configuration area.

- `VIDIOC_DQBUF` is replaced by a dedicated event.

- `VIDIOC_DQEVENT` is replaced by a dedicated event.

- `VIDIOC_G_JPEGCOMP` and `VIDIOC_S_JPEGCOMP` are deprecated and replaced by the

  controls of the JPEG class.

- `VIDIOC_LOG_STATUS` is a guest-only operation and shall not be implemented by

  the host.

## Events

Events are a way for the device to inform the driver about asynchronous events

that it should know about. In virtio-media, they are used as a replacement for

the `VIDIOC_DQBUF` and `VIDIOC_DQEVENT` ioctls and the polling mechanism, which

would be impractical to implement on top of virtio.

### Dequeued buffer events

A `struct virtio_media_event_dqbuf` event is queued on the eventq by the device

every time a buffer previously queued using the `VIDIOC_QBUF` ioctl is done

being processed and can be used by the driver again. This is like an implicit

`VIDIOC_DQBUF` ioctl.

Pointer values in the `struct v4l2_buffer` and `struct v4l2_plane` are

meaningless and must be ignored by the driver. It is recommended that the device

sets them to `NULL` in order to avoid leaking potential host addresses.

Note that in the case of a `USERPTR` buffer, the `struct v4l2_buffer` used as

event payload is not followed by the buffer memory: since that memory is the

same that the driver submitted with the `VIDIOC_QBUF`, it would be redundant to

have it here.

### Dequeued V4L2 event event

A `struct virtio_media_event_event` event is queued on the eventq by the device

every time an event the driver previously subscribed to using the

`VIDIOC_SUBSCRIBE_EVENT` ioctl has been signaled. This is like an implicit

`VIDIOC_DQEVENT` ioctl.

## Memory types

The semantics of the three V4L2 memory types (`MMAP`, `USERPTR` and `DMABUF`)

can easily be mapped to a guest/host context.

### MMAP

In virtio-media, `MMAP` buffers are provisioned by the host, just like they are

by the kernel in regular V4L2. Similarly to how userspace can map a `MMAP`

buffer into its address space using `mmap` and `munmap`, the virtio-media driver

can map host buffers into the guest space by queueing the

`struct virtio_media_cmd_mmap` and `struct virtio_media_cmd_munmap` commands to

the commandq.

### USERPTR

In virtio-media, `USERPTR` buffers and provisioned by the guest, just like they

are by userspace in regular V4L2. Instances of `struct v4l2_buffer` and

`struct v4l2_plane` of this type are followed by a series of descriptors mapping

the buffer backing memory in guest space.

For the host convenience, the backing memory must start with a new descriptor -

this allows the host to easily map the buffer memory to render into it instead

of having to do a copy.

The host must not alter the pointer values provided by the guest, i.e. the

`m.userptr` member of `struct v4l2_buffer` and `struct v4l2_plane` must be

returned to the guest with the same value as it was provided.

### DMABUF

In virtio-media, `DMABUF` buffers are provisioned by a virtio object, just like

they are by a DMABUF in regular V4L2. Virtio objects are 16-bytes UUIDs and do

not fit in the placeholders for file descriptors, so they follow their embedding

data structure as needed and the device must leave the V4L2 structure

placeholder unchanged. For instance, a 3-planar `struct v4l2_buffer` with the

`V4L2_MEMORY_DMABUF` memory type will have the following layout:

```text

+-------------------------------------+

| struct v4l2_buffer                  |

+-------------------------------------+

| struct v4l2_plane for plane 0       |

| struct v4l2_plane for plane 1       |

| struct v4l2_plane for plane 2       |

+-------------------------------------+

| 16 byte UUID for plane 0            |

+-------------------------------------+

| 16 byte UUID for plane 1            |

+-------------------------------------+

| 16 byte UUID for plane 2            |

+-------------------------------------+

```

Contrary to `USERPTR` buffers, virtio objects UUIDs need to be added in both the

device-readable and device-writable section of the descriptor chain.

Host-allocated buffers with the `V4L2_MEMORY_MMAP` memory type can also be

exported as virtio objects for use with another virtio device using the

`VIDIOC_EXPBUF` ioctl. The `fd` placefolder of `v4l2_exportbuffer` means that

space for the UUID needs to be reserved right after that structure, so the ioctl

layout will looks as follows:

```text

+-------------------------------------+

| struct virtio_media_cmd_ioctl       |

+-------------------------------------+

| struct v4l2_exportbuffer            |

+-------------------------------------+

| 16 bytes UUID for exported buffer   |

+=====================================+

| struct virtio_media_resp_ioctl      |

+-------------------------------------+

| struct v4l2_exportbuffer            |

+-------------------------------------+

| 16 bytes UUID for exported buffer   |

+-------------------------------------+

```