https://github.com/w3c/mediacapture-region
This document introduces an API for cropping a video track derived from display-capture of the current tab.
https://github.com/w3c/mediacapture-region
mediacapture screen-capture specification
Last synced: 8 months ago
JSON representation
This document introduces an API for cropping a video track derived from display-capture of the current tab.
- Host: GitHub
- URL: https://github.com/w3c/mediacapture-region
- Owner: w3c
- License: other
- Created: 2021-10-05T18:42:36.000Z (over 4 years ago)
- Default Branch: main
- Last Pushed: 2024-10-25T20:45:48.000Z (over 1 year ago)
- Last Synced: 2025-01-30T09:41:27.690Z (over 1 year ago)
- Topics: mediacapture, screen-capture, specification
- Language: HTML
- Homepage: http://w3c.github.io/mediacapture-region/
- Size: 178 KB
- Stars: 42
- Watchers: 14
- Forks: 10
- Open Issues: 21
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: CONTRIBUTING.md
- License: LICENSE.md
Awesome Lists containing this project
README
# Summary
## Problem Overview
Recall that applications may currently obtain a capture of the tab in which they run using [getDisplayMedia](https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getDisplayMedia), either with or without [preferCurrentTab](http://go/get-current-browsing-context-media). Moreover, soon another API will allow similar functionality - [getViewportMedia](https://github.com/w3c/mediacapture-screen-share/issues/155). In either case, the application may then also wish to crop the resulting video track so as to remove some content from it (typically before sharing it remotely). We introduce a performant and robust API for cropping a self-capture video track.
## Core Challenges
Layout can change asynchronously when the user scrolls, zooms or resizes the window. The application [cannot](https://docs.google.com/document/d/1dULARMnMZggfWqa_Ti_GrINRNYXGIli3XK9brzAKEV8/edit#heading=h.vtnc1viphmzf) robustly react to such changes without risking mis-cropping the video track on occasion. The browser therefore needs to step in and help.
## Sample Use Case
Consider a combo-application consisting of two major parts - a video-conferencing application and a productivity-suite application co-existing in a single tab. Assume the video-conferencing uses existing/upcoming APIs such as [getDisplayMedia](https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getDisplayMedia) and/or [getViewportMedia](https://github.com/w3c/mediacapture-screen-share/issues/155) and captures the entire tab. Now it needs to crop away everything other than a particular section of the productivity-suite. It needs to crop away its own video-conferencing content, any speaker notes and other private and/or irrelevant content in the productivity-suite, before transmitting the resulting cropped video remotely.
Moreover, consider that it is likely that the two collaborating applications are cross-origin from each other. They can post messages, but all communication is asynchronous, and it's easier and more performant if information is transmitted sparingly between them. That precludes solutions involving posting of entire frames, as well as solutions which are too slow to react to changes in layout (e.g. scrolling, zooming and window-size changes).
## Goals and Non-Goals
### Goals
* The new API we introduce allows an application which is already in possession of a self-capture video track, to crop that track to the contours of its desired element.
* The API allows this to be done performantly, consistently and robustly.
### Non-Goals
* This API does not introduce new ways to obtain a self-capture video track.
* This API does not introduce mechanisms by which a captured document may control what the capturing document can see.
# Solution
## Solution Overview
A two-pronged solution is presented:
* **CropTarget production:** A mechanism for tagging an HTMLElement as a potential target for the cropping mechanism.
* **Cropping mechanism**: A mechanism for instructing the user agent to start cropping a video track to the contours of a previously tagged HTMLElement, or to stop such cropping and revert a track to its uncropped state.
## CropTarget production
We introduce `navigator.mediaDevices.produceCropTarget()`.
```webidl
partial interface MediaDevices {
Promise
produceCropTarget(HTMLElement target);
};
```
Given an HTMLElement, `produceCropTarget()` produces an opaque class that uniquely identifies that element to our second mechanism - the cropping mechanism.
(The `Promise` returned by `produceCropTarget()` is only resolved when the CropTarget is ready for use, allowing the browser time to set up prerequisites and propagate state cross-process.)
## Cropping mechanism
We introduce a `cropTo()` method, which we expose on all video tracks derived of tab-capture.
```webidl
[Exposed = Window]
interface BrowserCaptureMediaStreamTrack : FocusableMediaStreamTrack {
Promise cropTo(CropTarget cropTarget);
};
```
Given a CropTarget, `cropTo()` starts cropping the video track to the contours of the referenced HTMLElement.
Given `undefined`, `cropTo()` reverts a video track to its uncropped state.
"On-the-fly" changing of crop-targets is possible.
# Code Samples
```js
/////////////////////////////////
// Code in the capture-target: //
/////////////////////////////////
const mainContentArea = navigator.getElementById('mainContentArea');
const cropTarget = await navigator.mediaDevices.produceCropTarget(mainContentArea);
sendCropTarget(cropTarget);
function sendCropTarget(cropTarget) {
// Can send the crop-target to another document in this browsing context
// using postMessage() or using any other means.
// Possibly there is no other document, and this is just consumed locally.
}
/////////////////////////////////////
// Code in the capturing-document: //
/////////////////////////////////////
async function startCroppedCapture(cropTarget) {
const stream = await navigator.mediaDevices.getDisplayMedia();
const [track] = stream.getVideoTracks();
if (!!track.cropTo) {
handleError(stream);
return;
}
await track.cropTo(cropTarget);
transmitVideoRemotely(track);
}
```
# Changelog
See [dedicated file](https://github.com/eladalon1983/region-capture/blob/main/CHANGELOG.md).