{"id":15288037,"url":"https://github.com/ndrean/rtc-hls","last_synced_at":"2025-07-25T05:30:35.916Z","repository":{"id":245863687,"uuid":"819408355","full_name":"ndrean/RTC-HLS","owner":"ndrean","description":"Broadcast your feed with WebRTC, ExRTC, HLS","archived":false,"fork":false,"pushed_at":"2024-11-11T12:49:54.000Z","size":33911,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-11-11T13:38:26.256Z","etag":null,"topics":["elixir","evision","face-api","hls-live-streaming","livebook","webrtc-video"],"latest_commit_sha":null,"homepage":"","language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ndrean.png","metadata":{"files":{"readme":null,"changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-06-24T12:51:04.000Z","updated_at":"2024-11-11T12:49:58.000Z","dependencies_parsed_at":"2024-07-16T00:54:47.891Z","dependency_job_id":null,"html_url":"https://github.com/ndrean/RTC-HLS","commit_stats":null,"previous_names":["ndrean/rtc-hls"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ndrean%2FRTC-HLS","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ndrean%2FRTC-HLS/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ndrean%2FRTC-HLS/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ndrean%2FRTC-HLS/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ndrean","download_url":"https://codeload.github.com/ndrean/RTC-HLS/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":227527528,"owners_count":17783752,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["elixir","evision","face-api","hls-live-streaming","livebook","webrtc-video"],"created_at":"2024-09-30T15:43:54.168Z","updated_at":"2024-12-01T09:26:33.496Z","avatar_url":"https://github.com/ndrean.png","language":"Elixir","funding_links":[],"categories":[],"sub_categories":[],"readme":"# WebRTC, HSL, DASH - Experiments with Elixir\n\n\u003ch1 align=\"center\"\u003e\u003cb\u003eDRAFT\u003c/b\u003e\u003c/h1\u003e\n\u003cbr/\u003e\n\nThis is a \"basic\" `LiveView` app where we experiment processing videos streams with different protocoles.\n\nWe explore the `WebRTC` API, the `ExRTC` (`Elixir` SFU implementation of WebRTC), HTTP Live Streaming with `HLS` or `DASH` and `MSE` (Media Source Extensions). We want to demonstrate how one can make and broadcast live transformations on images.\n\nOur transformation will be the \"Hello World\" of computer vision, **face contouring**.\n\nWe heavily use `FFmpeg` and the Elixir libraries `ExWebERTC`, `Evision` (port of `OpenCV`), `ExCmd` as the `FFmpeg` runner (on the OS level), and of course `Phoenix LiveView` and `Elixir.Channel`.\n\n\u003e **M**edia **S**ource **E**xtensions is a media player inside the browser. You create a [MediaSource object](https://developer.mozilla.org/en-US/docs/Web/API/MediaSource) and assign it to your video element, like `video.src = URL.createObjectURL(mediaSource)`. Your javascript code can fetch media segments from somewhere and supply it to the SourceBuffer attached to MediaSource.\n\u003e `WebRTC` is not just a player, it is also a capture, encoding and sending mechanism. You create another object, a [MediaStream](https://developer.mozilla.org/en-US/docs/Web/API/MediaStream) and assign it to your video element, like `video.srcObject = URL.createObjectURL(mediaStream)`. Notice that in this case the mediaStream object is not created directly by yourself, but supplied to you by WebRTC APIs such as `getUserMedia`.\n\nTo summarize, in both cases you use a video element to play, but with MSE you have to supply media segments by yourself, while with WebRTC you use the WebRTC API to supply media. WebRTC can do more: capture a user's webcam, encode it and send to another browser to play, enabling p2p video chat, for example.\n\nBrowser to browser video chat testing without WebRTC. How to Use the web socket server to send and receive data in real time.\n\n    Obtain media stream using getUserMedia to access webcams on the local computer.\n    MediaRecorder encodes media stream and converts it into blob data.(media segment)\n    Send blob data to the server via a web socket.(Blob data will be converted into arrayBuffer)\n    The server returns the data back to the client.\n    By using appendBuffer, appends the media segment to the SourceBuffer in MediaSource.\n\n\u003e **:hash: What are we building?**\n\nWe will use the camera and microphone of the device to exchange media streams. This LiveView based app has \"lobby\" home page that displays tabs that allow you to use differents protocoles to broadcast our feed.\n\n- We demonstrate the usage of WebRTC with an Elixir SFU backend: we establish a WebRTC connection between the browser and a ExWebRTC server. Another brower creates his browser/SFU server WebRTC link. Then we forward the flux between the servers. Note that we run a ML recognition process run in the browser using `mediaPipe`. This protocole gives very **low latency**.\n\n\u003e :exclamation: It works with Chrome and Safari but not work with Firefox because it uses `requestVideoFrameCallback`.\n\nYou can check this Livebook.\n\n[![ExWebRTC in Livebook](https://livebook.dev/badge/v1/blue.svg)](https://livebook.dev/run?url=https%3A%2F%2Fgithub.com%2Fdwyl%2FWebRTC-SFU-demo%2Fblob%2Fmain%2Flib%2Fecho_mediapipe.livemd)\n\n- We explore HTTP Live Streaming (HLS). We capture the stream from the browser, send it to the browser (WebSocket binary or HTTP POST). In the server, we transform the data with `Evision` running the Haar Casacade model.You will see that this protocole has **high latency** (15s).\n\n[![Run in Livebook](https://livebook.dev/badge/v1/blue.svg)](https://livebook.dev/run?url=https%3A%2F%2Fgithub.com%2Fdwyl%2FHLS-demo%2Fblob%2Fmain%2Flib%2Fhls-demo.livemd)\n\n- This Livebook demonstrates MPEG-DASH. It is very similar to HLS and has the same high latency.\n\n[![Run in Livebook](https://livebook.dev/badge/v1/blue.svg)](https://livebook.dev/run?url=https%3A%2F%2Fgithub.com%2Fdwyl%2FDASH-demo%2Fblob%2Fmain%2Flib%2Fdash-demo.livemd)\n\n- We explore the WebRTC, using the browser. It establishes a peer-to-peer connection. This means that once the connection established (this process needs a server to communicate between peers), the browsers communicaque directly via an UPD connection. You can organize for example a live session. We limited it to 3 participants.\n\n**:hash: Quick review of possible technologies, ([cf Wiki page](https://developer.mozilla.org/en-US/docs/Web/Media/Audio_and_video_delivery/Live_streaming_web_audio_and_video)):**\n\n- UPD based techs, for low latency and low quality: [RTP](https://en.wikipedia.org/wiki/Real-time_Transport_Protocol#:~:text=RTP%20typically%20runs%20over%20User,aids%20synchronization%20of%20multiple%20streams.) with [WebRTC](https://en.wikipedia.org/wiki/WebRTC),\n- HTTP based techs: [MPEG-DASH](https://developer.mozilla.org/en-US/docs/Web/Media/Audio_and_video_delivery/Setting_up_adaptive_streaming_media_sources#mpeg-dash_encoding) (playback in the browser with [Dash.js](https://github.com/Dash-Industry-Forum/dash.js/)), and [HLS](https://developer.mozilla.org/en-US/docs/Web/Media/Audio_and_video_delivery/Setting_up_adaptive_streaming_media_sources#hls_encoding) (playback in the browser with [hsl.js](https://github.com/video-dev/hls.js)).\n\n## Review of WebRTC\n\nThis technology is about making web apps capable of exchanging media content - audio and video - between browsers _without requiring an intermediary_. It is intended for peer-to-peer delivery within a limited number of browsers, like video conferencing, rather than large-scale broadcasting.\n\nIt is based on RTP. It uses codecs to compress data. The [WebRTC API](https://developer.mozilla.org/en-US/docs/Web/API/WebRTC_API) is natively implemented in (almsot every) web navigator.\n\nWe will also use an Elixir implementation - [Elixir WebRTC](https://github.com/elixir-webrtc/ex_webrtc) - of the WebRTC to connect clients (named `ExWebRTC` here). It is a WebRTC gateway on the server.\n\n**:hash: What is signaling?**\n\nThe WebRTC standards focus primarily on the media plane. Signaling functions – session setup and management – are left to the application.\n\nTo use WebRTC, you need to discover the IP address of the connected peers.\n\nThe _signaling process_ is the discovery of peers location and media format. You may need a third - mutually agreed-upon - server (STUN, TURN) for this.\nThe WebRTC process needs to discover the IP address of the clients to determine a way to exchange data between peers.\n\nThe _signaling server_ is the transport mechanism of the data exchange.\n\n![signaling](https://github.com/ndrean/RTC/blob/main/priv/static/images/signaling.png)\n\nFor the signaling process, we can:\n\n- use the LiveView \"/live\" socket. Check [this paragraph](#signaling-process-with-the-liveview),\n- use a custom WebSocket. We used this with `Elixir.Channel`, a process build on top of the custom WebSocket connection,\n- use HTTP requests (the WHEP and WHIP protocoles). This is demonstrated in the [Elixir-WebRTC/Broadcaster repo](https://github.com/elixir-webrtc/apps/tree/master/broadcaster). It provides a simplified signaling process because of the HTTP-friendly approach: you don't need to establish a WebSocket connection. You use WHIP (Ingress) for clients to _send_ media streams to the server, and WHEP (Egress) for clients to _receive_ media streams from the server.\n\n**:hash: What topopolgy?**\n\nThe native WebRTC uses a _full mesh_ topology: each user is connected with n-1 users, like the distributed Erlang.\nThe more connected users, the more bandwidth a single user will use as he has to send/receive data to/from n-1 users. Furthermore, each received stream has to be decoded, and each sent stream has to be encoded, very CPU demanding. Other topologies than mesh are needed, such as SFU and MCU.\n\nThe server based library `ex_webrtc` connects a client to a dedicated GenServer. To connect different peers, you exchange data between these GenServers, who will retransmit to their respective client.\n\n**🧐 Why would you implement a server?**\n\nWhen you need to process the streams, such as:\n\n- saving the media into a file,\n- using media processing algorithms or machine learning processing where some models need several Gb of RAM\n- things that might be hard to do this in/from the browser!\n\n\u003chr/\u003e\n\n## The TOC\n\n- [WebRTC, HSL, DASH - Experiments with Elixir](#webrtc-hsl-dash---experiments-with-elixir)\n  - [Review of WebRTC](#review-of-webrtc)\n  - [The TOC](#the-toc)\n  - [Broadcast face contouring with mediaPipe](#broadcast-face-contouring-with-mediapipe)\n    - [Push frames to the server](#push-frames-to-the-server)\n      - [Push using WebSocket](#push-using-websocket)\n      - [Push using HTTP request](#push-using-http-request)\n      - [Overview](#overview)\n      - [The \"frame\" hook](#the-frame-hook)\n    - [Push video chunks](#push-video-chunks)\n  - [Signaling process with the LiveView](#signaling-process-with-the-liveview)\n  - [WebRTC](#webrtc)\n    - [WebRTC signaling flow](#webrtc-signaling-flow)\n      - [Connexion and SDP exchange](#connexion-and-sdp-exchange)\n      - [Media streams](#media-streams)\n      - [The ICE exchange](#the-ice-exchange)\n    - [Flow for 3+ peers](#flow-for-3-peers)\n      - [WebRTC 3+ client code](#webrtc-3-client-code)\n      - [The Elixir signaling channel](#the-elixir-signaling-channel)\n      - [Phoenix Channel client side](#phoenix-channel-client-side)\n      - [Details of WebRTC objects](#details-of-webrtc-objects)\n  - [ExWebRTC](#exwebrtc)\n    - [Using channels](#using-channels)\n      - [The server WebRTC process](#the-server-webrtc-process)\n        - [Signaling module](#signaling-module)\n    - [RTC module](#rtc-module)\n    - [Example of ExWebRTC with an Echo server](#example-of-exwebrtc-with-an-echo-server)\n    - [Example of ExWebRTC with two connected clients](#example-of-exwebrtc-with-two-connected-clients)\n    - [Statistics and getting transfer rates with getStats](#statistics-and-getting-transfer-rates-with-getstats)\n    - [Details of the process supervision](#details-of-the-process-supervision)\n  - [HLS with an Elixir server](#hls-with-an-elixir-server)\n    - [What is HLS](#what-is-hls)\n    - [The process](#the-process)\n    - [FFmpeg commands](#ffmpeg-commands)\n    - [FileWatcher on the manifest file](#filewatcher-on-the-manifest-file)\n    - [Proxy or CDN](#proxy-or-cdn)\n  - [MPEG-DASH with an Elixir server](#mpeg-dash-with-an-elixir-server)\n  - [Basics on Channel and Presence](#basics-on-channel-and-presence)\n    - [Refresher (or not) on Erlang queue](#refresher-or-not-on-erlang-queue)\n    - [Refresher on Channels, Custom sockets, Presence](#refresher-on-channels-custom-sockets-presence)\n    - [Custom WebSocket connection](#custom-websocket-connection)\n      - [Client-side](#client-side)\n      - [Server-side](#server-side)\n    - [WS Security](#ws-security)\n    - [Channel set up](#channel-set-up)\n    - [Logs and local testing](#logs-and-local-testing)\n      - [Server logs](#server-logs)\n      - [Testing on local network](#testing-on-local-network)\n    - [LiveView navigation](#liveview-navigation)\n    - [Presence](#presence)\n      - [Set up](#set-up)\n      - [Stream Presence](#stream-presence)\n      - [A word on \"hooks\"](#a-word-on-hooks)\n    - [FFmpeg commands](#ffmpeg-commands-1)\n\n\u003chr/\u003e\n\n## Broadcast face contouring with mediaPipe\n\nWe have our video feed from our webcam. We want:\n\n- to get frames from this video stream and send them to the server to run some transformations server-side on them,\n- or upload these streams to the server as it is,\n- or add a face contouring layer on top of it with `mediaPipe` and send these transformed chunks to the server.\n\nOnce available, you can upload the chunks to the server:\n\n- through a `WebSocket` (via the existing `LiveSocket` or preferably via a custom WebSocket exposed by a `Channel`)\n- with an `HTTP POST` request\n- using a `RTCPeerConnection` and `RTCDataChannel`.\n\n**Get video streams**\nYou firstly get streams from the webcam with the WebRTC method `getUserMedia`. You get a `MediaStream`. You inject the stream into a `\u003cvideo\u003e` element (via the `srcObject`) and you preview your feed.\n\n```js\nthis.video = document.querySelector(\"#webcam\");\nconst stream = await navigator.mediaDevices.getUserMedia({ video: true });\nthis.video.srcObject = stream;\n```\n\n### Push frames to the server\n\nYou want to run some _object detection_ from your camera feed: you send a frame every (say) 500ms to run some heavy computations on it.\n\nTo capture a frame from a video stream, you \"draw\" an image from the `\u003cvideo\u003e` element into the `context` of a `\u003ccanvas\u003e`:\n\n```js\ncontext = canvas.getContext('2d')\ncontext.drawImage(video, ... coordinates, ...resizing coordinates)\n```\n\n\u003e You can resize the image during this operation. If you use this image for ML purposes, you may want to match the models requirements and minimise the size of the data.\n\n#### Push using WebSocket\n\nIf we want to use a WebSocket to send the data to the server, whether via the LiveSocket, or preferably via a custom WebSocket (Channel), you need to encode the data as a Base64 string.\n\n\u003e You could use `canvas.toDataURL` to convert the whole data into a B64 encoded string. However, the following is more efficient.\n\nIt is more efficient to use `canvas.toBlob` and work with the Blob. You can type the blob as \"image/webp\": this minimizes the weigth of the image a lot compared to PNG (the default) and eliminates the need to compress and decompress the data.\n\nTo transform a blob (immutable data), you need transform it into a `ArrayBuffer`: a chunk of memory with a fixed length (the length of the blob).\nThe ArrayBuffer can be mutated via types such has `Unit8Array`, typed arrays of usigned 8-bits integers.\nWe then can manipulate the Unit8Array by chunks of 32kB to produce a base64 encoded string.\nThis process lowers the memory impact and minimizes the data size.\n\nIf you use the LiveSocket, you receive the data in a `handle_event` callback in your LiveView. If you used a dedicated Channel (to separate concerns and let the LiveView handle only the UI), you receive the data in a `handle_in` callback in your Channel.\n\n#### Push using HTTP request\n\nYou need to transform the `blob` as a **`File`** to append it to a `FormData`. It can then be sent by `fetch` to a `Phoenix` controller.\nYou will get a `%Plug_Upload{}` struct that contains a temporary path to your file.\n\n#### Overview\n\n```mermaid\ngraph TD;\n    A[getUserMedia] --\u003e B[canvas.context.drawImage \u003cbr\u003e resize]\n    B-.-\u003eB1[canvas.toDataURL]\n    B1-.-\u003eD\n    B --\u003e C1[canvas.toBlob \u003cbr\u003etype image/webp]\n    C1--\u003eC2[ArrayBuffer]\n    C2 --btoa(Uint8Array)--\u003e D[b64 encoded string]\n    D -- push \u003cbr\u003ews:// --\u003e E[Elixir server b64 decode]\n    C1 -- new File(blob) --\u003eF[FormData]\n    F -- http:// POST  --\u003eE1[Elixir \u003cbr\u003e %Plug.Upload]\n```\n\n\u003cbr/\u003e\n\n#### The \"frame\" hook\n\n\u003cbr/\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003e\n    A hook to capture a frame and push to the server via liveSocket\n  \u003c/summary\u003e\n\n```js\nconst frame = {\n  intId: null,\n  video: null,\n  localStream: null\n\n  async mounted() {\n    const _this = this,\n      mediaConstraints = {\n      video: {\n        facingMode: \"user\",\n        frameRate: { ideal: 30 },\n        width: { ideal: 1900 },\n        height: { ideal: 1500 },\n      },\n      audio: false,\n    };\n\n    // setup channel\n    this.channel = streamSocket.channel(\"stream:frame\", { userId });\n    this.channel\n      .join()\n      .receive(\"error\", (resp) =\u003e {\n        console.log(\"Unable to join\", resp);\n      })\n      .receive(\"ok\", () =\u003e {\n        console.log(`Joined successfully stream:frame`);\n      });\n\n    this.video = document.querySelector(\"#webcam\");\n\n    const stream = await navigator.mediaDevices.getUserMedia(mediaConstraints);\n    this.intId = setInterval(captureFrame, 500, this.video);\n    this.video.srcObject = stream;\n\n    // to be able stop stream when leave the page (destroyed)\n    this.localStream = stream;\n\n    async function captureFrame(video) {\n      const canvas = document.createElement(\"canvas\"),\n        ctx = canvas.getContext(\"2d\"),\n        w = video.videoWidth,\n        h = video.videoHeight,\n        targetW = 244,\n        targetH = 244 * (h / w);\n\n      /* Capture a frame by drawing into a canvas and resize image\n      to the target dimensions to match the model requirements */\n      ctx.drawImage(video, 0, 0, w, h, 0, 0, targetW, targetH);\n\n      /* We need to pass the data as B64 encoded string as LiveView accepts only strings.\n      It is more efficient to canvas.toBlob and work on the Blob than directly convert the datanwith canvas.toDataURL into a B64 encoded string.\n      You also convert the canvas content to WEBP format in the canvas.toBlbob. */\n\n      // convert the data into a Blob typed as WEBP\n      const { promise, resolve } = Promise.withResolvers();\n      canvas.toBlob(resolve, \"image/webp\", 0.9);\n      const blob = await promise;\n\n      checkCapture(blob)\n\n      // convert immutable Blob into mutable object\n      const arrayBuffer = await blob.arrayBuffer();\n      //\n      const encodedB64 = arrayBufferToB64(arrayBuffer);\n\n      _this.channel.push(\"frame\",  msg)\n      // _this.pushEvent(\"frame\", { data: encodedB64 });\n      // or fetch(...)\n      // or via RTCDataChannel\n    }\n\n    function arrayBufferToB64(arrayBuffer) {\n      // convert the ArrayBuffer to a binary string\n      const bytes = new Uint8Array(arrayBuffer);\n      const chunkSize = 0x8000; // 32KB chunks\n      const chunks = [];\n      // convert chunks of Uint8Array to binary strings and encode them to base64\n      for (let i = 0; i \u003c bytes.byteLength; i += chunkSize) {\n        const chunk = bytes.subarray(i, i + chunkSize);\n        const binaryString = Array.from(chunk)\n          .map((byte) =\u003e String.fromCharCode(byte))\n          .join(\"\");\n        chunks.push(btoa(binaryString));\n      }\n      return chunks.join(\"\");\n    }\n  },\n\n  destroyed() {\n    clearInterval(this.intId);\n    this.localStream.getTracks().forEach((track) =\u003e track.stop());\n    this.localStream = null;\n    this.video = null;\n    if (this.channel) {\n      this.channel.leave();\n    }\n    console.log(\"destroyed\");\n  },\n};\n\nexport default frame\n```\n\n\u003c/details\u003e\n\u003cbr/\u003e\n\n\u003e You can check the captured image by creating an `\u003cimg\u003e` element in your DOM and pass it the data as dataURL:\n\n\u003cdetails\u003e\n  \u003csummary\u003e\n    Check your frame\n  \u003c/summary\u003e\n\n```js\nfunction checkCapture(blob) {\n  const imgURL = URL.createObjectURL(blob);\n  const imgElement = document.querySelector(\"#test-img\");\n  imgElement.src = imgURL;\n  imgElement.onload = () =\u003e {\n    URL.revokeObjectURL(imgURL);\n  };\n}\n```\n\n\u003c/details\u003e\n\u003cbr/\u003e\n\nFor example, we push a 10 kB image with a processing time (browser) less than 20 ms per image.\nWe could process this way 1000/20 = 50fps, transfering only 0.5MB/s per client through the socket.\n\n### Push video chunks\n\n\u003chttps://web.dev/articles/requestvideoframecallback-rvfc\u003e\n\nYou want to broadcast our feed and send **chunks**.\n\nOnce the `\u003cvideo\u003e` element has started to play the feed, we invoque `video.captureStream(20 fps)` and feed a `MediaRecorder`.\n\n```js\nmediaStream = video.captureStream(20);\nmediaRecorder = new MediaRecorder(mediaStream, { mimeType: \"video/webm\" });\n```\n\nWe have several ways to send these chunks to the server:\n\n- use `FileReader`, mainly used for static files. You must `captureStream` to get a blob.\n- use `Streams API`, for video streams: you can use _directly_ the stream from the video element.\n\nThen, either you can proceed with b64 encoded strings (and use a WebSocket) or files (and send an HTTP POST multipart request).\n\n- use `FileReader`, save the blob into a File, add it to a FormData and make a HTTP POST multipart request to an Elixir controller,\n- use the `Streams API`, open a ReadableStream, use a WriteableStream to make an HTTP POST multipart request to an Elixir controller.\n\nWe want to draw contours around the faces we found. We can do this in a canvas and superimpose the canvas upon the current video element. This gives the impression of contour detection.\n\nBut we want more: we want the video chunks and the contour overlay in the data.\nFor this, we draw an animation `requestAnimationFrame`. It takes a function as argument, the function that draws the update and recursively calls itself. This naturally comes with limitations.\n\nThe process is more easily visualized in a graph.\n\n```mermaid\ngraph TD;\n    A[getUserMedia] -- overlay\u003cbr\u003e face contouring --\u003e B1[canvas: draw contouring on frame]\n    B1--requestAnimationFrame--\u003eB1\n    B1 --\u003e C[canvas.captureStream 20 fps]\n    C --\u003e D[new MediaRecorder stream]\n    D --\u003e E[mediaRecorder.start x ms]\n    E -- onloadedend --\u003eF1[reader = new FileReader]\n    F1-- reader.readAsDataURL --\u003eG[b64 dataURL]\n    G-- push ws://--\u003eH1[Elixir]\n    H1--decode b64 --\u003e H1\n    G--http:// POST body --\u003eH2[Elixir]\n    H2 -- read_body \u003cbr\u003e decode b64 --\u003e H2\n\n    D --\u003e E2[mediaRecorder.start]\n    E2 --\u003e R[ReadableStreamProcessor chunks]\n    R --pipeTo --\u003e W[WritableStream]\n    W--\u003eG2\n    A -- no overlay --\u003eD\n    E -- onloadedend --\u003eF2[new File blob \u003cbr\u003e type: video/webm]\n    F2 --\u003e G2[FormData : append file]\n    G2 --http:// POST --\u003e H3[Elixir]\n    H3 -- %Plug.Upload --\u003e H3\n```\n\n\u003cbr/\u003e\n\nIt remains to send this to the server. We need to transform it into a base64 encoded string. We can use `canvas.toDataURL` which is available on the canvas. However, this increases the size (+2/6). The canvas element has also the `canvas.toBlob`. From there, we transform the immutable blob into an ArrayBuffer composed of Unit8Array on which we work to encode into b64 with `btoa` (which is limited to 16_000 characters). With this in place, we can push through the WebSocket.\n\nWhen we deal with chunks, we have blobs. We send them to the server with a POST HTTP request and use a `FormData`. We can then receive the data from a controller which has `:multipart` in his pipeline.\nOne important point is to use `new File(blob)` as Phoenix won't accept the blob as such, only containerized as a file.\n\n- you get a chunk when you `stream.captureStream(20 fps)`.\n\nget a video stream, capture a frame into a `\u003ccanvas\u003e` element, and push it to the server via the LiveSocket.\n\n\u003cdetails\u003e\n  \u003csummary\u003eHook to push video chunks via HTTP POST requests\u003c/summary\u003e\n\n```js\nexport const faceApi = {\n  localStream: null,\n  mediaRecorder: null,\n  requestId: null,\n\n  async mounted() {\n    // the webcam feed\n    const video = document.getElementById(\"webcam\"),\n      // the transformed video with the detected contours\n      overlayed = document.getElementById(\"overlayed\"),\n      displaySize = { width: video.width, height: video.height },\n      _this = this;\n\n    // we louad the libraries\n    const [faceapi, stream] = await Promise.all([\n      import(\"@vladmandic/face-api\"),\n      navigator.mediaDevices.getUserMedia({ video: true }),\n    ]);\n\n    await faceapi.nets.tinyFaceDetector.loadFromUri(\"/models/face-api\");\n\n    // display your webcam\n    video.srcObject = stream;\n    video.onloadeddata = video.play;\n\n    // keep a reference to stop the video stream once the component is destroyed\n    this.localStream = stream;\n\n    let canvas = null;\n\n    video.onplay = async () =\u003e {\n      //  draw a canvas\n      canvas = faceapi.createCanvasFromMedia(video);\n      faceapi.matchDimensions(canvas, displaySize);\n\n      await drawAnimationOnCanvas();\n      // capture the animation drawn in the canvas at 20 fps\n      const canvasStream = canvas.captureStream(20);\n      // reference to cancel the recording when the component is destroyed\n      this.mediaRecorder = new MediaRecorder(canvasStream);\n      // start recording chunks at 5 fps, ie of length 1000/5=200 ms\n      const fps = 5;\n      this.mediaRecorder.start(1000 / fps);\n      // given it to the MediaRecorder and HTTP request to the server\n      this.mediaRecorder.ondataavailable = sendBlobToServer;\n      // visualizing the animation in the second video\n      overlayed.srcObject = canvasStream;\n\n      // we can also broadcast the stream with RTCPeerConnection\n      // canvasStream.getTracks().forEach((track) =\u003e {...})\n    };\n\n    await drawAnimationOnCanvas();\n    // capture the animation drawn in the canvas at 20 fps\n    const canvasStream = canvas.captureStream(20);\n    // reference to cancel the recording when the component is destroyed\n    this.mediaRecorder = new MediaRecorder(canvasStream);\n    // start recording chunks at 5 fps, ie of length 1000/5=200 ms\n    const fps = 5;\n    this.mediaRecorder.start(1000 / fps);\n    // given it to the MediaRecorder and HTTP request to the server\n    this.mediaRecorder.ondataavailable = sendBlobToServer;\n    // visualizing the animation in the second video\n    overlayed.srcObject = canvasStream;\n\n    // we can also broadcast the stream with RTCPeerConnection\n    // canvasStream.getTracks().forEach((track) =\u003e {...})\n\n    async function sendBlobToServer({ data }) {\n      if (data.size \u003e 0) {\n        const file = new File([data], \"chunk.webm\", {\n          type: \"video/webm\",\n        });\n        const formData = new FormData();\n        formData.append(\"file\", file);\n\n        return fetch(`${window.location.origin}/face-api/upload`, {\n          method: \"POST\",\n          body: formData,\n        });\n      }\n    }\n  },\n\n  destroyed() {\n    console.log(\"destroyed\");\n    if (this.requestId) cancelAnimationFrame(id);\n\n    this.requestId = null;\n\n    if (this.localStream) {\n      this.localStream.getTracks().forEach((track) =\u003e track.stop());\n    }\n    if (this.mediaRecorder) this.mediaRecorder.stop();\n  },\n};\n```\n\n\u003c/details\u003e\n\u003cbr/\u003e\n\n## Signaling process with the LiveView\n\n`LiveView` uses a WebSocket connection between the client and the server.\n\nWhen we use the `ex_webrtc` library, each client communicates to the server. The \"live\" socket could be used for signaling.\n\nUpon a client connection, the server will start a `ex_webrtc` process. The diagram below describes the message passing, cf [LiveView client-server communication](https://hexdocs.pm/phoenix_live_view/js-interop.html#client-server-communication)\n\n```mermaid\nsequenceDiagram\n  participant Server\n  participant LiveView\n  participant Browser\n\n  Note right of Browser: client connects\n  Browser -\u003e\u003eLiveView: connects\n  LiveView -\u003e\u003e Server: calls Room.connect \u003cbr\u003e (lv_pid)\n  Note left of Server: start \u003cbr\u003eExWebRTC\n\n  Note right of Browser: WebRTC event\n  Browser -\u003e\u003e LiveView: this.pushEvent\u003cbr\u003e({:signal, msg})\n  LiveView -\u003e\u003e Server: Room.receive_signal\u003cbr\u003e{:signal, msg}\n  activate Server\n  Note left of Server: ExWebRTC\u003cbr\u003eprocess\n  Server -\u003e\u003e LiveView: send \u003cbr\u003e(lv_pid, {:signal, msg})\n  deactivate Server\n  Note right of LiveView: handle_info\u003cbr\u003e({:signal, msg})\n  LiveView -\u003e\u003e Browser: push_event\u003cbr\u003e(lv_socket, {:signal, msg})\n  Note left of Browser: this.handleEvent\u003cbr\u003e(\"event\", msg)\n```\n\nThe event handler in the LiveView to `this.pushEvent` from the client:\n\n```elixir\ndef handle_event(\"signal\", msg, socket) do\n  Rtc.Room.receive_signaling_msg(socket.assigns.room_id, msg)\n  {:noreply, socket |\u003e push_event(msg[\"type\"], msg)}\nend\n```\n\nThe message handler in the LiveView to a `Kernel.send` from the `ExWebRTC` server:\n\n```elixir\n@impl true\ndef handle_info({:signaling, %{\"type\" =\u003e type} = msg}, socket) do\n  {:noreply, socket |\u003e push_event(type, msg)}\nend\n```\n\nIn this configuration, we will push and receive data to other peers, ie other LiveViews: messages are not flowing between LiveViews.\n\nWe would need to **broadcast** messages to spread it among the different LiveView processes.\n\nTo separate the concerns, we used the Channel API since joining peers will connect to the same Channel topic. The primitives are easy: two Javascript methods `channel.push`, `channel.on`, and one Elixir listener `handle_in` that runs a `broadcast_from`.\n\n[:arrow_up:](#rtc---demo-of-elixir-and-webrtc)\n\n## WebRTC\n\n### WebRTC signaling flow\n\nSource :\u003chttps://developer.mozilla.org/en-US/docs/Web/API/WebRTC_API/Signaling_and_video_calling#signaling_transaction_flow\u003e\n\nWe have three flows of data to exchange between peers: SDP, streams and ICE.\n\n- the `ICE` protocol (Interactive Connectivity Establishment) is used to establish the path of the connections between peers. ICE candidates are delivered by a STUN server or TURN servers. In fact on localhost, you don't need anything!.\n- the `SDP` protocol (Session Description Protocol) is used to describe how to set up multimedia session between peers. The data contains informations such as the codecs. It negotiates the RTP (Real Time Protocol). The SCTCP (Stream Control Transmission Protocol) manages the data transport, in particular for the `DataChannel` API.\n- media streams captured by `mediaDevices.getUserMedia`.\n\nThe WebRTC process is fully managed by the browser's WebRTC API. You only need to code the sequence of the data exchange between peers.\n\n![signaling](https://github.com/ndrean/RTC/blob/main/priv/static/images/signaling.png)\n\n\u003e The signaling process that transports the data between peers can use WebSockets or HTTP requests.\n\u003e If we use WebSockets, we can use:\n\u003e\n\u003e - directly the LiveView socket. Check [this paragraph](#signaling-process-with-the-liveview),\n\u003e - use `Elixir.Channel`, a process running on top of a custom WebSocket connection between the browser and the Phoenix server.\n\n\u003e This connection is usefull only during the lifetime of the set up of the connection. You can even shut down the server afterwards, the RTC connection will persist.\n\n#### Connexion and SDP exchange\n\nSource: [MDN Session description](https://developer.mozilla.org/en-US/docs/Web/API/WebRTC_API/Connectivity#session_descriptions)\n\n1. The caller captures local Media via `MediaDevices.getUserMediagetUserMedia`.\n2. The caller creates `pc = new RTCPeerConnection()` and calls `RTCPeerConnection.addTrack()`.\n3. The caller calls `pc.createOffer()`to create an offer.\n4. The caller calls `pc.setLocalDescription()` to set that offer as the local description (that is, the description of the local end of the connection).\n5. After `setLocalDescription()`, the caller asks STUN servers to generate the ice candidates\n6. The caller uses the signaling server to transmit the offer to the intended receiver of the call.\n7. The recipient receives the offer and calls `pc.setRemoteDescription()` to record it as the remote description (the description of the other end of the connection).\n8. The recipient does any setup it needs to do for its end of the call: capture its local media, and attach each media tracks into the peer connection via `pc.addTrack()`.\n9. The recipient then creates an answer by calling `pc.createAnswer()`.\n10. The recipient calls `pc.setLocalDescription()`, passing in the created answer, to set the answer as its local description. The recipient now knows the configuration of both ends of the connection.\n11. The recipient uses the signaling server to send the answer to the caller.\n12. The caller receives the answer.\n13. The caller calls `pc.setRemoteDescription()` to set the answer as the remote description for its end of the call. It now knows the configuration of both peers.\n\nThe SDP flow between two peers:\n\n```mermaid\nsequenceDiagram\n  participant A as Peer A\n  participant C as Channel\n  participant B as Peer B\n\n  A --\u003e C: join\n  Note right of A: connection\n  A -\u003e\u003eA: streams = getUserMedia(audio, video)\n  A-\u003e\u003eA: \u003cvideo local srcObject=streams\u003e\n  A-\u003e\u003eA: pc = new RTCPeerConnection()\n  A-\u003e\u003eA: pc.addTrack(streams)\n\n\n  B --\u003e C: join\n  Note left of B: connection\n  B -\u003e\u003eB: streams = getUserMedia(audio, video)\n  B-\u003e\u003eB: \u003cvideo local srcObject=streams\u003e\n  B -\u003e\u003eB: pc = new RTCPeerConnection()\n  B -\u003e\u003eB: pc.addTrack(streams)\n  B -\u003e\u003eB: offer = createOffer()\n  B-\u003e\u003eB: setLocalDescription(offer)\n  B -\u003e\u003e C: OFFER event\n  C --\u003e\u003e A: broadcast OFFER \u003cbr\u003e(except to Peer B)\n  activate A\n  Note right of A: OFFER event listener\n  A-\u003e\u003eA: setRemoteDescription(offer)\n  A-\u003e\u003eA: answer = createAnswer()\n  A-\u003e\u003eA: setLocalDescription(answer)\n  A -\u003e\u003e C: ANSWER event\n  deactivate A\n  C --\u003e\u003e B: broadcast ANSWER \u003cbr\u003e (except to Peer A)\n  Note left of B: ANSWER event listener\n  B-\u003e\u003eB: setRemoteDescription(answer)\n  Note left of B: connection \u003cbr\u003ecomplete\n```\n\nThe code for two peers is [here](#rtc-module)\n\nThe WebRTC connection uses the `RTCPeerConnection` object. The final state of the object after the `SDP` exchange process and ICE process is described below.\n\n```mermaid\n  classDiagram\n  class RTCPeerConnection {\n    +currentocalDescription: RTCSessionDescription\n    +currentRemoteDescription: RTCSessionDescription\n\n    +iceConnectionState: RTCIceConnectionState\n    +connectionState: RTCPeerConnectionState\n    +signalingState: RTCSignalingState\n    +iceGatheringState: RTCIceGatheringState\n\n    pc.ontrack() =  set_stream_to_video_srcObj()\n    pc.onnegotiationneeded()= createOffer()\n    pc.onicecandidate() = signalCandidate()\n  }\n\n    class Peer_A  {\n        currentLocalDescription: \"answer\"\n        currentRemoteDescription: \"offer\"\n        +iceConnectionState: \"connected\"\n        +connectionState: \"connected\"\n        +signalingState: \"stable\"\n        +iceGatheringState: \"complete\"\n    }\n\n    class Peer_B {\n        currentLocalDescription: \"offer\"\n        currentRemoteDescription: \"answer\"\n        +iceConnectionState: \"connected\"\n        +connectionState: \"connected\"\n        +signalingState: \"stable\"\n        +iceGatheringState: \"complete\"\n    }\n    RTCPeerConnection --\u003e Peer_A\n    RTCPeerConnection --\u003e Peer_B\n```\n\n\u003cbr\u003e\n\n#### Media streams\n\nThe easiest process is the media stream. You invoque:\n\n```js\nnavigator.mediaDevices.getUserMedia;\n```\n\nto access your local camera and microphone and receive streams from them.\nYou pass the streams to the `srcObj` attribute of a `\u003cvideo\u003e` et voilà, you have your local stream.\n\nOnce the communication is established between peers, the `RTCPeerConnection` protocole will send a \"track\" event. It returns remote streams. Your callback will simply pass them to the `scrObj` attribute of your other `\u003cvideo\u003e` element of your page. This will reflect the data from the remote camera.\n\n#### The ICE exchange\n\nPeers exchange ICE candidates in both directions to maximize the chances of etablishing the best direct connection.\n\nTo be able to process a candidate, a peer must have set his remote description. We must therefor store the received candidates until the peer PC can process it.\n\n```mermaid\nsequenceDiagram\n  participant Peer A\n  participant Signaling Channel\n  participant Peer B\n\n  Peer A -\u003e\u003e Signaling Channel: ICE Candidate\n  Signaling Channel --\u003e\u003e Peer B: broadcast ICE \u003cbr\u003e(except to peer A)\n  activate Peer B\n  Note left of Peer B: process or enqueue\n  Peer B -\u003e\u003e Signaling Channel: ICE Candidate\n  deactivate Peer B\n  Signaling Channel --\u003e\u003e Peer A: broadcast ICE \u003cbr\u003e(except to peer B)\n\n  Note right of Peer A: process or enqueue\n```\n\n### Flow for 3+ peers\n\nWhen a new peer A connects to the channel, the channel will broadcast an event NEW (from the server-side).\nThe listeners of the connected user B will react by creating a `new PeerConnection` instance for the new peer A. He will also send a PING signal to the peer A for him to start the reverse connection A-\u003eB upon reception. Then the SDP and ICE transactions can start.\nWe need to trace the PeerConnections between peers. Each peer will store an object whose keys are the IDs of the other connected peers and the RTCPeerConnection object. For example, if A, B and C are connected, then A has something like:\n\n```js\npcs = {user_idB: RTC_pc(A-\u003eB), user_idC: RTC_pc(A-\u003eC)}\n```\n\n\u003e :exclamation: In order not to _double the offers_, we used an **ordering function** between peers identifiers. In our case, the identifiers are numbers so we used the following rule: if `Id(A)\u003cId(B)`, then B will send an offer in the \"negotiationneeded\" callback. This works because the roles of peers are _inverted when viewed by the other peer_ (B becomes A, and A is B).\n\n\u003e Note that the case of connecting just two peers is simplified as it doesn't need any ordering, nor keeping track of the connections.\n\n```mermaid\nsequenceDiagram\n    participant S as SignalingServer\n    participant A as userA\n    participant B as userB\n    participant C as userC\n\n    A--\u003e\u003e+S: join(roomId, A)\n    S--\u003e-C: broadcast_from(A): NEW\n\n\n    B--\u003e\u003e+S: channel.join(roomId, B)\n    S--\u003e-C: broadcast_from(B): NEW\n    activate A\n    Note right of A: A receives NEW, ({from: B})\n    A -\u003e\u003e+S: push PING\u003cbr\u003e ({from: A, to: B} )\n    A -xB: create PeerConnection with  B\n    deactivate A\n    S--\u003eC: broadcast({from: A, to: B}): PING\n    deactivate S\n    activate B\n    Note right of B: B matches PING from  A\n    B -x A: create PeerConnection with A\n    deactivate B\n\n    A-\u003e\u003eB: OFFER (SDP)\n    activate B\n    B-\u003e\u003eA: ANSWER (SDP)\n    deactivate B\n    activate A\n    Note right of A: RTC A \u003c-\u003e B established\n    deactivate A\n```\n\n\u003cbr\u003e\n\n#### WebRTC 3+ client code\n\nIn the code below, we expose to the `window` object the \"pcs\" object that tracks the peer connections.\nEach message passed through the channel will get a `{from, to}` object appended.\n\n\u003cdetails\u003e\u003csummary\u003eThe 3+ WebRTC implementation\u003c/summary\u003e\n\n```js\nimport setPlayer from \"./setPlayer.js\";\nimport joinChannel from \"./signalingChannel.js\";\n\nconst configuration = {\n  iceServers: [{ urls: \"stun:stun.l.google.com:19302\" }],\n};\nconst mediaConstraints = {\n  video: {\n    facingMode: \"user\",\n    frameRate: { ideal: 15 },\n    width: { ideal: 320 },\n    height: { ideal: 160 },\n  },\n  audio: true,\n};\n\nfunction order(userA, userB) {\n  BigInt(userA) \u003c BigInt(userB)\n}\n\nconst RTC = {\n  // global variables\n  pcs: {},\n  pc: null,\n  pc_curr: null,\n  channel: null,\n  localStream: null,\n\n  destroyed() {\n    if (this.localStream) {\n      this.localStream.getTracks().forEach((track) =\u003e track.stop());\n      this.localStream = null;\n    }\n\n    if (this.channel) {\n      this.channel.leave().receive(\"ok\", () =\u003e {\n        console.log(\"left room, closing channel\", this.channel.topic);\n      });\n      this.channel = null;\n    }\n    if (this.pc) {\n      this.pc.close();\n      this.pc = null;\n    }\n    delete window.pc;\n    delete window.pcs;\n    console.log(\"destroyed\");\n  },\n\n  async mounted() {\n    let rtc = this,\n      iceCandidatesQueue = [];\n\n    const userId = document.querySelector(\"#room-view\").dataset.userId;\n    const roomId = window.location.pathname.slice(1).toString();\n\n    async function handleOffer({ sdp, from, to }) {\n      if (to !== userId) return;\n\n      const pc = rtc.pcs[from];\n      await pc.setRemoteDescription(sdp);\n      const answer = await pc.createAnswer();\n      await pc.setLocalDescription(answer);\n\n      rtc.channel.push(\"answer\", {\n        sdp: pc.localDescription,\n        type: \"answer\",\n        from: to,\n        to: from,\n      });\n    }\n\n    async function handleAnswer({ from, to, sdp }) {\n      if (to !== userId) return;\n      const pc = rtc.pcs[from];\n\n      await pc.setRemoteDescription(sdp);\n      consumeIceCandidates(to);\n    }\n\n    async function handleCandidate({ candidate, from, to }) {\n      if (to !== userId || !candidate) return;\n\n      const pc = rtc.pcs[from];\n      if (pc) {\n        await pc.addIceCandidate(new RTCIceCandidate(candidate));\n      } else {\n        iceCandidatesQueue.push({ candidate, from });\n      }\n    }\n\n    function createConnection({ user, peer }, stream) {\n      const pc = new RTCPeerConnection(configuration);\n\n      stream.getTracks().forEach((track) =\u003e pc.addTrack(track, stream));\n\n      pc.onicecandidate = (event) =\u003e {\n        if (event.candidate) {\n          rtc.channel.push(\"ice\", {\n            candidate: event.candidate,\n            type: \"ice\",\n            from: user,\n            to: peer,\n          });\n        }\n      };\n\n      pc.ontrack = ({ streams }) =\u003e {\n        setPlayer(\"new\", streams[0], peer);\n      };\n\n      pc.onnegotiationneeded = async () =\u003e {\n        // only one of the 2 peers should create the offer\n        if order(user,peer) return;\n\n        const offer = await pc.createOffer();\n        await pc.setLocalDescription(offer);\n\n        rtc.channel.push(\"offer\", {\n          sdp: pc.localDescription,\n          type: \"offer\",\n          from: user,\n          to: peer,\n        });\n      };\n\n      pc.onconnectionstatechange = () =\u003e {\n        const state = pc.connectionState;\n        switch (state) {\n          case \"connected\":\n            console.log(\"~~\u003e Connection state: \", state, { user, peer });\n            console.log(rtc.pcs);\n            break;\n          case \"disconnected\":\n          case \"failed\":\n          case \"closed\":\n            console.log(\"~~\u003e Connection state: \", state, { user, peer });\n            delete rtc.pcs[peer];\n            rtc.destroyed();\n            break;\n          default:\n            console.log(\"~~\u003e Connection state: \", state, { user, peer });\n            break;\n        }\n      };\n\n      rtc.pcs[peer] = pc;\n      window.pcs = rtc.pcs;\n\n      return pc;\n    }\n\n    const handlers = {\n      offer: handleOffer,\n      answer: handleAnswer,\n      ice: handleCandidate,\n      ping: ({ from, to }) =\u003e {\n        if (to !== userId) return;\n\n        const peers = { user: userId, peer: from };\n        rtc.pc = createConnection(peers, rtc.localStream);\n      },\n      new: ({ from, to }) =\u003e {\n        const peers = { user: userId, peer: from };\n\n        if (from !== userId \u0026\u0026 to === undefined) {\n          rtc.channel.push(\"ping\", { from: userId, to: from });\n          rtc.pc = createConnection(peers, rtc.localStream);\n        }\n      },\n    };\n\n    this.localStream = await navigator.mediaDevices.getUserMedia(\n      mediaConstraints\n    );\n    setPlayer(\"local\", this.localStream);\n\n    this.channel = await joinChannel(roomId, userId, handlers);\n\n    function consumeIceCandidates(from) {\n      while (iceCandidatesQueue.length \u003e 0) {\n        iceCandidatesQueue = iceCandidatesQueue.filter((item) =\u003e {\n          if (item.from === from) {\n            rtc.pcs[from].addIceCandidate(item.candidate);\n            return false;\n          }\n          return true;\n        });\n      }\n    }\n  },\n};\n\nexport default RTC;\n```\n\n\u003c/details\u003e\n\u003cbr/\u003e\n\nand the video player module helper (to add dynamically video tags):\n\n\u003cdetails\u003e\u003csummary\u003eThe VideoPlayer module\u003c/summary\u003e\n\n```js\nexport default function setPlayer(eltId, stream, from = \"\") {\n  let video;\n\n  const remote = document.getElementById(from);\n\n  if (eltId === \"new\" \u0026\u0026 remote === null) {\n    video = document.createElement(\"video\");\n    video.id = from;\n    video.setAttribute(\"class\", \"w-full h-full object-cover rounded-lg\");\n\n    const fig = document.createElement(\"figure\");\n    const figcap = document.createElement(\"figcaption\");\n    figcap.setAttribute(\"class\", \"text-red-500\");\n    figcap.textContent = from;\n    document.querySelector(\"#videos\").appendChild(fig);\n    fig.appendChild(video);\n    video.after(figcap);\n  } else {\n    if (eltId === \"new\" \u0026\u0026 remote !== null) {\n      video = remote;\n    } else {\n      video = document.getElementById(eltId);\n    }\n  }\n  video.srcObject = stream;\n  video.controls = false;\n  video.muted = true;\n  video.playsInline = true;\n\n  video.onloadeddata = (e) =\u003e {\n    try {\n      video.play();\n    } catch (e) {\n      console.error(e);\n    }\n  };\n}\n```\n\n\u003c/details\u003e\n\u003cbr/\u003e\n\n#### The Elixir signaling channel\n\nThe \"signaling_channel\" Elixir implementation. It is the module that manages the Channel process attached to the custom WebSocket.\nIt uses `handle_in` callbacks from the client (the RTC.js module) and responds with `broadcast_from`.\nThe data just passes through.\n\n```elixir\ndefmodule RtcWeb.SignalingChannel do\n  use RtcWeb, :channel\n  require Logger\n\n  @impl true\n  def join(\"room:\" \u003c\u003e id = _room_id, payload, socket) do\n    send(self(), {:after_join, id})\n    {:ok, assign(socket, %{room_id: id, user_id: payload[\"userId\"]})}\n  end\n\n  @impl true\n  def handle_info({:after_join, id}, socket) do\n    :ok = broadcast_from(socket, \"new\", %{\"from\" =\u003e socket.assigns.user_id})\n    {:noreply, socket}\n  end\n\n  # 'broadcast_from' to send the  message to all OTHER clients in the room\n  @impl true\n  def handle_in(event, msg, socket) do\n    :ok = broadcast_from(socket, event, msg)\n    {:noreply, socket}\n  end\n\n  @impl true\n  def terminate(reason, socket) do\n    room_id = socket.assigns.room_id\n    Logger.warning(\"STOP Channel:#{room_id}, reason: #{inspect(reason)}\")\n    {:stop, reason}\n  end\nend\n```\n\n#### Phoenix Channel client side\n\nThis is the code of \"signalingChannel.js\", client-side implementation.\n\n\u003cdetails\u003e\u003csummary\u003esignalingChannel.js\u003c/summary\u003e\n\n```js\nimport roomSocket from \"./roomSocket\";\n\n// this function is async to ensure the channel is joined before starting the WebRTC process\nexport default async function joinChannel(roomId, userId, callbacks) {\n  return new Promise((resolve) =\u003e {\n    const channel = roomSocket.channel(\"room:\" + roomId, { userId });\n\n    channel\n      .join()\n      .receive(\"error\", (resp) =\u003e {\n        console.log(\"Unable to join\", resp);\n        window.location.href = \"/\";\n      })\n      .receive(\"ok\", () =\u003e {\n        console.log(`Joined successfully room:${roomId}`);\n        setHandlers(channel, handlers);\n        resolve(channel);\n      });\n  });\n}\n\nfunction setHandlers(channel, callbacks) {\n  for (let key in callbacks) {\n    channel.on(key, callbacks[key]);\n  }\n}\n```\n\n\u003c/details\u003e\n\u003cbr/\u003e\n\nIt attaches a channel to the custom `roomSocket`, calls `channel.join()` and set the listeners `channel.on()` with callbacks defined in RTC.js.\n\nIt is async to ensure that the channel is connected before starting the PeerConnection process.\n\n#### Details of WebRTC objects\n\n![detail webrtc objects](https://github.com/ndrean/RTC/blob/main/priv/static/images/detail-webrtc-objects.png)\n\n\u003cdetails\u003e\u003csummary\u003eDetail of the WebRTC objects\u003c/summary\u003e\n\n```mermaid\nclassDiagram\n  class RTCPeerConnection {\n    +localDescription: RTCSessionDescription\n    +remoteDescription: RTCSessionDescription\n    +iceConnectionState: RTCIceConnectionState\n    +connectionState: RTCPeerConnectionState\n    +signalingState: RTCSignalingState\n    +iceGatheringState: RTCIceGatheringState\n    +onicecandidate: RTCPeerConnectionIceEvent\n\n    pc.ontrack() =  \"append stream to video\"\n    pc.onnegotiationneeded()= createOffer()\n    pc.onicecandidate() = signalCandidate()\n  }\n\n  class RTCSessionDescription {\n    +type: RTCSdpType\n    +sdp: String\n  }\n\n  class RTCIceCandidate {\n    +candidate: String\n    +sdpMid: String\n    +sdpMLineIndex: Number\n  }\n\n  RTCPeerConnection \"1\" *-- \"1\" RTCSessionDescription : localDescription\n  RTCPeerConnection \"1\" *-- \"1\" RTCSessionDescription : remoteDescription\n  RTCPeerConnection \"1\" *-- \"*\" RTCIceCandidate : iceCandidates\n\n  class MediaStream {\n    +id: String\n    +active: Boolean\n    +getTracks(): MediaStreamTrack[]\n    +getAudioTracks(): MediaStreamTrack[]\n    +getVideoTracks(): MediaStreamTrack[]\n    +addTrack(track: MediaStreamTrack): void\n    +removeTrack(track: MediaStreamTrack): void\n  }\n\n  class MediaStreamTrack {\n    +id: String\n    +kind: String\n    +enabled: Boolean\n    +muted: Boolean\n    +readyState: MediaStreamTrackState\n    +stop(): void\n  }\n\n  RTCPeerConnection \"1\" *-- \"1\" MediaStream : localStream\n  RTCPeerConnection \"1\" *-- \"1\" MediaStream : remoteStream\n  MediaStream \"1\" *-- \"*\" MediaStreamTrack : tracks\n```\n\n\u003c/details\u003e\n\u003cbr/\u003e\n\n[:arrow_up:](#rtc---a-demo-of-webrtc-with-elixir)\n\n## ExWebRTC\n\nWe will now use the package `ex_webrtc` that provides a server side solution written in Elixir.\n\nWe start with the \"echo\" demo: the ExWebRTC server sends back to the user his own video streams. It sends the video in SRTP packets using VP8, so the browser can play it.\n\n### Using channels\n\n#### The server WebRTC process\n\n##### Signaling module\n\nWe will use `Elixir.Channel` for the signaling between the client and the server `ExWebRTC` processes. The message flow between the browser and the `ExWebRTC` process passes through a Channel. The LiveView process isn't involded.\n\n```mermaid\nsequenceDiagram\n  participant S as Server\n  participant C as Channel\n  participant B as Browser\n\n  Note right of B: client connects\n  B -\u003e\u003eC: join()\n  activate C\n  C -\u003e\u003e S: Room.connect \u003cbr\u003e (ch_pid)\n  deactivate C\n  Note left of S: start \u003cbr\u003eExWebRTC\n\n  Note right of B: WebRTC event\n  B -\u003e\u003e C: channel.on\u003cbr\u003e({:signal, msg})\n  activate C\n  Note right of C: handle_in\n  C -\u003e\u003e S: Room.receive_signal\u003cbr\u003e(ch_pid, {:signal, msg})\n  activate S\n  deactivate C\n  Note left of S: ExWebRTC\u003cbr\u003eprocess\n  S -\u003e\u003e C: Kernel.send \u003cbr\u003e(ch_pid, {:signal, msg})\n  deactivate S\n  activate C\n  Note right of C: handle_info\u003cbr\u003e({:signal, msg})\n  C -\u003e\u003e B: push\u003cbr\u003e({:signal, msg})\n  deactivate C\n  activate B\n  Note left of B: channel.on\u003cbr\u003e(\"event\", msg)\n  deactivate B\n```\n\n\u003cbr/\u003e\n\nIn the \"signaling_channel.ex\" module, we add a `handle_info` that will receive the messages sent from the server to the channel pid. We use `push` since we send to the client on the same socket. The messages sent from the client are received in the `handle_in`: it calls server code, the GenServer that controls the Room.\n\n```elixir\ndefmodule RtcWeb.SignalingChannel do\n  use RtcWeb, :channel\n  require Logger\n\n  @impl true\n  def join(\"room:\" \u003c\u003e id = _room_id, payload, socket) do\n    send(self(), {:after_join, id})\n    {:ok, assign(socket, %{room_id: id, user_id: payload[\"userId\"], users: []})}\n  end\n\n  @impl true\n  def handle_info({:after_join, id}, socket) do\n    # calls ExWebRTC.PeerConnection.start() on the server\n    :connected = Rtc.Room.connect(id, self())\n    {:noreply, socket}\n  end\n\n  @impl true\n  def handle_info({:signaling, %{\"type\" =\u003e type} = msg}, socket) do\n    :ok = push(socket, type, msg)\n    {:noreply, socket}\n  end\n\n  @impl true\n  def handle_in(\"signal\", msg, socket) do\n    Rtc.Room.receive_signaling_msg(socket.assigns.room_id, msg)\n    {:noreply, socket}\n  end\n```\n\nThe \"signalingChannel.js\" remains the same.\n\n[:arrow_up:](#rtc---demo-of-elixir-and-webrtc)\n\n### RTC module\n\nThe \"RTC.js\" module is simplified. Change the reference in \"app.js\".\n\n\u003cdetails\u003e\u003csummary\u003eWebRTC hook to communicate with the server\u003c/summary\u003e\n\n```js\n// /assets/js/serverRTC.js\nimport setPlayer from \"./setPlayer.js\";\nimport joinChannel from \"./signalingChannel.js\";\n\nconst RTC = {\n  // global variables\n  pc: null,\n  channel: null,\n  localStream: null,\n\n  destroyed() {\n    console.log(\"destroyed\");\n    if (this.localStream) {\n      this.localStream.getTracks().forEach((track) =\u003e track.stop());\n      this.localStream = null;\n    }\n\n    if (this.channel) {\n      this.channel.leave().receive(\"ok\", () =\u003e {\n        console.log(\"left room, closing channel\", this.channel.topic);\n      });\n      this.channel = null;\n    }\n    if (this.pc) {\n      this.pc.close();\n      this.pc = null;\n    }\n    window.pc = null;\n  },\n\n  async mounted() {\n    const configuration = {\n      iceServers: [{ urls: \"stun:stun.l.google.com:19302\" }],\n    };\n\n    const mediaConstraints = {\n      video: {\n        facingMode: \"user\",\n        frameRate: { ideal: 15 },\n        width: { ideal: 320 },\n        height: { ideal: 160 },\n      },\n      audio: true,\n    };\n\n    let iceCandidatesQueue = [];\n\n    const userId = document.querySelector(\"#room-view\").dataset.userId;\n    const roomId = window.location.pathname.slice(1).toString();\n\n    let rtc = this;\n\n    const handlers = {\n      offer: async (msg) =\u003e {\n        await rtc.pc.setRemoteDescription(msg.sdp);\n        const answer = await rtc.pc.createAnswer();\n        await rtc.pc.setLocalDescription(answer);\n\n        rtc.channel.push(\"answer\", {\n          sdp: rtc.pc.localDescription,\n          type: \"answer\",\n          from: userId,\n        });\n      },\n      answer: async (msg) =\u003e {\n        await rtc.pc.setRemoteDescription(msg.sdp);\n      },\n      ice: async (msg) =\u003e {\n        if (msg.candidate === null) {\n          return;\n        }\n        await rtc.pc.addIceCandidate(msg.candidate);\n      },\n    };\n\n    rtc.channel = await joinChannel(roomId, userId, handlers);\n    rtc.pc = new RTCPeerConnection(configuration);\n\n    const stream = await navigator.mediaDevices.getUserMedia(mediaConstraints);\n    // to close the Media when the user leaves the room\n    rtc.localStream = stream;\n    setPlayer(\"local\", stream);\n    stream.getTracks().forEach((track) =\u003e rtc.pc.addTrack(track, stream));\n\n    rtc.pc.onicecandidate = (event) =\u003e {\n      if (event.candidate) {\n        rtc.channel.push(\"ice\", {\n          candidate: event.candidate,\n          type: \"ice\",\n        });\n      }\n    };\n\n    rtc.pc.ontrack = ({ streams }) =\u003e {\n      setPlayer(\"remote\", streams[0]);\n    };\n\n    rtc.pc.onconnectionstatechange = listenConnectionState;\n\n    rtc.pc.onnegotiationneeded = async () =\u003e {\n      const offer = await rtc.pc.createOffer();\n      await rtc.pc.setLocalDescription(offer);\n      rtc.channel.push(\"offer\", { sdp: offer, type: \"offer\", from: userId });\n    };\n\n    function listenConnectionState() {\n      const state = rtc.pc.connectionState;\n      if (\n        state === \"disconnected\" ||\n        state === \"failed\" ||\n        state === \"closed\"\n      ) {\n        rtc.destroyed();\n      }\n    }\n  },\n};\n\nexport default RTC;\n```\n\n\u003c/details\u003e\n\n[:arrow_up:](#rtc---demo-of-elixir-and-webrtc)\n\n### Example of ExWebRTC with an Echo server\n\n[Source: ExWebRTC Echo example](https://github.com/elixir-webrtc/ex_webrtc/tree/master/examples/echo/lib/echo)\n\nWhen the user navigates to the Echo page, the Javascript hook will run. It will start a Channel which in turn will start an ExWebRTC PeerConnection server side. The hook will also instantiate a WebRTC connection with the ExWebRTC server. The signaling process will start.\nThe browser will display its own video, send it to the server who will echo it back and the browser will display it in another `\u003cvideo\u003e` element.\n\nThe key is to let the ExWebRTC server instance (named `pc` below) send back the packet received from the client - in a `handle_info(:rtp)` - under his own \"server_track_id\".\n\n```elixir\nPeerConnection.send_rtp(pc, server_track_id, client_packet)\n```\n\n```mermaid\nsequenceDiagram\n    participant A as Client A\n    participant PcA as PcA \u003cbr\u003e(instance A)\n\n    A-\u003e\u003ePcA: Offer SDP (A)\n    PcA-\u003e\u003eA: Answer SDP (PcA -\u003e A)\n    PcA-\u003e\u003eA: ICE Candidates (PcA -\u003e A)\n    A-\u003e\u003ePcA: ICE Candidates (A -\u003e PcA)\n\n\n    rect rgb(173, 201, 230)\n        A--\u003e\u003ePcA: A sends streams to PcA \u003cbr\u003e local source \u003cvideo\u003e\n\n        PcA--\u003e\u003eA: PcA forward streams \u003cbr\u003e remote source \u003cvideo\u003e\n\n        note over A,PcA: Streaming Process\n    end\n```\n\n\u003cdetails\u003e\u003csummary\u003eExWebRTC Echo server\u003c/summary\u003e\n\n```elixir\ndefmodule RTC.Room do\n  use GenServer, restart: :temporary\n\ndefp id(room_id), do:\n  {:via, Registry, {Rtc.Reg, room_id}}\n###\n\ndef start_link(room_id), do:\n  GenServer.start_link(__MODULE__, room_id, name: id(room_id))\n\ndef connect(room_id, channel_pid), do:\n  GenServer.call(id(room_id), {:connect, channel_pid})\n\ndef receive_signaling_msg(room_id, msg), do:\n  GenServer.cast(id(room_id), {:receive_signaling_msg, msg})\n\n#####\ndef init(room_id) do\n  {:ok,\n    %{\n      room_id: room_id,\n      pc: nil,\n      pc_id: nil,\n      channel: nil,\n      client_video_track: nil,\n      client_audio_track: nil\n    }}\nend\n\ndef handle_call({:connect, channel_pid}, _from, state) do\n\n  Process.monitor(channel_pid)\n  {:ok, pc} = PeerConnection.start_link(ice_servers: @ice_servers)\n\n  state =\n    state\n    |\u003e Map.put(:channel, channel_pid)\n    |\u003e Map.put(:pc, pc)\n\n  vtrack = MediaStreamTrack.new(:video)\n  atrack = MediaStreamTrack.new(:audio)\n  {:ok, _sender} \u003c- PeerConnection.add_track(pc, vtrack)\n  {:ok, _sender} \u003c- PeerConnection.add_track(pc, atrack)\n\n  new_track =\n    %{\n      serv_video_track: vtrack,\n      serv_audio_track: atrack\n    }\n  {:reply, :connected, Map.merge(state, new_track)}\n\nend\n\n#-- receive offer from client\ndef handle_cast({:receive_signaling_msg, %{\"type\" =\u003e \"offer\"} = msg}, state) do\n    with desc \u003c-\n           SessionDescription.from_json(msg[\"sdp\"]),\n         :ok \u003c-\n           PeerConnection.set_remote_description(state.pc, desc),\n         {:ok, answer} \u003c-\n           PeerConnection.create_answer(state.pc),\n         :ok \u003c-\n           PeerConnection.set_local_description(state.pc, answer),\n         :ok \u003c-\n           gather_candidates(state.pc) do\n      Logger.debug(\"--\u003e Server sends Answer to remote\")\n\n      #  the 'answer' is formatted into a struct, which can't be read by the JS client\n      sent_answer = %{\n        \"type\" =\u003e \"answer\",\n        \"sdp\" =\u003e %{type: answer.type, sdp: answer.sdp},\n        \"from\" =\u003e msg[\"from\"]\n      }\n\n      send(state.channel, {:signaling, sent_answer})\n      {:noreply, state}\n    else\n      error -\u003e\n        Logger.error(\"Server: Error creating answer: #{inspect(error)}\")\n        {:stop, :shutdown, state}\n    end\n  end\n\n  # -- receive ICE Candidate from client\n  def handle_cast({:receive_signaling_msg, %{\"type\" =\u003e \"ice\"} = msg}, state) do\n    case msg[\"candidate\"] do\n      nil -\u003e\n        {:noreply, state}\n\n      candidate -\u003e\n        candidate = ICECandidate.from_json(candidate)\n        :ok = PeerConnection.add_ice_candidate(state.pc, candidate)\n        Logger.debug(\"--\u003e Server processes remote ICE\")\n        {:noreply, state}\n    end\n  end\n\n#-- send ICE candidate to the client\ndef handle_info({:ex_webrtc, _pc, {:ice_candidate, candidate}}, state) do\n  candidate = ICECandidate.to_json(candidate)\n  send(state.channel, {:signaling, %{\"type\" =\u003e \"ice\", \"candidate\" =\u003e candidate}})\n  {:noreply, state}\nend\n\n# receive the client track_id per kind and save it in the state\ndef handle_info({:ex_webrtc, _pc, {:track, %{kind: :audio} = track}}, state) do\n    {:noreply, %{state | client_audio_track: track}}\n  end\n\n  def handle_info({:ex_webrtc, pc, {:track, %{kind: :video} = track}}, state) do\n    {:noreply, %{state | client_video_track: track}}\n  end\n\n# the server receives packets from the client.\n# We pick the packets with kind :audio by matching the received track_id with the\n# state.client_audio_track.id.\n# We send these packets to the PeerConnection under the server audio track id.\n\ndef handle_info(\n        {:ex_webrtc, pc, {:rtp, c_id, packet}},\n        %{client_audio_track: %{id: c_id, kind: :audio}} = state\n      ) do\n    PeerConnection.send_rtp(pc, state.serv_audio_track.id, packet)\n    {:noreply, state}\n  end\n\n  def handle_info(\n        {:ex_webrtc, pc, {:rtp, c_id, packet}},\n        %{client_video_track: %{id: c_id, kind: :video}} = state\n      ) do\n    PeerConnection.send_rtp(pc, state.serv_video_track.id, packet)\n    {:noreply, state}\n  end\nend\n```\n\n\u003c/details\u003e\n\u003cbr/\u003e\n\n### Example of ExWebRTC with two connected clients\n\nTwo clients A and B will connect to the server and will create their own PeerConnection on the server.\n\n```mermaid\nsequenceDiagram\n    participant A as Client A\n    participant PcA as PcA \u003cbr\u003e(instance A)\n    participant PcB as PcB \u003cbr\u003e(instance B)\n    participant B as Client B\n\n    note over PcA, PcB: WebRTC Server\n\n    A-\u003e\u003ePcA: SPD/ICE\n\n    B-\u003e\u003ePcB: SDP/ICE\n\n    rect rgb(255, 248, 230)\n        A--\u003e\u003eB: A sends streams to PcA,  forwards them to PcB, and then to B\n\n        B--\u003e\u003eA: B sends streams to PcA,  forwards them to PcA, and then to A\n        note over A,B: Streaming Process\n    end\n```\n\n\u003cbr/\u003e\n\nIn a `handle_info(:rtp)`, for each type of track (video or audio), you must forward the packets received by a server PeerConnection process from his client to the other PeerConnection process.\n\nIn the handler below, the current `ExWebRTC.PeerConnection` will receive packets from his client (so the value of `pc_current` below will approximatively alternate between `pc1` and `pc2`, once both peers are connected to the Gateway.).\n\n```elixir\ndef handle_info({:ex_webrtc, pc_current, {:rtp, id, packet}}, state)\n```\n\nYou must look for the PID (say `pc2`) of the other `PeerConnection` process and forward the packets with `send_rtp`:\n\n```elixir\nPeerConnection.send_rtp(pc2, server_track_id, client_packet)\n```\n\nWhen the first peer connects, it produces a [keyframe](https://en.wikipedia.org/wiki/Intra-frame_coding), but there are no other peers, so the keyframe dropped. When the second peer connects, the first one does not know that it has to produce a new keyframe without using PLI, thus the long freeze. You must renew it with `send_pli`.\n\nWhen the second peer `pc2` is connected, then you tell `pc1` to:\n\n```elixir\nPeerConnection.send_pli(pc1, pc1.client_v_track_id)\n```\n\nThe dual streaming should now happen.\n\n### Statistics and getting transfer rates with getStats\n\nWe can count the size of each packet we receive in the Room callback event \"rtp\" with `byte_size(packet)`.\nWebRTC provides directly stats with the `peerConnection.getStats()` method.\n\n\u003e This data is also collected by the ExWebRTC dashboard.\n\n\u003e You can also visit the pages `chrome://webrtc-internals` for Chrome and `about:webrtc` for Firefox.\n\nWe can use it to display directly the transfer rate in the browser without keeping the server busy nor round trip.\n\n\u003cdetails\u003e\u003csummary\u003eJavascript snippet of the bitrate\u003c/summary\u003e\n\n```js\nlet init = 0,\n  timeInt = 2_000;\n\nasync function logPacketSizes() {\n  try {\n    const stats = await rtc.pc.getStats();\n    stats.forEach((report) =\u003e {\n      if (report.type === \"outbound-rtp\" \u0026\u0026 report.kind === \"video\") {\n        let bytesChange = report.bytesSent - init;\n        init = report.bytesSent;\n        let rate = Math.round((bytesChange * 8) / timeInt);\n\n        document.querySelector(\"#stats\").textContent =\n          \"Video transfer rate: \" + rate + \" kBps\";\n      }\n    });\n  } catch (error) {\n    console.error(\"Error getting stats:\", error);\n  }\n}\n\n// use it in the WebRTC event listener:\nfunction listenConnectionState() {\n  const state = rtc.pc.connectionState;\n  if (state === \"connected\") {\n    rtc.int = setInterval(logPacketSizes, timeInt);\n  }\n}\n```\n\n\u003c/details\u003e\n\u003cbr/\u003e\n\n### Details of the process supervision\n\nWe use a Lobby GenServer to start dynamically supervised Room processes when a user enter a given room.\nA Room process is a GenServer that starts a ExWebRTC PeerConnection with the client.\nThe client is connected via the LiveView for the HTML rendering, and the Channel (via the custom RoomSocket) for the signaling.\n\nEach peer will create his own `ExWebRTC.PeerConnection` process.\n\nWe `Process.monitor` the Channel process from the Room process. When a client leaves the page, this stops the channel. The dynamic GenServer will consequently stop.\nThe Lobby monitors the dynamic supervisor, so the lobby will update its state.\nIn the state, we track the pid, the room number, and number of connected peers.\n\n```elixir\n\n# RoomLive.ex, handle_event(\"goto\")\nLobby.create_room(room_id)\n#=\u003e\nDynamicSupervisor.start_child(DynSup,{RoomServer, [id: room_id]})\n\n# SignalingChannel.ex, join/3\n:connect = Room.connect(room_id, self())\n# Room.ex, connect/2\n{:ok, pc} = PeerConnection.start_link(ice_servers: @ice_servers)\n```\n\n```mermaid\ngraph TB\n    subgraph Process connection flow\n    Application -- start_child --\u003e L[Lobby]\n    LVM[LiveView\u003cbr\u003e mount] -- roomSocket --\u003e RS[RoomSocket]\n    LVN[LV \u003cbr\u003enavigate] -- Lobby.create_room\u003cbr\u003eroom_id --\u003e M[Room\u003cbr\u003eroom_id]\n    LVN-- roomSocket\u003cbr\u003echannel--\u003e  Ch[Channel]\n    Ch -- Room\u003cbr\u003econnect --\u003e M\n    end\n```\n\n## HLS with an Elixir server\n\n### What is HLS\n\n[Source](https://obsproject.com/forum/resources/how-to-do-hls-streaming-in-obs-open-broadcast-studio.945/)\n\nHLS stands for [HTTP Live Streaming](https://en.wikipedia.org/wiki/HTTP_Live_Streaming). The protocol is based on standard HTTP transactions.\nIt allows you to stream live on any website; the website does not require special streaming server software to be installed.\n\nAlhough one of the key feature is **adaptative bitrate streaming**, we don't develop this here but focus on getting it working.\n\nHLS was designed to enable big live sporting events to be streamed on content delivery networks, which only supported simple static file serving. It is also useful if you have a website on very simple cheap shared hosting and can't install a streaming server.\n\nHow does HLS work? The streamer breaks the video into lots of small segments, which are uploaded as separate files.\nIt also frequently updates a .m3u8 playlist file which contains information about the stream and the location of the last few segments. JavaScript in the viewer's web browser downloads the segments in turn and stitches them together to play back seemlessly. The web browser repeatedly downloads the .m3u8 file to discover new segments as they appear.\n\nHTTP Live Streaming can traverse any firewall or proxy server that lets through standard HTTP traffic, unlike UDP-based protocols such as RTP. This also allows content to be offered from conventional HTTP servers and delivered over widely available HTTP-based CND (content delivery network). You have high latency (several seconds).\n\n### The process\n\nYou have a producer of video streams and viewers of these streams. Both use the `video` HTMLElement of their browser.\nThe producer get streams from his webcam with `MediaDevices.getUserMedia`.\nThe streams are then trasnformed with `mediaRecorder` into a `Blob` of type **webm** (VP8 /VP9 encoding).\nSince we want to send data to the LiveView backend via the LiveSocket, we need to build Base 64 encoded strings.\nThe Bas64 codec uses the `FileReader` for this. This data-url is then `Phoenix.LiveView.push` to the backend.\nThis is a continous process with a time interval of 1s (arbitrary).\n\nThe backend receives the event with the data. It decodes from Base64 back and sends the binary to an FFmpeg process.\nThis OS process is launched with `Porcelain`. Since the browser emits data regularly, we feed the OS stdin with the data and FFmpeg receives them as a buffer.\nFFmpeg transcodes the data from (webm) VP8/VP9 **into H.264/H.265 (MPEG)**. It produces 2 type of files: a manifest which is an index of files, and segments which contains the video chunks. HLS will also create duplicates at different quality levels.\n\nThese files are kept on the filsystem and Phoenix will serve them as static files.\n\nThe incoming data chunks are managed with a **queue** (using Erlang's `:queue`). This provides a backpressure mechanism to prevent the FFmpeg buffer from being overwhelmed by possibly too many chunks.\n\nA viewer connects to the app. On connection, he loads the library `hls.js`. It will continuously look for updates of the manifest file and fetch the corresponding segments. These segments are the input of his `video` HTMLElement.\n\n```mermaid\ngraph TD\n    subgraph Browser/Producer\n      A0[video src]\n    end\n\n     A0 -- Base64 encoded data-url --\u003e B1\n\n    subgraph Elixir/WebServer\n        B1[Decode Base64 to binary]\n        B2[Webserver \u003cbr\u003e static files]\n        B1 -- spawn FFmpeg OS process--\u003e FFmpeg\n        B1 -- binary data to FFmpeg --\u003e B3\n        subgraph FFmpeg\n          B3[Buffer Transcoding \u003cbr\u003e vp8/h264]\n          B3 -- HLS segments \u003cbr\u003e update manifest --\u003e B4[filesystem]\n        end\n        B2 --\u003e B4\n        B4 --\u003e B2\n    end\n\n\n    subgraph Browser/Viewer\n        C1[Request manifest \u003cbr\u003e stream.m3u8]  --\u003e B2\n        C2[Request segment \u003cbr\u003e segment_001.ts] -- http://domain/stream.m3u8 \u003cbr\u003ehttp://domain/stream_001.ts --\u003e B2\n    end\n```\n\n```mermaid\nsequenceDiagram\n    participant Browser/Producer\n    participant Elixir Server\n    participant FFmpeg process\n    participant Browser/Viewer\n\n    Browser/Producer-\u003e\u003eBrowser/Producer: getUserMedia -\u003e streams\n    loop Every interval (e.g., 1000ms)\n        Browser/Producer-\u003e\u003eBrowser/Producer: MediaRecorder produces webm chunks\n        Browser/Producer-\u003e\u003eBrowser/Producer: FileReader encodes to Base64\n        Browser/Producer-\u003e\u003eElixir Server: Send Base64 encoded data-url\n    end\n    Elixir Server-\u003e\u003eElixir Server: Decode Base64 to binary\n    Elixir Server-\u003e\u003eFFmpeg process: spawn OS process\n    loop Continuous\n        Elixir Server-\u003e\u003eFFmpeg process: Send binary data\n        FFmpeg process -\u003e\u003eFFmpeg process: transcoding vp8/h264\n        FFmpeg process-\u003e\u003eElixir Server: Write HLS/DASH segments and manifest to filesystem\n    end\n    Browser/Viewer-\u003e\u003eElixir Server: Request manifest \u003cbr\u003estream.m3u8\n    Elixir Server-\u003e\u003eBrowser/Viewer: Serve manifest\n    loop Continuous\n        Browser/Viewer-\u003e\u003eElixir Server: Request segment \u003cbr\u003e segment_001.ts\n        Elixir Server-\u003e\u003eBrowser/Viewer: Serve segment\n    end\n\n```\n\n### FFmpeg commands\n\n[FFmpeg Hls doc](https://ffmpeg.org/ffmpeg-formats.html#hls-2)\n\n### FileWatcher on the manifest file\n\n[Playlist construction](https://developer.apple.com/documentation/http-live-streaming/video-on-demand-playlist-construction)\n\n```txt\n  #EXTM3U\n  #EXT-X-VERSION:3\n  #EXT-X-TARGETDURATION:8\n  #EXT-X-MEDIA-SEQUENCE:0\n  #EXT-X-PLAYLIST-TYPE:EVENT\n  #EXTINF:8.356544,\n  segment_000.ts\n  #EXTINF:8.356544,\n  segment_001.ts\n  #EXTINF:8.356544,\n  segment_002.ts\n  #EXTINF:0.467911,\n  segment_003.ts\n  #EXT-X-ENDLIST\n```\n\n- EXTM3U: this indicates that the file is an extended m3u file. Every HLS playlist must start with this tag.\n- EXT-X-VERSION: indicates the compatibility version of the Playlist file.\n- EXT-X-TARGETDURATION: this specifies the maximum duration of the media file in seconds.\n- EXT-X-MEDIA-SEQUENCE: indicates the sequence number of the first URL that appears in a playlist file. Each media file URL in a playlist has a unique integer sequence number. The sequence number of a URL is higher by 1 than the sequence number of the URL that preceded it. The media sequence numbers have no relation to the names of the files.\n- EXTINF: tag specifies the duration of a media segment. It should be followed by the URI of the associated media segment — this is mandatory. You should ensure that the EXTINF value is less than or equal to the actual duration of the media file that it is referring to\n\n### Proxy or CDN\n\nNaturally, we can opt to use a dedicated Webserver - Nginx, Apache or Caddy - instead of Phoenix to server these files.\n\nWwe can also use a CDN. Instead of saving files, we can use the output streams of Ffmpeg and send them to a CDN.\nOnce we get a 201 back, we can forward the URL to the client.\n\n## MPEG-DASH with an Elixir server\n\nThe process is totally similar to the HLS, except from the FFmpeg command and the Javascript library that handles the streams.\n\n## Basics on Channel and Presence\n\n### Refresher (or not) on Erlang queue\n\nWe use 2 times a `:queue`. Used [the doc](https://hexdocs.pm/elixir/1.17.2/erlang-libraries.html#the-queue-module).\nIn resume, it is a FIFO, with `:queue.new`, `:queue.in` and `:queue.out`.\n\n\u003cdetails\u003e\n\u003csummary\u003eExamples of \":queue\" commands \u003c/summary\u003e\n\n```elixir\niex(38)\u003e q = :queue.new()\n{[], []}\niex(33)\u003e q = :queue.in(\"a\", q)\n{[\"a\"], []}\niex(34)\u003e q = :queue.in(\"b\", q)\n{[\"b\"], [\"a\"]}\niex(35)\u003e q = :queue.in(\"c\", q)\n{[\"c\", \"b\"], [\"a\"]}\n\niex(36)\u003e {{:value, value3}, q} = :queue.out(q)\n{{:value, \"a\"}, {[\"c\"], [\"b\"]}}\niex(37)\u003e {{:value, value2}, q} = :queue.out(q)\n{{:value, \"b\"}, {[], [\"c\"]}}\niex(37)\u003e {{:value, value3}, q} = :queue.out(q)\n{{:value, \"c\"}, {[], []}}\n\niex(39)\u003e :queue.out(q)\n{:empty, {[], []}}\n```\n\n\u003c/details\u003e\n\u003cbr/\u003e\n\n### Refresher on Channels, Custom sockets, Presence\n\nWe include a step-by-step reminder on Channels and Presence if you don't use this every day.\n\n```mermaid\nsequenceDiagram\n  participant Channel\n  participant Browser\n\n  Channel -\u003e Browser: roomSocket(ws://)\n  Note right of Browser: client connects\n  Browser -\u003e\u003e Channel: channel.join()\n  Note left of Channel: Channel.join\n  Note right of Browser: WebRTC \u003cbr\u003eevent\n  Browser -\u003e\u003e Channel: channel.push\u003cbr\u003e(event, msg)\n  activate Channel\n  Note left of Channel: handle_in\u003cbr\u003e(event, msg)\n  Channel -\u003e\u003e Browser: broadcast_from\n  deactivate Channel\n  Note right of Browser: channel.on\u003cbr\u003e(event, msg)\n```\n\n### Custom WebSocket connection\n\nWe will generate a custom WebSocket connection named `RoomSocket` that will support all the Channel `SignalingChannel` processes that are appended to this WS when you enter a \"room\".\n\nWe name-space with \"/room\":\n\n```bash\nws://localhost:4000/room/websocket?user_token=XYZ...\n```\n\n#### Client-side\n\nThe primitives come from [PhoenixJS](https://hexdocs.pm/phoenix/js/index.html#phoenix). This package is imported into our app.\n\nWe create a client module \"roomSocket.js\" that exports a `roomSocket` object. We append a \"user_token\" to the query string. It will be created by the server and passed to Javascript as an assign.\n\n\u003cdetails\u003e\n  \u003csummary\u003e\"roomSocket.js\" \u003c/summary\u003e\n\n```js\n// /assets.js/roomSocket.js\nimport { Socket } from \"phoenix\";\n\nexport defaut new Socket(\"/room\", {\n  params: { user_token: window.userToken },\n});\n```\n\n\u003c/details\u003e\n\n\u003cbr/\u003e\n\nThe usage of the `window.userToken` is explained [below](#ws-security).\n\nTo instantiate the WS, import it into the main \"app.js\" file and invoque the `connect` method as below:\n\n```js\n// /assets/js/app.js\nimport roomSocket from \"./roomSocket.js\";\n[...]\nroomSocket.connect();\n```\n\n#### Server-side\n\nWe finish this WebSocket connection server-side with two files: the endpoint and the module `RtcWeb.RoomSocket` it references.\n\n\u003e The URI should match the one defined client-side.\n\n\u003cdetails\u003e\n  \u003csummary\u003eServer Endpoint of the WS \"room_socket\"\u003c/summary\u003e\n\n```elixir\n#/lib/rtc_web/endpoint.ex\nsocket \"/room\", RtcWeb.RoomSocket,\n  websocket: true,\n  longpoll: false\n```\n\n\u003c/details\u003e\n\u003cbr/\u003e\n\nand the server module declared above:.\n\n\u003cdetails\u003e\n\u003csummary\u003eRoomSocket module\u003c/summary\u003e\n\n```elixir\ndefmodule RtcWeb.RoomSocket do\n  use Phoenix.Socket\n\n  @impl true\n  def connect(%{\"user_token\" =\u003e user_token} = _params, socket, _connect_info) do\n    case Phoenix.Token.verify(WebRtcWeb.Endpoint, \"user token\", user_token) do\n      {:ok, _} -\u003e\n        {:ok, socket}\n\n      {:error, reason} -\u003e\n        {:error, %{reason: reason}}\n    end\n  end\n\n  @impl true\n  def id(_socket), do: nil\nend\n```\n\n\u003c/details\u003e\n\u003cbr/\u003e\n\n\u003e In the `connect` callback, we verify that the token is valid (we used `Phoenix.Token` to generate it). The next paragraph explains more about this.\n\n### WS Security\n\nWe follow the [documentation](https://hexdocs.pm/phoenix/channels.html#using-token-authentication).\n\n- We create the \"user_token\" to authenticate the custom WebSocket connection.\n  We use the built-in module `Phoenix.Token` for this.\n- We create it in the `Router.ex` module with a `Plug`.\n- We pass it to the assigns so it is available in \"root.html.heex\" or \"app.html.heex\".\n- We pass it as a script, and Javascript will append it to the `window` object: any Javascript code will access it.\n- We now can use the `window.userToken` when the browser initiates the WebSocket \"RoomSocket\" connection. We pass the \"user_token\" in the query string of the WebSocket conection.\n\n\u003cdetails\u003e\n  \u003csummary\u003eProtect WS \"socket\" with a \"user token\" in Router\u003c/summary\u003e\n\n```elixir\n# /lib/rtc_web/router.ex\n\npipeline :browser do\n  ...\n  plug :put_user_token\nend\n\ndef put_user_token(conn, _) do\n  # dummay user_id\n  user_id = System.unique_integer() |\u003e abs() |\u003e Integer.to_string()\n\n  user_token =\n    Phoenix.Token.sign(WebRtcWeb.Endpoint, \"user token\", user_id)\n\n  conn\n  |\u003e Plug.Conn.fetch_session()\n  |\u003e Plug.Conn.put_session(:user_id, user_id)\n  |\u003e Plug.Conn.assign(:user_token, user_token)\nend\n```\n\n\u003c/details\u003e\n\u003cbr/\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003ePass the \"user token\" to Javascript\u003c/summary\u003e\n\n```html\nlib/rtc_web/templates/layout/root.html.heex\n\u003cscript\u003e\n  window.userToken = \"\u003c%= assigns[:user_token] %\u003e\";\n\u003c/script\u003e\n```\n\n\u003c/details\u003e\n\u003cbr/\u003e\n\n\u003cbr/\u003e\n\nWhen we run the server, we check that our custom socket is connected.\n\n```bash\n[info] CONNECTED TO RtcWeb.RoomSocket in 488µs\n  Transport: :websocket\n  Serializer: Phoenix.Socket.V2.JSONSerializer\n  Parameters: %{\"user_token\" =\u003e \"SFMyNTY.g2gDYW5uBgCcg3OLjwFiAAFRgA.0DV24hmkHsyemH-roK3o87ZGVgNoSWuss4YPC9bg6m4\", \"vsn\" =\u003e \"2.0.0\"}\n```\n\n### Channel set up\n\nThe channels processes work with pattern matching. In the `RtcWeb.RoomSocket` module, we firstly declare the pattern(s) we use and the linked server module `RtcWeb.SignalingChannel`:\n\n\u003cdetails\u003e\n\u003csummary\u003eThe RoomSocket module\u003c/summary\u003e\n\n```elixir\ndefmodule RtcWeb.RoomSocket do\n  use Phoenix.Socket\n  channel \"room:*\", RtcWeb.SignalingChannel\n  ...\n```\n\n\u003c/details\u003e\n\u003cbr/\u003e\n\nThe Channel has two parts, client and server.\n\nOn the client-side, we will append a channel to our custom socket, and on the server-side, we create a new module `SignalingChannel`.\n\nWe create a Javascript module to instantiate the channels (file named \"signalingChannel.js\").\n\n\u003cdetails\u003e\n\u003csummary\u003eClient-side signaling channel\u003c/summary\u003e\n\n```js\n// /assets/signalingChannel.js\nimport roomSocket from \"./roomSocket\";\n\nfunction joinChannel(roomId) {\n  const channel = roomSocket.channel(\"room:\" + roomId, {});\n\n  channel\n    .join()\n    .receive(\"ok\", (roomId) =\u003e\n      console.log(`Joined successfully room:${roomId}`)\n    )\n    .receive(\"error\", (resp) =\u003e {\n      console.log(\"Unable to join\", resp);\n      window.location.href = \"/\";\n    });\n}\n\njoinChannel(\"lobby\");\n```\n\nWe import it into \"app.js\" to run this code.\n\n```js\n// apps.js\nimport \"./signalingChannel\";\n```\n\n\u003c/details\u003e\n\u003cbr/\u003e\n\nServer-side, the SignalingChannel module includes the `join` alter ego callback.\n\n\u003cdetails\u003e\n\u003csummary\u003eServer SignalingChannel module\u003c/summary\u003e\n\n```elixir\ndefmodule RtcWeb.SignalingChannel do\n  use RtcWeb, :channel\n\n  @impl true\n  def join(\"room:\" \u003c\u003e id, payload, socket) do\n    {:ok, socket}\n  end\n\n  def id(_), do: nil\nend\n```\n\n\u003c/details\u003e\n\u003cbr/\u003e\n\nWe can check and run `mix phx.server`.\nWe should get the message below in the terminal:\n\n```bash\n[info] JOINED room:lobby in 228µs\n  Parameters: %{}\n```\n\nand the message below in the console:\n\n```js\nJoined successfully\n```\n\n### Logs and local testing\n\n#### Server logs\n\nWe can display the server logs in the browser with `web_console_logger: true` enabled in the \"config/dev.exs\" file and when you append the JS snippet below in \"app.js\",\n\n```js\nwindow.addEventListener(\"phx:live_reload:attached\", ({ detail: reloader }) =\u003e {\n  reloader.enableServerLogs();\n});\n```\n\nYou will see:\n\n```js\n// console logs\nJoined successfully\n\n// server logs\n[info] CONNECTED TO RtcWeb.RoomSocket in 3ms  Transport: :websocket  Serializer: Phoenix.Socket.V2.JSONSerializer  Parameters: %{\"user_token\" =\u003e \"SFMyNTY.g2gDYgAAARBuBgBsfdSLjwFiAAFRgA.YaxhoOEx_sZvmEVMnbg54labKwydi7XJKpYJ8Ksl1s4\", \"vsn\" =\u003e \"2.0.0\"}\nroom_channels.js:8 Joined successfully\n\n[info] JOINED room:lobby in 88µs  Parameters: %{}\n```\n\n#### Testing on local network\n\nWe follow the [documention](https://hexdocs.pm/phoenix/using_ssl.html#ssl-in-development).\n\nExcept your localhost, WebRTC requires HTTPS.\nIn order to test with a device (your phone or another computer) connected to the same network (such as the WIFI), you need to provide an HTTPS endpoint.\nYou can use a _self-signed certificate_ that can be generated by running the following Mix task:\n\n```elixir\nmix phx.gen.cert\n```\n\nThis adds two files in the \"/priv\" folder.\n\nThen, change the \"/config/devs.exs\" script to:\n\n```elixir\n# /config/dev.exs\n\nconfig :rtc, RtcWeb.Endpoint,\n  # Binding to loopback ipv4 address prevents access from other machines.\n  # Change to `ip: {0, 0, 0, 0}` to allow access from other machines.\n  http: [ip: {0, 0, 0, 0}, port: 4000],\n               ^^^\n  ...,\n  # NEW: add SSL Support in devs mode for mobile\n  https: [\n    port: 4001,\n    cipher_suite: :strong,\n    keyfile: \"priv/cert/selfsigned_key.pem\",\n    certfile: \"priv/cert/selfsigned.pem\"\n  ]\n```\n\nYour server provides two endpoints, HTTP on port 4000, and HTTPS on port 4001. This is enough to run your tests.\nYou can also [ngrok](https://ngrok.com/) your HTTPS endpoint for remote testing.\n\n### LiveView navigation\n\nWe render the HTML via `LiveView`.\n\nAll our routes will be under the same `live_session`.\n\nEach route calls the module RoomLive. We append the \"live_action\" as an atom to each route.\nThis is passed into the socket assigns so we can handle different actions in the same Liveview and render the corresponding HTML.\n\n:heavy_exclamation_mark: For Presence to detect the change of location of a user, you cannot use `patch`but only `navigate`.\n\n\u003e Recall that you get the params in the first argument of the LiveView `mount/3` and in the `handle_params` callback, callback before a `handle_event` if any (for example when you submit a form).\n\n\u003cdetails\u003e\n  \u003csummary\u003eThe Router.ex module\u003c/summary\u003e\n\n```elixir\n# /lib/rtc_web/router.ex\nscope \"/\", RtcWeb do\n  pipe_through :browser\n\n  live_session :default do\n    live \"/\", RoomLive, :lobby\n    # room that uses ExWebRTC\n    live \"/ex/:room_id\", RoomLive, :room\n    # room that uses WebRTC\n    live \"/web/:room_id\", RoomLive, :web\n  end\nend\n```\n\n\u003c/details\u003e\n\u003cbr/\u003e\n\nWe used tabs, for the fun but also for the UI. It is _shamelessly borrowed_ from the excellent - because simple - solution from [Tracey Onim](h.ttps://medium.com/@traceyonim22/how-to-create-tabs-in-phoenix-liveview-caf960b7c517).\n\n[:arrow_up:](#rtc---a-demo-of-webrtc-with-elixir)\n\n### Presence\n\nSource: \u003chttps://hexdocs.pm/phoenix/presence.html#usage-with-liveview\u003e\n\n:bangbang: Use `navigate`.\n\n#### Set up\n\nWe firstly run the generator to generate a `RtcWeb.Presence` _client process_ that we start in the `Application.ex` module.\n\n```bash\nmix phx.gen.presence Presence\n```\n\n\u003cdetails\u003e\u003csummary\u003eStart and supervise the Presence process\u003c/summary\u003e\n\n```elixir\n# /lib/rtc/Application.ex\nchildren = [\n  ...\n  RtcWeb.Presence,\n  ...\n]\n```\n\n\u003c/details\u003e\n\u003cbr/\u003e\n\nWe track users per room with `Presence` as an [_Elixir client process_](https://hexdocs.pm/phoenix/Phoenix.Presence.html#module-using-elixir-as-a-presence-client), defined in the `Rtc.Presence` module.\n\nWhen a user connects to the app, he is (pre)registered with a unique _user_id_.\n\nOur Presence client module defines the following functions:\n\n- `track_user` : used to start the `user_id` in the LiveView `mount`,\n- `list_users`: the Presence process keeps the state and we access it with `Presence.list`. It outputs the list of users with meta-data (the room he attends),\n- the `init` and `fetch` and `handle_metas` callbacks. When `Presence` detects a change, the `handle_metas` callback runs.\n  This callback uses the `fetch` callback. We re-wrote the `fetch` callback to insert a mandatory `id` key since we are using streams. Note that you _need_ to add the `metas` key.\n\n```elixir\ndef fetch(_topic, presences) do\n  for {tracking_key, %{metas: metas}} \u003c- presences, into: %{} do\n    {tracking_key, %{metas: metas, id: tracking_key}}\n  end\nend\n```\n\nWe then broadcast a `:join` or/and `:leave` event.\n\n\u003cdetails\u003e\u003csummary\u003ePresence tracking module\u003c/summary\u003e\n\n```elixir\ndefmodule RtcWeb.Presence do\n  use Phoenix.Presence,\n    otp_app: :rtc,\n    pubsub_server: Rtc.PubSub\n\n  require Logger\n\n  def track_user(key, params) do\n    Logger.info(\"Track #{key} with params #{inspect(params)}\")\n    track(self(), \"proxy:users\", key, params)\n  end\n\n  def list_users do\n    RtcWeb.Presence.list( \"proxy:users\")\n    |\u003e Enum.map(fn {_room_id, presence} -\u003e presence end)\n  end\n\n  @doc \"\"\"\n  We overwrite the callback to add the mandatory \"id\" key.\n  We set its value to \"tracking_key\", which is the user_id\n  \"\"\"\n  @impl true\n  def fetch(_topic, presences) do\n    for {tracking_key, %{metas: metas}} \u003c- presences, into: %{} do\n      {tracking_key, %{metas: metas, id: tracking_key}}\n    end\n  end\n\n  @impl true\n  def init(_opts) do\n    Logger.info(\"Presence process: #{inspect(self())}\")\n    {:ok, %{pid: self()}}\n  end\n\n  @impl true\n  def handle_metas(topic, %{leaves: leaves, joins: joins}, _presences, state) do\n    for {_user_id, presence} \u003c- joins do\n      :ok =\n        Phoenix.PubSub.local_broadcast(\n          Rtc.PubSub,\n          topic,\n          {:join, presence}\n        )\n    end\n\n    for {_user_id, presence} \u003c- leaves do\n      :ok =\n        Phoenix.PubSub.local_broadcast(\n          Rtc.PubSub,\n          topic,\n          {:leave, presence}\n        )\n    end\n\n    {:ok, state}\n  end\nend\n```\n\n\u003c/details\u003e\n\u003cbr/\u003e\n\n#### Stream Presence\n\nWe use `streams` because their handling and rendering is easy.\nChanges in the users' list will be pushed into the DOM - like delivering ephemeral messages - and no state is kept in the socket in a delcarative way: `stream_insert` or `stream_delete` upon Presence changes.\n\n```mermaid\ngraph TB\n    subgraph Tracking\n    Application -- start_child --\u003e P[Presence process]\n    LVM[LiveView \u003cbr\u003emount] -- Presence.track\u003cbr\u003e:user_id --\u003e M[Presence \u003cbr\u003e handle_metas]\n\n    B[Presence \u003cbr\u003e handle_meta] -- PubSub :join, :leave\u003cbr\u003e stream insert, delete --\u003e LV[DOM \u003cbr\u003e update]\n    end\n```\n\nWe define a `stream` in the Liveview assigns and call the tracking in the `mount` callback.\n\n\u003cdetails\u003e\u003csummary\u003eMount with Presence and streams\u003c/summary\u003e\n\n```elixir\ndefmodule Rtc.RoomLive do\n\n  alias Rtc.Presence\n\n  def mount(_params, session, socket) do\n  user_id = session[\"user_id\"]\n    room_id = Map.get(params, \"room_id\", \"lobby\")\n    room = \"room:#{room_id}\"\n\n    socket =\n      socket\n      |\u003e stream(:presences, Presence.list_users())\n      |\u003e assign(%{\n        form: to_form(%{\"room_id\" =\u003e room_id}),\n        min: 1,\n        max: 20,\n        room_id: room_id,\n        user_id: user_id,\n        room: room,\n        id: socket.id\n      })\n\n    socket =\n      if connected?(socket) do\n        Logger.info(\"LV connected --------#{socket.id}\")\n        # we subscribe to a specific topic for the broadcasting of join \u0026 leave data\n        subscribe(\"proxy:users\")\n        # you need to use the key \":id\"\n        Presence.track_user(user_id, %{\n          id: room_id,\n          user_id: user_id\n        })\n      end\n\n    {:ok, socket}\n  end\n\n  end\nend\n```\n\n\u003c/details\u003e\n\u003cbr/\u003e\n\nThe Presence process sends a \"presence_diff\" event that we have to handle (although we don't use it here).\nHowever, we handle the broadcasted `:leave` and `:join` messages to update the stream accordingly.\n\n\u003cdetails\u003e\u003csummary\u003ePresence handlers\u003c/summary\u003e\n\n```elixir\n# mandatory callback from RoomChannel \"handle_metas\"\n@impl true\ndef handle_info(%{topic: \"proxy:users\", event: \"presence_diff\"}, socket) do\n  {:noreply, socket}\nend\n# PubSub callbacks\ndef handle_info({:join, user_data}, socket) do\n  {:noreply, stream_insert(socket, :presences, user_data)}\nend\n\ndef handle_info({:leave, user_data}, socket) do\n  {:noreply, stream_delete(socket, :presences, user_data)}\nend\n```\n\n\u003c/details\u003e\n\u003cbr/\u003e\n\nYou can test this. Open 2 tabs:\n\n```elixir\n\u003e iex -S mix phx.server\niex\u003e RtcWeb.Presence.list_users()\n[\n  %{\n    id: \"576460752303421752\",\n    metas: [\n      %{id: \"lobby\", user_id: \"576460752303421752\", phx_ref: \"F9Cnz01URefvugbk\"}\n    ]\n  },\n  %{\n    id: \"576460752303421976\",\n    metas: [\n      %{id: \"lobby\", user_id: \"576460752303421976\", phx_ref: \"F9CnpfAVzmTvugaE\"}\n    ]\n  }\n]\n```\n\nand navigate each tab to say a different room:\n\n```elixir\niex(2)\u003e RtcWeb.Presence.list_users\n[\n  %{\n    id: \"576460752303421752\",\n    metas: [\n      %{id: \"2\", user_id: \"576460752303421752\", phx_ref: \"F9Cn0eXljnXvugEl\"}\n    ]\n  },\n  %{\n    id: \"576460752303421976\",\n    metas: [\n      %{id: \"1\", user_id: \"576460752303421976\", phx_ref: \"F9CnpfAVzmTvugaE\"}\n    ]\n  }\n]\n```\n\nIt remains to render the users per room on the screen. We have to follow the rules by adding a `phx-udpate=\"stream\"` and use an `id` exactly on the dom element we will interact on.\nWe define a rendering component where the list of users in a room is presented in a table.\n\n\u003cdetails\u003e\u003csummary\u003eRender list users per room\u003c/summary\u003e\n\n```elixir\ndefmodule UsersInRoom do\n  use Phoenix.Component\n\n  attr :room, :string\n  attr :room_id, :integer\n  attr :streams, :any\n\n  def list(assigns) do\n    ~H\"\"\"\n    \u003ch2\u003eUsers in \u003c%= @room %\u003e\u003c/h2\u003e\n    \u003cbr /\u003e\n    \u003ctable\u003e\n      \u003ctbody phx-update=\"stream\" id=\"room\"\u003e\n        \u003ctr\n          :for={{dom_id, %{metas: [%{id: id, user_id: user_id}]} = _metas} \u003c- @streams.presences}\n          id={dom_id}\n        \u003e\n          \u003ctd :if={@room_id == id}\u003e\n            \u003c%= user_id %\u003e\n          \u003c/td\u003e\n        \u003c/tr\u003e\n      \u003c/tbody\u003e\n    \u003c/table\u003e\n    \"\"\"\n  end\nend\n```\n\n\u003c/details\u003e\n\u003cbr/\u003e\n\nand we declare this component in the `render` callbacks of our LiveView as:\n\n```elixir\n\u003cUsersInRoom.list streams={@streams} room={@room} room_id={@room_id} /\u003e\n```\n\n![lobby page with user on line](https://github.com/ndrean/RTC/blob/main/priv/static/images/lobby.png)\n\n#### A word on \"hooks\"\n\nWe use `LiveView`. The custom WebRTC Javascript code is encapsulated in a so-called \"hook\": it allows you to run custom Javascript code. The \"hook\" object has a complete lifecycle, such as `mounted` and `destroyed` for the \"beforeunload\" event. It is also equipped with LiveView primitives (cf [phoenix_live_view](https://www.npmjs.com/package/phoenix_live_view)).\n\nIt is linked to a DOM element - a DOM id is required - and called when this DOM element is rendered. In our case, this happens when we navigate to a given room page.\n\n\u003e In particular, we can use LiveView's primitives such as `pushEvent` and `handleEvent` to communicate with the LiveView (cf [doc](https://hexdocs.pm/phoenix_live_view/1.0.0-rc.0/Phoenix.LiveView.html#push_event/3)). It will use the `LiveSocket` to push messages into it so the RoomLive will receive them.\n\nThis is how we declare it:\n\n```elixir\ndef render(assigns) when assigns.live_action == :room do\n  ...\n  \u003csection id=\"room-view\" phx-hook=\"rtc\"\u003e\n           ^^              ^^\n```\n\nWe import the file \"RTC.js\" in the \"app.js\" module and append it to the `LiveSocket` to the `hooks` object. The key is the name declared in the HTML, and the value is the function name exported by the module. For example:\n\n```js\n// /assets/js/RTC.js\nconst RTC =  {\n  mounted(){\n    ...\n  },\n  destroyed(){\n    ...\n  }\n}\nexport default RTC;\n\n\n// /assets/js/app.js\nimport RTC from \"./RTC.js\"\n[...]\nlet liveSocket = new LiveSocket(\"/live\", Socket, {\n  longPollFallbackMs: 2500,\n  params: {_csrf_token: csrfToken},\n  hooks: {rtc: RTC}\n             ^^^\n})\n\nliveSocket.connect()\n```\n\n[:arrow_up:](#rtc---demo-of-elixir-and-webrtc)\n\n### FFmpeg commands\n\n- capture the webcam and output raw images at 30 fps\n-\n\n```\nffmpeg -f avfoundation -framerate 30 -pixel_format uyvy422   -pixel_format uyvy422  -i \"0\"  demo/test_%03d.jpg\n```\n\n```\nffmpeg -f avfoundation -framerate 30 -pixel_format uyvy422   -pixel_format uyvy422  -i \"0\"  -f image2pipe pipe:1\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fndrean%2Frtc-hls","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fndrean%2Frtc-hls","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fndrean%2Frtc-hls/lists"}