{"id":15063631,"url":"https://github.com/dwyl/hls-demo","last_synced_at":"2026-02-09T06:06:44.420Z","repository":{"id":247995822,"uuid":"827402636","full_name":"dwyl/HLS-demo","owner":"dwyl","description":"Minimal example of HTTP Live Streaming with face recognition","archived":false,"fork":false,"pushed_at":"2024-08-11T14:12:09.000Z","size":84,"stargazers_count":2,"open_issues_count":0,"forks_count":1,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-04-10T11:40:00.283Z","etag":null,"topics":["elixir","evision","haar-cascade-classifier","hls-live-streaming","livebook","streaming-server"],"latest_commit_sha":null,"homepage":"","language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dwyl.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-11T15:33:26.000Z","updated_at":"2024-08-11T16:38:52.000Z","dependencies_parsed_at":"2024-08-11T15:29:41.429Z","dependency_job_id":"42ba1722-2c70-4682-93e5-f0b533cdaafc","html_url":"https://github.com/dwyl/HLS-demo","commit_stats":null,"previous_names":["dwyl/hls-demo"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/dwyl/HLS-demo","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dwyl%2FHLS-demo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dwyl%2FHLS-demo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dwyl%2FHLS-demo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dwyl%2FHLS-demo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dwyl","download_url":"https://codeload.github.com/dwyl/HLS-demo/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dwyl%2FHLS-demo/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":270041239,"owners_count":24516799,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-12T02:00:09.011Z","response_time":80,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["elixir","evision","haar-cascade-classifier","hls-live-streaming","livebook","streaming-server"],"created_at":"2024-09-25T00:05:07.465Z","updated_at":"2026-02-09T06:06:44.253Z","avatar_url":"https://github.com/dwyl.png","language":"Elixir","funding_links":[],"categories":[],"sub_categories":[],"readme":"# HTTP Live Streaming (HLS) with Elixir and Livebook and face recognition\n\n## What?\n\nWe illustrate [HTTP Live Streaming](https://en.wikipedia.org/wiki/HTTP_Live_Streaming) using - \u003cmark\u003e[hls.js](https://github.com/video-dev/hls.js)\u003c/mark\u003e.\n\n\u003cimg width=\"311\" alt=\"Screenshot 2024-08-09 at 11 55 47\" src=\"https://github.com/user-attachments/assets/3d08f4cc-34b3-4cdd-9c0c-2ec503ecb093\"\u003e\n\nBecause HLS is based on HTTP, any ordinary web server can originate the stream.\n\n\n`HLS` is a streaming protocol developed by Apple to deliver media content over the internet using HTTP. It breaks the overall stream into a sequence of small HTTP-based file downloads, each download loading one short chunk of an overall potentially unbounded transport stream. It uses a (unique) \"playlist\" file that describes the \"segments\" files to be played. It uses a dedicated library Once these files are available for reading (in the browser), the library \u003cmark\u003e`hls.js`\u003c/mark\u003e will download the playlist and consequently segments to be played. It handles entirely the playback process. `Elixir` will serve these files.\n\n:exclamation: This protocole has **high latency**: you will experience up to 20 seconds delay.\n\n## How does HLS work?\n\n[Cloudfare source](https://www.cloudflare.com/learning/video/what-is-http-live-streaming/)\n\n**Server**: An HLS stream originates from a server where (in on-demand streaming) the media file is stored, or where (in live streaming) the stream is created. Because HLS is based on HTTP, any ordinary web server can originate the stream.\n\nTwo main processes take place on the server:\n\n- Encoding: The video data is reformatted so that any device can recognize and interpret the data. HLS must use H.264 or H.265 encoding.\n- Segmenting: The video is divided up into segments a few seconds in length. The length of the segments can vary, although the default length is 6 seconds (until 2016 it was 10 seconds).\n\n## What are we doing?\n\nOur job here is to:\n\n- capture the built-in webcam stream\n- **transform** the images server-side. We ran the \"hello world\" of computer vision, namely **face detection** with the `Haar Cascade model`. This is powered by [Evision](https://github.com/cocoa-xu/evision) (\u003cmark\u003e[OpenCV](https://docs.opencv.org/4.10.0/)\u003c/mark\u003e). The model is present by default in the source code of `Evision` and has a loader for it.\n- send the transformed images back to the browser. They are played back by the Javacript library `hls.js`. It is based on the [MediaSource API](https://developer.mozilla.org/en-US/docs/Web/API/MediaSource).\n\nThis relies heavily on \u003cmark\u003e[FFmpeg](https://ffmpeg.org/ffmpeg-formats.html#hls-1)\u003c/mark\u003e to get frames from the input video source and build HLS segments and the playlist.\n\n## How?\n\nWe propose two versions: a `Livebook` and an `Elixir/Plug` app.\n\nWhy? The Livebook is an easy and self contained app but the code is slighlty different from the web app from whom you may borrow the code.\n\n:exclamation: We need to have `FFmpeg` installed but also `fsevent` on MacOS or `inotify` for Linux on which depends `FlieSystem`.\n\n:exclamation: You might encounter sometimes the error \"segmentation fault\". No further explanation on this.\n\n### A livebook\n\nThis gives a nice introduction on how to use the amazing `Kino.JS.Live` module.\n\n[![Run in Livebook](https://livebook.dev/badge/v1/blue.svg)](https://livebook.dev/run?url=https%3A%2F%2Fgithub.com%2Fdwyl%2FHLS-demo%2Fblob%2Fmain%2Flib%2Fhls-demo.livemd)\n\n\u003e The default directory to which all the files are saved is your home directory. All the files will be saved in 3 folders: \"./priv/input\", \"./priv/output\" and \"./priv/hls\". **You need to clean these folders**.\n\n### A web app\n\nTo run the web app, you fork the repo:\n\n```elixir\nopen http://localhost:4000 \u0026\u0026 mix run --no-halt\n```\n\nThe web app is minimal in the sense that it is is a `Plug` app. \n\nWe run a tpc listener on port 4000 with `Bandit` to communicate with the browser.\n\nWe use a _raw WebSocket_ in the browser. The backend uses the library [websock_adapter](https://github.com/phoenixframework/websock_adapter).\nWe use it to send binary data (an `ArryBuffer`) from the browser to the Elixir backend. Check [this blog](https://kobrakai.de/kolumne/bare-websockets).\nWe securized the WS connection with a CSRFToken.\n\nWe have a `Plug` router that:\n\n- serves the static files: the (unique) HTML page, the associated Javascript module and the HLS files,\n- handles the `WebSocket` connection.\n\nWe run `FFmpeg` as **\"kept alive\"** with `ExCmd.Process`. This is crucial for the process.\n\nWe run a **file watcher process** with `file_system`. It will detect when `FFmpeg` will have built the HLS playlist and segments.\n\n## Telemetry\n\nThe process is able to handle \"small\" frames (640x480) at 30 fps. We use approx 300 I/O per second.\nThe memory seems stable around 70MB during a 20 min test.\n\n![Screenshot 2024-07-11 at 13 27 44](https://github.com/dwyl/HLS-demo/assets/6793008/2227ea2c-2b51-4171-ab4a-414734134955)\n\n![Screenshot 2024-07-11 at 13 29 26](https://github.com/dwyl/HLS-demo/assets/6793008/4ade5976-4687-4d6c-8b0d-007547e4500e)\n\n## Process flow\n\nThe browser will ask to control your webcam.\n\nOnce you click on \"start\", a WebSocket connection is instantiated with the backend.\nThe browser will produce video chunks and send them to the server.\nThe server will extract all the frames, save them into files, pass the file to the Evision process for face detection.\nThe frames will be glued altogether (respecting the order) to produce video chunks ready for the browser to consume them.\nThey is a file watching process to detect when the playlist is ready, and we pass this message to the browser.\nThe HLS Javascript library will then ready the playlist and download on a regular basis the new segments it needs.\n\n```mermaid\ngraph TD\n    F --\u003e|mediaRecorder \u003cbr\u003esend video chunk| A\n    A -- WebSocketHandler.init --\u003e A\n    A[WebSocketHandler] --\u003e|ffmpeg_capture: \u003cbr\u003emake frames| B[WebSocketHandler]\n    B --\u003e|send: process_queue| C[WebSocketHandler]\n    C --\u003e|send: ffmpeg_rebuild\u003cbr\u003emake segments \u0026 playlist| E[WebSocketHandler]\n    E --\u003e|send: playlist_ready| F[Browser]\n\n    G[FileWatcher] --\u003e|send: playlist_created| E[WebSocketHandler]\n    F -- hls.js \u003cbr\u003e GET playlist / segments--\u003e H[Elixir / Plug]\n```\n\n### Notes on the code\n\n#### The \"web router\" module\n\nWe defined four routes.\n\nThe root \"/\" sends the HTML text. It is parsed to add a CSRFToken.\n\nThe \"/js/main.js\" route sends the Javascript file when the browser calls it.\n\nThe \"\"/hls/:file\" route sends the HLS segments when the browser calls them.\n\nThe \"/socket\" route upgrade to a WebSocket connection after the token validation (comparison between the token saved in the session and the one received in the query string).\n\n#### The \"controller\" module\n\nThis module serves the files. Since we want to pass a \"CSRFToken\" to the Javascript, we use `EEX.eval_file` to parse the \"index.html.heex\" text.\n\n```elixir\ndef serve_homepage(conn, %{csrf_token: token}) do\nhtml = EEx.eval_file(\"./priv/index.html.heex\", csrf_token: token)\nPlug.Conn.send_resp(conn, 200, html)\nend\n```\n\n#### The \"file watcher\" module\n\nIt essentially uses `FileSystem`. We declare which folder we want to monitor. Whenever a change in the file system occurs in this folder, an event is emitted. We exploit it to send to the caller process (the WebSocketHandler) a message.\nMore precisely, we monitor the creation of the \"playlist.m3u8\" file.\n\n```mermaid\ngraph LR\n    A[FileWatcher.init] --pid = FileSystem.start_link \u003cbr\u003edirs: priv/hls \u003cbr\u003e  --\u003e D[FileSystem.subscribe pid]\n\n    E[handle_info] -- :file_event \u003cbr\u003e .m3u8--\u003e G[send: playlist_created\u003cbr\u003e to WebSocketHandler]\n```\n\n#### The \"ffmpeg processor\" module\n\nIt runs `FFmpeg` as \"keep alive\" processes via `ExCmd`. This is crucial as we pipe the stdin to FFmpeg.\n\nWhen we receive video chunks in binary form via the WebSocket (approx 300KB, depending on the height/width of your video HTMLElement), we pass it to FFmpeg to extract all the frames, at 30 fps.\nEach frame is saved into a file (approx 10kB).\n\nThe other FFmpeg process is when we rebuild HLS video chunks from the transformed frames.\nFFmpeg will update a playlist and produce HLS segments. The FFmpeg process must not die in order not to append \"#EXT-X-ENDLIST\" at the end of the playlist.\n\nThe general form of a FFmpeg command is:\n\n```\nffmpeg [GeneralOptions] [InputFileOptions] -i input [OutputFileOptions] output\n```\n\n#### The \"image processor\" module\n\nWe use `Evision` to detect faces and draw rectangles around the region of interest found.\n\n:exclamation: When you run the code, you will see that the Haar Cascade models produces a lot of false positives.\n\n#### The \"websocket handler\" module\n\nA visualisation may perhaps help to understand.\n\n```mermaid\ngraph LR\n\n   E[handle_in: binary_msg] --\u003e F[FFmpeg process\u003cbr\u003e save frames into files]\n   F -- craete files \u003cbr\u003epriv/input--\u003e G[send: ffmpeg_process]\n\n   H[handle_info: ffmpeg_process] --read files \u003cbr\u003epriv/input--\u003e J[Enqueue new files]\n   J --\u003e K[send: process_queue]\n\n   L[handle_info: process_queue] --\u003e M[Process files with Evision\u003cbr\u003e\n       detect_and_draw_faces]\n   M -- create files\u003cbr\u003epriv/output--\u003e O[send: ffmpeg_rebuild \u003cbr\u003e when queue empty]\n\n   P[handle_info: ffmpeg_rebuild when chunk_id == 5] -- files in\u003cbr\u003epriv/output--\u003e Q[Pass files to FFmpeg segment process]\n   Q--\u003e R[files in priv/hls]\n\n   UU[FileWatcher] -- event\u003cbr\u003eplaylist created--\u003eU[handle_info: playlist_created, init: true]\n   U --\u003e V[send playlist_ready to browser]\n```\n\n##### Instantiate the WebSocket handler\n\nWhen the browser initiates a WebSocket connection, the backend will respond with:\n\n```mermaid\ngraph TD\n    A[WebSocketHandler.init] --\u003e B[FileWatcher]\n    A --\u003e C[FFmpegProcessor]\n    C --\u003e D[FFmpeg Capture Process]\n    C --\u003e E[FFmpeg Rebuild Process]\n    A --\u003e F[load_haar_cascade]\n    D --\u003e G[State]\n    E--\u003eG\n    F--\u003eG\n```\n\n### Javascript\n\nFor clarity, the Javascript module is separated into its own file. It is served by Elixir/Plug.\n\nWe load `hls.js` from a CDN:\n\n```html\n\u003cscript src=\"https://cdn.jsdelivr.net/npm/hls.js@latest\" defer\u003e\u003c/script\u003e\n```\n\nWe need to use `DOMContentLoaded`, so the \"main.js\" module starts with:\n\n```js\nwindow.addEventListener('DOMContentLoaded', ()=\u003e {...})\n```\n\nWe use the WebSocket API:\n\n```js\nlet socket = new WebSocket(\n  `ws://localhost:4000/socket?csrf_token=${csrfToken}`\n);\n```\n\nWe use the WebRTC API method `getUserMedia` to display the built-in into a `\u003cvideo\u003e` element:\n\n```js\nlet stream = await navigator.mediaDevices.getUserMedia({\n  video: { width: 640, height: 480 },\n  audio: false,\n});\n\nvideo1.srcObject = stream;\n```\n\nWe use the `MediaRecorder` API to record the streams and push chunks every 1000ms to the connected server via the WebSocket:\n\n```js\nlet mediaRecorder = new MediaRecorder(stream);\n\nmediaRecorder.ondataavailable = async ({ data }) =\u003e {\n  if (!isReady) return;\n  if (data.size \u003e 0) {\n    console.log(data.size);\n    const buffer = await data.arrayBuffer();\n    if (socket.readyState === WebSocket.OPEN) {\n      socket.send(buffer);\n    }\n  }\n};\nmediaRecorder.start(1_000);\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdwyl%2Fhls-demo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdwyl%2Fhls-demo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdwyl%2Fhls-demo/lists"}