{"id":15046042,"url":"https://github.com/cieslarmichal/transcoder","last_synced_at":"2025-04-22T16:42:02.427Z","repository":{"id":250952281,"uuid":"834621915","full_name":"cieslarmichal/transcoder","owner":"cieslarmichal","description":"Video transcoder system based on microservices with RabbitMQ, Redis and FFmpeg.","archived":false,"fork":false,"pushed_at":"2024-12-08T10:25:14.000Z","size":27877,"stargazers_count":6,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-17T07:16:54.825Z","etag":null,"topics":["ffmpeg","microservices","rabbitmq","redis","s3","video-preview","video-processing","video-thumbnail","video-transcoding"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cieslarmichal.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-27T20:49:16.000Z","updated_at":"2024-12-14T07:08:36.000Z","dependencies_parsed_at":null,"dependency_job_id":"e1ad1009-e969-4ce2-b09f-0f2e11c27669","html_url":"https://github.com/cieslarmichal/transcoder","commit_stats":null,"previous_names":["cieslarmichal/transcoder"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cieslarmichal%2Ftranscoder","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cieslarmichal%2Ftranscoder/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cieslarmichal%2Ftranscoder/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cieslarmichal%2Ftranscoder/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cieslarmichal","download_url":"https://codeload.github.com/cieslarmichal/transcoder/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250278959,"owners_count":21404294,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ffmpeg","microservices","rabbitmq","redis","s3","video-preview","video-processing","video-thumbnail","video-transcoding"],"created_at":"2024-09-24T20:52:37.730Z","updated_at":"2025-04-22T16:42:02.406Z","avatar_url":"https://github.com/cieslarmichal.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Video transcoder system\n\n## Introduction\n\nThe video transcoder system is a distributed system that allows users to upload videos and transcode them to different formats.\nOne video will generate following artifacts:\n\n- Original video\n- Encoded videos in different formats with HLS playlist(1080p, 720p, 480p, 360p)\n- Preview videos for each format (first 4 seconds)\n- Master HLS playlist\n- Thumbnails image - grid of images from the video taken every 10 seconds\n\n## Technologies\n\n- Node.js\n- Typescript\n- RabbitMQ\n- Redis\n- S3\n- FFmpeg\n\n## Transcoding process\n\n1. User uploads a video\n2. API service uploads the video to S3\n3. Downloader service downloads the video from S3 to shared filesystem storage\n4. Encoding director service decides the encoding profiles for the video\n5. Encoder service encodes the video to the desired formats\n6. Uploader service uploads the video artifacts to S3\n7. Playlist sticher service creates a master playlist for the encoded videos\n\nAll services communicate with each other via RabbitMQ.\n\n## Architecture\n\n```mermaid\nflowchart TB\n  API[API Service]:::service\n  DOWNLOADER[Downloader Service]:::service\n  ENCODING_DIRECTOR[Encoding Director Service]:::service\n  ENCODER[Encoder Service]:::service\n  UPLOADER[Uploader Service]:::service\n  PLAYLIST_STICHER[Playlist Sticher Service]:::service\n\n  USERS[Users]:::external\n  RABBITMQ[RabbitMQ]:::external\n  S3[S3 Storage]:::external\n\n  REDIS[Redis]:::db\n\n  USERS --\u003e|send video| API\n  USERS --\u003e|get progress| API\n  USERS --\u003e|get artifacts| API\n  API --\u003e|video upload| S3\n  API --\u003e|done| RABBITMQ\n  API --\u003e|get progress| REDIS\n\n  RABBITMQ --\u003e|url to download| DOWNLOADER\n  DOWNLOADER --\u003e|video download| S3\n  DOWNLOADER --\u003e|done| RABBITMQ\n\n  RABBITMQ --\u003e|video id| ENCODING_DIRECTOR\n  ENCODING_DIRECTOR --\u003e|video encoding spec| RABBITMQ\n\n  RABBITMQ --\u003e|video path with encoding spec| ENCODER\n  ENCODER --\u003e|save progress| REDIS\n  ENCODER --\u003e|done| RABBITMQ\n\n  RABBITMQ --\u003e|encoded file path| UPLOADER\n  UPLOADER --\u003e|file upload| S3\n  UPLOADER --\u003e|done| RABBITMQ\n\n  RABBITMQ --\u003e|encoding id| PLAYLIST_STICHER\n  PLAYLIST_STICHER --\u003e|master playlist upload| S3\n\n  classDef db color:#fff,fill:#ff9655,stroke:#ffa764,stroke-width:2px;\n  classDef external color:#fff,fill:#9b84d0,stroke:#9676d7,stroke-width:2px;\n  classDef service color:#fff,fill:#3b5dae,stroke:#97a9d3,stroke-width:2px;\n```\n\n## RabbitMQ Architecture\n\n```mermaid\nflowchart LR\n  API[API Service]:::service\n  DOWNLOADER[Downloader Service]:::service\n  ENCODING_DIRECTOR[Encoding Director Service]:::service\n  ENCODER[Encoder Service]:::service\n  UPLOADER[Uploader Service]:::service\n  PLAYLIST_STICHER[Playlist Sticher Service]:::service\n\n  INGESTED_VIDEOS[ingested-videos queue]:::queue\n  DOWNLOADED_VIDEOS[downloaded-videos queue]:::queue\n  ENCODING_REQUESTS[encoding-requests queue]:::queue\n  ENCODED_VIDEOS[encoded-videos queue]:::queue\n  UPLOADED_ARTIFACTS[uploaded-artifacts queue]:::queue\n\n  EXCHANGE[transcoder exchange]:::exchange\n\n  API --\u003e|video.ingested| EXCHANGE\n  EXCHANGE --\u003e INGESTED_VIDEOS\n  INGESTED_VIDEOS --\u003e DOWNLOADER\n\n  DOWNLOADER --\u003e|video.downloaded| EXCHANGE\n  EXCHANGE --\u003e DOWNLOADED_VIDEOS\n  DOWNLOADED_VIDEOS --\u003e ENCODING_DIRECTOR\n\n  ENCODING_DIRECTOR --\u003e|video.encoding.requested| EXCHANGE\n  EXCHANGE --\u003e ENCODING_REQUESTS\n  ENCODING_REQUESTS --\u003e ENCODER\n  \n  ENCODER --\u003e|video.encoded| EXCHANGE\n  EXCHANGE --\u003e ENCODED_VIDEOS\n  ENCODED_VIDEOS --\u003e UPLOADER\n\n  UPLOADER --\u003e|video.artifact.uploaded| EXCHANGE\n  EXCHANGE --\u003e UPLOADED_ARTIFACTS\n  UPLOADED_ARTIFACTS --\u003e PLAYLIST_STICHER\n\n  classDef queue color:#fff,fill:#ff9655,stroke:#ffa764,stroke-width:2px;\n  classDef exchange color:#fff,fill:#9b84d0,stroke:#9676d7,stroke-width:2px;\n  classDef service color:#fff,fill:#3b5dae,stroke:#97a9d3,stroke-width:2px;\n```\n\n## Services\n\n### API Service\n\n- Accepts a video from a user by HTTP\n- Uploads a video to S3\n- Sends a message with video id and download to RabbitMQ\n- Checks the encoding progress in Redis\n- Checks the encoding artifacts in S3\n\n### Downloader Service\n\n- Consumes messages with video URL to download from RabbitMQ\n- Downloads a video from S3 and saves it to the shared volume\n- Sends a downloading done message to RabbitMQ\n\n### Encoding Director Service\n\n- Consumes messages with video id from RabbitMQ\n- Decides the encoding profile for the video\n- Sends encoding request messages to RabbitMQ (one for each encoding profile)\n\n### Encoder Service\n\n- Consumes messages with encoding format and video path from RabbitMQ\n- Encodes a video to the desired format\n- Saves the encoding progress to Redis\n- Sends an encoding done message to RabbitMQ\n\n### Uploader Service\n\n- Consumes messages about downloading done from RabbitMQ\n- Uploads encoding artifacts from shared volume to S3\n- Sends an uploading done message to RabbitMQ\n\n### Playlist Sticher Service\n\n- Consumes messages with encoding id from RabbitMQ\n- Creates a master playlist for the encoded videos\n- Uploads the master playlist to S3\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcieslarmichal%2Ftranscoder","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcieslarmichal%2Ftranscoder","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcieslarmichal%2Ftranscoder/lists"}