https://github.com/cieslarmichal/transcoder
Video transcoder system based on microservices with RabbitMQ, Redis and FFmpeg.
https://github.com/cieslarmichal/transcoder
ffmpeg microservices rabbitmq redis s3 video-preview video-processing video-thumbnail video-transcoding
Last synced: about 1 year ago
JSON representation
Video transcoder system based on microservices with RabbitMQ, Redis and FFmpeg.
- Host: GitHub
- URL: https://github.com/cieslarmichal/transcoder
- Owner: cieslarmichal
- Created: 2024-07-27T20:49:16.000Z (almost 2 years ago)
- Default Branch: main
- Last Pushed: 2024-12-08T10:25:14.000Z (over 1 year ago)
- Last Synced: 2025-04-17T07:16:54.825Z (about 1 year ago)
- Topics: ffmpeg, microservices, rabbitmq, redis, s3, video-preview, video-processing, video-thumbnail, video-transcoding
- Language: TypeScript
- Homepage:
- Size: 26.6 MB
- Stars: 6
- Watchers: 1
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Video transcoder system
## Introduction
The video transcoder system is a distributed system that allows users to upload videos and transcode them to different formats.
One video will generate following artifacts:
- Original video
- Encoded videos in different formats with HLS playlist(1080p, 720p, 480p, 360p)
- Preview videos for each format (first 4 seconds)
- Master HLS playlist
- Thumbnails image - grid of images from the video taken every 10 seconds
## Technologies
- Node.js
- Typescript
- RabbitMQ
- Redis
- S3
- FFmpeg
## Transcoding process
1. User uploads a video
2. API service uploads the video to S3
3. Downloader service downloads the video from S3 to shared filesystem storage
4. Encoding director service decides the encoding profiles for the video
5. Encoder service encodes the video to the desired formats
6. Uploader service uploads the video artifacts to S3
7. Playlist sticher service creates a master playlist for the encoded videos
All services communicate with each other via RabbitMQ.
## Architecture
```mermaid
flowchart TB
API[API Service]:::service
DOWNLOADER[Downloader Service]:::service
ENCODING_DIRECTOR[Encoding Director Service]:::service
ENCODER[Encoder Service]:::service
UPLOADER[Uploader Service]:::service
PLAYLIST_STICHER[Playlist Sticher Service]:::service
USERS[Users]:::external
RABBITMQ[RabbitMQ]:::external
S3[S3 Storage]:::external
REDIS[Redis]:::db
USERS -->|send video| API
USERS -->|get progress| API
USERS -->|get artifacts| API
API -->|video upload| S3
API -->|done| RABBITMQ
API -->|get progress| REDIS
RABBITMQ -->|url to download| DOWNLOADER
DOWNLOADER -->|video download| S3
DOWNLOADER -->|done| RABBITMQ
RABBITMQ -->|video id| ENCODING_DIRECTOR
ENCODING_DIRECTOR -->|video encoding spec| RABBITMQ
RABBITMQ -->|video path with encoding spec| ENCODER
ENCODER -->|save progress| REDIS
ENCODER -->|done| RABBITMQ
RABBITMQ -->|encoded file path| UPLOADER
UPLOADER -->|file upload| S3
UPLOADER -->|done| RABBITMQ
RABBITMQ -->|encoding id| PLAYLIST_STICHER
PLAYLIST_STICHER -->|master playlist upload| S3
classDef db color:#fff,fill:#ff9655,stroke:#ffa764,stroke-width:2px;
classDef external color:#fff,fill:#9b84d0,stroke:#9676d7,stroke-width:2px;
classDef service color:#fff,fill:#3b5dae,stroke:#97a9d3,stroke-width:2px;
```
## RabbitMQ Architecture
```mermaid
flowchart LR
API[API Service]:::service
DOWNLOADER[Downloader Service]:::service
ENCODING_DIRECTOR[Encoding Director Service]:::service
ENCODER[Encoder Service]:::service
UPLOADER[Uploader Service]:::service
PLAYLIST_STICHER[Playlist Sticher Service]:::service
INGESTED_VIDEOS[ingested-videos queue]:::queue
DOWNLOADED_VIDEOS[downloaded-videos queue]:::queue
ENCODING_REQUESTS[encoding-requests queue]:::queue
ENCODED_VIDEOS[encoded-videos queue]:::queue
UPLOADED_ARTIFACTS[uploaded-artifacts queue]:::queue
EXCHANGE[transcoder exchange]:::exchange
API -->|video.ingested| EXCHANGE
EXCHANGE --> INGESTED_VIDEOS
INGESTED_VIDEOS --> DOWNLOADER
DOWNLOADER -->|video.downloaded| EXCHANGE
EXCHANGE --> DOWNLOADED_VIDEOS
DOWNLOADED_VIDEOS --> ENCODING_DIRECTOR
ENCODING_DIRECTOR -->|video.encoding.requested| EXCHANGE
EXCHANGE --> ENCODING_REQUESTS
ENCODING_REQUESTS --> ENCODER
ENCODER -->|video.encoded| EXCHANGE
EXCHANGE --> ENCODED_VIDEOS
ENCODED_VIDEOS --> UPLOADER
UPLOADER -->|video.artifact.uploaded| EXCHANGE
EXCHANGE --> UPLOADED_ARTIFACTS
UPLOADED_ARTIFACTS --> PLAYLIST_STICHER
classDef queue color:#fff,fill:#ff9655,stroke:#ffa764,stroke-width:2px;
classDef exchange color:#fff,fill:#9b84d0,stroke:#9676d7,stroke-width:2px;
classDef service color:#fff,fill:#3b5dae,stroke:#97a9d3,stroke-width:2px;
```
## Services
### API Service
- Accepts a video from a user by HTTP
- Uploads a video to S3
- Sends a message with video id and download to RabbitMQ
- Checks the encoding progress in Redis
- Checks the encoding artifacts in S3
### Downloader Service
- Consumes messages with video URL to download from RabbitMQ
- Downloads a video from S3 and saves it to the shared volume
- Sends a downloading done message to RabbitMQ
### Encoding Director Service
- Consumes messages with video id from RabbitMQ
- Decides the encoding profile for the video
- Sends encoding request messages to RabbitMQ (one for each encoding profile)
### Encoder Service
- Consumes messages with encoding format and video path from RabbitMQ
- Encodes a video to the desired format
- Saves the encoding progress to Redis
- Sends an encoding done message to RabbitMQ
### Uploader Service
- Consumes messages about downloading done from RabbitMQ
- Uploads encoding artifacts from shared volume to S3
- Sends an uploading done message to RabbitMQ
### Playlist Sticher Service
- Consumes messages with encoding id from RabbitMQ
- Creates a master playlist for the encoded videos
- Uploads the master playlist to S3