{"id":50994978,"url":"https://github.com/aayushkdev/rendition","last_synced_at":"2026-06-20T08:02:19.544Z","repository":{"id":362458165,"uuid":"1142462881","full_name":"aayushkdev/rendition","owner":"aayushkdev","description":"Distributed video transcoding system with FastAPI, PostgreSQL, RabbitMQ, and S3-compatible storage. Supports presigned uploads, worker-based ffmpeg HLS encoding, outbox-backed job   publishing, source-aware renditions, and local MinIO or production S3/R2 deployments.","archived":false,"fork":false,"pushed_at":"2026-06-04T08:35:54.000Z","size":364,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-04T10:13:16.894Z","etag":null,"topics":["distributed","ffmpeg","pipeline","rabbitmq","renditions","s3","streaming","transcoding","video-transcoding"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aayushkdev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-01-26T12:54:07.000Z","updated_at":"2026-06-04T08:52:17.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/aayushkdev/rendition","commit_stats":null,"previous_names":["aayushkdev/rendition"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/aayushkdev/rendition","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aayushkdev%2Frendition","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aayushkdev%2Frendition/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aayushkdev%2Frendition/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aayushkdev%2Frendition/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aayushkdev","download_url":"https://codeload.github.com/aayushkdev/rendition/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aayushkdev%2Frendition/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34561766,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-20T02:00:06.407Z","response_time":98,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["distributed","ffmpeg","pipeline","rabbitmq","renditions","s3","streaming","transcoding","video-transcoding"],"created_at":"2026-06-20T08:02:18.827Z","updated_at":"2026-06-20T08:02:19.538Z","avatar_url":"https://github.com/aayushkdev.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Rendition\n\nRendition is a distributed video transcoding system. It accepts large video\nuploads, stores the source video in S3-compatible object storage, queues encoding\njobs, generates HLS renditions with `ffmpeg`, and publishes a master playlist for\nplayback.\n\nThe local stack uses FastAPI, PostgreSQL, RabbitMQ, MinIO, a Python worker, an\noutbox publisher, and a Next.js frontend.\n\n## What It Does\n\n- Direct browser-to-object-storage multipart uploads using presigned URLs.\n- Upload validation for file size, content type, part count, part ordering, and\n  ETags.\n- Upload completion verification against object metadata before work is queued.\n- One encoding job per target rendition.\n- RabbitMQ job publishing through an outbox so jobs are not lost if RabbitMQ is\n  temporarily unavailable.\n- Worker-side job claiming with ownership, heartbeats, and manual ack/nack\n  behavior.\n- Stale job recovery through a reaper that returns abandoned running jobs to the\n  retry flow.\n- Bounded job retries with backoff and RabbitMQ dead-letter routing for terminal\n  worker failures.\n- Source probing with `ffprobe` for width, height, bitrate, and duration.\n- HLS encoding with `ffmpeg` for 1080p, 720p, 480p, 360p, 240p, and 144p\n  presets.\n- Source-aware rendition skipping. For example, a 720p source will not create a\n  1080p output.\n- HLS segment and rendition playlist upload to object storage.\n- Master playlist generation at `hls/{video_id}/master.m3u8`.\n- API-backed playback URLs so the frontend does not construct object-storage\n  URLs directly.\n- Request IDs and consistent API error responses.\n- A dashboard UI for uploads, progress, cancellation, retry, uploaded video\n  status, and HLS playback for completed videos.\n\n## Architecture\n\n```mermaid\nflowchart LR\n  frontend[Frontend] --\u003e api[API]\n  frontend --\u003e storage[(Object Storage)]\n  api --\u003e storage\n  api --\u003e postgres[(PostgreSQL)]\n  api --\u003e rabbitmq[(RabbitMQ)]\n  outbox[Outbox Publisher] --\u003e postgres\n  outbox --\u003e rabbitmq\n  rabbitmq --\u003e worker[Encoding Worker]\n  worker --\u003e postgres\n  worker --\u003e storage\n  reaper[Job Reaper] --\u003e postgres\n```\n\n### Services\n\n- `frontend`: Next.js app for upload testing and video/dashboard views.\n- `api`: FastAPI service exposing health checks, upload APIs, and video APIs.\n- `worker`: consumes encoding jobs and runs `ffprobe`/`ffmpeg`.\n- `reaper`: marks stale running jobs retryable when workers stop heartbeating.\n- `outbox`: periodically publishes pending queue messages from PostgreSQL to\n  RabbitMQ.\n- `postgres`: application database.\n- `rabbitmq`: job queue.\n- `minio`: local S3-compatible object storage for development.\n- `minio-init`: creates the private local bucket.\n\n## Upload And Encoding Flow\n\n1. The frontend asks the API for upload limits and allowed content types.\n2. The frontend starts an upload with filename, content type, size, and part\n   count.\n3. The API creates a `Video`, an `UploadSession`, and a multipart upload in\n   object storage.\n4. The frontend uploads each chunk directly to MinIO/S3 with presigned part URLs.\n5. The frontend completes the upload by sending part numbers and ETags to the\n   API.\n6. The API completes the multipart upload and verifies object size/content type.\n7. The API creates renditions, jobs, and outbox messages in the database.\n8. The API attempts immediate queue publishing; the outbox service retries any\n   pending messages every 30 seconds.\n9. The worker claims a pending job, records ownership, starts heartbeating, then\n   downloads the source video and probes it.\n10. If the requested rendition is not valid for the source, the rendition is\n    marked `skipped`.\n11. Valid renditions are encoded to HLS in a temporary per-job directory.\n12. Before uploading HLS output, the worker verifies it still owns the job so a\n    stale worker cannot overwrite files after another worker reclaimed the job.\n13. The worker uploads HLS segments and the rendition playlist.\n14. Failed jobs are retried with backoff until attempts are exhausted.\n15. The reaper moves stale running jobs back into the retry flow after the\n    heartbeat timeout.\n16. Once all rendition work is terminal, a master playlist is generated and\n    uploaded.\n17. `videos.playback_path` points at the master playlist.\n18. The frontend asks the API for playback metadata and uses the returned\n    playlist URL.\n\n## Local Development\n\nThe local setup runs with:\n\n- FastAPI API\n- Next.js frontend\n- PostgreSQL\n- RabbitMQ\n- MinIO\n- Python encoding worker\n- Job reaper\n- Outbox publisher\n\n\nCreate your local environment file:\n\n```bash\ncp .env.example .env\n```\n\nStart everything:\n\n```bash\ndocker compose up --build\n```\n\nApply migrations:\n\n```bash\nuv run alembic upgrade head\n```\n\nOpen:\n\n- Frontend: `http://localhost:3000`\n- API docs: `http://localhost:8000/docs`\n- MinIO console: `http://localhost:9001`\n- RabbitMQ console: `http://localhost:15672`\n\nDefault local credentials from `.env.example`:\n\n```text\nPostgres:  rendition / rendition\nRabbitMQ:  rendition / rendition\nObject storage: rendition / rendition-secret\nBucket:    rendition\n```\n\nUseful local worker defaults:\n\n```text\nWorker queue: jobs.encode\nWorker prefetch: 1\nWorker heartbeat interval: 60 seconds\nJob stale timeout: 300 seconds\nRetry backoff: 30, 120, 600 seconds\n```\n\n## Production Deployment\n\nUse `docker-compose.prod.yml` when object storage is provided externally by AWS\nS3, Cloudflare R2, or another S3-compatible provider.\n\nCreate a production environment file with working database, RabbitMQ, and\nstorage credentials, then start the stack with:\n\n```bash\ndocker compose -f docker-compose.prod.yml up --build\n```\n\nAt minimum, the production environment should provide:\n\n```text\nPOSTGRES_HOST=...\nPOSTGRES_PORT=5432\nPOSTGRES_DB=...\nPOSTGRES_USER=...\nPOSTGRES_PASSWORD=...\n\nRABBITMQ_DEFAULT_USER=...\nRABBITMQ_DEFAULT_PASS=...\n\nSTORAGE_ENDPOINT=...\nSTORAGE_PRESIGN_ENDPOINT=...\nSTORAGE_ACCESS_KEY_ID=...\nSTORAGE_SECRET_ACCESS_KEY=...\nSTORAGE_BUCKET=...\nSTORAGE_REGION=...\n\nWORKER_JOB_RETRY_COUNT=3\nWORKER_HEARTBEAT_INTERVAL_SECONDS=60\nJOB_REAPER_INTERVAL_SECONDS=120\nJOB_STALE_TIMEOUT_SECONDS=300\nJOB_RETRY_BACKOFF_SECONDS=30,120,600\n```\n\nTypical production flow:\n\n1. Create the object storage bucket ahead of time.\n2. Prepare the production environment file or exported environment variables.\n3. Start the stack with `docker compose -f docker-compose.prod.yml up --build -d`.\n4. Run database migrations with `uv run alembic upgrade head`.\n5. Verify the API health endpoint and frontend before sending traffic.\n\nUnlike the local stack, the production compose file does not provision MinIO or\ncreate a bucket for you. It assumes PostgreSQL, RabbitMQ, and your\nS3-compatible storage are already available and correctly configured.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faayushkdev%2Frendition","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faayushkdev%2Frendition","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faayushkdev%2Frendition/lists"}