{"id":17601118,"url":"https://github.com/esam091/youtube-aws","last_synced_at":"2026-02-05T09:39:28.822Z","repository":{"id":258905745,"uuid":"866375409","full_name":"esam091/youtube-aws","owner":"esam091","description":"An AWS full-stack project implementing YouTube's video processing pipeline. Also showcases various usages of AWS services.","archived":false,"fork":false,"pushed_at":"2024-10-21T03:03:26.000Z","size":382,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-12T12:12:24.106Z","etag":null,"topics":["amplify-js","aws","aws-cognito","aws-dynamodb","aws-elastic-beanstalk","aws-lambda","aws-s3","aws-sqs","nextjs","typescript"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/esam091.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-10-02T06:29:04.000Z","updated_at":"2024-11-10T02:12:18.000Z","dependencies_parsed_at":"2024-10-21T06:26:19.870Z","dependency_job_id":null,"html_url":"https://github.com/esam091/youtube-aws","commit_stats":null,"previous_names":["esam091/youtube-aws"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/esam091/youtube-aws","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/esam091%2Fyoutube-aws","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/esam091%2Fyoutube-aws/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/esam091%2Fyoutube-aws/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/esam091%2Fyoutube-aws/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/esam091","download_url":"https://codeload.github.com/esam091/youtube-aws/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/esam091%2Fyoutube-aws/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29118622,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-05T05:31:32.482Z","status":"ssl_error","status_checked_at":"2026-02-05T05:31:29.075Z","response_time":65,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["amplify-js","aws","aws-cognito","aws-dynamodb","aws-elastic-beanstalk","aws-lambda","aws-s3","aws-sqs","nextjs","typescript"],"created_at":"2024-10-22T12:08:40.204Z","updated_at":"2026-02-05T09:39:28.805Z","avatar_url":"https://github.com/esam091.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"## About\n\nThis is a backend focused project to demonstrate:\n- Video processing pipeline for streaming services.\n- Various usage of AWS services.\n\n## Technologies\n\n### Backend\n\n![Amazon Web Services](https://img.shields.io/badge/Amazon_Web_Services-232F3E?logo=amazonwebservices\u0026logoColor=white\u0026style=for-the-badge) \n![Amazon S3](https://img.shields.io/badge/S3-569A31?logo=amazons3\u0026logoColor=white\u0026style=for-the-badge)\n![Amazon SQS](https://img.shields.io/badge/SQS-FF4F8B?logo=amazonsqs\u0026logoColor=white\u0026style=for-the-badge)\n![AWS Lambda](https://img.shields.io/badge/Lambda-FF9900?logo=awslambda\u0026logoColor=white\u0026style=for-the-badge)\n![Amazon DynamoDB](https://img.shields.io/badge/DynamoDB-4053D6?logo=amazondynamodb\u0026logoColor=white\u0026style=for-the-badge)\n![Amazon Cognito](https://img.shields.io/badge/Cognito-DD344C?logo=amazoncognito\u0026logoColor=white\u0026style=for-the-badge)\n![AWS Media Convert](https://img.shields.io/badge/Media_Convert-232F3E?logo=amazonwebservices\u0026logoColor=white\u0026style=for-the-badge) \n![AWS Amplify](https://img.shields.io/badge/Amplify-FF9900?logo=awsamplify\u0026logoColor=white\u0026style=for-the-badge) \n![Amazon CloudWatch](https://img.shields.io/badge/CloudWatch-FF4F8B?logo=amazoncloudwatch\u0026logoColor=white\u0026style=for-the-badge)\n![Elastic Beanstalk](https://img.shields.io/badge/Elastic_Beanstalk-232F3E?logo=amazonwebservices\u0026logoColor=white\u0026style=for-the-badge)\n![Terraform](https://img.shields.io/badge/Terraform-844FBA?logo=terraform\u0026logoColor=white\u0026style=for-the-badge) \n\n### Frontend\n\n![Next.js](https://img.shields.io/badge/Next.js-black?logo=nextdotjs\u0026style=for-the-badge) \n![React](https://img.shields.io/badge/React-61DAFB?logo=react\u0026logoColor=black\u0026style=for-the-badge)\n![TypeScript](https://img.shields.io/badge/TypeScript-3178C6?logo=typescript\u0026logoColor=white\u0026style=for-the-badge) \n![Tailwind CSS](https://img.shields.io/badge/Tailwind_CSS-06B6D4?logo=tailwindcss\u0026logoColor=white\u0026style=for-the-badge) \n\n## Highlights\n✅ Infrastructure as Code with Terraform. Deploy everything with a single command.  \n✅ Secure services with strict roles and policies  \n✅ Video processing with AWS Elemental MediaConvert to produce HLS playlist  \n✅ S3 to store raw videos and processed HLS playlist  \n✅ Event processing with AWS Lambda and SQS  \n✅ CloudWatch event to filter finished Media Convert jobs  \n✅ User management with AWS Cognito  \n✅ Database with DynamoDB  \n✅ Frontend with Next.js and React  \n\n## Non goals\nBeautiful and feature complete front end: This is a backend focused project.  \nHigh availability: I use minimal infrastructure to keep costs down.\n\n## Deploying to AWS\n\nYou'll need an AWS account with credit card to use Media Convert.\n\nPlease install the following tools:\n- [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html)\n- [cURL](https://curl.se/download.html)\n- [Terraform CLI](https://developer.hashicorp.com/terraform/install)\n\nMake sure you are signed in to [AWS CLI](https://docs.aws.amazon.com/signin/latest/userguide/command-line-sign-in.html).\n\nThen run the following command:\n```\nterraform init\nterraform apply\n```\n\n## Video processing pipeline explained\n\n![Architecture](./assets/architecture.png)\n\n### 1. Authentication\n\n![Authentication](./assets/1.png)\n\nEvery uploaded video must have an owner. This means users must sign in before uploading videos.\n\nI use Amplify's React UI to handle sign in and sign up, store the access token in cookie storage.\n\n### 2. Request a presigned POST URL\n\n![Request a presigned POST URL](./assets/2.png)\n\nAfter the user is authenticated, the frontend will request a presigned POST URL from the backend.\n\nThe backend will:\n1. Generate a new ULID which is going to be used as the object key and also the video ID in database.\n2. Generate a presigned POST URL with `x-amz-user-id` set to the owner of the video, and also limit the upload size to 100MB.\n3. Return the presigned POST URL and the video ID to the frontend.\n\n### 3. Upload the video and write to database\n\n![Upload the video and write to database](./assets/3.png)\n\nThe video is uploaded to the presigned POST URL.\n\nOnce the video is uploaded, the front end ask the back end to create a new video record in database with the specified video ID.\n\nThe backend verifies whether the `x-amz-user-id` of the object matches the owner of the video.\n\nThe new video record will have these fields:\n- title\n- description\n- status set to `processing`. Videos with this status will not show up in the home page.\n- createdAt\n- userId\n\n### 4. Generate an HLS playlist from uploaded video\n\n![Generate an HLS playlist from uploaded video](./assets/4.png)\n\nOnce the video is uploaded, an event will be sent to an SQS queue which is consumed by a Lambda function.\n\nThe Lambda function will:\n1. Use `ffprobe` to get the video resolution.\n2. Calculate the resolutions of the HLS playlist.\n- A 720p source video will produce 720p, 480p, 360p and 240p resolutions.\n- A 370p source video will produce 360p and 240p resolutions.\n3. Send a job to AWS Elemental MediaConvert to generate the HLS playlist with the calculated resolutions.\n\n#### Lambda layer\n\nIn order to use `ffprobe`, I need to include the ffprobe binary in a separate Lambda layer.\n\nThe Lambda layer is packaged in a separate [repository](https://github.com/esam091/ffprobe-lambda-layer) and can be downloaded easily with curl.\n\n### 5. Media Convert post processing\n\n![Media Convert post processing](./assets/5.png)\n\nWhen a successful Media Convert job has finished, the results will be stored in a separate HLS playlist bucket.\n\nThis bucket is publicly readable so no signed URL required to get the thumbnail and the HLS playlist.\n\nThe video's status needs to be updated to either `done` or `failed` by:\n1. Using CloudWatch event, filtered only for `COMPLETE` or `ERROR` events coming from the video upload bucket.\n2. SQS and Lambda will handle the filtered CloudWatch event, get the video id from the event, and update the video's status.\n3. Aside from that, the raw video of the finished job will be deleted.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fesam091%2Fyoutube-aws","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fesam091%2Fyoutube-aws","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fesam091%2Fyoutube-aws/lists"}