{"id":19578228,"url":"https://github.com/sameersegal/video-concat","last_synced_at":"2025-10-28T11:07:01.831Z","repository":{"id":146095852,"uuid":"299829305","full_name":"sameersegal/video-concat","owner":"sameersegal","description":"Concatenate videos in Google Drive using FFMPEG and then places it back in Google Drive","archived":false,"fork":false,"pushed_at":"2021-06-24T10:19:12.000Z","size":478,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-02-26T11:46:42.476Z","etag":null,"topics":["ecs-fargate","google-drive-cli","lambda-functions","sqs-queue","terraform","video-processing"],"latest_commit_sha":null,"homepage":"","language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sameersegal.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-09-30T06:32:45.000Z","updated_at":"2021-06-24T10:19:15.000Z","dependencies_parsed_at":"2023-04-17T21:57:43.032Z","dependency_job_id":null,"html_url":"https://github.com/sameersegal/video-concat","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/sameersegal/video-concat","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sameersegal%2Fvideo-concat","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sameersegal%2Fvideo-concat/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sameersegal%2Fvideo-concat/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sameersegal%2Fvideo-concat/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sameersegal","download_url":"https://codeload.github.com/sameersegal/video-concat/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sameersegal%2Fvideo-concat/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":281425578,"owners_count":26499036,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-28T02:00:06.022Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ecs-fargate","google-drive-cli","lambda-functions","sqs-queue","terraform","video-processing"],"created_at":"2024-11-11T07:10:17.520Z","updated_at":"2025-10-28T11:07:01.823Z","avatar_url":"https://github.com/sameersegal.png","language":"HCL","readme":"# Video Concat\n\nWith every family video, we get more ambitious. It always is a frantic finish just before we need to play the video. \n\nThis project helps join/concatenate videos based on a sequence file. It get's triggered by API call and results in a final video published on Google Drive.\n\nVideo processing takes time (CPU intensive) and a lot of space. On Mac Air, you are severly constrained. That's why I picked AWS ECS with Fargate.\n\nI have used Terraform to quickly bring up the infrastructure and destroy when I am done. It's perfect for infrequent use like Family events. \n\nI assume you have an account with AWS and have a brief understanding of AWS, Terraform, Docker, etc\n\n## Get Started\n\n1. Ensure you have the pre-requisites:\n    1. AWS CLI [downloaded](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) and configured (`aws configure`) with significant previlige to create all the infrastructure. \n    2. Terraform [downloaded](https://www.terraform.io/downloads.html) and in path. \n    3. Docker [downloaded](https://www.docker.com/get-started). (On a Mac, `brew cask install docker` worked instead of the regular `brew install docker`)\n\n2. Generate a Service Account to access Google Drive\n    1. Follow the instructions [here](https://developers.google.com/identity/protocols/oauth2/service-account) to generate the json file and store it as `cotainer/credentials.json`\n    2. Remember to [enable](https://console.developers.google.com/apis/api/drive.googleapis.com/overview) Google Drive Apis for this service account.\n    3. On the `output` Google Drive folder, remember to add the email of the Service Account as a collaborator with edit previleges. \n\n3. Create the lambda function:\n    1. `npm install` in the lambda folder\n    2. zip and create the package\n        ```\n            zip -r ../trigger.zip main.js node_modules package.json\n        ```\n\n3. Build the infrastructure\n    1. The following commands have to be run in the `terraform` folder\n        ```\n            cd terraform\n        ```\n    1. Create a `my.tfvars` file with the following:\n        ```\n            ecr_name                   = \"ss-video-concate\"\n            ecs_cluster_name           = \"ss-video-cluster\"\n            ecs_service_name           = \"videoconcat-service\"\n            ecs_task_definition_family = \"VideoConcat\"\n            docker_image               = \"705594476693.dkr.ecr.ap-south-1.amazonaws.com/ss-video-concate\"\n            sqs_queue_name             = \"video-queue\"\n            api_path                   = \"video-concat\"\n        ```\n        Note: You may not have the docker_image just yet. Put in a dummy value (like above) and then update it after running the docker steps.\n\n    2. Run terraform\n        ```            \n            terraform init\n            terraform plan --var-file=\"my.tfvars\" -out=tfplan\n            terraform apply \"tfplan\"\n        ```\n\n    3. Note down the output variables from above\n        ```\n            base_url = \"https://XXXXXXXX.execute-api.ap-south-1.amazonaws.com/test\"\n            queue_url = \"https://sqs.ap-south-1.amazonaws.com/XXXXXXXXXXXX/video-queue\"\n        ```\n\n3. Create and publish the docker image\n    1. Create the docker image locally\n        ```\n            docker build -t video-concat .\n        ```\n\n    2. Follow the instructions on the ECR page to publish. It will substitute variables correctly.\n        ```\n            aws ecr get-login-password --region ap-south-1 | docker login --username AWS --password-stdin XXXXXXXXXX.dkr.ecr.ap-south-1.amazonaws.com\n            docker build -t ss-video-concate .\n            docker tag ss-video-concate:latest XXXXXXXXXX.dkr.ecr.ap-south-1.amazonaws.com/ss-video-concate:latest\n            docker push XXXXXXXXXX.dkr.ecr.ap-south-1.amazonaws.com/ss-video-concate:latest\n        ```\n    3. Copy the URI / ARN for the latest docker image and update the file `my.tfvars` created in step 3.2\n    \n4. And you are done! If you are stuck, keep repeating steps 4.1-4.3 to get the configuration right. \n\n5. Test your setup by creating a POST request to your API:\n\n    1. Find the input and output folder IDs. It's part that comes after the *folders* in a Google Drive link\n    ```\n        https://drive.google.com/drive/folders/XXXXXXXXXXXXXgtj9wZG3HIcb6b1dLLqg\n    ```\n    \n    2. Create a sequence file by mention each file name on a separate line. Remember that the sequence file should be a simple text file and not Google Docs or Doc or Docx or any other complex format. (We will be doing `cat sequence` in our script.)\n    ```\n        Opening.mp4    \n        Video 1.mp4\n        Video 2.mp4        \n        Credits.mp4        \n    ```\n    \n    3. Generate the video by calling the API:\n    ```\n        curl --location --request POST 'https://XXXXXXXX.execute-api.ap-south-1.amazonaws.com/test/video-concat/' \\\n            --header 'Content-Type: application/json' \\\n            --data-raw '{\n                \"input_folder\": \"XXXXXXXXXXTgtj9wZG3HIcb6b1dLLqg\",\n                \"output_folder\": \"XXXXXXXXXjjLvduXwQM4j5lPHY7_6z6a\",\n                \"sequence_file_name\": \"sequence\",\n                \"template_file_name\": \"template\",\n                \"output_file_prefix\": \"Short-Video-\",\n\t\t\"skip_download\":\"false\",\n\t\t\"delete_files\":\"template%20sequence\"\n            }'\n    ```\n    4. Check CloudWatch logs to see if there are any errors in Lambda or ECS. If not, you will see your video in the `output` folder\n\n6. When done you can destroy your infrastructure:\n    ```\n        terraform plan --var-file=\"my.tfvars\" -destroy -out=tfplan\n        terraform apply \"tfplan\"\n    ```\n\n### How does this work?\n\n![Architecture Diagram](./diagram.png)\n\n1. User's `POST` request is passed on to a Lambda Function via API Gateway\n2. The Lambda Function posts an event to SQS Queue and updates the ECS service's desired count to 1\n3. The ECS container reads from the queue in a while loop, until there are no messages.\n4. The ECS container downloads all the files in the `input` folder. It expects the sequence file to be present here.\n5. It uses `ffmpeg` to concatenate the videos using [ffmpeg filters](https://ffmpeg.org/ffmpeg-filters.html#concat)\n6. It then pushes the final video to the `output` folder\n7. It then deletes the message from the queue, and checks for another message\n\n### Example Sequence \u0026 Templates\n\n```\nBeginning Video with Music.mp4\nMom \\\u0026 Dad.mp4\nRelative1.mov\tName 1\tmandala2.png\t#032B60\tSN2s.jpg\nRelative2.MOV\tName 2\tmandala2.png\t#8F3A6F\tSN14s.jpg\nRelative3.mov\tName 3\tmandala1.png\t#70094A\tSN14s.jpg\nRelative4.mov\tName 4\tmandala2.png\t#3750A8\tSN13s.jpg\nRelative5.mp4\tName 5\tmandala3.png\t#A67761\tSN1s.jpg\nRelative6.mp4\tName 6\tmandala3.png\t#A67761\tSN1s.jpg\nEnding Video with Music.mp4\n\n```\n\n```\nif [[ -z \"$arg3\" ]];\nthen \n\n    ffmpeg -hide_banner -i \"$arg1\" -t 10 \\\n    -c:a aac -c:v libx264 -crf 23 \\\n    -filter_complex \"[0:v]scale=1920x1080:force_original_aspect_ratio=decrease,pad=1920:1080:0:0:color=black, \\\n    setdar=16/9,setsar=1/1,fps=fps=30,format=yuv420p; \\\n    [0:a]loudnorm=i=-24:tp=-2:lra=7\" \\\n    \"$i.ts\"\n\nelse\n\n    ffmpeg -hide_banner -i \"$arg1\" -t 10 -i $arg3 -i $arg5 \\\n    -filter_complex \"[0:v]scale=1600:900:force_original_aspect_ratio=decrease,pad=1920:1080:320:0:color=$arg4,setdar=16/9,setsar=1/1,fps=fps=30,format=yuv420p[V1]; \\\n    [V1][1:v]overlay=(overlay_w/2)*-1+40:main_h-(overlay_h/2)-40[V2]; \\\n    [V2][2:v]overlay=0:450-(overlay_h/2)[V3]; \\\n    [V3]drawtext=fontfile=./Sanchez-Regular.ttf: text='$arg2': fontcolor=white: fontsize=64: x=(w-text_w)/2: y=(h-90-(text_h/2)); \\\n    [0:a]loudnorm=i=-24:tp=-2:lra=7\" \\\n    -c:a aac -c:v libx264 -crf 23 \\\n    \"$i.ts\"\nfi\n```\nMore templates [here](templates/)\n\n### Further Reading\n\nI found the following tutorials and articles very helfpul while working on this project:\n\n1. [Better Together: Amazon ECS and AWS Lambda | AWS Compute Blog](https://aws.amazon.com/blogs/compute/better-together-amazon-ecs-and-aws-lambda/)\n2. [How to manage Terraform state. A guide to file layout, isolation, and… | by Yevgeniy Brikman | Gruntwork](https://blog.gruntwork.io/how-to-manage-terraform-state-28f5697e68fa)\n3. [Copy all files in a folder from Google Drive to AWS S3 (Example)](https://coderwall.com/p/rckamw/copy-all-files-in-a-folder-from-google-drive-to-aws-s3)\n4. [Docker Images : Part I - Reducing Image Size](https://www.ardanlabs.com/blog/2020/02/docker-images-part1-reducing-image-size.html)\n5. [Fargate as Batch Service. AWS Fargate can be a useful service for… | by Ava Chen | Aug, 2020 | Medium](https://medium.com/@avachen2005/fargate-as-batch-service-31a896ec1917)\n6. [Serverless Applications with AWS Lambda and API Gateway | Terraform - HashiCorp Learn](https://learn.hashicorp.com/tutorials/terraform/lambda-api-gateway)\n7. [Why does AWS Lambda need to pass ecsTaskExecutionRole to ECS task](https://serverfault.com/questions/945596/why-does-aws-lambda-need-to-pass-ecstaskexecutionrole-to-ecs-task)\n    * https://stackoverflow.com/questions/58686844/ecs-task-not-starting-stopped-cannotpullcontainererror-error-response-from\n    * https://medium.com/@paweldudzinski/creating-aws-ecs-cluster-of-ec2-instances-with-terraform-893c15d1116\n8. [EFS and ECS](https://aws.amazon.com/blogs/containers/developers-guide-to-using-amazon-efs-with-amazon-ecs-and-aws-fargate-part-1/)\n9. [FFMPEG documentation]()\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsameersegal%2Fvideo-concat","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsameersegal%2Fvideo-concat","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsameersegal%2Fvideo-concat/lists"}