{"id":28455577,"url":"https://github.com/uddithmachiraju/aws-serverless-data-processing-pipeline","last_synced_at":"2026-05-15T23:03:01.492Z","repository":{"id":297473297,"uuid":"992716920","full_name":"uddithmachiraju/AWS-Serverless-Data-Processing-Pipeline","owner":"uddithmachiraju","description":"AWS Serverless Data Processing Pipeline","archived":false,"fork":false,"pushed_at":"2025-06-12T17:52:07.000Z","size":23,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-06-27T02:39:30.433Z","etag":null,"topics":["api-gateway","aws","aws-lambda","rest-api","s3"],"latest_commit_sha":null,"homepage":"","language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/uddithmachiraju.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-05-29T15:46:47.000Z","updated_at":"2025-06-12T17:52:10.000Z","dependencies_parsed_at":"2025-06-05T17:40:20.470Z","dependency_job_id":null,"html_url":"https://github.com/uddithmachiraju/AWS-Serverless-Data-Processing-Pipeline","commit_stats":null,"previous_names":["uddithmachiraju/aws-serverless-data-processing-pipeline"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/uddithmachiraju/AWS-Serverless-Data-Processing-Pipeline","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uddithmachiraju%2FAWS-Serverless-Data-Processing-Pipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uddithmachiraju%2FAWS-Serverless-Data-Processing-Pipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uddithmachiraju%2FAWS-Serverless-Data-Processing-Pipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uddithmachiraju%2FAWS-Serverless-Data-Processing-Pipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/uddithmachiraju","download_url":"https://codeload.github.com/uddithmachiraju/AWS-Serverless-Data-Processing-Pipeline/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uddithmachiraju%2FAWS-Serverless-Data-Processing-Pipeline/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266270393,"owners_count":23902733,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api-gateway","aws","aws-lambda","rest-api","s3"],"created_at":"2025-06-06T22:09:47.195Z","updated_at":"2026-05-15T23:02:56.471Z","avatar_url":"https://github.com/uddithmachiraju.png","language":"HCL","funding_links":[],"categories":[],"sub_categories":[],"readme":"In this project, I was going to build a **serverless Data Processing Pipeline** in AWS. I havn't yet thought of the full fledged process yet. But, the planned steps are;\n1. Take input data via the public REST API \n2. Stores the data in the S3 bucket\n3. Trigers a Lambda function when new data added (process the data)\n4. Adds the data to the DynamoDB\n\nProject Goal:\nUser submits data via REST API -\u003e stored in S3 -\u003e Triggers Lambda -\u003e Process data -\u003e saves to DynamoDB\n\nPhase 1:\n- Project Setup:\n    1. Create a github repo \n    2. Create a API Gateway URL with MOCK Integration as of now \n\nPhase 2:\n- S3 Integration\n\nPhase 3:\n- Lambda Processing\n\nPhase 4:\n- DynamoDB Storage\n\nPhase 5:\n- Deploy API to AWS\n    1. Use API Gateway to Integrate \n\nPhase 6:\n- CloudWatch Logs\n\n\nInstall AWS CLI:\n    curl \"https://s3.amazonaws.com/aws-cli/awscli-bundle.zip\" -o \"awscli-bundle.zip\"\n    unzip awscli-bundle.zip \n    sudo ./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws (if not working saying python: no file or directory - use: sudo ln -s /usr/bin/python3 /usr/bin/python) \n\nConfigure AWS:\n    Go to Security Credientials and create an access key, and run\n    1. aws configure and paste all the things \n\nInstall Terraform:\n    https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli (go throught the documentation steps)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fuddithmachiraju%2Faws-serverless-data-processing-pipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fuddithmachiraju%2Faws-serverless-data-processing-pipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fuddithmachiraju%2Faws-serverless-data-processing-pipeline/lists"}