{"id":51605042,"url":"https://github.com/victorcezeh/aws-event-driven-data-platform-v1-project","last_synced_at":"2026-07-12T01:30:21.877Z","repository":{"id":343160500,"uuid":"1136089001","full_name":"victorcezeh/aws-event-driven-data-platform-v1-project","owner":"victorcezeh","description":"A production-style serverless event-driven data platform that ingests semi-structured JSON data from an external API, transforms and validates it using AWS Lambda inside a private VPC, and loads curated datasets into a Redshift Serverless data warehouse. Fully provisioned with CloudFormation and deployed via GitHub Actions CI/CD.","archived":false,"fork":false,"pushed_at":"2026-07-01T05:50:17.000Z","size":24667,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-07-01T07:16:35.876Z","etag":null,"topics":["aws-cloudformation","aws-lambda","aws-redshift","aws-s3","aws-sdk-python","aws-vpc","aws-vpc-subnet","ci-cd","git","github-actions","python","yaml"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/victorcezeh.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-01-17T03:36:09.000Z","updated_at":"2026-07-01T05:43:39.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/victorcezeh/aws-event-driven-data-platform-v1-project","commit_stats":null,"previous_names":["victorcezeh/aws-data-pipeline-starter","victorcezeh/aws-event-driven-data-platform","victorcezeh/aws-event-driven-data-platform-project","victorcezeh/aws-event-driven-data-platform-project_v1"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/victorcezeh/aws-event-driven-data-platform-v1-project","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/victorcezeh%2Faws-event-driven-data-platform-v1-project","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/victorcezeh%2Faws-event-driven-data-platform-v1-project/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/victorcezeh%2Faws-event-driven-data-platform-v1-project/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/victorcezeh%2Faws-event-driven-data-platform-v1-project/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/victorcezeh","download_url":"https://codeload.github.com/victorcezeh/aws-event-driven-data-platform-v1-project/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/victorcezeh%2Faws-event-driven-data-platform-v1-project/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35379590,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-11T02:00:05.354Z","response_time":104,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws-cloudformation","aws-lambda","aws-redshift","aws-s3","aws-sdk-python","aws-vpc","aws-vpc-subnet","ci-cd","git","github-actions","python","yaml"],"created_at":"2026-07-12T01:30:21.365Z","updated_at":"2026-07-12T01:30:21.872Z","avatar_url":"https://github.com/victorcezeh.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# aws-event-driven-data-platform 🏗️\n\n\u003e 🚧 In progress - built alongside AWS SAA exam prep 😁\n\n## Context\n\nThis project started as a way to make my AWS SAA studying concrete. Instead of just reading about S3, VPC, IAM, and CloudFormation, I wanted to actually build something with them. Something that resembles a real production data platform.\n\nThe result is an event-driven pipeline that ingests raw JSON from an external API, processes it with AWS Lambda inside a private VPC, and loads the results into a Redshift Serverless data warehouse. Fully provisioned with CloudFormation and deployed via GitHub Actions CI/CD.\n\n## Infrastructure 🔧\n\nEach stack is purpose-built and independently deployable via `main_stack.yaml`.\n\n| Stack | What it does |\n|---|---|\n| `vpc.yaml` | 🌐 Private VPC, subnets, route tables, VPC endpoints |\n| `iam_role.yaml` | 🔐 Least-privilege IAM roles for Lambda and Redshift |\n| `s3.yaml` | 🪣 S3 bucket with versioning, HTTPS enforcement, event trigger |\n| `secrets_manager.yaml` | 🔑 Auto-generated Redshift credentials |\n| `lambda.yaml` | ⚡ Python 3.14 Lambda inside private VPC |\n| `redshift.yaml` | 🏭 Redshift Serverless in private subnet |\n| `monitoring.yaml` | 📊 CloudWatch alarms, log groups, SNS email alerts |\n| `main_stack.yaml` | 🎯 Nested stack orchestrator |\n\n## Project Structure\n\n```\naws-event-driven-data-platform-v1-project/\n├── .github/\n│   └── workflows/\n│       └── ci.yaml\n├── config/\n│   ├── __init__.py\n│   ├── settings.py\n│   └── logging_config.py\n├── data/\n│   ├── processed/\n│   │   └── .gitkeep\n│   └── raw/\n│       └── .gitkeep\n├── docs/\n│   └── ARCHITECTURE.md\n├── infrastructure/\n│   ├── iam_role.yaml\n│   ├── lambda.yaml\n│   ├── main_stack.yaml\n│   ├── monitoring.yaml\n│   ├── redshift.yaml\n│   ├── s3.yaml\n│   ├── secrets_manager.yaml\n│   └── vpc.yaml\n├── lambda/\n│   └── handler.py\n├── logs/\n│   └── pipeline.log\n├── src/\n│   ├── __init__.py\n│   ├── extract.py\n│   ├── load_redshift.py\n│   ├── load_s3.py\n│   ├── s3_event_reader.py\n│   └── transform.py\n├── tests/\n│   ├── test_extract.py\n│   ├── test_load_s3.py\n│   └── test_transform.py\n├── .env.example\n├── .gitignore\n├── LICENSE\n├── pipeline.py\n├── README.md\n└── requirements.txt\n```\n\nMore Ideas to add to the documentation:\n- Problem: What problem does the platform solve?\n- Architecture: Why is each component there?\n- Implementation: Show the code and infrastructure.\n- Trade-offs: Explain why these choices were made.\n- Future improvements: Show awareness of what could be better either on this project or another\n- Show debugging stories and failures in the repo.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvictorcezeh%2Faws-event-driven-data-platform-v1-project","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvictorcezeh%2Faws-event-driven-data-platform-v1-project","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvictorcezeh%2Faws-event-driven-data-platform-v1-project/lists"}