https://github.com/tuni56/serverless-etl
Terraform-ready, plug-and-play serverless ETL repo
https://github.com/tuni56/serverless-etl
aws etl-pipeline serverless terraform
Last synced: 29 days ago
JSON representation
Terraform-ready, plug-and-play serverless ETL repo
- Host: GitHub
- URL: https://github.com/tuni56/serverless-etl
- Owner: tuni56
- Created: 2025-09-05T22:29:55.000Z (10 months ago)
- Default Branch: main
- Last Pushed: 2025-09-05T23:09:36.000Z (10 months ago)
- Last Synced: 2025-09-06T01:10:29.435Z (10 months ago)
- Topics: aws, etl-pipeline, serverless, terraform
- Language: HCL
- Homepage: https://medium.com/@intzabai/serverless-at-scale-how-i-built-a-fully-automated-data-processing-workflow-on-aws-a7b4c7d0d63b
- Size: 10.7 KB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Serverless ETL — AWS Lambda + Step Functions + EventBridge (Terraform-ready)
> Serverless ETL pipeline designed to process millions of events per day.
> Orchestration with Step Functions. Ingestion via EventBridge. Transformations in AWS Lambda. Infrastructure as Code with Terraform.
---
## Overview
This repository contains a production-grade serverless ETL pipeline.
EventBridge ingests events from APIs, S3 uploads, and scheduled triggers.
Step Functions orchestrates a state machine that runs modular Lambdas for validation, enrichment, transformation, and loading.
The final curated dataset is stored in S3. All components are deployable with reusable Terraform modules.
---
## Problem
The business needed to:
- Handle highly variable workloads (50K events per hour up to 5M+ during peak hours).
- Operate without a dedicated DevOps team.
- Guarantee traceability, modularity, and strict SLAs on freshness and reliability.
- Avoid managing EC2, containers, or batch servers.
---
## Goals
1. Fully serverless architecture.
2. Multi-step orchestration with retries and timeouts.
3. Automatic scaling for millions of daily events.
4. Reproducible deployments with Terraform.
5. Observability and fault tolerance (DLQ, idempotency).
---
# Serverless ETL — AWS Lambda + Step Functions + EventBridge (Terraform-ready)
> Serverless ETL pipeline designed to process millions of events per day.
> Orchestration with Step Functions. Ingestion via EventBridge. Transformations in AWS Lambda. Infrastructure as Code with Terraform.
---
## Overview
This repository contains a production-grade serverless ETL pipeline.
EventBridge ingests events from APIs, S3 uploads, and scheduled triggers.
Step Functions orchestrates a state machine that runs modular Lambdas for validation, enrichment, transformation, and loading.
The final curated dataset is stored in S3. All components are deployable with reusable Terraform modules.
---
## Problem
The business needed to:
- Handle highly variable workloads (50K events per hour up to 5M+ during peak hours).
- Operate without a dedicated DevOps team.
- Guarantee traceability, modularity, and strict SLAs on freshness and reliability.
- Avoid managing EC2, containers, or batch servers.
---
## Goals
1. Fully serverless architecture.
2. Multi-step orchestration with retries and timeouts.
3. Automatic scaling for millions of daily events.
4. Reproducible deployments with Terraform.
5. Observability and fault tolerance (DLQ, idempotency).
---
## Architecture diagram
---
## Workflow
1. **Ingestion**
Events arrive from multiple sources into EventBridge. Scheduled rules (cron) also trigger flows.
2. **Trigger**
EventBridge starts the Step Functions state machine.
3. **Processing (Step Functions)**
- **Validate**: schema validation and idempotency checks (S3 flag).
- **Enrich**: fetches additional data from external APIs.
- **Transform**: normalizes timestamps, flattens payloads, prepares curated record.
- **Upload**: writes curated record to S3 and returns the key.
4. **Completion**
The state machine emits a completion event back to EventBridge for notifications or downstream triggers.
5. **Error handling**
Each step defines retries and timeouts. Final failures are routed to SQS DLQ. Lambdas have dead-letter configs.
---
## Key Design Decisions
- **Step Functions orchestration** for visual debugging and parallelism.
- **Parallel execution**: each event runs independently.
- **Retries & timeouts** defined per state in ASL.
- **Idempotency** using record hash flags in S3.
- **Decoupled Lambdas**: each function is modular and replaceable.
- **Terraform modules** for reusability (`lambda_fn`, `s3_bucket`, `eventbridge_rule`, etc.).
- **Packaging** with `archive_file` or ZIPs.
- **Observability** with structured logging and CloudWatch metrics.
- **Cost efficiency**: pay-per-use with minimal operational overhead.
---
## Results
- Scale: millions of records per day.
- Reliability: >99.99% successful executions.
- Latency: <15s end-to-end per record (depending on flow).
- Operations: near-zero infrastructure maintenance.
- Deployment: fully automated in minutes with Terraform.
---
## Stack
- AWS Lambda (Python 3.11)
- AWS Step Functions (ASL JSON)
- Amazon EventBridge (ingestion + completion)
- Amazon S3 (raw, curated)
- Amazon SQS (DLQ)
- CloudWatch (logs, metrics, alarms)
- Terraform (IaC modules)
- GitHub Actions (CI/CD recommended)
- Optional: OpenSearch for error triage, Lambda Powertools for logging/tracing
---
## Deployment (Quick Start)
1. Configure AWS credentials.
2. Update `env/dev.tfvars` with `project`, `region`, `env`.
3. Initialize Terraform: make init
4. Plan and apply: make plan make apply
5. Manual test:
Trigger Step Functions execution:
aws stepfunctions start-execution \
--state-machine-arn $STATE_MACHINE_ARN \
--input file://event.json
Check logs: aws logs filter-log-events --log-group-name /aws/lambda/ --limit 50