{"id":35266554,"url":"https://github.com/johnathon-smith/real-time-data-pipeline-aws-snowflake","last_synced_at":"2026-04-17T05:31:56.414Z","repository":{"id":330212537,"uuid":"1121973859","full_name":"johnathon-smith/real-time-data-pipeline-aws-snowflake","owner":"johnathon-smith","description":"Production-style real-time data ingestion pipeline built with AWS and Snowflake, featuring pre-stream validation, fault isolation, and automated Snowpipe ingestion.","archived":false,"fork":false,"pushed_at":"2025-12-24T01:09:30.000Z","size":1489,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-12-25T12:00:17.884Z","etag":null,"topics":["aws","cloud-architecture","data-engineering","kinesis","lambda","real-time-streaming","snowflake","snowpipe"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/johnathon-smith.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-12-23T22:07:28.000Z","updated_at":"2025-12-24T01:09:34.000Z","dependencies_parsed_at":"2025-12-25T12:00:19.252Z","dependency_job_id":null,"html_url":"https://github.com/johnathon-smith/real-time-data-pipeline-aws-snowflake","commit_stats":null,"previous_names":["johnathon-smith/real-time-data-pipeline-aws-snowflake"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/johnathon-smith/real-time-data-pipeline-aws-snowflake","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/johnathon-smith%2Freal-time-data-pipeline-aws-snowflake","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/johnathon-smith%2Freal-time-data-pipeline-aws-snowflake/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/johnathon-smith%2Freal-time-data-pipeline-aws-snowflake/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/johnathon-smith%2Freal-time-data-pipeline-aws-snowflake/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/johnathon-smith","download_url":"https://codeload.github.com/johnathon-smith/real-time-data-pipeline-aws-snowflake/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/johnathon-smith%2Freal-time-data-pipeline-aws-snowflake/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31916771,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-16T18:22:33.417Z","status":"online","status_checked_at":"2026-04-17T02:00:06.879Z","response_time":62,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","cloud-architecture","data-engineering","kinesis","lambda","real-time-streaming","snowflake","snowpipe"],"created_at":"2025-12-30T10:47:13.438Z","updated_at":"2026-04-17T05:31:56.408Z","avatar_url":"https://github.com/johnathon-smith.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Real-Time Data Streaming Pipeline (AWS → Snowflake)\n\nA production-style, real-time data ingestion pipeline built using AWS managed streaming services and Snowflake, with an emphasis on data validation, fault isolation, and automated ingestion.\n\n---\n\n## Table of Contents\n\n- [Overview](#overview)\n- [Architecture](#architecture)\n- [End-to-End Data Flow](#end-to-end-data-flow)\n- [Data Validation \u0026 Error Handling](#data-validation--error-handling)\n- [Streaming \u0026 Delivery Design](#streaming--delivery-design)\n- [Snowflake Ingestion (Snowpipe)](#snowflake-ingestion-snowpipe)\n- [Testing \u0026 Verification](#testing--verification)\n- [Security \u0026 IAM Considerations](#security--iam-considerations)\n- [Cost Management \u0026 Teardown](#cost-management--teardown)\n- [Future Improvements](#future-improvements)\n- [Key Takeaways](#key-takeaways)\n\n---\n\n## Overview\n\nThis project demonstrates the design and implementation of a **real-time data ingestion pipeline** using AWS and Snowflake.\n\nThe primary goals of the pipeline were to:\n\n- Ingest JSON data from an external API\n- Validate incoming data before it enters the streaming layer\n- Isolate malformed or invalid records\n- Deliver clean, validated data to Snowflake with minimal latency\n- Preserve raw data for downstream analytics and transformation\n\nRather than focusing only on moving data from point A to point B, this project emphasizes **production-grade concerns** such as data quality, failure isolation, automation, and cost control.\n\n---\n\n## Architecture\n\n![Pipeline Architecture Diagram](architecture/pipeline-architecture.png)\n\n**High-level architecture:**\n\nExternal Client  \n↓  \nAPI Gateway  \n↓  \nLambda (Validation)  \n├── Invalid Data → S3 Error Bucket  \n↓  \nKinesis Data Streams  \n↓  \nKinesis Data Firehose  \n↓  \nS3 Landing Bucket  \n↓  \nSnowpipe (Auto-Ingest)  \n↓  \nSnowflake Raw Table  \n\nThis design ensures that **only validated data** enters the streaming system and downstream analytics environment.\n\n---\n\n## End-to-End Data Flow\n\n### API Ingestion\n\n![API Gateway Stage](screenshots/01_ingestion/api-gateway-stage-and-endpoint.png)\n\n1. An external client sends a JSON payload via an HTTP POST request.\n2. **API Gateway** receives the request and forwards it to AWS Lambda using a proxy integration.\n3. **Lambda** parses and validates the payload:\n   - If the payload is malformed or missing required fields, it is written to an **S3 error bucket**.\n   - If valid, the record is published to **Kinesis Data Streams**.\n4. **Kinesis Data Firehose** buffers streaming records and writes JSON files to an S3 landing bucket.\n5. **Snowpipe** automatically detects new files and ingests them into a raw table in Snowflake.\n\n---\n\n## Data Validation \u0026 Error Handling\n\n### Lambda Validation \u0026 Routing\n\n![Lambda Overview](screenshots/02_validation/lambda-overview-and-role.png)\n![Lambda Execution Role](screenshots/02_validation/lambda-executor-role.png)\n\nData validation occurs **before** the data enters the streaming layer.\n\n### Validation Rules\n\n- Payload must be valid JSON\n- Required `id` field must be present and non-null\n\n### Failure Handling Strategy\n\n- Invalid or malformed records are written directly to a dedicated **S3 error bucket**\n- Invalid data is prevented from entering Kinesis, Firehose, and Snowflake\n\n### S3 Error Isolation\n\n![S3 Error Bucket](screenshots/04_storage/bad-data-in-error-bucket.png)\n\nThis approach prevents:\n- Stream pollution\n- Downstream ingestion failures\n- Corruption of analytics tables\n\n---\n\n## Streaming \u0026 Delivery Design\n\n### Kinesis Data Streams\n\n![Kinesis Data Stream](screenshots/03_streaming/kinesis-data-stream.png)\n\n- Serves as the real-time ingestion backbone\n- Default shard configuration used for simplicity\n\n### Kinesis Data Firehose\n\n![Firehose Overview](screenshots/03_streaming/firehose-overview.png)\n![Firehose Buffer Settings](screenshots/03_streaming/firehose-buffer-settings.png)\n\n**Configuration highlights:**  \n- Source: Kinesis Data Streams  \n- Destination: S3 landing bucket  \n- Buffer size: **1 MiB**  \n- Buffer interval: **5 seconds**  \n- Output format: JSON (`.json`)  \n\nThese settings prioritize **low-latency delivery** while still producing file sizes suitable for efficient downstream ingestion.\nThe buffer size and buffer interval were only so small for the sake of this project. In production, unless otherwise required by the business needs, I would increase both the buffer size and \nthe buffer interval in order to minimize the number of writes to S3 and increase the size of each file while also reducing the number of files overall.\nThis would result in more efficient operation, making the pipeline both faster and cheaper to run.\n\n---\n\n## Snowflake Ingestion (Snowpipe)\n\n### S3 Landing Bucket\n\n![S3 Landing Bucket](screenshots/04_storage/good-data-in-s3-data-bucket.png)\n\nValidated data is delivered to S3 and automatically ingested into Snowflake using Snowpipe.\n\n---\n\n### Snowpipe Auto-Ingest\n\n![Snowpipe SHOW PIPES](screenshots/05_snowflake/snowpipe-show-pipes.png)\n![Snowpipe Running](screenshots/05_snowflake/snowpipe-running.png)\n\nSnowpipe is configured with auto-ingest enabled, triggered by SQS notifications on new S3 object creation.\n\n---\n\n### Raw Table Verification\n\n![Snowflake Raw Table](screenshots/05_snowflake/raw-table-select.png)\n\nIncoming JSON data is stored in a raw table exactly as received, preserving full fidelity for downstream transformations.\n\n---\n\n## Testing \u0026 Verification\n\nTesting was performed using **Postman** to validate both success and failure scenarios.\n\n### Valid Data Test\n\n![Postman Valid Request](screenshots/06_testing/postman-valid-request.png)\n\n- Sent JSON payload containing all required fields\n- Verified:\n  - File creation in the S3 landing bucket\n  - Successful ingestion into the Snowflake raw table\n\n### Invalid Data Test\n\n![Postman Invalid Request](screenshots/06_testing/postman-invalid-request.png)\n\n- Sent JSON payload with a missing `id` field\n- Verified:\n  - Record written to the S3 error bucket\n  - No ingestion into Snowflake\n\nThese tests confirmed correct behavior for both **happy-path ingestion** and **error isolation**.\n\n---\n\n## Security \u0026 IAM Considerations\n\nIAM roles were created for:\n\n- API Gateway and Lambda execution\n- Snowflake access to S3 via a storage integration\n\nWhile permissions were intentionally broad during development, this project demonstrates:\n\n- Cross-account IAM trust relationships\n- Use of external IDs for Snowflake integrations\n- Separation of ingestion and error-handling resources\n\n---\n\n## Cost Management \u0026 Teardown\n\nAfter validating the pipeline end-to-end, all AWS resources were **intentionally shut down** to avoid unnecessary cloud costs.\n\nThe project is preserved through:\n- Architecture diagrams\n- Configuration artifacts\n- Screenshots of deployed resources\n\nThis reflects real-world engineering practices where **infrastructure lifecycle management** is a core responsibility.\n\n---\n\n## Future Improvements\n\n- Introduce a schema registry for stricter validation\n- Add dead-letter queues for error handling\n- Implement CloudWatch metrics and alerts\n- Define infrastructure using Terraform\n- Add CI/CD for automated deployment\n\n---\n\n## Key Takeaways\n\n- Data validation is most effective **before** streaming, not after\n- Isolating bad data simplifies downstream systems\n- Event-driven ingestion reduces operational complexity\n- Cost awareness is an essential part of data engineering\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjohnathon-smith%2Freal-time-data-pipeline-aws-snowflake","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjohnathon-smith%2Freal-time-data-pipeline-aws-snowflake","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjohnathon-smith%2Freal-time-data-pipeline-aws-snowflake/lists"}