https://github.com/bhavanachitragar/data-pipeline-with-aws-dynamodb
This project uses AWS Lambda to fetch live weather data from an API and store it in DynamoDB. Changes in DynamoDB trigger another Lambda function to format and store the data in CSV files on Amazon S3. Snowflake then automatically ingests this data from S3 using Snowpipe for real-time analysis and visualization, ensuring efficient data processing
https://github.com/bhavanachitragar/data-pipeline-with-aws-dynamodb
dynamodb iam-role lambda-functions s3-bucket snowflake weather-api
Last synced: 7 months ago
JSON representation
This project uses AWS Lambda to fetch live weather data from an API and store it in DynamoDB. Changes in DynamoDB trigger another Lambda function to format and store the data in CSV files on Amazon S3. Snowflake then automatically ingests this data from S3 using Snowpipe for real-time analysis and visualization, ensuring efficient data processing
- Host: GitHub
- URL: https://github.com/bhavanachitragar/data-pipeline-with-aws-dynamodb
- Owner: bhavanachitragar
- Created: 2024-06-25T10:09:26.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2024-06-25T11:24:56.000Z (over 1 year ago)
- Last Synced: 2025-03-11T14:20:09.820Z (7 months ago)
- Topics: dynamodb, iam-role, lambda-functions, s3-bucket, snowflake, weather-api
- Language: Python
- Homepage:
- Size: 10.7 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Building a Real-Time Data Pipeline with AWS, DynamoDB, and Snowflake
-------------------------------------------------------------
This project builds a real-time data pipeline using AWS services and Snowflake. It starts with an AWS Lambda function that periodically fetches weather data from an API and stores it in a DynamoDB table. DynamoDB Streams capture changes and trigger another Lambda function to write the data to an S3 bucket in CSV format. Snowpipe in Snowflake automatically ingests the data from S3, making it available for analysis and visualization within Snowflake. This setup ensures continuous, automated data flow and real-time insights from the weather data.
#### AWS services used:
- AWS Lambda
- AWS DynamoDB
- S3
- IAM
### 1. AWS Lambda
In this project, one Lambda function fetches weather data from an external API and stores it in DynamoDB, while another processes DynamoDB Streams and writes data to an S3 bucket in CSV format.
### 2. Amazon DynamoDB
Amazon DynamoDB stores real-time weather data fetched by AWS Lambda. It uses DynamoDB Streams to capture changes in the data, triggering another Lambda function for further processing.
### 3. Amazon S3
Amazon S3 is used to store raw data files (in CSV format) from DynamoDB. These files are then ingested by Snowflake using Snowpipe for analysis.
### 4. AWS IAM
AWS IAM manages permissions and roles for AWS services. It ensures that Lambda functions have the necessary access to interact with DynamoDB and S3, maintaining secure and controlled access.
### 5. Snowflake
Snowflake is used to automatically ingest data from S3 via Snowpipe, making the data available for real-time analysis and reporting.
#### Sanpshots:
#### DynamoDB
### S3 Bucket
### Snowflake
### Output:
----------------------------------------------------------------
#### Credits: DateWithData