https://github.com/yai333/pythonserverlesssample
Build a serverless data pipeline with AWS S3, Lamba and DynamoDB
https://github.com/yai333/pythonserverlesssample
Last synced: 4 months ago
JSON representation
Build a serverless data pipeline with AWS S3, Lamba and DynamoDB
- Host: GitHub
- URL: https://github.com/yai333/pythonserverlesssample
- Owner: yai333
- Created: 2019-03-04T11:55:25.000Z (about 6 years ago)
- Default Branch: master
- Last Pushed: 2022-09-08T06:13:51.000Z (over 2 years ago)
- Last Synced: 2024-08-14T07:09:21.147Z (8 months ago)
- Language: Python
- Homepage:
- Size: 939 KB
- Stars: 6
- Watchers: 1
- Forks: 5
- Open Issues: 4
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
- jimsghstars - yai333/pythonserverlesssample - Build a serverless data pipeline with AWS S3, Lamba and DynamoDB (Python)
README
## A serverless data pipeline with AWS S3, Lamba and DynamoDB
https://medium.com/the-apps-team/build-a-serverless-data-pipeline-with-aws-s3-lamba-and-dynamodb-5ecb8c3ed23e
This is example of a serverless data pipeline using AWS Lambda Functions, S3 and DynamoDB!
### Prerequisites
- Serverless framework
- Python3.6
- Pandas
- docker
### How this pipeline works
On a daily basis, an external data source exports data of the pervious day in csv format to an S3 bucket. S3 event triggers an AWS Lambda Functions that do ETL process and save the data to DynamoDB.
### Deploying your service
```sls deploy```
### How to test
```$ aws s3 cp sample.csv s3://dev.document.files```