https://github.com/shiyis/aws-serverless-etl
This is a simple data pipeline simulation that I have gone through in learning AWS and GCP related concepts and could be deployed on aws.
https://github.com/shiyis/aws-serverless-etl
Last synced: 4 months ago
JSON representation
This is a simple data pipeline simulation that I have gone through in learning AWS and GCP related concepts and could be deployed on aws.
- Host: GitHub
- URL: https://github.com/shiyis/aws-serverless-etl
- Owner: shiyis
- Created: 2022-10-14T20:39:39.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2023-10-25T23:50:04.000Z (over 1 year ago)
- Last Synced: 2025-01-11T08:11:47.247Z (6 months ago)
- Language: Python
- Homepage:
- Size: 265 MB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# aws-sam-etl-cicd
This project contains source code and supporting files for a serverless application that you can deploy with the SAM CLI. It includes the following files and folders:
- functions - Code for the application's Lambda functions to poll data from twitter.
- statemachines - Definition for the state machine that orchestrates the data polling workflow.
- tests - Unit tests for the Lambda functions' application code.
- template.yaml - A template that defines the application's AWS resources.
(we are going to translate this into terraform)
This application creates a data polling workflow which runs on a pre-defined schedule (note that the schedule is disabled by default to avoid incurring charges).
## Tests
Tests are defined in the `tests` folder in this project. Use PIP to install the test dependencies and run tests.
```bash
sam-app$ pip install -r tests/requirements.txt --user
# unit test
sam-app$ python -m pytest tests/unit -v
# integration test, requiring deploying the stack first.
# Create the env variable AWS_SAM_STACK_NAME with the name of the stack we are testing
sam-app$ AWS_SAM_STACK_NAME= python -m pytest tests/integration -v
```