https://github.com/localstack/serverless-streaming-data-pipeline
A serverless streaming data pipeline running on LocalStack
https://github.com/localstack/serverless-streaming-data-pipeline
aws cloudwatch kinesis lambda localstack realtime-analytics serverless tinybird
Last synced: 3 months ago
JSON representation
A serverless streaming data pipeline running on LocalStack
- Host: GitHub
- URL: https://github.com/localstack/serverless-streaming-data-pipeline
- Owner: localstack
- Created: 2022-04-05T20:54:44.000Z (about 3 years ago)
- Default Branch: main
- Last Pushed: 2022-05-13T16:24:36.000Z (about 3 years ago)
- Last Synced: 2025-03-25T11:52:29.144Z (3 months ago)
- Topics: aws, cloudwatch, kinesis, lambda, localstack, realtime-analytics, serverless, tinybird
- Language: Python
- Homepage: https://localstack.cloud/blog/2022-04-04-develop-and-test-data-analytics-pipelines-on-localstack/
- Size: 10.7 KB
- Stars: 8
- Watchers: 2
- Forks: 3
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# LocalStack Streaming Data Pipeline Demo
A serverless ETL for demonstrating streaming structured analytics events from Lambda to Tinybird using CloudWatch and Kinesis.
## Requirements
* LocalStack Pro
* AWS [CDK](https://aws.amazon.com/cdk/)
* [cdklocal](https://github.com/localstack/aws-cdk-local)
* Python 3
## Running Locally
### Deploy Under LocalStack
1. Start LocalStack Pro: `LOCALSTACK_API_KEY="your_key" localstack start -d`
2. Install python dependencies: `make install`
3. Deploy locally: `make deploy-local`
### Testing
After following the local deployments steps above, you can run the sample integration test with `make test-integration`.
Alternatively, you can test end-to-end manually:
* Start the local Tinybird server mock with `make start-request-recorder`
* Emit an event by invoking the logger Lambda: `make invoke-test-logger-local`
* Observe the event payload arrive in the mock server output.
## Further Reading
* [Develop and Test Real-time Data Pipelines with LocalStack](https://localstack.cloud/blog/2022-04-04-develop-and-test-data-analytics-pipelines-on-localstack/)