https://github.com/harshcasper/serverless-streaming-data-pipeline

https://github.com/harshcasper/serverless-streaming-data-pipeline

Last synced: 3 days ago
JSON representation

• Host: GitHub
• URL: https://github.com/harshcasper/serverless-streaming-data-pipeline
• Owner: HarshCasper
• Created: 2024-03-14T11:08:43.000Z (10 months ago)
• Default Branch: main
• Last Pushed: 2024-12-15T00:55:02.000Z (19 days ago)
• Last Synced: 2024-12-15T01:32:36.537Z (18 days ago)
• Language: Python
• Size: 21.5 KB
• Stars: 0
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# LocalStack Streaming Data Pipeline Demo

A serverless ETL for demonstrating streaming structured analytics events from Lambda to Tinybird using CloudWatch and Kinesis.

## Requirements

* LocalStack Pro

* AWS [CDK](https://aws.amazon.com/cdk/)

* [cdklocal](https://github.com/localstack/aws-cdk-local)

* Python 3

## Running Locally

### Deploy Under LocalStack

1. Start LocalStack Pro: `LOCALSTACK_AUTH_TOKEN="your-auth-token" localstack start -d`

2. Install python dependencies: `make install`

3. Deploy locally: `make deploy-local`

### Testing

After following the local deployments steps above, you can run the sample integration test with `make test-integration`.

Alternatively, you can test end-to-end manually:

* Start the local Tinybird server mock with `make start-request-recorder`

* Emit an event by invoking the logger Lambda: `make invoke-test-logger-local`

* Observe the event payload arrive in the mock server output.