https://github.com/tinybirdco/dynamodbexporter
Open-source DynamoDB integration for Tinybird
https://github.com/tinybirdco/dynamodbexporter
Last synced: 9 months ago
JSON representation
Open-source DynamoDB integration for Tinybird
- Host: GitHub
- URL: https://github.com/tinybirdco/dynamodbexporter
- Owner: tinybirdco
- License: apache-2.0
- Created: 2023-10-27T11:46:12.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2024-07-24T15:24:54.000Z (almost 2 years ago)
- Last Synced: 2025-03-30T23:51:13.268Z (about 1 year ago)
- Language: Python
- Homepage:
- Size: 221 KB
- Stars: 6
- Watchers: 4
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: readme.md
- License: LICENSE
Awesome Lists containing this project
README
# Tinybird DynamoDB Export Utilities
This repository contains example solutions for replicating data out of DynamoDB into Tinybird.
Both solutions, scanToExport and DDBStreamCDC contain their own readme files on usage.
## DDBStreamCDC [Recommended]
This solution leverages AWS DynamoDB Export to S3 & AWS DynamoDB Streams functionalities via a Python Lambda to forward both snapshots and changes to a Tinybird Datasource with Keys automatically indexed, which is then deduplicated for the latest values and made available for users.
A simple implementation pathway is provided which may be expanded upon.
More detail in the dedicated [readme](DDBStreamCDC/readme.md)
## scanToExport [Legacy]
This solution uses a simple DynamoDB Scan to export a file to an S3 bucket, and then uses the Tinybird Datasource Replace functionality, implemented as a Python Lambda.
It is simpler in nature than the CDC solution, though it lacks the low latency updates and you may find a Scan is more expensive in practice.