https://github.com/moritzkoerber/local-data-stack
This project demos how to integrate DuckDB and Delta Lake with dbt, orchestrated by Dagster. All data is stored and processed locally.
https://github.com/moritzkoerber/local-data-stack
dagster dbt delta-lake delta-rs polars sql uv
Last synced: 2 months ago
JSON representation
This project demos how to integrate DuckDB and Delta Lake with dbt, orchestrated by Dagster. All data is stored and processed locally.
- Host: GitHub
- URL: https://github.com/moritzkoerber/local-data-stack
- Owner: moritzkoerber
- Created: 2025-04-27T13:52:29.000Z (11 months ago)
- Default Branch: master
- Last Pushed: 2025-12-12T18:30:57.000Z (4 months ago)
- Last Synced: 2025-12-14T09:18:31.305Z (4 months ago)
- Topics: dagster, dbt, delta-lake, delta-rs, polars, sql, uv
- Language: Python
- Homepage:
- Size: 862 KB
- Stars: 9
- Watchers: 1
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
- awesome-dagster - Dagster, dbt, Delta lake
README
# Local data stack
This project demos how to integrate DuckDB and Delta Lake with dbt, orchestrated by Dagster. All data is stored and processed locally/on-premise. The whole pipeline is partitioned by date range and scheduled to run daily. The stack queries stock exchange (Xetra) data through an API and upserts the result into a Delta table (= bronze layer). The table serves as a source for dbt, which does a layered incremental load into a DuckDB database: first into silver, then into gold. Finally, the gold table is queried with DuckDB to create a line chart in Plotly. Overview:
## How to run
The stack uses [uv](https://docs.astral.sh/uv/getting-started/installation/) to manage its dependencies. Simply run `uv run dg dev` to launch the dagster webserver.