https://github.com/kevinandersontech/ecommerce_data_pipeline
A batch data pipeline that generates synthetic e-commerce order data, validates it, loads into DuckDB, runs dbt models (staging, facts, marts), and exports daily metrics as CSV for downstream use.
https://github.com/kevinandersontech/ecommerce_data_pipeline
csv dag data-pipeline data-validation dbt duckdb etl python
Last synced: 6 months ago
JSON representation
A batch data pipeline that generates synthetic e-commerce order data, validates it, loads into DuckDB, runs dbt models (staging, facts, marts), and exports daily metrics as CSV for downstream use.
- Host: GitHub
- URL: https://github.com/kevinandersontech/ecommerce_data_pipeline
- Owner: kevinandersontech
- Created: 2025-09-17T04:22:56.000Z (7 months ago)
- Default Branch: main
- Last Pushed: 2025-09-17T04:34:59.000Z (7 months ago)
- Last Synced: 2025-09-17T06:25:46.720Z (7 months ago)
- Topics: csv, dag, data-pipeline, data-validation, dbt, duckdb, etl, python
- Language: Python
- Homepage:
- Size: 197 KB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# E-commerce Revenue Dashboard
A simple Streamlit app that connects to a DuckDB database created by the e-commerce data pipeline. It provides an interactive dashboard with daily revenue metrics, filters, and charts.
## Features
- Connects to a local DuckDB database
- Displays daily revenue metrics in a table and line chart
- Interactive filters for date range
- Easy one-command local run
## Prerequisites
This app expects a DuckDB database with a `mart_revenue_daily` table. By default, it looks for the file at `../ecommerce_data_pipeline/warehouse/ecom.duckdb`. You can change the path in `config.py` or set it with an environment variable.
## Quick Start
```bash
python3.12 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install -r requirements.txt
# Run the app
streamlit run app.py
```
Open the URL printed in your terminal (usually `http://localhost:8501`).
## Configuration
- Default database path: `../ecommerce_data_pipeline/warehouse/ecom.duckdb`
- Override with:
```bash
export DUCKDB_PATH=/absolute/path/to/ecom.duckdb
```
## Notes
The code is fully commented and written for clarity.