{"id":24016620,"url":"https://github.com/esadek/mini-mds","last_synced_at":"2025-04-15T14:14:36.631Z","repository":{"id":265850302,"uuid":"864677291","full_name":"esadek/mini-mds","owner":"esadek","description":"Lightweight, open source, locally-hosted Modern Data Stack","archived":false,"fork":false,"pushed_at":"2025-04-07T06:31:16.000Z","size":379,"stargazers_count":14,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-15T14:14:31.237Z","etag":null,"topics":["dash","dbt","dlt","duckdb","modern-data-stack","pandera","prefect"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/esadek.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-09-28T21:27:16.000Z","updated_at":"2025-04-15T07:43:44.000Z","dependencies_parsed_at":null,"dependency_job_id":"e15f6a8f-47e9-439d-8dc4-68b81abd5823","html_url":"https://github.com/esadek/mini-mds","commit_stats":null,"previous_names":["esadek/mini-mds"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/esadek%2Fmini-mds","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/esadek%2Fmini-mds/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/esadek%2Fmini-mds/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/esadek%2Fmini-mds/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/esadek","download_url":"https://codeload.github.com/esadek/mini-mds/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249085429,"owners_count":21210267,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dash","dbt","dlt","duckdb","modern-data-stack","pandera","prefect"],"created_at":"2025-01-08T08:52:07.360Z","updated_at":"2025-04-15T14:14:36.625Z","avatar_url":"https://github.com/esadek.png","language":"Python","readme":"# Mini MDS\n\n[![Build](https://img.shields.io/github/actions/workflow/status/esadek/mini-mds/ci.yml)](https://github.com/esadek/mini-mds/actions/workflows/ci.yml)\n[![Supported Python Versions](https://img.shields.io/badge/python-3.9_|_3.10_|_3.11_|_3.12_|_3.13-blue)](https://www.python.org/downloads/)\n[![License](https://img.shields.io/github/license/esadek/mini-mds)](https://github.com/esadek/mini-mds/blob/main/LICENSE)\n[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)\n\nLightweight, open source, locally-hosted Modern Data Stack\n\n- Extract and Load: [Polars](https://pola.rs/) and [dlt](https://dlthub.com/)\n- Data Quality: [Pandera](https://www.union.ai/pandera/)\n- Storage: [DuckDB](https://duckdb.org/)\n- Transformation: [dbt](https://www.getdbt.com/)\n- Orchestration: [Prefect](https://www.prefect.io/)\n- Visualization: [Dash](https://dash.plotly.com/)\n\n## Installation\n\nPrerequisites: Install [git](https://git-scm.com/) and [uv](https://docs.astral.sh/uv/).\n\nClone repository and change directory:\n\n```bash\ngit clone https://github.com/esadek/mini-mds.git\ncd mini-mds\n```\n\n## Usage\n\nExtract, validate, load and transform data:\n\n```bash\nuv run prefect/elt.py\n```\n\nVisualize data:\n\n```bash\nuv run dash/app.py\n```\n\n## Architecture\n\n```mermaid\nflowchart LR\n    A(CSV) --\u003e B[Polars]\n    subgraph Prefect\n        B --\u003e C[Pandera]\n        C --\u003e D[dlt]\n        E[dbt Core]\n    end\n    D --\u003e F[(DuckDB)]\n    E \u003c--\u003e F\n    F --\u003e G[Dash]\n```\n\n## Project Structure\n\n```\nmini-mds\n├── .github/                # GitHub workflows\n├── dash/                   # Dash application\n├── dbt/                    # dbt project\n├── duckdb/                 # DuckDB warehouse\n├── prefect/                # Prefect workflows\n├── .editorconfig           # Editor configuration\n├── .gitignore              # Untracked files to ignore\n├── .python-version         # Default Python version\n├── LICENSE                 # MIT license\n├── pyproject.toml          # Project metadata\n├── README.md               # Documentation\n└── uv.lock                 # Dependency lockfile\n```\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fesadek%2Fmini-mds","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fesadek%2Fmini-mds","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fesadek%2Fmini-mds/lists"}