{"id":30769887,"url":"https://github.com/space-lumps/oroboro-dw-dbt","last_synced_at":"2025-09-04T23:04:23.683Z","repository":{"id":312874191,"uuid":"1049086891","full_name":"space-lumps/oroboro-dw-dbt","owner":"space-lumps","description":"This project builds a clean, BI‑ready table user_base in BigQuery","archived":false,"fork":false,"pushed_at":"2025-09-02T13:42:59.000Z","size":8,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-02T15:34:44.817Z","etag":null,"topics":["analytics-engineering","bigquery","business-intelligence","ci-cd","data-mart","data-modeling","dbt","elt","gcp","geospatial","jinja","metabase","sql","yaml"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/space-lumps.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-02T13:18:58.000Z","updated_at":"2025-09-02T14:08:58.000Z","dependencies_parsed_at":"2025-09-02T15:44:53.939Z","dependency_job_id":null,"html_url":"https://github.com/space-lumps/oroboro-dw-dbt","commit_stats":null,"previous_names":["space-lumps/oroboro-dw-dbt"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/space-lumps/oroboro-dw-dbt","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/space-lumps%2Foroboro-dw-dbt","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/space-lumps%2Foroboro-dw-dbt/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/space-lumps%2Foroboro-dw-dbt/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/space-lumps%2Foroboro-dw-dbt/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/space-lumps","download_url":"https://codeload.github.com/space-lumps/oroboro-dw-dbt/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/space-lumps%2Foroboro-dw-dbt/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273685604,"owners_count":25149722,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-04T02:00:08.968Z","response_time":61,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["analytics-engineering","bigquery","business-intelligence","ci-cd","data-mart","data-modeling","dbt","elt","gcp","geospatial","jinja","metabase","sql","yaml"],"created_at":"2025-09-04T23:02:33.683Z","updated_at":"2025-09-04T23:04:23.668Z","avatar_url":"https://github.com/space-lumps.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# user\\_base\\_project — dbt + BigQuery\n\n## Overview\n\nThis project builds a clean, BI‑ready table `user_base` in BigQuery by:\n\n* Moving complex SQL out of BI and into dbt layers for **consistency, speed, and testability**.\n* Building a robust **location dimension** from component tables and **deduping** per source location to avoid conflicting geographies.\n* Unifying user linkage across roles/routes (learners, educators, invite codes, sites/partners) and preserving users without links via left joins, producing a **clear grain** for dashboards.\n* Documenting lineage with dbt Docs and enforcing quality with dbt tests.\n\n## Warehouse \u0026 naming\n\n* **Project:** `oroboro-dw`\n* **Datasets:** `bronze_raw` (sources), `analytics_dev` (models)\n* **dbt source name:** `raw`\n\n## Model layers\n\n* **staging** (optional/omitted here for brevity): thin cleans of raw sources\n* **intermediate**: reusable transforms and joins\n\n  * `intermediate/locations_clean.sql` — one row per `from_location_id` with `city/county/state/country` + coordinates\n  * `intermediate/stacked_users_partners.sql` — one row per user + partner/site/classroom\n* **marts**: BI‑ready outputs\n\n  * `marts/user_base.sql` — final table queried by BI\n\n## Data flow\n\n```\nbronze_raw (raw sources)\n  ├─ user_user\n  ├─ location_* (components, types, base)\n  └─ (classroom, invites, partners, sites)\n\nintermediate views\n  ├─ locations_clean               -- unique per from_location_id\n  └─ stacked_users_partners        -- user ↔ partner/site/classroom\n\nmart\n  └─ user_base (table)             -- one row per user_id + partner_id + site_id\n```\n\n## Location dimension — why multi‑step\n\nA single `from_location_id` can map to several `to_location_id`s representing different types (city/locality, county, state, country). The pipeline:\n\n1. **Collects candidates** and tags by type.\n2. **Ranks** city candidates (e.g., distance checks, non‑street heuristics).\n3. **Picks one value per type** and emits a single, consistent row per source location.\n   This prevents partial/ambiguous geography from appearing in BI and supports stable filtering and grouping.\n\n## User linkage — how it’s unified\n\nUsers can be associated via multiple routes (classroom membership, invitations, partner codes, etc.). The pipeline:\n\n* Normalizes these routes into a **stacked** intermediate with a consistent set of fields.\n* Joins back to `user_user` and the location dimension in the final mart.\n* Preserves users with **no current link** (via left joins) so global user counts remain accurate.\n\n**Grain of `user_base`:** `(user_id, partner_id, site_id)`.\n\n## Materialization strategy\n\n* `intermediate` models: **views** (lightweight, always reflect raw)\n* `marts.user_base`: **table** (fast dashboards, predictable cost)\n\nChange to a view by editing `dbt_project.yml`:\n\n```yaml\nmodels:\n  user_base_project:\n    marts:\n      +materialized: view\n```\n\n## Tests\n\n`models/marts/marts_schema.yml` includes:\n\n* `not_null` on `user_id`\n* `dbt_utils.unique_combination_of_columns` on `(user_id, partner_id, site_id)`\n  Add staging tests (e.g., `unique`/`not_null` on natural keys) as needed.\n\n## Build \u0026 run\n\n```bash\npip install dbt-bigquery\n# Configure ~/.dbt/profiles.yml from profiles.example.yml\n\ndbt deps        # if using packages (dbt_utils)\ndbt debug\n# Build\ndbt run --select intermediate.locations_clean intermediate.stacked_users_partners marts.user_base\n# Test\ndbt test\n# Docs\ndbt docs generate \u0026\u0026 dbt docs serve\n```\n\n## Metabase usage\n\n* Connect Metabase to dataset `analytics_dev` and query `user_base`.\n* Keep Metabase **dynamic filter** queries using **fully‑qualified names** when Metabase injects SQL. Aliases **inside dbt models** are fine and don’t affect this.\n\n## Repo structure\n\n```\n.\n├─ dbt_project.yml\n├─ profiles.example.yml\n├─ packages.yml\n├─ macros/\n│  └─ utils.sql\n├─ models/\n│  ├─ sources.yml\n│  ├─ intermediate/\n│  │  ├─ locations_clean.sql\n│  │  └─ stacked_users_partners.sql\n│  └─ marts/\n│     ├─ user_base.sql\n│     └─ marts_schema.yml\n└─ README.md\n```\n\n## Compatibility\n\n* dbt Core ≥ 1.5\n* Adapter: `dbt-bigquery`\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspace-lumps%2Foroboro-dw-dbt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fspace-lumps%2Foroboro-dw-dbt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspace-lumps%2Foroboro-dw-dbt/lists"}