{"id":51550192,"url":"https://github.com/scalefreecom/datavault4sqlmesh","last_synced_at":"2026-07-09T23:01:08.081Z","repository":{"id":369061093,"uuid":"1251202000","full_name":"ScalefreeCOM/datavault4sqlmesh","owner":"ScalefreeCOM","description":"A python package including loading patterns for Data Vault 2 automation with sqlmesh. Authored and maintained by Scalefree International GmbH.","archived":false,"fork":false,"pushed_at":"2026-07-03T11:03:43.000Z","size":148,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-07-03T13:07:56.498Z","etag":null,"topics":["automation","data-vault","data-vault-2","sqlmesh"],"latest_commit_sha":null,"homepage":"https://www.scalefree.com","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ScalefreeCOM.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-27T10:45:59.000Z","updated_at":"2026-07-03T11:03:46.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/ScalefreeCOM/datavault4sqlmesh","commit_stats":null,"previous_names":["scalefreecom/datavault4sqlmesh"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/ScalefreeCOM/datavault4sqlmesh","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ScalefreeCOM%2Fdatavault4sqlmesh","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ScalefreeCOM%2Fdatavault4sqlmesh/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ScalefreeCOM%2Fdatavault4sqlmesh/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ScalefreeCOM%2Fdatavault4sqlmesh/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ScalefreeCOM","download_url":"https://codeload.github.com/ScalefreeCOM/datavault4sqlmesh/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ScalefreeCOM%2Fdatavault4sqlmesh/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35314872,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-09T02:00:07.329Z","response_time":57,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automation","data-vault","data-vault-2","sqlmesh"],"created_at":"2026-07-09T23:01:06.224Z","updated_at":"2026-07-09T23:01:08.066Z","avatar_url":"https://github.com/ScalefreeCOM.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# datavault4sqlmesh\n\u003cimg width=\"1357\" height=\"300\" alt=\"DataVault4sqlmesh-Powered-by-Scalefree-White\" src=\"https://github.com/user-attachments/assets/5924cd48-3e71-4ccf-a8f9-e869661a0f3a\" /\u003e\n\n---\n`datavault4sqlmesh` is a Python library that brings Data Vault 2\nmodels to SQLMesh. It auto-derives the `columns={}` schema, \nsets the correct model kind (`INCREMENTAL_UNMANAGED` or `FULL`), and wires up the `execute` \nfunction — so each Data Vault entity is a single declarative call in your model file.\n\n## Installation\n\n```bash\npip install datavault4sqlmesh\n```\n\nRequires Python 3.9+ \u0026 `sqlmesh`.\n\n---\n\n## Getting Started\n\n### 1. Create `config.json` in your SQLMesh project root\n\n```json\n{\n  \"dialect\": \"snowflake\",\n  \"ldts_alias\": \"ldts\",\n  \"rsrc_alias\": \"rsrc\",\n  \"hash\": \"MD5\"\n}\n```\n\nSee [datavault4sqlglot configuration](https://github.com/ScalefreeCOM/datavault4sqlglot/blob/main/README.md#configuration). \n\n### 2. Load configuration once in `models/__init__.py`\n\n```python\n# models/__init__.py\nfrom datavault4sqlmesh.config import load_dv_config\n\nload_dv_config()  # reads config.json from the SQLMesh project root\n```\n\nAll model files in the package automatically inherit these settings.\n\n### 3. Write model files\n\n```python\n# models/stg_customer.py\nfrom datavault4sqlmesh import stage_model\n\nstage_model(\n    name=\"stage.stg_customer\",\n    source_table=\"customers\",\n    source_schema=\"raw\",\n    hashed_columns={\n        \"hk_customer_h\": [\"customer_id\"],\n        \"hd_customer_s\": {\n            \"is_hashdiff\": True,\n            \"columns\": [\"customer_name\", \"email\"],\n        },\n    },\n    column_overrides={\n        \"customer_id\":   \"VARCHAR\",\n        \"customer_name\": \"VARCHAR\",\n        \"email\":         \"VARCHAR\",\n    },\n)\n```\n\n```python\n# models/customer_h.py\nfrom datavault4sqlmesh import hub_model\n\nhub_model(\n    name=\"dv.customer_h\",\n    hashkey=\"hk_customer_h\",\n    business_keys=[\"customer_id\"],\n    source_schema=\"stage\",\n    source_table=\"stg_customer\",\n    rsrc_statics=[\"ERP/customers\"],\n)\n```\n\n```python\n# models/customer_0_s.py\nfrom datavault4sqlmesh import satellite_model\n\nsatellite_model(\n    name=\"dv.customer_0_s\",\n    parent_hash_key=\"hk_customer_h\",\n    hash_diff=\"hd_customer_s\",\n    payload=[\"customer_name\", \"email\"],\n    source_schema=\"stage\",\n    source_table=\"stg_customer\",\n)\n```\n\n```python\n# models/customer_1_s.py\nfrom datavault4sqlmesh import satellite_v1_model\n\nsatellite_v1_model(\n    name=\"dv.customer_1_s\",\n    parent_hash_key=\"hk_customer_h\",\n    hash_diff=\"hd_customer_s\",\n    payload=[\"customer_name\", \"email\"],\n    sat_v0_table=\"customer_0_s\",\n    sat_v0_schema=\"dv\",\n)\n```\n\n### 4. See a working example\n\nThe **[datavault4sqlmesh-demo](https://github.com/ScalefreeCOM/datavault4sqlmesh-demo)** repo is a\nminimal end-to-end project you can clone and run locally. It covers a customer-and-orders dataset\nthrough the full DV2.0 pipeline — staging, hubs, links, and all three satellite patterns (v0/v1/v2)\n— against a local PostgreSQL instance.\n\n```bash\ngit clone https://github.com/ScalefreeCOM/datavault4sqlmesh-demo\ncd datavault4sqlmesh-demo\npip install datavault4sqlmesh\nsqlmesh plan --auto-apply\n```\n\n---\n\n## Model functions\n\n### `hub_model`\n\nCreates an `INCREMENTAL_UNMANAGED` Hub model.\n\n```python\nfrom datavault4sqlmesh import hub_model\n\nhub_model(\n    name=\"dv.customer_h\",\n    hashkey=\"hk_customer_h\",\n    business_keys=[\"customer_id\"],\n    source_schema=\"stage\",\n    source_table=\"stg_customer\",\n    rsrc_statics=[\"ERP/customers\"],\n    # Optional\n    cron=\"@daily\",\n    column_overrides={\"customer_id\": \"BIGINT\"},\n)\n```\n\n| Parameter | Required | Description |\n|---|---|---|\n| `name` | yes | Qualified model name, e.g. `\"dv.customer_h\"` |\n| `hashkey` | yes | Hub hash key column name |\n| `business_keys` | yes | Business key column names (at least one) |\n| `source_schema` | no | Schema of the staging source table |\n| `source_table` | no | Staging source table name; triggers auto-generate mode |\n| `sources` | no | List of `SourceModel` objects for multi-source hubs |\n| `rsrc_statics` | no | LIKE-pattern strings for HWM scoping |\n| `cron` | no | SQLMesh cron expression |\n| `grain` | no | Unique-row columns; defaults to `[hashkey]` |\n| `column_overrides` | no | Override inferred column types |\n\n**Multi-source** hubs pass a list of `SourceModel` objects via `sources`. Each\nsource is unioned and deduplicated by earliest `ldts` per hash key. `source_table`\nand `sources` cannot be combined.\n\n---\n\n### `link_model`\n\nCreates an `INCREMENTAL_UNMANAGED` Link model.\n\n```python\nfrom datavault4sqlmesh import link_model\n\nlink_model(\n    name=\"dv.order_customer_l\",\n    link_hash_key=\"hk_order_customer_l\",\n    foreign_hash_keys=[\"hk_order_h\", \"hk_customer_h\"],\n    source_schema=\"stage\",\n    source_table=\"stg_orders\",\n    rsrc_statics=[\"OMS/orders\"],\n)\n```\n\n| Parameter | Required | Description |\n|---|---|---|\n| `name` | yes | Qualified model name |\n| `link_hash_key` | yes | Link hash key column name |\n| `foreign_hash_keys` | yes | Foreign hash key columns (at least two) |\n| `source_schema` | no | Schema of the staging source table |\n| `source_table` | no | Staging source table name; triggers auto-generate mode |\n| `sources` | no | List of `SourceModel` objects for multi-source links |\n| `rsrc_statics` | no | LIKE-pattern strings for HWM scoping |\n\n---\n\n### `satellite_model`\n\nCreates an `INCREMENTAL_UNMANAGED` Satellite v0 (current-record) model.\n\n**Single source only** — pass one `SourceModel` to `source_model`.\n\n```python\nfrom datavault4sqlmesh import satellite_model\n\nsatellite_model(\n    name=\"dv.customer_0_s\",\n    parent_hash_key=\"hk_customer_h\",\n    hash_diff=\"hd_customer_s\",\n    payload=[\"customer_name\", \"email\"],\n    source_schema=\"stage\",\n    source_table=\"stg_customer\",\n)\n```\n\n| Parameter | Required | Description |\n|---|---|---|\n| `name` | yes | Qualified model name |\n| `parent_hash_key` | yes | Hash key of the parent Hub or Link |\n| `hash_diff` | yes | Hash diff column name |\n| `payload` | no | Attribute column names |\n| `source_schema` | no | Schema of the staging source table |\n| `source_table` | no | Staging source table name; triggers auto-generate mode |\n\n---\n\n### `satellite_v1_model`\n\nCreates a `FULL` Satellite v1 (end-dated) model derived from a v0 satellite.\n\n```python\nfrom datavault4sqlmesh import satellite_v1_model\n\nsatellite_v1_model(\n    name=\"dv.customer_1_s\",\n    parent_hash_key=\"hk_customer_h\",\n    hash_diff=\"hd_customer_s\",\n    payload=[\"customer_name\", \"email\"],\n    sat_v0_table=\"customer_0_s\",\n    sat_v0_schema=\"dv\",\n)\n```\n\n| Parameter | Required | Description |\n|---|---|---|\n| `name` | yes | Qualified model name |\n| `parent_hash_key` | yes | Hash key of the parent Hub or Link |\n| `hash_diff` | yes | Hash diff column name |\n| `payload` | no | Attribute column names |\n| `sat_v0_table` | no | Table name of the v0 source satellite; omit to use decorator mode |\n| `sat_v0_schema` | no | Schema of the v0 source; defaults to the same schema as the v1 target |\n| `add_is_current` | no | Append an `is_current` boolean column (default `True`) |\n\n---\n\n### `stage_model`\n\nCreates a `FULL` (complete refresh) Stage model.\n\n```python\nfrom datavault4sqlmesh import stage_model\n\nstage_model(\n    name=\"stage.stg_customer\",\n    source_table=\"customers\",\n    source_schema=\"raw\",\n    hashed_columns={\n        \"hk_customer_h\": [\"customer_id\"],\n        \"hd_customer_s\": {\n            \"is_hashdiff\": True,\n            \"columns\": [\"customer_name\", \"email\"],\n        },\n    },\n    column_overrides={\n        \"customer_id\":   \"VARCHAR\",\n        \"customer_name\": \"VARCHAR\",\n        \"email\":         \"VARCHAR\",\n    },\n)\n```\n\nUse `derived_columns` for expressions not present in the raw source (e.g. a\nconstant record source):\n\n```python\nstage_model(\n    name=\"stage.stg_orders\",\n    source_table=\"orders\",\n    source_schema=\"raw\",\n    derived_columns={\"rsrc\": \"'OMS/orders'\"},\n    hashed_columns={\"hk_order_h\": [\"order_id\"]},\n)\n```\n\n---\n\n## Column type inference\n\nAll model functions automatically build the `columns={}` dict required by SQLMesh.\nDefault types:\n\n| Column category | Inferred type |\n|---|---|\n| Hash key / hash diff | `VARCHAR` |\n| Business keys / payload | `VARCHAR` |\n| Foreign hash keys | `VARCHAR` |\n| `ldts_alias` (load date) | `TIMESTAMP` |\n| `rsrc_alias` (record source) | `VARCHAR` |\n| `ledts_alias` (load end date, sat v1 only) | `TIMESTAMP` |\n| `is_current` (sat v1 only) | `BOOLEAN` |\n\nOverride specific columns with `column_overrides`:\n\n```python\nhub_model(..., column_overrides={\"customer_id\": \"BIGINT\"})\n```\n\n---\n\n## SQLMesh model kind mapping\n\n| DV entity | SQLMesh kind | Notes |\n|---|---|---|\n| Stage | `FULL` | Complete refresh every run |\n| Hub | `INCREMENTAL_UNMANAGED` | HWM-based incremental load |\n| Link | `INCREMENTAL_UNMANAGED` | HWM-based incremental load |\n| Satellite v0 | `INCREMENTAL_UNMANAGED` | HWM-based incremental load |\n| Satellite v1 | `FULL` | Complete refresh; see Limitations |\n\nOverride the kind via `kind={\"name\": \"FULL\"}` if needed.\n\n---\n\n## Limitations\n\n### Satellite v1 materializes as a table, not a view\n\nData Vault 2 specifies that a v1 (end-dated) satellite is a derived view over\nthe v0 base satellite. In SQLMesh, Python `execute` models can only produce\n**tables** — the `VIEW` model kind is only available for SQL models. For this\nreason, `satellite_v1_model` uses `FULL` (complete refresh table).\n\nThe semantic result is the same: every run reconstructs the full end-dated\nhistory from the v0 source. The cost is a full table rewrite on each run rather\nthan a no-op view.\n\n### Satellite v0 is single-source only\n\n`satellite_model` accepts a single `SourceModel`. Multi-source satellites are\nnot supported. If your source data comes from multiple staging tables, load\nthem into a single staging model first.\n\n### Stage source columns are not inferred automatically\n\n`StageGenerator` produces a `SELECT *` from the raw source, so the source table\ncolumns are not known at model-definition time. Any column that needs to appear\nin the SQLMesh `columns={}` schema — including all business key and payload\ncolumns — must be declared explicitly via `column_overrides`.\n\n---\n\n## Running tests\n\n```bash\ncd datavault4sqlmesh\npip install -e \".[test]\"\npytest tests/\n```\n\n---\n\n## 📄 License\n\nThis project is licensed under the **GNU Affero General Public License v3 (AGPL-3.0)** - see the [LICENSE](LICENSE) file for details.\n\n---\n## 🤝 Contributing\n\nWe welcome and appreciate community contributions! To keep the project sustainable while ensuring the software remains open and accessible, we follow a **Dual-Licensing** model.\n\n### 📜 Licensing \u0026 Open Source\nThis project is licensed under the **GNU Affero General Public License v3 (AGPL-3.0)**. \n\nThe AGPL is a \"strong copyleft\" license. If you modify this software and provide it as a service over a network (SaaS), you **must** make your modified source code available to your users under the same license.\n\n### ✍️ Contributor License Agreement (CLA)\nTo contribute code, all contributors are required to sign our **Contributor License Agreement (CLA)**. \n* **Why?** This ensures that you have the right to contribute the code and grants us the necessary rights to include your work in future versions of the project, including potential commercial or non-AGPL distributions.\n* **How?** **FIXME**\n\n### 💼 Commercial Usage \u0026 Licensing\nWe understand that the AGPL-3.0 may not be suitable for every organization's internal policies or proprietary products. \n\nIf you wish to use this project in a commercial or proprietary setting without the \"copyleft\" requirements of the AGPL, we offer **alternative commercial licenses**. This allows you to:\n* Use the software without disclosing your own source code.\n* Receive dedicated support and enterprise-grade warranties.\n* Support the development team.\n\nPlease contact us at **contact@scalefree.com** to discuss a commercial license tailored to your needs.\n\n---\n\nBuilt by [Scalefree](https://scalefree.com)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscalefreecom%2Fdatavault4sqlmesh","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fscalefreecom%2Fdatavault4sqlmesh","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscalefreecom%2Fdatavault4sqlmesh/lists"}