{"id":51550188,"url":"https://github.com/scalefreecom/datavault4sqlglot","last_synced_at":"2026-07-09T23:01:07.248Z","repository":{"id":369846456,"uuid":"1153459011","full_name":"ScalefreeCOM/datavault4sqlglot","owner":"ScalefreeCOM","description":"Collection of Data Vault loading patterns implemented for sqlglot. Published and maintained by Scalefree International GmbH.","archived":false,"fork":false,"pushed_at":"2026-07-07T06:24:06.000Z","size":304,"stargazers_count":0,"open_issues_count":6,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-07-07T08:11:14.272Z","etag":null,"topics":["automation","data-vault","data-vault-2","sqlglot"],"latest_commit_sha":null,"homepage":"https://www.scalefree.com","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ScalefreeCOM.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-02-09T10:22:21.000Z","updated_at":"2026-07-07T06:23:48.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/ScalefreeCOM/datavault4sqlglot","commit_stats":null,"previous_names":["scalefreecom/datavault4sqlglot"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/ScalefreeCOM/datavault4sqlglot","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ScalefreeCOM%2Fdatavault4sqlglot","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ScalefreeCOM%2Fdatavault4sqlglot/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ScalefreeCOM%2Fdatavault4sqlglot/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ScalefreeCOM%2Fdatavault4sqlglot/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ScalefreeCOM","download_url":"https://codeload.github.com/ScalefreeCOM/datavault4sqlglot/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ScalefreeCOM%2Fdatavault4sqlglot/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35314872,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-09T02:00:07.329Z","response_time":57,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automation","data-vault","data-vault-2","sqlglot"],"created_at":"2026-07-09T23:01:05.416Z","updated_at":"2026-07-09T23:01:07.242Z","avatar_url":"https://github.com/ScalefreeCOM.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# datavault4sqlglot\n\nA Python library for generating Data Vault 2 SQL using [sqlglot](https://github.com/tobymao/sqlglot).\nProduces dialect-agnostic SQL for Hubs, Links, Satellites, and Staging layers — no dbt, no live database connection required.\n\n## Quick Start\n\n```python\nfrom datavault4sqlglot.generators.hub import HubGenerator\nfrom datavault4sqlglot.metadata import SourceModel, SourceBinding\n\n# Describe the staged source table\nsource = SourceModel(\n    schema=\"stage\",\n    table_name=\"stg_orders\",\n    load_date_col=\"ldts\",\n    record_source_col=\"rsrc\",\n)\n\n# Bind the source to a Hub: declare which column is the business key\nbinding = SourceBinding(\n    source=source,\n    business_keys=[\"order_id\"],\n)\n\n# Generate the Hub SQL\ngenerator = HubGenerator(\n    sources=[binding],\n    hashkey=\"hk_order_h\",\n    target_schema=\"dv\",\n    target_table=\"order_h\",\n    is_incremental=True,\n)\n\nprint(generator.to_sql())\n```\n\n## Configuration\n\nThe library uses a global configuration object (`datavault4sqlglot.config`) with the following defaults:\n\n| Key | Default | Description |\n|---|---|---|\n| `dialect` | `snowflake` | Target SQL dialect |\n| `hash` | `MD5` | Hash algorithm (`MD5`, `SHA256`, `SHA1`) |\n| `ldts_alias` | `ldts` | Load date timestamp column name |\n| `rsrc_alias` | `rsrc` | Record source column name |\n| `ledts_alias` | `ledts` | Load end date column name |\n| `end_of_all_times` | `9999-12-31` | Sentinel for open-ended records |\n| `beginning_of_all_times` | `0001-01-01` | Sentinel for ghost records |\n| `hashkey_input_case_sensitive` | `false` | Apply `UPPER()` before hashing hash keys |\n| `hashdiff_input_case_sensitive` | `false` | Apply `UPPER()` before hashing hash diffs |\n| `use_trim` | `true` | Apply `TRIM()` before hashing |\n| `quote_identifiers` | `true` | Quote table and column identifiers |\n| `ghost_record_rsrc` | `SYSTEM` | Record source value for the unknown ghost row |\n| `ghost_record_error_rsrc` | `ERROR` | Record source value for the error ghost row |\n\nYou can override any of these by placing a `config.json` file in your current working directory.\nThe library will automatically load and apply these settings when imported.\n\n**Example `config.json`:**\n\n```json\n{\n  \"ldts_alias\": \"load_date_timestamp\",\n  \"hash\": \"SHA256\",\n  \"dialect\": \"bigquery\"\n}\n```\n\nYou can also manually load a configuration file from a specific path:\n\n```python\nfrom datavault4sqlglot.config import config, load_config\n\nload_config(config, \"/path/to/my/custom_config.json\")\n```\n\n## Models\n\nThree distinct classes, each for a different layer:\n\n### `StageModel` — used exclusively by `StageGenerator`\n\nDescribes a raw source table that needs to be hashed and prepared. It owns everything about that transformation: which columns to hash, derived expressions, missing columns for schema evolution, etc. It's a self-contained description of one staging job — no binding to anything else.\n\n```python\nStageModel(\n    table_name=\"raw.orders\",\n    hashed_columns={\"HK_ORDER_H\": [\"ORDER_ID\"]},\n    derived_columns={\"ldts\": \"CURRENT_TIMESTAMP()\"},\n)\n```\n\n### `SourceModel` — the physical table pointer used by vault generators (Hub, Link, Sat)\n\nJust says where to find the already-staged data: table name, optional schema/database, and which\ncolumns are `ldts`/`rsrc` if they differ from the config defaults. No transformation logic.\n\n```python\nSourceModel(\n    database=\"RAW_DB\",\n    schema=\"STAGE\",\n    table_name=\"STG_ORDERS\",\n    load_date_col=\"LOAD_DATE\",\n    record_source_col=\"RECORD_SOURCE\",\n)\n```\n\n### `SourceBinding` — wraps a `SourceModel` with DV-loading intent\n\nAnswers what to extract from that staged table for a specific vault entity: per-source physical\n`bk_columns` (when they differ from the hub's canonical names), foreign hash keys (for links),\n`rsrc_statics` for HWM scoping, etc. A single `SourceModel` can be wrapped in different\n`SourceBinding`s for different vault entities.\n\nThe hub-level *canonical* `business_keys` live on `HubGenerator` itself (not on the binding) —\nevery binding into the same hub maps onto that same canonical name set.\n\n```python\nSourceBinding(\n    source=_SRC_ORDERS_MODEL,        # the SourceModel\n    bk_columns=[\"SAP_ORDER_ID\"],     # only needed when the source's column\n                                     # name differs from the hub canonical\n    rsrc_statics=[\"ERP/ORDERS\"],\n)\n```\n\n## Conceptual split\n\n→ Staging table  \n\u0026emsp;→ `StageModel` → `StageGenerator` → staged table (to calculate hash keys)\n\n→ Raw Data Vault table  \n\u0026emsp;→ `SourceModel` — where is the data?  \n\u0026emsp;\u0026emsp;→ `SourceBinding` — what to extract, for which vault entity?  \n\u0026emsp;\u0026emsp;\u0026emsp;→ `HubGenerator` / `LinkGenerator` / `SatelliteGenerator`\n\n`StageModel` and `SourceModel` are both Pydantic models (validated on construction). `SourceBinding` is a plain dataclass — it's just a lightweight container pairing a `SourceModel` with extraction metadata.\n\n---\n\n## 📄 License\n\nThis project is licensed under the **GNU Affero General Public License v3 (AGPL-3.0)** - see the [LICENSE](LICENSE) file for details.\n\n---\n## 🤝 Contributing\n\nWe welcome and appreciate community contributions! To keep the project sustainable while ensuring the software remains open and accessible, we follow a **Dual-Licensing** model.\n\n### 📜 Licensing \u0026 Open Source\nThis project is licensed under the **GNU Affero General Public License v3 (AGPL-3.0)**. \n\nThe AGPL is a \"strong copyleft\" license. If you modify this software and provide it as a service over a network (SaaS), you **must** make your modified source code available to your users under the same license.\n\n### ✍️ Contributor License Agreement (CLA)\nTo contribute code, all contributors are required to sign our **Contributor License Agreement (CLA)**. \n* **Why?** This ensures that you have the right to contribute the code and grants us the necessary rights to include your work in future versions of the project, including potential commercial or non-AGPL distributions.\n* **How?** **FIXME**\n\n### 💼 Commercial Usage \u0026 Licensing\nWe understand that the AGPL-3.0 may not be suitable for every organization's internal policies or proprietary products. \n\nIf you wish to use this project in a commercial or proprietary setting without the \"copyleft\" requirements of the AGPL, we offer **alternative commercial licenses**. This allows you to:\n* Use the software without disclosing your own source code.\n* Receive dedicated support and enterprise-grade warranties.\n* Support the development team.\n\nPlease contact us at **contact@scalefree.com** to discuss a commercial license tailored to your needs.\n\n---\n\nBuilt by [Scalefree](https://scalefree.com)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscalefreecom%2Fdatavault4sqlglot","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fscalefreecom%2Fdatavault4sqlglot","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscalefreecom%2Fdatavault4sqlglot/lists"}