{"id":29323849,"url":"https://github.com/korawica/sqlplate","last_synced_at":"2026-02-15T20:31:10.729Z","repository":{"id":276981111,"uuid":"925526302","full_name":"korawica/sqlplate","owner":"korawica","description":":scroll: SQL template generator","archived":false,"fork":false,"pushed_at":"2025-08-28T02:42:37.000Z","size":91,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-12-01T09:42:46.938Z","etag":null,"topics":["data-quality","jinja2-templates","sql","sql-template"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/korawica.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-02-01T04:29:13.000Z","updated_at":"2025-08-28T02:42:37.000Z","dependencies_parsed_at":"2025-02-11T14:24:33.509Z","dependency_job_id":"eb677d53-55ed-4894-b457-b44f680ef380","html_url":"https://github.com/korawica/sqlplate","commit_stats":null,"previous_names":["korawica/sqlplate"],"tags_count":9,"template":false,"template_full_name":null,"purl":"pkg:github/korawica/sqlplate","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/korawica%2Fsqlplate","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/korawica%2Fsqlplate/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/korawica%2Fsqlplate/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/korawica%2Fsqlplate/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/korawica","download_url":"https://codeload.github.com/korawica/sqlplate/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/korawica%2Fsqlplate/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29488535,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-15T19:29:10.908Z","status":"ssl_error","status_checked_at":"2026-02-15T19:29:10.419Z","response_time":118,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-quality","jinja2-templates","sql","sql-template"],"created_at":"2025-07-07T16:41:07.958Z","updated_at":"2026-02-15T20:31:10.717Z","avatar_url":"https://github.com/korawica.png","language":"Python","readme":"# SQL Template\n\n[![test](https://github.com/korawica/sqlplate/actions/workflows/tests.yml/badge.svg?branch=main)](https://github.com/korawica/sqlplate/actions/workflows/tests.yml)\n[![pypi version](https://img.shields.io/pypi/v/sqlplate)](https://pypi.org/project/sqlplate/)\n[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/sqlplate?logo=pypi)](https://pypi.org/project/sqlplate/)\n[![size](https://img.shields.io/github/languages/code-size/korawica/sqlplate)](https://github.com/korawica/sqlplate)\n[![gh license](https://img.shields.io/github/license/korawica/sqlplate)](https://github.com/korawica/sqlplate/blob/main/LICENSE)\n\nA **SQL Template Generator** (aka `SqlPlate`) provide the generator object for\nSQL template statements via Python API object.\nAll SQL template files are store in the [Jinja template](https://jinja.palletsprojects.com/en/stable/templates/)\nformat that is the powerful template tool package.\n\n\u003e [!NOTE]\n\u003e This project keep all SQL use-case that use on Data Engineer works like SCD2\n\u003e with `MERGE` operator.\n\n**The layer of SQL template files will be:**\n\n```text\ntemplates/\n   ├─ databricks/\n   │     ├─ 📂macros/\n   │     │     ╰─ ⚙️ delta.jinja\n   │     ╰─ 📂latest/\n   │           ├─ 📜 etl.delta.sql\n   │           ├─ 📜 etl.scd2.sql\n   │           ╰─ 📜 select.sql\n   ├─ sqlite/\n   │     ╰─ 📂latest/\n   │           ╰─📜 etl.delta.sql\n   ├─ synapse/\n   │     ╰─ 📂latest/\n   │           ╰─📜 etl.delta.sql\n   ╰─ utils/\n         ╰─ ⚙️ etl_vars.jinja\n```\n\n\u003e [!IMPORTANT]\n\u003e The first object of this project is generate ETL statements for dynamic service\n\u003e change. You can change the SQL compute service while the SQL statement does not\n\u003e change.\n\n## :package: Installation\n\n```shell\npip install -U sqlplate\n```\n\n## :fork_and_knife: Usage\n\n### Generate SQL template\n\nStart passing option parameters before generate the Delta ETL SQL statement that\nwill use on the Azure Databricks service.\n\n```python\nfrom datetime import datetime\nfrom sqlplate import SQLPlate\n\nstatement: str = (\n    SQLPlate.format('databricks')\n    .template('etl.delta')\n    .option('catalog', 'catalog-name')\n    .option('schema', 'schema-name')\n    .option('table', 'table-name')\n    .option('pk', 'pk_col')\n    .option('columns', ['col01', 'col02'])\n    .option('query', 'SELECT * FROM catalog-name.schema-name.source-name')\n    .option('load_src', 'SOURCE_FOO')\n    .option('load_id', 1)\n    .option('load_date', datetime(2025, 2, 1, 10))\n    .option('only_main', True)\n    .load()\n)\nprint(statement.strip().strip('\\n'))\n```\n\nThe result SQL statement:\n\n```sql\nMERGE INTO catalog-name.schema-name.table-name AS target\nUSING (\n    WITH change_query AS (\n        SELECT\n            src.*,\n        CASE WHEN tgt.pk_col IS NULL THEN 99\n             WHEN hash(src.col01, src.col02) \u003c\u003e hash(tgt.col01, tgt.col02) THEN 1\n             ELSE 0 END AS data_change\n        FROM ( SELECT * FROM catalog-name.schema-name.source-name ) AS src\n        LEFT JOIN catalog-name.schema-name.table-name AS tgt\n            ON  tgt.col01 = src.col01\nAND tgt.col02 = src.col02\n    )\n    SELECT * EXCEPT( data_change ) FROM change_query WHERE data_change IN (99, 1)\n) AS source\n    ON  target.pk_col = source.pk_col\nWHEN MATCHED THEN UPDATE\n    SET target.col01            = source.col01\n    ,   target.col02            = source.col02\n    ,   target.updt_load_src    = 'SOURCE_FOO'\n    ,   target.updt_load_id     = 1\n    ,   target.updt_load_date   = to_timestamp('20250201', 'yyyyMMdd')\nWHEN NOT MATCHED THEN INSERT\n    (\n        col01, col02, pk_col, load_src, load_id, load_date, updt_load_src, updt_load_id, updt_load_date\n    )\n    VALUES (\n        source.col01,\n        source.col02,\n        source.pk_col,\n        'SOURCE_FOO',\n        1,\n        20250201,\n        'SOURCE_FOO',\n        1,\n        to_timestamp('20250201', 'yyyyMMdd')\n    )\n;\n```\n\n### Data Quality\n\nThis package handle generate SQL statement only.\nFor the data quality part, this package can use quality templates instead.\n\n```python\nfrom sqlplate import SQLPlate\n\nstatement: str = (\n    SQLPlate.format('databricks')\n    .template('quality.check')\n    .option('catalog', 'catalog-name')\n    .option('schema', 'schema-name')\n    .option('table', 'table-name')\n    .option('filter', \"load_date \u003e= to_timestamp('20250201', 'yyyyMMdd')\")\n    .option('unique', ['pk_col'])\n    .option('notnull', ['col01', 'col02'])\n    .check(\"contain\", [\"col01\"], \"IN ['A', 'B', 'C']\")\n    .check(\"gt_10000\", [\"col03\"], \"\u003e 10000\")\n    .load()\n)\nprint(statement.strip().strip('\\n'))\n```\n\nThe result SQL statement:\n\n```sql\nWITH source AS (\n    SELECT\n        *\n    FROM\n        catalog-name.schema-name.table-name\n    WHERE load_date \u003e= to_timestamp('20250201', 'yyyyMMdd')\n)\n, records AS (\n    SELECT COUNT(1) AS table_records FROM source\n)\nSELECT\n    (SELECT table_records FROM records) AS table_records\n    , (SELECT COUNT( DISTINCT pk_col ) FROM source) = table_records AS unique_pk_col\n    , (SELECT COUNT_IF( col01 IS NULL ) FROM source) = 0 AS notnull_col01\n    , (SELECT COUNT_IF( col02 IS NULL ) FROM source) = 0 AS notnull_col02\n    , (SELECT COUNT(1) FILTER(WHERE col01 IN ['A', 'B', 'C']) FROM source) = table_records AS contain_col01\n    , (SELECT COUNT(1) FILTER(WHERE col03 \u003e 10000) FROM source) = table_records AS gt_10000_col03\nFROM records\n```\n\n## :chains: Support Systems\n\n| System             | Progress Status  | System Integration Test | Remark                            |\n|:-------------------|:----------------:|:-----------------------:|:----------------------------------|\n| databricks         |        🟢        |           🟡            | Azure Databricks                  |\n| postgres           |        🔴        |           🔴            |                                   |\n| mysql              |        🔴        |           🔴            |                                   |\n| mssql              |        🔴        |           🔴            | Microsoft SQL Server              |\n| synapse            |        🔴        |           🔴            | Azure Synapse Dedicate SQL Pool   |\n| synapse-serverless |        🔴        |           🔴            | Azure Synapse Serverless SQL Pool |\n| bigquery           |        🟡        |           🔴            | Google BigQuery                   |\n| snowflake          |        🔴        |           🔴            |                                   |\n| sqlite             |        🟡        |           🟡            |                                   |\n| duckdb             |        🟡        |           🟡            |                                   |\n| redshift           |        🔴        |           🔴            | Amazon RedShift                   |\n| athena             |        🔴        |           🔴            | Amazon Athena                     |\n| trino              |        🔴        |           🔴            | Trino                             |\n\n\u003e [!NOTE]\n\u003e - 🟢 Complete\n\u003e - 🟡 In progress\n\u003e - 🔴 Does not develop yet\n\u003e - 🟣 Does not plan to support\n\n## :speech_balloon: Contribute\n\nI do not think this project will go around the world because it has specific propose,\nand you can create by your coding without this project dependency for long term\nsolution. So, on this time, you can open [the GitHub issue on this project :raised_hands:](https://github.com/korawica/sqlplate/issues)\nfor fix bug or request new feature if you want it.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkorawica%2Fsqlplate","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkorawica%2Fsqlplate","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkorawica%2Fsqlplate/lists"}