{"id":13703352,"url":"https://github.com/AxelThevenot/dbt-assertions","last_synced_at":"2025-05-05T07:30:46.013Z","repository":{"id":217359783,"uuid":"743632470","full_name":"AxelThevenot/dbt-assertions","owner":"AxelThevenot","description":"Package to assert rows in-line with dbt macros.","archived":false,"fork":false,"pushed_at":"2025-04-22T18:28:23.000Z","size":863,"stargazers_count":66,"open_issues_count":4,"forks_count":10,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-22T18:39:12.867Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AxelThevenot.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-01-15T16:45:21.000Z","updated_at":"2025-04-22T18:28:26.000Z","dependencies_parsed_at":"2024-01-29T10:51:06.233Z","dependency_job_id":"6e727752-3847-4b06-93a6-53991c7847f4","html_url":"https://github.com/AxelThevenot/dbt-assertions","commit_stats":null,"previous_names":["axelthevenot/dbt-assertions"],"tags_count":14,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AxelThevenot%2Fdbt-assertions","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AxelThevenot%2Fdbt-assertions/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AxelThevenot%2Fdbt-assertions/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AxelThevenot%2Fdbt-assertions/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AxelThevenot","download_url":"https://codeload.github.com/AxelThevenot/dbt-assertions/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252458338,"owners_count":21751019,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-02T21:00:53.769Z","updated_at":"2025-05-05T07:30:46.007Z","avatar_url":"https://github.com/AxelThevenot.png","language":null,"funding_links":[],"categories":["Packages"],"sub_categories":[],"readme":"![dbt-assertions-logo.png](./img/dbt-assertions-logo.png)\n\n\u003cp align=\"center\"\u003e\n    \u003cimg alt=\"License\" src=\"https://img.shields.io/badge/license-Apache--2.0-ff69b4?style=plastic\"/\u003e\n    \u003cimg alt=\"Static Badge\" src=\"https://img.shields.io/badge/dbt-package-orange\"\u003e\n    \u003cimg alt=\"GitHub Release\" src=\"https://img.shields.io/github/v/release/AxelThevenot/dbt-assertions\"\u003e\n    \u003cimg alt=\"GitHub (Pre-)Release Date\" src=\"https://img.shields.io/github/release-date-pre/AxelThevenot/dbt-assertions\"\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"https://img.shields.io/circleci/project/github/badges/shields/master\" alt=\"build status\"\u003e\n    \u003cimg alt=\"GitHub issues\" src=\"https://img.shields.io/github/issues/AxelThevenot/dbt-assert\"\u003e\n    \u003cimg alt=\"GitHub pull requests\" src=\"https://img.shields.io/github/issues-pr/AxelThevenot/dbt-assertions\"\u003e\n    \u003cimg src=\"https://img.shields.io/github/contributors/AxelThevenot/dbt-assertions\" /\u003e\n\u003c/p\u003e\n\n## Features\n\n✅ **Robust Data Quality Checks**\n\n`dbt-assertions` ensures thorough data quality assessments at the row level,\nenhancing the reliability of downstream models.\n\n🔍 **Efficient Exception Detection**\n\nGranular row-by-row exception detection identifies and\nflags specific rows that fails assertions, streamlining the resolution process.\n\n🛠️ **Customizable Assertions \u0026 Easy Integration**\n\nEasy-to-use macros `assertions()` and `assertions_filter()` empower users to\ncustomize without barriers data quality checks within the model YAML definition,\nadapting to specific data validation needs.\n\n🚀 **An Easy Shift from your Actual Workflows**\n\nA generic test `generic_assertions()` to perform dbt tests as usual,\ntesting the package easily without compromising your current workflows.\n**you can test the package with this generic test easily without having to rebuild you table**\n\n\n\u003ca href=\"https://drive.google.com/file/d/1REdVQufkBVYu2m8Z2zxnXgBTsWvQdyYa/view?usp=sharing\"\u003e\n  \u003cp align=\"center\" href=\"https://drive.google.com/file/d/1REdVQufkBVYu2m8Z2zxnXgBTsWvQdyYa/view?usp=sharing\"\u003e\n    \u003cimg src=\"img/dbt-assertions-video.png\" width=\"67%\"\u003e\n  \u003c/p\u003e\n\u003c/a\u003e\n\n## Content\n\n- [Features](#features)\n- [Content](#content)\n- [Install](#install)\n- [Supported databases](#supported-databases)\n- [Dependencies](#dependencies)\n- [Variables](#variables)\n- [Basic Example](#basic-example)\n- [Documentation](#documentation)\n  - [Macros](#macros)\n    - [assertions](#assertions)\n    - [assertions\\_filter](#assertions_filter)\n  - [Tests](#tests)\n    - [generic\\_assertions](#generic_assertions)\n  - [Model definition](#model-definition)\n    - [Yaml general definition](#yaml-general-definition)\n    - [Custom assertions](#custom-assertions)\n    - [`null_as_exception`](#null_as_exception)\n    - [`__unique__` helper](#__unique__-helper)\n    - [`__not_null__` helper](#__not_null__-helper)\n    - [Custom column name](#custom-column-name)\n- [Contribution](#contribution)\n- [Acknowledgments](#acknowledgments)\n- [Contact](#contact)\n\n\n## Install\n\n`dbt-assertions` currently supports `dbt 1.7.x` or higher.\n\n\nCheck [dbt package hub](https://hub.getdbt.com/calogica/dbt_expectations/latest/)\nfor the latest installation instructions,\nor [read the docs](https://docs.getdbt.com/docs/package-management)\nfor more information on installing packages.\n\nInclude in `packages.yml`\n\n```yaml\npackages:\n  - package: AxelThevenot/dbt_assertions\n    version: [\"\u003e=1.0.0\", \"\u003c2.0.0\"]\n    # \u003csee https://github.com/AxelThevenot/dbt-assertions/releases/latest\u003e for the latest version tag\n```\n\n## Supported databases\n\n|**Database**|[assertions](#assertions)|[assertions_filter](#assertions_filter)|[\\_\\_unique__](#__unique__-helper)|[\\_\\_not_null__](#__not_null__-helper)|[generic_assertions](#generic_assertions)\n|:---:|:---:|:---:|:---:|:---:|:---:|\n|BigQuery (default)|✅|✅|✅|✅|✅|\n|Snowflake|✅|✅|✅ \u003c/br\u003e(not nested fields)|✅ \u003c/br\u003e(not nested fields)|✅|\n|DuckDB|✅|✅|✅ \u003c/br\u003e(not nested fields)|✅ \u003c/br\u003e(not nested fields)|✅|\n|Databricks|✅|✅|✅ \u003c/br\u003e(not nested fields)|✅ \u003c/br\u003e(not nested fields)|✅|\n|Redshift|✅|✅\u003c/br\u003e (Include/Exclude Fields Not Supported)|✅ \u003c/br\u003e(not nested fields)|✅ \u003c/br\u003e(not nested fields)|✅|\n|Athena|✅|✅|✅ \u003c/br\u003e(not nested fields)|✅ \u003c/br\u003e(not nested fields)|✅|\n|Clickhouse|✅|✅|✅ \u003c/br\u003e(not nested fields)|✅ \u003c/br\u003e(not nested fields)|✅|\n|Spark|✅|✅|✅ \u003c/br\u003e(not nested fields)|✅ \u003c/br\u003e(not nested fields)|✅|\n|Others|ℹ️|ℹ️|ℹ️|ℹ️|ℹ️|\n\n- ✅: supported\n- ✔️: supported not tested\n- ❌: not supported\n- ℹ️: Opened to contributions ❤️ (see the [integration_tests](./integration_tests/) folder on how to test your adapter)\n\n\nFor latest release, see [https://github.com/AxelThevenot/dbt-assertions/releases](https://github.com/AxelThevenot/dbt-assertions/releases/latest)\n\n\n## Dependencies\n\nThis package do not have dependencies.\n\n## Variables\n\nThe following variable **can** be defined in your `dbt_project.yml` file to change the default `exceptions` column name used by the package.\n\n```yml\nvars:\n    \"dbt_assertions:default_column\": \"exceptions\"\n```\n\n## Basic Example\n\nCheck the [basic_example](examples/basic_example) example.\n\n## Documentation\n\n### Macros\n\n#### [assertions](macros/assertions.sql)\n\n`assertions()` macro generates a select expression for row-level assertions.\n\n**Arguments:**\n- **column (optional[str]):** column to read the assertions from.\n\n---\n\nThis macro parses the schema model YAML to extract row-level assertions;\n[custom assertions](#custom-assertions),\n[unique](#__unique__-helper),\nand [not-null](#__not_null__-helper).\nIt then constructs an array of exceptions for each row based on its assertions results.\n\n\nBy default, it will generate assertions based on\nyour [YAML model definition](#model-definition)\nreading configuration for a column named `exceptions`.\n\nYou can call the macro using `column` argument to change this default column.\n\n```sql\nSELECT\n    *,\n    {{ dbt_assertions.assertions(column='warnings') }},\nFROM {{ ref('my_model') }}\n```\n\n**Note:** this macro is made to generate assertions based on you query result.\nIt means it must be generated at the end of the query.\n\n```sql\nWITH\n    [...] -- Other CTEs\n    final AS (\n        SELECT\n            [...]\n        FROM {{ ref('my_model') }}\n    )\n\n-- After query results\nSELECT\n    *,\n    {{ dbt_assertions.assertions() }},\nFROM final\n```\n\n#### [assertions_filter](macros/assertions_filter.sql)\n\n`assertions_filter()` macro generates an expression to filter rows based on\nassertions results, generated with the [`assertions()`](#assertions) macro.\n\n**Arguments:**\n- **column (optional[str]):** Column to read the exceptions from.\n- **exclude_list (optional[list[str]]):** Assertions to exclude in the filter.\n- **include_list (optional[list[str]]):** Assertions to include in the filter.\n- **reverse (optional[bool]):** returns rows without exception when `reverse=false`,\n  and rows with exceptions when `reverse=true`.\n\n---\n\nBy default, each row with exception(s) will be filtered.\n\n\n```sql\nSELECT\n    *\nFROM {{ ref('my_model') }}\nWHERE {{ dbt_assertions.assertions_filter() }}\n```\n\nYou can change this default behaviour specifying an\noptional `exclude_list` or `include_list` argument (not both).\n\n```sql\nSELECT\n    *\nFROM {{ ref('my_model') }}\nWHERE {{ dbt_assertions.assertions_filter(exclude_list=['assertions_id']) }}\n```\n\n### Tests\n\n####  [generic_assertions](tests/generic/generic_assertions.sql)\n\nGenerates a test to get rows based on exceptions.\n\nIt will returns the rows without any exception by default.\nYou can change this default behaviour specifying a exclude_list or include_list (not both).\n\nYou must defined beforehand the assertions for the model. [More on YAML definition for assertions](#yaml-general-definition).\n\n**Arguments:**\n- **column (optional[str]):** Column to read the exceptions from.\n- **exclude_list (optional[list[str]]):** Assertions to exclude in the filter.\n- **include_list (optional[list[str]]):** Assertions to include in the filter.\n- **re_assert (optional[bool]):** to set to `true` if your assertion field\n  is not calculated in your table.\n\nConfigure the generic test in schema.yml with:\n\n```yml\nmodel:\n  name: my_model\n  tests:\n    - dbt_assertions.generic_assertions:\n      [column: \u003ccolumn_name\u003e]\n      [exclude_list: \u003clist(str_to_filter)\u003e]\n      [include_list: \u003clist(str_to_filter)\u003e]\n      [re_assert: true | false]\n\n  columns:\n    ...\n```\n\n`[]` represents optional parts. Yes everything is optional but let's see it by examples.\n\nIn the [basic test example](./examples/basic_test_example/)\nyou can easily create your test as follows then run your `dbt test` command.\n\n```yml\nmodels:\n  - name: basic_test_example_d_site\n    tests:\n      - dbt_assertions.generic_assertions:\n          column: exceptions\n          include_list:\n            - site_id_is_not_null\n          # `re_assert: true` to use only if your assertion's column\n          # is not computed and saved in your table.\n          re_assert: true\n\n    columns:\n      ...\n```\n\n### Model definition\n\n#### Yaml general definition\n\nThe assertions definition **must** be created\n**under a column definition of your model** and respects the following.\n\n```yml\nassertions:\n  [__unique__: \u003cunique_expression\u003e]\n  [__not_null__: __unique__ | \u003cnot_null_expression\u003e]\n\n  [\u003ccustom_assertion_id\u003e:\n    description: [\u003cstring\u003e]\n    expression: \u003cstring\u003e\n    null_as_exception: [\u003cbool\u003e]]\n  ...\n```\n\n`[]` represents optional parts.\nYes, everything is optional but let's see it by examples.\n\n#### Custom assertions\n\nCustom assertions are the basics assertions.\n\n\u003e The package is made to support every assertions as long as\n\u003e it is supported in a SELECT statement of your underlying database.\n\u003e **So you can do a lot of things**.\n\nIt is represented as key values. Keys are the ID of the assertions.\n\nEach assertions is defined by at least an `expression` which will be rendered\nto be evaluated as your test.\n\n`description` and [`null_as_exception`](#null_as_exception) are optional.\n\n```yml\nassertions:\n  unique:\n    description: \"Row must be unique.\"\n    expression: \"1 = COUNT(1) OVER(PARTITION by my_id)\"\n\n  site_creation_date_is_past:\n    description: \"Site must be created in the past.\"\n    expression: \"site_creation_date \u003c= CURRENT_DATE()\"\n\n  site_type_pick_list:\n    description: \"The site type be must in its known picklist.\"\n    expression: |\n        site_type IN (\n            'store',\n            'ecommerce',\n            'drive',\n            'pickup'\n        )\n```\n\n#### `null_as_exception`\n\n`null_as_exception` is an optional configuration for your assertion.\nDefault to `false` it is the return result if your expression is evaluated to `NULL`.\n\nDefault behaviour is set to `false` because one assertion must evaluate on thing.\nPrefer using the [`__not_null_`](#__not_null__-helper) helper instead.\n\nIn our previous, if we want to also avoid `NULL` site types.\n\n```yml\nassertions:\n  ...\n\n  site_type_pick_list:\n    description: \"The site type be must in its known picklist.\"\n    expression: |\n        site_type IN (\n            'store',\n            'ecommerce',\n            'drive',\n            'pickup'\n        )\n    null_as_exception: true\n```\n\n#### `__unique__` helper\n\nAs guaranteeing uniqueness of rows is a concern in most of the use cases,\nthe `__unique__` helper is here to avoid writing complex and repetitive queries.\n\n---\n\n```yml\nassertions:\n  unique:\n    description: \"Row must be unique.\"\n    expression: \"1 = COUNT(1) OVER(PARTITION by key_1, key_2)\"\n```\n\nThe above configuration is the same as writing\n\n```yml\nassertions:\n  __unique__:\n    - key_1\n    - key_2\n```\n\n---\n\nYou can also verify unique keys for nested/repeated structure. It will generate:\n- One assertion for the 0-depth guaranteeing uniqueness **across the rows**.\n- One assertion **for each** repeated field guaranteeing uniqueness **within the row**.\n\n\nThe following example will generate the assertions:\n- `unique`: Row must be unique over the unique keys.\n- `nested_1__unique`: Items must be unique **within** nested_1 in the row.\n- `nested_1.nested_2__unique`: Items must be unique **within** nested_1.nested_2 in the row.\n\n```yml\nassertions:\n  __unique__:\n    - key_1\n    - key_2\n    - nested_1:\n        - key_3\n        - key_4\n        - nested_2:\n            - key_5\n            - key_6\n```\n\n#### `__not_null__` helper\n\nAs guaranteeing not null values is also concern in most of the use cases,\nthe `__not_null__` helper is here to avoid writing complex and repetitive queries.\n\n---\n\n```yml\nassertions:\n  key_1__not_null:\n    description: \"key_1 is not null.\"\n    expression: \"key_1 IS NOT NULL\"\n\n  key_2__not_null:\n    description: \"key_2 is not null.\"\n    expression: \"key_2 IS NOT NULL\"\n```\n\nThe above configuration is the same as writing\n\n```yml\nassertions:\n  __unique__:\n    - key_1\n    - key_2\n\n  __not_null__:\n    - key_1\n    - key_2\n```\n\nAnd as the two helpers are often linked,\nyou can rewrite the assertions as follows, which is also the same.\n\n```yml\nassertions:\n  __unique__:\n    - key_1\n    - key_2\n\n  __not_null__: __unique__\n```\n\n---\n\nYou can also verify unique keys for nested/repeated structure. It will generate:\n- One assertion **for each column** of the 0-depth guaranteeing not null.\n- One assertion **for each column** under the repeated field guaranteeing\n  **all the values are not null within the row**.\n\nThe following example will generate the assertions:\n- `key_1__not_null`: key_1 is not null.\n- `key_2__not_null`: key_2 is not null.\n- `nested_1.key_3__not_null`: nested_1.key_3 **are** not null.\n\n```yml\nassertions:\n  __not_null__:\n    - key_1\n    - key_2\n    - nested_1:\n        - key_3\n```\n\n#### Custom column name\n\nIf `exceptions` column is not a naming convention you like,\nyou can still opt for a column name you choose and the macro will\nstill work with the `from_colum` argument.\n\nYou can also play with multiple columns.\n\n```yml\nmodel:\n  name: my_model\n  columns:\n    ...\n    - errors:\n      assertions:\n        __unique__:\n            - key_1\n            - key_2\n\n        __not_null__: __unique__\n\n    - warns:\n      assertions:\n        site_creation_date_is_past:\n          description: \"Site must be created in the past.\"\n          expression: \"site_creation_date \u003c= CURRENT_DATE()\"\n```\n\nAnd in your model query.\n\n```sql\nWITH final AS\n    (\n        SELECT ...\n    )\nSELECT\n    *,\n    {{ dbt_assertions.assertions(column='errors') }},\n    {{ dbt_assertions.assertions(column='warns') }}\nFROM {{ ref('my_model') }}\n```\n\n## Contribution\n\nIf you want to contribute, please open a Pull Request or an Issue on this repo.\nFeel free to reach me [Linkedin](https://www.linkedin.com/in/axel-thevenot/).\n\n## Acknowledgments\n\nSpecial thank to\n- [Victor Vaneecloo](https://www.linkedin.com/in/victorvaneecloo/) for the generic test\n- [Guillaume Blaquiere](https://www.linkedin.com/in/guillaume-blaquiere-38693b15/) for its help in terminology\n- [Daniel Bartley](https://www.linkedin.com/in/dbrtly/) for its help in terminology\n- [Benoit Perigaud](https://www.linkedin.com/in/benoit-perigaud/) for the Snowflake support\n- [Ferdy Hulzebos](https://www.linkedin.com/in/ferdyhulzebos/) for the Databricks support\n- [Rich Herman](https://www.linkedin.com/in/rich-herman-1149326/) for the Redshift support\n- [Vaisakh Mohan](https://www.linkedin.com/in/vaisakh-mohan-123b0936/) for the Athena support\n- [François Lenne](https://www.linkedin.com/in/fran%C3%A7ois-lenne-5975b9174/) for the Clickhouse support\n- [François Lenne](https://www.linkedin.com/in/fran%C3%A7ois-lenne-5975b9174/) for the Spark support\n\n## Contact\n\nIf you have any question, please open a new Issue or\nfeel free to reach out to [Linkedin](https://www.linkedin.com/in/axel-thevenot/)\n\n---\n\nHappy coding with **dbt-assertions**!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FAxelThevenot%2Fdbt-assertions","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FAxelThevenot%2Fdbt-assertions","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FAxelThevenot%2Fdbt-assertions/lists"}