{"id":13703333,"url":"https://github.com/kgmcquate/dbt-testgen","last_synced_at":"2025-05-05T07:30:45.910Z","repository":{"id":212118665,"uuid":"729854668","full_name":"kgmcquate/dbt-testgen","owner":"kgmcquate","description":"Generate DBT tests based on sample data","archived":false,"fork":false,"pushed_at":"2024-02-28T04:50:18.000Z","size":985,"stargazers_count":36,"open_issues_count":5,"forks_count":4,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-05-02T02:49:31.933Z","etag":null,"topics":["dbt","dbt-packages","sql","testing-tools"],"latest_commit_sha":null,"homepage":"https://kgmcquate.github.io/dbt-testgen/","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kgmcquate.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-12-10T15:07:34.000Z","updated_at":"2025-02-10T09:30:10.000Z","dependencies_parsed_at":"2024-02-28T04:23:57.389Z","dependency_job_id":"c29c81db-670b-455d-8627-be2f77a689be","html_url":"https://github.com/kgmcquate/dbt-testgen","commit_stats":null,"previous_names":["kgmcquate/dbt-testgen"],"tags_count":14,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kgmcquate%2Fdbt-testgen","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kgmcquate%2Fdbt-testgen/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kgmcquate%2Fdbt-testgen/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kgmcquate%2Fdbt-testgen/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kgmcquate","download_url":"https://codeload.github.com/kgmcquate/dbt-testgen/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252458339,"owners_count":21751019,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dbt","dbt-packages","sql","testing-tools"],"created_at":"2024-08-02T21:00:53.513Z","updated_at":"2025-05-05T07:30:45.392Z","avatar_url":"https://github.com/kgmcquate.png","language":"Shell","funding_links":[],"categories":["Packages"],"sub_categories":[],"readme":"# dbt-testgen\n\n- [dbt-testgen](#dbt-testgen)\n- [About](#about)\n- [Install](#install)\n- [Supported Databases](#supported-databases)\n- [Test types](#test-types)\n- [Usage](#usage)\n- [Macros](#macros)\n  - [get\\_test\\_suggestions](#get_test_suggestions)\n\n\n# About\n`dbt-testgen` is a [dbt](https://github.com/dbt-labs/dbt) package that autogenerates dbt test YAML based on real data.\n\nCode documentation available at [here](https://kgmcquate.github.io/dbt-testgen/)\n\nInspired by [dbt-codegen](https://github.com/dbt-labs/dbt-codegen) and [deequ Constraint Suggestion](https://github.com/awslabs/deequ/blob/master/src/main/scala/com/amazon/deequ/examples/constraint_suggestion_example.md).\n\n# Install\n`dbt-testgen` currently supports `dbt 1.2.x` or higher.\n\nInclude in `packages.yml`:\n```yaml\npackages:\n  - git: https://github.com/kgmcquate/dbt-testgen\n```\n\n# Supported Databases\nThe following databases are supported:\n- Snowflake\n- Databricks\n- RedShift\n- BigQuery\n- Postgres\n- DuckDB\n\nIntegration tests are run for each of these databases in [Actions](https://github.com/kgmcquate/dbt-testgen/actions).\n\n# Test types\ndbt-testgen can generate these types of tests, using [built-in tests](https://docs.getdbt.com/reference/resource-properties/data-tests), [dbt_utils](https://github.com/dbt-labs/dbt-utils), and [dbt-expectations](https://github.com/calogica/dbt-expectations/):\n- [uniqueness](https://github.com/dbt-labs/dbt-utils?tab=readme-ov-file#unique_combination_of_columns-source)\n- [not_null](https://docs.getdbt.com/reference/resource-properties/data-tests#not_null)\n- [string length](https://github.com/calogica/dbt-expectations/tree/main?tab=readme-ov-file#expect_column_value_lengths_to_be_between)\n- [range](https://github.com/dbt-labs/dbt-utils?tab=readme-ov-file#accepted_range-source)\n- [accepted_values](https://docs.getdbt.com/reference/resource-properties/data-tests#accepted_values)\n- [recency](https://github.com/dbt-labs/dbt-utils?tab=readme-ov-file#recency-source)\n\n# Usage\nThe DBT config YAML is generated by a Jinja macro, `get_test_suggestions`, which you can run like this:\n```powershell\ndbt compile -q --inline \"{{ testgen.get_test_suggestions(ref('mymodel')) }}\"\n```\nOutput:\n```yaml\nmodels:\n- name: mymodel\n  tests:\n  - dbt_utils.recency:\n      field: day\n      datepart: day\n      interval: 2\n  columns:\n  - name: user_id\n    description: Numeric range test generated by dbt-testgen\n    tests:\n    - unique\n    - not_null\n    - dbt_utils.accepted_range:\n        min_value: 1\n        max_value: 30\n  - name: username\n    tests:\n    - unique\n    - not_null\n    - dbt_expectations.expect_column_value_lengths_to_be_between:\n        min_value: 8\n        max_value: 15\n        row_condition: '\"username\" is not null'\n  - name: email\n    tests:\n    - unique\n    - not_null\n    - dbt_expectations.expect_column_value_lengths_to_be_between:\n        min_value: 18\n        max_value: 25\n        row_condition: '\"email\" is not null'\n  - name: user_status\n    tests:\n    - accepted_values:\n        values:\n        - active\n        - inactive\n    - dbt_expectations.expect_column_value_lengths_to_be_between:\n        min_value: 6\n        max_value: 8\n        row_condition: '\"user_status\" is not null'\n  - name: age\n    tests:\n    - dbt_utils.accepted_range:\n        min_value: 22\n        max_value: 35\n```\n\n\u003chr\u003e\n\u003cbr\u003e\n\nYou can output to a file like this:\n```yaml\ndbt compile -q --inline \"{{ testgen.get_test_suggestions(ref('mymodel')) }}\" \u003e\u003e models/schema.yml\n```\n\n\u003chr\u003e\n\u003cbr\u003e\n\nYou can also merge with an existing properties YAML file:\n```bash\nEXISTING_YAML_BODY=`cat models/schema.yml`\ndbt compile -q --inline \"{{ testgen.get_test_suggestions(ref('users'), dbt_config=fromyaml(\\\"${EXISTING_YAML_BODY}\\\")) }}\"\n```\n\n\u003chr\u003e\n\u003cbr\u003e\n\nHere's an example of more advanced usage:\n```bash\nEXISTING_YAML_BODY=$(cat \u003c\u003cEOF\nmodels:\n- name: stg_jaffle_shop__customers\n  config:\n    tags:\n    - pii\n  columns:\n  - name: customer_id\n    tests:\n    - unique\n    - not_null\n\nseeds:\n- name: stg_jaffle_shop__orders\n  config:\n    materialized: view\n  columns:\n  - name: order_id\n    tests:\n    - unique\n    - not_null\nEOF\n)\n\nJINJA_TEMPLATE=$(cat \u003c\u003cEOF\n{{ testgen.get_test_suggestions(\n    ref('users'),\n    sample = true,\n    limit = 1000,\n    resource_type = 'seeds',\n    column_config = {'quote': true, 'tags': ['generated']},\n    exclude_types = ['float'],\n    tests = ['uniqueness', 'accepted_values', 'range'],\n    uniqueness_composite_key_length = 2,\n    accepted_values_max_cardinality = 10,\n    range_stddevs = 1,\n    dbt_config=fromyaml(\"${EXISTING_YAML_BODY}\")\n) }}\nEOF\n)\n\ndbt compile -q --inline \"${JINJA_TEMPLATE}\"\n```\nOutput:\n```yaml\nmodels:\n- name: stg_jaffle_shop__customers\n  config:\n    tags:\n    - pii\n  columns:\n  - name: customer_id\n    tests:\n    - unique\n    - not_null\nseeds:\n- name: stg_jaffle_shop__orders\n  columns:\n  - name: order_id\n    tests:\n    - unique\n    - not_null\n- name: users\n  columns:\n  - name: user_id\n    tests:\n    - unique\n    - not_null\n    - dbt_utils.accepted_range:\n        min_value: -3.4017042154147523\n        max_value: 34.40170421541475\n    quote: true\n    tags:\n    - generated\n  - name: username\n    tests:\n    - unique\n    - not_null\n    quote: true\n    tags:\n    - generated\n  - name: email\n    tests:\n    - unique\n    - not_null\n    quote: true\n    tags:\n    - generated\n  - name: user_status\n    tests:\n    - accepted_values:\n        values:\n        - active\n        - inactive\n    quote: true\n    tags:\n    - generated\n  - name: age\n    tests:\n    - dbt_utils.accepted_range:\n        min_value: 20.462630250354053\n        max_value: 36.53736974964595\n    quote: true\n    tags:\n    - generated\n```\n\n\n# Macros\n## get_test_suggestions \n- [documentation](https://kgmcquate.github.io/dbt-testgen/#!/macro/macro.testgen.get_test_suggestions)\n- [source](macros/test_aggregation/get_test_suggestions.sql)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkgmcquate%2Fdbt-testgen","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkgmcquate%2Fdbt-testgen","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkgmcquate%2Fdbt-testgen/lists"}