{"id":18318676,"url":"https://github.com/autotraderuk/dbt-dry-run","last_synced_at":"2025-05-16T14:07:23.077Z","repository":{"id":38241280,"uuid":"458272485","full_name":"autotraderuk/dbt-dry-run","owner":"autotraderuk","description":"Dry run capability for dbt projects using BigQuery","archived":false,"fork":false,"pushed_at":"2025-04-11T15:30:24.000Z","size":872,"stargazers_count":97,"open_issues_count":10,"forks_count":13,"subscribers_count":11,"default_branch":"main","last_synced_at":"2025-05-14T08:57:48.899Z","etag":null,"topics":["bigquery","dbt","testing"],"latest_commit_sha":null,"homepage":"https://pypi.org/project/dbt-dry-run/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/autotraderuk.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGES.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-02-11T17:04:49.000Z","updated_at":"2025-04-29T06:32:54.000Z","dependencies_parsed_at":"2023-10-14T14:02:56.431Z","dependency_job_id":"287670b7-dea9-404b-8d47-6a1510070055","html_url":"https://github.com/autotraderuk/dbt-dry-run","commit_stats":{"total_commits":62,"total_committers":5,"mean_commits":12.4,"dds":0.3709677419354839,"last_synced_commit":"13938eb1ec5f68742dfc7133e8d54c381f5b6ae7"},"previous_names":[],"tags_count":29,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/autotraderuk%2Fdbt-dry-run","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/autotraderuk%2Fdbt-dry-run/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/autotraderuk%2Fdbt-dry-run/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/autotraderuk%2Fdbt-dry-run/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/autotraderuk","download_url":"https://codeload.github.com/autotraderuk/dbt-dry-run/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254544146,"owners_count":22088807,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bigquery","dbt","testing"],"created_at":"2024-11-05T18:11:14.207Z","updated_at":"2025-05-16T14:07:23.061Z","avatar_url":"https://github.com/autotraderuk.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# dbt-dry-run\n\n[dbt][dbt-home] is a tool that helps manage data transformations using templated SQL queries. These SQL queries are\nexecuted against a target data warehouse. It doesn't check the validity of SQL queries before it executes your project.\nThis dry runner uses BigQuery's [dry run][bq-dry-run] capability to allow you to check that SQL queries are valid before\ntrying to execute them.\n\nSee the [blog post][blog-post] for more information on how the dry runner works.\n\n![Terminal Recording of failing dry run](./terminal-demo.gif)\n\n## Quickstart\n\n### Installation\n\nThe dry runner can be installed via pip:\n\n`\npip install dbt-dry-run\n`\n\n### Running\n\nThe dry runner has a single command called `dbt-dry-run` in order for it to run you must first compile a dbt manifest\nusing `dbt compile`. \n\n\u003cdetails\u003e\n  \u003csummary\u003eHow much of the project should I compile?\u003c/summary\u003e\n      It is best practice to compile the entire dbt project when supplying a manifest for dry run. The \n      dry run loops through your project in the DAG order (staging -\u003e intermediate -\u003e mart) based on `ref` and predicts the \n      schema of each model as it progresses. If you dry run `marts` but have not compiled `staging` then it cannot \n      determine if `marts` will run as it does not know the predicted schema of the upstream models and you will see \n      `NotCompiledException` in the dry run output.\n\u003c/details\u003e\n\nThen on the same machine (So that the dry runner has access to your dbt project source and the\n`manifest.yml`) you can run the dry-runner in the same directory as our `dbt_project.yml`:\n\n```\ndbt-dry-run\n```\n\nLike dbt it will search for `profiles.yml` in `~/.dbt/` and use the default target specified. Just like in the dbt CLI\nyou can override these defaults:\n\n```\ndbt-dry-run default --project-dir /my_org_dbt/ --profiles-dir /my_org_dbt/profiles/ --target local\n```\n\nThe full CLI help is shown below, anything prefixed with [dbt] can be used in the same way as a normal dbt parameter:\n\n```\n  ❯ dbt-dry-run --help\n   Usage: dbt-dry-run [OPTIONS]\n   \n   Options:\n     --profiles-dir TEXT             [dbt] Where to search for `profiles.yml`\n                                     [default: /Users/connor.charles/.dbt]\n     --project-dir TEXT              [dbt] Where to search for `dbt_project.yml`\n                                     [default: /Users/connor.charles/Code/dbt-\n                                     dry-run]\n     --vars TEXT                     [dbt] CLI Variables to pass to dbt\n                                     [default: {}]\n     --target TEXT                   [dbt] Target profile\n     --target-path TEXT              [dbt] Target path\n     --verbose / --no-verbose        Output verbose error messages  [default: no-\n                                     verbose]\n     --report-path TEXT              Json path to dump report to\n     --skip-not-compiled             Whether or not the dry run should ignore\n                                     models that are not compiled. This has\n                                     several caveats that make this not a\n                                     recommended option. The dbt manifest should\n                                     generally be compiled with `--select *` to\n                                     ensure good  coverage\n     --full-refresh                  [dbt] Full refresh\n     --extra-check-columns-metadata-key TEXT\n                                     An extra metadata key that can be used in\n                                     place of `dry_run.check_columns` for\n                                     verifying column metadata has been specified\n                                     correctly. `dry_run.check_columns` will\n                                     always take precedence. The metadata key\n                                     should be of boolean type or it will be cast\n                                     to a boolean to be 'True/Falsey`\n     --version\n     --install-completion [bash|zsh|fish|powershell|pwsh]\n                                     Install completion for the specified shell.\n     --show-completion [bash|zsh|fish|powershell|pwsh]\n                                     Show completion for the specified shell, to\n                                     copy it or customize the installation.\n     --help                          Show this message and exit.\n```\n\n## Reporting Results \u0026 Failures\n\nIf the result is successful it will output the number of models that were tested like so:\n\n```\nDry running 3 models\n\nDRY RUN SUCCESS!\n```\n\nThe process will also return exit code 0\n\nIf there are failures it will print a summary table of the nodes that failed:\n\n```\nDry running 3 models\nNode model.test_models_with_invalid_sql.second_layer failed with exception:\n400 POST https://bigquery.googleapis.com/...: Column d in USING clause not found on left side of join at [6:88]\n\n(job ID: 5e336f32-273d-480a-b8bb-cdf4fca66a98)\n\nTotal 1 failures:\n1       :       model.test_models_with_invalid_sql.second_layer :       BadRequest      :       ERROR\nDRY RUN FAILURE!`\n```\n\nThe process will also return exit code 1\n\n### Column and Metadata Linting\n\nThe dry runner can also be configured to inspect your metadata YAML and assert that the predicted schema of your dbt\nprojects data warehouse matches what is documented in the metadata. To enable this for your models specify the key\n`dry_run.check_columns: true`. The dry runner will then fail if the model's documentation does not match. You can also\nspecify a custom extra key to enable `check_columns` by setting the CLI argument `--extra-check-columns-metadata-key`.\nFor example the full metadata for this model:\n\n```yaml\nmodels:\n  - name: badly_documented_model\n    description: This model is missing some columns in its docs\n    meta:\n      dry_run.check_columns: true\n    columns:\n      - name: a\n        description: This is in the model\n\n      - name: b\n        description: This is in the model\n\n      #      - name: c\n      #        description: Forgot to document c\n\n      - name: d\n        description: This shouldn't be here\n```\n\nThis model is badly documented as the predicted schema is 3 columns `a,b,c` the dry runner will therefore output the\nfollowing error and fail your CI/CD checks:\n\n```text\nDry running X models\nNode model.test_column_linting.badly_documented_model failed linting with rule violations:\n        UNDOCUMENTED_COLUMNS : Column not documented in metadata: 'c'\n        EXTRA_DOCUMENTED_COLUMNS : Extra column in metadata: 'd'\n\nTotal 1 failures:\n1       :       model.test_column_linting.badly_documented_model        :       LINTING :       ERROR\nDRY RUN FAILURE!\n```\n\nCurrently, these rules can cause linting failures:\n\n1. UNDOCUMENTED_COLUMNS: The predicted schema of the model will have extra columns that have not been documented in the\n   YAML\n2. EXTRA_DOCUMENTED_COLUMNS: The predicted schema of the model does not have this column that was specified in the\n   metadata\n\n### Usage with dbt-external-tables\n\nThe dbt package [dbt-external-tables][dbt-external-tables] gives dbt support for staging and managing\n[external tables][bq-external-tables]. These sources do not produce any compiled sql in the manifest, so it is not\npossible for the dry runner to predict their schema. Therefore, you must specify the resulting schema manually in the\nmetadata of the source. \n\nHowever, if the `columns` schema is already defined under the `name` in the yaml config, you do not need to specify `dry_run_columns` under `external`. The dry runner will use the `columns` schema if `dry_run_columns` is not specified. This avoids duplicated schema definitions.\n\nFor example if you were import data from a gcs bucket:\n\n```yaml\nversion: 2\n\nsources:\n  - name: source_dataset\n    tables:\n      - name: event\n        description: \"Some events bucket. If external is populated then the dry runner will assume it is using `dbt-external-tables`\"\n        external:\n          location: 'gs://bucket/path/*'\n            format: csv\n\n           dry_run_columns:\n             - name: string_field\n               data_type: STRING\n               description: \"Specify each column in the yaml for external sources\"\n             - name: record_array_field[]\n               data_type: RECORD[]\n               description: \"For struct/record fields specify the `data_type` as `RECORD`\"\n             - name: record_array_field.foo\n               data_type: NUMERIC\n               description: \"For record attributes use the dot notation\"\n             - name: integer_array\n               data_type: NUMERIC[]\n               description: \"For repeated fields suffix data_type with []\"\n```\n\nThe dry runner cannot predict the schema, therefore, it is up to you to accurately describe the schema in the YAML otherwise \nyou may get false positive/negative results from the dry run.\n\n### Report Artefact\n\nIf you specify `---report-path` a JSON file will be outputted regardless of dry run success/failure with detailed\ninformation of each node's predicted schema or error message if it has failed:\n\n```json\n{\n  \"success\": false,\n  \"node_count\": 3,\n  \"failure_count\": 1,\n  \"failed_node_ids\": [\n    \"model.test_models_with_invalid_sql.second_layer\"\n  ],\n  \"nodes\": [\n    {\n      \"unique_id\": \"seed.test_models_with_invalid_sql.my_seed\",\n      \"success\": true, \n      \"status\": \"SUCCESS\",\n      \"error_message\": null,\n      \"table\": {\n        \"fields\": [\n          ...\n        ]\n      }\n    },\n    {\n      \"unique_id\": \"model.test_models_with_invalid_sql.first_layer\",\n      \"success\": true,\n      \"status\": \"SUCCESS\",\n      \"error_message\": null,\n      \"table\": {\n        \"fields\": [\n          ...\n        ]\n      }\n    },\n    {\n      \"unique_id\": \"model.test_models_with_invalid_sql.second_layer\",\n      \"success\": false,\n      \"status\": \"FAILURE\",\n      \"error_message\": \"BadRequest\",\n      \"table\": null\n    }\n  ]\n}\n```\n\n## Capabilities and Limitations\n\n### Things this can catch\n\nThe dry run can catch anything the BigQuery planner can identify before the query has run. Which includes:\n\n1. Typos in SQL keywords:  `selec` instead of `select`\n2. Typos in columns names: `orders.produts` instead of `orders.products`\n3. Problems with incompatible data types: Trying to execute \"4\" + 4\n4. Incompatible schema changes to models: Removing a column from a view that is referenced by a downstream model\n   explicitly\n5. Incompatible schema changes to sources: Third party modifies schema of source tables without your knowledge\n6. Permission errors: The dry runner should run under the same service account your production job runs under. This\n   allows you to catch problems with table/project permissions as dry run queries need table read permissions just like\n   the real query\n7. Incorrect configuration of snapshots: For example a typo in the `unique_key` config. Or `check_cols` which do not\n   exist in the snapshot\n\n### Things this can't catch\n\nThere are certain cases where a syntactically valid query can fail due to the data in the tables:\n\n1. Queries that run but do not return intended/correct result. This is checked using [tests][dbt-tests]\n2. `NULL` values in `ARRAY_AGG` (See [IGNORE_NULLS bullet point][bq-ignore-nulls])\n3. Bad query performance that makes it too complex/expensive to run\n\n### Things still to do...\n\nThe implementation of seeds is incomplete as we don't use them very much in our own dbt projects. The dry runner\nwill just use the datatypes that `agate` infers from the CSV files. It will ignore any type overrides you add in the YAML.\n\nIf you see anything else that you think it should catch don't hesitate to raise an issue!\n\n[dbt-home]: https://www.getdbt.com/\n\n[bq-dry-run]: https://cloud.google.com/bigquery/docs/dry-run-queries\n\n[dbt-tests]: https://docs.getdbt.com/docs/building-a-dbt-project/tests\n\n[bq-ignore-nulls]: https://cloud.google.com/bigquery/docs/reference/standard-sql/aggregate_functions#array_agg\n\n[blog-post]: https://engineering.autotrader.co.uk/2022/04/06/dry-running-our-data-warehouse-using-bigquery-and-dbt.html\n\n[get-poetry]: https://python-poetry.org/\n\n[dbt-external-tables]: https://github.com/dbt-labs/dbt-external-tables\n\n[bq-external-tables]: https://cloud.google.com/bigquery/docs/external-tables\n\n## License\n\nCopyright 2022 Auto Trader Limited\n\nLicensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with the\nLicense. You may obtain a copy of the License at\n\nhttp://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"\nAS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific\nlanguage governing permissions and limitations under the License.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fautotraderuk%2Fdbt-dry-run","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fautotraderuk%2Fdbt-dry-run","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fautotraderuk%2Fdbt-dry-run/lists"}