{"id":13703793,"url":"https://github.com/brooklyn-data/dbt_artifacts","last_synced_at":"2025-05-05T07:31:42.167Z","repository":{"id":40269702,"uuid":"335676992","full_name":"brooklyn-data/dbt_artifacts","owner":"brooklyn-data","description":"A dbt package for modelling dbt metadata. https://brooklyn-data.github.io/dbt_artifacts","archived":false,"fork":false,"pushed_at":"2025-05-02T22:38:09.000Z","size":2472,"stargazers_count":358,"open_issues_count":57,"forks_count":139,"subscribers_count":21,"default_branch":"main","last_synced_at":"2025-05-02T23:27:41.730Z","etag":null,"topics":["dbt"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/brooklyn-data.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2021-02-03T15:57:10.000Z","updated_at":"2025-04-30T05:55:08.000Z","dependencies_parsed_at":"2022-07-09T06:46:07.464Z","dependency_job_id":"821e37af-8e1e-4bea-bde5-c51e37504a55","html_url":"https://github.com/brooklyn-data/dbt_artifacts","commit_stats":null,"previous_names":["tailsdotcom/dbt_artifacts"],"tags_count":51,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brooklyn-data%2Fdbt_artifacts","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brooklyn-data%2Fdbt_artifacts/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brooklyn-data%2Fdbt_artifacts/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brooklyn-data%2Fdbt_artifacts/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/brooklyn-data","download_url":"https://codeload.github.com/brooklyn-data/dbt_artifacts/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252458465,"owners_count":21751042,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dbt"],"created_at":"2024-08-02T21:01:00.201Z","updated_at":"2025-05-05T07:31:42.156Z","avatar_url":"https://github.com/brooklyn-data.png","language":"Shell","funding_links":[],"categories":["Packages","others"],"sub_categories":[],"readme":"# dbt Artifacts Package\n\nThis package builds a mart of tables and views describing the project it is installed in. In pre V1 versions of the package, the artifacts dbt produces were uploaded to the warehouse, hence the name of the package. That's no longer the case, but the name has stuck!\n\n[![Main branch test package](https://github.com/brooklyn-data/dbt_artifacts/actions/workflows/main_test_package.yml/badge.svg)](https://github.com/brooklyn-data/dbt_artifacts/actions/workflows/main_test_package.yml)\n[![Main branch lint package](https://github.com/brooklyn-data/dbt_artifacts/actions/workflows/main_lint_package.yml/badge.svg)](https://github.com/brooklyn-data/dbt_artifacts/actions/workflows/main_lint_package.yml)\n[![Documentation](https://github.com/brooklyn-data/dbt_artifacts/actions/workflows/publish_docs_on_release.yml/badge.svg)](https://github.com/brooklyn-data/dbt_artifacts/actions/workflows/publish_docs_on_release.yml)\n\n## Contributing\n\nContributions are welcome! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for more information.\n\n## Supported Data Warehouses\n\nThe package currently supports\n\n- Databricks :white_check_mark:\n- Spark :white_check_mark:\n- Snowflake :white_check_mark:\n- Google BigQuery :white_check_mark:\n- Postgres :white_check_mark:\n- SQL Server :white_check_mark:\n\nModels included:\n\n```\ndim_dbt__current_models\ndim_dbt__exposures\ndim_dbt__models\ndim_dbt__seeds\ndim_dbt__snapshots\ndim_dbt__sources\ndim_dbt__tests\nfct_dbt__invocations\nfct_dbt__model_executions\nfct_dbt__seed_executions\nfct_dbt__snapshot_executions\nfct_dbt__test_executions\n```\n\nSee the generated [dbt docs site](https://brooklyn-data.github.io/dbt_artifacts/#!/overview) for documentation on each model.\n\n## Quickstart\n\n1. Add this package to your `packages.yml`:\n\n```\npackages:\n  - package: brooklyn-data/dbt_artifacts\n    version: 2.9.2\n```\n\n:construction_worker: Make sure to fix at least the **minor** version, to avoid issues when a new release is open. See the notes on upgrading below for more detail.\n\n2. Run `dbt deps` to install the package\n\n3. Add an on-run-end hook to your `dbt_project.yml`\n\n    ```yml\n    on-run-end:\n      - \"{{ dbt_artifacts.upload_results(results) }}\"\n    ```\n\n    We recommend adding a conditional here so that the upload only occurs in your production environment, such as:\n\n    ```yml\n    on-run-end:\n      - \"{% if target.name == 'prod' %}{{ dbt_artifacts.upload_results(results) }}{% endif %}\"\n    ```\n\n4. Run the tables!\n\n    ```\n    dbt run --select dbt_artifacts\n    ```\n\n### Notes on upgrading\n\nDue to the structure of the project, when additional fields are added, the package needs to be re-run to ensure the tables include the new field, or it will simply error on the hook. These changes will always be implemented within a new **minor** version, so make sure that the version you use in `packages.yml` reflects this.\n\nTo upgrade and re-build, update the version number within `packages.yml` and then run:\n\n```\ndbt deps\ndbt run --select dbt_artifacts\n```\n\nMake sure this is updated in any database that you use your code base in.\n\n## Configuration\n\nThe following configuration can be used to specify where the raw (sources) data is uploaded, and where the dbt models are created:\n\n```yml\nmodels:\n  ...\n  dbt_artifacts:\n    +database: your_destination_database # optional, default is your target database\n    +schema: your_destination_schema # optional, default is your target schema\n    staging:\n      +database: your_destination_database # optional, default is your target database\n      +schema: your_destination_schema # optional, default is your target schema\n    sources:\n      +database: your_sources_database # optional, default is your target database\n      +schema: your sources_database # optional, default is your target schema\n```\n\nNote that model materializations and `on_schema_change` configs are defined in this package's `dbt_project.yml`, so do not set them globally in your `dbt_project.yml` ([see docs on configuring packages](https://docs.getdbt.com/docs/building-a-dbt-project/package-management#configuring-packages)):\n\n\u003e Configurations made in your dbt_project.yml file will override any configurations in a package (either in the dbt_project.yml file of the package, or in config blocks).\n\n### Environment Variables\n\nIf the project is running in dbt Cloud, the following five columns (\u003chttps://docs.getdbt.com/docs/dbt-cloud/using-dbt-cloud/cloud-environment-variables#special-environment-variables\u003e) will be automatically populated in the fct_dbt__invocations model:\n\n- dbt_cloud_project_id\n- dbt_cloud_job_id\n- dbt_cloud_run_id\n- dbt_cloud_run_reason_category\n- dbt_cloud_run_reason\n\nTo capture other environment variables in the fct_dbt__invocations model in the `env_vars` column, add them to the `env_vars` variable in your `dbt_project.yml`. Note that environment variables with secrets (`DBT_ENV_SECRET_`) can't be logged.\n\n```yml\nvars:\n  env_vars: [\n    'ENV_VAR_1',\n    'ENV_VAR_2',\n    '...'\n  ]\n```\n\n### dbt Variables\n\nTo capture dbt variables in the fct_dbt__invocations model in the `dbt_vars` column, add them to the `dbt_vars` variable in your `dbt_project.yml`.\n\n```yml\nvars:\n  dbt_vars: [\n    'var_1',\n    'var_2',\n    '...'\n  ]\n```\n\n## Creating custom marts tables\n\nMultiple modelled `dim` and `fct` models have been provided for ease of use, but we recognise that some use cases may require custom ones. To this end, you can disable all but the raw sources tables using the following in your `dbt_project.yml` file:\n\n```yml\n# dbt_project.yml\n\nmodels:\n  dbt_artifacts:\n    +enabled: false\n    sources:\n      +enabled: true\n```\n\nIn these sources tables, you will find a JSON column `all_results` which contains a JSON blob of the results object used, which you can use in your own analysis:\n\n- exposures\n- models\n- seeds\n- snapshots\n- sources\n- tests\n\nThis column can cause queries to become too long - particularly in BigQuery. Therefore, if you want to disable this column, you can make use of the `dbt_artifacts_exclude_all_results` variable, and set this to `true` in your `dbt_project.yml` file.\n\n```\n# dbt_project.yml\nvars:\n  dbt_artifacts_exclude_all_results: true\n```\n\n## Upgrading from 1.x to \u003e=2.0.0\n\nIf you were using the following variables:\n\n```yml\nvars:\n  dbt_artifacts_database: your_db\n  dbt_artifacts_schema: your_schema\n```\n\nYou must now move these to the following model configs:\n\n```yml\nmodels:\n  ...\n  dbt_artifacts:\n    sources:\n      +database: your_db\n      +schema: your_schema\n```\n\nThat's because the raw tables are now managed as dbt models. Be aware of any impact that [generate_database_name](https://docs.getdbt.com/docs/building-a-dbt-project/building-models/using-custom-databases#generate_database_name) and [generate_schema_name](https://docs.getdbt.com/docs/building-a-dbt-project/building-models/using-custom-schemas#how-does-dbt-generate-a-models-schema-name) macros may have on the final database/schema.\n\n## Migrating From \u003c1.0.0 to \u003e=1.0.0\n\nTo migrate your existing data from the `dbt-artifacts` versions \u003c=0.8.0, a helper macro and guide is provided. This migration uses the old `fct_*` and `dim_*` models' data to populate the new sources. The steps to use the macro are as follows:\n\n1. If not already completed, run `dbt run-operation create_dbt_artifacts_tables` to make your source tables.\n2. Run `dbt run-operation migrate_from_v0_to_v1 --args '\u003csee-below-for-arguments\u003e'`.\n3. Verify that the migration completes successfully.\n4. Manually delete any database objects (sources, staging models, tables/views) from the previous `dbt-artifacts` version.\n\nThe arguments for `migrate_from_v0_to_v1` are as follows:\n| argument      | description                                               |\n|-------------- |---------------------------------------------------------- |\n| `old_database`  | the database of the \u003c1.0.0 output (`fct_`/`dim_`) models  |\n| `old_schema`    | the schema of the \u003c1.0.0 output (`fct_`/`dim_`) models    |\n| `new_database`  | the target database that the artifact sources are in      |\n| `new_schema`    | the target schema that the artifact sources are in        |\n\nThe old and new database/schemas *do not* have to be different, but it is explicitly defined for flexible support.\n\nAn example operation is as follows:\n\n```bash\ndbt run-operation migrate_from_v0_to_v1 --args '{old_database: analytics, old_schema: dbt_artifacts, new_database: analytics, new_schema: artifact_sources}'\n```\n\n## Acknowledgements\n\nThank you to [Tails.com](https://tails.com/gb/careers/) for initial development and maintenance of this package. On 2021/12/20, the repository was transferred from the Tails.com GitHub organization to Brooklyn Data Co.\n\nThe macros in the early versions package were adapted from code shared by [Kevin Chan](https://github.com/KevinC-wk) and [Jonathan Talmi](https://github.com/jtalmi) of [Snaptravel](snaptravel.com).\n\nThank you for sharing your work with the community!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbrooklyn-data%2Fdbt_artifacts","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbrooklyn-data%2Fdbt_artifacts","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbrooklyn-data%2Fdbt_artifacts/lists"}