{"id":23885961,"url":"https://github.com/dpguthrie/disco-api-persist-docs","last_synced_at":"2026-06-22T07:30:18.117Z","repository":{"id":268009840,"uuid":"903010938","full_name":"dpguthrie/disco-api-persist-docs","owner":"dpguthrie","description":"A Python tool that extends dbt's persist_docs functionality by automatically syncing inherited column descriptions from dbt to BigQuery tables.","archived":false,"fork":false,"pushed_at":"2024-12-13T18:30:01.000Z","size":9,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-04T05:56:08.932Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dpguthrie.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-12-13T18:28:26.000Z","updated_at":"2024-12-13T18:30:06.000Z","dependencies_parsed_at":"2024-12-13T19:32:14.453Z","dependency_job_id":"200c0b9c-51bd-499a-af9a-ee0ea8ddc863","html_url":"https://github.com/dpguthrie/disco-api-persist-docs","commit_stats":null,"previous_names":["dpguthrie/disco-api-persist-docs"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpguthrie%2Fdisco-api-persist-docs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpguthrie%2Fdisco-api-persist-docs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpguthrie%2Fdisco-api-persist-docs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpguthrie%2Fdisco-api-persist-docs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dpguthrie","download_url":"https://codeload.github.com/dpguthrie/disco-api-persist-docs/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240265405,"owners_count":19774069,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-01-04T05:56:12.724Z","updated_at":"2026-06-22T07:30:18.045Z","avatar_url":"https://github.com/dpguthrie.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# dbt Column Description Inheritance\n\nThis tool solves a limitation in dbt's `persist_docs` functionality by persisting inherited column descriptions to BigQuery tables. While dbt's built-in `persist_docs` feature can persist manually written column descriptions, it doesn't handle descriptions that are inherited from upstream columns.\n\n## Overview\n\nWhen using dbt's column description inheritance (using `description:` in your `.yml` files), the descriptions are visible in dbt docs but don't get persisted to the actual BigQuery tables. This script:\n\n1. Queries the dbt Cloud Metadata API to get all models in your environment\n2. Identifies columns that have inherited descriptions from upstream columns\n3. Updates the BigQuery table schemas to include these inherited descriptions\n\n## Prerequisites\n\n- A dbt Cloud account with API access\n- Google Cloud credentials with BigQuery access\n- Python 3.x\n\n## Setup\n\n1. Install required dependencies:\n\n```bash\npip install -r requirements.txt\n```\n\n2. Set up environment variables in `.env`:\n```\nDBT_CLOUD_SERVICE_TOKEN=\nDBT_CLOUD_HOST=\nDBT_CLOUD_ACCOUNT_ID=\nDBT_CLOUD_ENVIRONMENT_ID=\n```\n\n3. Place your Google Cloud service account JSON key in the root directory as `service_account.json`\n\n## Usage\n\nRun the script:\n```bash\npython main.py\n```\n\nThe script will:\n- Fetch all models from your dbt Cloud environment\n- Identify columns with inherited descriptions\n- Update BigQuery table schemas for affected tables\n- Print confirmation messages for each updated table\n\n## How It Works\n\n1. The script uses the dbt Cloud Metadata API to query information about your models, including column descriptions and their origins\n2. For each model, it checks if any columns have inherited descriptions (indicated by `descriptionOriginColumnName`)\n3. If inherited descriptions are found, it updates the BigQuery table schema to include these descriptions\n4. The updates are made using the BigQuery API, ensuring the descriptions persist even after future dbt runs\n\n## Limitations\n\n- Only works with BigQuery\n- Requires dbt Cloud (not compatible with dbt Core)\n- Updates are made directly to BigQuery tables, outside of dbt's normal workflow\n- Descriptions will be overwritten by subsequent dbt runs, so this script should either:\n  - Be configured as a webhook to run after dbt Cloud job completion\n  - Be scheduled to run periodically after your dbt jobs complete\n  - Be integrated into your dbt job orchestration workflow\n\n## Contributing\n\nFeel free to open issues or submit pull requests if you have suggestions for improvements or bug fixes.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdpguthrie%2Fdisco-api-persist-docs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdpguthrie%2Fdisco-api-persist-docs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdpguthrie%2Fdisco-api-persist-docs/lists"}