{"id":23885970,"url":"https://github.com/dpguthrie/dbt-cloud-column-aware-ci","last_synced_at":"2026-06-22T07:30:18.229Z","repository":{"id":270051579,"uuid":"909195485","full_name":"dpguthrie/dbt-cloud-column-aware-ci","owner":"dpguthrie","description":"A tool for running dbt Cloud CI jobs with column-level change awareness","archived":false,"fork":false,"pushed_at":"2025-01-20T20:53:10.000Z","size":1128,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-20T21:28:12.382Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dpguthrie.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-12-28T01:48:36.000Z","updated_at":"2025-01-20T20:51:51.000Z","dependencies_parsed_at":"2025-01-13T01:19:02.216Z","dependency_job_id":"421fe207-8849-4a11-a28b-4ba5d49da840","html_url":"https://github.com/dpguthrie/dbt-cloud-column-aware-ci","commit_stats":null,"previous_names":["dpguthrie/dbt-cloud-column-aware-ci"],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpguthrie%2Fdbt-cloud-column-aware-ci","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpguthrie%2Fdbt-cloud-column-aware-ci/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpguthrie%2Fdbt-cloud-column-aware-ci/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpguthrie%2Fdbt-cloud-column-aware-ci/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dpguthrie","download_url":"https://codeload.github.com/dpguthrie/dbt-cloud-column-aware-ci/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240265406,"owners_count":19774069,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-01-04T05:56:13.231Z","updated_at":"2026-06-22T07:30:18.119Z","avatar_url":"https://github.com/dpguthrie.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# dbt Cloud Column-Aware CI Action\n\nThis GitHub Action enables column-aware CI for dbt Cloud Enterprise accounts by leveraging dbt Cloud's column-level lineage feature. It intelligently determines which downstream models need to be rebuilt based on column-level changes in your dbt models.\n\n## Demo\n\nhttps://www.loom.com/share/c9ddaaa259b8413c9ab09adb670fd996?sid=c03617bd-2743-43c8-bb68-c2e9502a0a2b\n\n## Overview\n\nTraditional dbt CI runs rebuild all downstream dependencies when a model changes. This action optimizes CI runs by:\n1. Analyzing which columns have changed in modified models\n2. Using dbt Cloud's column-level lineage to identify affected downstream models\n3. Excluding unaffected downstream models from the CI run\n\nThis results in faster CI runs and more efficient use of warehouse resources.\n\n## Benefits\n\n- **Faster CI Runs**: Only rebuild models that are actually impacted by column changes\n- **Resource Optimization**: Reduce warehouse costs by skipping unnecessary model runs\n- **Enhanced Developer Experience**: Get faster feedback on your PRs\n- **Enterprise Integration**: Seamlessly works with dbt Cloud Enterprise features\n\n## Prerequisites\n\n- dbt Cloud Enterprise account\n- A `dbt docs generate` command should be run in at least one job in your environment that the CI job defers to.  This is what enables column-level lineage.  More info [here](https://docs.getdbt.com/docs/collaborate/column-level-lineage#access-the-column-level-lineage).\n- A dbt Cloud [Personal access token](https://docs.getdbt.com/docs/dbt-cloud-apis/user-tokens#create-a-personal-access-token)\n- A dbt Cloud [Service token](https://docs.getdbt.com/docs/dbt-cloud-apis/service-tokens) with the following permissions:\n\n| Permission | Usage |\n|-------|-------------|\n| Metadata | Used to return column-level lineage and compiled code |\n| Job Runner | Used to trigger the CI job configured in the workflow |\n| Job Viewer | Used to infer the deferring environment ID if not given as part of the workflow inputs |\n\n## Inputs\n\n| Input | Description | Required | Default |\n|-------|-------------|----------|---------|\n| `dbt_cloud_account_id` | dbt Cloud Account ID | Yes | - |\n| `dbt_cloud_job_id` | dbt Cloud CI Job ID for the current project | Yes | - |\n| `dbt_cloud_service_token` | dbt Cloud Service Token | Yes | - |\n| `dbt_cloud_token_name` | Name of the personal API Key created in dbt Cloud | Yes | - |\n| `dbt_cloud_token_value` | dbt Cloud Personal API Key for use with the dbt Cloud CLI | Yes | - |\n| `dialect` | SQL dialect of your warehouse (e.g., 'snowflake') | Yes | - |\n| `dbt_cloud_host` | dbt Cloud host | No | cloud.getdbt.com |\n| `dry_run` | When true, analyzes changes but doesn't trigger dbt Cloud job | No | false |\n| `github_token` | GitHub token for API authentication | No | ${{ github.token }} |\n| `log_level` | Logging level (DEBUG, INFO, WARNING, ERROR, CRITICAL) | No | INFO |\n\nThe dialect refers to your data platform where your dbt project is being executed.  Valid dialects include:\n- athena\n- bigquery\n- databricks\n- postgres\n- redshift\n- snowflake\n- spark\n- trino\n\n## Usage\n\nHere's an example workflow that uses this action:\n\n```yaml\nname: dbt Cloud CI\n  on:\n    pull_request:\n      branches: [ main ]\n\njobs:\n  dbt-cloud-ci:\n    runs-on: ubuntu-latest\n    steps:\n    - uses: actions/checkout@v3\n    - name: Run Column-Aware dbt Cloud CI\n      uses: dpguthrie/dbt-cloud-column-aware-ci@0.5.2\n      with:\n        dbt_cloud_service_token: ${{ secrets.DBT_CLOUD_SERVICE_TOKEN }}\n        dbt_cloud_token_name: 'github-actions'\n        dbt_cloud_token_value: ${{ secrets.DBT_CLOUD_TOKEN_VALUE }}\n        dbt_cloud_account_id: '12345'\n        dbt_cloud_job_id: '98765'\n        dialect: 'snowflake'\n        log_level: 'DEBUG' # optional\n```\n\n## Recommended Use\n\nTo get the most out of column-aware CI, it's recommended to set up your dbt Cloud environment as follows:\n\n1. Create a merge job in your environment that the CI job defers to.  This will run when PRs are merged to your configured branch.\n2. Configure the merge job to run at minimum:\n   ```bash\n   dbt docs generate\n   ```\n   This step is crucial because it:\n   - Recalculates column-level lineage information\n   - Generates an updated `manifest.json` file\n   - Ensures accurate state comparison for subsequent CI runs using `state:modified`\n\nThe merge job keeps your deferred environment's state current, which enables this action to accurately determine which models need to be rebuilt based on column-level changes.\n\n## How It Works\n\n1. The action identifies modified models using dbt's state comparison\n2. For each modified model, it:\n   - Compiles the current and previous compiled code.  The current code is retrieved from running a `dbt compile` and the previous code is retrieved from the discovery API.\n   - Create a diff between the current and previous code.  Specifically, looking for changes to columns\n   - Queries dbt Cloud's Discovery API to find impacted downstream models from the columns that had changes\n3. Creates a filtered CI run that excludes unaffected downstream models\n4. Monitors the job run and reports status back to GitHub\n\nSee example of what the flow looks like below:\n\n![Example of flow](./assets/column_aware_ci_flow.png)\n\n## Examples\n\nThe following examples demonstrate different types of schema changes and their impact:\n\n### Non-Breaking Change\nAdding a new column to a table is considered a non-breaking change, which means no models downstream need to be run\n![Non-Breaking Change](assets/non_breaking_change.png)\n\n### Column Breaking Change\nModifying an existing column is a breaking change that **could** impact downstream models.  **Only models referencing that modified column will be run as part of CI.**\n![Column Breaking Change](assets/column_breaking_change.png)\n\n### Node Breaking Change\nModifying a where clause has the potential to break any models downstream of that change, so nothing downstream of this  model will be excluded.\n![Node Breaking Change](assets/node_breaking_change.png)\n\n## Codebase Structure\n\n### Core Components\n\n#### Configuration (`src/config.py`)\n- Manages application configuration through the `Config` class\n- Handles dbt Cloud credentials and settings\n- Provides environment variable parsing via `from_env()`\n\n#### Main Entry Point (`src/main.py`)\n- Sets up logging configuration\n- Initializes the application configuration\n- Creates and runs the CI orchestrator\n- Handles top-level error handling\n\n### Models (`src/models/`)\n\n- `Node`: Represents a dbt model with source and target code\n- `NodeFactory`: Creates Node instances from raw data\n- `NodeManager`: Manages collections of nodes and their dependencies\n- `BreakingChange`: Analyzes SQL changes to detect breaking modifications\n- `ColumnTracker`: Tracks column-level changes across models\n\n### Services (`src/services/`)\n\n#### Core Services\n- `CiOrchestrator`: Coordinates the entire CI workflow\n- `DbtRunner`: Handles dbt CLI command execution\n- `DiscoveryClient`: Interfaces with dbt Cloud's Discovery API\n- `LineageService`: Manages model lineage information\n\n#### Interface Definitions (`src/interfaces/`)\n- Defines protocol classes for key components\n- Ensures consistent implementation across services\n- Includes protocols for:\n  - `DbtRunnerProtocol`\n  - `DiscoveryClientProtocol`\n  - `LineageServiceProtocol`\n  - `OrchestratorProtocol`\n\n### Utilities\n\n#### Support Functions (`src/utils.py`)\n- Contains helper functions for:\n  - Creating dbt Cloud profiles\n  - Triggering dbt Cloud jobs\n  - Managing job run statuses\n\n#### GraphQL Queries (`src/discovery_api_queries.py`)\n- Defines GraphQL queries for the Discovery API\n- Includes queries for:\n  - Column lineage\n  - Compiled code\n  - Node lineage\n\n#### Logging Configuration (`src/logging_config.py`)\n- Sets up standardized logging across the application\n- Configures console output formatting\n- Defines log levels and handlers\n\n## Caveats\n\n- Only been tested with Snowflake\n- Assumes that your column names are **not** case sensitive.\n- The dbt Cloud CLI is used to run dbt commands `compile` and `ls`, which means that it needs a personal access token and is at the moment scoped to a particular user.  The job itself that is triggered at the end of the workflow would still use the credentials configured for the enviroment it's running in.\n- The `favor-state` flag is used when compiling the target SQL.  This is done to try and minimize any changes that are picked up solely because of environment separation (e.g. db.my_dev_schema.dim_customers vs. db.my_prod_schema.dim_customers).  However, this doesn’t apply if the node is also part of the selected nodes.  See example below when running `dbt compile -s state:modified --favor-state`:\n\n![Example of favor-state flag](./assets/favor_state.png)\n\n## Contributing\n\nContributions are welcome! Please feel free to submit a Pull Request.\n\n## License\n\nThis project is licensed under the terms of the MIT license.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdpguthrie%2Fdbt-cloud-column-aware-ci","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdpguthrie%2Fdbt-cloud-column-aware-ci","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdpguthrie%2Fdbt-cloud-column-aware-ci/lists"}