{"id":13703062,"url":"https://github.com/nicholasyager/dbt-loom","last_synced_at":"2026-01-24T22:28:51.926Z","repository":{"id":181311545,"uuid":"666556220","full_name":"nicholasyager/dbt-loom","owner":"nicholasyager","description":"A dbt-core plugin to weave together multi-project dbt-core deployments","archived":false,"fork":false,"pushed_at":"2025-01-28T17:26:22.000Z","size":7281,"stargazers_count":141,"open_issues_count":12,"forks_count":21,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-03-29T10:04:47.745Z","etag":null,"topics":["dbt","dbt-core","plugin","python3"],"latest_commit_sha":null,"homepage":"https://nicholasyager.github.io/dbt-loom/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"unlicense","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nicholasyager.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"docs/CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"docs/CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-07-14T20:39:14.000Z","updated_at":"2025-03-20T04:12:58.000Z","dependencies_parsed_at":"2023-07-14T22:39:30.686Z","dependency_job_id":"45f20112-4c77-4d59-a752-f64c4282a701","html_url":"https://github.com/nicholasyager/dbt-loom","commit_stats":null,"previous_names":["nicholasyager/dbt-loom"],"tags_count":19,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nicholasyager%2Fdbt-loom","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nicholasyager%2Fdbt-loom/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nicholasyager%2Fdbt-loom/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nicholasyager%2Fdbt-loom/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nicholasyager","download_url":"https://codeload.github.com/nicholasyager/dbt-loom/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247325693,"owners_count":20920714,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dbt","dbt-core","plugin","python3"],"created_at":"2024-08-02T21:00:49.768Z","updated_at":"2026-01-24T22:28:51.918Z","avatar_url":"https://github.com/nicholasyager.png","language":"Python","funding_links":[],"categories":["Utilities"],"sub_categories":[],"readme":"# dbt-loom\n\n[![pypi version shield](https://img.shields.io/pypi/v/dbt-loom)](https://img.shields.io/pypi/v/dbt-loom)\n\ndbt-loom is a dbt Core plugin that weaves together multi-project deployments. dbt-loom works by fetching public model definitions from your dbt artifacts, and injecting those models into your dbt project.\n\n```mermaid\nflowchart LR\n\n    classDef black fill:#f2f2ebff, stroke:#000, color:#000\n    classDef background fill:#f2f2ebff, stroke:#000, color:#000\n    classDef hidden fill:#BADC3F, stroke:#BADC3F, color:#BADC3F\n\n   style TOP fill:#BADC3F, stroke:#000\n\n  subgraph TOP[Your Infrastructure]\n    direction TB\n    dbt_runtime[dbt Core]:::background\n    proprietary_plugin[Open Source Metadata Plugin]:::background\n\n    files[Local and Remote Files]:::background\n    object_storage[Object Storage]:::background\n    data_warehouse_storage[Data Warehouse Storage]:::background\n    discovery_api[dbt-core Hosting Providers]:::background\n\n    discovery_api --\u003e proprietary_plugin\n    files --\u003e proprietary_plugin\n    object_storage --\u003e proprietary_plugin\n    data_warehouse_storage --\u003e proprietary_plugin\n    proprietary_plugin --\u003e dbt_runtime\n  end\n\n  Project:::black --\u003e TOP --\u003e Warehouse:::black\n```\n\ndbt-loom currently supports obtaining model definitions from:\n\n- Local manifest files\n- Remote manifest files via http(s)\n- `dbt-core` Hosting Providers\n  - dbt Cloud\n  - Datacoves\n  - Paradime\n- Object Storage\n  - GCS\n  - S3-compatible object storage services\n  - Azure Storage\n- Database Warehouse Storage\n  - Snowflake stages\n  - Databricks Volume, DBFS, and Workspace locations\n\n## Getting Started\n\nTo begin, install the `dbt-loom` python package.\n\n```console\npip install dbt-loom\n```\n\nNext, create a `dbt-loom` configuration file. This configuration file provides the paths for your\nupstream project's manifest files.\n\n```yaml\nmanifests:\n  - name: project_name # This should match the project's real name\n    type: file\n    config:\n      # A path to your manifest. This can be either a local path, or a remote\n      # path accessible via http(s).\n      path: path/to/manifest.json\n```\n\nBy default, `dbt-loom` will look for `dbt_loom.config.yml` in your working directory. You can also set the\n`DBT_LOOM_CONFIG` environment variable.\n\n## How does it work?\n\nAs of dbt-core 1.6.0-b8, there now exists a `dbtPlugin` class which defines functions that can\nbe called by dbt-core's `PluginManager`. During different parts of the dbt-core lifecycle (such as graph linking and\nmanifest writing), the `PluginManager` will be called and all plugins registered with the appropriate hook will be executed.\n\ndbt-loom implements a `get_nodes` hook, and uses a configuration file to parse manifests, identify public models, and\ninject those public models when called by `dbt-core`.\n\n## Advanced Features\n\n### Configuring artifact sources\n\n#### Object storage\n\n`dbt-loom` supports loading static manifest data stored within object storage\nproviders like S3, Azure Storage, and GCS. Note that these integrations use the\nstandard python libraries for each service (`boto3`, `gcs`, `azure`), and as\nsuch their standard environment variables are supported.\n\n```yml\nmanifests:\n  # AWS-Hosted Manifest objects\n  - name: aws_project\n    type: s3\n    config:\n      # The name of the bucket where your manifest is stored.\n      bucket_name: \u003cYOUR S3 BUCKET NAME\u003e\n\n      # The object name of your manifest file.\n      object_name: \u003cYOUR OBJECT NAME\u003e\n\n  # Google Cloud Storage\n  - name: gcs_project\n    type: gcs\n    config:\n      # The alphanumeric ID of the GCP project that contains your target bucket.\n      project_id: \u003cYOUR GCP PROJECT ID\u003e\n\n      # The name of the bucket where your manifest is stored.\n      bucket_name: \u003cYOUR GCS BUCKET NAME\u003e\n\n      # The object name of your manifest file.\n      object_name: \u003cYOUR OBJECT NAME\u003e\n\n      # The OAuth2 Credentials to use. If not passed, falls back to the default inferred from the environment.\n      credentials: \u003cPATH TO YOUR SERVICE ACCOUNT JSON CREDENTIALS\u003e\n\n  # Azure Storage\n  - name: azure_project\n    type: azure\n    config:\n      # The name of your Azure Storage account\n      account_name: \u003cYOUR AZURE STORAGE ACCOUNT NAME\u003e\n\n      # The name of your Azure Storage container\n      container_name: \u003cYOUR AZURE STORAGE CONTAINER NAME\u003e\n\n      # The object name of your manifest file.\n      object_name: \u003cYOUR OBJECT NAME\u003e\n\n      # Alternatively, Set the `AZURE_STORAGE_CONNECTION_STRING` environment\n      # variable to authenticate via a connection string.\n```\n\n#### `dbt-core` hosting providers\n\n`dbt-loom` supports calling the APIs for different `dbt-core` hosting providers\nto obtain manifest data, including dbt Cloud and Paradime. Each client requires\nspecific configuration values to operate correctly.\n\n```yaml\nmanifests:\n  - name: dbt_cloud_project\n    type: dbt_cloud\n    config:\n      account_id: \u003cYOUR DBT CLOUD ACCOUNT ID\u003e\n\n      # Job ID pertains to the job that you'd like to fetch artifacts from.\n      job_id: \u003cREFERENCE JOB ID\u003e\n\n      # dbt Cloud has multiple regions with different URLs. Update this to\n      # your appropriate dbt cloud endpoint.\n      api_endpoint: \u003cDBT CLOUD ENDPOINT\u003e\n\n      # If your job generates multiple artifacts, you can set the step from\n      # which to fetch artifacts. Defaults to the last step.\n      step_id: \u003cJOB STEP\u003e\n\n  - name: paradime_project\n    type: paradime\n    config:\n      # It is recommended to use environment variables to set your API credentials.\n      api_key: \u003cYOUR PARADIME API KEY\u003e\n      api_secret: \u003cYOUR PARADIME API SECRET\u003e\n      api_endpoint: \u003cPARADIME API ENDPOINT\u003e\n\n      # The name of the Paradime Bolt schedule to fetch artifacts from.\n      schedule_name: \u003cYOUR PARADIME SCHEDULE NAME\u003e\n\n      # (Optional) The index of the command to fetch the artifact from. If not provided,\n      # it will search through all commands in the schedule run starting from the last command.\n      command_index: \u003cYOUR PARADIME SCHEDULE COMMAND INDEX\u003e\n```\n\n#### Data Warehouses\n\nLastly, uou can use dbt-loom to fetch manifest files from Snowflake Stage and\nfrom Databricks Volumes, DBFS, and Workspace locations by setting up a `snowflake` or `databricks` manifest in your `dbt-loom` config.\n\n\u003e [!WARNING]\n\u003e The `dbt-databricks` adapter or Python SDK is required to use the `databricks` manifest type\n\n\u003e [!WARNING]\n\u003e Please note that these only work for dbt-core versions 1.8.0 and newer.\n\n```yaml\nmanifests:\n\nmanifests:\n  - name: databricks_project\n    type: databricks\n    config:\n      path: \u003cWORKSPACE, VOLUME, OR DBFS PATH TO MANIFEST FILE\u003e\n\n      # The `databricks` type implements Client Unified Authentication (https://docs.databricks.com/aws/en/dev-tools/auth/unified-auth), supporting all environment variables and authentication mechanisms.\n\n  - name: snowflake_project\n    type: snowflake\n    config:\n      stage: stage_name # Stage name, can include Database/Schema\n      stage_path: path/to/dbt/manifest.json # Path to manifest file in the stage\n```\n\n### Using environment variables\n\nYou can easily incorporate your own environment variables into the config file. This allows for dynamic configuration values that can change based on the environment. To specify an environment variable in the `dbt-loom` config file, use one of the following formats:\n\n`${ENV_VAR}` or `$ENV_VAR`\n\n#### Example:\n\n```yaml\nmanifests:\n  - name: revenue\n    type: gcs\n    config:\n      project_id: ${GCP_PROJECT}\n      bucket_name: ${GCP_BUCKET}\n      object_name: ${MANIFEST_PATH}\n```\n\n### Gzipped files\n\n`dbt-loom` natively supports decompressing gzipped manifest files. This is useful to reduce object storage size and to minimize loading times when reading manifests from object storage. Compressed file detection is triggered when the file path for the manifest is suffixed\nwith `.gz`.\n\n```yaml\nmanifests:\n  - name: revenue\n    type: s3\n    config:\n      bucket_name: example_bucket_name\n      object_name: manifest.json.gz\n```\n\n### Exclude nested packages\n\nIn some circumstances, like running `dbt-project-evaluator`, you may not want a\ngiven package in an upstream project to be imported into a downstream project.\nYou can manually exclude downstream projects from injecting assets from packages\nby adding the package name to the downstream project's `excluded_packages` list.\n\n```yaml\nmanifests:\n  - name: revenue\n    type: file\n    config:\n      path: ../revenue/target/manifest.json\n    excluded_packages:\n      # Provide the string name of the package to exclude during injection.\n      - dbt_project_evaluator\n```\n\n### Optional manifests\n\nIf you want to allow a manifest reference to be missing (e.g. using dbt-loom for an upstream project to see dependencies), you can set `optional: true` for that manifest entry. When `optional` is true and the manifest file does not exist, dbt-loom will skip loading it without raising an error. If `optional` is false or omitted (the default), missing manifests will cause an error.\n\n```yaml\nmanifests:\n  - name: revenue\n    type: file\n    config:\n      path: ../revenue/target/manifest.json\n    optional: true # If the manifest file is missing, do not raise an error\n```\n\n## Known Caveats\n\nCross-project dependencies are a relatively new development, and dbt-core plugins\nare still in beta. As such there are a number of caveats to be aware of when using\nthis tool.\n\n1. dbt plugins are only supported in dbt-core version 1.6.0-b8 and newer. This\n   means you must be using a dbt adapter compatible with this version.\n2. `PluginNodeArgs` are not fully-realized dbt `ManifestNode`s, so\n   documentation generated by `dbt docs generate` may\n   be sparse when viewing injected models.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnicholasyager%2Fdbt-loom","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnicholasyager%2Fdbt-loom","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnicholasyager%2Fdbt-loom/lists"}