{"id":13702869,"url":"https://github.com/PrefectHQ/prefect-dbt","last_synced_at":"2025-05-05T06:32:31.031Z","repository":{"id":38068972,"uuid":"496771859","full_name":"PrefectHQ/prefect-dbt","owner":"PrefectHQ","description":"Collection of Prefect integrations for working with dbt with your Prefect flows.","archived":true,"fork":false,"pushed_at":"2024-04-26T15:28:01.000Z","size":2928,"stargazers_count":83,"open_issues_count":0,"forks_count":9,"subscribers_count":9,"default_branch":"main","last_synced_at":"2025-04-26T23:37:59.095Z","etag":null,"topics":["dbt","prefect","python"],"latest_commit_sha":null,"homepage":"https://prefecthq.github.io/prefect-dbt/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/PrefectHQ.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-05-26T21:05:42.000Z","updated_at":"2024-12-28T17:05:49.000Z","dependencies_parsed_at":"2023-10-30T21:35:34.586Z","dependency_job_id":"ae782d87-7a14-473d-88a1-6f453766b793","html_url":"https://github.com/PrefectHQ/prefect-dbt","commit_stats":{"total_commits":123,"total_committers":10,"mean_commits":12.3,"dds":0.6585365853658536,"last_synced_commit":"4bfdda85940140eb640b2d7c50a8a3e3d5f01f90"},"previous_names":[],"tags_count":13,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PrefectHQ%2Fprefect-dbt","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PrefectHQ%2Fprefect-dbt/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PrefectHQ%2Fprefect-dbt/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PrefectHQ%2Fprefect-dbt/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/PrefectHQ","download_url":"https://codeload.github.com/PrefectHQ/prefect-dbt/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252451982,"owners_count":21750019,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dbt","prefect","python"],"created_at":"2024-08-02T21:00:44.827Z","updated_at":"2025-05-05T06:32:30.400Z","avatar_url":"https://github.com/PrefectHQ.png","language":"Python","funding_links":[],"categories":["Integrations"],"sub_categories":[],"readme":"\u003e [!NOTE]\n\u003e Active development of this project has moved within PrefectHQ/prefect. The code can be found [here](https://github.com/PrefectHQ/prefect/tree/main/src/integrations/prefect-dbt) and documentation [here](https://docs.prefect.io/latest/integrations/prefect-dbt).\n\u003e Please open issues and PRs against PrefectHQ/prefect instead of this repository.\n\n\n# prefect-dbt\n\n\u003cp align=\"center\"\u003e\n    \u003ca href=\"https://pypi.python.org/pypi/prefect-dbt/\" alt=\"PyPI version\"\u003e\n        \u003cimg alt=\"PyPI\" src=\"https://img.shields.io/pypi/v/prefect-dbt?color=26272B\u0026labelColor=090422\"\u003e\u003c/a\u003e\n    \u003ca href=\"https://github.com/prefecthq/prefect-dbt/\" alt=\"Stars\"\u003e\n        \u003cimg src=\"https://img.shields.io/github/stars/prefecthq/prefect-dbt?color=26272B\u0026labelColor=090422\" /\u003e\u003c/a\u003e\n    \u003ca href=\"https://pepy.tech/badge/prefect-dbt/\" alt=\"Downloads\"\u003e\n        \u003cimg src=\"https://img.shields.io/pypi/dm/prefect-dbt?color=26272B\u0026labelColor=090422\" /\u003e\u003c/a\u003e\n    \u003ca href=\"https://github.com/prefecthq/prefect-dbt/pulse\" alt=\"Activity\"\u003e\n        \u003cimg src=\"https://img.shields.io/github/commit-activity/m/prefecthq/prefect-dbt?ccolor=26272B\u0026labelColor=090422\" /\u003e\u003c/a\u003e\n    \u003cbr\u003e\n    \u003ca href=\"https://prefect-community.slack.com\" alt=\"Slack\"\u003e\n        \u003cimg src=\"https://img.shields.io/badge/slack-join_community-red.svg?color=26272B\u0026labelColor=090422\u0026logo=slack\" /\u003e\u003c/a\u003e\n\n\u003c/p\u003e\n\nVisit the full docs [here](https://PrefectHQ.github.io/prefect-dbt) to see additional examples and the API reference.\n\nWith prefect-dbt, you can easily trigger and monitor dbt Cloud jobs, execute dbt Core CLI commands, and incorporate other services, like Snowflake, into your dbt runs!\n\nCheck out the examples below to get started!\n\n## Getting Started\n\nBe sure to install [prefect-dbt](#installation) and [save a block](#saving-credentials-to-block) to run the examples below!\n\n### Integrate dbt Cloud jobs with Prefect flows\n\nIf you have an existing dbt Cloud job, take advantage of the flow, `run_dbt_cloud_job`.\n\nThis flow triggers the job and waits until the job run is finished.\n\nIf certain nodes fail, `run_dbt_cloud_job` efficiently retries the specific, unsuccessful nodes.\n\n```python\nfrom prefect import flow\n\nfrom prefect_dbt.cloud import DbtCloudJob\nfrom prefect_dbt.cloud.jobs import run_dbt_cloud_job\n\n@flow\ndef run_dbt_job_flow():\n    result = run_dbt_cloud_job(\n        dbt_cloud_job=DbtCloudJob.load(\"my-block-name\"),\n        targeted_retries=5,\n    )\n    return result\n\nrun_dbt_job_flow()\n```\n\n### Integrate dbt Core CLI commands with Prefect flows\n\n`prefect-dbt` also supports execution of dbt Core CLI commands.\n\nTo get started, if you don't have a `DbtCoreOperation` block already saved,\nset the commands that you want to run; it can include a mix of dbt and non-dbt commands.\n\nThen, optionally specify the `project_dir`.\n\nIf `profiles_dir` is unset, it will try to use the `DBT_PROFILES_DIR` environment variable.\nIf that's also not set, it will use the default directory `$HOME/.dbt/`.\n\n#### Using an existing profile\n\nIf you already have an existing dbt profile, specify the `profiles_dir` where `profiles.yml` is located.\n\n```python\nfrom prefect import flow\nfrom prefect_dbt.cli.commands import DbtCoreOperation\n\n@flow\ndef trigger_dbt_flow() -\u003e str:\n    result = DbtCoreOperation(\n        commands=[\"pwd\", \"dbt debug\", \"dbt run\"],\n        project_dir=\"PROJECT-DIRECTORY-PLACEHOLDER\",\n        profiles_dir=\"PROFILES-DIRECTORY-PLACEHOLDER\"\n    ).run()\n    return result\n\ntrigger_dbt_flow()\n```\n\n#### Writing a new profile\n\nTo setup a new profile, first [save and load a DbtCliProfile block](#saving-credentials-to-block) and use it in `DbtCoreOperation`.\n\nThen, specify `profiles_dir` where `profiles.yml` will be written.\n\n```python\nfrom prefect import flow\nfrom prefect_dbt.cli import DbtCliProfile, DbtCoreOperation\n\n@flow\ndef trigger_dbt_flow():\n    dbt_cli_profile = DbtCliProfile.load(\"DBT-CORE-OPERATION-BLOCK-NAME-PLACEHOLDER\")\n    with DbtCoreOperation(\n        commands=[\"dbt debug\", \"dbt run\"],\n        project_dir=\"PROJECT-DIRECTORY-PLACEHOLDER\",\n        profiles_dir=\"PROFILES-DIRECTORY-PLACEHOLDER\",\n        dbt_cli_profile=dbt_cli_profile,\n    ) as dbt_operation:\n        dbt_process = dbt_operation.trigger()\n        # do other things before waiting for completion\n        dbt_process.wait_for_completion()\n        result = dbt_process.fetch_result()\n    return result\n\ntrigger_dbt_flow()\n```\n\n## Resources\n\nIf you need help getting started with or using dbt, please consult the [dbt documentation](https://docs.getdbt.com/docs/building-a-dbt-project/documentation).\n\n### Installation\n\nTo use `prefect-dbt` with dbt Cloud:\n\n```bash\npip install prefect-dbt\n```\n\nTo use dbt Core (CLI):\n\n```bash\npip install \"prefect-dbt[cli]\"\n```\n\nTo use dbt Core with Snowflake profiles:\n\n```bash\npip install \"prefect-dbt[snowflake]\"\n```\n\nTo use dbt Core with BigQuery profiles:\n\n```bash\npip install \"prefect-dbt[bigquery]\"\n```\n\nTo use dbt Core with Postgres profiles:\n\n```bash\npip install \"prefect-dbt[postgres]\"\n```\n\n!!! warning \"Some dbt Core profiles require additional installation\"\n\n    According to dbt's [Databricks setup page](https://docs.getdbt.com/reference/warehouse-setups/databricks-setup), users must first install the adapter:\n\n    ```bash\n    pip install dbt-databricks\n    ```\n\n    Check out the [desired profile setup page](https://docs.getdbt.com/reference/profiles.yml) on the sidebar for others.\n\nRequires an installation of Python 3.8+.\n\nWe recommend using a Python virtual environment manager such as pipenv, conda or virtualenv.\n\nThese tasks are designed to work with Prefect 2. For more information about how to use Prefect, please refer to the [Prefect documentation](https://docs.prefect.io/).\n\n### Saving credentials to block\n\nNote, to use the `load` method on Blocks, you must already have a block document [saved through code](https://docs.prefect.io/concepts/blocks/#saving-blocks) or [saved through the UI](https://docs.prefect.io/ui/blocks/).\n\n!!! info \"Registering blocks\"\n\n    Register blocks in this module to\n    [view and edit them](https://docs.prefect.io/ui/blocks/)\n    on Prefect Cloud:\n\n    ```bash\n    prefect block register -m prefect_dbt\n    ```\n\nA list of available blocks in `prefect-dbt` and their setup instructions can be found [here](https://PrefectHQ.github.io/prefect-dbt/blocks_catalog).\n\n#### dbt Cloud\n\nTo create a dbt Cloud credentials block:\n\n1. Head over to your [dbt Cloud profile](https://cloud.getdbt.com/settings/profile).\n2. Login to your dbt Cloud account.\n3. Scroll down to \"API\" or click \"API Access\" on the sidebar.\n4. Copy the API Key.\n5. Click Projects on the sidebar.\n6. Copy the account ID from the URL: `https://cloud.getdbt.com/settings/accounts/\u003cACCOUNT_ID\u003e`.\n7. Create a short script, replacing the placeholders.\n\n```python\nfrom prefect_dbt.cloud import DbtCloudCredentials\n\nDbtCloudCredentials(\n    api_key=\"API-KEY-PLACEHOLDER\",\n    account_id=\"ACCOUNT-ID-PLACEHOLDER\"\n).save(\"CREDENTIALS-BLOCK-NAME-PLACEHOLDER\")\n```\n\nThen, to create a dbt Cloud job block:\n\n1. Head over to your [dbt home page](https://cloud.getdbt.com/).\n2. On the top nav bar, click on Deploy -\u003e Jobs.\n3. Select a job.\n4. Copy the job ID from the URL: `https://cloud.getdbt.com/deploy/\u003cACCOUNT_ID\u003e/projects/\u003cPROJECT_ID\u003e/jobs/\u003cJOB_ID\u003e`\n5. Create a short script, replacing the placeholders.\n\n```python\nfrom prefect_dbt.cloud import DbtCloudCredentials, DbtCloudJob\n\ndbt_cloud_credentials = DbtCloudCredentials.load(\"CREDENTIALS-BLOCK-NAME-PLACEHOLDER\")\ndbt_cloud_job = DbtCloudJob(\n    dbt_cloud_credentials=dbt_cloud_credentials,\n    job_id=\"JOB-ID-PLACEHOLDER\"\n).save(\"JOB-BLOCK-NAME-PLACEHOLDER\")\n```\n\nCongrats! You can now easily load the saved block, which holds your credentials:\n\n```python\nfrom prefect_dbt.cloud import DbtCloudJob\n\nDbtCloudJob.load(\"JOB-BLOCK-NAME-PLACEHOLDER\")\n```\n\n#### dbt Core CLI\n\n!!! info \"Available `TargetConfigs` blocks\"\n\n    The following may vary slightly depending on the service you want to incorporate.\n\n    Visit the [API Reference](cli/configs/base) to see other built-in `TargetConfigs` blocks.\n\n    If the desired service profile is not available, check out the\n    [Examples Catalog](examples_catalog/#clicredentials-module) to see how you can\n    build one from the generic `TargetConfigs` class.\n\nTo create dbt Core target config and profile blocks for BigQuery:\n\n1. Save and load a [`GcpCredentials` block](https://prefecthq.github.io/prefect-gcp/#saving-credentials-to-a-block).\n2. Determine the schema / dataset you want to use in BigQuery.\n3. Create a short script, replacing the placeholders.\n\n```python\nfrom prefect_gcp.credentials import GcpCredentials\nfrom prefect_dbt.cli import BigQueryTargetConfigs, DbtCliProfile\n\ncredentials = GcpCredentials.load(\"CREDENTIALS-BLOCK-NAME-PLACEHOLDER\")\ntarget_configs = BigQueryTargetConfigs(\n    schema=\"SCHEMA-NAME-PLACEHOLDER\",  # also known as dataset\n    credentials=credentials,\n)\ntarget_configs.save(\"TARGET-CONFIGS-BLOCK-NAME-PLACEHOLDER\")\n\ndbt_cli_profile = DbtCliProfile(\n    name=\"PROFILE-NAME-PLACEHOLDER\",\n    target=\"TARGET-NAME-placeholder\",\n    target_configs=target_configs,\n)\ndbt_cli_profile.save(\"DBT-CLI-PROFILE-BLOCK-NAME-PLACEHOLDER\")\n```\n\nThen, to create a dbt Core operation block:\n\n1. Determine the dbt commands you want to run.\n2. Create a short script, replacing the placeholders.\n\n```python\nfrom prefect_dbt.cli import DbtCliProfile, DbtCoreOperation\n\ndbt_cli_profile = DbtCliProfile.load(\"DBT-CLI-PROFILE-BLOCK-NAME-PLACEHOLDER\")\ndbt_core_operation = DbtCoreOperation(\n    commands=[\"DBT-CLI-COMMANDS-PLACEHOLDER\"],\n    dbt_cli_profile=dbt_cli_profile,\n    overwrite_profiles=True,\n)\ndbt_core_operation.save(\"DBT-CORE-OPERATION-BLOCK-NAME-PLACEHOLDER\")\n```\n\nCongrats! You can now easily load the saved block, which holds your credentials:\n\n```python\nfrom prefect_dbt.cloud import DbtCoreOperation\n\nDbtCoreOperation.load(\"DBT-CORE-OPERATION-BLOCK-NAME-PLACEHOLDER\")\n```\n\n### Feedback\n\nIf you encounter any bugs while using `prefect-dbt`, feel free to open an issue in the [prefect-dbt](https://github.com/PrefectHQ/prefect-dbt) repository.\n\nIf you have any questions or issues while using `prefect-dbt`, you can find help in the [Prefect Slack community](https://prefect.io/slack).\n\nFeel free to star or watch [`prefect-dbt`](https://github.com/PrefectHQ/prefect-dbt) for updates, too!\n\n### Contributing\n\nIf you'd like to help contribute to fix an issue or add a feature to `prefect-dbt`, please [propose changes through a pull request from a fork of the repository](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request-from-a-fork).\n\nHere are the steps:\n\n1. [Fork the repository](https://docs.github.com/en/get-started/quickstart/fork-a-repo#forking-a-repository)\n2. [Clone the forked repository](https://docs.github.com/en/get-started/quickstart/fork-a-repo#cloning-your-forked-repository)\n3. Install the repository and its dependencies:\n```\npip install -e \".[dev]\"\n```\n4. Make desired changes\n5. Add tests\n6. Insert an entry to [CHANGELOG.md](https://github.com/PrefectHQ/prefect-dbt/blob/main/CHANGELOG.md)\n7. Install `pre-commit` to perform quality checks prior to commit:\n```\npre-commit install\n```\n8. `git commit`, `git push`, and create a pull request\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FPrefectHQ%2Fprefect-dbt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FPrefectHQ%2Fprefect-dbt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FPrefectHQ%2Fprefect-dbt/lists"}