{"id":14065596,"url":"https://github.com/gocardless/airflow-dbt","last_synced_at":"2025-07-29T20:33:23.684Z","repository":{"id":37518087,"uuid":"242791495","full_name":"gocardless/airflow-dbt","owner":"gocardless","description":"Apache Airflow integration for dbt","archived":true,"fork":false,"pushed_at":"2024-05-17T06:46:00.000Z","size":68,"stargazers_count":410,"open_issues_count":31,"forks_count":62,"subscribers_count":61,"default_branch":"master","last_synced_at":"2025-07-15T16:45:46.072Z","etag":null,"topics":["airflow","airflow-dbt","dbt"],"latest_commit_sha":null,"homepage":"https://pypi.org/project/airflow-dbt/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gocardless.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-02-24T16:54:33.000Z","updated_at":"2025-07-11T20:04:28.000Z","dependencies_parsed_at":"2024-05-28T01:50:41.009Z","dependency_job_id":null,"html_url":"https://github.com/gocardless/airflow-dbt","commit_stats":{"total_commits":67,"total_committers":25,"mean_commits":2.68,"dds":0.6268656716417911,"last_synced_commit":"1eddf5266d151f09cafba57e21d9d140a587b5f1"},"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"purl":"pkg:github/gocardless/airflow-dbt","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gocardless%2Fairflow-dbt","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gocardless%2Fairflow-dbt/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gocardless%2Fairflow-dbt/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gocardless%2Fairflow-dbt/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gocardless","download_url":"https://codeload.github.com/gocardless/airflow-dbt/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gocardless%2Fairflow-dbt/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":267754868,"owners_count":24139435,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-29T02:00:12.549Z","response_time":2574,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow","airflow-dbt","dbt"],"created_at":"2024-08-13T07:04:35.113Z","updated_at":"2025-07-29T20:33:23.412Z","avatar_url":"https://github.com/gocardless.png","language":"Python","readme":"# airflow-dbt\n\nThis is a collection of [Airflow](https://airflow.apache.org/) operators to provide easy integration with [dbt](https://www.getdbt.com).\n\n```py\nfrom airflow import DAG\nfrom airflow_dbt.operators.dbt_operator import (\n    DbtSeedOperator,\n    DbtSnapshotOperator,\n    DbtRunOperator,\n    DbtTestOperator,\n    DbtCleanOperator,\n)\nfrom airflow.utils.dates import days_ago\n\ndefault_args = {\n  'dir': '/srv/app/dbt',\n  'start_date': days_ago(0)\n}\n\nwith DAG(dag_id='dbt', default_args=default_args, schedule_interval='@daily') as dag:\n\n  dbt_seed = DbtSeedOperator(\n    task_id='dbt_seed',\n  )\n\n  dbt_snapshot = DbtSnapshotOperator(\n    task_id='dbt_snapshot',\n  )\n\n  dbt_run = DbtRunOperator(\n    task_id='dbt_run',\n  )\n\n  dbt_test = DbtTestOperator(\n    task_id='dbt_test',\n    retries=0,  # Failing tests would fail the task, and we don't want Airflow to try again\n  )\n\n  dbt_clean = DbtCleanOperator(\n    task_id='dbt_clean',\n  )\n\n  dbt_seed \u003e\u003e dbt_snapshot \u003e\u003e dbt_run \u003e\u003e dbt_test \u003e\u003e dbt_clean\n```\n\n## Installation\n\nInstall from PyPI:\n\n```sh\npip install airflow-dbt\n```\n\nIt will also need access to the `dbt` CLI, which should either be on your `PATH` or can be set with the `dbt_bin` argument in each operator.\n\n## Usage\n\nThere are five operators currently implemented:\n\n* `DbtDocsGenerateOperator`\n  * Calls [`dbt docs generate`](https://docs.getdbt.com/reference/commands/cmd-docs)\n* `DbtDepsOperator`\n  * Calls [`dbt deps`](https://docs.getdbt.com/docs/deps)\n* `DbtSeedOperator`\n  * Calls [`dbt seed`](https://docs.getdbt.com/docs/seed)\n* `DbtSnapshotOperator`\n  * Calls [`dbt snapshot`](https://docs.getdbt.com/docs/snapshot)\n* `DbtRunOperator`\n  * Calls [`dbt run`](https://docs.getdbt.com/docs/run)\n* `DbtTestOperator`\n  * Calls [`dbt test`](https://docs.getdbt.com/docs/test)\n* `DbtCleanOperator`\n  * Calls [`dbt clean`](https://docs.getdbt.com/docs/clean)\n\n\nEach of the above operators accept the following arguments:\n\n* `env`\n  * If set as a kwarg dict, passed the given environment variables as the arguments to the dbt task\n* `profiles_dir`\n  * If set, passed as the `--profiles-dir` argument to the `dbt` command\n* `target`\n  * If set, passed as the `--target` argument to the `dbt` command\n* `dir`\n  * The directory to run the `dbt` command in\n* `full_refresh`\n  * If set to `True`, passes `--full-refresh`\n* `vars`\n  * If set, passed as the `--vars` argument to the `dbt` command. Should be set as a Python dictionary, as will be passed to the `dbt` command as YAML\n* `models`\n  * If set, passed as the `--models` argument to the `dbt` command\n* `exclude`\n  * If set, passed as the `--exclude` argument to the `dbt` command\n* `select`\n  * If set, passed as the `--select` argument to the `dbt` command\n* `selector`\n  * If set, passed as the `--selector` argument to the `dbt` command\n* `dbt_bin`\n  * The `dbt` CLI. Defaults to `dbt`, so assumes it's on your `PATH`\n* `verbose`\n  * The operator will log verbosely to the Airflow logs\n* `warn_error`\n  * If set to `True`, passes `--warn-error` argument to `dbt` command and will treat warnings as errors\n\nTypically you will want to use the `DbtRunOperator`, followed by the `DbtTestOperator`, as shown earlier.\n\nYou can also use the hook directly. Typically this can be used for when you need to combine the `dbt` command with another task in the same operators, for example running `dbt docs` and uploading the docs to somewhere they can be served from.\n\n## Building Locally\n\nTo install from the repository:\nFirst it's recommended to create a virtual environment:\n```bash\npython3 -m venv .venv\n\nsource .venv/bin/activate\n```\n\nInstall using `pip`:\n```bash\npip install .\n```\n\n## Testing\n\nTo run tests locally, first create a virtual environment (see [Building Locally](https://github.com/gocardless/airflow-dbt#building-locally) section)\n\nInstall dependencies:\n```bash\npip install . pytest\n```\n\nRun the tests:\n```bash\npytest tests/\n```\n\n## Code style\nThis project uses [flake8](https://flake8.pycqa.org/en/latest/).\n\nTo check your code, first create a virtual environment (see [Building Locally](https://github.com/gocardless/airflow-dbt#building-locally) section):\n```bash\npip install flake8\nflake8 airflow_dbt/ tests/ setup.py\n```\n\n## Package management\n\nIf you use dbt's package manager you should include all dependencies before deploying your dbt project.\n\nFor Docker users, packages specified in `packages.yml` should be included as part your docker image by calling `dbt deps` in your `Dockerfile`.\n\n## Amazon Managed Workflows for Apache Airflow (MWAA)\n\nIf you use MWAA, you just need to update the `requirements.txt` file and add `airflow-dbt` and `dbt` to it.\n\nThen you can have your dbt code inside a folder `{DBT_FOLDER}` in the dags folder on S3 and configure the dbt task like below:\n\n```python\ndbt_run = DbtRunOperator(\n  task_id='dbt_run',\n  dbt_bin='/usr/local/airflow/.local/bin/dbt',\n  profiles_dir='/usr/local/airflow/dags/{DBT_FOLDER}/',\n  dir='/usr/local/airflow/dags/{DBT_FOLDER}/'\n)\n```\n\n## Templating and parsing environments variables\n\nIf you would like to run DBT using custom profile definition template with environment-specific variables, like for example profiles.yml using jinja:\n```yaml\n\u003cprofile_name\u003e:\n  outputs:\n    \u003csource\u003e:\n      database: \"{{ env_var('DBT_ENV_SECRET_DATABASE') }}\"\n      password: \"{{ env_var('DBT_ENV_SECRET_PASSWORD') }}\"\n      schema: \"{{ env_var('DBT_ENV_SECRET_SCHEMA') }}\"\n      threads: \"{{ env_var('DBT_THREADS') }}\"\n      type: \u003ctype\u003e\n      user: \"{{ env_var('USER_NAME') }}_{{ env_var('ENV_NAME') }}\"\n  target: \u003csource\u003e\n```\n\nYou can pass the environment variables via the `env` kwarg parameter:\n\n```python\nimport os\n...\n\ndbt_run = DbtRunOperator(\n  task_id='dbt_run',\n  env={\n    'DBT_ENV_SECRET_DATABASE': '\u003cDATABASE\u003e',\n    'DBT_ENV_SECRET_PASSWORD': '\u003cPASSWORD\u003e',\n    'DBT_ENV_SECRET_SCHEMA': '\u003cSCHEMA\u003e',\n    'USER_NAME': '\u003cUSER_NAME\u003e',\n    'DBT_THREADS': os.getenv('\u003cDBT_THREADS_ENV_VARIABLE_NAME\u003e'),\n    'ENV_NAME': os.getenv('ENV_NAME')\n  }\n)\n```\n\n## License \u0026 Contributing\n\n* This is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).\n* Bug reports and pull requests are welcome on GitHub at https://github.com/gocardless/airflow-dbt.\n\nGoCardless ♥ open source. If you do too, come [join us](https://gocardless.com/about/jobs).\n","funding_links":[],"categories":["Python"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgocardless%2Fairflow-dbt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgocardless%2Fairflow-dbt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgocardless%2Fairflow-dbt/lists"}