{"id":21514840,"url":"https://github.com/getindata/dbt-airflow-factory","last_synced_at":"2025-04-09T20:11:38.152Z","repository":{"id":40349871,"uuid":"419245441","full_name":"getindata/dbt-airflow-factory","owner":"getindata","description":"Library to convert DBT manifest metadata to Airflow tasks","archived":false,"fork":false,"pushed_at":"2024-03-08T09:48:51.000Z","size":2045,"stargazers_count":48,"open_issues_count":2,"forks_count":6,"subscribers_count":7,"default_branch":"develop","last_synced_at":"2025-04-09T20:11:32.529Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/getindata.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null}},"created_at":"2021-10-20T08:25:48.000Z","updated_at":"2025-01-31T06:07:31.000Z","dependencies_parsed_at":"2023-02-06T03:15:55.350Z","dependency_job_id":null,"html_url":"https://github.com/getindata/dbt-airflow-factory","commit_stats":null,"previous_names":["getindata/dbt-airflow-manifest-parser"],"tags_count":35,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/getindata%2Fdbt-airflow-factory","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/getindata%2Fdbt-airflow-factory/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/getindata%2Fdbt-airflow-factory/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/getindata%2Fdbt-airflow-factory/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/getindata","download_url":"https://codeload.github.com/getindata/dbt-airflow-factory/tar.gz/refs/heads/develop","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248103872,"owners_count":21048245,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-23T23:53:11.660Z","updated_at":"2025-04-09T20:11:38.120Z","avatar_url":"https://github.com/getindata.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DBT Airflow Factory\n\n[![Python Version](https://img.shields.io/badge/python-3.8%20%7C%203.9%20%7C%203.10%20%7C%203.11-blue)](https://github.com/getindata/dbt-airflow-factory)\n[![PyPI Version](https://badge.fury.io/py/dbt-airflow-factory.svg)](https://pypi.org/project/dbt-airflow-factory/)\n[![Downloads](https://pepy.tech/badge/dbt-airflow-factory)](https://pepy.tech/project/dbt-airflow-factory)\n[![Maintainability](https://api.codeclimate.com/v1/badges/47fd3570c858b6c166ad/maintainability)](https://codeclimate.com/github/getindata/dbt-airflow-factory/maintainability)\n[![Test Coverage](https://api.codeclimate.com/v1/badges/47fd3570c858b6c166ad/test_coverage)](https://codeclimate.com/github/getindata/dbt-airflow-factory/test_coverage)\n[![Documentation Status](https://readthedocs.org/projects/dbt-airflow-factory/badge/?version=latest)](https://dbt-airflow-factory.readthedocs.io/en/latest/?badge=latest)\n\nLibrary to convert DBT manifest metadata to Airflow tasks\n\n## Documentation\n\nRead the full documentation at [https://dbt-airflow-factory.readthedocs.io/](https://dbt-airflow-factory.readthedocs.io/en/latest/index.html)\n\n## Installation\n\nUse the package manager [pip][pip] to install the library:\n\n```bash\npip install dbt-airflow-factory\n```\n\n## Usage\n\nThe library is expected to be used inside an Airflow environment with a Kubernetes image referencing **dbt**.\n\n**dbt-airflow-factory**'s main task is to parse `manifest.json` and create Airflow DAG out of it. It also reads config\nfiles from `config` directory and therefore is highly customizable (e.g., user can set path to `manifest.json`).\n\nTo start, create a directory with a following structure, where `manifest.json` is a file generated by **dbt**:\n```\n.\n├── config\n│   ├── base\n│   │   ├── airflow.yml\n│   │   ├── dbt.yml\n│   │   └── k8s.yml\n│   └── dev\n│       └── dbt.yml\n├── dag.py\n└── manifest.json\n```\n\nThen, put the following code into `dag.py`:\n```python\nfrom dbt_airflow_factory.airflow_dag_factory import AirflowDagFactory\nfrom os import path\n\ndag = AirflowDagFactory(path.dirname(path.abspath(__file__)), \"dev\").create()\n```\n\nWhen uploaded to Airflow DAGs directory, it will get picked up by Airflow, parse `manifest.json` and prepare a DAG to run.\n\n### Configuration files\n\nIt is best to look up the example configuration files in [tests directory][tests] to get a glimpse of correct configs.\n\nYou can use [Airflow template variables][airflow-vars] in your `dbt.yml` and `k8s.yml` files, as long as they are inside\nquotation marks:\n```yaml\ntarget: \"{{ var.value.env }}\"\nsome_other_field: \"{{ ds_nodash }}\"\n```\n\nAnalogously, you can use `\"{{ var.value.VARIABLE_NAME }}\"` in `airflow.yml`, but only the Airflow variable getter.\nAny other Airflow template variables will not work in `airflow.yml`.\n\n### Creation of the directory with data-pipelines-cli\n\n**DBT Airflow Factory** works best in tandem with [data-pipelines-cli][dp-cli] tool. **dp** not only prepares directory\nfor the library to digest, but also automates Docker image building and pushes generated directory to the cloud storage\nof your choice.\n\n[airflow-vars]: https://airflow.apache.org/docs/apache-airflow/stable/templates-ref.html#variables\n[dp-cli]: https://pypi.org/project/data-pipelines-cli/\n[pip]: https://pip.pypa.io/en/stable/\n[tests]: https://github.com/getindata/dbt-airflow-factory/tree/develop/tests/config\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgetindata%2Fdbt-airflow-factory","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgetindata%2Fdbt-airflow-factory","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgetindata%2Fdbt-airflow-factory/lists"}