{"id":14271694,"url":"https://github.com/airflow-laminar/airflow-config","last_synced_at":"2025-07-10T14:30:34.422Z","repository":{"id":251675953,"uuid":"741200612","full_name":"airflow-laminar/airflow-config","owner":"airflow-laminar","description":"A Configuration System for Airflow","archived":false,"fork":false,"pushed_at":"2025-07-02T03:56:03.000Z","size":1766,"stargazers_count":11,"open_issues_count":5,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-07-02T04:04:32.209Z","etag":null,"topics":["airflow","apache-airflow","configuration","hydra","omegaconf","pydantic","python","scheduler"],"latest_commit_sha":null,"homepage":"https://airflow-laminar.github.io/airflow-config/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/airflow-laminar.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":".github/CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-01-09T22:47:10.000Z","updated_at":"2025-07-02T03:55:05.000Z","dependencies_parsed_at":"2024-08-05T01:26:19.756Z","dependency_job_id":"e0a4e69c-d01a-400e-8471-dc9904b34c3a","html_url":"https://github.com/airflow-laminar/airflow-config","commit_stats":null,"previous_names":["airflow-laminar/airflow-config"],"tags_count":30,"template":false,"template_full_name":null,"purl":"pkg:github/airflow-laminar/airflow-config","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/airflow-laminar%2Fairflow-config","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/airflow-laminar%2Fairflow-config/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/airflow-laminar%2Fairflow-config/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/airflow-laminar%2Fairflow-config/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/airflow-laminar","download_url":"https://codeload.github.com/airflow-laminar/airflow-config/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/airflow-laminar%2Fairflow-config/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264590694,"owners_count":23633613,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow","apache-airflow","configuration","hydra","omegaconf","pydantic","python","scheduler"],"created_at":"2024-08-22T20:01:03.120Z","updated_at":"2025-07-10T14:30:34.417Z","avatar_url":"https://github.com/airflow-laminar.png","language":"Python","funding_links":[],"categories":["Libraries, Hooks, Utilities"],"sub_categories":[],"readme":"# airflow-config\n\n[Apache Airflow](https://airflow.apache.org) utilities for configuration of DAGs and DAG environments\n\n[![Build Status](https://github.com/airflow-laminar/airflow-config/actions/workflows/build.yaml/badge.svg?branch=main\u0026event=push)](https://github.com/airflow-laminar/airflow-config/actions/workflows/build.yaml)\n[![codecov](https://codecov.io/gh/airflow-laminar/airflow-config/branch/main/graph/badge.svg)](https://codecov.io/gh/airflow-laminar/airflow-config)\n[![License](https://img.shields.io/github/license/airflow-laminar/airflow-config)](https://github.com/airflow-laminar/airflow-config)\n[![PyPI](https://img.shields.io/pypi/v/airflow-config.svg)](https://pypi.python.org/pypi/airflow-config)\n\n## Overview\n\nThis library allows for `YAML`-driven configuration of Airflow, including DAGs, Operators, and declaratively defined DAGs (à la [dag-factory](https://github.com/astronomer/dag-factory)).\nIt is built with [Pydantic](https://pydantic.dev), [Hydra](https://hydra.cc), and [OmegaConf](https://omegaconf.readthedocs.io/).\n\nConsider the following basic DAG:\n\n```python\nfrom airflow import DAG\nfrom airflow.providers.standard.operators.bash import BashOperator\nfrom datetime import datetime, timedelta\n\nwith DAG(\n    dag_id=\"test-dag\",\n    default_args={\n        \"depends_on_past\": False,\n        \"email\": [\"my.email@myemail.com\"],\n        \"email_on_failure\": False,\n        \"email_on_retry\": False,\n        \"retries\": 0,\n    },\n    description=\"test that dag is working properly\",\n    schedule=timedelta(minutes=1),\n    start_date=datetime(2024, 1, 1),\n    catchup=False,\n    tags=[\"utility\", \"test\"],\n):\n    BashOperator(\n        task_id=\"test-task\",\n        bash_command=\"echo 'test'\",\n    )\n```\n\nWe can already see many options that we might want to drive centrally via config, perhaps based on some notion of environment (e.g. `dev`, `prod`, etc).\n\n- `\"email\": [\"my.email@myemail.com\"]`\n- `\"email_on_failure\": False`\n- `\"email_on_retry\": False`\n- `\"retries\": 0`\n- `schedule=timedelta(minutes=1)`\n- `tags=[\"utility\", \"test\"]`\n\nIf we want to change these in our DAG, we need to modify code. Now imagine we have hundreds of DAGs, this can quickly get out of hand, especially since Airflow DAGs are Python code, and we might easily inject a syntax error or a trailing comma or other common problem.\n\nNow consider the alternative, config-driven approach:\n\n`config/dev.yaml`\n\n```yaml\n# @package _global_\n_target_: airflow_config.Configuration\ndefault_args:\n  _target_: airflow_config.TaskArgs\n  owner: test\n  email: [myemail@myemail.com]\n  email_on_failure: false\n  email_on_retry: false\n  retries: 0\n  depends_on_past: false\ndefault_dag_args:\n  _target_: airflow_config.DagArgs\n  schedule: \"01:00\"\n  start_date: \"2024-01-01\"\n  catchup: false\n  tags: [\"utility\", \"test\"]\n```\n\n```python\nfrom airflow.providers.standard.operators.bash import BashOperator\nfrom airflow_config import DAG, load_config\n\nconfig = load_config(config_name=\"dev\")\n\nwith DAG(\n    dag_id=\"test-dag\",\n    description=\"test that dag is working properly\",\n    schedule=timedelta(minutes=1),\n    config=config\n):\n    BashOperator(\n        task_id=\"test-task\",\n        bash_command=\"echo 'test'\",\n    )\n```\n\nThis has a number of benefits:\n\n- Make changes without code changes, with static type validation\n- Make changes across any number of DAGs without having to copy-paste\n- Organize collections of DAGs into groups, e.g. via enviroment like `dev`, `prod`, etc\n\n## Features\n\n- Configure DAGs from a central config file or...\n- from multiple env-specific config files (e.g. `dev`, `uat`, `prod`)\n- Specialize DAGs by `dag_id` from a single file (e.g. set each DAG's `schedule` from a single shared file)\n- Generate entire DAGs declaratively, like [astronomer/dag-factory](https://github.com/astronomer/dag-factory)\n- Configure other extensions like:\n  - [airflow-priority](https://github.com/airflow-laminar/airflow-priority)\n  - [airflow-balancer](https://github.com/airflow-laminar/airflow-balancer)\n  - [airflow-supervisor](https://github.com/airflow-laminar/airflow-supervisor)\n  - or write your own pydantic-based model and get yaml-based configuration for free\n\n\n## Configuration\n\n```python\nclass Configuration(BaseModel):\n    # default task args\n    # https://airflow.apache.org/docs/apache-airflow/stable/_api/airflow/models/baseoperator/index.html#airflow.models.baseoperator.BaseOperator\n    default_task_args: TaskArgs\n\n    # default dag args\n    # https://airflow.apache.org/docs/apache-airflow/stable/_api/airflow/models/dag/index.html#airflow.models.dag.DAG\n    default_dag_args: DagArgs\n\n    # string (dag id) to Dag mapping\n    dags: Optional[Dict[str, Dag]]\n\n    # string (dag id) to Task mapping\n    tasks: Optional[Dict[str, Task]]\n\n    # used for extensions to inject arbitrary configuration.\n    # See e.g.: https://github.com/airflow-laminar/airflow-supervisor?tab=readme-ov-file#example-dag-airflow-config\n    extensions: Optional[Dict[str, BaseModel]]\n```\n\nHere is an example configuration defined via yaml:\n\n```yaml\n# @package _global_\n_target_: airflow_config.Configuration\ndefault_task_args:\n  _target_: airflow_config.TaskArgs\n  owner: blerg\n  email: []\n  email_on_failure: false\n  email_on_retry: false\n  retries: 0\n  depends_on_past: false\n\ndefault_dag_args:\n  _target_: airflow_config.DagArgs\n  start_date: [\"2025-01-01\", \"America/New_York\"]\n  catchup: false\n  max_active_runs: 1\n\ndags:\n  reboot:\n    tags: [\"reboot\", \"utility\"]\n    description: \"Reboot machines\"\n    schedule: \"0 0 * * *\"\n    max_active_tasks: 1\n  clean-logs:\n    tags: [\"celery\", \"utility\"]\n    description: \"Clean worker logs\"\n    schedule: \"0 4 * * *\"\n```\n\n\n## Examples\n\n- [Basic 1](https://airflow-laminar.github.io/airflow-config/docs/src/examples.html#load-defaults-from-config)\n- [Basic 2](https://airflow-laminar.github.io/airflow-config/docs/src/examples.html#load-more-defaults-from-config)\n- [Specialize DAGs](https://airflow-laminar.github.io/airflow-config/docs/src/examples.html#specialize-individual-dags)\n- [Declarative DAGs (DAG Factory)](https://airflow-laminar.github.io/airflow-config/docs/src/examples.html#declarative-dags-dag-factory)\n- [Test Suite Setups](https://github.com/airflow-laminar/airflow-config/tree/main/airflow_config/tests/setups)\n\n\n## License\n\nThis software is licensed under the Apache 2.0 license. See the [LICENSE](LICENSE) file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fairflow-laminar%2Fairflow-config","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fairflow-laminar%2Fairflow-config","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fairflow-laminar%2Fairflow-config/lists"}