{"id":21223100,"url":"https://github.com/airflow-laminar/airflow-ha","last_synced_at":"2025-07-10T14:30:34.579Z","repository":{"id":254480670,"uuid":"845737437","full_name":"airflow-laminar/airflow-ha","owner":"airflow-laminar","description":"High Availability (HA) DAG Utility","archived":false,"fork":false,"pushed_at":"2025-07-02T23:24:54.000Z","size":2172,"stargazers_count":8,"open_issues_count":1,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-07-02T23:31:55.244Z","etag":null,"topics":["airflow","apache-airflow","dag","high-availability","python","scheduler"],"latest_commit_sha":null,"homepage":"https://airflow-laminar.github.io/airflow-ha/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/airflow-laminar.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":".github/CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-08-21T20:41:45.000Z","updated_at":"2025-07-02T23:24:05.000Z","dependencies_parsed_at":"2024-09-14T10:45:06.269Z","dependency_job_id":"f1a64c2e-1292-48cb-a43a-05fc22ad18cb","html_url":"https://github.com/airflow-laminar/airflow-ha","commit_stats":null,"previous_names":["airflow-laminar/airflow-ha"],"tags_count":21,"template":false,"template_full_name":null,"purl":"pkg:github/airflow-laminar/airflow-ha","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/airflow-laminar%2Fairflow-ha","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/airflow-laminar%2Fairflow-ha/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/airflow-laminar%2Fairflow-ha/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/airflow-laminar%2Fairflow-ha/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/airflow-laminar","download_url":"https://codeload.github.com/airflow-laminar/airflow-ha/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/airflow-laminar%2Fairflow-ha/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264590694,"owners_count":23633613,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow","apache-airflow","dag","high-availability","python","scheduler"],"created_at":"2024-11-20T22:49:28.898Z","updated_at":"2025-07-10T14:30:34.573Z","avatar_url":"https://github.com/airflow-laminar.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# airflow-ha\n\nHigh Availability (HA) DAG Utility\n\n[![Build Status](https://github.com/airflow-laminar/airflow-ha/actions/workflows/build.yaml/badge.svg?branch=main\u0026event=push)](https://github.com/airflow-laminar/airflow-ha/actions/workflows/build.yaml)\n[![codecov](https://codecov.io/gh/airflow-laminar/airflow-ha/branch/main/graph/badge.svg)](https://codecov.io/gh/airflow-laminar/airflow-ha)\n[![License](https://img.shields.io/github/license/airflow-laminar/airflow-ha)](https://github.com/airflow-laminar/airflow-ha)\n[![PyPI](https://img.shields.io/pypi/v/airflow-ha.svg)](https://pypi.python.org/pypi/airflow-ha)\n\n## Overview\n\nThis library provides an operator called `HighAvailabilityOperator`, which inherits from `PythonSensor` and runs a user-provided `python_callable`.\nThe return value can trigger the following actions:\n\n| Return              | Result                                       | Current DAGrun End State |\n| :-----              | :-----                                       | :----------------------- |\n| `(PASS, RETRIGGER)` | Retrigger the same DAG to run again          | `pass`                   |\n| `(PASS, STOP)`      | Finish the DAG, until its next scheduled run | `pass`                   |\n| `(FAIL, RETRIGGER)` | Retrigger the same DAG to run again          | `fail`                   |\n| `(FAIL, STOP)`      | Finish the DAG, until its next scheduled run | `fail`                   |\n| `(*, CONTINUE)`     | Continue to run the Sensor                   | N/A                      |\n\n\u003e [!NOTE]\n\u003e Note: if the sensor times out, the behavior matches `(Result.PASS, Action.RETRIGGER)`.\n\n### Limiters\n\nArguments to `HighAvailabilityOperator` can be used to configure finishing behavior outside of the callable:\n\n- `runtime`: A `timedelta` or `int` (seconds). The operator will turn off cleanly after `dag.start_date + runtime` (`(PASS, STOP)`)\n- `endtime`: A `time` or `str` (isoformat time). The operator will turn off cleanly after `today + endtime` (`(PASS, STOP)`)\n- `maxretrigger`: An integer. The operator will turn off after `maxretrigger` retriggers (`(\u003cprevious status, STOP)`)\n\n\u003e [!NOTE]\n\u003e These can be configured as arguments to `HighAvailabilityOperator`, and will be automatically included as [DAG Params](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/params.html). This also allows them to be overriden by the DAG Config during a manual run. There is also a `force-run` option when running the DAG manually, which will cause the `HighAvailabilityOperator` to ignore the above 3 limiters.\n\n\n### Example - Always On\n\nConsider the following DAG:\n\n```python\nwith DAG(\n    dag_id=\"test-high-availability\",\n    description=\"Test HA Operator\",\n    schedule=timedelta(days=1),\n    start_date=datetime(2024, 1, 1),\n    catchup=False,\n):\n    ha = HighAvailabilityOperator(\n        task_id=\"ha\",\n        timeout=30,\n        poke_interval=5,\n        python_callable=lambda **kwargs: choice(\n            (\n                (Result.PASS, Action.CONTINUE),\n                (Result.PASS, Action.RETRIGGER),\n                (Result.PASS, Action.STOP),\n                (Result.FAIL, Action.CONTINUE),\n                (Result.FAIL, Action.RETRIGGER),\n                (Result.FAIL, Action.STOP),\n            )\n        ),\n    )\n    \n    pre = PythonOperator(task_id=\"pre\", python_callable=lambda **kwargs: \"test\")\n    pre \u003e\u003e ha\n    \n    retrigger_fail = PythonOperator(task_id=\"retrigger_fail\", python_callable=lambda **kwargs: \"test\")\n    ha.retrigger_fail \u003e\u003e retrigger_fail\n\n    stop_fail = PythonOperator(task_id=\"stop_fail\", python_callable=lambda **kwargs: fail_, trigger_rule=\"all_failed\")\n    ha.stop_fail \u003e\u003e stop_fail\n    \n    retrigger_pass = PythonOperator(task_id=\"retrigger_pass\", python_callable=lambda **kwargs: \"test\")\n    ha.retrigger_pass \u003e\u003e retrigger_pass\n\n    stop_pass = PythonOperator(task_id=\"stop_pass\", python_callable=lambda **kwargs: \"test\")\n    ha.stop_pass \u003e\u003e stop_pass\n```\n\nThis produces a DAG with the following topology:\n\n\u003cimg src=\"https://raw.githubusercontent.com/airflow-laminar/airflow-ha/main/docs/src/top.png\" /\u003e\n\nThis DAG exhibits cool behavior.\nIf the check returns `CONTINUE`, the DAG will continue to run the sensor.\nIf the check returns `RETRIGGER` or the interval elapses, the DAG will re-trigger itself and finish.\nIf the check returns `STOP`, the DAG will finish and not retrigger itself. \nIf the check returns `PASS`, the current DAG run will end in a successful state.\nIf the check returns `FAIL`, the current DAG run will end in a failed state.\n\nThis allows the one to build \"always-on\" DAGs without having individual long blocking tasks.\n\nThis library is used to build [airflow-supervisor](https://github.com/airflow-laminar/airflow-supervisor), which uses [supervisor](http://supervisord.org) as a process-monitor while checking and restarting jobs via `airflow-ha`.\n\n### Example - Recursive\n\nYou can also use this library to build recursive DAGs - or \"Cyclic DAGs\", despite the oxymoronic name.\n\nThe following code makes a DAG that triggers itself with some decrementing counter, starting with value 3:\n\n```python\n\nwith DAG(\n    dag_id=\"test-ha-counter\",\n    description=\"Test HA Countdown\",\n    schedule=timedelta(days=1),\n    start_date=datetime(2024, 1, 1),\n    catchup=False,\n):\n    \n    def _get_count(**kwargs):\n        # The default is 3\n        return kwargs['dag_run'].conf.get('counter', 3) - 1\n\n    get_count = PythonOperator(task_id=\"get-count\", python_callable=_get_count)\n\n    def _keep_counting(**kwargs):\n        count = kwargs[\"task_instance\"].xcom_pull(key=\"return_value\", task_ids=\"get-count\")\n        return (Result.PASS, Action.RETRIGGER) if count \u003e 0 else (Result.PASS, Action.STOP) if count == 0 else (Result.FAIL, Action.STOP)\n\n    keep_counting = HighAvailabilityOperator(\n        task_id=\"ha\",\n        timeout=30,\n        poke_interval=5,\n        python_callable=_keep_counting,\n        pass_trigger_kwargs={\"conf\": '''{\"counter\": {{ ti.xcom_pull(key=\"return_value\", task_ids=\"get-count\") }}}'''},\n    )\n\n    get_count \u003e\u003e keep_counting\n```\n\u003cimg src=\"https://raw.githubusercontent.com/airflow-laminar/airflow-ha/main/docs/src/rec.png\" /\u003e\n\n## License\n\nThis software is licensed under the Apache 2.0 license. See the [LICENSE](LICENSE) file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fairflow-laminar%2Fairflow-ha","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fairflow-laminar%2Fairflow-ha","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fairflow-laminar%2Fairflow-ha/lists"}