{"id":14271704,"url":"https://github.com/airflow-laminar/airflow-supervisor","last_synced_at":"2026-03-17T13:12:34.042Z","repository":{"id":252601218,"uuid":"601387725","full_name":"airflow-laminar/airflow-supervisor","owner":"airflow-laminar","description":"Airflow utilities for running long-running or always-on jobs with supervisord","archived":false,"fork":false,"pushed_at":"2025-07-02T23:22:44.000Z","size":2164,"stargazers_count":11,"open_issues_count":6,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-07-03T00:21:26.279Z","etag":null,"topics":["airflow","apache-airflow","process-manager","process-monitor","python","scheduler","supervisor","supervisord"],"latest_commit_sha":null,"homepage":"https://airflow-laminar.github.io/airflow-supervisor/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/airflow-laminar.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":".github/CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-02-14T00:33:50.000Z","updated_at":"2025-07-02T23:21:51.000Z","dependencies_parsed_at":"2024-09-15T08:13:24.264Z","dependency_job_id":"0c2d1caa-8468-4f37-affa-cc88ce49de02","html_url":"https://github.com/airflow-laminar/airflow-supervisor","commit_stats":null,"previous_names":["airflow-laminar/airflow-supervisor","timkpaine/airflow-supervisor"],"tags_count":36,"template":false,"template_full_name":null,"purl":"pkg:github/airflow-laminar/airflow-supervisor","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/airflow-laminar%2Fairflow-supervisor","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/airflow-laminar%2Fairflow-supervisor/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/airflow-laminar%2Fairflow-supervisor/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/airflow-laminar%2Fairflow-supervisor/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/airflow-laminar","download_url":"https://codeload.github.com/airflow-laminar/airflow-supervisor/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/airflow-laminar%2Fairflow-supervisor/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264590700,"owners_count":23633618,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow","apache-airflow","process-manager","process-monitor","python","scheduler","supervisor","supervisord"],"created_at":"2024-08-22T20:01:03.260Z","updated_at":"2026-03-17T13:12:34.035Z","avatar_url":"https://github.com/airflow-laminar.png","language":"Python","funding_links":[],"categories":["Libraries, Hooks, Utilities"],"sub_categories":[],"readme":"# airflow-supervisor\n\n[Apache Airflow](https://airflow.apache.org) utilities for running long-running or always-on jobs with [supervisord](http://supervisord.org)\n\n[![Build Status](https://github.com/airflow-laminar/airflow-supervisor/actions/workflows/build.yaml/badge.svg?branch=main\u0026event=push)](https://github.com/airflow-laminar/airflow-supervisor/actions/workflows/build.yaml)\n[![codecov](https://codecov.io/gh/airflow-laminar/airflow-supervisor/branch/main/graph/badge.svg)](https://codecov.io/gh/airflow-laminar/airflow-supervisor)\n[![License](https://img.shields.io/github/license/airflow-laminar/airflow-supervisor)](https://github.com/airflow-laminar/airflow-supervisor)\n[![PyPI](https://img.shields.io/pypi/v/airflow-supervisor.svg)](https://pypi.python.org/pypi/airflow-supervisor)\n\n## Overview\n\nThis library provides a configuration-driven way of generating [supervisor](http://supervisord.org) configurations and airflow [operators](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/operators.html)/[sensors](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/sensors.html) for long-running or always-on jobs. Configuration is managed by [Pydantic](https://pydantic.dev), [Hydra](https://hydra.cc), and [OmegaConf](https://omegaconf.readthedocs.io/) via the [supervisor-pydantic](https://github.com/airflow-laminar/supervisor-pydantic) library.\n\n## How To: Use in Airflow\n\n`airflow-supervisor` can be installed in your airflow server environment and imported in your dag files. It provides two convenient top level DAG subclasses:\n\n- `Supervisor`: creates a DAG representing a local supervisor instance running on the airflow worker node (underlying task will use `PythonOperator` and `BashOperator` to communicate between airflow and supervisor)\n- `SupervisorSSH`: creates a DAG representing a remote supervisor instance running on another machine (underlying tasks will use `SSHOperator` to communicate between airflow and supervisor)\n\nWe expose DAGs composed of a variety of tasks and sensors, which are exposed as a discrete pipeline of steps:\n\n1. Setup `supervisord` configuration\n1. Start the `supervisord` daemon\n1. Start the supervised programs with `supervisorctl`\n1. Start sensors to query the programs' state via [supervisor's XML-RPC API](http://supervisord.org/api.html)\n1. Evaluate and take action according to the program's state changes\n1. Restart programs if necessary\n1. Tear down the sensors from (4)\n1. Stop the supervised programs from (3)\n1. Stop the `supervisord` daemon from (2)\n1. Remove configuration from (1)\n\nThis setup provides maximal configureability with a minimal requirements on the machine (for example, no requirements on an existing `supervisord` daemon via e.g. `systemd`). It also lets you hook your own tasks into any step of the process. For example, if we detect a process has died in step (5), you could configure your own task to take some custom action before/instead of the default restart of step 6.\n\nHere is a nice overview of the DAG, with annotations for code paths and the actions taken by Supervisor:\n\n\u003cimg src=\"https://raw.githubusercontent.com/airflow-laminar/airflow-supervisor/main/docs/img/overview.png\" /\u003e\n\nMore docs and code examples coming soon!\n\n### Example Dag:\n\n```python\nfrom airflow import DAG\nfrom datetime import timedelta, datetime\nfrom airflow_supervisor import SupervisorAirflowConfiguration, Supervisor, ProgramConfiguration\n\n\n# Create supervisor configuration\ncfg = SupervisorAirflowConfiguration(\n    working_dir=\"/data/airflow/supervisor\",\n    config_path=\"/data/airflow/supervisor/supervisor.conf\",\n    program={\n        \"test\": ProgramConfiguration(\n            command=\"bash -c 'sleep 14400; exit 1'\",\n        )\n    },\n)\n\n# Create DAG as normal\nwith DAG(\n    dag_id=\"test-supervisor\",\n    schedule=timedelta(days=1),\n    start_date=datetime(2024, 1, 1),\n    catchup=False,\n) as dag:\n\n    # Link supervisor config to dag\n    supervisor = Supervisor(dag=dag, cfg=cfg)\n```\n\n## Example DAG: [`airflow-config`](https://github.com/airflow-laminar/airflow-config)\n\n```yaml\n# @package _global_\n_target_: airflow_config.Configuration\ndefault_args:\n  _target_: airflow_config.DefaultArgs\n  retries: 0\n  depends_on_past: false\nall_dags:\n  _target_: airflow_config.DagArgs\n  start_date: \"2024-01-01\"\n  catchup: false\nextensions:\n  supervisor:\n    _target_: airflow_supervisor.SupervisorAirflowConfiguration\n    port: 9091\n    working_dir: \"/data/airflow/supervisor\"\n    config_path: \"/data/airflow/supervisor/supervisor.conf\"\n    program:\n      test:\n        _target_: airflow_supervisor.ProgramConfiguration\n        command: \"bash -c 'sleep 14400; exit 1'\"\n```\n\n```python\nfrom datetime import timedelta\nfrom airflow_config import load_config, DAG\nfrom airflow_supervisor import Supervisor\n\nconfig = load_config(config_name=\"airflow\")\n\nwith DAG(\n    dag_id=\"test-supervisor\",\n    schedule=timedelta(days=1),\n    config=config,\n) as dag:\n    supervisor = Supervisor(dag=dag, cfg=config.extensions[\"supervisor\"])\n```\n\n## Configuration\n\nSee [supervisor-pydantic](http://airflow-laminar.github.io/supervisor-pydantic/) for reference.\n\n- `SupervisorAirflowConfiguration`: Wrapper around `supervisor_pydantic.SupervisorConvenienceConfiguration`, with added airflow-specific configuration\n- `SupervisorSSHAirflowConfiguration`: Wrapper around `SupervisorAirflowConfiguration`, with added parameters for airflow's `SSHOperator`\n\n```mermaid\nclassDiagram\n    SupervisorConvenienceConfiguration \u003c|-- SupervisorAirflowConfiguration\n    SupervisorAirflowConfiguration \u003c|-- SupervisorSSHAirflowConfiguration\n\n    class SupervisorConvenienceConfiguration {\n      supervisor_pydantic.SupervisorConvenienceConfiguration\n    }\n\n    class SupervisorAirflowConfiguration{\n        # PythonSensor arguments\n        check_interval: timedelta\n        check_timeout: timedelta\n\n        # HighAvailabilityOperator arguments\n        runtime: timedelta\n        endtime: time\n        maxretrigger: int\n        reference_date: str\n\n        # Airflow arguments\n        stop_on_exit: bool\n        cleanup: bool\n        restart_on_initial: bool\n        restart_on_retrigger: bool\n    }\n    class SupervisorSSHAirflowConfiguration {\n      command_prefix: str\n\n      # Airflow SSHOperator Arguments\n      ssh_operator_args: SSHOperatorArgs\n    }\n```\n\n\u003e [!NOTE]\n\u003e This library is built on [supervisor-pydantic](https://github.com/airflow-laminar/supervisor-pydantic), which provides configuration elements for all supervisor structures, as well as self-contained tools for interacting with supervisor instances.\n\n\u003e [!NOTE]\n\u003e This library is designed to work with [airflow-config](https://github.com/airflow-laminar/airflow-config) for YAML-driven DAG definitions, enabling fully declarative Airflow workflows.\n\n\u003e [!NOTE]\n\u003e This library was generated using [copier](https://copier.readthedocs.io/en/stable/) from the [Base Python Project Template repository](https://github.com/python-project-templates/base).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fairflow-laminar%2Fairflow-supervisor","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fairflow-laminar%2Fairflow-supervisor","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fairflow-laminar%2Fairflow-supervisor/lists"}