{"id":19886845,"url":"https://github.com/cdmikechen/airflow-dag-operator","last_synced_at":"2025-07-11T02:33:45.798Z","repository":{"id":168166742,"uuid":"415246957","full_name":"cdmikechen/airflow-dag-operator","owner":"cdmikechen","description":"Use K8s CustomResourceDefinition to define and create Airflow Dags","archived":false,"fork":false,"pushed_at":"2022-07-15T08:45:10.000Z","size":185,"stargazers_count":1,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-01T04:25:25.454Z","etag":null,"topics":["airflow","airflow-dags","operator","quarkus"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cdmikechen.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-10-09T08:17:06.000Z","updated_at":"2024-03-20T00:22:27.000Z","dependencies_parsed_at":null,"dependency_job_id":"87c4bcc8-7528-4416-a549-7914387358d7","html_url":"https://github.com/cdmikechen/airflow-dag-operator","commit_stats":null,"previous_names":["cdmikechen/airflow-dag-operator"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/cdmikechen/airflow-dag-operator","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cdmikechen%2Fairflow-dag-operator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cdmikechen%2Fairflow-dag-operator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cdmikechen%2Fairflow-dag-operator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cdmikechen%2Fairflow-dag-operator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cdmikechen","download_url":"https://codeload.github.com/cdmikechen/airflow-dag-operator/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cdmikechen%2Fairflow-dag-operator/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264713034,"owners_count":23652705,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow","airflow-dags","operator","quarkus"],"created_at":"2024-11-12T17:39:47.057Z","updated_at":"2025-07-11T02:33:45.506Z","avatar_url":"https://github.com/cdmikechen.png","language":"Java","readme":"# Airflow Dag Operator\n\nUse K8s [CustomResourceDefinition](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/) \nto replace Airflow Git Sync strategy. The main idea of the project is to start a synchronization service \nwith [Quarkus Operator](https://github.com/quarkiverse/quarkus-operator-sdk) \non each airflow pod to synchronize the DAG/files into the DAG folder.\n\nThis project has included the docker image packaging part (buildconfig or quarkus build) \nand the modified helm chart template based on the official airflow project (https://github.com/apache/airflow/tree/main/chart).\n\n## Operation Strategy\n\n* use `kubectl` to create rbac and `CustomResourceDefinition` (operator can automatically create resources)\n* create some dag resource instances\n* start quarkus operator\n* operator can list dag resources or automatically perceive changes in resources, then create/update or delete dags in DAG folder.\n\n## CRD Schema\n\nResource description can be referred to [02-crd.yaml](example/02-crd.yaml) .\nThere are several important attributes in `CRD`, which are described here:\n\nParameter | Description | Default\n--- | --- | ---\n`type` | Type of CRD, it can be `dag_file`, `file` or `dag_yaml`. `dag_file` must be a DAG description file. `file` can be a python or other text format file.`dag_yaml` reference [dag-factory](https://github.com/ajbosco/dag-factory), but add some changes | `dag_file`\n`path`| File path. If the file path is empty, it defaults to the root directory of `dags`, otherwise it is a subdirectory under `dags`  |\n`file_name` | If `type` is `file`, we need a `file_name`. |\n`dag_name` | If `type` is `dag_file` or `dag_yaml`, we need a `dag_name`. If `dag_name` don't have `.py` suffix, the operator will automatically append it. | crd name\n`content`| If `type` is `dag_file` or `file`, It is the content of the file. |\n`paused`| If `paused` is not empty, the operator will scan the DAG status and automatically pause / unpause the task. |\n`dag_yaml`| The described of DAG by yaml, For details, please refer to [dag-factory](https://github.com/ajbosco/dag-factory) |\n\n## Dev Or Test\n\nWe can run our application in dev mode that enables live coding using:\n```shell script\n./mvnw compile quarkus:dev\n```\n\nAn example has been in `/example` folder. In `/example`, it includes `RBAC`，`CRD`, some cases and `Deployment` for test.\n\n## Quarkus Image Build\n\nIf we use OpenShift, we can use `BuildConfig` or `Tekton/Pipline` to build a native image. \nOtherwise, we can create a native executable using:\n```shell script\n./mvnw package -Pnative\n# if use macOS, you should use -Dquarkus.native.container-build=true to build quarkus in docker with a linux environment\ndocker build -f src/main/docker/Dockerfile.native -t quarkus/airflow-dag-operator .\n```\n\nOr, if we don't have GraalVM installed, we can run the native executable build in a container using:\n```shell script\n./mvnw package -Pnative -Dquarkus.native.container-build=true\ndocker build -f src/main/docker/Dockerfile.native -t quarkus/airflow-dag-operator .\n```\n\n## Airflow Helm Chart\n\nHelm dependency update to add postgresql chart and `lint`. We need helm3 to build.\n```shell script\n# dependency update\nhelm dep update\n\n# lint\nhelm lint\n\n# debug\nhelm install --dry-run --debug  -f values.yaml airflow -n airflow .\n```\n\nDeploy Chart\n```shell script\n# install\nhelm install -f values.yaml airflow -n airflow .\n\n# upgrade\nhelm upgrade -f values.yaml airflow -n airflow .\n\n# uninstall\nhelm uninstall airflow\n```\n\n## Support pause\n\nWe need to rebuild the image\n```shell\n# if want to support pause, we need to build by change `quarkus.datasource.jdbc` from false to true\n./mvnw package -Pnative -Dquarkus.datasource.jdbc=true\n```\n\nDue to the complexity of parsing DAG's python codes, we need to ensure that `dag_name` and `dag_id` are consistent for now.  \nNote that the helm has not been modified yet right now! by design, the operator will only turn on `support pause` on the scheduler node to avoid repeated executions.\n\n# History\n\n* 2021-10-09 1.0.0 First Commit \n* 2021-11-16 1.0.1 Update to quarkus-operator-sdk 2.0.0\n* 2022-03-27 1.0.1 Update to quarkus-operator-sdk 3.0.5\n* 2022-07-15 1.0.2 Update to quarkus-operator-sdk 4.0.0.RC / Support paused \n\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcdmikechen%2Fairflow-dag-operator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcdmikechen%2Fairflow-dag-operator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcdmikechen%2Fairflow-dag-operator/lists"}