{"id":46988089,"url":"https://github.com/guidok91/airflow","last_synced_at":"2026-03-11T13:43:09.906Z","repository":{"id":45232299,"uuid":"156104335","full_name":"guidok91/airflow","owner":"guidok91","description":"Airflow setup with Kubernetes.","archived":false,"fork":false,"pushed_at":"2026-03-01T04:25:03.000Z","size":60,"stargazers_count":2,"open_issues_count":2,"forks_count":1,"subscribers_count":0,"default_branch":"master","last_synced_at":"2026-03-01T07:29:42.289Z","etag":null,"topics":["airflow","docker","helm","kubernetes","postgresql"],"latest_commit_sha":null,"homepage":"","language":"Makefile","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/guidok91.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2018-11-04T16:55:50.000Z","updated_at":"2026-02-27T09:21:50.000Z","dependencies_parsed_at":"2024-12-13T12:22:09.705Z","dependency_job_id":"7a299d03-a198-4d1a-bcc3-a6132ec4e5d8","html_url":"https://github.com/guidok91/airflow","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/guidok91/airflow","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/guidok91%2Fairflow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/guidok91%2Fairflow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/guidok91%2Fairflow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/guidok91%2Fairflow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/guidok91","download_url":"https://codeload.github.com/guidok91/airflow/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/guidok91%2Fairflow/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30382676,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-11T12:49:11.341Z","status":"ssl_error","status_checked_at":"2026-03-11T12:46:41.342Z","response_time":84,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow","docker","helm","kubernetes","postgresql"],"created_at":"2026-03-11T13:43:09.403Z","updated_at":"2026-03-11T13:43:09.896Z","avatar_url":"https://github.com/guidok91.png","language":"Makefile","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Sample Airflow setup with Kubernetes\n[Apache Airflow](https://airflow.apache.org/) sample setup with Kubernetes.\n\nSpins up a local Kubernetes cluster for Airflow with [Kind](https://kind.sigs.k8s.io/):\n  - Leverages [Airflow's official Helm chart](https://airflow.apache.org/docs/helm-chart/stable/index.html).\n  - Uses a PostgreSQL database for Airflow.\n  - Uses [Kubernetes Executor](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/kubernetes.html) to run each Airflow task on an isolated pod.\n\nThis is meant as a sample local setup. In order to run it in a production environment, please refer to the [Airflow Helm chart production guide](https://airflow.apache.org/docs/helm-chart/stable/production-guide.html).\n\n- [Requirements](#requirements)\n- [Instructions](#instructions)\n- [Configuration](#configuration)\n- [DAG deployment](#dag-deployment)\n- [Custom Docker image for pods](#custom-docker-image-for-pods)\n- [Logs](#logs)\n- [Dependency management](#dependency-management)\n\n## Requirements\n[Kind](https://kind.sigs.k8s.io/), [Docker](https://www.docker.com/) and [Helm](https://helm.sh/) for local Kubernetes cluster.\n\n## Instructions\nThe repo includes a `Makefile`. You can run `make help` to see usage.\n\nBasic setup:\n- Run `make k8s-cluster-up` to spin up local Kubernetes cluster with Kind.\n- Run `make airflow-k8s-add-helm-chart` to add the official Airflow Helm chart to the local repo.\n- Run `make airflow-k8s-create-namespace` to create a namespace for the Airflow deployment.\n- Run `make airflow-k8s-up` to deploy Airflow on the local Kubernetes cluster.\n- On a separate terminal, run `make airflow-api-server-port-forward` to be able to access the Airflow UI on http://localhost:8080.\n\nThe credentials for the UI are admin/admin.\n\n## Configuration\nIf you need to customize the Airflow deployment you can edit [values.yaml](k8s/values.yaml) accordingly.\n\nif you need to tune [Airflow configuration](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html) you can add the corresponding environment variables in the `env` section of `values.yaml`.\n\nThe default `values.yaml` of the source Helm chart can be seen [here](https://github.com/apache/airflow/blob/main/chart/values.yaml). \n\n## DAG deployment\nDAGs are deployed via GitSync.\n\nGitSync acts as a side car container alongside the other Airflow pods, synchronising the `dags/` folder in the pods with the DAGs located in a Git repo of your choice (in this case https://github.com/guidok91/airflow/tree/master/dags).\n\n## Custom Docker image for pods\nA custom [Docker image](Dockerfile) is provided for the pods. Here we can install the Airflow dependencies we need.\n\n## Logs\nSo that Airflow logs don't get lost every time a task finishes (e.g. the pod gets deleted), the setup provides a [PersistentVolume](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) that shares the logs with the host system in the `data/` folder.\n\n## Dependency management\nDependabot is configured to periodically upgrade repo dependencies. See [dependabot.yml](.github/dependabot.yml).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fguidok91%2Fairflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fguidok91%2Fairflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fguidok91%2Fairflow/lists"}