https://github.com/bhavaniravi/airflow-kube-setup
How to deploy airflow on Kubernetes
https://github.com/bhavaniravi/airflow-kube-setup
airflow docker kubernetes
Last synced: about 1 year ago
JSON representation
How to deploy airflow on Kubernetes
- Host: GitHub
- URL: https://github.com/bhavaniravi/airflow-kube-setup
- Owner: bhavaniravi
- Created: 2020-03-17T09:18:45.000Z (over 6 years ago)
- Default Branch: main
- Last Pushed: 2023-07-02T16:04:00.000Z (almost 3 years ago)
- Last Synced: 2024-11-07T20:41:50.857Z (over 1 year ago)
- Topics: airflow, docker, kubernetes
- Language: Shell
- Homepage:
- Size: 1.98 MB
- Stars: 39
- Watchers: 3
- Forks: 41
- Open Issues: 4
-
Metadata Files:
- Readme: Readme.md
Awesome Lists containing this project
README
# Airflow Kubernetes Setup
> This setup is used and tested in airflow 1.10.10. It's a good idea to use latest version of Airflow 2.3.x+. Need help setting up infrastructure and training your team? Contact me bhavanicodes@gmail.com
The setup files are copied directly from airflow's repo and modified to fit the requirements.
One major change is instead of building the docker image from source, we use `pip` to install airflow
## Create Docker Image
```
cd scripts/docker
docker build -t airflow .
```
> Push the image to your favorite registry and get the URL
## Deploy in kubernetes
1. Find $IMAGE in the repository
2. Find $TAG in the repository
2. Change it with the docker image URL and tag respectively
```
cd scripts/
./kube/deploy.sh -d persistent_mode
```
The other supported modes is `git_mode`
## Walkthrough blog
[Deploying airflow on kubernetes](https://bhavaniravi.com/blog/deploying-airflow-on-kubernetes/)