{"id":13651415,"url":"https://github.com/stwind/airflow-on-kubernetes","last_synced_at":"2025-04-11T01:10:35.717Z","repository":{"id":141904223,"uuid":"230403265","full_name":"stwind/airflow-on-kubernetes","owner":"stwind","description":"Bare minimal Airflow on Kubernetes (Local, EKS, AKS)","archived":false,"fork":false,"pushed_at":"2020-03-04T10:39:47.000Z","size":93,"stargazers_count":52,"open_issues_count":1,"forks_count":9,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-24T21:51:09.430Z","etag":null,"topics":["airflow","aks","aws","azure","eks","kubernetes"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/stwind.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2019-12-27T08:25:37.000Z","updated_at":"2024-09-14T19:17:22.000Z","dependencies_parsed_at":"2024-01-14T12:29:18.675Z","dependency_job_id":null,"html_url":"https://github.com/stwind/airflow-on-kubernetes","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stwind%2Fairflow-on-kubernetes","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stwind%2Fairflow-on-kubernetes/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stwind%2Fairflow-on-kubernetes/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stwind%2Fairflow-on-kubernetes/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/stwind","download_url":"https://codeload.github.com/stwind/airflow-on-kubernetes/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248322600,"owners_count":21084337,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow","aks","aws","azure","eks","kubernetes"],"created_at":"2024-08-02T02:00:49.233Z","updated_at":"2025-04-11T01:10:35.696Z","avatar_url":"https://github.com/stwind.png","language":"Python","funding_links":[],"categories":["Best practices, lessons learned and cool use cases","Boas praticas, lições aprendidas e bons usos de caso"],"sub_categories":[],"readme":"# Bare Minimal Airflow On Kubernetes\n\nAirflow and Kubernetes are perfect match, but they are complicated beasts to each their own. There are many [attempts](#references) to provide partial or complete deployment solution with custom helm charts. But usually one just look around for useful snippets and ideas to build their own solution instead of directly installing them.\n\nIn the repo, instead of providing another full feature helm chart or terraform module, I try to use just command line to setup a minimal Airflow on Kubernetes. Anyone interested could hopefully just copy and paste to reproduce the same results, maybe as a starting point or trouble shooting tool for their own solution.\n\n## \u003ca name=\"toc\"\u003e\u003c/a\u003eContents\n\n* [Prerequisites](#prerequisites)\n* [Preparation](#preparation)\n* [Environments](#environments)\n\t* [Local](#local)\n\t\t* [MySQL](#local-mysql)\n\t\t* [Initialize Database](#local-init-db)\n\t\t* [Start Airflow](#local-start)\n\t\t* [Testing](#local-test)\n\t\t* [Cleanup](#local-cleanup)\n  * [EKS](#eks)\n\t  * [IAM Setup](#eks-iam)\n\t  * [ECR Setup](#eks-ecr)\n\t  * [Create Cluster](#eks-cluster)\n\t  * [Create RDS](#eks-rds)\n\t  * [Connect VPCs](#eks-vpc)\n\t  * [Testing Airflow](#eks-test)\n\t  * [Cleanup](#eks-cleanup)\n  * [AKS](#aks)\n\t  * [ACR setup](#aks-acr)\n\t  * [Cluster Setup](#aks-cluster)\n\t  * [MySQL Setup](#aks-mysql)\n\t  * [Prepare Volume](#aks-vol)\n\t  * [Testing Airflow](#aks-test)\n\t  * [Cleanup](#aks-cleanup)\n\n## Prerequisites\n\n* common\n  * [jq](https://stedolan.github.io/jq/): `1.6`\n* Local\n  * Docker For Mac: `19.03.5`\n  * Kubernetes: `v1.15.5`\n  * Helm: `v3.0.1`\n* EKS\n  * [awscli](https://aws.amazon.com/cli/): `1.16.130`\n  * [eksctl](https://eksctl.io/): `0.11.1`\n* AKS\n  * [azure-cli](https://docs.microsoft.com/en-us/cli/azure/): `2.0.78`\n\n## Preparation\n\nAirflow does [not allow SQLite to be used with the kubernetes executor](https://github.com/apache/airflow/blob/6fffa5b0d7840727a96dc1765a0166656bc7ea52/airflow/configuration.py#L170), so you need to have a MySQL or PostgreSQL server. For this demostration, we use MySQL.\n\n### Build the docker image\n\n```shell\n$ docker build -t my/airflow -\u003c\u003c'EOF'\nFROM python:3.7.6-slim\n\nARG AIRFLOW_USER_HOME=/var/lib/airflow\nARG AIRFLOW_USER=\"airflow\"\nARG AIRFLOW_UID=\"1000\"\nARG AIRFLOW_GID=\"100\"\nENV AIRFLOW_HOME=$AIRFLOW_USER_HOME\n\nRUN mkdir $AIRFLOW_USER_HOME \u0026\u0026 \\\n  useradd -ms /bin/bash -u $AIRFLOW_UID airflow \u0026\u0026 \\\n  chown $AIRFLOW_USER:$AIRFLOW_GID $AIRFLOW_USER_HOME \u0026\u0026 \\\n  buildDeps='freetds-dev libkrb5-dev libsasl2-dev libssl-dev libffi-dev libpq-dev' \\\n  apt-get update \u0026\u0026 \\\n  apt-get install -yqq --no-install-recommends $buildDeps build-essential default-libmysqlclient-dev \u0026\u0026 \\\n  pip install --no-cache-dir 'apache-airflow[crypto,kubernetes,mysql]' \u0026\u0026 \\\n  apt-get purge --auto-remove -yqq $buildDeps \u0026\u0026 \\\n  apt-get autoremove -yqq --purge \u0026\u0026 \\\n  rm -rf /var/lib/apt/lists/*\n\nUSER $AIRFLOW_UID\n\nWORKDIR $AIRFLOW_USER_HOME\nEOF\n```\n\n## Environments\n\n### \u003ca name=\"local\"\u003e\u003c/a\u003eLocal\n\n#### \u003ca name=\"local-mysql\"\u003e\u003c/a\u003eMySQL [▲](#toc) \n\nIn case you don't have one yet, you can install with helm.\n\nSave the following the content as `mysql/values.yaml`\n\n```yaml\n---\n# put your more serious password here\nmysqlRootPassword: root\nmysqlUser: airflow\nmysqlPassword: airflow\nmysqlDatabase: airflow\n\nconfigurationFiles:\n  mysql.cnf: |-\n    [mysqld]\n    # https://airflow.apache.org/docs/stable/faq.html#how-to-fix-exception-global-variable-explicit-defaults-for-timestamp-needs-to-be-on-1\n    explicit_defaults_for_timestamp=1\n```\n\nInstall the [MySQL chart](https://github.com/helm/charts/tree/master/stable/mysql)\n\nBecause `helm init` no longer exists in Helm 3, we need to add stable repo manually\n\n```shell\n$ helm repo add stable https://kubernetes-charts.storage.googleapis.com/\n$ helm install -f mysql/values.yaml mysql stable/mysql\n```\n\nMake sure the `explicit_defaults_for_timestamp` is `ON`\n\n```sh\n$ kubectl exec -ti $(kubectl get po -l app=mysql,release=mysql -o jsonpath=\"{.items[0].metadata.name}\") -- mysql -u root -p -e \"SHOW VARIABLES LIKE 'explicit_defaults_for_timestamp'\"\nEnter password: root\n+---------------------------------+-------+\n| Variable_name                   | Value |\n+---------------------------------+-------+\n| explicit_defaults_for_timestamp | ON    |\n+---------------------------------+-------+\n```\n\n#### \u003ca name=\"local-init-db\"\u003e\u003c/a\u003eInitialize Database [▲](#toc)\n\n```sh\n$ kubectl run airflow-initdb \\\n    --restart=Never -ti --rm --image-pull-policy=IfNotPresent --generator=run-pod/v1 \\\n    --image=my/airflow \\\n    --env AIRFLOW__CORE__LOAD_EXAMPLES=False \\\n    --env AIRFLOW__CORE__SQL_ALCHEMY_CONN=mysql://airflow:airflow@mysql.default/airflow \\\n    --command -- airflow initdb\n```\n\n#### \u003ca name=\"local-start\"\u003e\u003c/a\u003eStart Airflow [▲](#toc)\n\n```sh\n$ kubectl run airflow -ti --rm --restart=Never --image=my/airflow --overrides='\n{\n  \"spec\": {\n    \"containers\":[{\n      \"name\": \"webserver\",\n      \"image\": \"my/airflow\",\n      \"imagePullPolicy\":\"IfNotPresent\",\n      \"command\": [\"airflow\",\"webserver\"],\n      \"stdin\": true,\n      \"tty\": true,\n      \"env\": [\n        {\"name\":\"AIRFLOW__CORE__LOAD_EXAMPLES\",\"value\":\"False\"},\n        {\"name\":\"AIRFLOW__CORE__SQL_ALCHEMY_CONN\",\"value\":\"mysql://airflow:airflow@mysql.default/airflow\"}, \n        {\"name\":\"AIRFLOW__CORE__EXECUTOR\",\"value\":\"KubernetesExecutor\"},\n        {\"name\":\"AIRFLOW__KUBERNETES__WORKER_CONTAINER_REPOSITORY\",\"value\":\"my/airflow\"},\n        {\"name\":\"AIRFLOW__KUBERNETES__WORKER_CONTAINER_TAG\",\"value\":\"latest\"},\n        {\"name\":\"AIRFLOW__KUBERNETES__DAGS_VOLUME_HOST\",\"value\":\"'$PWD/dags'\"}\n      ],\n      \"volumeMounts\": [{\"mountPath\": \"/var/lib/airflow/dags\",\"name\": \"store\"}]\n    },{\n      \"name\": \"scheduler\",\n      \"image\": \"my/airflow\",\n      \"imagePullPolicy\":\"IfNotPresent\",\n      \"command\": [\"airflow\",\"scheduler\"],\n      \"stdin\": true,\n      \"tty\": true,\n      \"env\": [\n        {\"name\":\"AIRFLOW__CORE__LOAD_EXAMPLES\",\"value\":\"False\"},\n        {\"name\":\"AIRFLOW__CORE__SQL_ALCHEMY_CONN\",\"value\":\"mysql://airflow:airflow@mysql.default/airflow\"}, \n        {\"name\":\"AIRFLOW__CORE__EXECUTOR\",\"value\":\"KubernetesExecutor\"},\n        {\"name\":\"AIRFLOW__KUBERNETES__WORKER_CONTAINER_REPOSITORY\",\"value\":\"my/airflow\"},\n        {\"name\":\"AIRFLOW__KUBERNETES__WORKER_CONTAINER_TAG\",\"value\":\"latest\"},\n        {\"name\":\"AIRFLOW__KUBERNETES__DAGS_VOLUME_HOST\",\"value\":\"'$PWD/dags'\"},\n        {\"name\":\"AIRFLOW__KUBERNETES__KUBE_CLIENT_REQUEST_ARGS\",\"value\":\"\"}\n      ],\n      \"volumeMounts\": [{\"mountPath\": \"/var/lib/airflow/dags\",\"name\": \"store\"}]\n    }],\n    \"volumes\": [{\"name\":\"store\",\"hostPath\":{\"path\":\"'$PWD/dags'\",\"type\":\"Directory\"}}]\n  }\n}'\n```\n\nThis will show the logs of webserver, you can also show the scheduler logs by running the following comamnd (on another shell):\n\n```sh\n$ kubectl logs -f airflow -c scheduler\n```\n\nAlso, open another shell and run this command to see how pods come and go\n\n```sh\n$ kubectl get po -w\n```\n\nOr if you want to see the UI\n\n```sh\n$ kubectl port-forward airflow 8080\n```\n\n#### \u003ca name=\"local-test\"\u003e\u003c/a\u003eTesting [▲](#toc)\n\nOpen another shell, list the dags by running the following \n\n```sh\n$ kubectl exec -ti airflow -c webserver airflow list_dags\n-------------------------------------------------------------------\nDAGS\n-------------------------------------------------------------------\nfoobar\n```\n\nUnpause and trigger the dag\n\n```sh\n$ kubectl exec -ti airflow -c webserver airflow unpause foobar\n$ kubectl exec -ti airflow -c webserver airflow trigger_dag foobar\n$ kubectl exec -ti airflow -c webserver airflow list_dag_runs foobar\n```\n\n#### \u003ca name=\"local-cleanup\"\u003e\u003c/a\u003eCleanup [▲](#toc)\n\nDelete the pod and MySQL\n\n```sh\n$ kubectl delete po airflow\n$ helm delete mysql\n```\n\n### \u003ca name=\"eks\"\u003e\u003c/a\u003eEKS\n\nSome environment variables we will use later\n\n```sh\n$ export AWS_REGION=$(aws configure get region)\n$ export AWS_ACCOUNT_ID=$(aws sts get-caller-identity | jq -r '.Account')\n```\n\n#### \u003ca name=\"eks-iam\"\u003e\u003c/a\u003eIAM Setup [▲](#toc) \n\nAttach the following managed IAM policies\n\n|Policy|Arn|\n|---|---|\n|[`AmazonEC2ContainerRegistryFullAccess`]((https://docs.aws.amazon.com/AmazonECR/latest/userguide/ecr_managed_policies.html#AmazonEC2ContainerRegistryFullAccess))|`arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryFullAccess`|\n|`AWSCloudFormationFullAccess`|`arn:aws:iam::aws:policy/AWSCloudFormationFullAccess `|\n|`AmazonEC2FullAccess`|`arn:aws:iam::aws:policy/AmazonEC2FullAccess`|\n|`AmazonECS_FullAccess`|`arn:aws:iam::aws:policy/AmazonECS_FullAccess`|\n|[`AmazonRDSFullAccess`](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_PerfInsights.access-control.html#USER_PerfInsights.access-control.managed-policy)|`arn:aws:iam::aws:policy/AmazonRDSFullAccess`|\n\nand create a custom policy called `AmazonEKSFullAccess` and attach to your user or group\n\n```json\n{\n  \"Version\": \"2012-10-17\",\n  \"Statement\": [\n    {\n      \"Effect\": \"Allow\",\n      \"Action\": \"eks:*\",\n      \"Resource\": \"*\"\n    }\n  ]\n}\n```\n\n#### \u003ca name=\"eks-ecr\"\u003e\u003c/a\u003eECR Setup [▲](#toc)\n\nFirst we have to setup a ECR repository to hold the image, you can create it in the web console or using the cli.\n\nHere we create a repository called `airflow`\n\n```sh\n$ aws ecr create-repository --repository-name airflow\n```\n\nlogin to the registry with the output of\n\n```sh\n$ aws ecr get-login --registry-ids $(aws ecr describe-repositories --repository-names airflow | jq '.repositories[0].registryId' -r)\n```\n\npush the image (replace `account_id` and `region` with your own)\n\n```sh\n$ docker tag my/airflow ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/airflow\n$ docker push ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/airflow\n```\n\nwhen done, your image should be in the repository\n\n```sh\n$ aws ecr describe-images --repository-name airflow\n```\n\n#### \u003ca name=\"eks-cluster\"\u003e\u003c/a\u003eCreate EKS cluster [▲](#toc)\n\nCreate a minimal EKS cluster called `airflow-test`\n\n```sh\n$ eksctl create cluster --name airflow-test --version 1.14 --nodes 1 --ssh-access\n```\n\nAfter a few minutes for cluster, you should have a new cluster in your kubectl contexts, run the following commands to make sure\n\n```sh\n$ kubectl config get-contexts\n$ kubectl config current-context\n$ kubectl cluster-info\n```\n\nget more information about the cluster\n\n```sh\n$ eksctl get cluster -n airflow-test\n```\n\n#### \u003ca name=\"eks-rds\"\u003e\u003c/a\u003eCreate RDS [▲](#toc)\n\nNow we are going to create an RDS to hold the data.\n\nThe cluster we just created is in its own VPC, but you probably don't want the RDS in the same VPC with the cluster. We are going to create another VPC and use [VPC Peering](https://docs.aws.amazon.com/vpc/latest/peering/what-is-vpc-peering.html) to connect them. Most of the following techniques are borrowed from this [great article](https://dev.to/bensooraj/accessing-amazon-rds-from-aws-eks-2pc3). \n\nTo summarize, an [RDS in VPC](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_VPC.WorkingWithRDSInstanceinaVPC.html#USER_VPC.Subnets) requires **an VPC with 2 subnets in different availability zones**.\n\nFirst we create the VPC and save the resulting VPC id to `RDS_VPC_ID` and enable DNS hostnames support\n\n```sh\n$ export RDS_VPC_CIDR_BLOCK=10.0.0.0/24\n$ aws ec2 create-vpc --cidr-block ${RDS_VPC_CIDR_BLOCK} | jq -r '{VpcId:.Vpc.VpcId,CidrBlock:.Vpc.CidrBlock}'\n{\n  \"VpcId\": \"vpc-0c92ff62833b93381\",\n  \"CidrBlock\": \"10.0.0.0/24\"\n}\n$ export RDS_VPC_ID=vpc-0c92ff62833b93381\n$ aws ec2 modify-vpc-attribute --vpc-id ${RDS_VPC_ID} --enable-dns-hostnames '{\"Value\":true}'\n$ aws ec2 create-tags --resources ${RDS_VPC_ID} --tags Key=Name,Value=airflow/VPC\n```\n\nNext let's check what availability zones are in the current region\n\n```sh\n$ aws ec2 describe-availability-zones --region $(aws configure get region) | jq -r '.AvailabilityZones[].ZoneName'\nap-northeast-1a\nap-northeast-1c\nap-northeast-1d\n```\n\nCreate two subnets in two different availability zones\n\n```sh\n$ aws ec2 create-subnet --availability-zone \"ap-northeast-1a\" --vpc-id ${RDS_VPC_ID} --cidr-block \"10.0.0.0/25\" | jq '{SubnetId:.Subnet.SubnetId,AvailabilityZone:.Subnet.AvailabilityZone,CidrBlock:.Subnet.CidrBlock,VpcId:.Subnet.VpcId}'\n{\n  \"SubnetId\": \"subnet-0cb408bee95aa7fb3\",\n  \"AvailabilityZone\": \"ap-northeast-1a\",\n  \"CidrBlock\": \"10.0.0.0/25\",\n  \"VpcId\": \"vpc-0c92ff62833b93381\"\n}\n$ aws ec2 create-tags --resources \"subnet-0cb408bee95aa7fb3\" --tags Key=Name,Value=airflow/Subnet1\n$ aws ec2 create-subnet --availability-zone \"ap-northeast-1c\" --vpc-id ${RDS_VPC_ID} --cidr-block 10.0.0.128/25 | jq -r '{SubnetId:.Subnet.SubnetId,AvailabilityZone:.Subnet.AvailabilityZone,CidrBlock:.Subnet.CidrBlock,VpcId:.Subnet.VpcId}'\n{\n  \"SubnetId\": \"subnet-04871a023fdf047ec\",\n  \"AvailabilityZone\": \"ap-northeast-1c\",\n  \"CidrBlock\": \"10.0.0.128/25\",\n  \"VpcId\": \"vpc-0c92ff62833b93381\"\n}\n$ aws ec2 create-tags --resources \"subnet-04871a023fdf047ec\" --tags Key=Name,Value=airflow-rds/Subnet1\n```\n\nAssociate the subnets with their VPC's default route table\n\n```sh\n$ export RDS_ROUTE_TABLE_ID=$(aws ec2 describe-route-tables --filters Name=vpc-id,Values=${RDS_VPC_ID} | jq -r '.RouteTables[0].RouteTableId')\n$ aws ec2 associate-route-table --route-table-id ${RDS_ROUTE_TABLE_ID} --subnet-id \"subnet-04871a023fdf047ec\"\n{\n    \"AssociationId\": \"rtbassoc-0d4b9e2d85994a074\"\n}\n$ aws ec2 associate-route-table --route-table-id ${RDS_ROUTE_TABLE_ID} --subnet-id \"subnet-0cb408bee95aa7fb3\"\n{\n    \"AssociationId\": \"rtbassoc-0ee558d300f3bb605\"\n}\n```\n\nCreate a subnet group\n\n```sh\n$ aws rds create-db-subnet-group --db-subnet-group-name \"airflow\" --db-subnet-group-description \"Subnet Group For Airflow\" --subnet-ids \"subnet-04871a023fdf047ec\" \"subnet-0cb408bee95aa7fb3\" | jq -r '{DBSubnetGroupName:.DBSubnetGroup.DBSubnetGroupName,VpcId:.DBSubnetGroup.VpcId,Subnets:.DBSubnetGroup.Subnets[].SubnetIdentifier}'\n{\n  \"DBSubnetGroupName\": \"airflow\",\n  \"VpcId\": \"vpc-0c92ff62833b93381\",\n  \"Subnets\": \"subnet-04871a023fdf047ec\"\n}\n{\n  \"DBSubnetGroupName\": \"airflow\",\n  \"VpcId\": \"vpc-0c92ff62833b93381\",\n  \"Subnets\": \"subnet-0cb408bee95aa7fb3\"\n}\n```\n\nCreate a security group\n\n```sh\n$ aws ec2 create-security-group --group-name airflow-rds/SecurityGroup --description \"Airflow RDS security group\" --vpc-id ${RDS_VPC_ID}\n{\n    \"GroupId\": \"sg-00ec4f82799271da0\"\n}\n$ export RDS_VPC_SECURITY_GROUP_ID=sg-00ec4f82799271da0\n```\n\nNow create the DB instance\n\n```sh\n$ aws rds create-db-instance \\\n  --db-name airflow \\\n  --db-instance-identifier airflow \\\n  --allocated-storage 10 \\\n  --db-instance-class db.t2.micro \\\n  --engine mysql \\\n  --engine-version \"5.7.16\" \\\n  --master-username airflowmaster \\\n  --master-user-password airflowmaster \\\n  --no-publicly-accessible \\\n  --no-multi-az \\\n  --no-auto-minor-version-upgrade \\\n  --vpc-security-group-ids ${RDS_VPC_SECURITY_GROUP_ID} \\\n  --db-subnet-group-name \"airflow\" \\\n  --backup-retention-period 0 \\\n  --port 3306 | jq -r '{DBInstanceIdentifier:.DBInstance.DBInstanceIdentifier,Engine:.DBInstance.Engine,DBName:.DBInstance.DBName,VpcSecurityGroups:.DBInstance.VpcSecurityGroups,EngineVersion:.DBInstance.EngineVersion,PubliclyAccessible:.DBInstance.PubliclyAccessible}'\n{\n  \"DBInstanceIdentifier\": \"airflow\",\n  \"Engine\": \"mysql\",\n  \"DBName\": \"airflow\",\n  \"VpcSecurityGroups\": [\n    {\n      \"VpcSecurityGroupId\": \"sg-00ec4f82799271da0\",\n      \"Status\": \"active\"\n    }\n  ],\n  \"EngineVersion\": \"5.7.16\",\n  \"PubliclyAccessible\": false\n}\n```\n\nThe default parameter group for RDS mysql 5.7 already have `explicit_defaults_for_timestamp` set to `ON`\n\n```sh\n$ aws rds describe-engine-default-parameters --db-parameter-group-family mysql5.7 | jq -r '.EngineDefaults.Parameters[] | select(.ParameterName==\"explicit_defaults_for_timestamp\") | {ParameterName:.ParameterName,ParameterValue:.ParameterValue}'\n{\n  \"ParameterName\": \"explicit_defaults_for_timestamp\",\n  \"ParameterValue\": \"1\"\n}\n```\n\n#### \u003ca name=\"eks-vpc\"\u003e\u003c/a\u003eConnect VPCs [▲](#toc)\n\nWe are now going to connect the two VPCs of the eks cluster and the RDS. \n\nLet's first figure out the VPC ID of the eks cluster\n\n```sh\n$ export EKS_VPC_ID=$(aws eks describe-cluster --name airflow-test | jq -r '.cluster.resourcesVpcConfig.vpcId')\n```\n\nInitiate a connection from the cluster to rds\n\n```sh\n$ aws ec2 create-vpc-peering-connection --vpc-id ${EKS_VPC_ID} --peer-vpc-id ${RDS_VPC_ID} | jq -r '{VpcPeeringConnectionId:.VpcPeeringConnection.VpcPeeringConnectionId}'\n{\n  \"VpcPeeringConnectionId\": \"pcx-008a8b86c35e07ff4\"\n}\n$ export VPC_PEERING_CONNECTION_ID=pcx-008a8b86c35e07ff4\n```\n\nAccept the connection\n\n```sh\n$ aws ec2 accept-vpc-peering-connection --vpc-peering-connection-id ${VPC_PEERING_CONNECTION_ID}\n```\n\nGive it a name `airflow/VpcPeering` and enable [DNS resolution](https://docs.aws.amazon.com/vpc/latest/peering/modify-peering-connections.html#vpc-peering-dns)\n\n```sh\n$ aws ec2 create-tags --resources ${VPC_PEERING_CONNECTION_ID} --tags Key=Name,Value=airflow/VpcPeering\n$ aws ec2 modify-vpc-peering-connection-options \\\n    --vpc-peering-connection-id ${VPC_PEERING_CONNECTION_ID} \\\n    --requester-peering-connection-options '{\"AllowDnsResolutionFromRemoteVpc\":true}' \\\n    --accepter-peering-connection-options '{\"AllowDnsResolutionFromRemoteVpc\":true}'\n```\n\nUpdate the route tables of the cluster and RDS to route mutual traffic\n\n```sh\n$ export EKS_ROUTE_TABLE_ID=$(aws ec2 describe-route-tables --filters 'Name=\"tag:aws:cloudformation:logical-id\",Values=\"PublicRouteTable\"' 'Name=\"tag:alpha.eksctl.io/cluster-name\",Values=\"airflow-test\"' | jq -r '.RouteTables[0].RouteTableId')\n$ export EKS_VPC_CIDR_BLOCK=$(aws ec2 describe-vpcs --vpc-id $(eksctl get cluster -n airflow-test -o json | jq -rj '.[0].ResourcesVpcConfig.VpcId') | jq -rj '.Vpcs[0].CidrBlock')\n$ aws ec2 create-route --route-table-id ${EKS_ROUTE_TABLE_ID} --destination-cidr-block ${RDS_VPC_CIDR_BLOCK} --vpc-peering-connection-id ${VPC_PEERING_CONNECTION_ID}\n$ aws ec2 create-route --route-table-id ${RDS_ROUTE_TABLE_ID} --destination-cidr-block ${EKS_VPC_CIDR_BLOCK} --vpc-peering-connection-id ${VPC_PEERING_CONNECTION_ID}\n```\n\nUpdate RDS's security group to allow mysql connection \n\n```sh\n$ aws ec2 authorize-security-group-ingress --group-id ${RDS_VPC_SECURITY_GROUP_ID} --protocol tcp --port 3306 --cidr ${EKS_VPC_CIDR_BLOCK}\n```\n\nLet's test the connection. First create an [ExternalName](https://kubernetes.io/docs/concepts/services-networking/service/#externalname) service as an alias of the RDS endpoint\n\n```sh\n$ export RDS_ENDPOINT=$(aws rds describe-db-instances --filters Name=db-instance-id,Values=airflow | jq -r '.DBInstances[0].Endpoint.Address')\n$ kubectl create service externalname mysql --external-name ${RDS_ENDPOINT}\n```\n\nTest the RDS port with `nc` with and alpine container\n\n```sh\n$ kubectl run test-mysql --restart=Never -ti --rm \\\n  --image=alpine --generator=run-pod/v1 --image-pull-policy=IfNotPresent \\\n  --command -- sh -c 'nc -zv mysql 3306 \u0026\u003e /dev/null \u0026\u0026 echo \"online\" || echo \"offline\"'\nonline\npod \"test-mysql\" deleted\n```\n\n#### \u003ca name=\"eks-test\"\u003e\u003c/a\u003eTesting Airflow [▲](#toc)\n\nWe are now ready to test airflow. First let's initialize the database\n\n```sh\n$ kubectl run airflow-initdb \\\n    --restart=Never -ti --rm --image-pull-policy=IfNotPresent --generator=run-pod/v1 \\\n    --image=${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/airflow \\\n    --env AIRFLOW__CORE__LOAD_EXAMPLES=False \\\n    --env AIRFLOW__CORE__SQL_ALCHEMY_CONN=mysql://airflowmaster:airflowmaster@mysql/airflow \\\n    --command -- airflow initdb\n```\n\nCopy dags onto every nodes\n\n```sh\n$ export EKS_AUTOSCALLING_GROUP=$(aws cloudformation describe-stack-resources --stack-name $(eksctl get nodegroup --cluster airflow-test -o json | jq -r '.[0].StackName') | jq -r '.StackResources[] | select(.ResourceType==\"AWS::AutoScaling::AutoScalingGroup\") | .PhysicalResourceId')\n$ declare -a EKS_NODE_INSTANCE_IDS=(`aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names ${EKS_AUTOSCALLING_GROUP} | jq -r '.AutoScalingGroups[0].Instances[].InstanceId'`)\nfor EKS_NODE_INSTANCE_ID in \"${EKS_NODE_INSTANCE_IDS[@]}\"\ndo\n    EKS_NODE_HOST=$(aws ec2 describe-instances --instance-ids ${EKS_NODE_INSTANCE_ID} | jq -r '.Reservations[0].Instances[0].PublicDnsName')\n    ssh -q -o \"UserKnownHostsFile=/dev/null\" -o \"StrictHostKeyChecking=no\" ec2-user@${EKS_NODE_HOST} \"sudo rm -f /opt/dags/*\"\n    tar -cf - dags/*.py | ssh -q -o \"UserKnownHostsFile=/dev/null\" -o \"StrictHostKeyChecking=no\" ec2-user@${EKS_NODE_HOST} \"sudo tar -x --no-same-owner -C /opt\"\ndone\n```\n\nCreate a service account, and give it the necessary permissions\n\n```sh\n$ kubectl apply -f - \u003c\u003c'EOF'\n---\napiVersion: v1\nkind: ServiceAccount\nmetadata:\n  name: airflow\n---\napiVersion: rbac.authorization.k8s.io/v1\nkind: Role\nmetadata:\n  name: airflow\nrules:\n  - apiGroups: [\"\"]\n    resources: [\"configmaps\"]\n    verbs: [\"get\", \"watch\", \"list\"]\n  - apiGroups: [\"\"]\n    resources: [\"secrets\"]\n    verbs: [\"get\", \"watch\", \"list\"]\n  - apiGroups: [\"\"]\n    resources: [\"pods\"]\n    verbs: [\"get\", \"list\", \"watch\", \"create\", \"update\", \"patch\", \"delete\"]\n  - apiGroups: [\"\"]\n    resources: [\"pods/exec\"]\n    verbs: [\"get\", \"create\"]\n  - apiGroups: [\"\"]\n    resources: [\"pods/log\"]\n    verbs: [\"get\"]\n---\nkind: RoleBinding\napiVersion: rbac.authorization.k8s.io/v1\nmetadata:\n  name: airflow\nsubjects:\n  - kind: ServiceAccount\n    name: airflow\nroleRef:\n  kind: Role\n  name: airflow\n  apiGroup: rbac.authorization.k8s.io\nEOF\n```\n\nNow run the scheduler (skipping the webserver this time)\n\n```sh\n$ kubectl run airflow -ti --rm --restart=Never \\\n    --image=${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/airflow --overrides='\n{\n  \"spec\": {\n    \"serviceAccountName\":\"airflow\",\n    \"containers\":[{\n      \"name\": \"scheduler\",\n      \"image\": \"'${AWS_ACCOUNT_ID}'.dkr.ecr.'${AWS_REGION}'.amazonaws.com/airflow\",\n      \"imagePullPolicy\":\"IfNotPresent\",\n      \"command\": [\"airflow\",\"scheduler\"],\n      \"stdin\": true,\n      \"tty\": true,\n      \"env\": [\n        {\"name\":\"AIRFLOW__CORE__LOAD_EXAMPLES\",\"value\":\"False\"},\n        {\"name\":\"AIRFLOW__CORE__SQL_ALCHEMY_CONN\",\"value\":\"mysql://airflowmaster:airflowmaster@mysql/airflow\"}, \n        {\"name\":\"AIRFLOW__CORE__EXECUTOR\",\"value\":\"KubernetesExecutor\"},\n        {\"name\":\"AIRFLOW__KUBERNETES__WORKER_CONTAINER_REPOSITORY\",\"value\":\"'${AWS_ACCOUNT_ID}'.dkr.ecr.'${AWS_REGION}'.amazonaws.com/airflow\"},\n        {\"name\":\"AIRFLOW__KUBERNETES__WORKER_CONTAINER_TAG\",\"value\":\"latest\"},\n        {\"name\":\"AIRFLOW__KUBERNETES__WORKER_SERVICE_ACCOUNT_NAME\",\"value\":\"airflow\"},\n        {\"name\":\"AIRFLOW__KUBERNETES__KUBE_CLIENT_REQUEST_ARGS\",\"value\":\"\"},\n        {\"name\":\"AIRFLOW__KUBERNETES__DAGS_VOLUME_HOST\",\"value\":\"/opt/dags\"}\n      ],\n      \"volumeMounts\": [{\"mountPath\": \"/var/lib/airflow/dags\",\"name\": \"dags\"}]\n    }],\n    \"volumes\": [{\"name\":\"dags\",\"hostPath\":{\"path\":\"/opt/dags\",\"type\":\"Directory\"}}]\n  }\n}'\n```\n\nOpen another terminal to watch pods status\n\n```sh\n$ kubectl get po -w\n```\n\nWe are now ready to test airflow functionalities\n\n```sh\n$ kubectl exec -ti airflow airflow list_dags\n$ kubectl exec -ti airflow airflow unpause foobar\n$ kubectl exec -ti airflow airflow trigger_dag foobar\n$ kubectl exec -ti airflow airflow list_dag_runs foobar\n```\n\nthe pods life cycle would be something like\n\n```\nNAME                                         READY   STATUS    RESTARTS   AGE\nfoobarfoo-b3bbbef345334d58863d0da4e11faba9   0/1     Pending   0          \u003cinvalid\u003e\nfoobarfoo-b3bbbef345334d58863d0da4e11faba9   0/1     Pending   0          \u003cinvalid\u003e\nfoobarfoo-b3bbbef345334d58863d0da4e11faba9   0/1     ContainerCreating   0          \u003cinvalid\u003e\nfoobarfoo-b3bbbef345334d58863d0da4e11faba9   1/1     Running             0          0s\nfoobarbar-d5457f56526a4084bb964942a18f95d9   0/1     Pending             0          \u003cinvalid\u003e\nfoobarbar-d5457f56526a4084bb964942a18f95d9   0/1     Pending             0          \u003cinvalid\u003e\nfoobarbar-d5457f56526a4084bb964942a18f95d9   0/1     ContainerCreating   0          \u003cinvalid\u003e\nfoobarbar-d5457f56526a4084bb964942a18f95d9   1/1     Running             0          0s\nfoobarfoo-b3bbbef345334d58863d0da4e11faba9   0/1     Completed           0          9s\nfoobarfoo-b3bbbef345334d58863d0da4e11faba9   0/1     Terminating         0          10s\nfoobarfoo-b3bbbef345334d58863d0da4e11faba9   0/1     Terminating         0          10s\nfoobarbar-d5457f56526a4084bb964942a18f95d9   0/1     Completed           0          9s\nfoobarbar-d5457f56526a4084bb964942a18f95d9   0/1     Terminating         0          10s\nfoobarbar-d5457f56526a4084bb964942a18f95d9   0/1     Terminating         0          10s\n\n```\n\n#### \u003ca name=\"eks-cleanup\"\u003e\u003c/a\u003eCleanup [▲](#toc)\n\nCleanup the VPC routes\n\n```sh\n$ aws ec2 delete-route --route-table-id ${RDS_ROUTE_TABLE_ID} --destination-cidr-block ${EKS_VPC_CIDR_BLOCK}\n$ aws ec2 delete-route --route-table-id ${EKS_ROUTE_TABLE_ID} --destination-cidr-block ${RDS_VPC_CIDR_BLOCK}\n$ aws ec2 delete-vpc-peering-connection --vpc-peering-connection-id ${VPC_PEERING_CONNECTION_ID}\n```\n\nDelete the cluster\n\n```sh\n$ eksctl delete cluster -n airflow-test\n```\n\nDelete the RDS and related VPC resources\n\n```sh\n$ aws rds delete-db-instance --db-instance-identifier airflow --skip-final-snapshot\n$ aws rds delete-db-subnet-group --db-subnet-group-name airflow\n$ aws ec2 delete-subnet --subnet-id subnet-0cb408bee95aa7fb3\n$ aws ec2 delete-subnet --subnet-id subnet-04871a023fdf047ec\n$ aws ec2 delete-security-group --group-id ${RDS_VPC_SECURITY_GROUP_ID}\n$ aws ec2 delete-vpc --vpc-id ${RDS_VPC_ID}\n```\n\nDelete images and repository\n\n```sh\n$ aws ecr batch-delete-image --repository-name airflow --image-ids $(aws ecr list-images --repository-name airflow | jq -r '.imageIds | reduce .[] as $img (\"\"; . + \"imageDigest=\\($img.imageDigest) \")')\n$ aws ecr delete-repository --repository-name airflow\n```\n\n### \u003ca name=\"aks\"\u003e\u003c/a\u003eAKS\n\nFirst create an `airflow` resource group\n\n```sh\n$ az group create --name airflow\n```\n\n#### \u003ca name=\"aks-acr\"\u003e\u003c/a\u003eACR Setup [▲](#toc)\n\nCreate a `Basic` registry and login\n\n```sh\n$ az acr create -g airflow -n airflow --sku Basic\n$ az acr login -n airflow\n```\n\nPush the image\n\n```sh\n$ docker tag my/airflow airflow.azurecr.io/airflow\n$ docker push airflow.azurecr.io/airflow\n```\n\nConfirm the image is successfully pushed\n\n```sh\n$ az acr repository show-tags -n airflow --repository airflow\n```\n\n#### \u003ca name=\"aks-cluster\"\u003e\u003c/a\u003eCluster Setup [▲](#toc)\n\nCreate a cluster with the ACR attached\n\n```sh\n$ az aks create -g airflow -n airflow \\\n  --kubernetes-version 1.15.5 \\\n  --node-count 1\n  --attach-acr airflow\n```\n\nConfigure local context\n\n```sh\n$ az aks get-credentials -g airflow --name airflow\n$ kubectl cluster-info\n```\n\nRun a simple pod\n\n```sh\n$ kubectl run echo -ti --rm --image=alpine --generator=run-pod/v1 --image-pull-policy=IfNotPresent --command -- echo hello\n```\n\n#### \u003ca name=\"aks-mysql\"\u003e\u003c/a\u003eMySQL Setup [▲](#toc)\n\nCreate the server and the database\n\n```sh\n$ az mysql server create -g airflow -n airflow \\\n  --admin-user airflowmaster \\\n  --admin-password airflowMa5ter \\\n  --sku-name GP_Gen5_2 \\\n  --version 5.7\n$ az mysql db create -g airflow -s airflow -n airflow\n```\n\nEnable a `Microsoft.SQL` service endpoint on the cluster subnet\n\n```sh\n$ export AKS_NODE_RESOURCE_GROUP=$(az aks show -n airflow -g airflow --query nodeResourceGroup -o tsv)\n$ export AKS_VNET_NAME=$(az resource list -g ${AKS_NODE_RESOURCE_GROUP} --query \"[?type=='Microsoft.Network/virtualNetworks'] | [0].name\" -o tsv)\n$ az network vnet subnet update --vnet-name ${AKS_VNET_NAME} -n aks-subnet -g ${AKS_NODE_RESOURCE_GROUP} --service-endpoints Microsoft.SQL\n```\n\nEnable MySQL access from the cluster\n\n```sh\n$ export AKS_SUBNET_ID=$(az network vnet show -n ${AKS_VNET_NAME} -g ${AKS_NODE_RESOURCE_GROUP} --query 'subnets[0].id' -o tsv)\n$ az mysql server vnet-rule create -n aks-mysql -g airflow -s airflow --subnet ${AKS_SUBNET_ID}\n```\n\nCheck connection and confirm `explicit_defaults_for_timestamp` is `ON`\n\n```sh\n$ kubectl run test-mysql -ti --rm --image=alpine --generator=run-pod/v1 --image-pull-policy=IfNotPresent --command -- \\\n    sh -c \"apk -U add mysql-client \u0026\u0026 mysql -h airflow.mysql.database.azure.com -u airflowmaster -pairflowMa5ter --ssl -e \\\"SHOW VARIABLES LIKE 'explicit_defaults_for_timestamp'\\\"\"\n+---------------------------------+-------+\n| Variable_name                   | Value |\n+---------------------------------+-------+\n| explicit_defaults_for_timestamp | ON    |\n+---------------------------------+-------+\n```\n\n#### \u003ca name=\"aks-vol\"\u003e\u003c/a\u003ePrepare Volume [▲](#toc)\n\nCreate a storage account and use a Secret to hold it\n\n```sh\n$ export AKS_STORAGEACCT=$(az storage account create -g airflow -n airflow$RANDOM --sku Standard_LRS --query \"name\" -o tsv)\n$ export AKS_STORAGEKEY=$(az storage account keys list -g airflow -n ${AKS_STORAGEACCT}  --query \"[0].value\" -o tsv)\n$ kubectl create secret generic azure-secret \\\n  --from-literal=azurestorageaccountname=${AKS_STORAGEACCT} \\\n  --from-literal=azurestorageaccountkey=${AKS_STORAGEKEY}\n```\n\nCreate a File Share and upload the dag file\n\n```sh\n$ az storage share create --account-name ${AKS_STORAGEACCT} --account-key ${AKS_STORAGEKEY} -n dags\n$ az storage file upload --account-name ${AKS_STORAGEACCT} --account-key ${AKS_STORAGEKEY} -s dags --source dags/foobar.py --path foobar.py\n$ az storage file list --account-name ${AKS_STORAGEACCT} --account-key ${AKS_STORAGEKEY} -s dags\n```\n\nCreate the StorageClass, PersistentVolume and PersistentVolumeClaim\n\n```sh\n$ kubectl apply -f - \u003c\u003cEOF\n---\nkind: StorageClass\napiVersion: storage.k8s.io/v1\nmetadata:\n  name: azurefile\nprovisioner: kubernetes.io/azure-file\nmountOptions:\n  - dir_mode=0755\n  - file_mode=0755\nparameters:\n  skuName: Standard_LRS\n  storageAccount: ${AKS_STORAGEACCT}\n  resourceGroup: airflow\n---\napiVersion: v1\nkind: PersistentVolume\nmetadata:\n  name: airflow-dags\n  labels:\n    usage: airflow-dags\nspec:\n  capacity:\n    storage: 1Gi\n  accessModes:\n    - ReadOnlyMany\n  storageClassName: azurefile\n  azureFile:\n    secretName: azure-secret\n    shareName: dags\n    readOnly: true\n---\napiVersion: v1\nkind: PersistentVolumeClaim\nmetadata:\n  name: airflow-dags\nspec:\n  accessModes:\n    - ReadOnlyMany\n  storageClassName: azurefile\n  resources:\n    requests:\n      storage: 1Gi\nEOF\n```\n\nMake sure the pvc is bound to the volume\n\n```sh\n$ kubectl describe pvc airflow-dags\n```\n\n#### \u003ca name=\"aks-test\"\u003e\u003c/a\u003eTesting Airflow [▲](#toc)\n\nCreate the service account\n\n```sh\n$ kubectl apply -f - \u003c\u003c'EOF'\n---\napiVersion: v1\nkind: ServiceAccount\nmetadata:\n  name: airflow\n---\napiVersion: rbac.authorization.k8s.io/v1\nkind: Role\nmetadata:\n  name: airflow\nrules:\n  - apiGroups: [\"\"]\n    resources: [\"configmaps\"]\n    verbs: [\"get\", \"watch\", \"list\"]\n  - apiGroups: [\"\"]\n    resources: [\"secrets\"]\n    verbs: [\"get\", \"watch\", \"list\"]\n  - apiGroups: [\"\"]\n    resources: [\"pods\"]\n    verbs: [\"get\", \"list\", \"watch\", \"create\", \"update\", \"patch\", \"delete\"]\n  - apiGroups: [\"\"]\n    resources: [\"pods/exec\"]\n    verbs: [\"get\", \"create\"]\n  - apiGroups: [\"\"]\n    resources: [\"pods/log\"]\n    verbs: [\"get\"]\n---\nkind: RoleBinding\napiVersion: rbac.authorization.k8s.io/v1\nmetadata:\n  name: airflow\nsubjects:\n  - kind: ServiceAccount\n    name: airflow\nroleRef:\n  kind: Role\n  name: airflow\n  apiGroup: rbac.authorization.k8s.io\nEOF\n```\n\nInitialize the database\n\n```sh\n$ kubectl run airflow-initdb \\\n    --restart=Never -ti --rm --image-pull-policy=IfNotPresent --generator=run-pod/v1 \\\n    --image=airflow.azurecr.io/airflow \\\n    --env AIRFLOW__CORE__LOAD_EXAMPLES=False \\\n    --env AIRFLOW__CORE__SQL_ALCHEMY_CONN=\"mysql://airflowmaster:airflowMa5ter@airflow.mysql.database.azure.com/airflow?ssl=true\" \\\n    --command -- airflow initdb\n```\n\nStart the scheduler\n\n```sh\n$ kubectl run airflow -ti --rm --restart=Never --image=airflow.azurecr.io/airflow --overrides='\n{\n  \"spec\": {\n    \"serviceAccountName\":\"airflow\",\n    \"containers\":[{\n      \"name\": \"scheduler\",\n      \"image\": \"airflow.azurecr.io/airflow\",\n      \"imagePullPolicy\":\"IfNotPresent\",\n      \"command\": [\"airflow\",\"scheduler\"],\n      \"stdin\": true,\n      \"tty\": true,\n      \"env\": [\n        {\"name\":\"AIRFLOW__CORE__LOAD_EXAMPLES\",\"value\":\"False\"},\n        {\"name\":\"AIRFLOW__CORE__SQL_ALCHEMY_CONN\",\"value\":\"mysql://airflowmaster:airflowMa5ter@airflow.mysql.database.azure.com/airflow?ssl=true\"}, \n        {\"name\":\"AIRFLOW__CORE__EXECUTOR\",\"value\":\"KubernetesExecutor\"},\n        {\"name\":\"AIRFLOW__KUBERNETES__WORKER_CONTAINER_REPOSITORY\",\"value\":\"airflow.azurecr.io/airflow\"},\n        {\"name\":\"AIRFLOW__KUBERNETES__WORKER_CONTAINER_TAG\",\"value\":\"latest\"},\n        {\"name\":\"AIRFLOW__KUBERNETES__WORKER_SERVICE_ACCOUNT_NAME\",\"value\":\"airflow\"},\n        {\"name\":\"AIRFLOW__KUBERNETES__KUBE_CLIENT_REQUEST_ARGS\",\"value\":\"\"},\n        {\"name\":\"AIRFLOW__KUBERNETES__DELETE_WORKER_PODS\",\"value\":\"True\"},\n        {\"name\":\"AIRFLOW__KUBERNETES__DAGS_VOLUME_CLAIM\",\"value\":\"airflow-dags\"}\n      ],\n      \"volumeMounts\": [{\"mountPath\": \"/var/lib/airflow/dags\",\"name\": \"dags\"}]\n    }],\n    \"volumes\": [{\"name\":\"dags\",\"persistentVolumeClaim\":{\"claimName\":\"airflow-dags\"}}]\n  }\n}'\n```\n\nWatch the pods\n\n```sh\n$ kubectl get po -w\n```\n\nRun the dag\n\n```sh\n$ kubectl exec -ti airflow airflow list_dags\n$ kubectl exec -ti airflow airflow unpause foobar\n$ kubectl exec -ti airflow airflow trigger_dag foobar\n$ kubectl exec -ti airflow airflow list_dag_runs foobar\n```\n\nHere is the pods logs\n\n```\nNAME                                         READY   STATUS    RESTARTS   AGE\nairflow                                      1/1     Running   0          48s\nfoobarbar-00d0e6c0d92c4848a45a29941035f076   1/1     Running   0          8s\nfoobarfoo-fb28dd8a155541379fd1a6261f41ae0b   1/1     Running   0          14s\nfoobarfoo-fb28dd8a155541379fd1a6261f41ae0b   0/1     Completed   0          15s\nfoobarfoo-fb28dd8a155541379fd1a6261f41ae0b   0/1     Terminating   0          16s\nfoobarfoo-fb28dd8a155541379fd1a6261f41ae0b   0/1     Terminating   0          16s\nfoobarbar-00d0e6c0d92c4848a45a29941035f076   0/1     Completed     0          14s\nfoobarbar-00d0e6c0d92c4848a45a29941035f076   0/1     Terminating   0          14s\nfoobarbar-00d0e6c0d92c4848a45a29941035f076   0/1     Terminating   0          14s\n```\n\n#### \u003ca name=\"aks-cleanup\"\u003e\u003c/a\u003eCleanup [▲](#toc)\n\nDelete the resource group and remove the context\n\n```sh\n$ az group delete -g airflow\n$ kubectl config delete-context airflow\n```\n\n## References\n\n* [Bitnami Apache Airflow Multi-Tier now available in Azure Marketplace | Blog | Microsoft Azure](https://azure.microsoft.com/en-us/blog/bitnami-apache-airflow-multi-tier-now-available-in-azure-marketplace/)\n* [tekn0ir/airflow-chart: Helm chart for deploying Apache Airflow in kubernetes](https://github.com/tekn0ir/airflow-chart)\n* [puckel/docker-airflow: Docker Apache Airflow](https://github.com/puckel/docker-airflow)\n* [GoogleCloudPlatform/airflow-operator: Kubernetes custom controller and CRDs to managing Airflow](https://github.com/GoogleCloudPlatform/airflow-operator)\n* [villasv/aws-airflow-stack: Turbine: the bare metals behind a complete Airflow setup](https://github.com/villasv/aws-airflow-stack)\n* [mumoshu/kube-airflow: A docker image and kubernetes config files to run Airflow on Kubernetes](https://github.com/mumoshu/kube-airflow)\n* [rolanddb/airflow-on-kubernetes: A guide to running Airflow on Kubernetes](https://github.com/rolanddb/airflow-on-kubernetes)\n* [EamonKeane/airflow-GKE-k8sExecutor-helm: Quickly get a kubernetes executor airflow environment provisioned on GKE. Azure Kubernetes Service instructions included also as are instructions for docker-for-mac.](https://github.com/EamonKeane/airflow-GKE-k8sExecutor-helm)\n* [Accessing Amazon RDS From AWS EKS - DEV Community 👩‍💻👨‍💻](https://dev.to/bensooraj/accessing-amazon-rds-from-aws-eks-2pc3)\n* [airflow-rbac.yml](https://gist.github.com/noqcks/04d4f4a2846ec1e0ed2fbda58907ca6d)\n* [pahud/amazon-eks-workshop: Amazon EKS workshop](https://github.com/pahud/amazon-eks-workshop)\n* [The Ultimate Kubernetes Cost Guide: Adding Persistent Volumes to the Mix](https://www.replex.io/blog/the-ultimate-kubernetes-cost-guide-adding-persistent-storage-to-the-mix)\n* [Deploying Apache Airflow on Azure Kubernetes Service](https://blog.godatadriven.com/airflow-on-aks)\n* [examples/staging/volumes/azure_file at master · kubernetes/examples](https://github.com/kubernetes/examples/tree/master/staging/volumes/azure_file)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstwind%2Fairflow-on-kubernetes","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstwind%2Fairflow-on-kubernetes","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstwind%2Fairflow-on-kubernetes/lists"}