{"id":43933075,"url":"https://github.com/scottypate/airflow-aws","last_synced_at":"2026-02-07T00:19:39.908Z","repository":{"id":189363333,"uuid":"669445269","full_name":"scottypate/airflow-aws","owner":"scottypate","description":"Run Airflow locally or on AWS in Kubernetes","archived":false,"fork":false,"pushed_at":"2024-01-04T19:44:48.000Z","size":35,"stargazers_count":5,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-01-04T20:43:42.524Z","etag":null,"topics":["airflow","airflow-kubernetes","aws","k8s","kube"],"latest_commit_sha":null,"homepage":"","language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/scottypate.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2023-07-22T09:53:54.000Z","updated_at":"2024-04-15T11:04:26.377Z","dependencies_parsed_at":"2023-08-19T16:56:54.547Z","dependency_job_id":"41e45fa3-9fcc-47e9-9c98-1e115565d7e5","html_url":"https://github.com/scottypate/airflow-aws","commit_stats":null,"previous_names":["scottypate/airflow-aws"],"tags_count":0,"template":null,"template_full_name":null,"purl":"pkg:github/scottypate/airflow-aws","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scottypate%2Fairflow-aws","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scottypate%2Fairflow-aws/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scottypate%2Fairflow-aws/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scottypate%2Fairflow-aws/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/scottypate","download_url":"https://codeload.github.com/scottypate/airflow-aws/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scottypate%2Fairflow-aws/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29181331,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-06T23:15:33.022Z","status":"ssl_error","status_checked_at":"2026-02-06T23:15:09.128Z","response_time":59,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow","airflow-kubernetes","aws","k8s","kube"],"created_at":"2026-02-07T00:19:39.303Z","updated_at":"2026-02-07T00:19:39.892Z","avatar_url":"https://github.com/scottypate.png","language":"HCL","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Airflow on Kubernetes\n\nRun Airflow on K8s in AWS. We are all running the same stuff. This won't be exactly what you need. This is meant to be a starting point to waste less engineering time.\n\n## EKS Deployment Networking\n\nThe example EKS cluster in this repo is configured to allow both private and public access. Private access is done through private subnetting via a NAT gateway. Public access is controlled via allowlisting in the [EKS Security Group](./terraform/modules/airflow/variables.tf#L23). If you have a VPN or want to use a jumpbox, you can restrict EKS to only private access and configure access from your private network.\n\n## Database\n\nThe main production database for this example is RDS - Postgres. There is a local version of a postgres database which runs inside of k8s. This is only for local development.\n\n## Getting Started\n\n### Prerequisites\n\n1. [AWS account CLI configured locally](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html).\n2. [Install Terraform](https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli).\n3. [Install Make](https://formulae.brew.sh/formula/make)\n4. Install Local Dependencies (Optional) - `make install-requirements`\n\n### Run Airflow locally\n\n1. Create and Export the .env file variables - `export $(cat .env)`. __Hint: the [.env template](.env.template) file shows which variables you will need to define in your .env file__. \n2. Build the Airflow image locally - `make build-minikube-airflow-image`\n3. Apply the dev k8s config - `make set-kube-secrets-local`\n4. Apply the dev k8s config - `make apply-kube-manifests-local`\n5. Port forward to the local webserver et voilà - `kubectl port-forward svc/airflow-webserver 8080:8080 --namespace airflow`\n\n### Run Airflow on AWS\n\n1. Create and Export the .env file variables - `export $(cat .env)`. __Hint: the [.env template](.env.template) file shows which variables you will need to define in your .env file__.\n2. Build the Terraform plans, `cd terraform/us-west-2 \u0026\u0026 terraform apply`.  __Hint: don't forget to [allowlist](./terraform/modules/k8s/variables.tf#L23) your IP or connect via a private network__.\n3. Update local kubeconfig to point to the AWS EKS cluster, `make update-kubeconfig`.\n4. Build and push the Airflow image to ECR, `make push-ecr-airflow-image`.\n5. Set secrets in EKS, `make set-kube-secrets-prod`.\n6. Apply kustomize manifests, `make apply-kube-manifests-prod`\n7. Port forward to the local webserver et voilà - `kubectl port-forward svc/airflow-webserver 8080:8080 --namespace airflow`\n\n## Common Commands\n\n```bash\n# Get all pods in the airflow namespace\nkubectl get pods -n airflow\n\n# Port forward the webserver to localhost:8080\nkubectl port-forward svc/airflow-webserver 8080:8080 --namespace airflow\n\n# Port forward the PG database to localhost:5432\nkubectl port-forward svc/postgres 5432:5432 --namespace airflow\n\n# Get the los for a pod\nkubectl logs ${POD_NAME} -n airflow --all-containers --follow\n\n# Open an interactive shell in a running container\nkubectl exec --stdin --tty -n airflow -c scheduler ${POD_NAME} -- /bin/sh\n\n# Print the plaintext of a kube secret\nkubectl get secret postgres -n airflow -o json | jq -r '.data.connection' | base64 --decode\n\n# Restart a pod\nkubectl rollout restart deployment airflow-webserver -n airflow\n\n# Delete all pods in a namespace\nkubectl delete pods --all -n airflow\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscottypate%2Fairflow-aws","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fscottypate%2Fairflow-aws","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscottypate%2Fairflow-aws/lists"}