{"id":16652468,"url":"https://github.com/nicor88/aws-ecs-airflow","last_synced_at":"2025-04-13T00:47:32.568Z","repository":{"id":51323729,"uuid":"115554893","full_name":"nicor88/aws-ecs-airflow","owner":"nicor88","description":"Run Airflow in AWS ECS(Elastic Container Service) using Fargate tasks","archived":false,"fork":false,"pushed_at":"2024-10-24T22:43:55.000Z","size":97,"stargazers_count":160,"open_issues_count":10,"forks_count":77,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-04-13T00:47:10.362Z","etag":null,"topics":["airflow","airflow-ecs","aws","dag","docker","ecr","ecs","etl","fargate","terraform"],"latest_commit_sha":null,"homepage":"","language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nicor88.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"custom":"paypal.com/paypalme/NicolaCorda"}},"created_at":"2017-12-27T20:24:45.000Z","updated_at":"2025-01-31T09:01:55.000Z","dependencies_parsed_at":"2024-10-27T11:46:27.870Z","dependency_job_id":"f7b5c6f2-ffc6-4a99-a714-c735bf22f312","html_url":"https://github.com/nicor88/aws-ecs-airflow","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nicor88%2Faws-ecs-airflow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nicor88%2Faws-ecs-airflow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nicor88%2Faws-ecs-airflow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nicor88%2Faws-ecs-airflow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nicor88","download_url":"https://codeload.github.com/nicor88/aws-ecs-airflow/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248650434,"owners_count":21139672,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow","airflow-ecs","aws","dag","docker","ecr","ecs","etl","fargate","terraform"],"created_at":"2024-10-12T09:28:49.809Z","updated_at":"2025-04-13T00:47:32.541Z","avatar_url":"https://github.com/nicor88.png","language":"HCL","funding_links":["paypal.com/paypalme/NicolaCorda"],"categories":[],"sub_categories":[],"readme":"# airflow-ecs\nSetup to run Airflow in AWS ECS containers\n\n## Requirements\n\n### Local\n* Docker\n\n### AWS\n* AWS IAM User for the infrastructure deployment, with admin permissions\n* [awscli](https://aws.amazon.com/cli/), intall running `pip install awscli`\n* [terraform \u003e= 0.13](https://www.terraform.io/downloads.html)\n* setup your IAM User credentials inside `~/.aws/credentials`\n* setup these env variables in your .zshrc or .bashrc, or in your the terminal session that you are going to use\n  \u003cpre\u003e\n  export AWS_ACCOUNT=your_account_id\n  export AWS_DEFAULT_REGION=us-east-1 # it's the default region that needs to be setup also in infrastructure/config.tf\n  \u003c/pre\u003e\n\n\n## Local Development\n* Generate a Fernet Key:\n  \u003cpre\u003e\n  pip install cryptography\n  export AIRFLOW_FERNET_KEY=$(python -c \"from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())\")\n  \u003c/pre\u003e\n  More about that [here](https://cryptography.io/en/latest/fernet/)\n\n* Start Airflow locally simply running:\n  \u003cpre\u003e\n  docker-compose up --build\n  \u003c/pre\n\nIf everything runs correctly you can reach Airflow navigating to [localhost:8080](http://localhost:8080).\nThe current setup is based on [Celery Workers](https://airflow.apache.org/howto/executor/use-celery.html). You can monitor how many workers are currently active using Flower, visiting [localhost:5555](http://localhost:5555)\n\n## Deploy Airflow on AWS ECS\nTo run Airflow in AWS we will use ECS (Elastic Container Service).\n\n### Deploy Infrastructure using Terraform\nRun the following commands:\n\u003cpre\u003e\nmake infra-init\nmake infra-plan\nmake infra-apply\n\u003c/pre\u003e\n\nor alternatively\n\u003cpre\u003e\ncd infrastructure\nterraform get\nterraform init -upgrade;\nterraform plan\nterraform apply\n\u003c/pre\u003e\n\nBy default the infrastructure is deployed in `us-east-1`.\n\nWhen the infrastructure is provisioned (the RDS metadata DB will take a while) check the if the ECR repository is created then run:\n\u003cpre\u003e\nbash scripts/push_to_ecr.sh airflow-dev\n\u003c/pre\u003e\nBy default the repo name created with terraform is `airflow-dev`\nWithout this command the ECS services will fail to fetch the `latest` image from ECR\n\n### Deploy new Airflow application\nTo deploy an update version of Airflow you need to push a new container image to ECR.\nYou can simply doing that running:\n\u003cpre\u003e\n./scripts/deploy.sh airflow-dev\n\u003c/pre\u003e\n\nThe deployment script will take care of:\n* push a new ECR image to your repository\n* re-deploy the new ECS services with the updated image\n\n## TODO\n* Create Private Subnets\n* Move ECS containers to Private Subnets\n* Use ECS private Links for Private Subnets\n* Improve ECS Task and Service Role\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnicor88%2Faws-ecs-airflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnicor88%2Faws-ecs-airflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnicor88%2Faws-ecs-airflow/lists"}