https://github.com/nicor88/aws-ecs-airflow

Run Airflow in AWS ECS(Elastic Container Service) using Fargate tasks
https://github.com/nicor88/aws-ecs-airflow

airflow airflow-ecs aws dag docker ecr ecs etl fargate terraform

Last synced: 3 days ago
JSON representation

Run Airflow in AWS ECS(Elastic Container Service) using Fargate tasks

• Host: GitHub
• URL: https://github.com/nicor88/aws-ecs-airflow
• Owner: nicor88
• License: mit
• Created: 2017-12-27T20:24:45.000Z (about 7 years ago)
• Default Branch: master
• Last Pushed: 2024-10-24T22:43:55.000Z (3 months ago)
• Last Synced: 2025-01-11T19:07:30.618Z (10 days ago)
• Topics: airflow, airflow-ecs, aws, dag, docker, ecr, ecs, etl, fargate, terraform
• Language: HCL
• Homepage:
• Size: 94.7 KB
• Stars: 159
• Watchers: 5
• Forks: 76
• Open Issues: 10
• Metadata Files:
  • Readme: README.md
  • Funding: .github/FUNDING.yml
  • License: LICENSE

Awesome Lists containing this project

README

        
# airflow-ecs

Setup to run Airflow in AWS ECS containers

## Requirements

### Local

* Docker

### AWS

* AWS IAM User for the infrastructure deployment, with admin permissions

* [awscli](https://aws.amazon.com/cli/), intall running `pip install awscli`

* [terraform >= 0.13](https://www.terraform.io/downloads.html)

* setup your IAM User credentials inside `~/.aws/credentials`

* setup these env variables in your .zshrc or .bashrc, or in your the terminal session that you are going to use

  

  export AWS_ACCOUNT=your_account_id

  export AWS_DEFAULT_REGION=us-east-1 # it's the default region that needs to be setup also in infrastructure/config.tf

  


## Local Development

* Generate a Fernet Key:

  

  pip install cryptography

  export AIRFLOW_FERNET_KEY=$(python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())")

  

  More about that [here](https://cryptography.io/en/latest/fernet/)


* Start Airflow locally simply running:

  

  docker-compose up --build

  

make infra-init

make infra-plan

make infra-apply


or alternatively


cd infrastructure

terraform get

terraform init -upgrade;

terraform plan

terraform apply



By default the infrastructure is deployed in `us-east-1`.

When the infrastructure is provisioned (the RDS metadata DB will take a while) check the if the ECR repository is created then run:


bash scripts/push_to_ecr.sh airflow-dev



By default the repo name created with terraform is `airflow-dev`

Without this command the ECS services will fail to fetch the `latest` image from ECR


### Deploy new Airflow application

To deploy an update version of Airflow you need to push a new container image to ECR.

You can simply doing that running:


./scripts/deploy.sh airflow-dev



The deployment script will take care of:

* push a new ECR image to your repository

* re-deploy the new ECS services with the updated image

## TODO

* Create Private Subnets

* Move ECS containers to Private Subnets

* Use ECS private Links for Private Subnets

* Improve ECS Task and Service Role