{"id":13626191,"url":"https://github.com/aws/aws-mwaa-local-runner","last_synced_at":"2025-05-14T14:10:24.973Z","repository":{"id":38184761,"uuid":"361984915","full_name":"aws/aws-mwaa-local-runner","owner":"aws","description":"This repository provides a command line interface (CLI) utility that replicates an Amazon Managed Workflows for Apache Airflow (MWAA) environment locally.","archived":false,"fork":false,"pushed_at":"2025-01-30T22:29:46.000Z","size":180,"stargazers_count":755,"open_issues_count":128,"forks_count":745,"subscribers_count":26,"default_branch":"v2.10.3","last_synced_at":"2025-05-08T00:07:58.873Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit-0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aws.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-04-27T04:55:27.000Z","updated_at":"2025-05-07T14:35:37.000Z","dependencies_parsed_at":"2023-11-09T00:25:42.645Z","dependency_job_id":"c41dbc73-2450-466b-971f-d9e0a30a0a6c","html_url":"https://github.com/aws/aws-mwaa-local-runner","commit_stats":{"total_commits":52,"total_committers":17,"mean_commits":"3.0588235294117645","dds":0.5769230769230769,"last_synced_commit":"4852eda2e4d98374e344ad874ac71cea4d739e0d"},"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aws%2Faws-mwaa-local-runner","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aws%2Faws-mwaa-local-runner/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aws%2Faws-mwaa-local-runner/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aws%2Faws-mwaa-local-runner/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aws","download_url":"https://codeload.github.com/aws/aws-mwaa-local-runner/tar.gz/refs/heads/v2.10.3","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254160558,"owners_count":22024571,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T21:02:12.551Z","updated_at":"2025-05-14T14:10:19.964Z","avatar_url":"https://github.com/aws.png","language":"Shell","funding_links":[],"categories":["Shell","Building"],"sub_categories":["Workflows"],"readme":"## Note\nStarting from Airflow version 2.9, MWAA has open-sourced the original Docker image used in our production deployments. You can refer to our open-source image repository at https://github.com/aws/amazon-mwaa-docker-images to create a local environment identical to that of MWAA.\nYou can also continue to use the MWAA Local Runner for testing and packaging requirements for all Airflow versions supported on MWAA.\n\n# About aws-mwaa-local-runner\n\nThis repository provides a command line interface (CLI) utility that replicates an Amazon Managed Workflows for Apache Airflow (MWAA) environment locally.\n\n_Please note: MWAA/AWS/DAG/Plugin issues should be raised through AWS Support or the Airflow Slack #airflow-aws channel. Issues here should be focused on this local-runner repository._\n\n_Please note: The dynamic configurations which are dependent on the class of an environment are\naligned with the Large environment class in this repository._\n\n## About the CLI\n\nThe CLI builds a Docker container image locally that’s similar to a MWAA production image. This allows you to run a local Apache Airflow environment to develop and test DAGs, custom plugins, and dependencies before deploying to MWAA.\n\n## What this repo contains\n\n```text\ndags/\n  example_lambda.py\n  example_dag_with_taskflow_api.py\n  example_redshift_data_execute_sql.py\ndocker/\n  config/\n    airflow.cfg\n    constraints.txt\n    mwaa-base-providers-requirements.txt\n    webserver_config.py\n    .env.localrunner\n  script/\n    bootstrap.sh\n    entrypoint.sh\n    systemlibs.sh\n    generate_key.sh\n  docker-compose-local.yml\n  docker-compose-resetdb.yml\n  docker-compose-sequential.yml\n  Dockerfile\nplugins/\n  README.md\nrequirements/\n  requirements.txt\n.gitignore\nCODE_OF_CONDUCT.md\nCONTRIBUTING.md\nLICENSE\nmwaa-local-env\nREADME.md\nVERSION\n```\n\n## Prerequisites\n\n- **macOS**: [Install Docker Desktop](https://docs.docker.com/desktop/).\n- **Linux/Ubuntu**: [Install Docker Compose](https://docs.docker.com/compose/install/) and [Install Docker Engine](https://docs.docker.com/engine/install/).\n- **Windows**: Windows Subsystem for Linux (WSL) to run the bash based command `mwaa-local-env`. Please follow [Windows Subsystem for Linux Installation (WSL)](https://docs.docker.com/docker-for-windows/wsl/) and [Using Docker in WSL 2](https://code.visualstudio.com/blogs/2020/03/02/docker-in-wsl2), to get started.\n\n## Get started\n\n```bash\ngit clone https://github.com/aws/aws-mwaa-local-runner.git\ncd aws-mwaa-local-runner\n```\n\n### Step one: Building the Docker image\n\nBuild the Docker container image using the following command:\n\n```bash\n./mwaa-local-env build-image\n```\n\n**Note**: it takes several minutes to build the Docker image locally.\n\n### Step two: Running Apache Airflow\n\n#### Local runner\n\nRuns a local Apache Airflow environment that is a close representation of MWAA by configuration.\n\n```bash\n./mwaa-local-env start\n```\n\nTo stop the local environment, Ctrl+C on the terminal and wait till the local runner and the postgres containers are stopped.\n\n### Step three: Accessing the Airflow UI\n\nBy default, the `bootstrap.sh` script creates a username and password for your local Airflow environment.\n\n- Username: `admin`\n- Password: `test`\n\n#### Airflow UI\n\n- Open the Apache Airlfow UI: \u003chttp://localhost:8080/\u003e.\n\n### Step four: Add DAGs and supporting files\n\nThe following section describes where to add your DAG code and supporting files. We recommend creating a directory structure similar to your MWAA environment.\n\n#### DAGs\n\n1. Add DAG code to the `dags/` folder.\n2. To run the sample code in this repository, see the `example_dag_with_taskflow_api.py` file.\n\n#### Requirements.txt\n\n1. Add Python dependencies to `requirements/requirements.txt`.\n2. To test a requirements.txt without running Apache Airflow, use the following script:\n\n```bash\n./mwaa-local-env test-requirements\n```\n\nLet's say you add `aws-batch==0.6` to your `requirements/requirements.txt` file. You should see an output similar to:\n\n```bash\nInstalling requirements.txt\nCollecting aws-batch (from -r /usr/local/airflow/dags/requirements.txt (line 1))\n  Downloading https://files.pythonhosted.org/packages/5d/11/3aedc6e150d2df6f3d422d7107ac9eba5b50261cf57ab813bb00d8299a34/aws_batch-0.6.tar.gz\nCollecting awscli (from aws-batch-\u003e-r /usr/local/airflow/dags/requirements.txt (line 1))\n  Downloading https://files.pythonhosted.org/packages/07/4a/d054884c2ef4eb3c237e1f4007d3ece5c46e286e4258288f0116724af009/awscli-1.19.21-py2.py3-none-any.whl (3.6MB)\n    100% |████████████████████████████████| 3.6MB 365kB/s\n...\n...\n...\nInstalling collected packages: botocore, docutils, pyasn1, rsa, awscli, aws-batch\n  Running setup.py install for aws-batch ... done\nSuccessfully installed aws-batch-0.6 awscli-1.19.21 botocore-1.20.21 docutils-0.15.2 pyasn1-0.4.8 rsa-4.7.2\n```\n\n3. To package the necessary WHL files for your requirements.txt without running Apache Airflow, use the following script:\n\n```bash\n./mwaa-local-env package-requirements\n```\n\nFor example usage see [Installing Python dependencies using PyPi.org Requirements File Format Option two: Python wheels (.whl)](https://docs.aws.amazon.com/mwaa/latest/userguide/best-practices-dependencies.html#best-practices-dependencies-python-wheels).\n\n#### Custom plugins\n\n- There is a directory at the root of this repository called plugins.\n- In this directory, create a file for your new custom plugin.\n- Add any Python dependencies to `requirements/requirements.txt`.\n\n**Note**: this step assumes you have a DAG that corresponds to the custom plugin. For example usage [MWAA Code Examples](https://docs.aws.amazon.com/mwaa/latest/userguide/sample-code.html).\n\n#### Startup script\n\n- There is a sample shell script `startup.sh` located in a directory at the root of this repository called `startup_script`.\n- If there is a need to run additional setup (e.g. install system libraries, setting up environment variables), please modify the `startup.sh` script.\n- To test a `startup.sh` without running Apache Airflow, use the following script:\n\n```bash\n./mwaa-local-env test-startup-script\n```\n\n## What's next?\n\n- Learn how to upload the requirements.txt file to your Amazon S3 bucket in [Installing Python dependencies](https://docs.aws.amazon.com/mwaa/latest/userguide/working-dags-dependencies.html).\n- Learn how to upload the DAG code to the dags folder in your Amazon S3 bucket in [Adding or updating DAGs](https://docs.aws.amazon.com/mwaa/latest/userguide/configuring-dag-folder.html).\n- Learn more about how to upload the plugins.zip file to your Amazon S3 bucket in [Installing custom plugins](https://docs.aws.amazon.com/mwaa/latest/userguide/configuring-dag-import-plugins.html).\n\n## FAQs\n\nThe following section contains common questions and answers you may encounter when using your Docker container image.\n\n### Can I test execution role permissions using this repository?\n\n- You can setup the local Airflow's boto with the intended execution role to test your DAGs with AWS operators before uploading to your Amazon S3 bucket. To setup aws connection for Airflow locally see [Airflow | AWS Connection](https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/connections/aws.html)\n  To learn more, see [Amazon MWAA Execution Role](https://docs.aws.amazon.com/mwaa/latest/userguide/mwaa-create-role.html).\n- You can set AWS credentials via environment variables set in the `docker/config/.env.localrunner` env file. To learn more about AWS environment variables, see [Environment variables to configure the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html) and [Using temporary security credentials with the AWS CLI](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_use-resources.html#using-temp-creds-sdk-cli). Simply set the relevant environment variables in `.env.localrunner` and `./mwaa-local-env start`.\n\n### How do I add libraries to requirements.txt and test install?\n\n- A `requirements.txt` file is included in the `/requirements` folder of your local Docker container image. We recommend adding libraries to this file, and running locally.\n\n### What if a library is not available on PyPi.org?\n\n- If a library is not available in the Python Package Index (PyPi.org), add the `--index-url` flag to the package in your `requirements/requirements.txt` file. To learn more, see [Managing Python dependencies in requirements.txt](https://docs.aws.amazon.com/mwaa/latest/userguide/best-practices-dependencies.html).\n\n## Troubleshooting\n\nThe following section contains errors you may encounter when using the Docker container image in this repository.\n\n### My environment is not starting\n\n- If you encountered [the following error](https://issues.apache.org/jira/browse/AIRFLOW-3678): `process fails with \"dag_stats_table already exists\"`, you'll need to reset your database using the following command:\n\n```bash\n./mwaa-local-env reset-db\n```\n\n- If you are moving from an older version of local-runner you may need to run the above reset-db command, or delete your `./db-data` folder. Note, too, that newer Airflow versions have newer provider packages, which may require updating your DAG code.\n\n### Fernet Key InvalidToken\n\nA Fernet Key is generated during image build (`./mwaa-local-env build-image`) and is durable throughout all\ncontainers started from that image. This key is used to [encrypt connection passwords in the Airflow DB](https://airflow.apache.org/docs/apache-airflow/stable/security/secrets/fernet.html).\nIf changes are made to the image and it is rebuilt, you may get a new key that will not match the key used when\nthe Airflow DB was initialized, in this case you will need to reset the DB (`./mwaa-local-env reset-db`).\n\n## Security\n\nSee [CONTRIBUTING](CONTRIBUTING.md#security-issue-notifications) for more information.\n\n## License\n\nThis library is licensed under the MIT-0 License. See the LICENSE file.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faws%2Faws-mwaa-local-runner","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faws%2Faws-mwaa-local-runner","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faws%2Faws-mwaa-local-runner/lists"}