{"id":21050160,"url":"https://github.com/astronomer/ap-airflow","last_synced_at":"2025-05-15T20:32:17.213Z","repository":{"id":37548338,"uuid":"221298116","full_name":"astronomer/ap-airflow","owner":"astronomer","description":"Astronomer Core Docker Images","archived":false,"fork":false,"pushed_at":"2024-05-22T20:09:10.000Z","size":34564,"stargazers_count":106,"open_issues_count":10,"forks_count":30,"subscribers_count":38,"default_branch":"master","last_synced_at":"2025-04-03T14:43:36.971Z","etag":null,"topics":["airflow","apache","docker","kubernetes"],"latest_commit_sha":null,"homepage":"https://astronomer.io","language":"Jinja","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/astronomer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS","security":"SECURITY.md","support":null}},"created_at":"2019-11-12T19:43:16.000Z","updated_at":"2024-09-16T16:10:33.000Z","dependencies_parsed_at":"2023-02-15T12:01:04.757Z","dependency_job_id":null,"html_url":"https://github.com/astronomer/ap-airflow","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/astronomer%2Fap-airflow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/astronomer%2Fap-airflow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/astronomer%2Fap-airflow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/astronomer%2Fap-airflow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/astronomer","download_url":"https://codeload.github.com/astronomer/ap-airflow/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254418590,"owners_count":22068103,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow","apache","docker","kubernetes"],"created_at":"2024-11-19T15:22:35.288Z","updated_at":"2025-05-15T20:32:16.424Z","avatar_url":"https://github.com/astronomer.png","language":"Jinja","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Astronomer Core Docker Images\n\n[![docker-pulls](https://img.shields.io/docker/pulls/astronomerinc/ap-airflow.svg)](https://hub.docker.com/r/astronomerinc/ap-airflow)\n\nAstronomer makes it easy to run, monitor, and scale [Apache Airflow](https://github.com/apache/airflow) deployments in our cloud or yours. Source code is made available for the benefit of customers.\n\n#### Terminology\n\n| Terms         | Example                  | Description                                                                                         |\n| :------------ | :----------------------- | :-------------------------------------------------------------------------------------------------- |\n| `edge` build  | `main-dev`               | Built from the current `main` branch of [astronomer/airflow](https://github.com/astronomer/airflow) |\n| dev build     | `2.2.4-4-dev`            | Development build, released during ap-airflow changes, including pre-releases and version releases  |\n| nightly build | `2.2.4-nightly-20220314` | Nightly builds, regularly triggered by a CircleCI pipeline sometime during the midnight hour UTC    |\n| release build | `2.2.4-4`                | Release builds, triggered by a release PR                                                           |\n\nNote: Edge builds are always development builds\n\n#### Build matrix\n\n| Build         | Nightly                   | Pre-release PR     | Release PR         |\n| :------------ | :------------------------ | :----------------- | :----------------- |\n| `edge` build  | :white_check_mark:        | :white_check_mark: | :white_check_mark: |\n| nightly build | :white_check_mark:        | :white_check_mark: |                    |\n| dev build     | (only during pre-release) | :white_check_mark: | :white_check_mark: |\n| release build |                           |                    | :white_check_mark: |\n\n## Docker images\n\nDocker images for deploying and running Astronomer Core are currently available on\n[Quay](https://quay.io/repository/astronomer/ap-airflow?tab=tags).\n\nWe publish 2 variants for each AC Version (example: `2.3.4-7`):\n\n1. `quay.io/astronomer/ap-airflow:2.3.4-7`\n2. `quay.io/astronomer/ap-airflow:2.3.4-7-onbuild`\n\nThe only difference between them is that the `-onbuild` images uses Docker `ONBUILD` commands to\ncopy `packages.txt`, `requirements.txt` and the entire project directory (including `dags`,\n`plugins` folders etc) in the docker file.\n\nWe also publish a \"floating\" or movable tag that points at the latest release of the Airflow version:\n\n1. `quay.io/astronomer/ap-airflow:2.3.4`\n2. `quay.io/astronomer/ap-airflow:2.3.4-onbuild`\n\n## Version Life Cycle \u0026 Maintenance policy\n\nThe support and maintenance of the  Docker images are described in\nthe [Version Life Cycle](https://docs.astronomer.io/software/ac-support-policy/).\n\n## Contents of this repo\n\n* The official Dockerfiles that build Astronomer Core Images\n* Example docker-compose files for running various pieces and configurations of\n  the platform.\n\n## Contribute\n\n* Source Code: \u003chttps://github.com/astronomer/ap-airflow\u003e\n* Issue Tracker: \u003chttps://github.com/astronomer/ap-airflow/issues\u003e\n\n## Step-by-step instructions for common activities\n\n### Release a new Astronomer Certified major, minor, or bugfix version (eg: X.Y.Z)\n\n\u003cdetails\u003e\n\u003csummary\u003eClick to expand Step-By-Step instructions\u003c/summary\u003e\n\n1. Remove the `-dev` part of the relevant version in `IMAGE_MAP` in `.circleci/common.py`.\n\n   Example:\n   The latest dev version is `2.2.1-1-dev`, and we want to release `2.2.1-1`.\n   ```diff\n   diff --git a/.circleci/common.py b/.circleci/common.py\n   index xxxxxxx..yyyyyyy 100644\n   --- a/.circleci/common.py\n   +++ b/.circleci/common.py\n   @@ -35,7 +35,7 @@ IMAGE_MAP = collections.OrderedDict([\n        (\"2.1.3-2\", [\"buster\"]),\n        (\"2.1.4-2\", [\"buster\"]),\n        (\"2.2.0-3-dev\", [\"bullseye\", \"buster\"]),\n   -    (\"2.2.1-1-dev\", [\"bullseye\", \"buster\"]),\n   +    (\"2.2.1-1\", [\"bullseye\", \"buster\"]),\n   ])\n\n   # Airflow Versions for which we don't publish Python Wheels\n   ```\n2. Run the `update-dockerfiles` pre-commit hook (this should fail but it should change the\n   relevant Dockerfile).\n\n   Example:\n   ```bash\n   pre-commit run update-dockerfiles\n   ```\n3. Add the changed Dockerfile and commit (this should succeed).\n\n   Example: The `update-dockerfiles` hook updated `2.2.1/bullseye/Dockerfile`:\n   ```bash\n   git add 2.2.1/bullseye/Dockerfile; git commit\n   ```\n\n\u003c/details\u003e\n\n### Release an existing Astronomer Certified version with an updated version of Airflow\n\n\u003cdetails\u003e\n\u003csummary\u003eClick to expand Step-By-Step instructions\u003c/summary\u003e\n\n1. Update the postfix version of the relevant version in `IMAGE_MAP` in `.circleci/common.py`.\n\n   Example:\n   The latest AC version is `2.2.0-1` and we want to release `2.2.0-2`.\n   ```diff\n   diff --git a/.circleci/common.py b/.circleci/common.py\n   index xxxxxxx..yyyyyyy 100644\n   --- a/.circleci/common.py\n   +++ b/.circleci/common.py\n   @@ -35,7 +35,7 @@ IMAGE_MAP = collections.OrderedDict([\n        (\"2.1.3-2\", [\"buster\"]),\n        (\"2.1.4-2\", [\"buster\"]),\n        (\"2.2.0-3-dev\", [\"bullseye\", \"buster\"]),\n   -    (\"2.2.1-1\", [\"bullseye\", \"buster\"]),\n   +    (\"2.2.1-2\", [\"bullseye\", \"buster\"]),\n    ])\n\n    # Airflow Versions for which we don't publish Python Wheels\n   ```\n2. Run the `update-dockerfiles` pre-commit hook (this should fail but it should change the\n   relevant Dockerfile).\n\n   Example:\n   ```bash\n   pre-commit run update-dockerfiles\n   ```\n3. Add the changed Dockerfile and commit (this should succeed).\n\n   Example:\n   The `update-dockerfiles` hook updated `2.2.0/bullseye/Dockerfile`:\n   ```bash\n   git add 2.2.0/bullseye/Dockerfile; git commit\n   ```\n\n\u003c/details\u003e\n\n### Add new Astronomer Certified development version\n\n\u003cdetails\u003e\n\u003csummary\u003eClick to expand Step-By-Step instructions\u003c/summary\u003e\n\n1. Add the Astronomer Certified version to `IMAGE_MAP` in `.circleci/common.py`.\n\n   Example:\n   The latest previous release was `2.2.1-1` and we're adding `2.3.0-1-dev`.\n   ```diff\n   diff --git a/.circleci/common.py b/.circleci/common.py\n   index xxxxxxx..yyyyyyy 100644\n   --- a/.circleci/common.py\n   +++ b/.circleci/common.py\n   @@ -36,6 +36,7 @@ IMAGE_MAP = collections.OrderedDict([\n        (\"2.1.4-2\", [\"buster\"]),\n        (\"2.2.0-3-dev\", [\"bullseye\", \"buster\"]),\n        (\"2.2.1-1\", [\"bullseye\", \"buster\"]),\n   +    (\"2.3.0-1-dev\", [\"bullseye\"]),\n    ])\n\n    # Airflow Versions for which we don't publish Python Wheels\n   ```\n4. Edit the new `CHANGELOG.md` to show what has changed in this release.\n\n   Example:\n   ```bash\n   nano 2.3.0/CHANGELOG.md\n   ```\n5. Add the new directory to the Git staging area.\n\n   Example:\n   ```bash\n   git add 2.3.0\n   ```\n6. Run the `update-dockerfiles` pre-commit hook (this should fail but it should change the\n   relevant Dockerfile).\n\n   Example:\n   ```bash\n   pre-commit run update-dockerfiles\n   ```\n\n   The pre-commit hook should change some lines in the new `Dockerfile`.\n   ```diff\n   diff --git a/2.3.0/bullseye/Dockerfile b/2.3.0/bullseye/Dockerfile\n   index xxxxxxx..yyyyyyy 100644\n   --- a/2.3.0/bullseye/Dockerfile\n   +++ b/2.3.0/bullseye/Dockerfile\n   @@ -110,10 +110,10 @@ RUN apt-get update \\\n        \u0026\u0026 apt-get clean \\\n        \u0026\u0026 rm -rf /var/lib/apt/lists/*\n\n   -ARG VERSION=\"2.2.1-1\"\n   +ARG VERSION=\"2.3.0-1.*\"\n    ARG SUBMODULES=\"async,azure,amazon,elasticsearch,google,password,cncf.kubernetes,mysql,postgres,redis,slack,ssh,statsd,virtualenv\"\n    ARG AIRFLOW_MODULE=\"astronomer_certified[${SUBMODULES}]==$VERSION\"\n   -ARG AIRFLOW_VERSION=\"2.2.1\"\n   +ARG AIRFLOW_VERSION=\"2.3.0\"\n\n    # Make pip look at our pip repo too, and force it to install these specific\n    # versions when ever it installs a module.\n   @@ -145,8 +145,8 @@ RUN apt-get update \\\n        \u0026\u0026 apt-get clean \\\n        \u0026\u0026 rm -rf /var/lib/apt/lists/*\n\n   -ARG VERSION=\"2.2.1-1\"\n   -ARG AIRFLOW_VERSION=\"2.2.1\"\n   +ARG VERSION=\"2.3.0-1.*\"\n   +ARG AIRFLOW_VERSION=\"2.3.0\"\n    LABEL io.astronomer.docker.airflow.version=\"${AIRFLOW_VERSION}\"\n    LABEL io.astronomer.docker.ac.version=\"${VERSION}\"\n\n   ```\n7. Stage the changes to the Dockerfile and commit (this should succeed).\n\n   Example:\n   ```bash\n   git add 2.3.0/bullseye/Dockerfile \u0026\u0026 git commit\n   ```\n\n\u003c/details\u003e\n\n### Add a new base build image (eg: new Debian stable release)\n\n\u003cdetails\u003e\n\u003csummary\u003eClick to expand Step-By-Step instructions\u003c/summary\u003e\n\n1. Add or adjust the Debian release name in `IMAGE_MAP`.\n\n   Example:\n   Previous Astronomer Certified versions only built with Debian Buster, but Debian Bullseye has\n   just been released as the new Debian stable version and we'd like to add support for that.\n   ```diff\n   diff --git a/.circleci/common.py b/.circleci/common.py\n   index xxxxxxx..yyyyyyy 100644\n   --- a/.circleci/common.py\n   +++ b/.circleci/common.py\n   @@ -36,7 +36,7 @@ IMAGE_MAP = collections.OrderedDict([\n        (\"2.1.4-2\", [\"buster\"]),\n        (\"2.2.0-3-dev\", [\"bullseye\", \"buster\"]),\n        (\"2.2.1-1\", [\"bullseye\", \"buster\"]),\n   -    (\"2.3.0-1-dev\", [\"buster\"]),\n   +    (\"2.3.0-1-dev\", [\"bullseye\", \"buster\"]),\n    ])\n\n    # Airflow Versions for which we don't publish Python Wheels\n   ```\n2. Add a new version directory for it.\n\n   Example:\n   There is currently a `2.3.0/buster` directory that we need to copy to `2.3.0/bullseye` and\n   then modify that `Dockerfile` to use Debian Bullseye.\n   ```bash\n   cp -a 2.3.0/buster 2.3.0/bullseye\n   ```\n3. Adjust the relevant Dockerfile.\n\n   Example:\n   Update the `2.3.0/bullseye/Dockerfile` to use the upstream Debian Bullseye image.\n   ```diff\n   diff --git a/2.3.0/bullseye/Dockerfile b/2.3.0/bullseye/Dockerfile\n   index xxxxxxx..yyyyyyy 100644\n   --- a/2.3.0/bullseye/Dockerfile\n   +++ b/2.3.0/bullseye/Dockerfile\n   @@ -14,7 +14,7 @@\n    # limitations under the License.\n    ARG APT_DEPS_IMAGE=\"airflow-apt-deps\"\n    ARG PYTHON_MAJOR_MINOR_VERSION=\"3.9\"\n   -ARG PYTHON_BASE_IMAGE=\"python:${PYTHON_MAJOR_MINOR_VERSION}-slim-buster\"\n   +ARG PYTHON_BASE_IMAGE=\"python:${PYTHON_MAJOR_MINOR_VERSION}-slim-bullseye\"\n\n    FROM ${PYTHON_BASE_IMAGE} as airflow-apt-deps\n\n   ```\n4. Stage the changes to the Dockerfile and commit (the pre-commit hooks should all succeed).\n\n   Example:\n   ```bash\n   git add .circleci/common.py 2.3.0/bullseye \u0026\u0026 git commit\n   ```\n\n\u003c/details\u003e\n\n\u003c!-- CHANGELOG START --\u003e\n## Changelog\n\nAll changes applied to available point releases will be documented in the `CHANGELOG.md` files within each version folder:\n- [2.3.4 Changelog](https://github.com/astronomer/ap-airflow/blob/master/2.3.4/CHANGELOG.md)\n- [2.4.3 Changelog](https://github.com/astronomer/ap-airflow/blob/master/2.4.3/CHANGELOG.md)\n\u003c!-- CHANGELOG END --\u003e\n\n## Testing\n\n### Local testing\n\nThis testing will run automatically in CI, but it will save some time to try it out locally first.\n\nAirflow is launched into a local Kubernetes cluster using the project \"kind\" and the most recent\nversion of the Astronomer airflow chart. Python's 'testinfra' module is used to perform system\ntesting on the components while they are running in \"kind\".\n\n### Ensure prerequisites are met:\n\n- docker\n- python3\n- virtualenv\n\nEnsure docker installed, and user has permissions\n```\ndocker run -it --rm hello-world\n```\n\nEnsure Python3 is installed and in PATH\n```\npython3 -c \"print('Confirmed python3 installed.')\"\n```\n\nEnsure virtualenv is installed\n```\nwhich virtualenv\n```\n\n### Set up virtual environment\n\n```\nvirtualenv --python=python3 venv\nsource venv/bin/activate\npip install -r .circleci/test-requirements.txt\n```\n\n### Run system testing\n\nBuild the image you want to test\n```\ndocker build -t airflow ./1.10.5/buster\n```\n\nRun system testing\n```\n.circleci/bin/test-airflow airflow\n```\n\nThe first time you do the build, and the first time you do the system test it will take longer\nthan subsequent runs. The system testing will install the tested versions of CI tools in /tmp/bin\n(helm, kubectl, kind). It will leave an airflow cluster running on your kind cluster in\n'test-cluster'. When you run it again, it will delete the namespace of your most recent deployment\nand redeploy into a new namespace. If you make changes in the image, don't forget to re-build the\nimage before testing it.\n\nUse the newly installed tools\n```\nexport PATH=/tmp/bin:$PATH\n```\n\nEnsure kubectl configured to use kind\n```\nkubectl cluster-info --context kind-test-cluster\n```\n\nLook at the pods\n```\nkubectl get pods --all-namespaces\n```\n\nClean up\n```\nkind delete cluster --name test-cluster\n```\n\n## Scheduled Tasks\n\nThe regularly scheduled tasks are:\n\n#### Edge Builds\n\n* Rebase the `astro-main` branch of [`astronomer/airflow`](https://github.com/astronomer/airflow)\n  onto `main` and push it to `astronomer/airflow:astro-main` (this then kicks off a GitHub Actions\n  workflow that builds Airflow and Astronomer Certified Python packages/wheels) and pushes them to\n  our PyPI package repository\n* Build nightly Docker images for QA (using those nightly Airflow and Astronomer Certified wheels)\n  and push them to the dev image repository\n\n### CircleCI Schedules\n\nThe [CircleCI documentation](https://github.com/CircleCI-Public/api-preview-docs/blob/master/docs/scheduled-pipelines.md)\non scheduled pipelines is very new and in slight disarray.\n\nSee also:\n* The [announcement](https://discuss.circleci.com/t/scheduled-pipelines-are-here/41684/2) for\n  scheduled pipelines\n* This [relevant (for now) PR](https://github.com/CircleCI-Public/api-preview-docs/pull/104),\n  which includes an example of branch filtering\n* Documentation on [parameter syntax](https://circleci.com/docs/2.0/reusing-config/#parameter-syntax)\n* Documentation on [logic statements](https://circleci.com/docs/2.0/configuration-reference/#logic-statements)\n\nThe API for manipulating schedules is documented [here](https://circleci.com/docs/api/v2/#tag/Schedule),\nincluding examples.\n\nHere is an example listing all schedules with HTTPie (and colorizing the response with `jq`):\n\n```bash\n$ http https://circleci.com/api/v2/project/gh/astronomer/ap-airflow/schedule \\\n       circle-token:\u003cCIRCLECI_PERSONAL_ACCESS_TOKEN\u003e \\\n       | jq\n```\n\nTo create a new schedule (refer to the [HTTPie docs about raw JSON](https://httpie.io/docs#json)):\n\n```bash\n$ http --verbose \\\n       https://circleci.com/api/v2/project/gh/astronomer/ap-airflow/schedule \\\n       circle-token:\u003cCIRCLECI_PERSONAL_ACCESS_TOKEN\u003e \\\n       name=\"every-morning-0200-UTC\" \\\n       description=\"Every morning at 02:00 UTC\" \\\n       attribution-actor=\"system\" \\\n       parameters:='{ \"branch\": \"master\" }' \\\n       timetable:='{ \"per-hour\": 1, \"hours-of-day\": [2], \"days-of-week\": [\"SUN\", \"MON\", \"TUE\", \"WED\", \"THU\", \"FRI\", \"SAT\"]}'\n```\n\nNote that updating and deleting schedules uses a different URL path:\n\n```bash\n$ http PATCH \\\n       https://circleci.com/api/v2/schedule/\u003cschedule_uuid\u003e \\\n       circle-token:\u003cCRCLECI_PERSONAL_ACCESS_TOKEN\u003e \\\n       name=every-sunday-0100-utc \\\n       description=\"Every Sunday at 01:00 UTC\"\n````\n\nYou can create a CircleCI personal API token [in your CircleCI user settings](https://app.circleci.com/settings/user/tokens).\nDo note that a PAT will authenticate as you, and have full, read and write access on CircleCI, so\nkeep your PAT secret and do not publish it anywhere!\n\n## License\n\nApache 2.0 with Commons Clause\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fastronomer%2Fap-airflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fastronomer%2Fap-airflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fastronomer%2Fap-airflow/lists"}