{"id":20984440,"url":"https://github.com/abhioncbr/docker-airflow","last_synced_at":"2026-04-07T05:31:47.182Z","repository":{"id":76998228,"uuid":"137235789","full_name":"abhioncbr/docker-airflow","owner":"abhioncbr","description":"Repo for building docker based airflow image. Containers support multiple features like writing logs to local or S3 folder and Initializing GCP while container booting. https://abhioncbr.github.io/docker-airflow/","archived":false,"fork":false,"pushed_at":"2019-06-02T23:17:25.000Z","size":1094,"stargazers_count":32,"open_issues_count":3,"forks_count":5,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-05-14T17:39:54.360Z","etag":null,"topics":["airflow","airflow-docker","aws-s3","celery","distributed-systems","docker","docker-airflow","gcp-container-builder","redis","scheduler","work-manager"],"latest_commit_sha":null,"homepage":"https://hub.docker.com/u/abhioncbr/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/abhioncbr.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-06-13T15:36:12.000Z","updated_at":"2024-05-29T01:07:09.000Z","dependencies_parsed_at":null,"dependency_job_id":"90f033ef-284f-4d20-b459-e0914b4bedac","html_url":"https://github.com/abhioncbr/docker-airflow","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/abhioncbr/docker-airflow","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abhioncbr%2Fdocker-airflow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abhioncbr%2Fdocker-airflow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abhioncbr%2Fdocker-airflow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abhioncbr%2Fdocker-airflow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/abhioncbr","download_url":"https://codeload.github.com/abhioncbr/docker-airflow/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abhioncbr%2Fdocker-airflow/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31501903,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-07T03:10:19.677Z","status":"ssl_error","status_checked_at":"2026-04-07T03:10:13.982Z","response_time":105,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow","airflow-docker","aws-s3","celery","distributed-systems","docker","docker-airflow","gcp-container-builder","redis","scheduler","work-manager"],"created_at":"2024-11-19T05:53:31.248Z","updated_at":"2026-04-07T05:31:47.175Z","avatar_url":"https://github.com/abhioncbr.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[\u003cimg src=\"https://github.com/abhioncbr/docker-airflow/raw/master/airflow-logo.png\" align=\"right\"\u003e](https://airflow.apache.org/)\n# docker-airflow\n[![CircleCI](https://circleci.com/gh/abhioncbr/docker-airflow/tree/master.svg?style=svg)](https://circleci.com/gh/abhioncbr/docker-airflow/tree/master)\n[![License](http://img.shields.io/:license-Apache%202-blue.svg)](http://www.apache.org/licenses/LICENSE-2.0.txt)\n[![Code Climate](https://codeclimate.com/github/codeclimate/codeclimate/badges/gpa.svg)](https://codeclimate.com/github/abhioncbr/docker-airflow)\n\nThis is a repository for building [Docker](https://www.docker.com/) container of [Apache Airflow](https://airflow.apache.org/) ([incubating](https://incubator.apache.org/)).\n\n* For understanding \u0026 knowing more about Airflow, please follow [curated list of resources](https://github.com/jghoman/awesome-apache-airflow).\n* Similarly, for Docker follow [curated list of resources](https://github.com/veggiemonk/awesome-docker).\n\n## Images\n\n|Image|Pulls|Tags|\n|:---|:---:|:---:|\n|abhioncbr/docker-airflow|[![Docker Pulls](https://img.shields.io/docker/pulls/abhioncbr/docker-airflow.svg)](https://cloud.docker.com/u/abhioncbr/repository/docker/abhioncbr/docker-airflow)|[tags](https://cloud.docker.com/repository/docker/abhioncbr/docker-airflow/tags)|\n\n## Airflow components stack\n- Airflow version: Notation for representing version `XX.YY.ZZ`\n- Execution Mode: `standalone`(simple container for exploration purpose, based on sqlite as airflow metadata db \u0026 SequentialExecutor ) or `prod`(single node based, LocalExecutor amd mysql as airflow metadata db) and `cluster` (for distributed production long run use-cases, container runs as either `server` or `worker` )\n- Backend database: standalone- Sqlite, prod \u0026 cluster- Mysql\n- Scheduler: standalone- Sequential, prod- LocalExecutor and Cluster- Celery\n- Task queue: cluster- Redis\n- Log location: local file system (Default) or AWS S3 (through `entrypoint-s3.sh`)\n- User authentication: Password based \u0026 support for multiple users with `superuser` privilege.\n- Code enhancement: password based multiple users supporting super-user(can see all dags of all owner) feature. Currently, Airflow is working on the password based multi user feature.\n- Other features: support for google cloud platform packages in container.\n\n## Airflow ports\n- airflow portal port: 2222\n- airflow celery flower: 5555\n- redis port: 6379\n- log files exchange port: 8793\n\n## Airflow services information\n- In server container: redis, airflow webserver \u0026 scheduler is running.\n- In worker container: airflow worker \u0026 celery flower ui service is running.\n\n## How to build images\n* [DockerFile](docker-files/Dockerfile) uses `airflow-version` as a `build-arg`.\n* build image, if you want to do some customization - \n    ```shell \n       docker build -t abhioncbr/docker-airflow:$IMAGE_VERSION --build-arg AIRFLOW_VERSION=$AIRFLOW_VERSION\n                  --build-arg AIRFLOW_PATCH_VERSION=$AIRFLOW_PATCH_VERSION -f ~/docker-airflow/docker-files/DockerFile .\n    ```\n    * Arg IMAGE_VERSION value should be airflow version for example, 1.10.3 or 1.10.2\n    * Arg AIRFLOW_PATCH_VERSION value should be the major release version of airflow for example for 1.10.2 it should be 1.10.\n\n## How to run using Kitmatic\n* Simplest way for exploration purpose, using [Kitematic](https://kitematic.com)(Run containers through a simple, yet powerful graphical user interface.) \n    * Search abhioncbr/docker-airflow Image on [docker-hub](https://hub.docker.com/r/abhioncbr/docker-airflow/) \n        [\u003cimg src=\"search-docker-airflow-Kitematic.png\" alt=\"search-docker-airflow-Kitematic\"\u003e](search-docker-airflow-Kitematic.png)\n    \n    * Start a container through Kitematic UI.\n        [\u003cimg src=\"run-docker-airflow-Kitematic.png\" alt=\"run-docker-airflow-Kitematic\"\u003e](run-docker-airflow-Kitematic.png)\n\n## How to run\n* General commands -\n    * starting airflow image as a `airflow-standalone` container in a standalone mode-\n        ```shell\n        docker run --net=host -p 2222:2222 --name=airflow-standalone abhioncbr/airflow-XX.YY.ZZ -m=standalone \u0026\n        ```\n    \n    * Starting airflow image as a `airflow-server` container in a cluster mode-\n        ```shell\n        docker run --net=host -p 2222:2222 -p 6379:6379 --name=airflow-server \\\n        abhioncbr/airflow-XX.YY.ZZ -m=cluster -t=server -d=mysql://user:password@host:3306/db-name \u0026\n        ```\n\n    * Starting airflow image as a `airflow-worker` container in a cluster mode-\n        ```shell\n        docker run --net=host -p 5555:5555 -p 8739:8739 --name=airflow-worker \\\n        abhioncbr/airflow-XX.YY.ZZ -m=cluster -t=worker -d=mysql://user:password@host:3306/db-name -r=redis://\u003cairflow-server-host\u003e:6379/0 \u0026\n        ```\n\n* In Mac using [docker for mac](https://docs.docker.com/docker-for-mac/install/) -\n    * Standalone Mode - starting airflow image in a standalone mode \u0026 mounting dags, code-artifacts \u0026 logs folder to host machine -\n        ```shell\n        docker run -p 2222:2222 --name=airflow-standalone \\\n        -v ~/airflow-data/code-artifacts:/code-artifacts \\\n        -v ~/airflow-data/logs:/usr/local/airflow/logs \\\n        -v ~/airflow-data/dags:/usr/local/airflow/dags \\\n        abhioncbr/airflow-XX.YY.ZZ -m=standalone \u0026\n        ```     \n    \n    * Cluster Mode\n        * starting airflow image as a server container \u0026 mounting dags, code-artifacts \u0026 logs folder to host machine -\n            ```shell\n            docker run -p 2222:2222 -p 6379:6379 --name=airflow-server \\\n            -v ~/airflow-data/code-artifacts:/code-artifacts \\\n            -v ~/airflow-data/logs:/usr/local/airflow/logs \\\n            -v ~/airflow-data/dags:/usr/local/airflow/dags \\\n            abhioncbr/airflow-XX.YY.ZZ \\\n            -m=cluster -t=server -d=mysql://user:password@host.docker.internal:3306:3306/\u003cairflow-db-name\u003e \u0026\n            ```\n     \n        * starting airflow image as a worker container \u0026 mounting dags, code-artifacts \u0026 logs folder to host machine - \n            ```shell\n            docker run -p 5555:5555 -p 8739:8739 --name=airflow-worker \\\n            -v ~/airflow-data/code-artifacts:/code-artifacts \\\n            -v ~/airflow-data/logs:/usr/local/airflow/logs \\\n            -v ~/airflow-data/dags:/usr/local/airflow/dags \\\n            abhioncbr/airflow-XX.YY.ZZ \\\n            -m=cluster -t=worker -d=mysql://user:password@host.docker.internal:3306:3306/\u003cairflow-db-name\u003e -r=redis://host.docker.internal:6379/0 \u0026   \n            ``` \n    [\u003cimg src=\"docker-airflow-entrypoint-args.png\" alt=\"Airflow\"\u003e](docker-airflow-entrypoint-args.png)      \n    \n## Distributed execution of airflow\n* As mentioned above, docker image of airflow can be leveraged to run in complete distributed run\n    * single docker-airflow container in `server` mode for serving the UI of the airflow, redis for celery task \u0026 scheduler.\n    * multiple docker-airflow containers in `worker` mode for executing tasks using celery executor.\n    * centralised airflow metadata database.\n* Image below depicts the docker-airflow distributed platform:\n    [\u003cimg src=\"airflow-aws-deployment.png\" alt=\"Distributed-Airflow\"\u003e](airflow-aws-deployment.png)   \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fabhioncbr%2Fdocker-airflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fabhioncbr%2Fdocker-airflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fabhioncbr%2Fdocker-airflow/lists"}