{"id":16135301,"url":"https://github.com/xnuinside/airflow_in_docker_compose","last_synced_at":"2025-04-09T15:06:31.074Z","repository":{"id":40390123,"uuid":"199626460","full_name":"xnuinside/airflow_in_docker_compose","owner":"xnuinside","description":"Apache Airflow in Docker Compose (for both versions 1.10.* and 2.*)","archived":false,"fork":false,"pushed_at":"2023-11-23T13:19:03.000Z","size":525,"stargazers_count":185,"open_issues_count":3,"forks_count":78,"subscribers_count":11,"default_branch":"main","last_synced_at":"2025-04-09T15:06:25.039Z","etag":null,"topics":["airflow","apache-airflow","celery-executor","docker","docker-airflow","docker-compose","docker-compose-files","docker-compose-template"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/xnuinside.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-07-30T10:05:50.000Z","updated_at":"2025-04-08T14:19:22.000Z","dependencies_parsed_at":"2025-01-03T22:02:20.408Z","dependency_job_id":null,"html_url":"https://github.com/xnuinside/airflow_in_docker_compose","commit_stats":{"total_commits":45,"total_committers":4,"mean_commits":11.25,"dds":"0.19999999999999996","last_synced_commit":"c1f4f2e338895c151ca458d37b2589735128f2a5"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xnuinside%2Fairflow_in_docker_compose","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xnuinside%2Fairflow_in_docker_compose/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xnuinside%2Fairflow_in_docker_compose/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xnuinside%2Fairflow_in_docker_compose/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/xnuinside","download_url":"https://codeload.github.com/xnuinside/airflow_in_docker_compose/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248055284,"owners_count":21040157,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow","apache-airflow","celery-executor","docker","docker-airflow","docker-compose","docker-compose-files","docker-compose-template"],"created_at":"2024-10-09T23:06:53.501Z","updated_at":"2025-04-09T15:06:31.053Z","avatar_url":"https://github.com/xnuinside.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Maybe you will be also interested\n\n* [airflow-helper](https://github.com/xnuinside/airflow-helper) - Pretty Fresh command line tool to set up Apache Airflow connections, variables \u0026 pools from yaml config. Support config inheritance \u0026 feature to get settings from existed server.\n\n# Official Docker-Compose\nPay attention that in current time already exists official Docker-Compose.yml https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html (maybe better to use it)\n\n### Apache Airflow version 2.0.0\n(2.0 not 100% bacward compatible to 1.10+ this is because I move it to separate compose file):\n\nBy default now RBAC is turn on and this mean, that to use Airflow UI you need create user first, for this in db_init service was added also command to create default user:\n\u003e\u003e *airflow users create --firstname admin --lastname admin --email admin --password admin --username admin --role Admin*\n\nChange your user password and login as you want. By default it is login: admin, password: admin.\n\n![New Apache Airflow 2.0](/docs/img/2.0.png?raw=true \"Apache Airflow 2.0\")\n\n**Note:**\nIf you will run docker-compose for 2nd and more times in init_db you will see log:\n\n```\n    initdb_1     | admin already exist in the db\n    airflow_in_docker_compose_initdb_1 exited with code 0\n```\n\n**[docker-compose-with-celery-executor.yml](docker-compose-2.0-with-celery-executor.yml)**\n\n**NOTE: if you previous run Airflow 1.10 - remove your DB volume files before run 2.0 or change db init command to db upgrade.**\n\n```bash\n\n    git clone https://github.com/xnuinside/airflow_in_docker_compose\n    cd airflow_in_docker_compose\n    \n    docker-compose -f docker-compose-2.0-with-celery-executor.yml up --build\n\n```\n### Apache Airflow 2.* with 2 Celery Workers (or more)\n\nBecause was issue about run Apache 2.0 with 2 Celery workers I think will be not bad to have docker-compose with such set up. \n\nI added it as separate compose file:\n\n**[docker-compose-2.0-with-celery-executor-2-workers.yml](docker-compose-2.0-with-celery-executor-2-workers.yml)**\n\nTo check that your workers up\u0026run well - use flower UI (it exists in docker-compose setup):\n![Flower UI with 2 workers](/docs/img/flower.png?raw=true \"Flower UI with 2 workers\")\n\n\n### Apache Airflow version 1.10.14:\n```bash\n\n    git clone https://github.com/xnuinside/airflow_in_docker_compose\n    cd airflow_in_docker_compose\n\n    # to run airflow with 1 Celery worker\n    docker-compose up --build\n\n```\n\nWait until all services will succesfull up \u0026 open http://localhost:8080/admin. \n\n### FAQ \u0026 Help\n\nExists different behaviour of Docker Compose on different OS relative to file system specifications, work with access rights \u0026 etc. This docker-compose file tested by me in MacOS mostly, some time I can up \u0026 run it on wsl (but not each update).\n\nIn issues you can find some cases when something goes wrong and maybe it will help you to solve own issue.\n\n**Ubuntu Issues:**\n\n1. [Permission denied error](https://github.com/xnuinside/airflow_in_docker_compose/issues/4)\n\n**WSL Issues:**\n\n1. [No DAGs in UI in Airflow 2.0 \u0026 failed airflow init on second runs](https://github.com/xnuinside/airflow_in_docker_compose/issues/10) - Not resolved yet\n\nAlso at the end of this README.md file exists section https://github.com/xnuinside/airflow_in_docker_compose#for-windows-10-users with some information wor WSL users. Maybe it also can help. \n\n**Problem with connection to PostgreSQL (at first time run)**:\n\nIf you share low count of resources for Docker or you have a machine with low perfomance, up\u0026run PostgreSQL for the first time can take a significant time. And you can see the errors like this:\n\n``` \nIs the server running on host \"postgres\" (172.25.0.3) and accepting\ninitdb_1     |  TCP/IP connections on port 5432?\n```\n\nIn normal behaviour - in docker-compose I added autorestarts so after 10-15 secs all servers will be up\u0026run, but sometimes 3 retries can be not enough. \n\nI can recommend in this case at first time run postgres service separate until you will see information that Postgres is up \u0026 ready to accept connections.:\n\n```\n    docker-compose -f docker-compose-2.0-with-celery-executor-2-workers.yml up --build postgres\n```\n\nIf you had any troubles \u0026 you successfully solve it - please open an issue with solution, I will add it to this readme.md file. Thank you!\n\n\n# Apache Airflow with Docker Compose examples\n**UPD from July 2020: \nThose articles was created before release of official Apache Airflow Docker image and they use puckel/docker-airflow. \nNow, already exist official image apache/airflow. So this docker-compose files became 'legacy' \nand all sources moved to 'docker_with_puckel_image'.\nMain Docker Compose Cluster based on  apache/airflow Image**\n\nDocker-compose config based on official image (required docker-compose version 3.7 and higher):\n\n**[docker-compose-with-celery-executor.yml](docker-compose-with-celery-executor.yml)**\n\n\nAnd env file with config setting for Airflow (used in docker-compose-with-celery-executor.yml):\n**[.env](.env)**\n\n\nSource files for article with description on Medium.\n\n**Apache Airflow with LocalExecutor:**\n    \u003chttps://medium.com/@xnuinside/quick-guide-how-to-run-apache-airflow-cluster-in-docker-compose-615eb8abd67a\u003e\n    \n**Apache Airflow with CeleryExecutor:**\n    \u003chttps://medium.com/@xnuinside/quick-tutorial-apache-airflow-with-3-celery-workers-in-docker-composer-9f2f3b445e4\u003e\n\n**Install Python dependencies to docker-compose cluster without re-build images**\n    \u003chttps://medium.com/@xnuinside/install-python-dependencies-to-docker-compose-cluster-without-re-build-images-8c63a431e11c\u003e\n\n\n![Main Apache Airflow UI](/docs/img/main.png?raw=true \"Main Apache Airflow UI\")\n![Version](/docs/img/version.png?raw=true \"Version Screen\")\n\n### 10.12.2022:\n1. Updated version to 2.5.0\n\n### 20.09.2022:\n1. Updated version to 2.4.0\n2. all files with version 1.* \u0026 puckel images moved to \"archive\" folder\n3. 2* became default version\n4. Updated docker-compose version\n\n### 20.09.2022:\n1. Updated version to 2.4.0\n2. all files with version 1.* \u0026 puckel images moved to \"archive\" folder\n3. 2* became default version\n4. Updated docker-compose version\n\n### 03.02.2021:\n1. In docker-compose files for Airflow 2.0 **scheduler** service restart police changed to 'any' because for some reason it exist with 0 if error in DB and init is not finished yet, so restart policy 'on-failure' does not works. \n2. Added example for Apached Airflow 2.0 with 2 workers. \n\n### 02.02.2021:\n1. Added FAQ section with issues that might help \n2. Updated fernet key in .env\n\n\n### 18.12.2020:\n1. Added separate docker-compose file for Apache Airflow 2.0 version\n\n\n### 16.12.2020:\n1. Update Apache Airflow version to 1.10.14\n2. Change  init db command to \"airflow db init\"\n\n\n### 29.11.2020:\n1. Update Apache Airflow version to 1.10.12\n2. Update PostgreSQL DB to 13.1\n3. Added restart_policy to services in docker-compose\n\n\n### 07.2020:\n1. All compose files with puckel_image moved to docker_with_puckel_image\n2. Creted docker-compose config based on official image (required docker-compose version 3.7 and higher):\n**[docker-compose-with-celery-executor.yml](docker-compose-with-celery-executor.yml)**\nAnd env file with config setting for Airflow (used in docker-compose-with-celery-executor.yml):\n**[.env](.env)**\n3. At the bottom of readme added note for Windows 10 users\n\n\n### 21.07.2020:\n1. Docker Compose files with puckel images moved to docker_with_puckel_image\n2. Added docker-compose-with-celery.yml based on official image.\n\n### 18.12.19 changes:\n1. added samples for article https://medium.com/@xnuinside/install-python-dependencies-to-docker-compose-cluster-without-re-build-images-8c63a431e11c (docker-compose-volume-packages.yml, packages.pth, added commented lines to Dockerfile)\n2. added .dockerignore\n\n### 29.11.19 changes:\n1. Apache Airflow Image was updated to version 1.10.6\n2. Added test_dag into airflow_files\n\n## For Windows 10 Users\nIf you try to work on Windows 10 \u0026 run docker-compose on it you will got an issue for **postgres** service:\n\nFATAL:  data directory \"/var/lib/postgresql/data/pgdata\" has wrong ownership\n\nTo solve this issue you must do additional steps (unfortanutely there is no more quick workaround, check: https://forums.docker.com/t/data-directory-var-lib-postgresql-data-pgdata-has-wrong-ownership/17963/23 and https://forums.docker.com/t/trying-to-get-postgres-to-work-on-persistent-windows-mount-two-issues/12456/5?u=friism):\n\n1. Create docker volume:\n\n    docker volume create --name volume-postgresql -d local\n\n\n2. in docker-compose.yml:\n    2.1 add volume at thetop of the file, under 'networks' defining like this:\n\n    ``` \n    networks:\n      airflow:\n\n    volumes:\n      volume-postgresql:\n        external: true\n    ```\n\n    2.2 change *postgres* service volumes:\n\n        was:  \n    ```\n      - ./database/data:/var/lib/postgresql/data/pgdata\n      - ./database/logs:/var/lib/postgresql/data/log\n    ```\n\n        become:\n    ```\n      - volume-postgresql:/var/lib/postgresql/data/pgdata\n      - volume-postgresql:/var/lib/postgresql/data/log\n    ```\n\nOr use WSL and run docker under it. \n\nIf you never use docker with mount local folders as volumes under WSL possible you need first follow up this article: https://nickjanetakis.com/blog/setting-up-docker-for-windows-and-wsl-to-work-flawlessly#ensure-volume-mounts-work because by default volumes are not mounted correct and you will not see any 'dags' in Airflow. \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxnuinside%2Fairflow_in_docker_compose","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fxnuinside%2Fairflow_in_docker_compose","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxnuinside%2Fairflow_in_docker_compose/lists"}