{"id":17429854,"url":"https://github.com/pacuna/airflow-docker","last_synced_at":"2025-04-16T02:19:42.688Z","repository":{"id":73891758,"uuid":"140973444","full_name":"pacuna/airflow-docker","owner":"pacuna","description":"Run Apache Airflow using Docker containers","archived":false,"fork":false,"pushed_at":"2018-09-05T13:14:34.000Z","size":7,"stargazers_count":8,"open_issues_count":0,"forks_count":6,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-29T04:04:55.210Z","etag":null,"topics":["airflow","airflow-docker","apache-airflow","containers","docker","docker-compose"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pacuna.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-07-14T19:44:05.000Z","updated_at":"2022-02-14T00:46:05.000Z","dependencies_parsed_at":"2023-09-21T12:02:15.954Z","dependency_job_id":null,"html_url":"https://github.com/pacuna/airflow-docker","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pacuna%2Fairflow-docker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pacuna%2Fairflow-docker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pacuna%2Fairflow-docker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pacuna%2Fairflow-docker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pacuna","download_url":"https://codeload.github.com/pacuna/airflow-docker/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249183449,"owners_count":21226202,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow","airflow-docker","apache-airflow","containers","docker","docker-compose"],"created_at":"2024-10-17T07:10:14.235Z","updated_at":"2025-04-16T02:19:42.675Z","avatar_url":"https://github.com/pacuna.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# airflow-docker\n\nRun Airflow using docker containers\n\n## Run\n\n```sh\ndocker-compose up --build\n```\n\n## Description\n\nThis project creates a full working Airflow environment using Docker containers.\n\nIt uses the Celery Executor with Redis as the broker/backend and PostgreSQL as the core Airflow database.\n\nAll the configuration is done via environmental variables. If you want change these values or add\nnew variables, you can modify the `env/airflow.env` file using the Airflow convention of\n`AIRFLOW__SECTION__VAR_NAME` where the section and var name are defined in the `airflow.cfg` file that\ncomes with the airflow installation. The environmental variables will override any value from the cfg file.   \n\nThe airflow.env file contains an example fernet key that is used by all containers so they can all decrypt encrypted values. For any production use, this should be changed by creating a new value using\n\n```\npython -c \"from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())\"\n```\n\n\nAll the necessary Airflow processes including `airflow initdb` command, webserver, scheduler and worker are\nisolated in their own Docker container and with a proper startup control using custom scripts (`sh/wait-for-migrations.sh`, `sh/wait-for-postgres.sh`).\nSo the project should start with not errors the first time. If not, please create an issue describing your case.\n\nBy default, the webserver forwards the port 8080 to the host so you can go to `localhost:8080` after starting the project.\n\nThe python dependencies are specified in the Dockerfile for now in case you need to add more packages.\n\nThe Dags and Plugins folder are mounted into the webserver, worker and scheduler so you can modify them and see the changes immediatly. If you see\nsome file stuck in an old state, it's better to restart (and maybe rebuild) the whole project.\n\n## Tips\n\n- Use `docker-compose logs -f` to troubleshoot errors\n- You can open bash sessions inside the containers to test airflow commands: `docker-compose exec worker bash` or `docker-compose run --rm worker airflow list_dags`\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpacuna%2Fairflow-docker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpacuna%2Fairflow-docker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpacuna%2Fairflow-docker/lists"}