{"id":13650014,"url":"https://github.com/noelmcloughlin/airflow-component","last_synced_at":"2025-07-06T15:33:54.938Z","repository":{"id":90036115,"uuid":"400440419","full_name":"noelmcloughlin/airflow-component","owner":"noelmcloughlin","description":"Lightweight IaC Installer of Federated Apache-Airflow","archived":false,"fork":false,"pushed_at":"2021-09-09T12:30:44.000Z","size":1064,"stargazers_count":6,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-02-15T01:28:29.122Z","etag":null,"topics":["airflow","celery-workers","federated","rabbitmq-cluster","salt"],"latest_commit_sha":null,"homepage":"","language":"Jinja","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/noelmcloughlin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":"SUPPORT.md","governance":null,"roadmap":null,"authors":null}},"created_at":"2021-08-27T08:24:23.000Z","updated_at":"2023-02-14T21:15:23.000Z","dependencies_parsed_at":"2023-07-10T09:19:13.869Z","dependency_job_id":null,"html_url":"https://github.com/noelmcloughlin/airflow-component","commit_stats":{"total_commits":32,"total_committers":2,"mean_commits":16.0,"dds":0.28125,"last_synced_commit":"80505234b8de7ab635ccf8b15d15b4b185bc2048"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/noelmcloughlin%2Fairflow-component","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/noelmcloughlin%2Fairflow-component/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/noelmcloughlin%2Fairflow-component/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/noelmcloughlin%2Fairflow-component/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/noelmcloughlin","download_url":"https://codeload.github.com/noelmcloughlin/airflow-component/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247994122,"owners_count":21030049,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow","celery-workers","federated","rabbitmq-cluster","salt"],"created_at":"2024-08-02T02:00:32.620Z","updated_at":"2025-04-09T07:14:04.218Z","avatar_url":"https://github.com/noelmcloughlin.png","language":"Jinja","funding_links":[],"categories":["Soluções de deployment do Airflow","Airflow deployment solutions"],"sub_categories":[],"readme":"# Lightweight federated Apache-Airflow Installer\n\nProvision federated (or single-node) reference deployment architecture of Apache Airflow (RabbitMQ, Postgres), via lightweight single-source-of-truth installer. Heavy lifting/CI by [Saltstack-formulas community](https://github.com/saltstack-formulas).\n![Airflow-Component](/templates/img/airflow-component.png?raw=true \"Federated Airflow, Reference Deployment Architecture\")\n\n    primary:   controller01.main.net  user: main\\airflowservice  - Active Scheduler, UI, worker\n    secondary: controller02.main.net  user: main\\airflowservice  - Standby Scheduler, UI, worker\n    worker01/02: apples, applesdev, applestest\n    worker01/02: oranges, orangesdev, orangestest\n    worker01/02: edge\n    worker01/02: fog\n\n# TL'DR\n\n    ~/airflow-component/installer.sh | tee ~/iac-installer.log\n\n# PREPARE\n\nDeclare your configuration in [sitedata.j2](https://github.com/noelmcloughlin/airflow-component/blob/master/sitedata.j2)\n\nCommission your infrastructure inline with [our infra ticket guidelines](https://github.com/noelmcloughlin/airflow-component/blob/master/INFRA.md)\n\nLogon as airflowservice on each participating host and user.\n\nEnsure proxy is published ~/.bashrc and ~/airflow-component/installer.sh files if applicable:\n\n    export HTTP_PROXY=\"http://myproxy:8080\"\n    export HTTPS_PROXY=\"http://myproxy:8080\"\n    export http_proxy=\"${HTTP_PROXY}\"\n    export https_proxy=\"${HTTPS_PROXY}\"\n    export no_proxy=\"localhost,*.net\"\n    export PATH=${PATH}:/usr/local/bin\n\nPlan to deploy primary/secondary hosts before workers.\n\nOptionally wipe data on any-all servers before reinstall. Normally this is not needed!!\n\n    sudo rm -fr ~/.local /var/lib/rabbitmq/ /var/log/rabbitmq/ /usr/lib/systemd/system/rabbitmq-serv* /usr/lib/systemd/system/airflow-* /etc/rabbitmq/ /var/lib/pgsql /srv/salt \u0026\u0026 sudo reboot\n\n\n# (RE)INSTALL/UPGRADE\n\nLogon as airflowservice on participating hosts and users. Get the software: For hosts without network connectivity to your git (i.e. from fog), use another method, i.e. sftp, see [SUPPORT](https://github.com/noelmcloughlin/airflow-component/blob/master/SUPPORT.md)\n\n    cd \u0026\u0026 rm -fr airflow-component airflow-dags\n    for name in component dags; do\n        git clone https://github.com/noelmcloughlin/airflow-${name}\n    done \u0026\u0026 cd ~/dags \u0026\u0026 rm -fr * \u0026\u0026 cp -Rp ../airflow-dags/dags/* .; chmod +x $( find . -name *.py)\n\nOn each participating host (begin with primary/secondary), install Airflow. Duration is ~15-30mins depending on compute resources:\n\n    ~/airflow-component/installer.sh | tee ~/iac-installer.log\n\nNote, the installation summary may indicate failures. Evaluate result as follows. For failures see [SUPPORT](https://github.com/noelmcloughlin/airflow-component/blob/master/SUPPORT.md)\n\n    - Success if 0 task fails: cluster join worked too. OK!\n    - Success if 1 task fails: cluster join is best effort, other node was not ready (race condition). OK!\n    - Retryable if \u003e1 task fails: sometimes the 2nd attempt just works! NOK!\n    - All other outcomes are failures.\n\nImport variables:\n\n    airflow variables import ~/airflow-dags/variables.json\n\n# USER INTERFACE\n\nAirflow:\n- http://primary.main.net:18080    (user/pass: airflow/airflow or custom)\n- http://secondary.main.net:18080  (user/pass: ditto)\n\nRabbitMQ:\n- http://primary.main.net:15672    (user/pass: airflow/airflow)\n- http://secondary.main.net:15672  (user/pass: airflow/airflow)\n- http://[worker-ipaddr]:15672           (user/pass: airflow/airflow)\n\nCelery Flower:\n- http://primary.main.net:5555\n- http://secondary.main.net:5555\n- http://[worker-ipaddr]:5555\n\n# TROUBLESHOOTING\n\nSee [SUPPORT](https://github.com/noelmcloughlin/airflow-component/blob/master/SUPPORT.md)\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnoelmcloughlin%2Fairflow-component","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnoelmcloughlin%2Fairflow-component","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnoelmcloughlin%2Fairflow-component/lists"}