{"id":20485526,"url":"https://github.com/knowsuchagency/docker-compose-airflow","last_synced_at":"2025-04-13T15:06:29.612Z","repository":{"id":86109528,"uuid":"170421778","full_name":"knowsuchagency/docker-compose-airflow","owner":"knowsuchagency","description":"An airflow deployment configuration with sane defaults","archived":false,"fork":false,"pushed_at":"2019-06-06T02:50:15.000Z","size":49,"stargazers_count":10,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-13T15:06:23.763Z","etag":null,"topics":["airflow","airflow-docker","devops","docker","docker-compose","docker-swarm","gcp"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/knowsuchagency.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"license.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-02-13T01:53:22.000Z","updated_at":"2024-01-17T09:22:14.000Z","dependencies_parsed_at":null,"dependency_job_id":"9032257c-8c13-41ab-a20d-3ff90b60f5fc","html_url":"https://github.com/knowsuchagency/docker-compose-airflow","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/knowsuchagency%2Fdocker-compose-airflow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/knowsuchagency%2Fdocker-compose-airflow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/knowsuchagency%2Fdocker-compose-airflow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/knowsuchagency%2Fdocker-compose-airflow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/knowsuchagency","download_url":"https://codeload.github.com/knowsuchagency/docker-compose-airflow/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248732485,"owners_count":21152852,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow","airflow-docker","devops","docker","docker-compose","docker-swarm","gcp"],"created_at":"2024-11-15T16:31:24.296Z","updated_at":"2025-04-13T15:06:29.600Z","avatar_url":"https://github.com/knowsuchagency.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Airflow\n\nThis is meant as a template for getting up-and running with\napache airflow quickly using docker compose for local development\nand docker swarm on Google Cloud for deployment.\n\nWhat this is meant to do is help you establish a baseline deployment/development\nenvironment with sane defaults.\n\nThere are many things that could be improved, but this should get you\nup-and-running quickly with some good patterns.\n\n## Some of the features\n* [invoke] for orchestration and configuration\n* [traefik] as edge proxy\n* [grafana] as a metrics front-end for your cluster\n* `pip install`able `flow_toolz` package for library code\n* a recipe for creating new dags that can easily be extended `inv new-dag`\n\n\n# Requirements\n\n* docker `brew cask install docker-edge`\n* python3 `brew install python`\n\n## Quickstart\n```bash\n# create a virtual environment\npython3 -m venv venv\n# activate virtual environment\n. venv/bin/activate\n# install the flow_toolz package\npip install 'airflow/[dev]'\n# generate self-signed tls cert and other filestubs\ninv bootstrap\n# bring up the server for local development\ndocker-compose up\n```\n\n# Authentication\n\n\nYou'll need to create two files at the project root for the purposes of authentication.\n**They can be empty at first**, just to get the server running,\nsince docker-compose will expect them to exist.\n\n* `aws-credentials.ini`\n* `default-service-account.json`\n\n## AWS\n\n`aws-credentials.ini`\n```.ini\n[default]\naws_access_key_id = \u003cyour access key\u003e\naws_secret_access_key = \u003cyour secret key\u003e\n```\n\n## GCP\n`default-service-account.json`\n\nThe `default-service-account.json` [service account key] at the project root will be used\nto authenticate with Google cloud by default.\n\n---\n\n## TLS\n\nIn the [reverse-proxy](./reverse-proxy) folder, you will need \na `certificate.crt` and `key.key` file that you can generate\nwith the `inv create-certificate` command.\n\nThis is really here just to get you started, you'll want to configure\ntraefik to use [letsencrypt] or other means to establish HTTPS on your \nproduction deployment.\n\n## Other\n\nFor other string-based secrets, you'll need a `.secrets.env`[./airflow/.secrets.env] i.e.:\n```\nAIRFLOW_CONN_POSTGRES_MASTER={{password}}\n```\n\nIn general:\n\nAuthentication **strings** should be a [the secrets file](./airflow/.secrets.env)\n\nAuthentication **files** should be set as a docker secret in the [docker compose file](docker-compose.yaml)\n\nSecrets SHOULD NOT be checked into version control.\n\n# Local Development\n\nInitialize the development server (once you have the authentication files described earlier)\n\n``docker-compose up``\n\nNote: it may take some time for the docker images to build at first\n\nThe airflow ui will now render to [localhost](localhost:80)\n\nThe reverse proxy admin panel will be at [localhost:8080](localhost:8080)\n\nGrafana dashboard will be at [localhost:3000](localhost:3000)\n```\nuser: admin\npw: admin\n```\n\nDAGs, and libraries in the [airflow](airflow) folder will automatically\nbe mounted onto the the services on your local deployment and updated on \nthe running containers in real-time.\n\n## Writing a new DAG\n\nThere exists a [handy dag template](airflow/flow_toolz/templates/dag_template.py)\nfor new dags.\n\nYou can use this template to quickly write new dags by using the [task runner](tasks.py):\n```bash\n# invoke the new-dag task\n# you will be prompted to provide parameters \n# such as `dag_id` and `owner`\ninv new-dag\n```\n\n# Library Code\n\nIn the [airflow](./airflow) folder, there is a [flow_toolz](airflow/flow_toolz) directory.\nThat directory is a Python package, meaning it can be `pip install`ed.\n\nCode that is shared between dags, or that you want to use outside of airflow (for testing/development) purposes\nshould be put there.\n\n```\n. venv/bin/activate\n\npip install -e './airflow'\n\n# in python, I can now\n\nimport flow_toolz\n...\n\n```\n\n# Configuration\n\nThe infrastructure -- services and how they'll communicate --\nare all described in `docker-compose.yaml`\n\nCross-service configuration -- environment variables that will exist across different services/machines -- \nwill be in either a `.env` file or `.secrets.env` -- the latter for sensitive information that\nshould not exist in version control.\n\nYou'll notice some of these environment variables follow the pattern `AIRFLOW__{foo}__{bar}`.\n\nThat tells airflow to configure itself with those variables as opposed to their analog in its default config file.\nMore information on how Airflow reads configuration can be found [at this link](https://airflow.apache.org/howto/set-config.html)\n\nFor configuration related to automated cli tasks [executed via invoke](http://www.pyinvoke.org/),\nthose are in `invoke.yaml` files and can be overridden by environment variables as well.\nFor more information on how `invoke` configuration works, [follow this link](http://docs.pyinvoke.org/en/0.11.1/concepts/configuration.html).\n\n# Deployment\n\n### Create the swarm (spins up machines on GCP)\n```bash\ninv swarm-up\n```\n\n### Deploy to our swarm\n```bash\ninv deploy --prod\n```\n\n## Notes\n\n* you'll want to change the names of the images in the docker-compose file for your own deployment\n* invoke tasks that make use of google cloud i.e. `inv deploy` will expect\na `project` element in the configuration. I have this set in my `/etc/invoke.yaml`\n\nHere's an example:\n```yaml\ngcp:\n  project: myproject\n```\n\nYou'll likely also want to change your default host bind **ip** in Docker for Mac\n\n![Imgur](https://i.imgur.com/Vu8vgqA.png)\n\n[invoke]:http://www.pyinvoke.org/\n[traefik]:https://traefik.io/\n[grafana]:https://grafana.com/\n[letsencrypt]:https://letsencrypt.org/\n[service account key]:https://cloud.google.com/iam/docs/creating-managing-service-account-keys\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fknowsuchagency%2Fdocker-compose-airflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fknowsuchagency%2Fdocker-compose-airflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fknowsuchagency%2Fdocker-compose-airflow/lists"}