{"id":42925104,"url":"https://github.com/tulibraries/airflow-docker-dev-setup","last_synced_at":"2026-01-30T18:07:48.753Z","repository":{"id":37953479,"uuid":"239641939","full_name":"tulibraries/airflow-docker-dev-setup","owner":"tulibraries","description":"Docker setup for dag repositories so they can run in isolation","archived":false,"fork":false,"pushed_at":"2025-05-02T15:05:01.000Z","size":73,"stargazers_count":5,"open_issues_count":0,"forks_count":0,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-05-02T16:22:42.456Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Makefile","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tulibraries.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2020-02-11T00:22:34.000Z","updated_at":"2025-05-02T15:05:02.000Z","dependencies_parsed_at":"2023-11-07T02:43:04.785Z","dependency_job_id":"43fc398f-da22-4e8c-bc5b-a84b55cfc87b","html_url":"https://github.com/tulibraries/airflow-docker-dev-setup","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/tulibraries/airflow-docker-dev-setup","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tulibraries%2Fairflow-docker-dev-setup","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tulibraries%2Fairflow-docker-dev-setup/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tulibraries%2Fairflow-docker-dev-setup/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tulibraries%2Fairflow-docker-dev-setup/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tulibraries","download_url":"https://codeload.github.com/tulibraries/airflow-docker-dev-setup/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tulibraries%2Fairflow-docker-dev-setup/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28917034,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-30T16:37:38.804Z","status":"ssl_error","status_checked_at":"2026-01-30T16:37:37.878Z","response_time":66,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-01-30T18:07:47.778Z","updated_at":"2026-01-30T18:07:48.738Z","avatar_url":"https://github.com/tulibraries.png","language":"Makefile","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Shared Docker Development Setup for Airflow DAGs\n\n\nThis repo contains all of the setup needed to run airflow in docker, mount your local dag repo into the containers that need it, and allow you to develop dags hopefully quickly!\n\n\n## Expected Setup\n\nThis repo expects:\n\n* to be a git submodule living in the root of your dags repo.\n* dag files are in a directory in the top level.\n* a `local.env` file with `DAG_DIR=some_name` with the name of the directory containing your dag files.\n\n\n## Setup\n\nIn the root of you dags repository, run `git submodule add git@github.com:tulibraries/airflow-docker-dev-setup docker`, and then commit the changes to .gitmodules and the new `docker` directory that has just been created.\n\nNext, in the root of your dags repository, create a file called `local.env`. Add the line `DAG_DIR=my_dag_dir` with the actual name of the directory in the root of the repo containing your dag files. If your dags do any python import statements, this directory name needs to match the top level package in those import statements.\n\nFinally, you should ensure that you have set the following environment varibles, `TUPSFTP_PASSWORD`, `WORKER_SSH_KEY_PATH`, `TUP_ACCOUNT_NAME`, `TUP_SSH_KEY_PATH`, and `TUP_SFTP_ACCOUNT_NAME`. For example:\n\n    export TUPSFTP_PASSWORD=\"REINDEER FLOTILLA\"\n    export WORKER_SSH_KEY_PATH=\"/home/flynn/.ssh/id_rsa\"\n    export TUP_ACCOUNT_NAME=\"flynn_the_deployer\"\n    export TUP_SSH_KEY_PATH=\"/usr/local/airflow/.ssh/flynn_the_deployer\"\n    export TUP_SFTP_ACCOUNT_NAME=\"flynnsplace\"\n\n## Usage\n\nTo use the docker setup, `cd` into the `docker` directory. This contains the `docker-compose.yml` and some other docker configurations, a `docker-requirement.txt` for pypi packages you want installed on the container, and a Makefile defining some useful commands.\n\n```\n\n$ make up\n\n```\n\nThis spins up an Airflow stack using Postgres for the metadata database; Celery, Redis \u0026 Flower for job management; CeleryExecutor, Scheduler, Web-Server and Worker Airflow services; and mounting the local `dags` directory as the Airflow stack's DAGs directory. That DAGs directory has cob_datapipeline and manifold_airflow_dags cloned into it if these subdirectories do not already exist. This will also create some known Variables and Connections, based off of the `variables.json` file found in the dag repository. (the task copies this into `data/local-dev-variables.json` if that file doesn't exist, then loads variables into Airflow from there).\n\nAirflow 2 requires users to login, so an initial user is created as part of this workflow.  The username is test-user and the password is \"password\".\n\nGive this up to 1 minute to start up. You can check the Airflow web-server health-check state by running:\n\n```\n$ docker-compose -p infra ps\n```\n\n#### Reload Docker Instances\n\nIf you change something in the docker setup, e.g. an airflow worker build step, you may want to restart the docker instances (restarts, does not destroy and rebuild):\n\n```\n$ make reload\n```\n\n#### Stop Local Airflow Stack (but do not delete)\n\n```\n $ make stop\n```\n\nThis will stop but not delete the Airflow docker stack, for ease of restart if you want to continue using these instances.\n\n#### Delete Local Airflow Stack\n\n```\n $ make down\n```\n\n#### Start Bash in Given Docker Instance\n\nRun shell in Airflow Worker instance:\n\n```\n $ make tty-worker\n```\n\nRun shell in Airflow Webserver instance:\n\n```\n$ make tty-webserver\n```\n\nRun shell in Airflow Scheduler instance:\n\n```\n$ make tty-scheduler\n```\n\n### Start Bash as Root in Given Docker Instance\n\nRun shell as root in Airflow Worker instance:\n\n```\n$ make tty-root-worker\n```\n\nRun shell as root in Airflow Webserver instance:\n\n```\n$ make tty-root-webserver\n```\nRun shell as root in Airflow Scheduler instance:\n\n```\n$ make tty-root-scheduler\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftulibraries%2Fairflow-docker-dev-setup","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftulibraries%2Fairflow-docker-dev-setup","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftulibraries%2Fairflow-docker-dev-setup/lists"}