{"id":16514763,"url":"https://github.com/andgineer/airflow","last_synced_at":"2026-05-06T13:14:01.376Z","repository":{"id":145706234,"uuid":"288424679","full_name":"andgineer/airflow","owner":"andgineer","description":"Apache Airflow 2 docker-compose environment with scheduler, workers, DB and live reload of DAGs","archived":false,"fork":false,"pushed_at":"2025-01-05T13:47:45.000Z","size":367,"stargazers_count":0,"open_issues_count":2,"forks_count":0,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-01-13T01:36:38.943Z","etag":null,"topics":["airflow","anaconda","docker","docker-compose","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/andgineer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-08-18T10:27:51.000Z","updated_at":"2025-01-05T13:47:17.000Z","dependencies_parsed_at":"2024-12-24T06:25:30.852Z","dependency_job_id":"50247c1e-a4cb-42e1-8ba4-d1b0415d0f0d","html_url":"https://github.com/andgineer/airflow","commit_stats":null,"previous_names":[],"tags_count":0,"template":true,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andgineer%2Fairflow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andgineer%2Fairflow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andgineer%2Fairflow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andgineer%2Fairflow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/andgineer","download_url":"https://codeload.github.com/andgineer/airflow/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241511796,"owners_count":19974468,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow","anaconda","docker","docker-compose","python"],"created_at":"2024-10-11T16:13:44.762Z","updated_at":"2026-05-06T13:14:01.369Z","avatar_url":"https://github.com/andgineer.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![CI status](https://github.com/andgineer/airflow/workflows/ci/badge.svg)](https://github.com/andgineer/airflow/actions)\n[![Coverage](https://raw.githubusercontent.com/andgineer/airflow/python-coverage-comment-action-data/badge.svg)](https://htmlpreview.github.io/?https://github.com/andgineer/airflow/blob/python-coverage-comment-action-data/htmlcov/index.html)\n# Apache Airflow 3 + Anaconda\n\nDocker Compose environment for local debugging of Apache Airflow DAGs with live reload.\n\nIncludes Airflow scheduler, Celery workers, PostgreSQL database, and Miniconda for using machine learning and data science packages from Anaconda in your ETL pipelines.\n\n[Apache Airflow](https://airflow.apache.org/docs/stable/) is a workflow management platform for building and monitoring data pipelines. Pipelines are configured as Python code, enabling dynamic pipeline generation.\n\n## Quick Start\n\n```bash\n./compose.sh build\n./up.sh\n```\n\n**Access:**\n- Airflow UI: http://127.0.0.1:8080/home (username: `admin`, password: `admin`)\n- Celery Flower: http://127.0.0.1:5551\n\nDAGs are in `etl/` and mounted into containers for live updates.\n\n## Demo DAG\n\nThe repository includes a demo DAG `HelloPandas` to verify everything works.\nCheck the `merge` task logs for: `Done. Returned value was: ('Hello', 'Pandas')`\n\n## Database Connections\n\nThe environment creates a PostgreSQL database for ETL tasks (same server as Airflow metadata DB in `airflow-db` container).\n\n**Required Airflow Connections:**\n- `etl_db` - ETL tasks database\n- `db_dev` - Development/business database for ETL operations\n\n## Configuration\n\n**Scaling Workers:** Use `docker-compose --scale` or deploy workers on separate machines.\n\n**Email Notifications:** Configure SMTP server in `airflow.cfg`.\n\n## Development\n\n### Testing\n\n```bash\n# Create/activate conda environment\n. ./activate.sh\n\n# Run tests\npytest\n```\n\n### Database Migrations\n\nDefine SQLAlchemy models in `etl/db/models/` (inherit from `db.models.Base`).\n\n```bash\n# Generate migration script\n./alembic.sh revision --autogenerate -m \"Schema changes.\"\n\n# Apply migrations\n./alembic.sh upgrade head\n```\n\n## Coverage Reports\n- [Codecov](https://app.codecov.io/gh/andgineer/airflow/tree/master/etl)\n- [Coveralls](https://coveralls.io/github/andgineer/airflow)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fandgineer%2Fairflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fandgineer%2Fairflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fandgineer%2Fairflow/lists"}