{"id":21489579,"url":"https://github.com/kooznitsa/airflow_project","last_synced_at":"2026-04-13T22:32:02.873Z","repository":{"id":242032673,"uuid":"808472737","full_name":"kooznitsa/airflow_project","owner":"kooznitsa","description":"Apache Airflow project","archived":false,"fork":false,"pushed_at":"2024-05-31T07:59:12.000Z","size":143,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-06T08:06:24.660Z","etag":null,"topics":["aiohttp","airflow","asyncio","docker","fastapi","postgresql","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kooznitsa.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-31T06:29:20.000Z","updated_at":"2024-05-31T07:59:15.000Z","dependencies_parsed_at":"2024-05-31T09:19:50.793Z","dependency_job_id":null,"html_url":"https://github.com/kooznitsa/airflow_project","commit_stats":null,"previous_names":["kooznitsa/airflow_project"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/kooznitsa/airflow_project","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kooznitsa%2Fairflow_project","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kooznitsa%2Fairflow_project/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kooznitsa%2Fairflow_project/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kooznitsa%2Fairflow_project/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kooznitsa","download_url":"https://codeload.github.com/kooznitsa/airflow_project/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kooznitsa%2Fairflow_project/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31774025,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-13T20:17:16.280Z","status":"ssl_error","status_checked_at":"2026-04-13T20:17:08.216Z","response_time":93,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aiohttp","airflow","asyncio","docker","fastapi","postgresql","python"],"created_at":"2024-11-23T14:22:08.337Z","updated_at":"2026-04-13T22:32:02.855Z","avatar_url":"https://github.com/kooznitsa.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Apache Airflow Project\n\nImportant notes:\n\n- Apache Airflow is not intended for Windows. Run in Docker or in Linux environment.\n- Your Python modules should be within dags/ directory.\n- Do not set ```datetime.now()``` or ```date.today()``` as DAG start_date.\n\nNotes on Airflow dependencies:\n\n- [apache-airflow-providers-celery](https://airflow.apache.org/docs/apache-airflow-providers-celery/3.7.0/index.html): for Celery workers.\n- [sqlalchemy==1.4](https://docs.sqlalchemy.org/en/14/): Apache Airflow throws an error with SQLAlchemy \u003e 2.0 (```sqlalchemy.exc.ArgumentError: Type annotation for \"TaskInstance.dag_model\" can't be correctly interpreted for Annotated Declarative Table form```). Since SQLModel works only with SQLAlchemy \u003e 2.0, the use of both libraries is impossible. Use SQLAlchemy 1.4 instead.\n\n## Environment variables\n\n- POSTGRES_DB: database used by Airflow and Celery (same as in AIRFLOW__DATABASE__SQL_ALCHEMY_CONN and AIRFLOW__CELERY__RESULT_BACKEND).\n- POSTGRES_API_DB: database for your API.\n- AIRFLOW__CORE__FERNET_KEY: generate a fernet key:\n\n```python\nfrom cryptography.fernet import Fernet\n\nfernet_key = Fernet.generate_key()\nprint(fernet_key.decode())\n```\n\n## Docker\n\n1. Create Docker network:\n\n```bash\ndocker network create airflow-net\n```\n\n2. Build Docker containers:\n\n```bash\ndocker compose up -d --build\n```\n\n3. Register a server with PGAdmin:\n\n- Port: 5433\n- User: postgres\n- Host: localhost\n\nThere should be 2 databases: \"airflowDB\" for Airflow DAGs and Celery backend, and \"apiDB\" for API.\n\n4. Apply migrations:\n\n```bash\ndocker exec -it api poetry run alembic upgrade head\n```\n\nAirflow dashboard: http://localhost:8080/home\n\nAPI: http://127.0.0.1:8000/docs\n\n## Optionally: Install Airflow with Linux WSL\n\n```bash\n# 1. Create directory\n\ncd home\nmkdir airflow_project\ncd airflow_project\n\n# 2. Install pip and virtualenv\n\napt-get update\napt install python3-pip\npip install virtualenv\npip install virtualenv\nvirtualenv airflow_env\nsource airflow_env/bin/activate\n\n# 3. Install Airflow\n\npip install apache-airflow\nairflow db init\n\nairflow users create \\\n          --username admin \\\n          --firstname admin \\\n          --lastname admin \\\n          --role Admin \\\n          --email email@gmail.com\n          \n# Password: insert password\n\n# 4. Run the scheduler\n\nairflow scheduler\n\n# 5. Another terminal: run the webserver\n\ncd airflow_project\nvirtualenv airflow_env\nsource airflow_env/bin/activate\n\nairflow webserver\n# to change the default port 8080:\n# airflow webserver –port \u003cport number\u003e\n```\n\nAirflow dashboard will be available at http://localhost:8080/home\n\n## Optionally: Install with a script (not tested)\n\n```bash\n# install_airflow.sh\n\nAIRFLOW_VERSION=2.9.1\n\n# Extract the version of Python you have installed. If you're currently using a Python version that is not supported by Airflow, you may want to set this manually.\n# See above for supported versions.\nPYTHON_VERSION=\"$(python -c 'import sys; print(f\"{sys.version_info.major}.{sys.version_info.minor}\")')\"\n\nCONSTRAINT_URL=\"https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt\"\n# For example this would install 2.9.1 with python 3.8: https://raw.githubusercontent.com/apache/airflow/constraints-2.9.1/constraints-3.8.txt\n\npip install \"apache-airflow==${AIRFLOW_VERSION}\" --constraint \"${CONSTRAINT_URL}\"\n```\n\n```python\n# main.py\n\nimport subprocess\n\nsubprocess.call('install_airflow.sh', shell=True)\n```\n\n## Useful resources\n\n### Airflow\n\n- [Apache Airflow Docs](https://airflow.apache.org/docs/apache-airflow/stable/index.html)\n- [Apache Airflow Docs: Best Practices](https://airflow.apache.org/docs/apache-airflow/2.9.1/best-practices.html)\n- [Apache Airflow Docs: Cron Presets](https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/cron.html#cron-presets)\n- [GitHub: Apache Airflow Tutorial](https://github.com/coder2j/airflow-docker/tree/main)\n- [GitHub: docker-airflow](https://github.com/marclamberti/docker-airflow/tree/main)\n- [GitHub: Airflow Celery Workers (with bash scripts)](https://github.com/himewel/airflow_celery_workers/tree/main)\n- [GitHub: Airflow Tutorial](https://github.com/dmlogv/airflow-tutorial/tree/master)\n- [GitHub: Airflow Made Easy | Local Setup Using Docker](https://github.com/anilkulkarni87/airflow-docker/tree/main)\n- [GitHub: docker-compose-CeleryExecutor.yml](https://github.com/puckel/docker-airflow/blob/master/docker-compose-CeleryExecutor.yml)\n- [GitHub: Airflow Logging](https://github.com/apache/airflow/issues/38479)\n- [GitHub: airflow-repo-template + Makefile](https://github.com/soggycactus/airflow-repo-template/tree/master)\n- [freeCodeCamp: How to Install Apache Airflow on Windows without Docker](https://www.freecodecamp.org/news/install-apache-airflow-on-windows-without-docker/)\n- [Habr: Airflow — инструмент, чтобы удобно и быстро разрабатывать и поддерживать batch-процессы обработки данных](https://habr.com/ru/companies/vk/articles/339392/)\n- [Habr: Apache Airflow: делаем ETL проще](https://habr.com/ru/articles/512386/)\n- [7 Common Errors to Check When Debugging Airflow DAGs](https://www.astronomer.io/blog/7-common-errors-to-check-when-debugging-airflow-dag/)\n\n### Other\n\n- [GitHub: Free Currency Exchange Rates API](https://github.com/fawazahmed0/exchange-api)\n- [Free Weather API](https://open-meteo.com/)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkooznitsa%2Fairflow_project","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkooznitsa%2Fairflow_project","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkooznitsa%2Fairflow_project/lists"}