{"id":16135287,"url":"https://github.com/xnuinside/piter_py_2020_apache_airflow","last_synced_at":"2025-04-06T16:18:19.416Z","repository":{"id":43453034,"uuid":"284225790","full_name":"xnuinside/piter_py_2020_apache_airflow","owner":"xnuinside","description":"PiterPy 2020: Apache Airflow Workshop","archived":false,"fork":false,"pushed_at":"2023-05-08T20:27:20.000Z","size":3931,"stargazers_count":1,"open_issues_count":1,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-12T22:17:39.639Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/xnuinside.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-08-01T09:07:22.000Z","updated_at":"2020-08-03T20:24:55.000Z","dependencies_parsed_at":"2024-11-01T13:02:43.693Z","dependency_job_id":"fa562551-b0bd-458b-915b-07806fd32be2","html_url":"https://github.com/xnuinside/piter_py_2020_apache_airflow","commit_stats":{"total_commits":6,"total_committers":3,"mean_commits":2.0,"dds":0.5,"last_synced_commit":"d10eaa955d1fdb16b87ef3e7a4f86d82bc1483cf"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xnuinside%2Fpiter_py_2020_apache_airflow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xnuinside%2Fpiter_py_2020_apache_airflow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xnuinside%2Fpiter_py_2020_apache_airflow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xnuinside%2Fpiter_py_2020_apache_airflow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/xnuinside","download_url":"https://codeload.github.com/xnuinside/piter_py_2020_apache_airflow/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247509241,"owners_count":20950232,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-09T23:06:46.517Z","updated_at":"2025-04-06T16:18:19.399Z","avatar_url":"https://github.com/xnuinside.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# PiterPy 2020 Apache Airflow Workshop\n\n## Overview\nThis repository contains Airflow infrastructure setup scripts and example of simple DAG.  \nWe will use it for building of \"Rocket launcher\" pipeline.\n\n## Prerequisites\n1. [Python 3.6](https://www.python.org/downloads/)\n2. [Docker](https://www.docker.com/products/docker-desktop)\n3. [Docker Compose](https://docs.docker.com/compose/install/)\n\n## Local environment setup (optional)\nCreate Python virtual environment:\n```\npython3 -m venv airflow-workshop-env\nsource airflow-workshop-env/bin/activate\n```\n\nInstall packages from `requirements-dev.txt`:\n```\npip3 install -r requirements-dev.txt\n```\n\n## Airflow setup\n1. Run `docker-compose up`\n2. Open Airflow Web UI [http://localhost:8080/admin/](http://localhost:8080/admin/)\n3. Activate DAG (toggle button near the DAG's name)\n3. Click `simple_dag -\u003e Trigger DAG -\u003e Trigger`\n4. Go to `Graph View`, check progress and task's logs\n\n## External APIs setup\n1. Create [OpenWeatherMap](https://openweathermap.org/appid) account, get API key\n2. Create your own [Telegram Bot](https://core.telegram.org/bots#6-botfather)\n\n## Resources\n* [Slides](https://github.com/xnuinside/piter_py_2020_apache_airflow/blob/master/slides/PiterPy%202020%20Airflow%20Workshop%20Slides.pdf)\n* [Apache Airflow documentation](https://airflow.apache.org/docs/stable/)\n\n## Contacts\nJoin our [Telegram Group](https://t.me/piter_py_2020_aw) to receive pipeline's notifications\n\n## For Ubuntu Users\nPossible, that at the start you will got an error:\n\nPermissionError: [Errno 13] Permission denied: '/opt/airflow/logs/scheduler'\n\nTo fix it in docker-compose.yml file change all volumes\n\n  - ./logs:/opt/airflow/logs\n\nto:\n\n  - /opt/airflow/logs\n\n\n## For Windows 10 Users\nIf you try to work on Windows 10 \u0026 run docker-compose on it you will got an issue for **postgres** service:\n\nFATAL:  data directory \"/var/lib/postgresql/data/pgdata\" has wrong ownership\n\nTo solve this issue you must do additional steps (unfortunately there is no more quick workaround, check: https://forums.docker.com/t/data-directory-var-lib-postgresql-data-pgdata-has-wrong-ownership/17963/23 and https://forums.docker.com/t/trying-to-get-postgres-to-work-on-persistent-windows-mount-two-issues/12456/5?u=friism):\n\n1. Create docker volume:\n\n    docker volume create --name volume-postgresql -d local\n\n2. in docker-compose.yml:\n    2.1 add volume at the top of the file, under 'networks' defining like this:\n\n    ``` \n    networks:\n      airflow:\n\n    volumes:\n      volume-postgresql:\n        external: true\n    ```\n\n    2.2 change *postgres* service volumes:\n\n        was:  \n    ```\n      - ./database/data:/var/lib/postgresql/data/pgdata\n      - ./database/logs:/var/lib/postgresql/data/log\n    ```\n\n        become:\n    ```\n      - volume-postgresql:/var/lib/postgresql/data/pgdata\n      - volume-postgresql:/var/lib/postgresql/data/log\n    ```\n\nOr use WSL and run docker under it. \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxnuinside%2Fpiter_py_2020_apache_airflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fxnuinside%2Fpiter_py_2020_apache_airflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxnuinside%2Fpiter_py_2020_apache_airflow/lists"}