{"id":18620336,"url":"https://github.com/natural-intelligence/airflow-infra-plugins","last_synced_at":"2025-11-03T12:30:17.989Z","repository":{"id":149504656,"uuid":"344619966","full_name":"Natural-Intelligence/airflow-infra-plugins","owner":"Natural-Intelligence","description":null,"archived":false,"fork":false,"pushed_at":"2022-11-03T08:24:50.000Z","size":6,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-12-27T04:24:55.896Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Natural-Intelligence.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-03-04T21:55:57.000Z","updated_at":"2022-11-03T08:24:55.000Z","dependencies_parsed_at":null,"dependency_job_id":"e47c9cac-ef5a-4657-992e-f3580cde9c56","html_url":"https://github.com/Natural-Intelligence/airflow-infra-plugins","commit_stats":{"total_commits":4,"total_committers":1,"mean_commits":4.0,"dds":0.0,"last_synced_commit":"544570269efc269343a231f0d1ffc7d8ba242c54"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Natural-Intelligence%2Fairflow-infra-plugins","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Natural-Intelligence%2Fairflow-infra-plugins/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Natural-Intelligence%2Fairflow-infra-plugins/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Natural-Intelligence%2Fairflow-infra-plugins/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Natural-Intelligence","download_url":"https://codeload.github.com/Natural-Intelligence/airflow-infra-plugins/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239413531,"owners_count":19634263,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-07T04:05:47.586Z","updated_at":"2025-11-03T12:30:17.884Z","avatar_url":"https://github.com/Natural-Intelligence.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Getting started\n\n* [Install Airflow](#create-service)\n* [Add a plugin](#add-a-plugin)\n* [Watch the Infrastracture](#watch-the-infrastracture)\n   * [Distribution List](#distribution-list)\n   * [Tree view](#tree-view)\n\n## Install Airflow\nChoose one or the other of the two:\n * Follow after the [Quick Start](https://airflow.apache.org/docs/apache-airflow/2.0.1/start/docker.html)\n * Do the below steps\n\n**In both options you should do step 4 and step 5**\n\n#### Step 1\n```bash\ncurl -LfO 'https://airflow.apache.org/docs/apache-airflow/2.0.1/docker-compose.yaml\n```\n#### Step 2\n```bash\nmkdir ./dags ./logs ./plugins\necho -e \"AIRFLOW_UID=$(id -u)\\nAIRFLOW_GID=0\" \u003e .env\n```\n#### Step 3\n```bash\ndocker-compose up airflow-init\n```\n#### Step 4\nEdit the docker-compose.yaml and change the image to 1.10.13 (That's what we use in NI)\n```bash\nimage: ${AIRFLOW_IMAGE_NAME:-apache/airflow:1.10.13-python3.6}\n```\n#### Step 5\nDefine the plugin folder to be under the DAGs folder.\nWe defined it under DAGs folder because our EFS mounts the DAGs and we want to keep having the plugins folder while Airflow got restarted\n```bash\nAIRFLOW__CORE__PLUGINS_FOLDER: '/opt/airflow/dags/plugins'\n```\n#### Step 6\n```bash\ndocker-compose up -d\n```\n\n## Add the plugin\nPlace the plugin folder in the dags folder\n```bash\ncp plugins dags/\n```\nRestart Airflow web and scheduler\n```bash\ndocker container restart airflow_airflow-scheduler_1 \u0026\u0026 docker container restart airflow_airflow-webserver_1\n```\n\n## Watch the Infrastracture\n\n##### Distribution List\nWatch the [distribution list](http://localhost:8080/admin/distribution/)\n\n##### Tree view\nWatch the [tree view](http://localhost:8080/admin/dags/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnatural-intelligence%2Fairflow-infra-plugins","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnatural-intelligence%2Fairflow-infra-plugins","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnatural-intelligence%2Fairflow-infra-plugins/lists"}