{"id":18802172,"url":"https://github.com/oracle-quickstart/oci-airflow","last_synced_at":"2025-04-13T18:10:27.902Z","repository":{"id":45265229,"uuid":"257413505","full_name":"oracle-quickstart/oci-airflow","owner":"oracle-quickstart","description":"Apache Airflow on Oracle Cloud Infrastructure","archived":false,"fork":false,"pushed_at":"2024-01-23T14:08:49.000Z","size":201,"stargazers_count":15,"open_issues_count":3,"forks_count":4,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-03-27T08:58:17.991Z","etag":null,"topics":["oracle-led"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"upl-1.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/oracle-quickstart.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-04-20T21:58:34.000Z","updated_at":"2024-03-27T12:02:03.000Z","dependencies_parsed_at":"2024-11-07T22:30:49.662Z","dependency_job_id":"6401af99-d982-424f-9d99-9becace1d4b5","html_url":"https://github.com/oracle-quickstart/oci-airflow","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oracle-quickstart%2Foci-airflow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oracle-quickstart%2Foci-airflow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oracle-quickstart%2Foci-airflow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oracle-quickstart%2Foci-airflow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/oracle-quickstart","download_url":"https://codeload.github.com/oracle-quickstart/oci-airflow/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248758418,"owners_count":21156957,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["oracle-led"],"created_at":"2024-11-07T22:26:52.960Z","updated_at":"2025-04-13T18:10:27.865Z","avatar_url":"https://github.com/oracle-quickstart.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DEPRECATED\n\nNote that use of this repository is deprecated in favor of [OKE-Airflow](https://github.com/oracle-quickstart/oke-airflow).   This is primarily due to lack of scheduler HA in an IaaS deployment of Airflow, along with the flexibility you get by using KubernetesExecutor for DAG execution, which is a much more efficient use of resources.\n\n# OCI-Airflow\n[Apache Airflow](https://airflow.apache.org/) on Oracle Cloud Infrastructure\n\nThis Quick Start uses [OCI Resource Manager](https://docs.cloud.oracle.com/iaas/Content/ResourceManager/Concepts/resourcemanager.htm)(ORM) to make deployment quite easy.  Simply [download the latest .zip](https://github.com/oracle-quickstart/oci-airflow/archive/master.zip) and follow the [Resource Manager instructions](https://docs.cloud.oracle.com/en-us/iaas/Content/ResourceManager/Tasks/managingstacksandjobs.htm) for how to build a stack.\n\n**This deployment requires using the included schema.yaml file with ORM.**   The GitHub zip must be repackaged so that it's contents are top-level prior to creating the ORM Stack for the schema to be read.  This is a straight forward process:\n\n                unzip oci-airflow-master.zip\n                cd oci-airflow-master\n                zip -r oci-airflow.zip *\n\nUse the `oci-airflow.zip` file created in the last step to create the ORM Stack.  The schema file can even be customized for your use, enabling you to build a set of approved variables for deployment if desired.\n\nThis template will build VCN/Subnets as part of deployment, but also has options for using pre-existing VCN/Subnets.  If using pre-existing network topology, ensure you have a security list entry allowing port TCP 8080 ingress/egress for access to the Airflow UI.   Also ensure a gateway is present to allow Internet access for the Airflow host, as Airflow is downloaded and compiled as part of deployment using options selected in the Resource Manager schema.\n\n## Deployment customization\nThe schema file offers advanced deployment options.   When enabled you can select which airflow libraries are installed during deployment, choose which executor you want to use, and customize other deployment parameters for metadata database and web UI security.   Default libraries are for SSH, Oracle, and MySQL. Note that the apache-airflow[mysql] package is required for installation.   If disabled this will result in a deployment failure.\n\n## Metadata Database\n\n### mysql-local\nThis template uses a community edition of MySQL for Airflow metadata.   This is downloaded and installed during provisioning.   The default root database password is set in the [master_boot.sh](https://github.com/oracle-quickstart/oci-airflow/blob/master/scripts/master_boot.sh#L256) which is run in CloudInit.  It's highly suggested you change the password either prior to deployment, or afterwards to something more secure.\n\n### mysql-oci\nThis deploys a MySQL DB instance on OCI and uses it for metadata in Airflow.  You will need to set some Secret Vault values prior to deployment for this to work, see the Security section below.\n\n### oracle\n*In Development* - This requires some updates to Alembic to work properly, bootstrapping the database currently fails.\n\n## Celery for parallelized execution\nThis template also supports celery executor to parallelize execution among multiple workers.  If using celery and pre-existing VCN/Subnet, ensure a security list entry is present allowing TCP 5555 ingress/egress for the Flower UI on the Airflow master.\n\nSee the Security section below for detail on synchronization of Fernet Key among cluster hosts.\n\n### FSS\nOCI Filesystem Service is offered when using celery.   Enabling this will create an NFS mount on each host in the cluster for `/opt/airflow/dags`.  This provides a single location to manage DAGs in the cluster, and ensures any changes will be in sync among all cluster hosts.\n\n## OCI Hooks, Operators, Sensors\nThis template automatically downloads and installs hooks, operators, and sensors for OCI services into `/opt/airflow/plugins`.   These plugins are fetched remotely by the airflow master instance from this github repository using `wget` as part of the CloudInit deployment.   Long term these hooks, operators and sensors will be committed upstream to Apache Airflow and be included as part of the native deployment.  When using Celery, workers will also fetch these files during deployment.  When using FSS, example Dags are only fetched by the Master host - otherwise all hosts will also download these to `/opt/airflow/dags`.\n\n## Security\n[Instance Principals](https://docs.cloud.oracle.com/en-us/iaas/Content/Identity/Tasks/callingservicesfrominstances.htm) needs to be enabled for all functionality below.  This is offered as part of deployment, but you may need to have your tenancy administrator enable policies for you if you don't have privileges to the tenancy root. \n\nThis template offers basic [Airflow security](https://airflow.apache.org/docs/stable/security.html) when deploying using ORM.   Click Advanced Options \u003e Enable Security to enable local password auth for the Airflow UI.   The password for this needs to be setup in OCI Secrets Vault prior to deployment.\n\nSee [Overview of Vault](https://docs.cloud.oracle.com/en-us/iaas/Content/KeyManagement/Concepts/keyoverview.htm) for more information on how to setup and configure a Vault.   \n\nThe Secrets Vault should be in the same compartment where you are deploying Airflow, and should use the following syntax:\n\n![Airflow Secrets](images/SecretsExample.png) \n\n* AirflowPassword - Password for the Airflow Web UI\n* AirflowUsername - Username for the Airflow Web UI\n* AirflowFernetKey - [Generate a Fernet Key](https://bcb.github.io/airflow/fernet-key) which is synchronized for celery deployments.\n* AirflowDBUsername - Username for the Metadata Database (not used in mysql-local)\n* AirflowDBPassword - Password for the Metadata Database (not useed in mysql-local)\n\n## Logging\nDeployment activities are logged to `/var/log/OCI-airflow-initialize.log`\n\nThis should provide some detail on installation process.  Note that the Airflow UI is not immediately available, as the binaries are compiled as part of deployment.   Watching the log file will tell you when the deployment is complete and the Airflow UI is available:\n\n`sudo tail -f /var/log/OCI-airflow-initialize.log`\n\n## SystemD \nThere are daemon scripts setup as part of deployment.  Airflow can be controlled using systemd commands:\n\n\tsystemctl (start|stop|status|restart) airflow-webserver\n\tsystemctl (start|stop|status|restart) airflow-scheduler\n\nAlso if using celery, the flower service is present on the airflow master, as well as an airflow-worker service on worker nodes.\n\n\tsystemctl (start|stop|status|restart) flower\n\tsystemctl (start|stop|status|restart) airflow-worker\n\nAll services are started during deployment and set to start at boot using chkconfig.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foracle-quickstart%2Foci-airflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foracle-quickstart%2Foci-airflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foracle-quickstart%2Foci-airflow/lists"}