https://github.com/KissConsult/Apache-Airflow
Detailed tutorial for installing Apache Airflow on IBM Cloud
https://github.com/KissConsult/Apache-Airflow
Last synced: 3 months ago
JSON representation
Detailed tutorial for installing Apache Airflow on IBM Cloud
- Host: GitHub
- URL: https://github.com/KissConsult/Apache-Airflow
- Owner: KissConsult
- Created: 2020-10-12T12:44:17.000Z (over 4 years ago)
- Default Branch: main
- Last Pushed: 2020-10-13T14:54:45.000Z (over 4 years ago)
- Last Synced: 2024-10-30T20:45:38.675Z (8 months ago)
- Size: 895 KB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
- awesome-apache-airflow - Installing Airflow on IBM Cloud - Quick and easy deployment on IBM Cloud with IBM [Bitnami Charts](https://github.com/bitnami/charts) (Airflow deployment solutions)
- awesome-apache-airflow-br - Tutorial de como instalar Apache Airflow na IBM Cloud - Tutorial de deploy feito de forma rápida e fácil na nuvem IBM com IBM [Bitnami Charts](https://github.com/bitnami/charts) (Soluções de deployment do Airflow)
README
# Get Apache Airflow on IBM Cloud
We will deploy Apache Airflow on an IBM Cloud Kubernetes Cluster
* Prerequisites :
* You should have an IBM Cloud account, otherwise you can [register here].
1. Provisioning a new Kubernetes Cluster, if already have one skip to step **2**
2. Deploying the IBM Cloud Block Storage plug-in, if already have it skip to step **3**
3. Deploying Apache Airflow
## Step 1 provisioning a new Kubernetes Cluster
* Click the **Catalog** button on the top
* Select **Service** from the left in the catalog
* Search for **Kubernetes Service** and click on it
* At the Kubernetes deployment page, we will specify our deployment details
* Choose a plan **standard** or **free**, the free plan only has one worker node and no subnet, to provision a standard cluster, you will need to upgrade you account to Pay-As-You-Go
* To upgrade to a Pay-As-You-Go account, complete the following steps:
* In the console, go to Manage > Account.
* Select Account settings, and click Add credit card.
* Enter your payment information, click Next, and submit your information
* Choose **classic** or **VPC**, read the [docs] and choose the most suitable type for yourself
* Please decide on your deployment's location parameters , for more information please visit [Locations]
* Choose **Geography** (continent)
* Choose **Single** or **Multizone**, in single zone your data is only kept in one datacenter, with Multizone your data is kept on multiple sites for more security
* Choose a **Worker Zone** if using Single zones or **Metro** if Multizone
* If you wish to use Multizone please set up your account with [VRF] or [enable Vlan spanning]
* At your current location selection, it is possible there is no Virtual LAN currently available, then a new Vlan will be created for you
* Choose a **Worker node setup** or use the preselected one, set **Worker node amount per zone**
* Choose **Master Service Endpoint**, In VRF-enabled accounts, you can choose private-only to make your master accessible on the private network or via VPN tunnel. Choose public-only to make your master publicly accessible. When you have a VRF-enabled account, your cluster is set up by default to use both private and public endpoints. For more information visit [endpoints].
* Give cluster a **name**
* Give desired **tags** to your cluster, for more information visit [tags]
* Click **create**
* Wait for you cluster to be provisioned
* Your cluster is ready for usage
## Step 2 deploy IBM Cloud Block Storage plug-in
The Block Storage plug-in is a persistent, high-performance iSCSI storage that you can add to your apps by using Kubernetes Persistent Volumes (PVs).
* Click the **Catalog** button on the top
* Select **Software** from the catalog
* Search for **IBM Cloud Block Storage plug-in** and click on it
* On the application page Click in the _dot_ next to the cluster, you wish to use
* Click on **Enter or Select Namespace** and choose the default Namespace or use a custom one (if you get error please wait 30 minutes for the cluster to finalize)
* Give a **name** to this workspace
* Click **install** and wait for the deployment
## Step 3 Deploy Apache Airflow
In this step we will deploy Apache Airflow on our cluster
* Click the **Catalog** button on the top
* Select **Software** from the left in the catalog
* Search for **Apache Aifrlow** and click on it
* On the application page Click in the _dot_ next to the cluster we just created or use an existing one
* Click on **Enter or Select Namespace** and choose one of the default Namespaces or use a custom one
* Give a unique **name** to your workspace
* Select which resource group you want to use, it is for access controll and billing purposes. For more information please visit [resource groups]
* Here you can give **tags** to your apache airflow workspace, which will affect your deployment. For more information visit [tags]
* Click on **Parameters with default values**, You can set deployment values or use the default ones
* Please **tick** the box next to the agreements and click **install**
* Your apache airflow workspace will start installing, please wait a couple of minutes for the deployment to finish
* You apache airflow workspace has been successfully deployed
## Verify Apache Airflow installation
* Go to [Resources] in your browser
* Click on **Clusters**
* Click on your Cluster
* Now you are at you cluster's overview, here Click on **Actions** on the top right and click on **Web terminal** from the dropdown menu
* Click **install**, then wait couple of minutes
* Click on **Actions**
* Click **Web terminal** --> a terminal will open up
* **Type** in the terminal, please change NAMESPACE to the namespace you choose at the deployment setup:
```sh
$ kubectl get ns
```
```sh
$ kubectl get pod -n NAMESPACE -o wide
```
```sh
$ kubectl get service -n NAMESPACE
```
* Your running Apache Airflow services will be visible
You successfully deployed Apache Airflow on IBM Cloud!
[IBM Cloud]:
[Register Here]:
[guide]:
[here]:
[resource groups]:
[tags]:
[Resources]:
[Locations]:
[VRF]:
[enable Vlan spanning]:
[endpoints]:
[docs]: