{"id":19329785,"url":"https://github.com/outerbounds/airflow-on-minikube","last_synced_at":"2026-05-14T08:32:23.156Z","repository":{"id":39072699,"uuid":"482687869","full_name":"outerbounds/airflow-on-minikube","owner":"outerbounds","description":null,"archived":false,"fork":false,"pushed_at":"2022-08-16T23:34:26.000Z","size":26,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-01-06T08:17:52.236Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/outerbounds.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-04-18T02:22:35.000Z","updated_at":"2022-04-20T19:31:17.000Z","dependencies_parsed_at":"2022-08-25T17:22:57.388Z","dependency_job_id":null,"html_url":"https://github.com/outerbounds/airflow-on-minikube","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/outerbounds%2Fairflow-on-minikube","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/outerbounds%2Fairflow-on-minikube/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/outerbounds%2Fairflow-on-minikube/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/outerbounds%2Fairflow-on-minikube/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/outerbounds","download_url":"https://codeload.github.com/outerbounds/airflow-on-minikube/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240434233,"owners_count":19800548,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-10T02:29:56.544Z","updated_at":"2026-05-14T08:32:23.125Z","avatar_url":"https://github.com/outerbounds.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Metaflow with Airflow on Minikube\n\n\nBefore proceeding please configure an Amazon S3 bucket and ensure you have AWS credentials (access keys, secret keys etc.) set in your environment that allow you to issue get/put/list requests to this bucket. \n\nTo run Metaflow flows with Airflow on Minikube follow these steps in order: \n\n1. [Setup Helm, Minikube, and a dag folder mount point to the minikube cluster](#setup-helm-and-minikube)\n2. [Create Kubernetes namespaces and set AWS related Kubernetes secrets.](#namespace-and-authentication-setup)\n3. [Setup Metaflow and airflow on the minikube cluster](#setup-metaflow-and-airflow-in-minikube-cluster)\n4. [Setup metaflow related configurations](#setting-up-auth-and-metaflow-configurations)\n    1. [Setup metaflow configuration file](#metaflow-configuration-setup)\n    2. [Run a test flow after setting up the metaflow configuration file](#creating-a-test-airflow-dag-from-a-metaflow-flow)\n    3. Access the [Airflow](#getting-access-to-airflow-ui) and [Metaflow UI](#getting-access-to-metaflow-ui)\n    4. [Access the artifacts generated by your flow programmatically](#accessing-artifacts-generated-from-the-flow)\n\n## Setup Helm and Minikube\n\n1. Install Helm and Minikube. You can get these using `brew`: `brew install minikube helm`. We recommend helm version \u003e= 2.5.0.\n2. Start Minikube: `minikube start --cpus 6 --memory 10240` . We recommend at least 6 cpus for the entire deployment. The more resources, the smoother the overall user-experience. \n3. Update Helm with the airflow repo: `helm repo add apache-airflow https://airflow.apache.org`\n4. In a separate terminal window run: ``minikube mount ./dags:/data/dags``. This will create a volume on Minikube that points to the dags folder in this directory. Any file added in the dags folder will be available to containers that have this volume mounted. This step is required for [setting up Airflow on the Minikube cluster](#setup-metaflow-and-airflow-in-minikube-cluster). You can change `./dags` to any directory where you will host Airflow DAGs. \n\n## Namespace and Authentication Setup\n1. Install the `requirements` file in this repository with `pip`. \n```bash\npip install -r requirements.txt\n```\n2. Create a namespace for `metaflow` and `airflow` deployment\n```bash\nkubectl create namespace metaflow\nkubectl create namespace airflow\n```\n3. Once the namespaces have been created, we will setup AWS related credentials in the `metaflow` and `airflow` namespaces. The below command creates secrets for all environment variables in your local shell that start with `AWS`. These secrets will be essential when running Metaflow flows via Airflow for this setup. You wouldn't necessarily need to do the same for a production deployment of Metaflow on Airflow.\n```bash\npython metaflow_configure.py setup-aws-secrets afsecret airflow\npython metaflow_configure.py setup-aws-secrets afsecret metaflow\n```\n\n## Setup Metaflow and Airflow in Minikube Cluster\n\n1. The below command deploys Airflow using the Helm chart configuration provided in [airflow-minikube-config.yml](./airflow-minikube-config.yml). The configuration values attach a volume to the dags folder in the Airflow containers. Currently, the [dags](./dags) folder in the root of this repository is attached as the common volume. Any new files added to this folder will be automatically made available inside Airflow containers.\n    \n    ```bash\n    helm upgrade --install airflow apache-airflow/airflow \\\n        -f airflow-minikube-config.yml \\\n        --timeout 10m0s \\\n        --namespace airflow --create-namespace\n    ```\n2. Wait for 10 minutes after the helm deployment has finished, since all components may not be in the ready state immediately.\n3. Install `nginx-ingress` in the Minikube cluster : `minikube addons enable ingress`\n4. Clone the metaflow tools repository : `git clone git@github.com:outerbounds/metaflow-tools.git`\n5. Deploy the Helm chart for Metaflow from the metaflow-tools repo using the below command. The namespace of the below deployment is `metaflow`. Change the path for `s3://mybucket` to the path of your Amazon S3 bucket. Change `metaflow-ui.envFrom[0].secretRef.name` from `afsecret` to something else only if you have set a different secret name.\n    \n    ```bash\n    helm upgrade --install metaflow metaflow-tools/k8s/helm/metaflow \\\n    \t--timeout 15m0s \\\n    \t--namespace metaflow \\\n    \t--create-namespace \\\n        --set metaflow-ui.METAFLOW_DATASTORE_SYSROOT_S3=s3://mybucket/metaflow \\\n        --set \"metaflow-ui.envFrom[0].secretRef.name=afsecret\" \\\n        --set metaflow-ui.ingress.className=nginx \\\n        --set metaflow-ui.ingress.enabled=true \\\n        --set metaflow-ui.image.tag=latest\n    ```\n- Wait for a few minutes so that the metaflow deployment finishes\n    \n\n## Metaflow Configurations\n\n### Metaflow Configuration Setup\n\nCreate `~/.metaflowconfig` folder if it doesn’t exist and then run the below command to extract a Metaflow configuration for the Minikube cluster and store it in the `~/.metaflow_config` folder. The command requires an Amazon S3 bucket path. If you changed the name of the secret in when [setting up metaflow](#setup-metaflow-and-airflow-in-minikube-cluster) then add `--metaflow-secret \u003cSECRETNAME\u003e` option to the below command\n\n```bash\npython metaflow_configure.py export-metaflow-config s3://mybucket \u003e ~/.metaflowconfig/config.json\n```\n\n### Creating a Test Airflow Dag from a Metaflow flow\n1. Install Metaflow fork with Airflow support: \n```bash\npip install metaflow\n```\n\n2. Since we have [added the AWS related environment variables](#namespace-and-authentication-setup) to `afsecret` we can just run the below command to create the `firstdag.py`  :\n\n```bash\npython flows/card_flow.py --with kubernetes airflow create dags/firstdag.py\n```\n\n### Getting Access to Airflow UI\nThe below command forwards traffic from port 8080 on the airflow-webserver container to port 8080 on the local machine. After running the command you can access the Airflow UI at `http://localhost:8080`\n```bash\nkubectl port-forward svc/airflow-webserver 8080:8080 --namespace airflow\n```\n\n### Getting Access to Metaflow UI\nThe below command opens a tunnel to the Metaflow UI. After running the command you can access the Metaflow UI at `http://localhost`\n```bash\nminikube tunnel\n```\nIf Minikube tunnel doesn't work you can always ssh-port forward port 80 on the Minikube shell.:\n```bash\n# Running this will have UI available at http://localhost:8008\nssh -i $(minikube ssh-key) docker@$(minikube ip) -L 8008:localhost:80 \n```\n\n### Accessing Artifacts generated from the flow\nThe below command forwards traffic from port 8080 of the metaflow-service to port 8089 on the local machine. \n```sh\nkubectl port-forward svc/metaflow-metaflow-service 8089:8080 --namespace metaflow\n```\n\nAfter running this command you can access the artifacts of your flows in the following way : \n```python\nfrom metaflow import Flow,metadata\nmetadata('http://localhost:8089')\n\nflow = Flow('CardFlow') # Change the flow name to get a different flow. \nrun = flow.latest_run # Get the run object\nsteps = list(run.steps())\n\ntask = steps[-1].task # Getting task \n\ndata_artifacts = list(task) # Get the data artifacts. \n```\n\n## [User API Documentation](./user-api.md)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fouterbounds%2Fairflow-on-minikube","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fouterbounds%2Fairflow-on-minikube","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fouterbounds%2Fairflow-on-minikube/lists"}