{"id":50111394,"url":"https://github.com/outerbounds/metaflow-kubeflow","last_synced_at":"2026-06-09T04:00:48.841Z","repository":{"id":328896592,"uuid":"1112593471","full_name":"outerbounds/metaflow-kubeflow","owner":"outerbounds","description":null,"archived":false,"fork":false,"pushed_at":"2026-01-27T06:48:11.000Z","size":62,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-01-27T19:52:58.823Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/outerbounds.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-12-08T20:55:38.000Z","updated_at":"2026-01-27T06:46:40.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/outerbounds/metaflow-kubeflow","commit_stats":null,"previous_names":["outerbounds/metaflow-kubeflow"],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/outerbounds/metaflow-kubeflow","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/outerbounds%2Fmetaflow-kubeflow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/outerbounds%2Fmetaflow-kubeflow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/outerbounds%2Fmetaflow-kubeflow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/outerbounds%2Fmetaflow-kubeflow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/outerbounds","download_url":"https://codeload.github.com/outerbounds/metaflow-kubeflow/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/outerbounds%2Fmetaflow-kubeflow/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34090751,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-09T02:00:06.510Z","response_time":63,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-05-23T12:32:22.513Z","updated_at":"2026-06-09T04:00:48.836Z","avatar_url":"https://github.com/outerbounds.png","language":"Python","funding_links":[],"categories":["Orchestration \u0026 Scheduling"],"sub_categories":[],"readme":"# Kubeflow Pipelines extension for Metaflow\n\nCompile and run Metaflow flows on Kubeflow Pipelines (**argo workflows** backend).\n\n## Basic Usage\n\n- Have access to a Kubeflow Pipelines instance with the API server URL.\n- Use the CLI commands to compile your flow into a Kubeflow Pipeline and deploy it.\n\n## Youtube Screencast\n\n[![metaflow kubeflow demo](https://img.youtube.com/vi/ALg0A9SzRG8/0.jpg)](https://www.youtube.com/watch?v=ALg0A9SzRG8)\n\n## Compiling and Deploying a Pipeline\n\n```py\npython my_flow.py kubeflow-pipelines --url https://my-kubeflow-instance.com create\n```\n\nThis command will:\n\n- Compile your Metaflow flow into a Kubeflow Pipeline YAML specification\n- Upload it to your Kubeflow Pipelines instance\n- Create a new version of the pipeline\n\n### Accessing Kubeflow Pipelines for Deployment\n\nMetaflow needs to be able to connect to Kubeflow Pipelines for deployment. If you have connectivity already set up, you don't need to do anything.\n\nIf you can't connect to the service directly, you can set up a port forward to the service:\n```bash\nkubectl port-forward -n kubeflow svc/ml-pipeline-ui 8081:80\n```\n\nAfter this, you can specify the service URL as `http://localhost:8081` in one of these ways:\n\n- On the CLI for `kubeflow-pipelines` with the `--url` option\n- In the Metaflow config, specify `\"METAFLOW_KUBEFLOW_PIPELINES_URL\": \"http://localhost:8081\"`\n- Set an environment variable, `METAFLOW_KUBEFLOW_PIPELINES_URL=http://localhost:8081`\n\n## Available Commands\n\n### 1. **create** - Compile and/or Deploy Pipeline\n\nCompile a new version of your flow to Kubeflow Pipelines:\n\n**Recurring Runs**: If your flow is decorated with `@schedule`, this command will automatically create or update the corresponding Recurring Run in Kubeflow Pipelines.\n\n```py\npython my_flow.py kubeflow-pipelines \\\n    --url https://my-kubeflow-instance.com \\\n    create \\\n    --version-name v1.0.0 \\\n    --experiment \"My Production Experiment\" \\\n    --alpha 0.5\n```\n\nOptions:\n- `--experiment`: The experiment name to create the recurring run under (if @schedule is present). Defaults to \"Default\".\n- `--version-name`: Allows one to deploy a custom version name. Else, a new version with UTC timestamp is created.\n- `--only-yaml`: Print the YAML specification to stdout and exit without uploading to Kubeflow Pipelines.\n- Flow Parameters: Any flow parameters (e.g., `--alpha`) passed here will be baked into the recurring run configuration (if @schedule is present), overriding the defaults defined in your code.\n\nUse `--help` for all available options including `tags`, `namespace`, `max-workers`, and production token management.\n\n### 2. **trigger** - Execute Pipeline\n\nTrigger an execution of your deployed pipeline:\n\n```py\npython my_flow.py kubeflow-pipelines \\\n    --url https://my-kubeflow-instance.com \\\n    trigger \\\n    --experiment my-experiment \\\n    --alpha 0.1 \\\n    --max-epochs 100\n```\n\nFlow parameters can be passed as command-line arguments. Use `--help` for all available options.\n\nBy default, the latest version of the deployed pipeline is used for the trigger. Else, one can also pass in a custom version using `--version-name`.\n\n### 3. **status** - Check Execution Status\n\nFetch the status of a running or completed pipeline execution:\n\n```py\npython my_flow.py kubeflow-pipelines \\\n    --url https://my-kubeflow-instance.com \\\n    status \\\n    --kfp-run-id abc-123-def-456\n```\n\nUse `--help` for all available options.\n\n### 4. **terminate** - Terminate Execution\n\nTerminate a running pipeline execution:\n\n```py\npython my_flow.py kubeflow-pipelines \\\n    --url https://my-kubeflow-instance.com \\\n    terminate \\\n    --kfp-run-id abc-123-def-456\n```\n\nUse `--help` for all available options.\n\n### 5. **delete** - Delete a Deployed Pipeline\n\nDelete the flow definition and all its associated versions from Kubeflow Pipelines.\n\nThis command also searches for and deletes any associated Recurring Runs (Schedules) to ensure no orphaned schedules continue trying to trigger deleted pipelines.\n\nIn essence, this undeploys the pipeline but preserves execution history (runs) and artifacts.\n\n```py\npython my_flow.py kubeflow-pipelines \\\n    --url https://my-kubeflow-instance.com \\\n    delete\n```\n\nUse `--help` for all available options.\n\n### Fin.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fouterbounds%2Fmetaflow-kubeflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fouterbounds%2Fmetaflow-kubeflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fouterbounds%2Fmetaflow-kubeflow/lists"}