{"id":36688072,"url":"https://github.com/converged-computing/flux-operator-component","last_synced_at":"2026-01-12T11:16:39.400Z","repository":{"id":234037645,"uuid":"656468263","full_name":"converged-computing/flux-operator-component","owner":"converged-computing","description":"Developing a KubeFlow componen to burst jobs to the Flux Operator (under development)","archived":false,"fork":false,"pushed_at":"2023-06-23T00:02:56.000Z","size":22,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-09-10T14:50:08.888Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/converged-computing.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2023-06-21T02:48:33.000Z","updated_at":"2023-06-21T22:12:29.000Z","dependencies_parsed_at":"2024-04-18T02:20:33.432Z","dependency_job_id":"019b843a-5252-4fed-9761-95b674b98a46","html_url":"https://github.com/converged-computing/flux-operator-component","commit_stats":null,"previous_names":["converged-computing/flux-operator-component"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/converged-computing/flux-operator-component","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/converged-computing%2Fflux-operator-component","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/converged-computing%2Fflux-operator-component/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/converged-computing%2Fflux-operator-component/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/converged-computing%2Fflux-operator-component/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/converged-computing","download_url":"https://codeload.github.com/converged-computing/flux-operator-component/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/converged-computing%2Fflux-operator-component/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28338970,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-12T10:58:46.209Z","status":"ssl_error","status_checked_at":"2026-01-12T10:58:42.742Z","response_time":98,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-01-12T11:16:39.325Z","updated_at":"2026-01-12T11:16:39.391Z","avatar_url":"https://github.com/converged-computing.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Flux Operator Component\n\nWe will be following the [guide here](https://www.kubeflow.org/docs/components/pipelines/v1/sdk/component-development/)\nto create a custom component for Kubeflow to \"burst\" off a job (a Flux MiniCluster)\nfrom a KubeFlow pipeline. This isn't traditional bursting as we've imagined with Flux\n(extending a running cluster) but rather creating a scoped job to handle one specific\ntask in a workflow (and then go away).\n\n## Setup\n\nCreate your environment with the Kubeflow Pipelines sdk:\n\n```bash\npython -m venv env\nsource env/bin/activate\npip install kfp --upgrade\n```\n\n**under development**\n\n## Development\n\n### Local\n\nThe basis of this component is to launch a job to a cluster, and then\nmonitor it until it finishes, and get output to return. Since local development\ntypically means we have a cluster already running (e.g., kind) we can\ntest this out with an argument to indicate existing. In this case, you don't need to\nbuild the container, but you will need the dependencies installed locally:\n\n```bash\npip install -r requirements.txt\n```\n\nThen, create a local kind cluster. \n\n```bash\nkind create cluster\n```\n\nAt this point we can emulate (manually) submitting a workflow, running the job, monitoring it, and then cleaning up.\nWe will tell the script that the cluster already exists and is local (meaning the kubectl will hit it!)\n\n```bash\npython ./src/deploy.py --local --image ghcr.io/flux-framework/flux-restful-api:latest \\\n       --command \"echo hello world\" \\\n       --nnodes 4 \\\n       --ntasks 4 \\\n       --outfile hello-world.out \\\n       --quiet\n```\n\nYou'll see the Flux Operator install, create the MiniCluster, wait and see the pods running, and get the output!\n\n```bash\n⭐️ Creating the minicluster flux-sample in flux-operator...\nflux-sample-0-fjb9p is in phase Pending\nflux-sample-1-g2sjv is in phase Pending\nflux-sample-2-tmb45 is in phase Pending\nflux-sample-0-fjb9p is in phase Running\nflux-sample-1-g2sjv is in phase Running\nflux-sample-2-tmb45 is in phase Running\nflux-sample-3-zgsv6 is in phase Running\nAll pods are in states \"Running\" or \"Succeeded\" or \"Completed\"\nhello world\nhello world\nhello world\nhello world\nAll pods are terminated.\n```\n\nThe output will be saved to the output file you specified:\n\n```bash\n$ cat hello-world.out \nhello world\nhello world\nhello world\nhello world\n```\n\nWhen you are done, you can delete the cluster:\n\n```bash\n$ kind delete cluster\n```\n\n### Google Cloud \n\nNow we can try doing the same, but via Google Cloud! This will emulate already running from\na container (in KubeFlow) and create the Google Cloud one. \n\n```bash\nGOOGLE_PROJECT=llnl-flux\npython ./src/deploy.py --project ${GOOGLE_PROJECT} --image ghcr.io/flux-framework/flux-restful-api:latest \\\n       --command \"echo hello world\" \\\n       --nnodes 4 \\\n       --ntasks 4 \\\n       --outfile hello-world.out \\\n       --quiet\n```\n\nYou'll see a similar interaction - running the jobs, getting output, but (since we are bursting to Google Cloud)\nyou'll create the cluster beforehand, and destroy it after!\n\n```console\nCommand is echo hello world\n⭐️ Creating the minicluster flux-sample in flux-operator...\nflux-sample-0-htkh7 is in phase Pending\nflux-sample-0-htkh7 is in phase Pending\nflux-sample-1-wwzj7 is in phase Running\nflux-sample-2-gkhrq is in phase Running\nflux-sample-3-nf2vl is in phase Pending\nflux-sample-0-htkh7 is in phase Running\nflux-sample-1-wwzj7 is in phase Running\nflux-sample-2-gkhrq is in phase Running\nflux-sample-3-nf2vl is in phase Running\nAll pods are in states \"Running\" or \"Succeeded\" or \"Completed\"\nhello world\nhello world\nhello world\nhello world\nAll pods are terminated.\n🧿️ Destroying Kubernetes Cluster, we are done!\n```\n\n### KubeFlow on GKE\n\nNow let's test fully in KubeFlow! First, build the container:\n\n```bash\n$ docker build -t ghcr.io/converged-computing/flux-operator-component .\n```\n\n**under development**\n\n## License\n\nHPCIC DevTools is distributed under the terms of the MIT license.\nAll new contributions must be made under this license.\n\nSee [LICENSE](https://github.com/converged-computing/cloud-select/blob/main/LICENSE),\n[COPYRIGHT](https://github.com/converged-computing/cloud-select/blob/main/COPYRIGHT), and\n[NOTICE](https://github.com/converged-computing/cloud-select/blob/main/NOTICE) for details.\n\nSPDX-License-Identifier: (MIT)\n\nLLNL-CODE- 842614\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fconverged-computing%2Fflux-operator-component","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fconverged-computing%2Fflux-operator-component","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fconverged-computing%2Fflux-operator-component/lists"}