{"id":36576003,"url":"https://github.com/converged-computing/fluxqueue","last_synced_at":"2026-01-12T07:35:24.455Z","repository":{"id":270873673,"uuid":"911467017","full_name":"converged-computing/fluxqueue","owner":"converged-computing","description":"Job scheduling in Kubernetes with Flux","archived":false,"fork":false,"pushed_at":"2025-01-31T02:20:37.000Z","size":475,"stargazers_count":0,"open_issues_count":2,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-09-10T14:49:55.264Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/converged-computing.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-01-03T04:56:00.000Z","updated_at":"2025-01-19T13:26:40.000Z","dependencies_parsed_at":"2025-01-19T05:23:10.962Z","dependency_job_id":"ec94718a-f38f-44b6-a759-c8f937b015ad","html_url":"https://github.com/converged-computing/fluxqueue","commit_stats":null,"previous_names":["converged-computing/fluxqueue"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/converged-computing/fluxqueue","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/converged-computing%2Ffluxqueue","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/converged-computing%2Ffluxqueue/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/converged-computing%2Ffluxqueue/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/converged-computing%2Ffluxqueue/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/converged-computing","download_url":"https://codeload.github.com/converged-computing/fluxqueue/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/converged-computing%2Ffluxqueue/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28336544,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-12T06:09:07.588Z","status":"ssl_error","status_checked_at":"2026-01-12T06:05:18.301Z","response_time":98,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-01-12T07:35:23.345Z","updated_at":"2026-01-12T07:35:24.449Z","avatar_url":"https://github.com/converged-computing.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Fluxqueue\n\n\u003e Under development!\n\n![img/fluxqueue.png](img/fluxqueue.png)\n\nI'm still thinking over improvements to fluxnetes, fluence, and related projects, and this is the direction I'm currently taking. I've been thinking of a design where Flux works as a controller, as follows:\n\n1. The controller has an admission webhook that intercepts jobs and pods being submit. For jobs, they are suspended. For all other abstractions, [scheduling gates](https://kubernetes.io/blog/2022/12/26/pod-scheduling-readiness-alpha/) are used.\n2. The jobs are wrapped as `FluxJob` and parsed into Flux Job specs and passed to a part of the controller, the Flux Queue.\n3. The Flux Queue, which runs in a loop, moves through the queue and interacts with a Fluxion service to schedule work.\n4. When a job is scheduled, it is unsuspended and/or targeted for the fluxqueue custom scheduler plugin that will assign exactly to the nodes it has been intended for.\n5. We will need an equivalent cleanup process to receive when pods are done, and tell fluxion and update the queue. Likely those will be done in the same operation.\n\nThis project comes out of [fluxqueue](https://github.com/converged-computing/fluxqueue), which was similar in design, but did the implementation entirely inside of Kubernetes. fluxqueue was a combination of Kubernetes and [Fluence](https://github.com/flux-framework/flux-k8s), both of which use the HPC-grade pod scheduling [Fluxion scheduler](https://github.com/flux-framework/flux-sched) to schedule pod groups to nodes. For our queue, we use [river](https://riverqueue.com/docs) backed by a Postgres database. The database is deployed alongside fluence and could be customized to use an operator instead.\n\n**Important** This is an experiment, and is under development. I will change this design a million times - it's how I tend to learn and work. I'll share updates when there is something to share. It deploys but does not work yet!\nSee the [docs](docs) for some detail on design choices.\n\n## Design\n\n### Containers\n\nFluxqueue builds three primary containers:\n\n - `ghcr.io/converged-computing/fluxqueue`: contains the webhook and operator with a flux queue for pods and groups that interacts with fluxion\n - `ghcr.io/converged-computing/fluxqueue-scheduler`: (TBA) will provide a simple scheduler plugin\n - `ghcr.io/converged-computing/fluxqueue-postgres`: holds the worker queue and provisional queue tables\n\nAnd we use `ghcr.io/converged-computing/fluxion` for the fluxion service.\n\n### Choices\n\n- **Duration of job comes from Kubernetes** Right now, we don't allow a special or different duration to be given to Fluxion. Any duration or deletion needs to come from Kubernetes first, by way of an object deletion. Otherwise we would need to orchestrate deletion from the cluster and Fluxion, and it's easier to ask the user to delete with a job duration or other mechanism.\n- **ungate** is done as a retryable task, the reason being that API operations to Kubernetes are not always reliable.\n\n## Deploy\n\nCreate a kind cluster. You need more than a control plane.\n\n```bash\nkind create cluster --config ./examples/kind-config.yaml\n```\n\nInstall the certificate manager:\n\n```bash\nkubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.1/cert-manager.yaml\n```\n\nThen you can deploy as follows:\n\n```bash\n./hack/quick-build-kind.sh\n```\n\nYou'll then have the fluxqueue service running, a postgres database (for the job queue), along with (TBA) the scheduler plugins controller, which we\ncurrently have to use PodGroup.\n\n```bash\n$ kubectl get pods -n fluxqueue-system\nNAME                                                 READY   STATUS    RESTARTS   AGE\nfluxqueue-chart-controller-manager-6dd6f95c6-z9qdk   0/1     Running   0          9s\npostgres-5dc8c6b49d-llv2s                            0/1     Running   0          9s\n```\n\nYou can then create a job or a pod (note that only pod is fully implemented, job coming soon):\n\n```bash\nkubectl apply -f test/job.yaml\nkubectl apply -f test/pod.yaml\n```\n\nWhich will currently each be suspended (job) or schedule gated (pod) to prevent scheduling. A FluxJob to wrap them is also created:\n\n```bash\n$ kubectl get fluxjobs.jobs.converged-computing.org \nNAME      AGE\njob-pod   4s\npod-pod   6s\n```\n\nHere is where we see the pod moving through the queue, getting sent to Fluxion to schedule, and then being ungated.\n\n```bash\n$ kubectl logs -n fluxqueue-system fluxqueue-controller-manager-5564dc7c9-4fjkx \n```\n\n\u003cdetails\u003e\n\n\u003csummary\u003e Controller Manager Log \u003c/summary\u003e\n\n```console\n2025/01/10 07:21:03 🦩️ starting client (127.0.0.1:4242)...\nNumber nodes  2\nSkipping control plane node  kind-control-plane\n\n📦️ kind-worker\n      allocated cpu: 1\n      available cpu: 15\n      allocated mem: 1193279488\n      available mem: 61682343936\n       running pods: 8\n\n2025-01-10T07:21:03Z    INFO    fluxqueue       match policy    {\"Policy\": \"lonode\"}\n2025-01-10T07:21:03Z    INFO    fluxqueue       ⭐️ Init cluster status  {\"Status\": \"INIT_SUCCESS\"}\n2025-01-10T07:21:03Z    INFO    controller-runtime.webhook      Registering webhook     {\"path\": \"/mutate-v1-sidecar\"}\n2025-01-10T07:21:03Z    INFO    setup   starting manager\n2025-01-10T07:21:03Z    INFO    controller-runtime.metrics      Starting metrics server\n2025-01-10T07:21:03Z    INFO    setup   disabling http/2\n2025-01-10T07:21:03Z    INFO    controller-runtime.webhook      Starting webhook server\n2025-01-10T07:21:03Z    INFO    setup   disabling http/2\n2025-01-10T07:21:03Z    INFO    starting server {\"name\": \"health probe\", \"addr\": \"[::]:8081\"}\n2025-01-10T07:21:03Z    INFO    controller-runtime.certwatcher  Updated current TLS certificate\n2025-01-10T07:21:03Z    INFO    controller-runtime.webhook      Serving webhook server  {\"host\": \"\", \"port\": 9443}\n2025-01-10T07:21:03Z    INFO    controller-runtime.certwatcher  Starting certificate watcher\nI0110 07:21:03.973120       1 leaderelection.go:254] attempting to acquire leader lease fluxqueue-system/b321c34b.converged-computing.org...\n2025-01-10T07:21:04Z    INFO    controller-runtime.metrics      Serving metrics server  {\"bindAddress\": \":8443\", \"secure\": true}\nI0110 07:21:20.494678       1 leaderelection.go:268] successfully acquired lease fluxqueue-system/b321c34b.converged-computing.org\n2025-01-10T07:21:20Z    DEBUG   events  fluxqueue-controller-manager-5564dc7c9-4fjkx_20ecc1fb-82f4-45d5-b074-3a6f9fc8ad6e became leader     {\"type\": \"Normal\", \"object\": {\"kind\":\"Lease\",\"namespace\":\"fluxqueue-system\",\"name\":\"b321c34b.converged-computing.org\",\"uid\":\"b4bd2847-851a-44d1-976f-757f50d4ad4b\",\"apiVersion\":\"coordination.k8s.io/v1\",\"resourceVersion\":\"21215\"}, \"reason\": \"LeaderElection\"}\n2025-01-10T07:21:20Z    INFO    Starting EventSource    {\"controller\": \"fluxjob\", \"controllerGroup\": \"jobs.converged-computing.org\", \"controllerKind\": \"FluxJob\", \"source\": \"kind source: *v1alpha1.FluxJob\"}\n2025-01-10T07:21:20Z    INFO    Starting Controller     {\"controller\": \"fluxjob\", \"controllerGroup\": \"jobs.converged-computing.org\", \"controllerKind\": \"FluxJob\"}\n2025-01-10T07:21:20Z    INFO    Starting workers        {\"controller\": \"fluxjob\", \"controllerGroup\": \"jobs.converged-computing.org\", \"controllerKind\": \"FluxJob\", \"worker count\": 1}\n2025-01-10T07:21:45Z    INFO    webhook Enqueue pod     {\"Name\": \"pod\", \"Namespace\": \"default\"}\n2025-01-10T07:21:45Z    INFO    webhook received pod and added gate     {\"Name\": \"pod\"}\n2025-01-10T07:21:45Z    INFO    submit  Creating flux job       {\"Namespace\": \"default\", \"Name\": \"pod-pod\"}\n2025-01-10T07:21:45Z    INFO    submit  Created flux job        {\"Namespace\": \"default\", \"Name\": \"pod-pod\"}\n2025-01-10T07:21:45Z    INFO    fluxqueue       🌀 Event received by FluxJob controller!\n2025-01-10T07:21:45Z    INFO    fluxqueue       Request:        {\"req\": {\"name\":\"pod-pod\",\"namespace\":\"default\"}}\n2025-01-10T07:21:45Z    INFO    fluxqueue       Found FluxJob   {\"Name\": \"pod-pod\", \"Namespace\": \"default\", \"Status\": \"\"}\n2025-01-10T07:21:45Z    INFO    fluxqueue       Preparing to submit FluxJob     {\"Namespace\": \"default\", \"Name\": \"pod-pod\"}\n2025-01-10T07:21:45Z    INFO    webhook Admission or new or seen pod success.\nfalse\n2025-01-10T07:21:45Z    INFO    fluxqueue       Enqueue for job was successful  {\"Namespace\": \"default\", \"Name\": \"pod-pod\"}\n2025-01-10T07:21:45Z    INFO    fluxqueue       Job was added to pending        {\"Namespace\": \"default\", \"Name\": \"pod-pod\"}\nI0110 07:21:45.357258       1 queue.go:277] [0xc000766330]\n2025-01-10T07:21:45Z    INFO    worker  [WORK] Asking Fluxion running for job   {\"Namespace\": \"default\", \"Name\": \"pod\", \"Args\": {\"jobspec\":\"attributes:\\n  system: {}\\nresources:\\n- count: 1\\n  type: node\\n  with:\\n  - count: 1\\n    label: pod\\n    type: slot\\n    with:\\n    - count: 1\\n      type: core\\ntasks:\\n- command:\\n  - echo\\n  - default\\n  - pod\\n  count:\\n    per_slot: 1\\n  slot: pod\\nversion: 1\\n\",\"object\":null,\"name\":\"pod\",\"namespace\":\"default\",\"flux_job_name\":\"pod-pod\",\"type\":\"0\",\"reservation\":0,\"duration\":0,\"size\":1,\"nodes\":\"\"}}\n2025/01/10 07:21:45 🦩️ starting client (127.0.0.1:4242)...\nattributes:\n  system: {}\nresources:\n- count: 1\n  type: node\n  with:\n  - count: 1\n    label: pod\n    type: slot\n    with:\n    - count: 1\n      type: core\ntasks:\n- command:\n  - echo\n  - default\n  - pod\n  count:\n    per_slot: 1\n  slot: pod\nversion: 1\n\nallocation:\"{\\\"graph\\\": {\\\"nodes\\\": [{\\\"id\\\": \\\"3\\\", \\\"metadata\\\": {\\\"type\\\": \\\"core\\\", \\\"id\\\": 0, \\\"rank\\\": -1, \\\"exclusive\\\": true, \\\"paths\\\": {\\\"containment\\\": \\\"/cluster0/0/kind-worker1/core0\\\"}}}, {\\\"id\\\": \\\"2\\\", \\\"metadata\\\": {\\\"type\\\": \\\"node\\\", \\\"basename\\\": \\\"kind-worker\\\", \\\"id\\\": 1, \\\"rank\\\": -1, \\\"paths\\\": {\\\"containment\\\": \\\"/cluster0/0/kind-worker1\\\"}}}, {\\\"id\\\": \\\"1\\\", \\\"metadata\\\": {\\\"type\\\": \\\"subnet\\\", \\\"basename\\\": \\\"\\\", \\\"id\\\": 0, \\\"rank\\\": -1, \\\"paths\\\": {\\\"containment\\\": \\\"/cluster0/0\\\"}}}, {\\\"id\\\": \\\"0\\\", \\\"metadata\\\": {\\\"type\\\": \\\"cluster\\\", \\\"id\\\": 1, \\\"rank\\\": -1, \\\"paths\\\": {\\\"containment\\\": \\\"/cluster0\\\"}}}], \\\"edges\\\": [{\\\"source\\\": \\\"2\\\", \\\"target\\\": \\\"3\\\"}, {\\\"source\\\": \\\"1\\\", \\\"target\\\": \\\"2\\\"}, {\\\"source\\\": \\\"0\\\", \\\"target\\\": \\\"1\\\"}]}}\\n\" jobid:1 overhead:0.0003170967\n2025-01-10T07:21:45Z    INFO    worker  Parsing fluxion nodes   {\"Nodes\": [{\"id\":\"3\",\"metadata\":{\"type\":\"core\",\"id\":0,\"rank\":-1,\"basename\":\"\",\"exclusive\":true,\"paths\":{\"containment\":\"/cluster0/0/kind-worker1/core0\"}}},{\"id\":\"2\",\"metadata\":{\"type\":\"node\",\"id\":1,\"rank\":-1,\"basename\":\"kind-worker\",\"exclusive\":false,\"paths\":{\"containment\":\"/cluster0/0/kind-worker1\"}}},{\"id\":\"1\",\"metadata\":{\"type\":\"subnet\",\"id\":0,\"rank\":-1,\"basename\":\"\",\"exclusive\":false,\"paths\":{\"containment\":\"/cluster0/0\"}}},{\"id\":\"0\",\"metadata\":{\"type\":\"cluster\",\"id\":1,\"rank\":-1,\"basename\":\"\",\"exclusive\":false,\"paths\":{\"containment\":\"/cluster0\"}}}]}\n2025-01-10T07:21:45Z    INFO    worker  Allocation response     {\"Nodes\": [\"kind-worker\"]}\n2025-01-10T07:21:45Z    INFO    fluxqueue       Updated FluxJob {\"Name\": \"pod-pod\", \"Namespace\": \"default\", \"Status\": \"statusSubmit\"}\n2025-01-10T07:21:45Z    INFO    fluxqueue       🌀 Event received by FluxJob controller!\n2025-01-10T07:21:45Z    INFO    fluxqueue       Request:        {\"req\": {\"name\":\"pod-pod\",\"namespace\":\"default\"}}\n2025-01-10T07:21:45Z    INFO    fluxqueue       Found FluxJob   {\"Name\": \"pod-pod\", \"Namespace\": \"default\", \"Status\": \"statusSubmit\"}\n{\"metadata\": {\"labels\": {\"fluxqueue/fluxion-nodes\": \"kind-worker\"}}}\n2025-01-10T07:21:45Z    INFO    webhook Enqueue pod     {\"Name\": \"pod\", \"Namespace\": \"default\"}\n2025-01-10T07:21:45Z    INFO    webhook Admission or new or seen pod success.\n2025-01-10T07:21:45Z    INFO    webhook Enqueue pod     {\"Name\": \"pod\", \"Namespace\": \"default\"}\n2025-01-10T07:21:45Z    INFO    webhook Admission or new or seen pod success.\n2025-01-10T07:21:45Z    INFO    worker  Success ungating pod    {\"Namespace\": \"default\", \"Name\": \"pod\"}\n2025-01-10T07:21:45Z    INFO    worker  [WORK] nodes allocated for job  {\"JobId\": 1, \"Nodes\": [\"kind-worker\"], \"Namespace\": \"default\", \"Name\": \"pod\"}\n```\n\n\u003c/details\u003e\n\n\nAnd the output from the scheduler plugin:\n\n```bash\nI0110 07:21:45.377076       1 scheduler.go:65] \"PreFilter received contender pod\" pod=\"default/pod\"\nI0110 07:21:45.377102       1 scheduler.go:93] \"PreFilter node assignment\" pod=\"default/pod\" node=\"kind-worker\"\nI0110 07:21:45.377158       1 scheduler.go:105] \"Filter received pod assignment\" pod=\"default/pod\" node=\"kind-worker\"\n```\n\nAnd of course the pod running on the node.\n\n```bash\nNAME   READY   STATUS    RESTARTS   AGE    IP            NODE          NOMINATED NODE   READINESS GATES\npod    1/1     Running   0          119s   10.244.1.40   kind-worker   \u003cnone\u003e           \u003cnone\u003e\n```\n\nNext I'll finish up job (not a lot to do) and work on edge cases of cancel to Fluxion, etc.\n\n## Development\n\n### Reservations\n\n\u003e How do reservations work?\n\n - We have a queue with one worker that sends jobs to the scheduler. This is important because Fluxion is single threaded, and we need the \"clear reservation\" job to run last.\n - The scheduler does everything that it can, and then it starts reserving things (e.g., easy allows 1 reservation)\n - Reservations block off resources in fluxion and give an estimated start time.\n - Reservations are cleared after the loop. This means that later jobs (smaller) aren't given resources that should go to larger jobs (with higher priority)\n - A single job to clear reservations is added to the end of a schedule loop. E.g.,\n   - We retrieve reservation ids\n   - We issue a cancel to fluxion\n   - On success, we delete the reservation ID from the table\n\n\n### Debugging Postgres\n\nIt is often helpful to shell into the postgres container to see the database directly:\n\n```bash\nkubectl exec -n fluxqueue-system -it postgres-597db46977-9lb25 bash\npsql -U postgres\n\n# Connect to database \n\\c\n\n# list databases\n\\l\n\n# show tables\n\\dt\n\n# test a query\nSELECT * from pending_queue;\nSELECT * from reservations;\n```\n\n### TODO\n\n- [ ] Pod creation needs better orchestration\n- [ ] In the case of jobs that are changing (e.g., pods deleting, but we don't want to kill entire job) what should we do?\n  - we need to use shrink or partial cancel here. And a shrink down to size 0 I assume is a cancel.\n- [ ] For cancel, we would issue a cancel for every pod associated with a job. How can we avoid that (or is that OK?)\n- [ ] we will eventually need another mechanism to move schedule queue aside from new submission\n- [ ] scheduleAt can be used to AskFlux in the future\n- [ ] Nodes that are currently assigned need to be taken into account\n   - Right now they aren't included in resources, but instead should be \"given\" to Fluxion.\n   - Can we use the bypass that I used for the container scheduler?\n   - Analogous to restarting cluster when jobs already running\n   - Resource representation of what is running gets sent back to flux-sched.\n   - [UpdateAllocate](https://github.com/flux-framework/fluxion-go/blob/bbe5b38ff747eba76e4eda8205a7bfba5f6aee82/pkg/fluxcli/reapi_cli.go#L206) \n   - make up job ids\n   - take system / operator pods that are running, convert into JGF, and then figure out which resources each are running on. \n   - at end of loop need to go through jobs, for those that aren't scheduled cancel.\n- [ ] kubectl plugin to get fluxion state?\n\n## License\n\nHPCIC DevTools is distributed under the terms of the MIT license.\nAll new contributions must be made under this license.\n\nSee [LICENSE](https://github.com/converged-computing/cloud-select/blob/main/LICENSE),\n[COPYRIGHT](https://github.com/converged-computing/cloud-select/blob/main/COPYRIGHT), and\n[NOTICE](https://github.com/converged-computing/cloud-select/blob/main/NOTICE) for details.\n\nSPDX-License-Identifier: (MIT)\n\nLLNL-CODE- 842614\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fconverged-computing%2Ffluxqueue","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fconverged-computing%2Ffluxqueue","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fconverged-computing%2Ffluxqueue/lists"}