{"id":50385116,"url":"https://github.com/stackhpc/slurm-k8s-cluster","last_synced_at":"2026-05-30T14:31:00.298Z","repository":{"id":176604729,"uuid":"656622805","full_name":"stackhpc/slurm-k8s-cluster","owner":"stackhpc","description":"A Slurm cluster for Kubernetes","archived":false,"fork":false,"pushed_at":"2023-09-13T09:41:47.000Z","size":1968,"stargazers_count":18,"open_issues_count":9,"forks_count":5,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-04-14T22:50:22.951Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":"giovtorres/slurm-docker-cluster","license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/stackhpc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-06-21T10:02:01.000Z","updated_at":"2024-04-08T09:01:43.000Z","dependencies_parsed_at":null,"dependency_job_id":"ae051ef6-3aad-4fd5-af23-4a6b19cbc9b1","html_url":"https://github.com/stackhpc/slurm-k8s-cluster","commit_stats":null,"previous_names":["stackhpc/slurm-docker-cluster","stackhpc/slurm-k8s-cluster"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/stackhpc/slurm-k8s-cluster","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stackhpc%2Fslurm-k8s-cluster","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stackhpc%2Fslurm-k8s-cluster/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stackhpc%2Fslurm-k8s-cluster/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stackhpc%2Fslurm-k8s-cluster/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/stackhpc","download_url":"https://codeload.github.com/stackhpc/slurm-k8s-cluster/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stackhpc%2Fslurm-k8s-cluster/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33696681,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-05-30T02:00:06.278Z","response_time":92,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-05-30T14:30:59.666Z","updated_at":"2026-05-30T14:31:00.293Z","avatar_url":"https://github.com/stackhpc.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Slurm Docker Cluster\n\nThis is a multi-container Slurm cluster using Kubernetes. The Slurm cluster Helm chart creates a named volume for persistent storage of MySQL data files. By default, it also installs the\nRookNFS Helm chart (also in this repo) to provide shared storage across the Slurm cluster nodes.\n\n## Dependencies\n\nRequires:\n\n* A Kubernetes cluster\n* Local installations of\n  * Helm\n  * kubectl\n\n## Containers and Volumes\n\nThe Helm chart will run the following containers:\n\n* login\n* mysql\n* slurmdbd\n* slurmctld\n* slurmd (2 replicas by default)\n\nThe Helm chart will create the following named volumes:\n\n* var_lib_mysql     ( -\u003e /var/lib/mysql )\n\nA named ReadWriteMany (RWX) volume mounted to `/home` is also expected, this can be external or can be deployed using the provided `rooknfs` chart directory (See \"Deploying the Cluster\").\n\n## Configuring the Cluster\n\nAll config files in `slurm-cluster-chart/files` will be mounted into the container to configure their respective services on startup. Note that changes to these files will not all be propagated to existing deployments (see \"Reconfiguring the Cluster\"). Additional parameters can be found in the `values.yaml` file for the Helm chart. Note that some of these values will also not propagate until the cluster is restarted (see \"Reconfiguring the Cluster\").\n\n## Deploying the Cluster\n\n### Generating Cluster Secrets\n\nOn initial deployment ONLY, run\n```console\n./generate-secrets.sh [\u003ctarget-namespace\u003e]\n```\nThis generates a set of secrets in the target namespace to be used by the Slurm cluster. If these need to be regenerated, see \"Reconfiguring the Cluster\"\n\nBe sure to take note of the Open Ondemand credentials, you will need them to access the cluster through a browser\n\n### Connecting RWX Volume\n\nA ReadWriteMany (RWX) volume is required for shared storage across cluster nodes. By default, the Rook NFS Helm chart is installed as a dependency of the Slurm cluster chart in order to provide a RWX capable Storage Class for the required shared volume. If the target Kubernetes cluster has an existing storage class which should be used instead, then `storageClass` in `values.yaml` should be set to the name of this existing class and the RookNFS dependency should be disabled by setting `rooknfs.enabled = false`. In either case, the storage capacity of the provisioned RWX volume can be configured by setting the value of `storage.capacity`.\n\nSee the separate RookNFS chart [values.yaml](./rooknfs/values.yaml) for further configuration options when using the RookNFS to provide the shared storage volume.\n\n### Supplying Public Keys\n\nTo access the cluster via `ssh`, you will need to make your public keys available. All your public keys from localhost can be added by running\n\n```console\n./publish-keys.sh [\u003ctarget-namespace\u003e]\n```\nwhere `\u003ctarget-namespace\u003e` is the namespace in which the Slurm cluster chart will be deployed (i.e. using `helm install -n \u003ctarget-namespace\u003e ...`). This will create a Kubernetes Secret in the appropriate namespace for the Slurm cluster to use. Omitting the namespace arg will install the secrets in the default namespace.\n\n### Deploying with Helm\n\nAfter configuring `kubectl` with the appropriate `kubeconfig` file, deploy the cluster using the Helm chart:\n```console\nhelm install \u003cdeployment-name\u003e slurm-cluster-chart\n```\n\nNOTE: If using the RookNFS dependency, then the following must be run before installing the Slurm cluster chart\n```console\nhelm dependency update slurm-cluster-chart\n```\n\nSubsequent releases can be deployed using:\n\n```console\nhelm upgrade \u003cdeployment-name\u003e slurm-cluster-chart\n```\n\nNote: When updating the cluster with `helm upgrade`, a pre-upgrade hook will prevent upgrades if there are running jobs in the Slurm queue. Attempting to upgrade will set all Slurm nodes to `DRAINED` state. If an upgrade fails due to running jobs, you can undrain the nodes either by waiting for running jobs to complete and then retrying the upgrade or by manually undraining them by accessing the cluster as a privileged user. Alternatively you can bypass the hook by running `helm upgrade` with the `--no-hooks` flag (may result in running jobs being lost)\n\n## Accessing the Cluster\n\nRetrieve the external IP address of the login node using:\n```console\nLOGIN=$(kubectl get service login -o jsonpath=\"{.status.loadBalancer.ingress[0].ip}\")\n```\nand connect to the cluster as the `rocky` user with\n```console\nssh rocky@$LOGIN\n```\n\nFrom the shell, execute slurm commands, for example:\n\n```console\n[root@slurmctld /]# sinfo\nPARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST\nnormal*      up 5-00:00:00      2   idle c[1-2]\n```\n\n## Running MPI Benchmarks\n\nThe Intel MPI Benchmarks are included in the containers. These can be run both with mpirun and srun. They can also be run as a containerised workload using apptainer.\n\nExample job scripts:\n* srun:\n```console\n#!/usr/bin/env bash\n\n#SBATCH -N 2\n#SBATCH --ntasks-per-node=1\n\necho $SLURM_JOB_ID: $SLURM_JOB_NODELIST\nsrun /usr/lib64/openmpi/bin/mpitests-IMB-MPI1 pingpong\n```\n* mpirun\n```console\n#!/usr/bin/env bash\n\n#SBATCH -N 2\n#SBATCH --ntasks-per-node=1\n\necho $SLURM_JOB_ID: $SLURM_JOB_NODELIST\n/usr/lib64/openmpi/bin/mpirun --prefix /usr/lib64/openmpi mpitests-IMB-MPI1 pingpong\n```\n* apptainer\n```console\n#!/usr/bin/env bash\n#SBATCH -N2\n#SBATCH --ntasks-per-node=1\nMPI_CONTAINER_TAG=\"main\"\necho SLURM_JOB_NAME: $SLURM_JOB_NAME\necho $SLURM_JOB_ID: $SLURM_JOB_NODELIST\nsrun singularity exec docker://ghcr.io/stackhpc/mpitests-container:${MPI_CONTAINER_TAG} /usr/lib64/openmpi/bin/mpitests-IMB-MPI1 pingpong\n```\n\nNote: The mpirun script assumes you are running as user 'rocky'. If you are running as root, you will need to include the --allow-run-as-root argument\n\n## Reconfiguring the Cluster\n\n### Changes to config files\n\nChanges to the Slurm configuration in `slurm-cluster-chart/files/slurm.conf` will be propagated (it may take a few seconds) to `/etc/slurm/slurm.conf` for all pods except the `slurmdbd` pod by running\n\n```console\nhelm upgrade \u003cdeployment-name\u003e slurm-cluster-chart/\n```\n\nThe new Slurm configuration can then be read by running `scontrol reconfigure` as root inside a Slurm pod. The [slurm.conf documentation](https://slurm.schedmd.com/slurm.conf.html) notes that some changes require a restart of all daemons, which here requires redeploying the Slurm pods as described below.\n\nChanges to other configuration files (e.g. Munge key etc) require a redeploy of the appropriate pods.\n\nTo redeploy pods use:\n```console\nkubectl rollout restart deployment \u003cdeployment-names ...\u003e\n```\nfor the `slurmdbd`, `login` and `mysql` pods and\n\n```\nkubectl rollout restart statefulset \u003cstatefulset-names ...\u003e\n```\nfor the `slurmd` and `slurmctld` pods\n\nGenerally restarts to `slurmd`, `slurmctld`, `login` and `slurmdbd` will be required.\n\n### Changes to secrets\n\nRegenerate secrets by rerunning\n```console\n./generate-secrets.sh\n```\nSome secrets are persisted in volumes, so cycling them requires a full teardown and reboot of the volumes and pods which these volumes are mounted on. Run\n```console\nkubectl delete deployment mysql\nkubectl delete pvc var-lib-mysql\nhelm upgrade \u003cdeployment-name\u003e slurm-cluster-chart\n```\nand then restart the other dependent deployments to propagate changes:\n```console\nkubectl rollout restart deployment slurmd slurmctld login slurmdbd\n```\n\n# Known Issues\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstackhpc%2Fslurm-k8s-cluster","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstackhpc%2Fslurm-k8s-cluster","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstackhpc%2Fslurm-k8s-cluster/lists"}