https://github.com/vantagecompute/slurm-factory
Slurm build factory using Spack + Docker.
https://github.com/vantagecompute/slurm-factory
build-tool cicd docker hpc slurm spack
Last synced: 4 months ago
JSON representation
Slurm build factory using Spack + Docker.
- Host: GitHub
- URL: https://github.com/vantagecompute/slurm-factory
- Owner: vantagecompute
- License: other
- Created: 2025-08-09T15:41:42.000Z (10 months ago)
- Default Branch: main
- Last Pushed: 2026-01-15T19:50:47.000Z (5 months ago)
- Last Synced: 2026-01-15T21:55:08.409Z (5 months ago)
- Topics: build-tool, cicd, docker, hpc, slurm, spack
- Language: Python
- Homepage: https://vantagecompute.github.io/slurm-factory/
- Size: 901 KB
- Stars: 3
- Watchers: 0
- Forks: 1
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Slurm Factory
[](LICENSE)
[](https://python.org)
[](https://pypi.org/project/slurm-factory/)
[](https://docker.com)
Build relocatable, **GPG-signed** Slurm packages using Docker and Spack.
[Documentation](https://vantagecompute.github.io/slurm-factory) | [Buildcache](https://slurm-factory-spack-binary-cache.vantagecompute.ai)
## Quick Start
### Option 1: Install Pre-built Slurm from Buildcache (Fastest!)
Use Spack to install GPG-signed pre-built binaries (**no slurm-factory tool needed**):
```bash
# Install Spack
git clone --depth 1 --branch v1.0.0 https://github.com/spack/spack.git
source spack/share/spack/setup-env.sh
# Add mirrors for dependencies and Slurm
SLURM_VERSION=25.11
TOOLCHAIN=noble # or: jammy, resolute, rockylinux9, rockylinux10, rockylinux8
CLOUDFRONT_URL=https://slurm-factory-spack-binary-cache.vantagecompute.ai
spack mirror add slurm-factory-deps "${CLOUDFRONT_URL}/${TOOLCHAIN}/slurm/deps/"
spack mirror add slurm-factory-slurm "${CLOUDFRONT_URL}/${TOOLCHAIN}/slurm/${SLURM_VERSION}/"
# Import GPG keys and install Slurm (5-15 min!)
spack buildcache keys --install --trust
spack install slurm@${SLURM_VERSION}
```
**→ Full guide:** [Installing Slurm from Buildcache](https://vantagecompute.github.io/slurm-factory/installing-slurm-from-buildcache)
### Option 2: Build Custom Slurm with slurm-factory Tool
Install the slurm-factory tool to build custom packages:
```bash
# Install Docker and the slurm-factory build tool
pip install slurm-factory
# Build Slurm with default compiler (GCC 13.4.0)
slurm-factory build-slurm --slurm-version 25.11
```
**→ Full guide:** [Installing slurm-factory Tool](https://vantagecompute.github.io/slurm-factory/installation)
## Supported Versions
### Slurm × Operating System Matrix
All combinations use **OS-provided compilers** and are available in the public buildcache:
| Slurm Version | Status | Supported Operating Systems |
|---------------|-------------|-------------------------------------------------------------|
| **25.11** | Latest | Rocky 10, Rocky 9, Rocky 8, **Ubuntu 24.04**, Ubuntu 22.04, Ubuntu 25.04 |
| **24.11** | LTS | Rocky 10, Rocky 9, Rocky 8, **Ubuntu 24.04**, Ubuntu 22.04, Ubuntu 25.04 |
| **23.11** | Stable | Rocky 10, Rocky 9, Rocky 8, **Ubuntu 24.04**, Ubuntu 22.04, Ubuntu 25.04 |
**Default**: Ubuntu 24.04 (Noble) - recommended for most users
### Operating System Compiler Toolchains
Slurm is built using the default system compiler from each OS distribution:
| OS | Codename | GCC Version | GLIBC | Use Case |
|----|----------|-------------|-------|---------|
| **Ubuntu 24.04** | **Noble** | **13.3.0** | **2.39** | **Recommended - Modern stable** |
| Ubuntu 26.04 | Resolute | 15.2.0 | 2.42 | Latest features (development) |
| Ubuntu 22.04 | Jammy | 11.4.0 | 2.35 | LTS - Wide compatibility |
| Rocky Linux 10 | - | 14.3.1 | 2.39 | RHEL 10 compatible |
| Rocky Linux 9 | - | 11.5.0 | 2.34 | RHEL 9 compatible |
| Rocky Linux 8 | - | 8.5.0 | 2.28 | RHEL 8 compatible |
## GPG Package Signing
All packages are **cryptographically signed with GPG** for security and integrity.
### Why GPG Signing?
- ✅ **Authenticity**: Verify packages were built by Vantage Compute
- ✅ **Integrity**: Detect tampering or corruption during download
- ✅ **Security**: Prevent man-in-the-middle attacks
- ✅ **Trust Chain**: Establish provenance for production deployments
### GPG Key Information
```text
Key ID: DFB92630BCA5AB71
Owner: Vantage Compute Corporation (Slurm Factory Spack Cache Signing Key)
Email: info@vantagecompute.ai
```
### Importing GPG Keys
Keys are automatically imported when using the buildcache:
```bash
# Automatic import and trust
spack buildcache keys --install --trust
# Packages are verified during installation
spack install slurm@25.11%gcc@13.4.0
```
## Features
- **🔐 GPG-Signed Packages** - All Slurm packages cryptographically signed
- **⚡ 10-15x Faster** - Pre-built packages install in 5-15 minutes vs 45-90 minutes
- **📦 Relocatable** - Deploy to any path, no host dependencies
- **🌍 CDN Distribution** - CloudFront-distributed buildcache for fast global access
- **🐧 6 OS Platforms** - Rocky Linux 8/9/10, Ubuntu 22.04/24.04/25.04
- **🎯 3 Slurm Versions** - 25.11, 24.11, 23.11
- **🏭 OS-Native Compilers** - Uses system-provided GCC for maximum compatibility
- **🚀 Optimized** - Architecture-specific compilation (x86_64_v3)
- **🐳 Clean Builds** - Docker isolation, no system pollution
## Build Options
### Build from Source with slurm-factory
```bash
# Default build (CPU-only, Ubuntu 24.04 toolchain)
slurm-factory build-slurm --slurm-version 25.11
# GPU support (CUDA/ROCm)
slurm-factory build-slurm --slurm-version 25.11 --gpu
# Different OS toolchain
slurm-factory build-slurm --slurm-version 25.11 --toolchain jammy
# Build and publish to buildcache with GPG signing
slurm-factory build-slurm --slurm-version 25.11
```
### Install Pre-built from Buildcache (Fastest!)
```bash
# Install Spack
git clone --depth 1 --branch v1.0.0 https://github.com/spack/spack.git
source spack/share/spack/setup-env.sh
# Configure mirrors for buildcache
SLURM_VERSION=25.11
TOOLCHAIN=noble # or: jammy, resolute, rockylinux9, rockylinux10, rockylinux8
CLOUDFRONT_URL=https://slurm-factory-spack-binary-cache.vantagecompute.ai
spack mirror add slurm-factory-deps "${CLOUDFRONT_URL}/${TOOLCHAIN}/slurm/deps/"
spack mirror add slurm-factory-slurm "${CLOUDFRONT_URL}/${TOOLCHAIN}/slurm/${SLURM_VERSION}/"
# Import GPG signing keys and trust
spack buildcache keys --install --trust
# Install signed package (5-15 minutes!)
spack install slurm@${SLURM_VERSION} target=x86_64_v3
# Deploy
spack load slurm@${SLURM_VERSION}
```
### Download Pre-built Tarball (Alternative)
Download complete Slurm installation as a tarball with GPG signature verification:
```bash
# Set versions
SLURM_VERSION=25.11
TOOLCHAIN=noble # or: jammy, rockylinux9, etc.
CLOUDFRONT_URL=https://slurm-factory-spack-binary-cache.vantagecompute.ai
# Download tarball and signature
wget "${CLOUDFRONT_URL}/builds/${SLURM_VERSION}/${TOOLCHAIN}/slurm-${SLURM_VERSION}-${TOOLCHAIN}-software.tar.gz"
wget "${CLOUDFRONT_URL}/builds/${SLURM_VERSION}/${TOOLCHAIN}/slurm-${SLURM_VERSION}-${TOOLCHAIN}-software.tar.gz.asc"
# Import GPG key
gpg --keyserver keyserver.ubuntu.com --recv-keys DFB92630BCA5AB71
# Verify signature
gpg --verify slurm-${SLURM_VERSION}-${TOOLCHAIN}-software.tar.gz.asc \
slurm-${SLURM_VERSION}-${TOOLCHAIN}-software.tar.gz
# Extract and install
sudo tar -xzf slurm-${SLURM_VERSION}-${TOOLCHAIN}-software.tar.gz -C /opt/
cd /opt && sudo ./data/slurm_assets/slurm_install.sh --full-init
```
## Package Structure
All builds produce GPG-signed relocatable tarballs:
```text
slurm-25.11-noble-software.tar.gz
├── view/ # Slurm binaries & libraries
├── modules/slurm/25.11.lua # Lmod module (relocatable)
└── data/slurm_assets/ # Config templates & install script
```
## Deployment
```bash
# Extract (from tarball build)
sudo tar -xzf slurm-25.11-noble-software.tar.gz -C /opt/
# Install (creates users, configs, services)
cd /opt && sudo ./data/slurm_assets/slurm_install.sh --full-init
# Use
module load slurm/25.11
```
## Buildcache Structure
The public buildcache is organized by OS toolchain for optimal performance:
```text
https://slurm-factory-spack-binary-cache.vantagecompute.ai/
├── /
│ └── slurm/
│ ├── deps/ # Slurm dependencies (GPG-signed)
│ ├── 25.11/ # Slurm 25.11 packages (GPG-signed)
│ ├── 24.11/ # Slurm 24.11 packages (GPG-signed)
│ └── 23.11/ # Slurm 23.11 packages (GPG-signed)
├── noble/ # Ubuntu 24.04
│ └── slurm/
│ ├── deps/
│ ├── 25.11/
│ ├── 24.11/
│ └── 23.11/
├── jammy/ # Ubuntu 22.04
├── resolute/ # Ubuntu 25.04
├── rockylinux10/ # Rocky Linux 10
├── rockylinux9/ # Rocky Linux 9
├── rockylinux8/ # Rocky Linux 8
└── builds/
└── /
└── /
├── slurm---software.tar.gz # Complete tarball
└── slurm---software.tar.gz.asc # GPG signature
```
```
### Mirror Architecture
The buildcache uses a **toolchain-first structure** organized by OS for efficient caching:
1. **`/slurm/deps/`** - Slurm runtime dependencies (OpenMPI, PMIx, Munge, etc.)
2. **`/slurm//`** - Slurm packages for each version
This separation allows:
- ✅ **Better caching** - Dependencies shared across Slurm versions within a toolchain
- ✅ **OS compatibility** - Packages built with native system compilers
- ✅ **Reduced storage** - No duplication of common packages per toolchain
- ✅ **Parallel downloads** - Spack can fetch from multiple mirrors simultaneously
## Requirements
- Python 3.12+
- Docker 24.0+ (for building from source)
- 50GB disk space
- 4+ CPU cores (8+ recommended)
- 16GB RAM (32GB+ recommended)
For buildcache installs: only Spack required (no Docker needed)
## Documentation
**[vantagecompute.github.io/slurm-factory](https://vantagecompute.github.io/slurm-factory)**
- [Overview](https://vantagecompute.github.io/slurm-factory/overview) - Architecture and features
- [Installation](https://vantagecompute.github.io/slurm-factory/installation) - Setup and quick start
- [Examples](https://vantagecompute.github.io/slurm-factory/examples) - Common use cases
- [Buildcache Guide](https://vantagecompute.github.io/slurm-factory/slurm-factory-spack-build-cache) - GPG signing and distribution
- [API Reference](https://vantagecompute.github.io/slurm-factory/api-reference) - Python API
## Development
```bash
# Install from source
git clone https://github.com/vantagecompute/slurm-factory.git
cd slurm-factory
pip install -e .
# Run tests
pytest
# Build docs
cd docusaurus && npm run build
```
## License
Apache License 2.0 - See [LICENSE](LICENSE) file.
## Support
- Issues: [github.com/vantagecompute/slurm-factory/issues](https://github.com/vantagecompute/slurm-factory/issues)
- Docs: [vantagecompute.github.io/slurm-factory](https://vantagecompute.github.io/slurm-factory)
- Website: [vantagecompute.ai](https://www.vantagecompute.ai)
---
Built with ❤️ by [Vantage Compute](https://www.vantagecompute.ai)