Projects in Awesome Lists tagged with slurm
A curated list of projects in awesome lists tagged with slurm .
https://github.com/stas00/ml-engineering
Machine Learning Engineering Open Book
ai inference large-language-models llm machine-learning machine-learning-engineering mlops pytorch scalability slurm training transformers
Last synced: 14 May 2025
https://github.com/schedmd/slurm
Slurm: A Highly Scalable Workload Manager
slurm slurm-job-scheduler slurm-workload-manager
Last synced: 13 May 2025
https://github.com/nextflow-io/nextflow
A DSL for data-driven computational pipelines
aws bioinformatics cloud dataflow docker groovy hello hpc nextflow pipeline pipeline-framework reproducible-research reproducible-science sge singularity singularity-containers slurm workflow-engine
Last synced: 13 May 2025
https://github.com/SchedMD/slurm
Slurm: A Highly Scalable Workload Manager
slurm slurm-job-scheduler slurm-workload-manager
Last synced: 26 Mar 2025
https://github.com/dstackai/dstack
dstack is an open-source alternative to Kubernetes and Slurm, designed to simplify GPU allocation and AI workload orchestration for ML teams across top clouds, on-prem clusters, and accelerators.
amd aws azure cloud docker fine-tuning gcp gpu inference k8s kubernetes llms machine-learning nvidia orchestration python slurm training
Last synced: 23 Oct 2025
https://github.com/facebookincubator/submitit
Python 3.8+ toolbox for submitting jobs to Slurm
Last synced: 14 May 2025
https://github.com/databiosphere/toil
A scalable, efficient, cross-platform (Linux/macOS) and easy-to-use workflow engine in pure Python.
aws common-workflow-language cwl gridengine kubernetes mesos pipeline python slurm wdl workflow workflow-description-language
Last synced: 29 Apr 2025
https://github.com/DataBiosphere/toil
A scalable, efficient, cross-platform (Linux/macOS) and easy-to-use workflow engine in pure Python.
aws common-workflow-language cwl gridengine kubernetes mesos pipeline python slurm wdl workflow workflow-description-language
Last synced: 07 Apr 2025
https://github.com/lambdalabsml/distributed-training-guide
Best practices & guides on how to write distributed pytorch training code
cluster cuda deepspeed distributed-training fsdp gpu gpu-cluster kuberentes lambdalabs mpi nccl pytorch sharding slurm
Last synced: 16 May 2025
https://github.com/pipefunc/pipefunc
Lightweight fast function pipeline (DAG) creation in pure Python for scientific workflows 🕸️🧪
dag hpc parallel-computing pipeline-framework pipelines reproducible-research slurm workflow-engine
Last synced: 16 Dec 2025
https://github.com/LambdaLabsML/distributed-training-guide
Best practices & guides on how to write distributed pytorch training code
cluster cuda deepspeed distributed-training fsdp gpu gpu-cluster kuberentes lambdalabs mpi nccl pytorch sharding slurm
Last synced: 08 Mar 2025
https://github.com/giovtorres/slurm-docker-cluster
A Slurm cluster using docker-compose
docker-compose hpc slurm slurm-cluster
Last synced: 07 Apr 2025
https://github.com/pytorch/torchx
TorchX is a universal job launcher for PyTorch applications. TorchX is designed to have fast iteration time for training/research and support for E2E production ML pipelines when you're ready.
airflow aws-batch components deep-learning distributed-training kubernetes machine-learning pipelines python pytorch ray slurm
Last synced: 14 May 2025
https://github.com/elasticluster/elasticluster
Create clusters of VMs on the cloud and configure them with Ansible.
ansible azure cloud cluster clustering ec2 gcp gridengine hadoop hpc python slurm spark
Last synced: 07 Apr 2025
https://github.com/rackslab/Slurm-web
Open source web dashboard for Slurm HPC clusters
Last synced: 06 Mar 2025
https://github.com/rackslab/slurm-web
Open source web dashboard for Slurm HPC clusters
Last synced: 07 Apr 2025
https://github.com/justanhduc/task-spooler
A scheduler for GPU/CPU tasks
c cpp debian gpu-support job-scheduler linux makefile slurm slurm-job slurm-job-scheduler task-spooler
Last synced: 06 Apr 2025
https://github.com/Azure/batch-shipyard
Simplify HPC and Batch workloads on Azure
azure azure-batch azure-functions batch-processing containers docker glusterfs gpu hpc infiniband mpi nfs rdma serverless singularity slurm windows-containers
Last synced: 29 Jul 2025
https://github.com/azure/batch-shipyard
Simplify HPC and Batch workloads on Azure
azure azure-batch azure-functions batch-processing containers docker glusterfs gpu hpc infiniband mpi nfs rdma serverless singularity slurm windows-containers
Last synced: 07 Oct 2025
https://github.com/dell/omnia
An open-source toolkit for deploying and managing high performance clusters for HPC, AI, and data analytics workloads.
ansible ansible-playbooks dell-emc dellemc hpc hpc-clusters k8s-cluster kubernetes slurm slurm-cluster
Last synced: 15 May 2025
https://github.com/zhenrong-wang/hpc-now
A Cross-Platform, Multi-Cloud High-Performance Computing Platform
aliyun aws azure baiduyun c cloud cluster devops google-cloud hpc huaweicloud linux opentofu scripts slurm tencent-cloud terraform
Last synced: 02 Apr 2025
https://github.com/TUM-DAML/seml
SEML: Slurm Experiment Management Library
experiment-manager experiment-tracking hyperparameter-optimization orchestration slurm slurm-workload-manager utility
Last synced: 08 May 2025
https://github.com/jdblischak/smk-simple-slurm
A simple Snakemake profile for Slurm without --cluster-config
bioinformatics slurm snakemake snakemake-profile
Last synced: 06 May 2025
https://github.com/mschubert/clustermq
R package to send function calls as jobs on LSF, SGE, Slurm, PBS/Torque, or each via SSH
cluster high-performance-computing lsf r-package sge slurm ssh
Last synced: 15 May 2025
https://github.com/gdikov/hypertunity
A toolset for black-box hyperparameter optimisation.
bayesian-optimization gpyopt hyperparameter-optimization slurm tensorboard
Last synced: 18 Apr 2025
https://github.com/NREL/HPC
A collection of various resources, examples, and executables for the general NREL HPC user community's benefit. Use the following website for accessing documentation.
computing energy high hpc lab laboratory national nrel performance renewable slurm training
Last synced: 27 Mar 2025
https://github.com/ohsu-comp-bio/funnel
Funnel is a toolkit for distributed task execution via a simple, standard API.
aws-batch docker ga4gh google-cloud gridengine kubernetes pbs-torque slurm
Last synced: 06 Apr 2025
https://github.com/neilmunday/slurm-mail
Slurm-Mail is a drop in replacement for Slurm's e-mails to give users much more information about their jobs compared to the standard Slurm e-mails.
email email-template python slurm
Last synced: 23 Apr 2025
https://github.com/mil-ad/stui
A Slurm dashboard for the terminal.
python slurm slurm-cluster slurm-utility terminal-based tui urwid
Last synced: 11 Dec 2025
https://github.com/futureverse/future.batchtools
:rocket: R package future.batchtools: A Future API for Parallel and Distributed Processing using batchtools
distributed-computing hpc job-scheduler package parallel pbs r sge slurm torque
Last synced: 12 Dec 2025
https://github.com/aws-samples/aws-hpc-recipes
Contains example recipes that demonstrate how to build HPC systems using AWS services and solutions.
aws batch cloudformation fsx-lustre parallelcluster pcs res slurm terraform
Last synced: 02 Apr 2025
https://github.com/mbhall88/ssubmit
Submit slurm sbatch jobs without a script
Last synced: 05 May 2025
https://github.com/juliaparallel/slurmclustermanager.jl
Julia package for running code on Slurm clusters
distributed-computing julia slurm
Last synced: 09 Sep 2025
https://github.com/USCbiostats/slurmR
slurmR: A Lightweight Wrapper for Slurm
bioinformatics hpc rpackage rstats slurm
Last synced: 30 Jul 2025
https://github.com/uscbiostats/slurmr
slurmR: A Lightweight Wrapper for Slurm
bioinformatics hpc rpackage rstats slurm
Last synced: 28 Apr 2025
https://github.com/JuliaParallel/SlurmClusterManager.jl
Julia package for running code on Slurm clusters
distributed-computing julia slurm
Last synced: 31 Mar 2025
https://github.com/fgci-org/fgci-ansible
:microscope: Collection of the Finnish Grid and Cloud Infrastructure Ansible playbooks
ansible ansible-roles cluster cscfi grid hpc provisioning slurm wlcg
Last synced: 10 Apr 2025
https://github.com/cihga39871/jobschedulers.jl
A Julia-based job scheduler and workload manager inspired by Slurm, PBS and Crontab.
cron crontab job-queue job-scheduler julia pbs pipeline pipelines queue scheduled-tasks scheduler slurm task-management task-manager task-queue task-scheduler workflow workload workload-managers
Last synced: 01 Sep 2025
https://github.com/jacopopan/a-minimalist-guide
Walkthroughs for DSL, AirSim, the Vector Institute, and more
airsim anaconda brax dji-tello-talent mujoco nvidia ray rllib robomaster-s1 robomaster-sdk slurm tensorflow torch tutorials ubuntu unreal-engine-4
Last synced: 27 Apr 2025
https://github.com/deepmodeling/dpdispatcher
generate HPC scheduler systems jobs input scripts and submit these scripts to HPC systems and poke until they finish
hpc job-scheduler lsf pbs python slurm
Last synced: 16 May 2025
https://github.com/Ensembl/ensembl-hive
EnsEMBL Hive - a system for creating and running pipelines on a distributed compute resource
docker docker-swarm ehive ensembl high-performance-computing htcondor java lsf mysql pbs-pro pbspro perl pipeline postgresql python sge slurm sqlite workflow-management-system
Last synced: 03 Aug 2025
https://github.com/bsc-wdc/compss
COMP Superscalar (COMPSs) is a framework which aims to ease the development and execution of applications for distributed infrastructures, such as Clusters, Grids and Clouds.
c distributed-computing docker hpc java pipeline-framework python singularity slurm workflow-management-system workflows
Last synced: 05 Apr 2025
https://github.com/natefoo/slurm-drmaa
DRMAA for Slurm: Implementation of the DRMAA C bindings for Slurm
cluster clusters distributed-computing drmaa hpc resource-management slurm slurm-job-scheduler
Last synced: 05 Apr 2025
https://github.com/clip-hpc/goslmailer
GoSlurmMailer - drop in replacement for default slurm MailProg. Delivers slurm job messages to various destinations.
discord hpc mail matrix mattermost msteams notifications slack slurm telegram
Last synced: 12 Oct 2025
https://github.com/ploomber/soopervisor
☁️ Export Ploomber pipelines to Kubernetes (Argo), Airflow, AWS Batch, SLURM, and Kubeflow.
airflow argo argo-workflows aws data-science kubeflow kubeflow-pipelines kubernetes machine-learning slurm workflow
Last synced: 21 Aug 2025
https://github.com/vultr/slik
Slurm in Kubernetes
kubernetes slurm slurm-operator
Last synced: 10 Aug 2025
https://github.com/CLIP-HPC/goslmailer
GoSlurmMailer - drop in replacement for default slurm MailProg. Delivers slurm job messages to various destinations.
discord hpc mail matrix mattermost msteams notifications slack slurm telegram
Last synced: 07 May 2025
https://github.com/ceems-dev/ceems
A Prometheus exporter and a REST API server to export metrics of compute units of resource managers like SLURM, Openstack, k8s, _etc_
cloud containers dashboards ebpf emissions energy-monitor grafana green-computing hpc json-api kubernetes metrics-server metrics-visualization monitoring observability openstack performance-monitoring prometheus prometheus-exporter slurm
Last synced: 23 Jul 2025
https://github.com/MHH-RCUG/Wochenende
Deprecated see https://github.com/MHH-RCUG/nf_wochenende : A whole Genome/Metagenome Sequencing Alignment Pipeline in Python3
alignment bioinformatics conda-environment genomics metagenomics nanopore pipeline slurm
Last synced: 19 Nov 2025
https://github.com/slinkyproject/slurm-bridge
Run Slurm as a Kubernetes scheduler. A Slinky project.
hpc kubernetes scheduler slinky slurm
Last synced: 07 Oct 2025
https://github.com/psteinb/hpc-in-a-day
a full day lesson material to teach the basics of using a HPC cluster to novices
carpentries hpc lsf mpi parallel pbs python slurm training
Last synced: 12 Apr 2025
https://github.com/akkornel/mpi4py
Example of using MPI in Python with mpi4py (and SLURM!)
Last synced: 01 Jul 2025
https://github.com/basnijholt/adaptive-scheduler
Run many functions (adaptively) on many cores (>10k-100k) using mpi4py.futures, ipyparallel, loky, or dask-mpi. :tada:
active-learning adaptive adaptive-learning dask distributed-computing interactive ipyparallel loky mpi4py parallel-computing pbs python slurm
Last synced: 06 Apr 2025
https://github.com/francois-rozet/dawgz
Unleash the true power of scheduling
directed-acyclic-graph hpc python reproducible-science scheduling slurm workflow
Last synced: 16 Sep 2025
https://github.com/pyiron/pysqa
Simple HPC queuing system adapter for Python on based jinja templates to automate the submission script creation.
hpc lsf moab python queue-manager sge slurm torque
Last synced: 21 Oct 2025
https://github.com/agnostiqhq/covalent-slurm-plugin
Executor plugin interfacing Covalent with Slurm
covalent data-pipeline etl hpc hpc-applications machinelearning machinelearning-python parallelization pipelines python python3 quantum-computing quantum-machine-learning slurm workflow workflow-automation
Last synced: 09 Oct 2025
https://github.com/saforem2/ezpz
Train across all your devices, ezpz 🍋
deepspeed distributed-training launcher machine-learning mpi mpi4py parallelism python pytorch rich slurm
Last synced: 27 Dec 2025
https://github.com/vsoch/ood-compose
Docker compose to bring up Open OnDemand with SLURM, Centos 7
docker-compose on-demand open-on-demand slurm
Last synced: 06 Oct 2025
https://github.com/daylily-informatics/daylily
A NGS analysis framework for WGS data, which automates the entire process of spinning up AWS EC2 spot instances and processing FASTQ to snvVCF in <60m, for dollars a sample and achieving Fscores of 0.998.
aws bioinformatics bioinformatics-pipeline budgeting cwl ec2 ephemeral genomic-data-analysis giab human-genome informatics ngs parallel-cluster scalable scaling slurm snakemake snv-call wgs
Last synced: 19 Jun 2025
https://github.com/mikedacre/fyrd
Submit functions and shell scripts to torque and slurm clusters or local machines using python.
bioinformatics-pipeline library python python2 python3 python3-library slurm slurm-cluster torque
Last synced: 30 Jun 2025
https://github.com/schlosslab/great_lakes_slurm
Using the Great Lakes cluster and batch computing with SLURM
Last synced: 30 Oct 2025
https://github.com/ulhpc/puppet-slurm
A Puppet module designed to configure and manage SLURM(see https://slurm.schedmd.com/), an open source, fault-tolerant, and highly scalable cluster management and job scheduling system for large and small Linux clusters
job-scheduler munge puppet slurm
Last synced: 05 Apr 2025
https://github.com/princetonuniversity/mousemotionmapper
Matlab pipeline for semi-supervised mouse behavioral classification
k-means-clustering matlab slurm tsne wavelet-transform
Last synced: 10 Sep 2025
https://github.com/ck37/ck37r
R functions for project setup, data cleaning, machine learning, SuperLearner, parallelization, and targeted learning.
h2oai parallelization slurm superlearner targeted-learning tmle
Last synced: 13 Jul 2025
https://github.com/JoeriHermans/awflow
Reproducible research and reusable acyclic workflows in Python. Execute code on HPC systems as if you executed them on your personal computer!
automation hpc hpc-tools python reproducible-research reproducible-science slurm workflow workflow-engine
Last synced: 08 May 2025
https://github.com/joerihermans/awflow
Reproducible research and reusable acyclic workflows in Python. Execute code on HPC systems as if you executed them on your personal computer!
automation hpc hpc-tools python reproducible-research reproducible-science slurm workflow workflow-engine
Last synced: 21 Sep 2025
https://github.com/cea-list/rpcdataloader
A variant of the PyTorch Dataloader using remote workers.
data-science dataloader distributed-computing hpc machine-learning preprocessing pytorch slurm
Last synced: 21 Jun 2025
https://github.com/silx-kit/jupyterhub_moss
Jupyterhub MOdular Slurm Spawner
Last synced: 27 Jul 2025
https://github.com/dirkpetersen/froster
Froster is a user-friendly archiving tool for teams that move data between Posix file systems and S3 like object storage systems such as AWS Glacier
archiving aws boto3 cli duckdb glacier hpc metadata petabyte pwalk python rclone s3 slurm storage tui
Last synced: 21 Sep 2025
https://github.com/link89/oh-my-batch
A toolkit to manipulate batch tasks with command line. Designed for scientific computing community.
bash batch command-line-tool hpc job-scheduler pipeline python3 scientific-computing shell slurm workflow
Last synced: 14 Apr 2025
https://github.com/ulhpc/launcher-scripts
(DEPRECATED) A set of launcher scripts to be used with OAR and Slurm for running jobs on the UL HPC platform
Last synced: 14 Apr 2025
https://github.com/zaccharieramzi/submission-scripts
All the submission scripts used for my work on Jean Zay and the TGCC
dask-jobqueue hpc slurm slurm-utility submitit
Last synced: 12 Jun 2025
https://github.com/epigen/cemm.slurm.sm
CeMM's Snakemake SLURM cluster profile
cluster-profile slurm snakemake
Last synced: 15 Apr 2025
https://github.com/goerz/clusterjob
Manage traditional HPC cluster workflows in Python
Last synced: 28 Oct 2025
https://github.com/mahendrapaipuri/ceems
A Prometheus exporter and a REST API server to export metrics of compute units of resource managers like SLURM, Openstack, k8s, _etc_
cloud containers dashboards ebpf emissions energy-monitor grafana green-computing hpc json-api kubernetes metrics-server metrics-visualization monitoring observability openstack performance-monitoring prometheus prometheus-exporter slurm
Last synced: 30 Mar 2025
https://github.com/pc2/slurm_jupyter_kernel
Manage (create, list, modify and delete) and starting jupyter kernels using sbatch
hpc ijulia ipython jupyter jupyter-kernel jupyterhub jupyterlab slurm slurm-jupyter-kernel srun
Last synced: 11 Oct 2025
https://github.com/patonlab/paton_group_workflows
Python Code, shell scripts, templates, submission scripts and compchem specific workflows for use in the Paton Lab
bash monitoring python slurm submission
Last synced: 27 Jul 2025
https://github.com/tdegeus/gooseslurm
SLURM command line tools and scripts
batch-script examples slurm wrapper
Last synced: 30 Apr 2025
https://github.com/protortyp/melon
A lightweight work manager.
distributed-systems job-scheduler rust slurm
Last synced: 05 Sep 2025
https://github.com/lululxvi/sumsjob
A simple Linux command-line utility which submits a job to one of the multiple GPU servers
command-line gpu resource-manager server slurm submitter workload-management workstation
Last synced: 04 Aug 2025
https://github.com/cwatson/mri_library
Scripts for MRI preprocessing
brain diffusion-mri dti dwi hpc-applications mri neuroimaging pipeline slurm tractography
Last synced: 02 May 2025
https://github.com/hill/lazyslurm
like lazygit/lazydocker but for slurm
cli hpc lazydocker lazygit slurm terminal
Last synced: 08 Oct 2025
https://github.com/justanhduc/messenger
A plugin that enables controlling Task Spoolers from multiple servers remotely
gpu slurm task-manager task-scheduler task-spooler
Last synced: 13 Apr 2025
https://github.com/ilri/hpc-infrastructure-scripts
Scripts used in ILRI's research computing infrastructure
Last synced: 10 Sep 2025
https://github.com/uscbiostats/slurmr-workshop
Workshop on HPC with Slurm, R, and the slurmR package
hpc rprogramming rstats slurm workshop
Last synced: 28 Apr 2025
https://github.com/henrikbengtsson/future.batchjobs
:rocket: R package: future.BatchJobs: A Future API for Parallel and Distributed Processing using BatchJobs [Intentionally archived on CRAN on 2021-01-08]
distributed-computing hpc job-scheduler package parallel parallel-computing pbs r sge slurm torque
Last synced: 10 Apr 2025
