Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
awesome-institutional-hpc
List of tools you can use to get the most from your institutional HPC resources
https://github.com/frankier/awesome-institutional-hpc
Last synced: 3 days ago
JSON representation
-
Tools provided for you
-
Workflow managers and distributed computing frameworks
- this tutorial
- Ray - Python-based distributed computing framework including tools for hyperparamter search
- Nextflow
- its built-in SLURM executor
- toil - 3.6.x/batchSystem.html)
- Snakemake - File based processing pipeline framework with workflows written on Python but additional support for CLI programs, Julia and R with SLURM support through either:
- SLURM Snakemake profile
- smk-simple-slurm
- singslurm2
- Guide at Software Carpentries incubator
- bpipe
-
Fancy shell sessions
- mosh - Drop free, low latency faked SSH sessions
- kitty - Terminal supporting image display
- EternalTerminal - Another approach to drop free SSH sessions
- xxh - Bring your own shell and config to shared computing environments
- Euporie - Jupyter notebooks in the terminal including support for image display with kitty
- tmux - Terminal multiplexers, which allow running multiple sessions and detatching and reattaching to terminal sessions
-
Containers
- Singularity - Container runtime which must be installed by your cluster administrator. It can run convert Docker images.
- udocker - Docker compatible CLI, which does not require action from your cluster administrator, but can use Singularity as a backend if it's available.
- singreqrun - Shim to allow running programs from the host from within a Singularity container
-
SLURM wrappers and libraries
- yaspi - A Python-based SLURM wrapper based on job arrays
- SEML - SEML: Slurm Experiment Management Library. Integration of the sacred experiment manager and SLURM.
- pyslurm - Python library for interfacing with SLURM
- slurmpy - Small Python library for starting SLURM jobs
- stubl - Utilities for SLURM including wrappers for job launching and reporting
-
Networking
- sshuttle - This program can help if you need to connect from outside of campus and the VPN doesn't get you where you want.
- lsyncd - A program to sync your files to a remote server, e.g. a HPC login node. This allows a workflow where you can develop and run smoke tests locally, and then seemlessly switch to HPC for full runs.
-
Guides
Categories
Sub Categories
Keywords
slurm
3
snakemake
3
python
2
ssh
2
terminal
2
workflow-management
2
hpc
2
batch
1
cython
1
cluster
1
chroot
1
containers
1
utility
1
slurm-workload-manager
1
orchestration
1
hyperparameter-optimization
1
experiment-tracking
1
experiment-manager
1
lesson
1
english
1
carpentries-incubator
1
alpha
1
snakemake-profile
1
deep-hybrid-datacloud
1
bioinformatics
1
docker
1
docker-containers
1
reproducibility
1
user
1
runc
1
root-privileges
1
proot
1
indigo
1
emulation
1
grid
1
fakechroot
1
eosc-hub
1
powerlevel9k
1
shell
1
xonsh
1
xxh
1
zsh
1
euporie
1
ipynb
1
ipython
1
ipython-notebook
1
jupyter
1
jupyter-notebook
1
kitty
1
prompt-toolkit
1