Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

awesome-hpc

A collection of Awesome HPC software and tools
https://github.com/dstdev/awesome-hpc

Last synced: 1 day ago
JSON representation

  • Provisioning

    • Grendel - Bare Metal Provisioning system for HPC Linux clusters ([Source Code](https://github.com`ubccr/grendel)) `GPL-3`.
    • XCat - xCAT is a toolkit for deployment and administration of clusters of all sizes ([Source Code](https://github.com/xcat2/xcat-core)) `EPL-1.0`.
    • Warewulf - Warewulf is a stateless and diskless container operating system provisioning system for large clusters of bare metal and/or virtual systems ([Source Code](https://github.com/hpcng/warewulf)) `BSD-3`.
    • Rocks - A Linux distribution for developing Linux clusters `other`.
    • Cobbler - Cobbler is a Linux installation server that allows for rapid setup of network installation environments ([Source Code](https://github.com/cobbler/cobbler)) `GPL-2.0`.
    • Bright - Bright Cluster Manager allows administrator to quickly build and manage heterogeneous clusters `Proprietary`.
    • BlueBanquise - BlueBanquise is an open source cluster deployment and management stack built on Python and Ansible ([Source Code](https://github.com/bluebanquise/bluebanquise)) `MIT`.
    • Grendel - Bare Metal Provisioning system for HPC Linux clusters ([Source Code](https://github.com`ubccr/grendel)) `GPL-3`.
    • Bright - Bright Cluster Manager allows administrator to quickly build and manage heterogeneous clusters `Proprietary`.
    • BlueBanquise - BlueBanquise is an open source cluster deployment and management stack built on Python and Ansible ([Source Code](https://github.com/bluebanquise/bluebanquise)) `MIT`.
    • Base Command Manager - Base Command Manager allows administrator to quickly build and manage heterogeneous clusters `Proprietary`.
    • Base Command Manager - Base Command Manager allows administrator to quickly build and manage heterogeneous clusters `Proprietary`.

  • Workload Managers

    • Slurm - A free and open source job scheduler ([Source Code](https://github.com/SchedMD/slurm)) `OSS`.
    • LSF - A job scheduler and workload management software developed by IBM `Proprietary`.
    • Moab - Moab is a workload management and job scheduler `other`.
    • Torque - Torque is a workload management and job scheduler `other`.
    • OpenLava - OpenLava is a workload management and job scheduler `other`.
    • UGE/SGE - Univa Grid Engine is a workload management engine for HPC `Proprietary`.
    • Volcano - Volcano is a batch system built on Kubernetes `Apache-2.0`.
    • Maui - Maui is a workload management and job scheduler `other`.
    • Kube Batch - A batch scheduler of kubernetes for high performance workload, e.g. AI/ML, BigData, HPC `Apache-2.0`.
    • OpenPBS - OpenPBS® software optimizes job scheduling and workload management in high-performance computing (HPC) environments ([Source Code](https://github.com/openpbs/openpbs)) `other`.

  • Pipelines

    • Nextflow - Data drive computational pipelines `Apache-2.0`.
    • Cromwell - Scientific workflow engine designed for simplicity & scalability ([Source Code](https://github.com/broadinstitute/cromwell)) `BSD-3`.
    • Nextflow - Data drive computational pipelines `Apache-2.0`.
    • Pegasus - A configurable system for mapping and executing scientific workflows over a wide range of computational infrastructure ([Source Code](https://github.com/pegasus-isi/pegasus))`Apache-2.0`.

  • Applications

    • Spack - A flexible package manager that supports multiple versions, configurations, platforms, and compilers ([Source Code](https://github.com/spack/spack)) `other`.
    • EasyBuild - EasyBuild - building software with ease ([Source Code](https://github.com/easybuilders/easybuild)) `GPL-2`.
    • Spack - A flexible package manager that supports multiple versions, configurations, platforms, and compilers ([Source Code](https://github.com/spack/spack)) `other`.

  • Compilers

    • Nvidia - NVIDIA HPC compiler suite for Fortran, C/C++ with OpenACC `Proprietary`.
    • Portland Group - The Portland Group compilers were Fortran, C/C++ compilers now integrated into NVIDIA HPC SDK `Proprietary`.
    • Intel - The Intel compiler suite offers many language compilers for use in the HPC space `Proprietary`.
    • Cray - A suite of compilers designed and optimized to target the AMD interlagos instruction set `Proprietary`.
    • GNU - The GNU Compiler Collection is a suite of compilers targeting many languages ([Source Code](https://gcc.gnu.org/git.html)) `GPL-3`.
    • LLVM - The LLVM project is a collection of modular compilers and toolchains ([Source Code](https://github.com/llvm/llvm-project)) `OSS`.
    • Portland Group - The Portland Group compilers were Fortran, C/C++ compilers now integrated into NVIDIA HPC SDK `Proprietary`.

  • MPI

    • OpenMPI - OpenMPI is an open source implementation of the MPI-3.1 standard ([Source Code](https://github.com/open-mpi/ompi)) `BSD`.
    • MPICH - MPICH is a high-performance and widely portable implementation of the MPI-3.1 standard ([Source Code](https://github.com/pmodels/mpich)) `other`.
    • MVAPICH - MVAPICH is an open source implementation of the MPI-3.1 standard developed by Ohio State University `BSD`.
    • Intel-MPI - Intel-MPI is Intel's MPI-3.1 implementation included in their compiler suite `other`.

  • Parallel Computing

    • ArrayFire - A general purpose tensor library that simplifies the process of software development for parallel architectures `other`.
    • OpenMP - OpenMP is an application programming interface that supports multi-platform shared-memory multiprocessing programming `other`.

  • Benchmarking

    • OSU Benchmarks - A collection of benchmarking tools for MPI developed by Ohio State University `other`.
    • Intel MPI Benchmarks - A set of benchmarks developed by Intel for use with their Intel MPI `other`.
    • HPCC Systems - HPCC Systems (High Performance Computing Cluster) is an open source, massive parallel-processing computing platform for big data processing and analytics ([Source Code](https://github.com/hpcc-systems/HPCC-Platform)) `other`.
    • LINPACK - LINPACK is a set of efficient fortran subroutines for solving linear systems which benchmarks are useful for HPC `other`.
    • IOzone - IOzone is a filesystem benchmark tool `OSS`.
    • IOR - Interleaved or Random is a useful benchmarking tool for testing parallel filesystems `other`.
    • MDtest - MDtest is an MPI-based application for evaluating the metadata performance of a file system `other`.
    • FIO - Flexible I/O is an advanced disk benchmark that depends upon the kernel's AIO access library ([Source Code](https://git.kernel.dk/cgit/fio/)) `GPL-2`.
    • elbencho - A distributed storage benchmark for files, objects & blocks with support for GPUs `GPL-3`.

  • Miscellaneous

    • OpenOnDemand - Open OnDemand helps computational researchers and students efficiently utilize remote computing resources by making them easy to access from any device ([Source Code](https://github.com/OSC/openondemand.org)) `MIT`.
    • Open XDMod - Open XDMoD is an open source tool to facilitate the management of high performance computing resources ([Source Code](https://github.com/ubccr/xdmod/)) `LGPL-3`.
    • Coldfront - ColdFront is an open source resource allocation system designed to provide a central portal for administration, reporting, and measuring scientific impact of HPC resources ([Source Code](https://github.com/ubccr/coldfront)) `GPL-3`.
    • Pavilion2 - Pavilion is a Python 3 (3.6+) based framework for running and analyzing tests targeting HPC systems ([Source Code](https://github.com/hpc/pavilion2)) `other`.
    • Open XDMod - Open XDMoD is an open source tool to facilitate the management of high performance computing resources ([Source Code](https://github.com/ubccr/xdmod/)) `LGPL-3`.
    • Pavilion2 - Pavilion is a Python 3 (3.6+) based framework for running and analyzing tests targeting HPC systems ([Source Code](https://github.com/hpc/pavilion2)) `other`.
    • OLCF Test Harness - The OLCF Test Harness (OTH) helps automate the testing of applications, tools, and other system software ([Source Code](https://github.com/olcf/olcf-test-harness)) `other`.
    • GoSlmailer - Goslmailer is a drop-in notification delivery solution for slurm that can do slack, mattermost, teams, and more.
    • Reframe - A powerful Python framework for writing and running portable regression tests and benchmarks for HPC systems. ([Source Code](https://github.com/reframe-hpc/reframe)) `BSD-3`.

  • Performance

    • TotalView - TotalView is a debugging tool for HPC applications `Proprietary`.
    • Valgrind - Valgrind is a tool designed to profile programs to determine memory leaks ([Source Code](https://sourceware.org/git/?p=valgrind.git)) `GPL-2`.
    • Paraver - Paraver is a very flexible data browser that is part of the CEPBA-Tools toolkit `other`.
    • PAPI - Performance Application Programming Interface (PAPI) is a performance analysis tool ([Source Code](https://bitbucket.org/icl/papi/src/master/)) `other`.
    • Valgrind - Valgrind is a tool designed to profile programs to determine memory leaks ([Source Code](https://sourceware.org/git/?p=valgrind.git)) `GPL-2`.
    • PAPI - Performance Application Programming Interface (PAPI) is a performance analysis tool ([Source Code](https://bitbucket.org/icl/papi/src/master/)) `other`.
    • Tau - TAU Performance System® is a portable profiling and tracing toolkit for performance analysis of parallel programs written in Fortran, C, C++, UPC, Java, Python `other`.

  • Parallel Shells

    • pdsh - pdsh runs terminal commands across multiple hosts in parallel ([Source Code](https://github.com/chaos/pdsh)) `GPL-2`.
    • ClusterShell - Scalable cluster administration Python framework ([Source Code](https://github.com/cea-hpc/clustershell)) `LGPL-2.1` .

  • Containers

    • Apptainer - Apptainer is an open source container system ([Source Code](https://github.com/apptainer/apptainer)) `BSD`.
    • Docker - Docker is a set of platform as a service products that use OS-level virtualization to deliver software in packages called containers `other`.
    • Shifter - Shifter is Linux containers for HPC ([Source Code](https://github.com/NERSC/shifter)) `other`.
    • Singularity HPC - Singularity Registry HPC (shpc) allows you to install containers as modules ([Source Code](https://github.com/singularityhub/singularity-hpc)) `MPL 2.0`.
    • Apptainer - Apptainer is an open source container system ([Source Code](https://github.com/apptainer/apptainer)) `BSD`.
    • Singularity HPC - Singularity Registry HPC (shpc) allows you to install containers as modules ([Source Code](https://github.com/singularityhub/singularity-hpc)) `MPL 2.0`.
    • uDocker - A basic user tool to execute simple docker containers in batch or interactive systems without root privileges ([Source Code](https://github.com/indigo-dc/udocker)) `Apache-2.0`.
    • Charliecloud - Charliecloud provides user-defined software stacks (UDSS) for high-performance computing (HPC) centers ([Source Code](https://github.com/hpc/charliecloud)) `Apache-2.0`.
    • HPC Container Maker - HPC Container Maker is an open source tool to make it easier to generate container specification files. `Apache-2.0`.
    • Scarus - An OCI-compatible container engine for HPC `BSD`.

  • Environment Management

    • Lmod - Lmod: An Environment Module System based on Lua, Reads TCL Modules, Supports a Software Hierarchy ([Source Code](https://github.com/TACC/Lmod)) `other`.
    • Environment Modules - Environment Modules: provides dynamic modification of a user's environment ([Source Code](https://github.com/cea-hpc/modules)) `GPL-2`.
    • Anaconda - Anaconda is a Python and R distribution for use in computational science `other`.
    • Mamba - Mamba is a reimplementation of the conda package manager in C++ ([Source Code](https://github.com/mamba-org/mamba)) `BSD`.

  • Visualization

    • Paraview - ParaView is an open-source, multi-platform data analysis and visualization application based on Visualization Toolkit (VTK) ([Source Code](https://github.com/Kitware/ParaView)) `BSD-3`.
    • Visit - VisIt - Visualization and Data Analysis for Mesh-based Scientific Data ([Source Code](https://github.com/visit-dav/visit)) `BSD-3`.

  • Parallel Filesystems

    • GPFS - GPFS is a high-performance clustered file system software developed by IBM `Proprietary`.
    • Quobyte - A high performance filesystem `Proprietary`.
    • Ceph - Ceph is a distributed object, block, and file storage platform ([Source Code](https://github.com/ceph/ceph)) `other`.
    • Weka - A file system designed for HPC `Proprietary` .
    • Lustre/Exascaler - Lustre is an open-source, distributed parallel file system software platform designed for scalability, high-performance, and high-availability ([Source Code](https://git.whamcloud.com/fs/lustre-release.git)) `other`.
    • BeeGFS - BeeGFS is a hardware-independent POSIX parallel file system developed with a strong focus on performance and designed for ease of use, simple installation, and management `Proprietary`.
    • OrangeFS - OrangeFS is a next generation parallel file system for Linux clusters ([Source Code](https://github.com/waltligon/orangefs)) `other`.
    • MooseFS - Moose File System is an Open-source, POSIX-compliant distributed file system developed by Core Technology ([Source Code](https://github.com/moosefs/moosefs)) `GPL-2.0`.
    • Quobyte - A high performance filesystem `Proprietary`.

  • Programming Languages

    • Julia - Julia is a high-level, high-performance dynamic language for technical computing `MIT`.
    • Futhark - Futhark is a purely functional data-parallel programming language in the ML family `isc`.
    • Chapel - Chapel is a programming language designed for productive parallel computing at scale `Apache-2.0`.

  • Journals

  • Blogs

      • The Register HPC - The Register is a leading and trusted global online enterprise technology news publication, reaching roughly 40 million readers worldwide.
      • The Register HPC - The Register is a leading and trusted global online enterprise technology news publication, reaching roughly 40 million readers worldwide.

    • Prometheus Based

      • HPCWire - Since 1987 covering the fastest computers in the world and the people who run them.
      • InsideHPC - insideHPC is a global publication recognized for its comprehensive and insightful coverage of the HPC-AI community, linking vendors, end-users and HPC strategists.
      • The Next Platform - Offers in-depth coverage of high-end computing at large enterprises, supercomputing centers, hyperscale data centers, and public clouds.
      • HPC at Dell - High-Performance Computing knowledge base articles from Dell.

  • Conferences

      • CCGrid - IEEE/ACM International Symposium on Cluster, Cloud and Internet Computing.
      • Hot Chips - Semiconductor industry's leading conference on high-performance microprocessors and related circuits.
      • Hot Interconnects - IEEE conference on software architectures and implementations for interconnection networks of all scales.
      • ESSA - Workshop on Extreme-Scale Storage and Analysis.
      • Hot Chips - Semiconductor industry's leading conference on high-performance microprocessors and related circuits.
      • Hot Interconnects - IEEE conference on software architectures and implementations for interconnection networks of all scales.

    • Prometheus Based

      • IEEE-IPDPS - IEEE International Parallel & Distributed Processing Symposium.
      • Pearc - Practice & Experience in Advanced Research Computing.
      • Supercomputing (SC) - The International Conference for High Performance Computing, Networking, Storage, and Analysis.
      • Supercomputing International (ISC) - The International Conference for High Performance Computing, Networking, Storage, and Analysis.
      • IEEE-HPEC - IEEE High Performance Embedded Computing.
      • ESPM2 Workshop - International Workshop on Extreme Scale Programming Models and Middleware.
      • LCI Workshops - The Linux Clusters Institute (LCI) is providing education and advanced technical training for the deployment and use of computing clusters to the high performance computing community worldwide.
      • HPC Carpentry - Teaching basic skills for high-performance computing.
      • CCGrid - IEEE/ACM International Symposium on Cluster, Cloud and Internet Computing.

  • Websites

    • Top500 - The TOP500 project ranks and details the 500 most powerful non-distributed computer systems in the world.
    • Top500 - The TOP500 project ranks and details the 500 most powerful non-distributed computer systems in the world.

  • User Groups

    • Prometheus Based

      • MVAPICH - The MUG conference provides an open forum for all attendees (users, system administrators, researchers, engineers, and students) to discuss and share their knowledge on using MVAPICH libraries.
      • Slurm - The annual Slurm user group meeting.

  • Monitoring

    • Prometheus Based

      • Slurm Exporter - Prometheus exporter for performance metrics from Slurm `GPL-3.0`.
      • Slurm Exporter - Slurm Exporter for Prometheus using Rest API `GPL-3.0`.
      • Infiniband Exporter - The InfiniBand exporter collects counters from InfiniBand switches and HCAs `Apache-2.0`.
      • Cgroup Exporter - Produces metrics from cgroups `Apache-2.0`.
      • Cgroup Exporter - A Prometheus exporter for cgroup-level metrics `unknown`.
      • GPFS Exporter - The GPFS exporter collects metrics from the GPFS filesystem `Apache-2.0`.
      • Lustre Exporter - Prometheus exporter for use with the Lustre parallel filesystem `GPL-3.0`.
      • DCGM Exporter - NVIDIA GPU metrics exporter for Prometheus leveraging DCGM `Apache-2.0`.

  • Podcasts

    • Prometheus Based

      • This week in HPC - Each week, Intersect360 Research CEO Addison Snell and HPCwire editor Tiffany Trader dissect the week's top HPC stories.
      • Exascaler Project - ECP's Let's Talk Exascale podcast goes behind the scenes to chat with some of the people who are bringing a capable and sustainable exascale computing ecosystem to fruition.
      • @HPCpodcast - Join Shahin Khan and Doug Black as they discuss Supercomputing technologies and the applications, markets, and policies that shape them.
Programming Languages
Go 9 Python 3 C++ 2 JavaScript 1 Shell 1
Categories
Conferences 15 Provisioning 12 Workload Managers 10 Containers 10 Benchmarking 9 Miscellaneous 9 Parallel Filesystems 9 Monitoring 8 Compilers 7 Performance 7 Blogs 6 Pipelines 4 Environment Management 4 MPI 4 Podcasts 3 Applications 3 Programming Languages 3 Parallel Computing 2 User Groups 2 Parallel Shells 2 Websites 2 Visualization 2 Journals 1
Sub Categories
Prometheus Based 27
Keywords
hpc 4 containers 3 docker 3 runc 2 linux 2 msteams 1 mattermost 1 matrix 1 mail 1 discord 1 windows 1 storage 1 s3 1 parallel 1 nvme 1 mdtest 1 live-stats 1 ior 1 hdfs 1 gpu 1 fio 1 file-systems 1 distributed 1 deep-learning 1 block-storage 1 benchmark 1 prometheus-exporter 1 metrics 1 lustre-exporter 1 lustre 1 oci 1 cpp11 1 singularity 1 user 1 root-privileges 1 proot 1 indigo 1 grid 1 fakechroot 1 eosc-hub 1 emulation 1 docker-containers 1 deep-hybrid-datacloud 1 chroot 1 batch 1 telegram 1 slurm 1 slack 1 notifications 1