{"id":18925634,"url":"https://github.com/jm1/gentoo-linux-on-hpc-clusters","last_synced_at":"2025-08-30T09:37:56.921Z","repository":{"id":69767708,"uuid":"279068117","full_name":"JM1/gentoo-linux-on-hpc-clusters","owner":"JM1","description":"Deploying large software stacks on HPC clusters using Gentoo Prefix","archived":false,"fork":false,"pushed_at":"2021-06-25T10:43:21.000Z","size":94,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-05-25T00:19:23.551Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/JM1.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2020-07-12T13:16:38.000Z","updated_at":"2022-05-10T02:22:57.000Z","dependencies_parsed_at":null,"dependency_job_id":"8a7cb333-5191-474d-b7d7-3ac2da5f2c57","html_url":"https://github.com/JM1/gentoo-linux-on-hpc-clusters","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/JM1/gentoo-linux-on-hpc-clusters","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JM1%2Fgentoo-linux-on-hpc-clusters","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JM1%2Fgentoo-linux-on-hpc-clusters/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JM1%2Fgentoo-linux-on-hpc-clusters/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JM1%2Fgentoo-linux-on-hpc-clusters/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/JM1","download_url":"https://codeload.github.com/JM1/gentoo-linux-on-hpc-clusters/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JM1%2Fgentoo-linux-on-hpc-clusters/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":272833295,"owners_count":25000870,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-30T02:00:09.474Z","response_time":77,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-08T11:12:40.623Z","updated_at":"2025-08-30T09:37:56.912Z","avatar_url":"https://github.com/JM1.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"[//]: # ( vim:set syntax=markdown fileformat=unix shiftwidth=4 softtabstop=4 expandtab textwidth=120: )\n[//]: # ( kate: syntax markdown; end-of-line unix; space-indent on; indent-width 4; word-wrap-column 120; )\n[//]: # ( kate: word-wrap on; remove-trailing-spaces modified; )\n\n# Deploying large software stacks on HPC clusters using Gentoo Prefix\n\nThis project is about how to build and run your own software stack on HPC clusters easily and rapidly with the help of\n[Gentoo Linux](https://www.gentoo.org/) and the [Gentoo Prefix](https://wiki.gentoo.org/wiki/Project:Prefix) project.\n\n:warning:\n**NOTE:**\nThe step-by-step guide listed below is work-in-progress! In its current state it resembles more of a personal sketchpad\nthan a complete guide, it is still missing important steps as I did not have the time to edit them properly and hence\nit is not ready for a broad audience yet!\n:warning:\n\n**NOTE:** Please, feel free to [open issues](https://github.com/JM1/hbrs-mpl/issues) if you encounter bugs or problems!\n\n## Background\n\n**NOTE:** You might skip to the next paragraph as this is about me and my research only.\n\nI am doing research on generic programming methodology. The ultimate goal is to provide non-HPC-experts and scientists\nfrom other domains, such as math, physics and fluid dynamics, a set of reusable and efficient software components. These\ncomponents are declared using domain-specific terms and constraints only without leaking implementation details. E.g.\nfor linear algebra we define matrices, associated operations like the singular value decomposition and pre- and post-\nconditions. We do not make assumptions about e.g. memory layout or memory allocation. These hard-to-get-right details\nare handled on lower layers hidden from application developers. Scientists can freely compose these components (and\nextend them if necessary) to build parallel, distributed and high performance applications for HPC clusters. They can\nfocus on their primary research topics again instead of having to get a programming veteran with decent Fortran, MPI\nand C skills.\n\nPart of my research involves the development, maintenance and deployment of a growing C++ and Python code base (e.g.\n[`hbrs-mpl`](https://github.com/jm1/hbrs-mpl/) and [`hbrs-theta_utils`](https://github.com/JM1/hbrs-theta_utils/)) and\nmany related tools (e.g. [TAU (and THETA)](http://tau.dlr.de/), [ParaView](https://www.paraview.org/)) and libraries\n(e.g. [Boost.Hana](https://github.com/boostorg/hana), [Elemental](https://github.com/elemental/Elemental),\n[OpenMPI](https://www.open-mpi.org/), [Visualization Toolkit (VTK)](https://vtk.org/)).\n\n## Motivation\n\nDeploying large software stacks like ours on a HPC cluster takes a considerably amount of time and effort. You have to\ndownload, configure, build and install dozens (hundreds?) of libraries and applications in the correct order, while\ntrying to satisfy all their dependencies. Some you will identify preliminarily, others you will discover during\nthe deployment process. You better get a package manager like [apt](https://en.wikipedia.org/wiki/APT_(software)) or\n[yum](https://en.wikipedia.org/wiki/Yum_(software))/[dnf](https://en.wikipedia.org/wiki/DNF_(software)) to do this for\nyou. Unfortunately, you cannot.\n\nOn HPC clusters, usually you do not have root/`su`/`sudo` rights to use the OS package manager. But this would not help\nanyway because major cluster operating systems, such as [Scientific Linux](https://scientificlinux.org/), probably do\nnot have the latest or any specific software revision installed that you would like to or have to use. Especially\nexperimental (and) scientific libraries are often not available from the package repositories at all, examples are\n[Elemental](https://github.com/elemental/Elemental) and [libflame](https://github.com/flame/libflame). Other\nusers might want to use different versions of the same libraries and tools, which is only partly (libraries only)\nsupported by package managers. Partial workarounds for different software stacks exist, e.g. [Environment\nModules](https://modules.readthedocs.io/en/latest/), but they do not lift the burden of the manual build process and\ndependency management.\n\nContainers or virtualization are often not available on HPC clusters, one reason is they require privileged access\nregular users do not have. Or they are no feasible alternative to native hardware access because of their negative\nimpact on performance and latency and their increased memory usage.\n\n## Solution: Gentoo Prefix and HPC clusters\n\nOur [HPC cluster at Bonn-Rhein-Sieg University](https://wr0.wr.inf.h-brs.de/wr/usage.html) runs `Scientific Linux\nrelease 7.8 (Nitrogen)`, does not provide containers or virtualization and lacks most software of our stack listed\nabove. Using the OS package manager `yum` is not possible due to insufficient permissions. Manually deploying the whole\nsoftware stack is best to be avoided due to the amount of work involved. What to do?\n\n[Gentoo Prefix](https://wiki.gentoo.org/wiki/Project:Prefix) comes to the rescue!\n\n**NOTE:** If it is possible to use containers or virtual machines, then deeply think before deciding on Gentoo Prefix!\n\n**NOTE:** Before installing Gentoo Linux, you might consider using the [Spack package\nmanager](#alternative-spack-package-manager) or [EasyBuild](#alternative-easybuild)!\n\n\u003e To bring out the virtues of Gentoo Linux on different operating systems, the Gentoo Prefix project develops and\n\u003e maintains a way of installing Gentoo systems in a non-standard location, designated by a \"prefix\".\n\u003e\n\u003e Usually, Gentoo Linux's package manager (portage) installs in the root of the filesystem hierarchy known as /. On\n\u003e systems other than Gentoo Linux, this usually results in problems, due to conflicts of software packages, unless the\n\u003e OS is adapted like Gentoo FreeBSD. Instead, Gentoo Prefix installs within an offset, known as a prefix, allowing\n\u003e users to install Gentoo in another location in the filesystem hierarchy, hence avoiding conflicts. Next to this\n\u003e offset, Gentoo Prefix runs unprivileged, meaning no root user or rights are required to use it.\n\u003e\n\u003e By using an offset (the \"prefix\" location), it is possible for many \"alternative\" user groups to benefit from a large\n\u003e part of the packages in the Gentoo Linux Portage tree. Currently users of the following systems successfully run\n\u003e Gentoo Prefix: Mac OS X on Intel, Linux on x86, x86_64 and arm, Solaris 10 on Sparc, Sparc/64, x86 and x86_64, AIX on\n\u003e PPC, Windows on x86 and x86_64. Other platforms have been successfully used in the past.\n\nThe Gentoo Prefix project allows users to install [Gentoo Linux](https://www.gentoo.org/), [a complete Linux\ndistribution](https://en.wikipedia.org/wiki/Gentoo_Linux), quickly and without privileged (root/`su`/`sudo`) rights\nto their home directories or any other writeable directory on HPC clusters. It enables you to run Gentoo Linux\nnatively from within Scientific Linux without any containers or virtualization. Hence, installing ParaView on our HPC\ncluster boils down to `emerge sci-visualization/paraview` and then Gentoo Linux will fetch, compile and install dozens\nof required libraries and tools automatically.\n\nActually, the only software used from the host operating system are its kernel, ssh server, bash for user login and\nslurmd. The latter, slurmd, is a job system that is required to distribute e.g. MPI applications across all cluster\nnodes. Everything else is from Gentoo Linux. Hence it should be straight forward to apply this guide to other HPC\nclusters as well. No other Linux distribution (i am aware of) supports this. One reason is, that unlike binary software\ndistributions such as Debian, Ubuntu, Red Hat Enterprise Linux / CentOS or SUSE Linux Enterprise Server, with Gentoo\nLinux (most) source code is compiled locally.\n\n[Gentoo's package repositories](https://packages.gentoo.org/) provide a large collection of software that can be\ninstalled easily using Gentoo's package manager [`Portage`](https://en.wikipedia.org/wiki/Portage_(software)). The\n[Gentoo Science Project](https://wiki.gentoo.org/wiki/Project:Science/Overlay) maintains many packages for scientific\nand high performance clustering applications. To get new software running on Gentoo, which is not yet available in any\npackage repository, either build and install it manually or better [write an `ebuild`\nfile](https://wiki.gentoo.org/wiki/Ebuild) and let `Portage` handle the process.\n\n\u003e An `ebuild` file is a text file, used by Gentoo package managers, which identifies a specific software package and\n\u003e how the Gentoo package manager should handle it. It uses a bash-like syntax style and is standardized through the\n\u003e EAPI version.\n\nWriting `ebuild` files is well documented in e.g. [Quickstart Ebuild Guide](https://devmanual.gentoo.org/quickstart/)\nor [Basic guide to write Gentoo Ebuilds](https://wiki.gentoo.org/wiki/Basic_guide_to_write_Gentoo_Ebuilds), and requires\nless effort than developing e.g. `deb` packages for apt-based distributions or `rpm` packages for yum-/dnf-based\ndistributions. Along with the guide you will find several `ebuild` files for packages that are not available in Gentoo\nLinux, such as [Elemental](https://github.com/elemental/Elemental), [libflame](https://github.com/flame/libflame),\n[`hbrs-mpl`](https://github.com/jm1/hbrs-mpl/) and [`hbrs-theta_utils`](https://github.com/JM1/hbrs-theta_utils/))\n\nSince the user's setup of Gentoo Linux is highly decoupled from the host operating system, changes to the cluster like\nsoftware updates or OS upgrades do not alter the Gentoo Prefix environment. Backup and restore of the environment is\nreduced to a plain `tar` of the Gentoo Linux install directory. Hence, the Gentoo Prefix project helps to maintain a\nconsistent environment on HPC clusters, which otherwise is hard, because e.g. backups of the cluster operating system\nis not feasible for regular users in most cases.\n\nUsing Gentoo Linux has downsides. First, it is probably faster and easier to setup a container or virtual machine than\ninstalling Gentoo Linux. If you can use those, think twice before choosing Gentoo Prefix. Not all packages from the\nofficial Gentoo reporitories are compatible to Gentoo Prefix out of the box. System integrity is not at risk, because\nGentoo will raise an error before an incompatible package tries to write to non-prefix directories. Most often it is\nsufficient to patch the `ebuild` file of broken packages, which is done easily due to Gentoo's source distribution\nmodel. When packages get updated upstream, those patches probably have to be refreshed, so better submit your prefix\npatches to Gentoo asap. The biggest issue is software outside of Gentoo Prefix. Depending on the actual software, it\nmight be difficult to compile and link against code or use software of the host OS. The host OS provides its own set of\ne.g. compilers, libraries, environment variables, pkg-config and CMake configurations. Incompatibilities between both\nsoftware stacks will cause compilation bugs at best or silently introduce undetected runtime bugs at worst. Please note,\nthis problem is not specific to Gentoo Prefix but can happen as soon as you use two different C++ compilers for separate\ncode parts. For example, the Visualization Toolkit (VTK) from Gentoo's package repository failed to compile, because its\nCMake-based build scripts erroneously picked up Qt5 headers and CMake exports from the host OS. This has been fixed by\nproviding a global `CMakeLists.txt`, which excludes paths from outside the Gentoo Prefix installation. You will find\nthis CMake configuration and all necessary patches for Gentoo along with this project.\n\nPros:\n+ quick setup compared to manual source code deployment\n+ simple and powerful package management system `Portage` handles package dependencies, their build and install process\n+ no privileged permissions, no root, `su` or `sudo` rights required\n+ minimal requirements and dependencies on host operation system and preexisting software:\n  kernel, ssh server, bash for user logins and slurmd for job scheduling.\n+ native, no performance degredation due to containers or virtualization\n+ many packages available in Gentoo's package repositories, also for scientific and HPC software\n+ easily extendable package system using bash-like `ebuild` text files\n+ consistent environment across host OS changes like software updates or system upgrades\n+ simple backup and restore compared to host OS backups\n\nCons:\n- harder than containers or virtual machines, in case those are viable options for you, think twice on Gentoo Prefix\n- not all official Gentoo packages are compatible to Gentoo Prefix and require patches\n- using software from outside the Gentoo Prefix installation, e.g. from the host OS, is difficult\n\n### Step-by-step guide\n\nFile [`INSTALL`](https://github.com/JM1/gentoo-linux-on-hpc-clusters/blob/master/INSTALL) describes how to setup Gentoo\nPrefix at an HPC cluster properly, using the example of our university's cluster [`wr0.wr.inf.h-brs.de`](\nhttps://wr0.wr.inf.h-brs.de/). [`USAGE`](https://github.com/JM1/gentoo-linux-on-hpc-clusters/blob/master/USAGE) shows\nhow to interact with Gentoo Linux while being logged into the HPC cluster, how to run applications from within the\nGentoo Prefix environment and how to schedule jobs with the host OS scheduler.\n\n## Alternative: Spack Package Manager\n\nFrom the [Spack package manager](https://spack.readthedocs.io) docs:\n\n\u003e Spack is a package management tool designed to support multiple versions and configurations of software on a wide\n\u003e variety of platforms and environments. It was designed for large supercomputing centers, where many users and\n\u003e application teams share common installations of software on clusters with exotic architectures, using libraries that\n\u003e do not have a standard ABI.\n\nCheck out its [package list](https://spack.readthedocs.io/en/latest/package_list.html) to find out if it builds all\nyour required packages.\n\n## Alternative: EasyBuild\n\nFrom the [EasyBuild documentation](https://easybuild.readthedocs.io) docs:\n\n\u003e EasyBuild is a software build and installation framework that allows you to manage (scientific) software on High\n\u003e Performance Computing (HPC) systems in an efficient way.\n\nCheck out its [package list](https://easybuild.readthedocs.io/en/latest/version-specific/Supported_software.html) to\nfind out if it builds all your required packages.\n\n## License\n\nGNU General Public License v3.0 or later\n\nSee [LICENSE.md](https://github.com/JM1/gentoo-linux-on-hpc-clusters/blob/master/LICENSE.md) to see the full text.\n\n## Author\n\nJakob Meng\n@jm1 ([github](https://github.com/jm1), [web](http://www.jakobmeng.de))\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjm1%2Fgentoo-linux-on-hpc-clusters","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjm1%2Fgentoo-linux-on-hpc-clusters","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjm1%2Fgentoo-linux-on-hpc-clusters/lists"}