{"id":27918756,"url":"https://github.com/juliaparallel/deploymentsonhpc","last_synced_at":"2026-01-24T04:06:24.941Z","repository":{"id":279161483,"uuid":"937880604","full_name":"JuliaParallel/DeploymentsOnHPC","owner":"JuliaParallel","description":"Central repository of Julia deployments on several HPC centers","archived":false,"fork":false,"pushed_at":"2025-04-30T20:57:49.000Z","size":120,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":16,"default_branch":"main","last_synced_at":"2025-04-30T21:40:20.777Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Euphoria","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/JuliaParallel.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-02-24T04:03:45.000Z","updated_at":"2025-04-30T20:57:52.000Z","dependencies_parsed_at":"2025-03-19T16:28:39.435Z","dependency_job_id":"eba57290-4852-4598-83c1-dabec718559d","html_url":"https://github.com/JuliaParallel/DeploymentsOnHPC","commit_stats":null,"previous_names":["juliaparallel/deploymentsonhpc"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JuliaParallel%2FDeploymentsOnHPC","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JuliaParallel%2FDeploymentsOnHPC/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JuliaParallel%2FDeploymentsOnHPC/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JuliaParallel%2FDeploymentsOnHPC/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/JuliaParallel","download_url":"https://codeload.github.com/JuliaParallel/DeploymentsOnHPC/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252742610,"owners_count":21797278,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-05-06T18:25:17.283Z","updated_at":"2026-01-24T04:06:24.928Z","avatar_url":"https://github.com/JuliaParallel.png","language":"Euphoria","funding_links":[],"categories":[],"sub_categories":[],"readme":"# A Collection of Julia Deployments on HPC Systems\n\nThis is a collection of module-centric Julia deployments on HPC systems. A\ncommon challenge faced by software deployments on shared systems is interacting\nwith other installed system software. Most HPC systems therefore come with an\nenvironment modules system -- which is the preferred way to manage HPC software\nenvironments. This repo shows how templated environment modules and settings\nfiles can be used to install and manage Julia on a shared file systems.\n\n## Objectives, Challenges, and how to use this Repository\n\nThe environment modules on HPC systems can be very complex, often presenting\nusers with different choices in toolchains and versions. A common example is\nthat users can select different MPI implementations, and different GPU runtime\nversions. This can lead to a combinatorial explosion of different system\ndependencies.\n\nHence the solution presented here is to \"tie\" the Julia configuration into the\nenvironment module system.\n\nRequirements:\n1. [Lua](https://www.lua.org/download.html) and\n   [luaposix](https://github.com/luaposix/luaposix)\n2. [Lmod](https://lmod.readthedocs.io/en/latest/030_installing.html) or\n   [Environment Modules](https://modules.readthedocs.io/en/latest/INSTALL.html)\n\n### Use the Entrypoint\n\nWe want to ensure that the installers have have predictable behavior,\nregardless of the execution context (e.g. where scripts are being run from).\nBash kinda sucks at this (especially when handling relative paths) -- an\nimperfect (but acceptable) solution is to use a launcher script:\n[entrypoint.sh](./entrypoint.sh). For example, if you want to render all Julia\nenvironment configuration files, please run:\n```\n$ ./entrypoint.sh nersc/environments/templates/render.sh \n```\nThis can be run from anywhere because `entrypoint.sh` sets two environment\nvariables `__PREFIX__`, and `__DIR__`:\n* `__PREFIX__` contains the location of the project root (i.e. Where\n`entrypoint.sh` is saved)\n* `__DIR__` contains the location of he script being run\n\n### Rendering Templates\n\nThis project uses [Simple\nTemplates](https://gitlab.blaschke.science/nersc/simple-templates) to render\ntemplates. A standalone version of Simple Templates is located at:\n`./opt/bin/simple-templates.ex` and only requires a reasonably modern version\nof Lua to run (I didn't check how modern it needs to be though).\n\nSimple Templates takes template files formatted in the [mustache templating\nlanguage](https://mustache.github.io/) and populates them with values from a\nsettings ([TOML](https://toml.io/en/) version 0.4) file. This allows us to\nadapt Julia settings and modules whenever systems are reconfigured and\nupgraded.\n\n#### Why not something like Jinja?\n\nWith the exception of Lua and Lmod, the objective of this approach is to be as\nself-contained as possible. More powerful templating engines need other\ndependencies to be installed (e.g. a modern version of Python + a virtualenv),\nwhich are often not present on bare-bones systems. The reason for going with\nLua is to not need a lengthy install procedure. Furthermore, more powerful\ntemplating languages are just not needed to generate what are basically just a\nbunch of settings and module files.\n\n### Installing Modules\n\nThis project uses [Simple\nModules](https://gitlab.blaschke.science/nersc/simple-modules) to download and\ninstall software, and to render the corresponding module files. A standalone\nversion of Simple Modules is located at: `./opt/bin/simple-modules.ex` and\nrequires a reasonably modern version of luaposix (which is used by Lmod also).\n\nSimple Modules is a small utility that lets you easily download, build, and\ndeploy software modules.\n\n### Use Modules (Lmod) As Much As Possible\n\nWe avoid activation and wrapper scripts as much as possible -- Lmod is the best\ntool for modifying the runtime environment, and it is capable of unwinding\nthose changes when requested.\n\nA consequence of this choice is that all configurations need to be defined when\nmodules are loaded, which can lead to some deployment quirks listed below.\n\n#### Configuration Files\n\nWe use configuration files exclusively. Consequently, every conceivable\nscenario needs its own set of configuration files. For example, every\ncombination of CUDA version and MPI toolchain needs to be accounted for. The\napproach we take is to generate settings (using templates -- see above) for\nevery combination (the NERSC deployment is a [good example of\nthis](./nersc/environments/rendered/)). A module will then put the right\nsettings files into the `JULIA_LOAD_PATH` like so:\n```lua\nlocal PE_ENV = os.getenv(\"PE_ENV\"):lower()\nlocal FAMILY_MPI = os.getenv(\"LMOD_FAMILY_MPI\"):lower()\nlocal FAMILY_CUDATOOLKIT = os.getenv(\"LMOD_FAMILY_CUDATOOLKIT_VERSION\"):lower()\n\nlocal ENV_NAME = PE_ENV .. \".\" .. FAMILY_MPI .. \".cuda\" .. FAMILY_CUDATOOLKIT\nlocal JULIA_LOAD_PATH = \":{{{JULIA_LOAD_PATH_PREFIX}}}/\" .. ENV_NAME\n\nappend_path(\"JULIA_LOAD_PATH\", JULIA_LOAD_PATH)\n```\n(**Note:** the string `{{{JULIA_LOAD_PATH_PREFIX}}}` is a template parameter\nwhich is filled out by Simple Modules)\n\n#### Pitfalls Due to Interactions Between Modules\n\nOut approach is not perfect -- one common issue is when Lmod changes dependent\nmodules, the values for the environment variables `PE_ENV`, `LMOD_FAMILY_MPI`,\nand `LMOD_FAMILY_CUDATOOLKIT_VERSION` in the example above might change, making\nLmod \"loose track\" of `JULIA_LOAD_PATH`. Only imperfect (but simple) solution\nis to keep track of the set value of `JULIA_LOAD_PATH` like so:\n```lua\nlocal JULIA_LOAD_PATH\nif (\"unload\" == mode()) then\n    JULIA_LOAD_PATH = os.getenv(\"__JULIAUP_MODULE_JULIA_LOAD_PATH\")\nelse\n    JULIA_LOAD_PATH = \":{{{JULIA_LOAD_PATH_PREFIX}}}/\" .. ENV_NAME\nend\nsetenv(\"__JULIAUP_MODULE_JULIA_LOAD_PATH\", JULIA_LOAD_PATH)\n\nappend_path(\"JULIA_LOAD_PATH\", JULIA_LOAD_PATH)\n```\nWhile not automagically updating `JULIA_LOAD_PATH` when `ENV_NAME` changes --\nthis approach allows a `module reload` to fix the `JULIA_LOAD_PATH`.\n\n## Deployments on HPC Facilities\n\n### Common Components\n\nSeveral components of this install are common across many HPC facilities. You\ncan find a [description of these here](./common/README.md).\n\n### NERSC\n\n[Description here](./nersc/README.md)\n\n### PSC\n\n[Description here](./psc/README.md)\n\n### TACC\n\nComing soon\n\n### DIII-D\n\nComing soon\n\n## FAQ\n\n### How is this different from [JUHPC](https://github.com/JuliaParallel/JUHPC)?\n\n[JUHPC](https://github.com/JuliaParallel/JUHPC) is a user space wrapper that\nhelps in downlaoding and configuring (via `Pkg`) Julia. So it is great for\nconfiguring Julia based on clues from the environment. However many HPC systems\nhave several versions of critical system software, so we still need a framework\nfor interacting with the module system. In practise this leads to the following\ndifferences:\n\n1. Our version builds [juliaup](https://github.com/JuliaLang/juliaup) without\n   `selfupdate`. You need to build juliaup from source in order to disable it\n   from attempting to grab write access to files that is owned by the user.\n2. We have a wrapper called juliaup-hpc which generates a module file for any\n   user-installed Julia versions (ideally that will make it upstream into\n   juliaup some time).\n3. We generate module files which know how to interact with the system modules.\n4. We manage globally installed Jupyter kernels\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjuliaparallel%2Fdeploymentsonhpc","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjuliaparallel%2Fdeploymentsonhpc","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjuliaparallel%2Fdeploymentsonhpc/lists"}