{"id":26232672,"url":"https://github.com/pc2/jhub-hpc-interface","last_synced_at":"2025-09-11T17:20:26.085Z","repository":{"id":62572353,"uuid":"336209811","full_name":"pc2/JHub-HPC-Interface","owner":"pc2","description":"JupyterHub + High-Performance Computing","archived":false,"fork":false,"pushed_at":"2021-05-18T07:32:49.000Z","size":460,"stargazers_count":6,"open_issues_count":0,"forks_count":1,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-09-10T13:53:02.580Z","etag":null,"topics":["hpc","jupyter","jupyterhub","singularity"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pc2.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-02-05T08:28:07.000Z","updated_at":"2025-03-03T19:06:05.000Z","dependencies_parsed_at":"2022-11-03T18:26:39.443Z","dependency_job_id":null,"html_url":"https://github.com/pc2/JHub-HPC-Interface","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/pc2/JHub-HPC-Interface","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pc2%2FJHub-HPC-Interface","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pc2%2FJHub-HPC-Interface/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pc2%2FJHub-HPC-Interface/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pc2%2FJHub-HPC-Interface/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pc2","download_url":"https://codeload.github.com/pc2/JHub-HPC-Interface/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pc2%2FJHub-HPC-Interface/sbom","scorecard":{"id":725015,"data":{"date":"2025-08-11","repo":{"name":"github.com/pc2/JHub-HPC-Interface","commit":"6c205d2269d134dc3cbac55375eeec0ec586d921"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":3,"checks":[{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Code-Review","score":0,"reason":"Found 0/30 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"SAST","score":0,"reason":"no SAST tool detected","details":["Warn: no pull requests merged into dev branch"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Pinned-Dependencies","score":-1,"reason":"no dependencies found","details":null,"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}}]},"last_synced_at":"2025-08-22T12:31:51.630Z","repository_id":62572353,"created_at":"2025-08-22T12:31:51.630Z","updated_at":"2025-08-22T12:31:51.630Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":274676489,"owners_count":25329295,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-11T02:00:13.660Z","response_time":74,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["hpc","jupyter","jupyterhub","singularity"],"created_at":"2025-03-13T00:38:06.561Z","updated_at":"2025-09-11T17:20:26.022Z","avatar_url":"https://github.com/pc2.png","language":"Jupyter Notebook","readme":"# JupyterHub + High-Performance Computing\n\n**High performance Jupyter Notebooks**\n\nThe aim of this project is to connect JupyterHub to a high-performance computer (HPC). By automatically offloading the computations in a Jupyter notebook to the HPC system, even complex calculations are possible. While JupyterHub is deployed on a regular server, the notebooks themselves are spawned and run on the remote HPC system using a workload manager, such as Slurm.\n\n**Motivation**\n\nThe technical core of this project is the transparent integration of digital worksheets (Jupyter notebooks), in which learning content and programs can be displayed, edited and executed on the students' own laptops, with current cloud and high-performance computing (HPC) technologies. This provides the conditions for innovative, digital teaching that encourages independent and interactive development of, for example, data science applications, without imposing the complexity of using a high-performance computer system on the students. Instead, particularly computationally and data-intensive calculations are automatically offloaded to a high-performance computer, enabling even sophisticated analyses to be performed that would otherwise not be feasible on students' laptops.\n\n**Features and use cases**\n\n* Starting a jupyter notebook server on a remote HPC system in a pre-defined singularity container\n* Quick config setup when using the Slurm configuration wizard\n* Automatically create a singularity overlay so that user changes are persistent\n* Great for managing courses with external participants\n* Possibility to include files in the notebook directory using WebDAV\n* Suitable for HPC users who have their own JupyterHub instance running and want to use HPC resources\n\n---\n\n## Table of Contents\n\n- [JupyterHub + High-Performance Computing](#jupyterhub--high-performance-computing)\n  - [Table of Contents](#table-of-contents)\n  - [Installation of JupyterHub Server](#installation-of-jupyterhub-server)\n    - [JupyterHub and BatchSpawner](#jupyterhub-and-batchspawner)\n    - [SSH tunnel user](#ssh-tunnel-user)\n    - [Node mapping](#node-mapping)\n  - [Installation on HPC System](#installation-on-hpc-system)\n    - [Requirements](#requirements)\n    - [Install using pip](#install-using-pip)\n    - [Singularity Container](#singularity-container)\n      - [Build Singularity Container](#build-singularity-container)\n        - [Compute](#compute)\n        - [GPU (Tensorflow)](#gpu-tensorflow)\n    - [The configuration file](#the-configuration-file)\n    - [Slurm configuration wizard](#slurm-configuration-wizard)\n  - [Examples](#examples)\n    - [Debug mode](#debug-mode)\n  - [Shibboleth Integration](#shibboleth-integration)\n  - [NBGrader Integration](#nbgrader-integration)\n    - [Installation](#installation)\n    - [Changing the Student ID to the JupyterHub logged in user name](#changing-the-student-id-to-the-jupyterhub-logged-in-user-name)\n    - [Create nbgrader_config.py](#create-nbgrader_configpy)\n  - [Security Precautions](#security-precautions)\n    - [Singularity Host Filesystems](#singularity-host-filesystems)\n    - [JupyterHub API (HTTPS)](#jupyterhub-api-https)\n      - [HTTPS](#https)\n    - [tunnelbot user](#tunnelbot-user)\n  - [Troubleshooting](#troubleshooting)\n\n---\n\n## Installation of JupyterHub Server\n\nThis section describes the required installations and configurations on the JupyterHub server.\n\n### JupyterHub and BatchSpawner\n\nThe first thing you should do is install JupyterHub and BatchSpawner. For this purpose we provide an Ansible playbook which can be found in `/jupyterhub-deployment/`. See the README for details. Alternatively, you can follow the official installation instructions.\n\nIf you decide to do the installations yourself, please proceed as follows:\n\n- install [JupyterHub](https://jupyterhub.readthedocs.io/en/stable/installation-guide-hard.html)\n- install [BatchSpawner](https://github.com/jupyterhub/batchspawner)\n- install [WrapSpawner](https://github.com/jupyterhub/wrapspawner) (make sure to install it in the right environment: `/opt/jupyterhub/bin/pip3 install git+https://github.com/jupyterhub/wrapspawner`)\n- copy the JupyterHub configuration file `/jupyterhub-deployment/config_files/jupyterhub_config.py` to `/opt/jupyterhub/etc/jupyterhub/` (you will most likely have to edit this file afterwards to make it fit your needs)\n- restart the JupyterHub service\n\n### SSH tunnel user\n\nA user called `tunnelbot` is needed on the JupyterHub server. This user is responsible for starting an SSH tunnel between the compute node and the JupyterHub server. An SSH key pair for the above mentioned purpose must be generated. See `/examples/jupyterhub_config.py` for more information.\n\n### Node mapping\n\nJupyterHub extracts the execution host name of the HPC system (e.g. `node01-002`). When a notebook server is started, an SSH tunnel is established using the notebook port.\n\nIn order for JupyterHub to be able to resolve the compute nodes host name, the `/etc/hosts` file must be edited. An example entry might look like the following:\n\n```\n127.0.0.1 node01-001\n127.0.0.1 node01-002\n127.0.0.1 node01-003\n...\n127.0.0.1 node12-048\n```\n\nThe actual node names depend on your HPC system of course.\n\n---\n\n## Installation on HPC System\n\nThis section describes the required installations and configurations of the HPC system to enable the interaction with the JuypterHub server.\n\n### Requirements\n\n* You need a user who is allowed to allocate resources on the HPC system\n  * With a SSH key pair. The public part must be deposited on the JupyterHub serer (`tunnelbot` user)\n  * The public key part of the `tunnelbot`-user created on the JupyterHub (-\u003e _~/.ssh/authorized_keys_)\n* Singularity (\u003e v.3.7.0)\n* mkfs/e2fsprogs with following option:\n  * https://git.kernel.org/pub/scm/fs/ext2/e2fsprogs.git/commit/?id=217c0bdf17899c0f79b73f76feeadd6d55863180\n\n### Install using pip\n\nYou can download and install the required files with pip.\n\nYou may want to build a small Python environment, or install the tools with `--user`.\n\n```bash\npython3 -m pip install --user jh-hpc-interface\n```\n\n### Singularity Container\n\nSingularity recipe examples are in the directory singularity/.\n\nIf you do not want to use singularity, then change the value of `use_singularity` in jh_config.ini to false.\n\n#### Build Singularity Container\n\nTo build the container with the recipe files in singularity/ you have to clone this repository.\n\nThe following commands replace USER_ID in the recipes to the output of `id -u`, create a new hidden file and build the singularity container with the new created file.\n \n##### Compute\n\n```bash\nUSER_ID=$(id -u) \u0026\u0026 sed \"s/USER_ID/$USER_ID/\" \u003c singularity/Singularity \u003e singularity/.recipefile_compute \u0026\u0026 singularity build --remote singularity/compute_jupyter.sif singularity/.recipefile_compute\n```\n\n##### GPU (Tensorflow)\n\n```bash\nUSER_ID=$(id -u) \u0026\u0026 sed \"s/USER_ID/$USER_ID/\" \u003c singularity/Singularity_Tensorflow \u003e singularity/.recipefile_gpu \u0026\u0026 singularity build --remote singularity/gpu_jupyter.sif singularity/.recipefile_gpu\n```\n\n_singularity build help section_:\n\u003e __-r, --remote__            build image remotely (does not require root)\n\nPlease refer to the official docs on how to use the remote build feature: https://sylabs.io/docs/\n\n### The configuration file \n\nIn the directory __bin/__ is a script, which is deposited after the installation on the system.\n\nWith the following call you can display the location of the configuration file:\n\n```bash\n$ jh_wrapper getconfig\n```\n\nTo learn more about the configuration file, see [docs/jh_config.ini.md](docs/jh_config.ini.md)\n\n### Slurm configuration wizard\n\nWith the configuration wizard you can prepare your HPC environment.\n\nThe script interactively goes through the configuration file and creates a temporary file which can be copied with a simple `cp`.\n\nTo start the wizard type the following:\n\n```bash\n$ jh_slurm_wizard\n```\n\n---\n\n## Examples\n\nYou will find examples for the configuration files __jh_config.ini__ and __jupyterhub_config.py__ in the directory _examples/_.\n\n---\n\n### Debug mode\n\nBy default the logs contain only information such as warnings or error messages.\nIt is also possible to switch on the debug mode, which writes extended information into the log files.\n\nJust set `log_level` in the configuration file to 'DEBUG'.\n\n---\n\n## Shibboleth Integration\n\nShibboleth authentication was set up for a JupyterHub server in a test environment. See `./shibboleth/` for an example configuration.\n\n---\n\n## NBGrader Integration\n\n### Installation\n\nInstallation instructions:\nhttps://nbgrader.readthedocs.io/en/latest/configuration/jupyterhub_config.html\n\nTo create an exchange directory for every user, just create an empty directory in `$scratch_dir` and mount it into the container with `$singularity_bind_extra`.\n\n### Changing the Student ID to the JupyterHub logged in user name\n\nSince the containers run as user `jovyan`, the value from the `$JUPYTERHUB_USER` variable is automatically used.\n\nSee here for more information: \nhttps://jupyter.readthedocs.io/en/latest/community/content-community.html#what-is-a-jovyan\n\n### Create nbgrader_config.py\n\nSee here: https://nbgrader.readthedocs.io/en/stable/configuration/nbgrader_config.html#use-case-3-nbgrader-and-jupyterhub\n\nTo make _nbgrader_config.py_ available in the container, just append the file in `$singularity_bind_extra`.\n\n---\n\n## Security Precautions\n\n### Singularity Host Filesystems\n\nIn case you are using Singularity, the host file system may be automatically mounted into the container when you start a Singularity Container.\n\nA possible cause is the option `mount hostfs` in _singularity.conf_\n\nSee here: https://sylabs.io/guides/3.5/admin-guide/configfiles.html#singularity-conf\n\n### JupyterHub API (HTTPS)\n\n#### HTTPS\n\nSee here for more information:\nhttps://jupyterhub.readthedocs.io/en/stable/reference/websecurity.html\n\n### tunnelbot user\n\nYou can increase the security by deactivating shell access for this user.\n\nJust type:\n\n```bash\nusermod -s /bin/false tunnelbot\n```\n\n---\n\n## Troubleshooting\n\nWhen problems occur with the JupyterHub, some information can be obtained from the logs when debug mode is enabled:\n\nhttps://github.com/jupyterhub/jupyterhub/wiki/Debug-Jupyterhub\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpc2%2Fjhub-hpc-interface","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpc2%2Fjhub-hpc-interface","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpc2%2Fjhub-hpc-interface/lists"}