{"id":22531082,"url":"https://github.com/pc2/slurm_jupyter_kernel","last_synced_at":"2025-10-11T01:04:58.936Z","repository":{"id":45601319,"uuid":"346978615","full_name":"pc2/slurm_jupyter_kernel","owner":"pc2","description":"Manage (create, list, modify and delete) and starting jupyter kernels using sbatch","archived":false,"fork":false,"pushed_at":"2023-03-20T10:40:55.000Z","size":786,"stargazers_count":11,"open_issues_count":1,"forks_count":5,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-09-29T23:56:29.677Z","etag":null,"topics":["hpc","ijulia","ipython","jupyter","jupyter-kernel","jupyterhub","jupyterlab","slurm","slurm-jupyter-kernel","srun"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pc2.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-03-12T07:26:20.000Z","updated_at":"2025-06-22T20:28:47.000Z","dependencies_parsed_at":"2024-06-12T17:22:01.901Z","dependency_job_id":"a02318b6-3c2f-42e4-9c47-5dd9da2383b6","html_url":"https://github.com/pc2/slurm_jupyter_kernel","commit_stats":{"total_commits":106,"total_committers":3,"mean_commits":"35.333333333333336","dds":"0.037735849056603765","last_synced_commit":"283873f3f3825b6602a65c33019b8dd9c10cefb4"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/pc2/slurm_jupyter_kernel","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pc2%2Fslurm_jupyter_kernel","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pc2%2Fslurm_jupyter_kernel/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pc2%2Fslurm_jupyter_kernel/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pc2%2Fslurm_jupyter_kernel/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pc2","download_url":"https://codeload.github.com/pc2/slurm_jupyter_kernel/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pc2%2Fslurm_jupyter_kernel/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279005909,"owners_count":26083982,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-10T02:00:06.843Z","response_time":62,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["hpc","ijulia","ipython","jupyter","jupyter-kernel","jupyterhub","jupyterlab","slurm","slurm-jupyter-kernel","srun"],"created_at":"2024-12-07T08:05:57.560Z","updated_at":"2025-10-11T01:04:58.920Z","avatar_url":"https://github.com/pc2.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Slurm Jupyter Kernel\n\nManage (create, list, modify and delete) and starting jupyter slurm kernels using `sbatch`.\n\nslurmkernel is able to connect to a kernel started on a compute node using SSH port forwarding.\nYou can specify a SSH proxy jump, if you have to jump over two hosts (e.g. a loadbalancer)\n\n![How it works](imgs/how_it_works.png)\n\n```\nSlurm job 3251854 is in state \"RUNNING\"\nSlurm job is in state running on compute node cn213\nStarting SSH tunnel to forward kernel ports to localhost\nYour started kernel is now ready to use on compute node cn213\n```\n\n## Features \u0026 Use-Cases\n\n* Start Remote Jupyter kernel using sbatch (Slurm)\n  * Access to your local filesystem with remote code execution\n* Manage existing Slurm Jupyter kernel\n* Use the template module to use pre-defined script templates for remote installation and local kernel creation\n* Custom environment variables supported (e.g. JULIA_NUM_THREADS)\n\n## Table of Contents\n\n- [Slurm Jupyter Kernel](#slurm-jupyter-kernel)\n  - [Features \\\u0026 Use-Cases](#features--use-cases)\n  - [Table of Contents](#table-of-contents)\n  - [Installation](#installation)\n    - [Install using pip](#install-using-pip)\n  - [Requirements for usage](#requirements-for-usage)\n  - [Create a new kernel](#create-a-new-kernel)\n    - [Template module (Script templates)](#template-module-script-templates)\n      - [Example](#example)\n    - [IPython Example](#ipython-example)\n      - [Remote Host](#remote-host)\n      - [Localhost](#localhost)\n    - [Set kernel-specific environment](#set-kernel-specific-environment)\n  - [Using the kernel with Applications](#using-the-kernel-with-applications)\n    - [Quarto Example](#quarto-example)\n  - [Troubleshooting](#troubleshooting)\n    - [Kernel exceptions](#kernel-exceptions)\n    - [Debugging](#debugging)\n    - [Get help](#get-help)\n\n## Installation\n\n`slurm_jupyter_kernel` must be installed locally where the Jupyter notebooks will run.\n\n### Install using pip\n\n```bash\npython3 -m pip install slurm_jupyter_kernel\n```\n\n## Requirements for usage\n\n* SSH-Key based authentication\n\nYou need a running SSH agent with the loaded key file to access the loginnode without a password.\n\n## Create a new kernel\n\nWe assume to install the Jupyter kernel tools into your `$HOME` directory on your cluster.\n\n### Template module (Script templates)\n\nWith `$ slurmkernel template {list, use, add, edit}` you can use pre-defined script templates to initialize your remote environment (IJulia, IPython, ...), add new script templates or edit existing templates.\n\nIf you want to create your own script templates, see here: [Create Script Templates](https://github.com/pc2/slurm_jupyter_kernel/wiki/Create-Script-Templates)\n\n#### Example\n\n**Note:** Add the parameter `--dry-run` to check the commands that will be executed!\n\n```bash\n$ slurmkernel template use --proxyjump lb.hpc.pc2.de --loginnode login001 --user hpcuser1 --template ipython\n````\n\nYou will be interactively asked for the required information if you do not pass any arguments when calling `slurmkernel template use`\n\n### IPython Example\n\n#### Remote Host\n\n1. load required software (if necessary)\n2. Create a Python virtual environment\n3. Install the IPython package (ipython, ipykernel)\n4. Create a wrapper script and mark it as executable\n\n```bash\nremotehost ~$ module load lang Python\nremotehost ~$ python3 -m venv remotekernel/\nremotehost ~$ source remotekernel/bin/activate\n(remotekernel) remotehost ~$ python3 -m pip install ipython ipykernel; deactivate\nremotehost ~$ echo -e '#!/bin/bash\\nmodule load lang Python\\n\\nsource remotekernel/bin/activate\\n\"$@\"' \u003e remotekernel/ipy_wrapper.sh \u0026\u0026 chmod +x remotekernel/ipy_wrapper.sh\n```\n\n#### Localhost\n\n5. Kernel Remote Slurm kernel with command `slurmkernel`\n\n```bash\nnotebook ~$ slurmkernel create --displayname \"Python 3.8.2\" \\\n--slurm-parameter=\"account=slurmaccount,time=00:30:00,partition=normal\" \\\n--kernel-cmd=\"\\$HOME/remotekernel/ipy_wrapper.sh ipython kernel -f {connection_file}\" \\\n--proxyjump=\"lb.n1.pc2.uni-paderborn.de\" \\\n--loginnode=\"login-0001\" \\\n--language=\"python\"\n```\n\n![Example](imgs/example.png)\n\n### Set kernel-specific environment\n\nIf you want to set kernel specific environment variables (e.g. `JULIA_NUM_THREADS` for the number of threads) just extend the jupyter kernelspec file with `env`.\n\nParameter for `slurmkernel`:\n\n`--environment=\"JULIA_NUM_THREADS=4\"`\n\nMore information here: https://jupyter-client.readthedocs.io/en/stable/kernels.html\n\n## Using the kernel with Applications\n\n* Install kernel as shown above \n  *  Make sure that you pass the `--language` flag as well (e.g. `python` or `julia`)\n\n### Quarto Example\n\u003cimg src=\"imgs/quarto_example.png\" width=\"600\"\u003e\n\n## Troubleshooting\n\n### Kernel exceptions\n\nWhen you start a Jupyter slurm kernel, it throws an excpetion, depending on the error case. You can read the exception from the graphical user interface like JupyterLab.\nIf you start a kernel in the classic notebook view, you can click on \"Error\" on the left of the kernel status. There you can also find the exception.\n\n![Exception Example 1](imgs/exception_example_1.png)\n![Exception Example 2](imgs/exception_example_2.png)\n\n### Debugging\n\nIf your Slurm jupyter kernel does not start, it can have many causes.\nBefore we turn on the debug mode, check following things:\n\n* SSH-Agent is active/running and my key is loaded\n  * If you can log in in to the loginnode of the HPC system passwordless (Shell) that should work\n* Correct Proxyjump (Loadbalancer), Loginnode\n\n### Get help\n\n```bash\n$ slurmkernel --help\n\nusage: Tool to manage (create, list, modify and delete) and starting jupyter slurm kernels using srun [-h] [--version] {create,list,edit,delete,template} ...\n\npositional arguments:\n  {create,list,edit,delete,template}\n    create              create a new slurm kernel\n    list                list available slurm kernel\n    edit                edit an existing slurm kernel\n    delete              delete an existing slurm kernel\n    template            manage script templates (list, use, add, edit)\n\noptional arguments:\n  -h, --help            show this help message and exit\n  --version             show program's version number and exit\n\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpc2%2Fslurm_jupyter_kernel","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpc2%2Fslurm_jupyter_kernel","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpc2%2Fslurm_jupyter_kernel/lists"}