{"id":16519741,"url":"https://github.com/zonca/nbsubmit","last_synced_at":"2025-10-28T07:31:27.485Z","repository":{"id":57445184,"uuid":"116991057","full_name":"zonca/nbsubmit","owner":"zonca","description":null,"archived":false,"fork":false,"pushed_at":"2018-09-26T19:07:02.000Z","size":36,"stargazers_count":2,"open_issues_count":2,"forks_count":2,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-02-01T12:45:29.443Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zonca.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-01-10T17:47:46.000Z","updated_at":"2019-10-25T22:08:21.000Z","dependencies_parsed_at":"2022-09-26T17:30:48.573Z","dependency_job_id":null,"html_url":"https://github.com/zonca/nbsubmit","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zonca%2Fnbsubmit","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zonca%2Fnbsubmit/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zonca%2Fnbsubmit/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zonca%2Fnbsubmit/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zonca","download_url":"https://codeload.github.com/zonca/nbsubmit/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":238614522,"owners_count":19501476,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-11T16:48:08.737Z","updated_at":"2025-10-28T07:31:27.153Z","avatar_url":"https://github.com/zonca.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"Submit Jupyter Notebooks for remote execution\n=============================================\n\nThis Python package provides a simplified interface for running a Jupyter Notebook\nnon-interactively on a remote machine.\nIt currently supports Supercomputers running SLURM, in the future I'd like to add\nalso HTCondor support, possibly interfacing with OpenScienceGrid.\n\n## Use case\n\nThe idea is that you develop and test on a small dataset your custom analysis Jupyter Notebook locally.\nOnce the Jupyter Notebook is finalized, you would like to run it on a large dataset or on a\nlarge amount of medium-sized dataset.\nYou need access to a large amount of memory and disk, therefore you would like to\nuse a Supercomputer you already have access to. For example Comet at the San Diego Supercomputer Center.\n\n## Installation\n\nInstall from pypi:\n\n    pip install nbsubmit\n\n## How it works\n\nThis package takes care of copying the Notebook to the Supercomputer, prepare a SLURM\njob and submit it to the scheduler.\nOnce the job starts, the Jupyter Notebook is executed non-interactively with `jupyter nbconvert` inside\na Singularity Container.\nYou can monitor its execution without ever leaving the Notebook on your local machine.\nOnce the job completes, `nbsubmit` allows you to copy back the executed Notebook (with plots included)\nand any output that was produced in the same folder.\n\n## Large data files\n\nIf you are processing files larger than ~100GB, it is better to transfer them first to the Supercomputer\nwith Globus Online to your SCRATCH space (e.g. on Comet it is `/oasis/scratch/comet/$USER/temp_project/`),\nand then point the Notebook to that folder.\n\nSame for the output files, you can save and automatically retrieve small files located in the same folder\nas the Notebook. Save instead large files on your SCRATCH space and copy them with Globus Online.\n\n## Mount a remote filesystem\n\nIt can be convenient to mount the filesystem locally so that we do not need to copy the results of our computation\nback to the local machine.\n\n`nbsubmit` provides some convenience functions to simplify this. However it is necessary to first setup a\nmultiplexed SSH connection, so that you can interactively start a SSH connection and authenticate just once,\nfor example this is feasible also when the Supercomputer requires 2 factor authentication.\n\nIn order to configure this, you need to copy into your `.ssh/config` the configuration available in `ssh_configuration/config`.\n\nThen before using `nbsubmit`, open a terminal on your local machine and access the target Supercomputer via SSH\nand leave that terminal open.\n\nThen we can mount the filesystem locally with:\n\n```\n\u003e from nbsubmit import cluster\n\u003e comet = cluster.get(\"comet\")\n\u003e comet.mount()\nMounted comet.sdsc.edu:/oasis/scratch/comet/zonca/temp_project to /home/zonca/comet\n\u003e !df -h | tail -1\ncomet.sdsc.edu:/oasis/scratch/comet/zonca/temp_project  2.5P  2.3P  240T  91% /home/zonca/comet\n```\n\nSee `example/mounted_filesystem` for a complete example.\n\n## Examples\n\nSee the `examples/` folder for example Notebooks and more documentation.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzonca%2Fnbsubmit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzonca%2Fnbsubmit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzonca%2Fnbsubmit/lists"}