{"id":13533912,"url":"https://github.com/SvenMarcus/hpc-rocket","last_synced_at":"2025-04-01T22:31:03.280Z","repository":{"id":38190577,"uuid":"358363996","full_name":"SvenMarcus/hpc-rocket","owner":"SvenMarcus","description":null,"archived":false,"fork":false,"pushed_at":"2024-02-22T08:34:32.000Z","size":2245,"stargazers_count":24,"open_issues_count":6,"forks_count":2,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-04-24T13:22:24.088Z","etag":null,"topics":["ci","continuous-integration","hpc"],"latest_commit_sha":null,"homepage":"https://svenmarcus.github.io/hpc-rocket/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SvenMarcus.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS","dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-04-15T19:02:00.000Z","updated_at":"2024-06-05T10:13:43.070Z","dependencies_parsed_at":"2024-01-09T11:23:35.458Z","dependency_job_id":"d8cc763d-feee-4815-8b52-698451e17afe","html_url":"https://github.com/SvenMarcus/hpc-rocket","commit_stats":{"total_commits":283,"total_committers":3,"mean_commits":94.33333333333333,"dds":0.04946996466431097,"last_synced_commit":"6c3f4b9e8dc4fa357eec2179aa12e744e5225e72"},"previous_names":["svenmarcus/ssh-slurm-runner"],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SvenMarcus%2Fhpc-rocket","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SvenMarcus%2Fhpc-rocket/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SvenMarcus%2Fhpc-rocket/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SvenMarcus%2Fhpc-rocket/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SvenMarcus","download_url":"https://codeload.github.com/SvenMarcus/hpc-rocket/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246720463,"owners_count":20822908,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ci","continuous-integration","hpc"],"created_at":"2024-08-01T07:01:24.237Z","updated_at":"2025-04-01T22:31:02.924Z","avatar_url":"https://github.com/SvenMarcus.png","language":"Python","funding_links":[],"categories":["Software"],"sub_categories":["Trends"],"readme":"## 🚀 HPC Rocket\n\n[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.7355862.svg)](https://doi.org/10.5281/zenodo.7355862)\n[![Python application](https://github.com/SvenMarcus/ssh-slurm-runner/actions/workflows/python-app.yml/badge.svg)](https://github.com/SvenMarcus/ssh-slurm-runner/actions/workflows/python-app.yml)\n[![Quality Gate Status](https://sonarcloud.io/api/project_badges/measure?project=SvenMarcus_ssh-slurm-runner\u0026metric=alert_status)](https://sonarcloud.io/dashboard?id=SvenMarcus_ssh-slurm-runner)\n[![Coverage](https://sonarcloud.io/api/project_badges/measure?project=SvenMarcus_ssh-slurm-runner\u0026metric=coverage)](https://sonarcloud.io/dashboard?id=SvenMarcus_ssh-slurm-runner)\n[![Python](docs/source/_static/pythonversions.svg)](https://python.org)\n\n\nHPC Rocket is a tool to send slurm commands to a remote machine and monitor the job progress. It was primarily written to launch slurm jobs from a CI pipeline.\n\n![](docs/source/_static/hpc-rocket-demo.gif)\n\n### Installation\n\nYou can get the latest version of HPC Rocket on PyPI:\n```\npython3 -m pip install hpc-rocket\n```\n\n### Authentication\n\nHPC Rocket does support authentication via password and private key. Both can be set via environment variables.\n\n### Slurm configuration\n\nCurrently all `sbatch` configuration must happen in the job file.\nHPC Rocket does not offer any other way of configuring your batch jobs.\n\n### Configuration file\n\nHPC Rocket uses a configuration file in YAML format containing credentials to connect to the remote machine. Additionally it allows copying files to the remote machine, copying results back to the local machine (collecting) and eventually cleaning up copied or produced files.\nNote that all paths in the configuration file must be relative paths.\nOn the local machine paths are evaluated from the current working directory, on the remote machine from the user's home directory, unless absolute paths are specified.\nIf you want to overwrite existing files on the remote machine, make sure to specify the `overwrite` instruction for each file you would like to overwrite.\nHPC Rocket will evaluate environment variables on the **LOCAL** machine in the form of `${VAR}` and `$VAR` when parsing the config file.\n\n```yaml\nhost: cluster.example.com\nuser: myuser\nprivate_keyfile: ~/.ssh/id_rsa\n\nproxyjumps:\n  - host: myproxy.example.com\n    user: myproxy-user\n    private_keyfile: ~/.ssh/proxy_keyfile\n\ncopy:\n  - from: jobs/slurm.job\n    to: slurm.job\n    overwrite: true\n\n  - from: bin/myexecutable\n    to: myexecutable\n\ncollect:\n  - from: remote_slurmresult.out\n    to: local_slurmresult.out\n    overwrite: true\n\nclean:\n  - slurm.job\n  - myexecutable\n\nsbatch: slurm.job\n```\n\n### Usage\n\n#### Launching a job on the remote machine\n\nUse the `launch` command to launch a job on the remote machine. You must provide a configuration file. The optional `--watch` flag makes `hpc-rocket` wait until your job is finished (defaults to `false`). The collection and cleaning steps in the configuration file are only executed if `--watch` is set.\n\n```bash\nhpc-rocket launch --watch config.yml\n```\n\n#### Checking a job's status\n\nIf a job was launched without `--watch` you can still check its status using the `status` command.\nYou will need to provide a configuration file with connection data and a job ID to check.\n\n```bash\nhpc-rocket status config.yml 12345\n```\n\n#### Monitoring a job until it finishes\n\nSimilar to the `status` command, `hpc-rocket` also provides the `watch` command to monitor a job's status continuously by entering a config file and a job id.\n\n```bash\nhpc-rocket watch config.yml 12345\n```\n\n#### Canceling a running job\n\nJobs may also be canceled using the `cancel` command. Like the previous commands it accepts a config file and the id of a running job.\n\n```bash\nhpc-rocket cancel config.yml 12345\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FSvenMarcus%2Fhpc-rocket","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FSvenMarcus%2Fhpc-rocket","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FSvenMarcus%2Fhpc-rocket/lists"}