{"id":18439413,"url":"https://github.com/idiap/gridtk","last_synced_at":"2025-08-23T06:11:41.722Z","repository":{"id":247441087,"uuid":"824489049","full_name":"idiap/gridtk","owner":"idiap","description":"GridTK: Slurm Job Management for Humans","archived":false,"fork":false,"pushed_at":"2025-05-01T20:36:01.000Z","size":578,"stargazers_count":5,"open_issues_count":4,"forks_count":0,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-08-22T15:18:09.646Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/idiap.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"COPYING","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-05T08:40:31.000Z","updated_at":"2025-07-22T13:45:52.000Z","dependencies_parsed_at":"2024-07-08T20:56:19.602Z","dependency_job_id":"3d71845a-9d82-4e9a-b939-d62374895479","html_url":"https://github.com/idiap/gridtk","commit_stats":null,"previous_names":["idiap/gridtk"],"tags_count":5,"template":false,"template_full_name":null,"purl":"pkg:github/idiap/gridtk","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/idiap%2Fgridtk","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/idiap%2Fgridtk/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/idiap%2Fgridtk/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/idiap%2Fgridtk/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/idiap","download_url":"https://codeload.github.com/idiap/gridtk/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/idiap%2Fgridtk/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271745694,"owners_count":24813517,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-23T02:00:09.327Z","response_time":69,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-06T06:24:42.259Z","updated_at":"2025-08-23T06:11:41.708Z","avatar_url":"https://github.com/idiap.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003c!--\nSPDX-FileCopyrightText: 2024 Idiap Research Institute \u003ccontact@idiap.ch\u003e\nSPDX-FileContributor: Amir Mohammadi  \u003camir.mohammadi@idiap.ch\u003e\n\nSPDX-License-Identifier: GPL-3.0-or-later\n--\u003e\n\n[![docs](https://img.shields.io/badge/docs-stable-orange.svg)](https://gridtk.readthedocs.io/en/stable/)\n[![tests](https://github.com/idiap/gridtk/actions/workflows/tests.yml/badge.svg)](https://github.com/idiap/gridtk/actions/workflows/tests.yml)\n[![coverage](https://raw.githubusercontent.com/idiap/gridtk/python-coverage-comment-action-data/badge.svg)](https://htmlpreview.github.io/?https://github.com/idiap/gridtk/blob/python-coverage-comment-action-data/htmlcov/index.html)\n[![repository](https://img.shields.io/badge/github-project-0000c0.svg)](https://github.com/idiap/gridtk)\n\n# GridTK: Slurm Job Management for Humans\n\n## Introduction\n\nGridTK is a powerful command-line tool designed to simplify the management of\nSlurm jobs. At its core, GridTK provides a drop-in replacement for `sbatch`,\n`gridtk submit`, which allows you to get started quickly. This\ntutorial will guide you through the process of using the `gridtk` script to\nefficiently manage your Slurm workloads. We will cover the basics of\ninstallation, submission, monitoring, and various commands provided by GridTK.\n\n## Prerequisites\n\nBefore diving into GridTK, ensure you have the following prerequisites:\n\n1. A working Slurm setup.\n2. [pipx](https://pipx.pypa.io/stable/) installed.\n3. GridTK installed (instructions provided below).\n\n## Installation\n\nTo install GridTK, open your terminal and run the following command:\n\n```bash\n$ pipx install gridtk\n```\nIt is **not recommended** to install GridTK using `pip install gridtk` in the\nsame environment as your experiments. GridTK does not need to be installed in\nthe same environment as your experiments and its dependencies may conflict with\nyour experiments' dependencies.\n\n## Basic Usage\n\nIn this section, we will cover the basic commands and usage of the GridTK\nscript. The primary goal is to help you get familiar with submitting,\nmonitoring, and managing your Slurm jobs using GridTK.\n\n### Submitting a Job\n\nTo submit a job script, use `gridtk submit`. For example, given the script (`job.sh`) below:\n\n```bash\n#!/bin/bash\n\necho \"Hello, GridTK!\"\n```\n\nSubmit the job using `gridtk submit`:\n\n```bash\n$ gridtk submit job.sh\n1\n```\nwhere `1` is the local job id (not the slurm job id) for your job. The job\nnumbers always start with 1 which is easier to remember than the slurm job id.\n\n`gridtk submit` is a drop-in replacement for `sbatch` and accepts the same options while adding its own.\nRun `gridtk submit --help` to see the list of `gridtk submit` specific options and run `sbatch --help` to see the full list of options for `sbatch`.\n\nNote that your slurm cluster may require you to specify a partition, an account,\nor another option. You can do so by adding them to `gridtk submit --account=myaccount --partition=mypartition job.sh`\nor setting default values using environment variables such as\n`SBATCH_ACCOUNT` and `SBATCH_PARTITION`.\n\n### Monitoring Jobs\n\nUse the `gridtk list` command to view the status of your jobs:\n```bash\n$ gridtk list\n  job-id    grid-id  nodes    state        job-name    output                  dependencies    command\n--------  ---------  -------  -----------  ----------  ----------------------  --------------  --------------------\n       1     136132  None     PENDING (0)  gridtk      logs/gridtk.136132.out                  gridtk submit job.sh\n```\n`gridtk list` will only show jobs that are submitted using `gridtk submit` **in the current folder**.\nYou can see the submitted job got a local job id of `1` and a slurm job id of\n`136132`.\nIt is in the `PENDING` state and its name is `gridtk` by default (it is\nrecommended to give a meaningful name using the `gridtk submit --job-name`\noption).\nThe output files are written to the `logs/` directory by default (you may change\nthe directory with the `gridtk --logs-dir` option).\nGridTK manages the log files for you, so you don't have to worry about knowing\nwhere they are stored or cleaning them up.\n\nFor detailed information about a specific job, use the `report` command:\n```bash\n$ gridtk report -j 1\nJob ID: 1\nName: gridtk\nState: COMPLETED (0)\nNodes: None\nSubmitted command: ['sbatch', '--job-name', 'gridtk', '--output', 'logs/gridtk.%j.out', '--error', 'logs/gridtk.%j.out', 'job.sh']\nOutput file: logs/gridtk.136132.out\nHello, GridTK!\n```\nwhere you can see the exact sbatch command that was used to submit the job and\nthe output of the job.\n\n### Stopping and Deleting a Job\n\nTo stop a running or pending job, use the `gridtk stop` command:\n\n```bash\n$ gridtk stop -j 1\nStopped job 1 with slurm id 136132\n```\n\nStopped jobs will be still available in the job list:\n```bash\n$ gridtk list\n  job-id    slurm-id  nodes    state          job-name    output                  dependencies    command\n--------  ----------  -------  -------------  ----------  ----------------------  --------------  --------------------\n       1      136137  None     CANCELLED (0)  gridtk      logs/gridtk.136137.out                  gridtk submit job.sh\n```\nand can be resubmitted using the `gridtk resubmit` command (more details on\nresubmit further down) and you can still view their output using the `gridtk report`\ncommand.\n\nTo delete a job (and its log file), use the `gridtk delete` command:\n```bash\n$ gridtk delete -j 1\nDeleted job 1 with slurm id 136137\n```\n\n### Resubmitting a Job\n\nIf a job fails or is stopped, you can resubmit it using the `gridtk resubmit` command:\n```bash\n$ gridtk submit job.sh\n1\n\n$ gridtk stop -j 1\nStopped job 1 with slurm id 136139\n\n$ gridtk resubmit -j 1\nResubmitted job 1\n\n$ gridtk list\n  job-id    slurm-id  nodes    state        job-name    output                  dependencies    command\n--------  ----------  -------  -----------  ----------  ----------------------  --------------  --------------------\n       1      136140  None     PENDING (0)  gridtk      logs/gridtk.136140.out                  gridtk submit job.sh\n```\nNotice how the resubmitted job got a new slurm job id of `136140`.\n\n## Advanced Usage\n\nGridTK provides several advanced commands to help with more complex job\nmanagement tasks. These include job dependencies, array jobs, and resource\nmanagement.\n\n### Job Submission without a Script\n\nSince GridTK keeps track of both the sbatch options and the command to run, you\ncan skip creating a script and submit a job directly from the command line.\nThis is done by using `---` (3 dashes) to separate the sbatch options from the command to\nrun:\n```bash\n$ gridtk submit --job-name=gridtk-no-script --- echo 'Hello, GridTK!'\n2\n```\nThis syntax is unique to `gridtk submit` and is not supported by `sbatch`.\n```bash\n$ gridtk list\n  job-id    slurm-id  nodes    state        job-name          output                            dependencies    command\n--------  ----------  -------  -----------  ----------------  --------------------------------  --------------  ------------------------------------\n       1      136140  None     PENDING (0)  gridtk            logs/gridtk.136140.out                            gridtk submit job.sh\n       2      136142  None     PENDING (0)  gridtk-no-script  logs/gridtk-no-script.136142.out                  gridtk submit --- echo Hello, GridTK!\n```\nWhat happens is that `gridtk submit` creates a temporary script with the command to run and\nsubmits it to slurm. The temporary script is deleted after the job is submitted. The content of\nthis temporary script can be viewed using the `gridtk report` command:\n```bash\n$ gridtk report -j 2\nJob ID: 2\nName: gridtk-no-script\nState: PENDING (0)\nNodes: None\nSubmitted command: ['sbatch', '--job-name', 'gridtk-no-script', '--output', 'logs/gridtk-no-script.%j.out', '--error', 'logs/gridtk-no-script.%j.out', '/tmp/tmpegoy2ma1.sh']\nContent of the temporary script:\n#!/bin/bash\necho 'Hello, GridTK!'\n\nOutput file: logs/gridtk-no-script.136142.out\n```\nThis is a fast, convenient, and **recommended** way to submit a job without having to create a\nscript and since everything is tracked by GridTK, you still benefit from the same\nreproducibility guarantees.\n\n### Environment Variables for CLI Options\n\nWhile sbatch already allows providing values for some options through `SBATCH_` prefixed environment variables, not all options are supported. GridTK adds support for all CLI options through the `GRIDTK_SUBMIT_` prefix. For example:\n\n```bash\n# Set default email notification settings\nexport GRIDTK_SUBMIT_MAIL_USER=your.email@example.com\nexport GRIDTK_SUBMIT_MAIL_TYPE=END\n\n# Use debug partition by default\nexport GRIDTK_SUBMIT_PARTITION=debug\n\n# Now submit a job - it will use these settings automatically\ngridtk submit job.sh\n```\n\nThis is useful when you have a set of options that you always want to use, but don't want to specify them every time you submit a job.\n\nOf course, other gridtk commands such as `gridtk resubmit` options can also be set using environment variables like: `export GRIDTK_RESUBMIT_STATE=ALL`.\n\n### Job Dependencies\n\nTo submit a job that depends on another job, use the `--dependency` flag:\n\n```bash\n$ gridtk submit --dependency=\u003cjob_id\u003e job.sh\n```\nThe `--dependency` flag takes the same values as in `sbatch` except that you\nneed to specify local job ids instead of slurm job ids.\n\n### Repeat Jobs\n\nYou can submit the same script N times using the `--repeat` flag:\n```bash\n$ gridtk submit --repeat=3 job.sh\n```\nThis will submit 3 jobs with the same script and the same options where each job\nwill depend on the previous one. This is useful if your script can resume from\na checkpoint and you want to run it effectively for a longer time than allowed\nby policy.\n\n### Monitoring Jobs\n\nWhile `gridtk list` and `gridtk report` are useful for checking the status of jobs,\nyou might get more information about your jobs using `squeue`, `scontrol`, and `sacct`.\nHere are some useful commands:\n\n* Get information about a specific job: `scontrol show job \u003cslurm_job_id\u003e`\n* Get information about a completed or failed job: `sacct -j \u003cslurm_job_id\u003e`.\n* See ALL your jobs: `squeue --me`\n* Cancel ALL your jobs: `scancel --me`\n* View current QOS policies:\n  ```bash\n  sacctmgr show qos format=Name%20,Priority,Flags%30,MaxWall,MaxTRESPU%20,MaxJobsPU,MaxSubmitPU,MaxTRESPA%25\n  ```\n* Find out which accounts your username has access to:\n  ```bash\n  sacctmgr list associations\n  # or\n  sacctmgr -n -p list assoc where user=$USER | awk '-F|' '{print \"   \"$2}'\n  ```\n\n### Tab Completion\n\nGridTK supports tab completion for the `gridtk` command. To enable it, add the following\nline to your `~/.bashrc` file:\n```bash\neval \"$(_GRIDTK_COMPLETE=bash_source gridtk)\"\n```\nor for `zsh` add the following line to your `~/.zshrc` file:\n```bash\neval \"$(_GRIDTK_COMPLETE=zsh_source gridtk)\"\n```\n\n### Adjusting `gridtk list` Output\n\nBy default, `gridtk list` outputs a table which migh not fit the terminal width.\nYou can adjust the output using the `--wrap` and `--truncate` flags. The `--wrap`\nflag wraps the output to fit the terminal width, while the `--truncate` flag\ntruncates the output to fit the terminal width.\n\n```bash\n$ gridtk list\n  job-id    slurm-id  nodes    state          job-name    output                  dependencies    command\n--------  ----------  -------  -------------  ----------  ----------------------  --------------  --------------------\n       1      506994  hcne01   COMPLETED (0)  gridtk      logs/gridtk.506994.out                  gridtk submit job.sh\n\n$ gridtk list --wrap  # --wrap or -w\n  job-id    slurm-  nodes    state     job-name    output             depende    command\n                id                                                    ncies\n--------  --------  -------  --------  ----------  -----------------  ---------  -------------\n       1    506994  hcne0    COMPLETE  gridtk      logs/gridtk.50699             gridtk submit\n                    1        D (0)                 4.out                         job.sh\n\n$ gridtk list --truncate # --truncate or -t\n  job-id    slur..  nodes    state    job-name    output            depe..    command\n--------  --------  -------  -------  ----------  ----------------  --------  -------------\n       1    506994  hc..     COMPL..  gridtk      logs/gridtk.50..            gridtk subm..\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fidiap%2Fgridtk","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fidiap%2Fgridtk","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fidiap%2Fgridtk/lists"}