{"id":13732935,"url":"https://github.com/jdblischak/smk-simple-slurm","last_synced_at":"2025-05-06T21:28:48.045Z","repository":{"id":42063156,"uuid":"363290003","full_name":"jdblischak/smk-simple-slurm","owner":"jdblischak","description":"A simple Snakemake profile for Slurm without --cluster-config","archived":false,"fork":false,"pushed_at":"2024-06-20T20:29:02.000Z","size":109,"stargazers_count":153,"open_issues_count":0,"forks_count":19,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-03-31T03:08:59.827Z","etag":null,"topics":["bioinformatics","slurm","snakemake","snakemake-profile"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"cc0-1.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jdblischak.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-05-01T00:24:58.000Z","updated_at":"2025-03-21T09:48:10.000Z","dependencies_parsed_at":"2024-02-06T03:06:35.211Z","dependency_job_id":"5993aa13-41a6-43a3-a3dd-c7fb869df8d5","html_url":"https://github.com/jdblischak/smk-simple-slurm","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jdblischak%2Fsmk-simple-slurm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jdblischak%2Fsmk-simple-slurm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jdblischak%2Fsmk-simple-slurm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jdblischak%2Fsmk-simple-slurm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jdblischak","download_url":"https://codeload.github.com/jdblischak/smk-simple-slurm/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252771474,"owners_count":21801718,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bioinformatics","slurm","snakemake","snakemake-profile"],"created_at":"2024-08-03T03:00:35.687Z","updated_at":"2025-05-06T21:28:48.023Z","avatar_url":"https://github.com/jdblischak.png","language":"Shell","funding_links":[],"categories":["Workflow managers and  distributed computing frameworks"],"sub_categories":[],"readme":"# Simple Slurm\n\n\u003e A simple Snakemake profile for Slurm without `--cluster-generic-*-cmd`\n\n- [Simple Slurm](#simple-slurm)\n  - [Features](#features)\n  - [Limitations](#limitations)\n  - [Quick start](#quick-start)\n  - [Customizations](#customizations)\n    - [A fixed argument to `sbatch`, e.g. `--account`](#a-fixed-argument-to-sbatch-eg---account)\n    - [A variable argument to `sbatch`, e.g. `--time`](#a-variable-argument-to-sbatch-eg---time)\n    - [Using a cluster status script](#using-a-cluster-status-script)\n    - [Multiple clusters](#multiple-clusters)\n  - [Use speed with caution](#use-speed-with-caution)\n  - [License](#license)\n\nThe option [`--cluster-config`][cluster-config] is removed in `snakemake\u003e8.0.0`, but it's still\npossible to set default and rule-specific resources for [submitting jobs to a\nremote scheduler][cluster-execution] using a combination of\n`--default-resources` and the `resources` field in individual rules. This\nprofile is a simplified alternative to the more comprehensive [official Slurm\nprofile for Snakemake][slurm-official]. For more background, this [blog\npost][sichong-post] by Sichong Peng nicely explains this strategy for replacing\n`--cluster-config`.\n\n\u003e [!WARNING]\n\u003e The Slurm profile and documentation in this repository have been updated to only support\n\u003e Snakemake versions \u003e= 8.0.0. This is because Snakemake 8\n\u003e completely overhauled how it submits jobs to external clusters, which broke\n\u003e this and all the other existing profiles.\n\u003e If you plan to continue to use Snakemake 7, you can find the Snakemake 7\n\u003e version of the docs in the [v7 branch of this repository][v7].\n\n## Features\n\n* Only requires a single configuration file to get started submitting jobs to\n  Slurm\n\n* Easily add or remove options to pass to `sbatch`\n\n* Automatically saves the log files as `logs/{rule}/{rule}-{wildcards}-%j.out`,\n  where `{rule}` is the name of the rule, `{wildcards}` is any wildcards passed\n  to the rule, and `%j` is the job number\n\n* Automatically names jobs using the pattern `smk-{rule}-{wildcards}`\n\n* Fast! It can quickly submit jobs and check their status because it doesn't\n  invoke a Python script for these steps, which adds up when you have thousands\n  of jobs (however, please see the section [Use speed with\n  caution](#use-speed-with-caution))\n\n* No reliance on the now-removed option `--cluster-config` or hard-to-read command-line flags (`--cluster-generic-*-cmd`)\n  to customize job resources\n\n* By default it relies on Snakemake's built-in ability to understand the job\n  statuses PENDING, RUNNING, COMPLETING, and OUT_OF_MEMORY\n\n* (Optional, but recommended) You can pass a simple script (see\n  [`extras/`](extras/)) to [`--cluster-generic-status-cmd`][cluster-status] to additionally\n  handle the job statuses TIMEOUT and CANCELED\n\n* **New** Support for [cluster-cancel][] feature introduced in Snakemake 7.0.0\n  (see [`examples/cluster-cancel/`](examples/cluster-cancel/))\n\n* **New** Full support for [multi-cluster setups][multi_cluster] (using a custom\n  status script requires Snakemake 7.1.1+). See the section [Multiple\n  clusters](#multiple-clusters) below\n\n* **New** Adaptable for use with [AWS ParallelCluster][aws-parallelcluster]. See\n  [Christian Brueffer's][cbrueffer] profile\n  [snakemake-aws-parallelcluster-slurm][]\n\n## Limitations\n\n* If you use [job grouping][grouping], then you can't dynamically name the jobs\n  and log files based on the name of the rules. This doesn't prevent you from\n  using this profile and benefiting from its other features, but it is less\n  convenient. Also note that [job grouping isn't easy to use in the first\n  place][grouping-issue], since it sums resources like `mem_mb` and `threads`,\n  but that is a limitation of Snakemake itself, and not anything in particular\n  with this profile **UPDATE:** As of Snakemake 7.11, there is improved support\n  for [managing the maximum resources requested when submitting a grouped\n  job][resources-and-group-jobs] that executes multiple rules. It's still\n  non-trivial, but now at least possible. See the example in\n  [`examples/job-grouping/`](examples/job-grouping/) for a demonstration of how\n  to use the new features\n\n* Wildcards can't contain `/` if you want to use them in the name of the Slurm\n  log file. This is a Slurm requirement (which makes sense, since it has to\n  create a file on the filesystem). You'll either have to change how you manage\n  the wildcards or remove the `{wildcards}` from the pattern passed to `--output`,\n  e.g. `--output=logs/{rule}/{rule}-%j.out`.\n  Note that you can still submit wildcards containing `/` to `--job-name`\n\n* Requires Snakemake version 8.0.0 or later (released 2023-12-20, see\n  [changelog](https://github.com/snakemake/snakemake/blob/main/CHANGELOG.md#800-2023-12-20)). You can test this directly in your `Snakefile` with\n  [`min_version()`][min_version]. If you require an older version of Snakemake, please see the [`v7` branch](https://github.com/jdblischak/smk-simple-slurm/tree/v7)\n\n## Quick start\n\n1. Download the configuration file [`config.v8+.yaml`](simple/config.v8+.yaml) to your\n   Snakemake project. It has to be in a subdirectory, e.g. `simple/`\n\n1. Open it in your favorite text editor and replace all the placeholders\n   surrounded in angle brackets (`\u003c\u003e`) with the options you use to submit jobs\n   on your cluster\n\n1. You can override any of the defaults by adding a `resources` field to a rule,\n   e.g.\n\n    ```python\n    rule much_memory:\n        resources:\n            mem_mb=64000\n    ```\n\n1. Invoke snakemake with the profile:\n\n    ```sh\n    snakemake --profile simple/\n    ```\n\n## Customizations\n\nSee the directory [`examples/`](examples/) for examples you can experiment with\non your cluster.\n\n### A fixed argument to `sbatch`, e.g. `--account`\n\nTo pass an additional argument to `sbatch` that will be fixed across all job\nsubmissions, add it directly to the arguments passed to `sbatch` in the field\n`cluster-generic-submit-cmd`. For example, to specify an account to use for all job submissions, you can add the `--account` argument as shown below:\n\n```yaml\nexecutor: cluster-generic\ncluster-generic-submit-cmd:\n  mkdir -p logs/{rule} \u0026\u0026\n  sbatch\n    --partition={resources.partition}\n    --qos={resources.qos}\n    --cpus-per-task={threads}\n    --mem={resources.mem_mb}\n    --job-name=smk-{rule}-{wildcards}\n    --output=logs/{rule}/{rule}-{wildcards}-%j.out\n    --account=myaccount\n```\n\n### A variable argument to `sbatch`, e.g. `--time`\n\nTo pass an additional argument to `sbatch` that can vary across job submissions,\nadd it to the arguments passed to `sbatch` in the field `cluster`, list a\ndefault value in the field `default-resources`, and update any rules that\nrequire a value different from the default.\n\nFor example, the `config.v8+.yaml` below sets a default time of 1 hour, and the\nexample rule overrides this default for a total of 3 hours. Note that the quotes\naround the default time specification are required, even though you don't need\nquotes when specifying the default for either `partition` or `qos`.\n\n```yaml\nexecutor: cluster-generic\ncluster-generic-submit-cmd:\n  mkdir -p logs/{rule} \u0026\u0026\n  sbatch\n    --partition={resources.partition}\n    --qos={resources.qos}\n    --cpus-per-task={threads}\n    --mem={resources.mem_mb}\n    --job-name=smk-{rule}-{wildcards}\n    --output=logs/{rule}/{rule}-{wildcards}-%j.out\n    --time={resources.time}\ndefault-resources:\n  - partition=\u003cname-of-default-partition\u003e\n  - qos=\u003cname-of-quality-of-service\u003e\n  - mem_mb=1000\n  - time=\"01:00:00\"\n```\n\n```python\n# A rule in Snakefile\nrule more_time:\n    resources:\n        time = \"03:00:00\"\n```\n\nNote that `sbatch` accepts time defined using various formats. Above I used\n`hours:minutes:seconds`, but the simple slurm profile is agnostic to how you\nchoose to configure this. It's a good idea to be consistent across rules, but\nit's not required. From Slurm 19.05.7:\n\n\u003e A time limit of zero requests that no time limit be imposed. Acceptable time\n\u003e formats include \"minutes\", \"minutes:seconds\", \"hours:minutes:seconds\",\n\u003e \"days-hours\", \"days-hours:minutes\" and \"days-hours:minutes:seconds\".\n\nThus to instead use `minutes`, you could achieve the same effect as above with:\n\n```yaml\nexecutor: cluster-generic\ncluster-generic-submit-cmd:\n  mkdir -p logs/{rule} \u0026\u0026\n  sbatch\n    --partition={resources.partition}\n    --qos={resources.qos}\n    --cpus-per-task={threads}\n    --mem={resources.mem_mb}\n    --job-name=smk-{rule}-{wildcards}\n    --output=logs/{rule}/{rule}-{wildcards}-%j.out\n    --time={resources.time}\ndefault-resources:\n  - partition=\u003cname-of-default-partition\u003e\n  - qos=\u003cname-of-quality-of-service\u003e\n  - mem_mb=1000\n  - time=60\n```\n\n```python\n# A rule in Snakefile\nrule more_time:\n    resources:\n        time = 180\n```\n\nSee [`examples/time-integer/`](examples/time-integer/) and\n[`examples/time-string/`](examples/time-string/) for examples you can play with.\nNote that specifying the time as a string requires a minimum Snakemake version\nof 5.15.0.\n\n### Using a cluster status script\n\nBy default, snakemake can monitor jobs submitted to slurm. I realized this when\nreading this detailed [blog post][no-cluster-status], in which the author\ndecided not to use the `cluster-status.py` script provided by the [official\nSlurm profile][slurm-official]. Thus if you don't find that your jobs are\nsilently failing often, then there's no need to worry about this extra\nconfiguration step.\n\nHowever, if you start to have jobs silently fail often, e.g. with status\n`TIMEOUT` for exceeding their time limit, then you can add a custom script to\nmonitor the job status with the option [`--cluster-generic-status-cmd`][cluster-status].\n\nThe directory [`extras/`](extras/) contains multiple options for checking the\nstatus of the jobs. You can choose which one you'd like to use:\n\n* `status-sacct.py` - This is the example from the [Snakemake\n  documentation][cluster-status]. It uses `sacct` to query the status of each\n  job by its ID\n\n* `status-sacct.sh` - (**recommended**) This is a Bash translation of the example\n  from the [Snakemake documentation][cluster-status]. The Python script is\n  simply shell-ing out to `sacct`, so running Bash directly removes the overhead\n  of repeatedly starting Python each time you check a job\n\n* `status-scontrol.sh` - This is a Bash script that uses `scontrol` to query the\n  status of each job by its ID. The `scontrol` command is from\n  [`slurm-status.py`](https://github.com/Snakemake-Profiles/slurm/blob/master/%7B%7Bcookiecutter.profile_name%7D%7D/slurm-status.py#L35)\n  in the [official profile][slurm-official]. If your HPC cluster doesn't have\n  `sacct` configured, you can use this option\n\n* `status-sacct-multi.sh` - Support for multi-cluster setup (see section\n  [Multiple clusters](#multiple-clusters))\n\nTo use one of these status scripts:\n\n1. Download the script to your profile directory where `config.yaml` is located\n\n1. Make the script executable, e.g. `chmod +x status-sacct.sh`\n\n1. Add the field `cluster-generic-status-cmd` to your `config.yaml`, e.g. `cluster-generic-status-cmd: status-sacct.sh`\n\n1. Add the flag `--parsable` to your `sbatch` command (requires Slurm version\n   14.03.0rc1 or later)\n\n### Multiple clusters\n\nIt's possible for Slurm to submit jobs to [multiple different\nclusters][multi_cluster]. Below is my advice on how to configure this. However,\nI've worked with multiple HPC clusters running Slurm, and have never encountered\nthis situation. Thus I'd appreciate any contributions to improve the\ndocumentation below.\n\n1. If you have access to multiple clusters, but only need to submit jobs to the\n   default cluster, then you shouldn't have to modify anything in this profile\n\n1. If you want to always submit your jobs to a cluster other than the default,\n   or use multiple clusters, then pass the option `--clusters` to `sbatch`, e.g.\n   to submit your jobs to either cluster \"c1\" or \"c2\"\n\n    ```yaml\n    # config.v8+.yaml\n    executor: cluster-generic\n    cluster-generic-submit-cmd:\n      mkdir -p logs/{rule} \u0026\u0026\n      sbatch\n        --clusters=c1,c2\n    ```\n\n1. To set a default cluster and override it for specific rules, use\n   `--default-resources`. For example, to run on \"c1\" by default but \"c2\" for a\n   specific rule:\n\n    ```yaml\n    # config.v8+.yaml\n    executor: cluster-generic\n    cluster-generic-submit-cmd:\n      mkdir -p logs/{rule} \u0026\u0026\n      sbatch\n        --clusters={resources.clusters}\n    default-resources:\n      - clusters=c1\n    ```\n\n    ```python\n    # Snakefile\n    rule different_cluster:\n        resources:\n            clusters=\"c2\"\n    ```\n\n1. Using a custom cluster status script in a multi-cluster setup requires\n   Snakemake 7.1.1+ (or Snakemake 8.0.0+ if you are using the new `--cluster-generic-*-cmd` flags). After you add the flag `--parsable` to `sbatch`, it will return `jobid;cluster_name`. I adapted `status-sacct.sh` to handle this\n   situation. Please see [`examples/multi-cluster/`](examples/multi-cluster) to\n   try out `status-sacct-multi.sh`\n\n## Use speed with caution\n\nA big benefit of the simplicity of this profile is the speed in which jobs can\nbe submitted and their statuses checked. The [official Slurm profile for\nSnakemake][slurm-official] provides a lot of extra fine-grained control, but\nthis is all defined in Python scripts, which then have to be invoked for each\njob submission and status check. I needed this speed for a pipeline that had an\naggregation rule that needed to be run tens of thousands of times, and the run\ntime for each job was under 10 seconds. In this situation, the job submission\nrate and status check rate were huge bottlenecks.\n\nHowever, you should use this speed with caution! On a shared HPC cluster, many\nusers are making requests to the Slurm scheduler. If too many requests are made\nat once, the performance will suffer for all users. If the rules in your\nSnakemake pipeline take at least more than a few minutes to complete, then it's\noverkill to constantly check the status of multiple jobs in a single second. In\nother words, only increase `max-jobs-per-second` and/or\n`max-status-checks-per-second` if either the submission rate or status checks to\nconfirm job completion are clear bottlenecks.\n\n## License\n\nThis is all boiler plate code. Please feel free to use it for whatever purpose\nyou like. No need to attribute or cite this repo, but of course it comes with no\nwarranties. To make it official, it's released under the [CC0][] license. See\n[`LICENSE`](LICENSE) for details.\n\n[aws-parallelcluster]: https://aws.amazon.com/hpc/parallelcluster/\n[cbrueffer]: https://github.com/cbrueffer\n[CC0]: https://creativecommons.org/publicdomain/zero/1.0/\n[changelog]: https://snakemake.readthedocs.io/en/stable/project_info/history.html\n[cluster-cancel]: https://snakemake.readthedocs.io/en/stable/tutorial/additional_features.html#using-cluster-cancel\n[cluster-config]: https://snakemake.readthedocs.io/en/stable/snakefiles/configuration.html#cluster-configuration-deprecated\n[cluster-execution]: https://snakemake.readthedocs.io/en/stable/executing/cluster.html\n[cluster-status]: https://snakemake.readthedocs.io/en/stable/tutorial/additional_features.html#using-cluster-status\n[grouping]: https://snakemake.readthedocs.io/en/stable/executing/grouping.html\n[grouping-issue]: https://github.com/snakemake/snakemake/issues/872\n[min_version]: https://snakemake.readthedocs.io/en/stable/snakefiles/writing_snakefiles.html#depend-on-a-minimum-snakemake-version\n[multi_cluster]: https://slurm.schedmd.com/multi_cluster.html\n[no-cluster-status]: http://bluegenes.github.io/Using-Snakemake_Profiles/\n[resources-and-group-jobs]: https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#resources-and-group-jobs\n[sichong-post]: https://www.sichong.site/workflow/2021/11/08/how-to-manage-workflow-with-resource-constraint.html\n[slurm-official]: https://github.com/Snakemake-Profiles/slurm\n[snakemake-aws-parallelcluster-slurm]: https://github.com/cbrueffer/snakemake-aws-parallelcluster-slurm\n[v7]: https://github.com/jdblischak/smk-simple-slurm/tree/v7\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjdblischak%2Fsmk-simple-slurm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjdblischak%2Fsmk-simple-slurm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjdblischak%2Fsmk-simple-slurm/lists"}