https://github.com/kdm9/snakemake_executor_plugin_mpcdf

snakemake_executor_plugin_mpcdf
https://github.com/kdm9/snakemake_executor_plugin_mpcdf

Last synced: 19 days ago
JSON representation

snakemake_executor_plugin_mpcdf

• Host: GitHub
• URL: https://github.com/kdm9/snakemake_executor_plugin_mpcdf
• Owner: kdm9
• License: mit
• Created: 2024-08-09T17:28:10.000Z (3 months ago)
• Default Branch: main
• Last Pushed: 2024-10-17T11:53:55.000Z (30 days ago)
• Last Synced: 2024-10-19T15:45:06.983Z (28 days ago)
• Language: Python
• Size: 28.3 KB
• Stars: 2
• Watchers: 1
• Forks: 0
• Open Issues: 1
• Metadata Files:
  • Readme: README.md
  • Changelog: CHANGELOG.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# snakemake-executor-plugin-mpcdf

A Snakemake executor plugin for the raven/viper/cobra systems at MPCDF, Garching, DE (the HPC for all Max Planck Institutes).

These clusters only permit running a very small number of Slurm jobs concurrently. This means that with the default limit of 8 jobs, you're limited to at most 8*72 cores. This is profile allows one to submit one massive job, requesting e.g. 24 nodes, and then use job steps to run jobs in parallel within the massive job. This effectively treats the single big job as your own private mini-cluster, within which your jobs can operate almost as per normal. There is one caveat: jobs are limited to at most about 3000 job steps, so use snakemake groups and `--group-components` liberally to batch up smaller jobs.

Nothing about this should be unique to MPCDF systems, so if your slurm cluster has similar restrictions, then this might work well for you too. It's also a bit of a hack, so if it breaks, you get to keep both pieces.

NB: this is not an official MPCDF product, nor is its use sanctioned by them. Their documentation and some direct communication at least indicates that this is allowed, but there could well be a better way to handle this problem.