https://github.com/brentp/slurmpy

submit jobs to slurm with quick-and-dirty python
https://github.com/brentp/slurmpy

Last synced: about 2 months ago
JSON representation

submit jobs to slurm with quick-and-dirty python

• Host: GitHub
• URL: https://github.com/brentp/slurmpy
• Owner: brentp
• License: mit
• Created: 2015-05-22T17:49:14.000Z (over 9 years ago)
• Default Branch: master
• Last Pushed: 2023-01-03T22:49:42.000Z (over 1 year ago)
• Last Synced: 2024-07-14T20:20:55.916Z (2 months ago)
• Language: Python
• Size: 14.6 KB
• Stars: 92
• Watchers: 7
• Forks: 33
• Open Issues: 7
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

• awesome-institutional-hpc - slurmpy - Small Python library for starting SLURM jobs (SLURM wrappers and libraries)

README

        
quick and dirty lib for submitting jobs to slurm via python2/python3.

```Python

from slurmpy import Slurm

s = Slurm("job-name", {"account": "my-account", "partition": "my-parition"})

s.run("""

do

lots

of

stuff

""")

```

The above will submit the job to `sbatch` automatically write the script to `scripts/`

and automatically write logs/{name}.err and logs/{name}.out. It will have today's

date in the log and script names.

The script to run() can also contain `$variables` which are filled with the cmd_kwarg dict.

E.g. `echo $name` could be filled with `cmd_kwargs={'name': 'sally'}`

A command can be tested (not sent to queue) by setting the `_cmd` are to `run` as e.g. "ls".

The default is `sbatch` which submits jobs to slurm.

Dependencies

============

Each time `slurmpy.Slurm().run()` is called, it returns the job-id of the submitted job. One or more

of these can be provided as arguments to a subsequent job:

```

s = Slurm()

s.run(..., depends_on=[job_id])

# or for multiple dependencies

s.run(..., depends_on=[job_id_1, job_id_2, job_id_3])

```

to indicate that this job should not run until the the job with `job_id` (or ids `job_id_1-3`) has finished successfully.

Additionally, SLURM allows you to define how a job depends on previous jobs e.g. should it run 

only on success or always etc. For this `slurmpy.Slurm().run()` provides an additional argument `depends_how`:

```

s = Slurmp()

s.run(..., depends_on=[job_id], depends_how='afterany')

```

currently supported are the following options with "afterok" as default: 

"after", "afterany", "afterburstbuffer", "aftercorr", "afternotok", "afterok", "expand"

More details can be found here: https://slurm.schedmd.com/sbatch.html in the `--dependency` section.

Install

=======

```Shell

pip install slurmpy --user

```