https://github.com/harryscholes/qsubber
Like qsub, but better
https://github.com/harryscholes/qsubber
Last synced: 3 months ago
JSON representation
Like qsub, but better
- Host: GitHub
- URL: https://github.com/harryscholes/qsubber
- Owner: harryscholes
- Created: 2018-06-05T08:52:47.000Z (about 7 years ago)
- Default Branch: master
- Last Pushed: 2018-08-10T15:08:23.000Z (almost 7 years ago)
- Last Synced: 2025-02-05T10:49:15.551Z (4 months ago)
- Language: Python
- Size: 52.7 KB
- Stars: 0
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
```
_ _
___ ___ _ _| |__ | |__ ___ _ __ TM
/ _` / __| | | | '_ \| '_ \ / _ \ '__|
| (_| \__ \ |_| | |_) | |_) | __/ |
\__, |___/\__,_|_.__/|_.__/ \___|_|
|_| Like qsub, but better
```
`qsubber` is a submission script compiler and job submitter for [SGE](https://en.wikipedia.org/wiki/Oracle_Grid_Engine)/qsub.
## Why?
Submission scripts accumulate quickly when using SGE. These scripts are difficult to organise and maintain.
Most submission scripts contain the specific job being run and some boilerplate setup/teardown code. For example, a typical Python job might look something like this:
```bash
# Setup
...
# Job
python ...
# Teardown
...
```
`qsubber` solves these problems by centralising the boilerplate code and providing an intuitive API to compile submission scripts and submit them to SGE.
## Usage
```
./qsubber JOB_NAME SCRIPT_AND_ARGS [--KWARGS ...]
```
Example:
```
./qsubber long_gpu_job "~/gpu_script.py --foo bar --baz qux" --gpu 1 --rt 5:00:0 --mem 15.5G
```
## Dependencies
* [Fire](https://github.com/google/python-fire)
## Installation
```
pip install fire
echo "export QSUBBERMACHINE=" >> ~/.bash_profile
```
Edit the following files:
**File** | **Contents**
--- | ---
`prologue.sh` | Shell commands to set up the job environment
`gpu.sh` | Source CUDA libraries and set `LD_LIBRARY_PATH`
`epilogue.sh` | Save the job output and tear down the job environment
## Options
Specify options like this: `--PARAMETER VALUE`
**Parameter** | **Alias** | **Default** | **Example value**
--- | --- | --- | ---
`shell` | | `/bin/bash`
`resource_request` | `R` | `y`
`working_directory` | `wd` | `$HOME/results`
`run_time` | `rt` | `1:00:0`
`memory` | `mem` | `1.8G`
`tmpfs` | `scratch` | `10G`
`parallel` | `pe` | `None` | `4`
`gpu` | | `None` | `1`
`submit_command` | `qsub`
`submission_script_path` | | `None` | `$HOME/qsub.sh`
`virtual_env` | `venv` | `None` | `conda_env`
`commit` | | `None` | `2fa78q1`
`results_dir` | `res` | `$HOME/results`
`output_dir` | `out` | `/scratch0/$USER/${JOB_NAME}_${JOB_ID}${sti}`
`buffered` | | `True` | `False` to make `STDOUT` & `STDERR` unbuffered
Setting `submission_script_path` to a path will save the compiled submission script
# 👾