https://github.com/giannisdaras/slurmify

A minimal python library for handling SLURM jobs
https://github.com/giannisdaras/slurmify

Last synced: about 1 month ago
JSON representation

A minimal python library for handling SLURM jobs

• Host: GitHub
• URL: https://github.com/giannisdaras/slurmify
• Owner: giannisdaras
• License: gpl-3.0
• Created: 2024-08-08T07:36:58.000Z (3 months ago)
• Default Branch: main
• Last Pushed: 2024-09-20T17:50:09.000Z (about 2 months ago)
• Last Synced: 2024-09-21T08:42:56.892Z (about 2 months ago)
• Language: Python
• Homepage:
• Size: 59.6 KB
• Stars: 1
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# 🚀 slurmify: A Python Library to ease your SLURM Workflow! 🚀

Welcome to slurmify, a python library for managing SLURM jobs with style and efficiency! 🎉

## 🌟 Features

- 📊 Submit parametric array jobs with ease

- 🔄 Automatic job resubmission

- 📝 Simplified CLI for common SLURM tasks

## 🛠 Installation

```bash

pip install slurmify

```

## 🚀 Quick Start

Here's a taste of what SLURM Utils can do:

```bash

slurmify submit-parametric-array \

  --job-name awesome_experiment \

  --script-path examples/run_experiment.py \

  --time-limit 01:00:00 \

  --parameter "learning_rate:0.001,0.01,0.1" \

  --parameter "batch_size:32,64,128" \

  --partition "gpu" \

  --nodes=1

```

## 📚 How It Works

1. Create your Python script (`run_experiment.py`) with two essential functions:

   - `setup()`: Prepare your environment. This should be a function that returns a string with the setup commands.

   - `run()`: Define your experiment logic. This should be a function that returns a string with the command to run your experiment.

2. smurmify takes care of the rest! It creates a parametric array job, manages submissions, and handles resubmissions if needed.

## 🎭 Example Script

Here's a simple template for your `run_experiment.py`:

```python

import os

def setup():

    setup_cmd = """

    source ~/.bashrc

    conda activate myenv

    module load cuda/11.3

    """

    print(setup_cmd)

    return setup_cmd

def run():

    learning_rate = float(os.environ["LEARNING_RATE"])

    batch_size = int(os.environ["BATCH_SIZE"])

    

    cmd = f"python train.py --lr {learning_rate} --batch-size {batch_size}"

    print(cmd)

    return cmd

if __name__ == "__main__":

    run()

```

## 🎉 Happy SLURMing!

Now go forth and conquer those clusters! 🏆