https://github.com/cdancette/pyrunner

Python task runner : wrap your scripts to make them idempotent and avoid unwanted parallel runs
https://github.com/cdancette/pyrunner

Last synced: 10 days ago
JSON representation

Python task runner : wrap your scripts to make them idempotent and avoid unwanted parallel runs

• Host: GitHub
• URL: https://github.com/cdancette/pyrunner
• Owner: cdancette
• Created: 2018-05-24T09:16:51.000Z (over 6 years ago)
• Default Branch: master
• Last Pushed: 2018-06-04T15:39:21.000Z (over 6 years ago)
• Last Synced: 2024-10-12T18:27:17.512Z (about 1 month ago)
• Language: Python
• Homepage:
• Size: 13.7 KB
• Stars: 5
• Watchers: 3
• Forks: 1
• Open Issues: 2
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# pyrunner

A simple Python task runner. Use it as a CLI, or wrap your own scripts with 

the python interface.

Features

- Prevent a task to be running twice simultaneously.

- Prevent a task to be launched again when it is already completed.

It implements this by writing token files `.done` and `.running` in a 

specified task directory. One directory = One task.

### Example use cases

##### Crash recovery

Say you launched multiple tasks for all subdirectories: `ls | xargs -I % command %`, 

and the command crashes in the middle. Now how do you run only the unfinished tasks ?

You can use pyrunner to run the your tasks, and then the second run will only start 

unfinished tasks.

##### Parallel

It can also be used as a poor man's parallel library.

Instead of `ls | xargs -P 2 -I % command %` you can run `ls | xargs -I % pyrunner %/.tokens command %` in two shells.

Each shell will pick up the pending tasks.

## Installation

`pip install pyrunner`

## CLI usage

You can use it as a cli, like this : 

```bash

pyrunner  

```

For example

```bash

pyrunner .tokens/ touch file.txt

```

Will create a file called file.txt.

Another example : 

```bash

pyrunner .tokens/ bash -c "date > date.txt"

```

This command will save the current date in the file `date.txt`. 

If you run it again it will not run.

## Python Usage

You can also wrap your script in a python class.

The advantage is that you can define your own arguments, and use them to define the tokens folder.  (and avoid duplicating it as an argument like with the `pyrunner` cli).

For this, you need to wrap your script in a python class.

A basic task that creates a file

```python

from pyrunner import Task

class TouchTask(Task):

    def command(self):

        return "touch"

    

    def experiment_folder(self, args):

        return "." # return current folder

if __name__ == "__main__":

    TouchTask().run()

```

More complex examples are in the examples/ directory.