https://github.com/cdancette/pyrunner
Python task runner : wrap your scripts to make them idempotent and avoid unwanted parallel runs
https://github.com/cdancette/pyrunner
Last synced: 7 months ago
JSON representation
Python task runner : wrap your scripts to make them idempotent and avoid unwanted parallel runs
- Host: GitHub
- URL: https://github.com/cdancette/pyrunner
- Owner: cdancette
- Created: 2018-05-24T09:16:51.000Z (over 7 years ago)
- Default Branch: master
- Last Pushed: 2018-06-04T15:39:21.000Z (over 7 years ago)
- Last Synced: 2025-04-12T05:54:29.783Z (7 months ago)
- Language: Python
- Homepage:
- Size: 13.7 KB
- Stars: 5
- Watchers: 2
- Forks: 1
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# pyrunner
A simple Python task runner. Use it as a CLI, or wrap your own scripts with
the python interface.
Features
- Prevent a task to be running twice simultaneously.
- Prevent a task to be launched again when it is already completed.
It implements this by writing token files `.done` and `.running` in a
specified task directory. One directory = One task.
### Example use cases
##### Crash recovery
Say you launched multiple tasks for all subdirectories: `ls | xargs -I % command %`,
and the command crashes in the middle. Now how do you run only the unfinished tasks ?
You can use pyrunner to run the your tasks, and then the second run will only start
unfinished tasks.
##### Parallel
It can also be used as a poor man's parallel library.
Instead of `ls | xargs -P 2 -I % command %` you can run `ls | xargs -I % pyrunner %/.tokens command %` in two shells.
Each shell will pick up the pending tasks.
## Installation
`pip install pyrunner`
## CLI usage
You can use it as a cli, like this :
```bash
pyrunner
```
For example
```bash
pyrunner .tokens/ touch file.txt
```
Will create a file called file.txt.
Another example :
```bash
pyrunner .tokens/ bash -c "date > date.txt"
```
This command will save the current date in the file `date.txt`.
If you run it again it will not run.
## Python Usage
You can also wrap your script in a python class.
The advantage is that you can define your own arguments, and use them to define the tokens folder. (and avoid duplicating it as an argument like with the `pyrunner` cli).
For this, you need to wrap your script in a python class.
A basic task that creates a file
```python
from pyrunner import Task
class TouchTask(Task):
def command(self):
return "touch"
def experiment_folder(self, args):
return "." # return current folder
if __name__ == "__main__":
TouchTask().run()
```
More complex examples are in the examples/ directory.