https://github.com/mtli/parscript
Parallel or distributed execution of jobs :parking:
https://github.com/mtli/parscript
Last synced: about 1 year ago
JSON representation
Parallel or distributed execution of jobs :parking:
- Host: GitHub
- URL: https://github.com/mtli/parscript
- Owner: mtli
- License: mit
- Created: 2019-05-13T22:56:59.000Z (about 7 years ago)
- Default Branch: master
- Last Pushed: 2022-08-09T00:58:04.000Z (almost 4 years ago)
- Last Synced: 2025-03-28T09:36:28.239Z (about 1 year ago)
- Language: Python
- Homepage:
- Size: 17.6 KB
- Stars: 5
- Watchers: 1
- Forks: 2
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE.txt
Awesome Lists containing this project
README
# Parscript: Parallel or distributed execution of jobs
Simple concept: write your jobs (shell, python, batch or anything!) in a txt file, one job one line!
```
python train.py --name=exp1 --param=1
python train.py --name=exp2 --param=2
python train.py --name=exp3 --param=3
```
And use `python -m parscript.worker job-list.txt` to run the jobs sequentially, or `python -m parscript.dispatcher job-list.txt` to run them in parallel or distributed fashion.
Usage:
- Specify the number of GPUs through `-g` (default 1)
- Specify the number of workers (per GPU) through `-w` (default 1)
- Reset the job counter through `-r`
- Directly edit `job-list.txt` after launched to add more jobs to it
- If one job fails, it will not affect other jobs and failed jobs will be recorded and reported at the end.
- Use `-s` to shutdown the machine after all jobs are finished (useful for running jobs on AWS).
- Specify which GPUs using `CUDA_VISIBLE_DEVICES`. You still need to set `-g` correspondingly for the dispatcher.
For example:
`python -m parscript.dispatcher job-list.txt -g 4 -w 2` means running 8 jobs at a time from the job list using 4 GPUs with 2 jobs on each GPU.
Hint: for complicated jobs, you might want to write a job/script generator.
## Installation
```
pip install parscript
```
[](https://badge.fury.io/py/parscript)
Useful bash aliases (add them to `~/.bash_aliases`):
```
alias parworker='python -m parscript.worker'
alias pardispatch='python -m parscript.dispatcher'
```