https://github.com/bgamari/tpar
A simple distributed parallel job execution engine in Haskell
https://github.com/bgamari/tpar
Last synced: over 1 year ago
JSON representation
A simple distributed parallel job execution engine in Haskell
- Host: GitHub
- URL: https://github.com/bgamari/tpar
- Owner: bgamari
- License: bsd-3-clause
- Created: 2013-11-17T01:32:44.000Z (over 12 years ago)
- Default Branch: master
- Last Pushed: 2016-08-06T19:47:09.000Z (almost 10 years ago)
- Last Synced: 2025-04-06T13:17:24.841Z (over 1 year ago)
- Language: Haskell
- Homepage:
- Size: 117 KB
- Stars: 7
- Watchers: 4
- Forks: 1
- Open Issues: 3
-
Metadata Files:
- Readme: README.mkd
- License: LICENSE
Awesome Lists containing this project
README
# tpar — simple parallel job scheduling
`tpar` is a simple tool for concurrent job scheduling. Say you have a
directory full of files which need processing,
```bash
$ ls
file1 file2 file3 file4 file5
...
```
Usually one could use a `bash` for loop,
```bash
$ for f in *; do process $f; done;
```
But if `process` is a long-running task and you have many cores at
your disposal, it would be nice to speed things up a bit,
```bash
$ tpar server -N8
$ for f in *; do tpar enqueue -- process $f; done;
```
If you have multiple machines with the data mounted over, say, NFS, they can
also help with churning through the queue,
```bash
$ for m in worker1 worker2 worker3; do
> ssh $m -- tpar worker -H`hostname`;
> done
```
## Commands
`tpar` has several subcommands,
* `tpar server` starts a local queue server.
* `tpar worker` starts a worker associated with
the given queue
* `tpar enqueue -- $cmd` enqueues a job in the given queue
* `tpar status` allows you to query for the status of the queue. You can also provide a job match expression
* `tpar kill` kills a running task (specified by a job match expression)
* `tpar watch` is analogous `tail -f`, watching the output of a set of running tasks
* `tpar dump` dumps a JSON representation of the queue state.
Nearly all of these commands will require that the `-H` option be provided
specifying the canonical hostname of the queue server (the machine running
`tpar server`).
## Job match expressions
Several `tpar` commands accept a *job match expression*, which specifies the
subset of jobs on which the command should act. For instance (note the quotes to
ensure that `bash` doesn't interpret our symbols),
```
$ tpar status '*' # This is equivalent to `tpar status` run without an argument
$ tpar status id=2
$ tpar status state=running
$ tpar status 'name="my-job" or name="my-other-job"'
```
These expressions consist of the primitive matches,
* `name="STRING"`, which matches on the job name provided in the `--name` of
`tpar enqueue`.
* `id=`, which matches on the job ID
* `state=`, which matches on the current state of the job (`queued`, `running`,
`finished`, `failed`, `killed`, or `code=N`)
* `*`, which matches all jobs
These matches can be connected with the `and` and `or` operators, and inverted
with `!`.