https://github.com/veupathdb/djob
https://github.com/veupathdb/djob
Last synced: 4 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/veupathdb/djob
- Owner: VEuPathDB
- License: apache-2.0
- Created: 2019-07-29T13:09:33.000Z (almost 7 years ago)
- Default Branch: master
- Last Pushed: 2025-04-02T17:43:13.000Z (about 1 year ago)
- Last Synced: 2025-08-06T01:31:58.987Z (11 months ago)
- Language: Perl
- Homepage:
- Size: 12.2 MB
- Stars: 1
- Watchers: 24
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
DistribJob: a distributed job controller
=============================
DistribJob is a distributed job controller: it distributes elements of an input set to nodes in a compute cluster for processing, and merges the result. DistribJob sits on top of queueing systems such as SGE and PBS.
More specifically, the controller runs on a server. It breaks a large input set into smaller sets called subtasks, and assigns the subtasks to compute slots on nodes (machines) as they become available. On the node it runs a command on the subtask's input, and merges the result into the main result on the server. It robustly tracks failures and can be restarted.
DistribJob does not do scheduling or load balancing. For a particular job, it is given a static set of nodes, each with a static number of slots. When a slot is vacated, DistribJob fills it with the next subtask.
For details, please have a look at the [DistribJob User Manual](http://www.google.com/url?q=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fpub%3Fid%3D1BixZ5t2c0hnOZES-Rk2wG2loAQqclcmRj7AeKQjZMHA).