Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/try2code/jobqueue
Parallelize Ruby things and shell jobs on a defined number of threads
https://github.com/try2code/jobqueue
Last synced: about 2 months ago
JSON representation
Parallelize Ruby things and shell jobs on a defined number of threads
- Host: GitHub
- URL: https://github.com/try2code/jobqueue
- Owner: Try2Code
- License: other
- Created: 2011-12-12T09:19:45.000Z (about 13 years ago)
- Default Branch: master
- Last Pushed: 2018-09-09T10:47:22.000Z (over 6 years ago)
- Last Synced: 2024-03-15T06:46:28.511Z (10 months ago)
- Language: Ruby
- Homepage:
- Size: 62.5 KB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# JobQueue / ParallelQueue - Let run your jobs in parallel
This repository contains JobQueue and ParallelQueue. Both libraries provide
similar functionality: Push arbitrary blocks of code to a Queue and execute
them on a user-defined number of Threads.
JobQueue offers no support for returning the results of the blocks, i.e. the
user has to collect them. ParallelQueue is based on [parallel]
(https://github.com/grosser/parallel, https://rubygems.org/gems/parallel) which
handles the return values internally and users can get the results
out-of-the-box.
For the moment I keep both classes, but the ParallelQueue class is a lot
cleaner and also includes the slightly faster JobQueue implentation. That's why
I'll remove the JobQueue class sometime in the future.
jobQueue/parallelQueue can do the following things:
* Run blocks, Procs and Lambdas
* Run instance and class methods
* Respect user definded locks (not needed on ParallelQueue)
* Parallelizing System commands is removed, because it can easily be implemented with 'parallel'
I started a python2 implementation of this, which can be installed via [pip]
(https://pypi.python.org/pypi/jobqueue). But I stopped because the
multiprocessing.Pool module nearly does what I need ... and blocks do _not_
exist in python ;-)
## Installation
### Gem Installation
Download and install jobQueue with the following.
```
gem install jobQueue
gem install parallelQueue
```
### Requirements
JobQueue requires Ruby only, but versions 1.9.x are needed to make use of system threads.
## Usage
### Parallelize Ruby's blocks, procs, lambdas and things
Create a JobQueue with nThreads worker with:
```ruby
jq = JobQueue.new(nThreads)
pq = ParallelQueue.new
```
ParallelQueue does not need the number of workers in the constructor. It has to
be provided in the run methods.
Use its push method to put in something to do
* For blocks:
```ruby
jq.push do
myObject.method0(...)
myObject.method1(...)
myObject.method3(...)
end
```
* For procs and lambdas:
```ruby
jp.push(myProc,arg0,arg1,...)
```
* For object methods:
```ruby
jq.push([myObject,[:method,arg0,arg1,...])
```
* Same code can be used for class methods:
```ruby
jq.push(myClass,[:myClassMethod,arg0,arg1,...])
```
To start the workers, call
```ruby
jq.run
results = qp.run(8)
pq.justRun(8) # no results
```
That's it. You might have look at tests.
## Support, Issues, Bugs, ...
please use personal mail, ruby-lang mailing list or github
## Changelog
JobQueue:
* 1.0.11: prun.rb now ignores empty lines
* 1.0.10: more flexible logging control (new switches '-l' and '-b')
* 1.0.9: print out stdout and stderr from the jobs given to prun.rb, use '-D' to avoid printing
* 1.0.8: support AIX for getting the maximum number of processors, improve processor count for jruby and rbx
ParallelQueue:
* 1.0.0: parallel-based child-class of Queue, JobQueue.run is implementation as justRun
## Credits
[Robert Klemme] For the first hints: https://www.ruby-forum.com/topic/68001#86298
## License
jobQueue use the BSD License
parallelQueue use the IRC License