https://github.com/alindeman/rector

Rector coordinates parallelized jobs that generate metrics or other data together
https://github.com/alindeman/rector

Last synced: 2 months ago
JSON representation

Rector coordinates parallelized jobs that generate metrics or other data together

• Host: GitHub
• URL: https://github.com/alindeman/rector
• Owner: alindeman
• Created: 2012-02-04T04:16:15.000Z (almost 13 years ago)
• Default Branch: master
• Last Pushed: 2012-02-09T20:20:54.000Z (almost 13 years ago)
• Last Synced: 2024-04-24T20:43:33.736Z (8 months ago)
• Language: Ruby
• Homepage:
• Size: 118 KB
• Stars: 7
• Watchers: 3
• Forks: 0
• Open Issues: 1
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# Rector

[![Build Status](https://secure.travis-ci.org/alindeman/rector.png)](http://travis-ci.org/alindeman/rector)

Rector allows coordination of a number of jobs spawned with a mechanism

like Resque (though any job manager will do). If you are able to parallelize

the processing of a task, yet all these tasks are generating metrics,

statistics, or other data that need to be combined, Rector might be for you.

## Requirements

* Ruby >= 1.9.2 (or 1.9 mode of JRuby or Rubinius)

## Configuration

Rector currently supports Redis as a backend for job coordination and

data storage.

### Redis Server

```ruby

Rector.configure do |c|

  c.redis = Redis.new(:host => "10.0.1.1", :port => 6380)

end

```

## Job Creation (Master)

Rector requires that some process be designated as the "master" process.

This is usually the process that is also responsible for spawning the

worker jobs.

```ruby

job = Rector::Job.new

# e.g., processing files in parallel

files.each do |file|

  worker = job.workers.create

  # e.g., using Resque for job management; Rector doesn't really care

  Resque.enqueue(WordCounterJob, worker.id, file)

end

# wait for all the workers to complete

job.join

# get aggregated data from all the jobs

job.data.each do |word, count|

  puts "#{word} was seen #{count} times across all files"

end

job.cleanup

```

## Job Processing (Workers)

```ruby

class ProcessFileJob

  def self.perform(worker_id, file)

    worker = Rector::Worker.new(worker_id)

    words = File.read(file).split(/\W/)

    words.reject(&:blank?).each do |word|

      worker.data[word] ||= 0 

      worker.data[word]  += 1

    end

    worker.finish

  end

end

```