https://github.com/preaction/beam-worker
Task runner and Map-Reduce for the Beam framework
https://github.com/preaction/beam-worker
Last synced: about 1 year ago
JSON representation
Task runner and Map-Reduce for the Beam framework
- Host: GitHub
- URL: https://github.com/preaction/beam-worker
- Owner: preaction
- License: other
- Created: 2015-02-28T07:06:14.000Z (over 11 years ago)
- Default Branch: master
- Last Pushed: 2016-11-23T23:15:43.000Z (over 9 years ago)
- Last Synced: 2025-06-12T11:03:14.029Z (about 1 year ago)
- Language: Perl
- Size: 15.6 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README
- Changelog: CHANGES
- Contributing: CONTRIBUTING.md
- License: LICENSE
Awesome Lists containing this project
README
NAME
Beam::Runner - Task runner and Map-Reduce framework for Beam
VERSION
version 0.001
SYNOPSIS
# Job is a series of Steps
# Steps accepts input (arguments) to become a Task
# Task is the intersection of Step and its arguments
# Run a job locally
my $job = Beam::Blast::Job->new(
wire => $wire,
steps => [
{
ref => 'foo',
method => 'bar',
},
{
ref => 'baz',
method => 'fuzz',
args => [], # Don't pass the output from the prior task
},
{
ref => 'fizz',
method => 'buzz',
args => [ 'moo', { '$ref': '@_' } ], # Append the output vars (curry)
},
],
);
my ( $output_1, $output_2 ) = $job->run( $input_1, $input_2 );
# Run using the distributed workers
my $runner = Beam::Blast::Runner->new(
host => 'localhost',
port => 2132,
);
my $job = $runner->job(
wire => $wire,
steps => [
{
ref => 'foo',
method => 'bar',
},
{
ref => 'baz',
method => 'fuzz',
},
],
);
my ( $output_1, $output_2 ) = $job->run( $input_1, $input_2 );
# Run distributed and async
my @output;
my $cb = sub {
my ( $job, $output ) = @_;
push @output, $output;
};
$job->run( $input_1, $input_2, $cb );
# Run a worker
my $worker = Beam::Blast::Worker->new(
port => 2132,
);
$worker->start;
# Run a master
# Masters can run map-reduce jobs, parcelling out work to other workers
# Otherwise, they appear from the outside to be workers
my $master = Beam::Blast::Master->new(
port => 2132,
workers => 10, # fork 10 child workers why not?
);
my $job = $master->job( ... );
my @output = $job->run( 1..500 ); # Run all 500 tasks
# Connect any random worker to a master
Beam::Blast::Worker->new(
master => 'tcp://example.com:2132',
workers => 10, # fork 10 more child workers
);
# When connected to a master, we don't need to accept any random job by
# opening a port. But we can, and then we would transmit our status
# to the master
# If you want to build a status monitor, you can use the Master's event
# callbacks
DESCRIPTION
AUTHOR
Doug Bell
COPYRIGHT AND LICENSE
This software is copyright (c) 2015 by Doug Bell.
This is free software; you can redistribute it and/or modify it under
the same terms as the Perl 5 programming language system itself.