Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/sysread/proc-tored-pool

Managed worker pool with Proc::tored and Parallel::ForkManager
https://github.com/sysread/proc-tored-pool

Last synced: about 1 month ago
JSON representation

Managed worker pool with Proc::tored and Parallel::ForkManager

Awesome Lists containing this project

README 

        
=pod



=encoding UTF-8



=head1 NAME



Proc::tored::Pool - managed work pool with Proc::tored and Parallel::ForkManager



=head1 VERSION



version 0.08



=head1 SYNOPSIS



  use Proc::tored::Pool;



  # Create a worker pool service

  my $pool = pool 'thing-doer', in '/var/run', capacity 10,

    on success, call {

      my ($me, $id, @results) = @_;

      print "thing $id complete: @results";

    },

    on failure, call {

      my ($me, $id, $message) = @_;

      warn "thing $id failed: $message";

    };



  # Do things with the pool

  run {

    my ($thing_id, $thing) = get_next_thing();

    process { do_things($thing) } $pool, $thing_id;

  } $pool;



  # Control the pool as a Proc::tored service

  pause $pool;

  resume $pool;

  stop $pool;

  zap $pool, 15 or die "timed out after 15 seconds waiting for pool to stop";

  start $pool;



=head1 DESCRIPTION



Provides a simple and fast interfact to build and manage a pool of forked

worker processes. The process is controlled using a pidfile and touch file.



=head1 EXPORTED SUBROUTINES



As a C is a L service, it by default exports

the same functions as L.



In addition, the following subroutines are exported by default.



=head2 pool



Creates the pool (an instance of L). Requires a

C<$name> as its first argument.



  my $pool = pool 'the-proletariat', ...;



=head2 capacity



Sets the max number of forked worker processes to be permitted at any given

time.



  my $pool = pool 'the-proletariat', capacity 16, ...;



=head2 on



Builds an event callback with one of L, L, or L.



  my $pool = pool 'the-proletariat', capacity 16,

    on success, call { ... };



=head2 call



Defines the code to be called by an event callback. See L.



=head2 pending



Returns the number of tasks that have been assigned to worker processes but

have not yet completed.



=head2 process



Sends a task (a C ref) to the pool, optionally specifying a task id to

identify the results in callbacks. The return value of the supplied code ref is

passed as is to the L callback (if supplied).



  process { seize_the_means_of_production() } $pool;

  process { seize_the_means_of_production() } $pool, $task_id;



=head2 sync



For situations in which a task or tasks must be completed before program

execution can continue, C may be used to block until all pending tasks

have completed. After calling sync, there will be no pending tasks and all

callbacks for previously submitted tasks will have been called.



  process { seize_the_means_of_production() } $pool;

  sync $pool;



=head1 EVENTS



=head2 assignment



Triggered immediately after a task is assigned to a worker process. Receives

the pool object and the task id (if provided when calling L).



  my $pool = pool 'thing-doer', ...,

    on assignment, call {

      my ($self, $task_id) = @_;

      $assigned{$task_id} = 1;

    };



  process { do_things() } $pool, $task_id;



=head2 success



Triggered after the completion of a task. Receives the pool object, task id (if

provided when calling L), and the return value of the code block.



  my $pool = pool 'thing-doer', ...,

    on success, call {

      my ($self, $task_id, @results) = @_;

      ...

    };



  process { do_things() } $pool, $task_id;



If L performs an C, no data is returned from the worker process

(because C never returns). In this case, if the process had a zero exit

status, the C callback is triggered with the results value of C<"zero

but true">. A non-zero exit status is handled by C as in the general

case.



=head2 failure



Triggered if the code block dies or the forked worker exits abnormally.

Recieves the pool object, task id (if provided when calling L), and the

error message generated by the code ref.



  my $pool = pool 'thing-doer', ...,

    on failure, call {

      my ($self, $task_id, $error) = @_;

      warn "Error executing task $task_id: $error";

    };



  process { do_things() } $pool, $task_id;



=head1 BUGS AND LIMITATIONS



=head2 Proc::tored



Warnings and limitations for L also apply to this module,

especially the notes regarding service control from within a running service

which also apply to code executing in forked worker processes. See

L for details.



=head2 Parallel::ForkManager



Warnings and limitations for L also apply to this

module, including the injunction against using two pools simultaneously from

the same process. See L and

L for details.



=head1 SEE ALSO



=over



=item L



=item L



=back



=head1 AUTHOR



Jeff Ober 



=head1 COPYRIGHT AND LICENSE



This software is copyright (c) 2017 by Jeff Ober.



This is free software; you can redistribute it and/or modify it under

the same terms as the Perl 5 programming language system itself.



=cut