{"id":20315222,"url":"https://github.com/sysread/coro-processpool","last_synced_at":"2026-02-12T04:03:14.614Z","repository":{"id":12100957,"uuid":"14689737","full_name":"sysread/Coro-ProcessPool","owner":"sysread","description":"An asynchronous process pool for Perl and Coro","archived":false,"fork":false,"pushed_at":"2017-11-14T02:59:09.000Z","size":237,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-08-15T10:52:12.863Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Perl 6","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sysread.png","metadata":{"files":{"readme":"README.pod","changelog":"Changes","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2013-11-25T15:44:44.000Z","updated_at":"2017-11-14T02:54:48.000Z","dependencies_parsed_at":"2022-07-29T10:19:26.965Z","dependency_job_id":null,"html_url":"https://github.com/sysread/Coro-ProcessPool","commit_stats":null,"previous_names":[],"tags_count":16,"template":false,"template_full_name":null,"purl":"pkg:github/sysread/Coro-ProcessPool","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sysread%2FCoro-ProcessPool","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sysread%2FCoro-ProcessPool/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sysread%2FCoro-ProcessPool/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sysread%2FCoro-ProcessPool/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sysread","download_url":"https://codeload.github.com/sysread/Coro-ProcessPool/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sysread%2FCoro-ProcessPool/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":270856775,"owners_count":24657700,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-17T02:00:09.016Z","response_time":129,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-14T18:18:21.765Z","updated_at":"2026-02-12T04:03:14.569Z","avatar_url":"https://github.com/sysread.png","language":"Perl 6","funding_links":[],"categories":[],"sub_categories":[],"readme":"=pod\n\n=encoding UTF-8\n\n=head1 NAME\n\nCoro::ProcessPool - An asynchronous pool of perl processes\n\n=head1 VERSION\n\nversion 0.30\n\n=head1 SYNOPSIS\n\n  use Coro::ProcessPool;\n  use Coro;\n\n  my $pool = Coro::ProcessPool-\u003enew(\n    max_procs =\u003e 4,\n    max_reqs  =\u003e 100,\n    include   =\u003e ['/path/to/my/task/classes', '/path/to/other/packages'],\n  );\n\n  my $double = sub { $_[0] * 2 };\n\n  #-----------------------------------------------------------------------\n  # Process in sequence, waiting for each result in turn\n  #-----------------------------------------------------------------------\n  my %result;\n  foreach my $i (1 .. 1000) {\n    $result{$i} = $pool-\u003eprocess($double, $i);\n  }\n\n  #-----------------------------------------------------------------------\n  # Process as a batch\n  #-----------------------------------------------------------------------\n  my @results = $pool-\u003emap($double, 1 .. 1000);\n\n  #-----------------------------------------------------------------------\n  # Defer waiting for result\n  #-----------------------------------------------------------------------\n  my %deferred;\n\n  $deferred{$_} = $pool-\u003edefer($double, $_)\n    foreach 1 .. 1000;\n\n  # Later\n  foreach my $i (keys %deferred) {\n    print \"$i = \" . $deferred{$i}-\u003e() . \"\\n\";\n  }\n\n  #-----------------------------------------------------------------------\n  # Use a \"task class\" implementing 'new' and 'run'\n  #-----------------------------------------------------------------------\n  my $result = $pool-\u003eprocess('Task::Doubler', 21);\n\n  #-----------------------------------------------------------------------\n  # Pipelines (work queues)\n  #-----------------------------------------------------------------------\n  my $pipe = $pool-\u003epipeline;\n\n  # Start producer thread to queue tasks\n  my $producer = async {\n    while (my $task = get_next_task()) {\n      $pipe-\u003equeue('Some::TaskClass', $task);\n    }\n\n    # Let the pipeline know no more tasks are coming\n    $pipe-\u003eshutdown;\n  };\n\n  # Collect the results of each task as they are received\n  while (my $result = $pipe-\u003enext) {\n    do_stuff_with($result);\n  }\n\n  $pool-\u003eshutdown;\n\n=head1 DESCRIPTION\n\nProcesses tasks using a pool of external Perl processes.\n\n=head1 CONSTRUCTOR\n\n  my $pool = Coro::ProcessPool-\u003enew(\n    max_procs =\u003e 4,\n    max_reqs  =\u003e 100,\n    include   =\u003e ['path/to/my/packages', 'some/more/packages'],\n  );\n\n=head2 max_procs\n\nThe maximum number of processes to run within the process pool. Defaults\nto the number of CPUs on the ssytem.\n\n=head2 max_reqs\n\nThe maximum number of tasks a worker process may run before being terminated\nand replaced with a fresh process. This is useful for tasks that might leak\nmemory over time.\n\n=head2 include\n\nAn optional array ref of directory paths to prepend to the set of directories\nthe worker process will use to find Perl packages.\n\n=head1 METHODS\n\n=head2 join\n\nCedes control to the event loop until the pool has completed all remaining\ntasks and woken up any threads watching them.\n\n=head2 defer\n\nQueues a task to be processed by the pool. Tasks may specified in either of two\nforms, as a code ref or the fully qualified name of a perl class which\nimplements two methods, C\u003cnew\u003e and C\u003crun\u003e. Any remaining arguments to C\u003cdefer\u003e\nare passed unchanged to the code ref or the C\u003cnew\u003e method of the task class.\n\nC\u003cdefer\u003e will immediately return an L\u003cAnyEvent/condvar\u003e that will wait for and\nreturn the result of the task (or croak if the task generated an error).\n\n  # Using a code ref\n  my $cv = $pool-\u003edefer(\\\u0026func, $arg1, $arg2, $arg3);\n  my $result = $cv-\u003erecv;\n\n  # With a task class\n  my $cv = $pool-\u003edefer('Some::Task::Class', $arg1, $arg2, $arg3);\n  my $result = $cv-\u003erecv;\n\n=head2 process\n\nCalls defer and immediately calls C\u003crecv\u003e on the returned condvar, returning\nthe result. This is useful if your workflow includes multiple threads which\nshare the same pool. All arguments are passed unchanged to C\u003cdefer\u003e.\n\n=head2 map\n\nLike perl's C\u003cmap\u003e, applies a code ref to a list of arguments. This method will\ncede until all results have been returned by the pool, returning the result as\na list. The order of arguments and results is preserved as expected.\n\n  my @results = $pool-\u003emap(\\\u0026func, $arg1, $arg2, $arg3);\n\n=head2 pipeline\n\nReturns a L\u003cCoro::ProcessPool::Pipeline\u003e object which can be used to pipe\nrequests through to the process pool. Results then come out the other end of\nthe pipe, not necessarily in the order in which they were queued. It is up to\nthe calling code to perform task accounting (for example, by passing an id in\nas one of the arguments to the task class).\n\n  my $pipe = $pool-\u003epipeline;\n\n  my $producer = async {\n    foreach my $args (@tasks) {\n      $pipe-\u003equeue('Some::Class', $args);\n    }\n\n    $pipe-\u003eshutdown;\n  };\n\n  while (my $result = $pipe-\u003enext) {\n    ...\n  }\n\nAll arguments to C\u003cpipeline()\u003e are passed transparently to the constructor of\nL\u003cCoro::ProcessPool::Pipeline\u003e. There is no limit to the number of pipelines\nwhich may be created for a pool.\n\n=head1 A NOTE ABOUT IMPORTS AND CLOSURES\n\nCode refs are serialized using L\u003cData::Dump::Streamer\u003e, allowing closed over\nvariables to be available to the code being called in the sub-process. Mutated\nvariables are I\u003cnot\u003e updated when the result is returned.\n\nSee L\u003cData::Dump::Streamer/Caveats-Dumping-Closures-(CODE-Refs)\u003e for important\nnotes regarding closures.\n\n=head2 Use versus require\n\nThe C\u003cuse\u003e pragma is run at compile time, whereas C\u003crequire\u003e is evaluated at\nruntime. Because of this, the use of C\u003cuse\u003e in code passed directly to the\nC\u003cprocess\u003e method can fail in the worker process because the C\u003cuse\u003e statement\nhas already been evaluated in the parent process when the calling code was\ncompiled.\n\nThis will not work:\n\n  $pool-\u003eprocess(sub {\n    use Foo;\n    my $foo = Foo-\u003enew();\n  });\n\nThis will work:\n\n  $pool-\u003eprocess(sub {\n    require Foo;\n    my $foo = Foo-\u003enew();\n  });\n\nIf C\u003cuse\u003e is necessary (for example, to import a method or transform the\ncalling code via import), it is recommended to move the code into its own\nmodule (or to expliticly call require and import in the subroutine), which can\nthen be called in the anonymous routine:\n\n  package Bar;\n\n  use Foo;\n\n  sub dostuff {\n    ...\n  }\n\nThen, in your caller:\n\n  $pool-\u003eprocess(sub {\n    require Bar;\n    Bar::dostuff();\n  });\n\nAlternately, a task class may be used if dependency management is causing a\nheadaches:\n\n  my $result = $pool-\u003eprocess('Task::Class', @args);\n\n=head1 COMPATIBILITY\n\nC\u003cCoro::ProcessPool\u003e will likely break on Win32 due to missing support for\nnon-blocking file descriptors (Win32 can only call C\u003cselect\u003e and C\u003cpoll\u003e on\nactual network sockets). Without rewriting this as a network server, which\nwould impact performance and be really annoying, it is likely this module will\nnot support Win32 in the near future.\n\nThe following modules will get you started if you wish to explore a synchronous\nprocess pool on Windows:\n\n=over\n\n=item L\u003cWin32::Process\u003e\n\n=item L\u003cWin32::IPC\u003e\n\n=item L\u003cWin32::Pipe\u003e\n\n=back\n\n=head1 SEE ALSO\n\n=over\n\n=item L\u003cCoro\u003e\n\n=item L\u003cAnyEvent/condvar\u003e\n\n=back\n\n=head1 AUTHOR\n\nJeff Ober \u003csysread@fastmail.fm\u003e\n\n=head1 COPYRIGHT AND LICENSE\n\nThis software is copyright (c) 2017 by Jeff Ober.\n\nThis is free software; you can redistribute it and/or modify it under\nthe same terms as the Perl 5 programming language system itself.\n\n=cut\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsysread%2Fcoro-processpool","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsysread%2Fcoro-processpool","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsysread%2Fcoro-processpool/lists"}