https://github.com/dotandimet/mojo-useragent-role-queued

A role for Mojo::UserAgent that processes non-blocking requests in a rate-limiting queue.
https://github.com/dotandimet/mojo-useragent-role-queued

mojo mojolicious perl perl5 user-agent web-crawler

Last synced: 8 days ago
JSON representation

A role for Mojo::UserAgent that processes non-blocking requests in a rate-limiting queue.

• Host: GitHub
• URL: https://github.com/dotandimet/mojo-useragent-role-queued
• Owner: dotandimet
• License: other
• Created: 2017-12-26T18:23:02.000Z (about 7 years ago)
• Default Branch: master
• Last Pushed: 2019-08-08T15:24:58.000Z (over 5 years ago)
• Last Synced: 2024-04-18T04:02:20.617Z (10 months ago)
• Topics: mojo, mojolicious, perl, perl5, user-agent, web-crawler
• Language: Perl
• Homepage:
• Size: 65.4 KB
• Stars: 3
• Watchers: 2
• Forks: 2
• Open Issues: 1
• Metadata Files:
  • Readme: README.md
  • Changelog: Changes
  • License: LICENSE

Awesome Lists containing this project

README

        
[![Build Status](https://travis-ci.org/dotandimet/Mojo-UserAgent-Role-Queued.svg?branch=master)](https://travis-ci.org/dotandimet/Mojo-UserAgent-Role-Queued) [![MetaCPAN Release](https://badge.fury.io/pl/Mojo-UserAgent-Role-Queued.svg)](https://metacpan.org/release/Mojo-UserAgent-Role-Queued)

# NAME

Mojo::UserAgent::Role::Queued - A role to process non-blocking requests in a rate-limiting queue.

# SYNOPSIS

       use Mojo::UserAgent;

       my $ua = Mojo::UserAgent->new->with_roles('+Queued');

       $ua->max_redirects(3);

       $ua->max_active(5); # process up to 5 requests at a time

       for my $url (@big_list_of_urls) {

       $ua->get($url, sub {

               my ($ua, $tx) = @_;

               if (! $tx->error) {

                   say "Page at $url is titled: ",

                     $tx->res->dom->at('title')->text;

               }

              });

      };

      Mojo::IOLoop->start unless Mojo::IOLoop->is_running;

      # works with promises, too:

     my @p = map {

       $ua->get_p($_)->then(sub { pop->res->dom->at('title')->text })

         ->catch(sub { say "Error: ", @_ })

     } @big_list_of_urls;

      Mojo::Promise->all(@p)->wait;

    

# DESCRIPTION

Mojo::UserAgent::Role::Queued manages all non-blocking requests made through [Mojo::UserAgent](https://metacpan.org/pod/Mojo::UserAgent) in a queue to limit the number of simultaneous requests.

[Mojo::UserAgent](https://metacpan.org/pod/Mojo::UserAgent) can make multiple concurrent non-blocking HTTP requests using Mojo's event loop, but because there is only a single process handling all of them, you must take care to limit the number of simultaneous requests you make.

Some discussion of this issue is available here

[http://blogs.perl.org/users/stas/2013/01/web-scraping-with-modern-perl-part-1.html](http://blogs.perl.org/users/stas/2013/01/web-scraping-with-modern-perl-part-1.html)

and in Joel Berger's answer here:

[http://stackoverflow.com/questions/15152633/perl-mojo-and-json-for-simultaneous-requests](http://stackoverflow.com/questions/15152633/perl-mojo-and-json-for-simultaneous-requests).

[Mojo::UserAgent::Role::Queued](https://metacpan.org/pod/Mojo::UserAgent::Role::Queued) tries to generalize the practice of managing a large number of requests using a queue, by embedding the queue inside [Mojo::UserAgent](https://metacpan.org/pod/Mojo::UserAgent) itself.

# ATTRIBUTES

[Mojo::UserAgent::Role::Queued](https://metacpan.org/pod/Mojo::UserAgent::Role::Queued) has the following attributes:

## max\_active

    $ua->max_active(5);  # execute no more than 5 transactions at a time.

    print "Execute no more than ", $ua->max_active, " concurrent transactions"

Parameter controlling the maximum number of transactions that can be active at the same time.

# EVENTS

[Mojo::UserAgent::Role::Queued](https://metacpan.org/pod/Mojo::UserAgent::Role::Queued) adds the following event to those emitted by [Mojo::UserAgent](https://metacpan.org/pod/Mojo::UserAgent):

## queue\_empty

    $ua->on(queue_empty => sub { my ($ua) = @_; .... })

Emitted when the queue has been emptied of all pending jobs. In previous releases, this event was called `stop_queue` (**this is a breaking change**).

## 

# LICENSE AND COPYRIGHT

This software is Copyright (c) 2017-2019 by Dotan Dimet .

This library is free software; you can redistribute it and/or modify

it under the terms of the Artistic License version 2.0.

# AUTHOR

Dotan Dimet