Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/raku-community-modules/http-useragent

Web user agent class for Raku
https://github.com/raku-community-modules/http-useragent

cookie cookiejar rakulang ssl user-agent

Last synced: 5 months ago
JSON representation

Web user agent class for Raku

Awesome Lists containing this project

README 

          
[![Actions Status](https://github.com/raku-community-modules/HTTP-UserAgent/actions/workflows/linux.yml/badge.svg)](https://github.com/raku-community-modules/HTTP-UserAgent/actions) [![Actions Status](https://github.com/raku-community-modules/HTTP-UserAgent/actions/workflows/macos.yml/badge.svg)](https://github.com/raku-community-modules/HTTP-UserAgent/actions)



NAME

====



HTTP::UserAgent - Web user agent class



SYNOPSIS

========



```raku

use HTTP::UserAgent;



my $ua = HTTP::UserAgent.new;

$ua.timeout = 10;



my $response = $ua.get("URL");



if $response.is-success {

    say $response.content;

}

else {

    die $response.status-line;

}

```



DESCRIPTION

===========



This module provides functionality to crawling the web witha handling cookies and correct User-Agent value.



It has TLS/SSL support.



METHODS

=======



method new

----------



```raku

method new(HTTP::UserAgent:U: :$!useragent, Bool :$!throw-exceptions, :$!max-redirects = 5, :$!debug)

```



Default constructor.



There are four optional named arguments:



  * useragent



A string that specifies what will be provided in the `User-Agent` header in the request. A number of standard user agents are described in [HTTP::UserAgent::Common](HTTP::UserAgent::Common), but a string that is not specified there will be used verbatim.



  * throw-exceptions



By default the `request` method will not throw an exception if the response from the server indicates that the request was unsuccesful, in this case you should check `is-success` to determine the status of the [HTTP::Response](HTTP::Response) returned. If this is specified then an exception will be thrown if the request was not a success, however you can still retrieve the response from the `response` attribute of the exception object.



  * max-redirects



This is the maximum number of redirects allowed for a single request, if this is exceeded then an exception will be thrown (this is not covered by `no-exceptions` above and will always be throw,) the default value is 5.



  * debug



It can etheir be a Bool like simply `:debug` or you can pass it a IO::Handle or a file name. Eg `:debug($*ERR)` will ouput on stderr `:debug("mylog.txt")` will ouput on the file.



method auth

-----------



```raku

method auth(HTTP::UserAgent:, Str $login, Str $password)

```



Sets username and password needed to HTTP Auth.



method get

----------



```raku

multi method get(Str $url is copy, :bin?, *%headers)

multi method get(URI $uri, :bin?, *%headers)

```



Requests the $url site, returns HTTP::Response, except if throw-exceptions is set as described above whereby an exception will be thrown if the response indicates that the request wasn't successfull.



If the Content-Type of the response indicates that the content is text the `content` of the Response will be a decoded string, otherwise it will be left as a [Blob](Blob).



If the ':bin' adverb is supplied this will force the response `content` to always be an undecoded [Blob](Blob)



Any additional named arguments will be applied as headers in the request.



method post

-----------



```raku

multi method post(URI $uri, %form, *%header ) -> HTTP::Response

multi method post(Str $uri, %form, *%header ) -> HTTP::Response

```



Make a POST request to the specified uri, with the provided Hash of %form data in the body encoded as "application/x-www-form-urlencoded" content. Any additional named style arguments will be applied as headers in the request.



An [HTTP::Response](HTTP::Response) will be returned, except if throw-exceptions has been set and the response indicates the request was not successfull.



If the Content-Type of the response indicates that the content is text the `content` of the Response will be a decoded string, otherwise it will be left as a [Blob](Blob).



If the ':bin' adverb is supplied this will force the response `content` to always be an undecoded [Blob](Blob)



If greater control over the content of the request is required you should create an [HTTP::Request](HTTP::Request) directly and populate it as needed,



method request

--------------



```raku

method request(HTTP::Request $request, :bin?)

```



Performs the request described by the supplied [HTTP::Request](HTTP::Request), returns a [HTTP::Response](HTTP::Response), except if throw-exceptions is set as described above whereby an exception will be thrown if the response indicates that the request wasn't successful.



If the response has a 'Content-Encoding' header that indicates that the content was compressed, then it will attempt to inflate the data using [Compress::Zlib](Compress::Zlib), if the module is not installed then an exception will be thrown. If you do not have or do not want to install [Compress::Zlib](Compress::Zlib) then you should be able to send an 'Accept-Encoding' header with a value of 'identity' which should cause a well behaved server to send the content verbatim if it is able to.



If the Content-Type of the response indicates that the content is text the `content` of the Response will be a decoded string, otherwise it will be left as a [Blob](Blob). The content-types that are always considered to be binary (and thus left as a [Blob](Blob) ) are those with the major-types of 'image','audio' and 'video', certain 'application' types are considered to be 'text' (e.g. 'xml', 'javascript', 'json').



If the ':bin' adverb is supplied this will force the response `content` to always be an undecoded [Blob](Blob)



You can use the helper subroutines defined in [HTTP::Request::Common](HTTP::Request::Common) to create the [HTTP::Request](HTTP::Request) for you or create it yourself if you have more complex requirements.



routine get :simple

-------------------



```raku

sub get(Str $url) returns Str is export(:simple)

```



Like method get, but returns decoded content of the response.



routine head :simple

--------------------



```raku

sub head(Str $url) returns Parcel is export(:simple)

```



Returns values of following header fields:



  * Content-Type



  * Content-Length



  * Last-Modified



  * Expires



  * Server



routine getstore :simple

------------------------



```raku

sub getstore(Str $url, Str $file) is export(:simple)

```



Like routine get but writes the content to a file.



routine getprint :simple

------------------------



```raku

sub getprint(Str $url) is export(:simple)

```



Like routine get but prints the content and returns the response code.



SUPPORT MODULES

===============



HTTP::Cookie - HTTP cookie class

--------------------------------



This module encapsulates single HTTP Cookie.



```raku

use HTTP::Cookie;



my $cookie = HTTP::Cookie.new(:name, :value);

say ~$cookie;

```



The following methods are provided:



### method new



```raku

my $c = HTTP::Cookie.new(:name, :value, :secure, fields => (a => b));

```



A constructor, it takes these named arguments:



key description



 name name of a cookie   value value of a cookie   secure Secure param   httponly HttpOnly param   fields list of field Pairs (field => value) 



### method Str



Returns a cookie as a string in readable (RFC2109) form.



HTTP::Cookies - HTTP cookie jars

--------------------------------



This module provides a bunch of methods to manage HTTP cookies.



```raku

use HTTP::Cookies;

my $cookies = HTTP::Cookies.new(

  :file<./cookies>,

  :autosave

);

$cookies.load;

```



### method new



```raku

my $cookies = HTTP::Cookies.new(

  :file<./cookies.here>

  :autosave,

);



Constructor, takes named arguments:

```



key description



 file where to write cookies   autosave save automatically after every operation on cookies or not 



### method set-cookie



```raku

my $cookies = HTTP::Cookies.new;

$cookies.set-cookie('Set-Cookie: name1=value1; HttpOnly');

```



Adds a cookie (passed as an argument $str of type Str) to the list of cookies.



### method save



```raku

my $cookies = HTTP::Cookies.new;

$cookies.set-cookie('Set-Cookie: name1=value1; HttpOnly');

$cookies.save;

```



Saves cookies to the file ($.file).



### method load



```raku

my $cookies = HTTP::Cookies.new;

$cookies.load;

```



Loads cookies from file specified at instantiation ($.file).



### method extract-cookies



```raku

my $cookies = HTTP::Cookies.new;

my $response = HTTP::Response.new(Set-Cookie => "name1=value; Secure");

$cookies.extract-cookies($response);

```



Gets cookies ('Set-Cookie: ' lines) from the HTTP Response and adds it to the list of cookies.



### method add-cookie-header



```raku

my $cookies = HTTP::Cookies.new;

my $request = HTTP::Request.new;

$cookies.load;

$cookies.add-cookie-header($request);

```



Adds cookies fields ('Cookie: ' lines) to the HTTP Request.



### method clear-expired



```raku

my $cookies = HTTP::Cookies.new;

$cookies.set-cookie('Set-Cookie: name1=value1; Secure');

$cookies.set-cookie('Set-Cookie: name2=value2; Expires=Wed, 09 Jun 2021 10:18:14 GMT');

$cookies.clear-expired; # contains 'name1' cookie only

```



Removes expired cookies.



### method clear



```raku

my $cookies = HTTP::Cookies.new;

$cookies.load;   # contains something

$cookies.clear;  # will be empty after this action

```



Removes all cookies.



### method push-cookie



```raku

my $c = HTTP::Cookie.new(:name, :value, :httponly);

my $cookies = HTTP::Cookies.new;

$cookies.push-cookie: $c;

```



Pushes cookies (passed as an argument $c of type HTTP::Cookie) to the list of cookies.



### method Str



Returns all cookies in human (and server) readable form.



HTTP::UserAgent::Common - the most commonly used User-Agents

------------------------------------------------------------



This module provides a list of the most commonly used User-Agents.



```raku

use HTTP::UserAgent::Common;

say get-ua('chrome_linux');

```



### routine get-ua



```raku

say get-ua('chrome_linux');

```



Returns correct UserAgent or unchaged passed argument if UserAgent could not be found.



Available UserAgents:



    chrome_w7_64 firefox_w7_64 ie_w7_64 chrome_w81_64 firefox_w81_64 mob_safari_osx

    safari_osx chrome_osx firefox_linux chrome_linux



HTTP::Header - class encapsulating HTTP message header

------------------------------------------------------



This module provides a class with a set of methods making us able to easily handle HTTP message headers.



```raku

use HTTP::Header;

my $h = HTTP::Header.new;

$h.field(Accept => 'text/plain');

say $h.field('Accept');

$h.remove-field('Accept');

```



### method new



```raku

my $head = HTTP::Header.new(:h1, :h2);

```



A constructor. Takes name => value pairs as arguments.



### method header



```raku

my $head = HTTP::Header.new(:h1, :h2);

say $head.header('h1');



my $head = HTTP::Header.new(:h1, :h2);

$head.header(:h3);

```



Gets/sets header field.



### method init-field



```raku

my $head = HTTP::Header.new;

$head.header(:h1);

$head.init-header(:h1, :h2); # it doesn't change the value of 'h1'

say ~$head;

```



Initializes a header field: adds a field only if it does not exist yet.



### method push-header



```raku

my $head = HTTP::Header.new;

$head.push-header( HTTP::Header::Field.new(:name, :value) );

say ~$head;

```



Pushes a new field. Does not check if exists.



### method remove-header



```raku

my $head = HTTP::Header.new;

$head.header(:h1);

$head.remove-header('h1');

```



Removes a field of name $field.



### method header-field-names



```raku

my $head = HTTP::Header.new(:h1, :h2);

my @names = $head.header-field-names;

say @names; # h1, h2

```



Returns a list of names of all fields.



### method clear



```raku

my $head = HTTP::Header.new(:h1, :h2);

$head.clear;

```



Removes all fields.



### method Str



Returns readable form of the whole header section.



### method parse



```raku

my $head = HTTP::Header.new.parse("h1: v1\r\nh2: v2\r\n");

say $head.raku;

```



Parses the whole header section.



HTTP::Header::Field

-------------------



This module provides a class encapsulating HTTP Message header field.



```raku

use HTTP::Header::Field;

my $header = HTTP::Header::Field.new(:name, values => (123, 456));

```



### method new



Constructor. Takes these named arguments:



key description



 name name of a header field   values array of values of a header field 



### method Str



Stringifies an HTTP::Header::Field object. Returns a header field in a human (and server) readable form.



HTTP::Request - class encapsulating HTTP request message

--------------------------------------------------------



Module provides functionality to easily manage HTTP requests.



```raku

use HTTP::Request;

my $request = HTTP::Request.new(GET => 'http://www.example.com/');

```



### method new



A constructor, the first form takes parameters like:



  * method => URL, where method can be POST, GET ... etc.



  * field => values, header fields



```raku

my $req = HTTP::Request.new(:GET, :h1);

```



The second form takes the key arguments as simple positional parameters and is designed for use in places where for example the request method may be calculated and the headers pre-populated.



### method set-method



```raku

my $req = HTTP::Request.new;

$req.set-method: 'POST';

```



Sets a method of the request.



### method uri



```raku

my $req = HTTP::Request.new;

$req.uri: 'example.com';

```



Sets URL to request.



### method add-cookies



```raku

method add-cookies(HTTP::Cookies $cookies)

```



This will cause the appropriate cookie headers to be added from the supplied HTTP::Cookies object.



### method add-form-data



```raku

multi method add-form-data(%data, :$multipart)

multi method add-form-data(:$multipart, *%data);

multi method add-form-data(Array $data, :$multipart)

```



Adds the form data, supplied either as a `Hash`, an `Array` of `Pair`s, or in a named parameter style, to the POST request (it doesn't make sense on most other request types).



The default is to use 'application/x-www-form-urlencoded' and 'multipart/form-data' can be used by providing the ':multipart' named argument. Alternatively a previously applied "content-type" header of either 'application/x-www-form-urlencoded' or 'multipart/form-data' will be respected and in the latter case any applied boundary marker will be retained.



As a special case for multipart data if the value for some key in the data is an `Array` of at least one item then it is taken to be a description of a file to be "uploaded" where the first item is the path to the file to be inserted, the second (optional) an alternative name to be used in the content disposition header and the third an optional `Array` of `Pair`s that will provide additional header lines for the part.



### method Str



Returns stringified object.



### method parse



```raku

method parse(Str $raw_request --> HTTP::Request:D)

```



Parses raw HTTP request. See `HTTP::Message`



HTTP::Request::Common - Construct common HTTP::Request objects

--------------------------------------------------------------



```raku

use HTTP::Request::Common;



my $ua       = HTTP::UserAgent.new;

my $response = $ua.request(GET 'http://google.com/');

```



This module provide functions that return newly created `HTTP::Request` objects. These functions are usually more convenient to use than the standard `HTTP::Request` constructor for the most common requests. The following functions are provided:



### GET $url, Header => Value...



The `GET` function returns an `HTTP::Request` object initialized with the "GET" method and the specified URL.



### HEAD $url, Header => Value,...



Like `GET` but the method in the request is "HEAD".



### DELETE $url, Header => Value,...



Like `GET` but the method in the request is "DELETE".



### `PUT $url, Header =` Value,..., content => $content>



Like `GET` but the method in the request is "PUT".



HTTP::Response - class encapsulating HTTP response message

----------------------------------------------------------



```raku

use HTTP::Response;

my $response = HTTP::Response.new(200);

say $response.is-success; # it is

```



Module provides functionality to easily manage HTTP responses.



Response object is returned by the .get() method of [HTTP::UserAgent](HTTP::UserAgent).



### method new



```raku

my $response = HTTP::Response.new(200, :h1);

```



A constructor, takes named arguments:



key description



 code code of the response   fields header fields (field_name => values) 



### method is-success



```raku

my $response = HTTP::Response.new(200);

say 'YAY' if $response.is-success;

```



Returns True if response is successful (status == 2xx), False otherwise.



method set-code

---------------



```raku

my $response = HTTP::Response.new;

$response.set-code: 200;

```



Sets code of the response.



### method Str



Returns stringified object.



### method parse



See `HTTP::Message`.



HTTP::Message - class encapsulating HTTP message

------------------------------------------------



```raku

use HTTP::Message;

my $raw_msg = "GET / HTTP/1.1\r\nHost: somehost\r\n\r\n";

my $mess = HTTP::Message.new.parse($raw_msg);

say $mess;

```



This module provides a bunch of methods to easily manage HTTP message.



### method new



```raku

my $msg = HTTP::Message.new('content', :field);

```



A constructor, takes these named arguments:



key description



 content content of the message (optional)   fields fields of the header section 



### method add-content



```raku

my $msg = HTTP::Message.new('content', :field);

$msg.add-content: 's';

say $msg.content; # says 'contents'

```



Adds HTTP message content. It does not remove the existing value, it concats to the existing content.



### method decoded-content



```raku

my $msg = HTTP::Message.new();

say $msg.decoded-content;

```



Returns decoded content of the message (using [Encode](Encode) module to decode).



### method field



See `HTTP::Header`.



### method init-field



See `HTTP::Header`.



### method push-field



See `HTTP::Header`.



### method remove-field



See `HTTP::Header`.



### method clear



```raku

my $msg = HTTP::Message.new('content', :field);

$msg.clear;

say ~$msg; # says nothing

```



Removes the whole message, both header and content section.



### method parse



```raku

my $msg = HTTP::Message.new.parse("GET / HTTP/1.1\r\nHost: example\r\ncontent\r\n");

say $msg.raku;

```



Parses the whole HTTP message.



It takes the HTTP message (with \r\n as a line separator) and obtains the header and content sections, creates a `HTTP::Header` object.



### method Str



Returns HTTP message in a readable form.



AUTHOR

======



  * Filip Sergot



Source can be located at: https://github.com/raku-community-modules/HTTP-UserAgent . Comments and Pull Requests are welcome.



COPYRIGHT AND LICENSE

=====================



Copyright 2014 - 2022 Filip Sergot



Copyright 2023 - 2025 The Raku Community



This library is free software; you can redistribute it and/or modify it under the MIT License.