"Net::Async::HTTP" - use HTTP with "IO::Async"
use IO::Async::Loop;
use Net::Async::HTTP;
use URI;
my $loop = IO::Async::Loop->new();
my $http = Net::Async::HTTP->new();
$loop->add( $http );
my ( $response ) = $http->do_request(
uri => URI->new( "http://www.cpan.org/" ),
)->get;
print "Front page of http://www.cpan.org/ is:\n";
print $response->as_string;
This object class implements an asynchronous HTTP user agent. It sends requests
to servers, returning Future instances to yield responses when they are
received. The object supports multiple concurrent connections to servers, and
allows multiple requests in the pipeline to any one connection. Normally, only
one such object will be needed per program to support any number of requests.
As well as using futures the module also supports a callback-based interface.
This module optionally supports SSL connections, if IO::Async::SSL is installed.
If so, SSL can be requested either by passing a URI with the "https"
scheme, or by passing a true value as the "SSL" parameter.
There are three ways in which connections to HTTP server hosts are managed by
this object, controlled by the value of "max_connections_per_host".
This controls when new connections are established to servers, as compared to
waiting for existing connections to be free, as new requests are made to them.
They are:
- max_connections_per_host = 1
- This is the default setting. In this mode, there will be one connection
per host on which there are active or pending requests. If new requests
are made while an existing one is outstanding, they will be queued to wait
for it.
If pipelining is active on the connection (because both the
"pipeline" option is true and the connection is known to be an
HTTP/1.1 server), then requests will be pipelined into the connection
awaiting their response. If not, they will be queued awaiting a response
to the previous before sending the next.
- max_connections_per_host > 1
- In this mode, there can be more than one connection per host. If a new
request is made, it will try to re-use idle connections if there are any,
or if they are all busy it will create a new connection to the host, up to
the configured limit.
- max_connections_per_host = 0
- In this mode, there is no upper limit to the number of connections per
host. Every new request will try to reuse an idle connection, or else
create a new one if all the existing ones are busy.
These modes all apply per hostname / server port pair; they do not affect the
behaviour of connections made to differing hostnames, or differing ports on
the same hostname.
The following named parameters may be passed to "new" or
"configure":
A string to set in the "User-Agent" HTTP header. If not supplied, one
will be constructed that declares "Net::Async::HTTP" and the version
number.
Optional. How many levels of redirection to follow. If not supplied, will
default to 3. Give 0 to disable redirection entirely.
Optional. The maximum number of in-flight requests to allow per host when
pipelining is enabled and supported on that host. If more requests are made
over this limit they will be queued internally by the object and not sent to
the server until responses are received. If not supplied, will default to 4.
Give 0 to disable the limit entirely.
Optional. Controls the maximum number of connections per hostname/server port
pair, before requests will be queued awaiting one to be free. Give 0 to
disable the limit entirely. See also the "Connection Pooling"
section documented above.
Currently, if not supplied it will default to 1. However, it has been found in
practice that most programs will raise this limit to something higher, perhaps
3 or 4. Therefore, a future version of this module may set a higher value.
To test if your application will handle this correctly, you can set a different
default by setting an environment variable:
$ NET_ASYNC_HTTP_MAXCONNS=3 perl ...
Optional. How long in seconds to wait before giving up on a request. If not
supplied then no default will be applied, and no timeout will take place.
Optional. How long in seconds to wait after each write or read of data on a
socket, before giving up on a request. This may be more useful than
"timeout" on large-file operations, as it will not time out provided
that regular progress is still being made.
Optional. Default values to apply to each "request" method.
Optional. A reference to a HTTP::Cookies object. Will be used to set cookies in
requests and store them from responses.
Optional. If false, disables HTTP/1.1-style request pipelining.
Optional. Parameters to pass on to the "connect" method used to
connect sockets to HTTP servers. Sets the socket family and local socket
address to "bind()" to. For more detail, see the documentation in
IO::Async::Connector.
Optional. Affects the behaviour of response handling when a "4xx" or
"5xx" response code is received. When false, these responses will be
processed as other responses and yielded as the result of the future, or
passed to the "on_response" callback. When true, such an error
response causes the future to fail, or the "on_error" callback to be
invoked.
The HTTP response and request objects will be passed as well as the code and
message, and the failure name will be "http".
( $code_message, "http", $response, $request ) = $f->failure
$on_error->( "$code $message", $response, $request )
Optional. Used to set the reading and writing buffer lengths on the underlying
"IO::Async::Stream" objects that represent connections to the
server. If not define, a default of 64 KiB will be used.
Optional. Used to set the "IP_TOS" socket option on client sockets. If
given, should either be a "IPTOS_*" constant, or one of the string
names "lowdelay", "throughput", "reliability" or
"mincost". If undefined or left absent, no option will be set.
Optional. If true, incoming responses that have a recognised
"Content-Encoding" are handled by the module, and decompressed
content is passed to the body handling callback or returned in the
"HTTP::Response". See "CONTENT DECODING" below for details
of which encoding types are recognised. When this option is enabled, outgoing
requests also have the "Accept-Encoding" header added to them if it
does not already exist.
Currently the default is false, because this behaviour is new, but it may
default to true in a later version. Applications which care which behaviour
applies should set this to a defined value to ensure it doesn't change.
Additionally, any parameters whose names start with "SSL_" will be
stored and passed on requests to perform SSL requests. This simplifies
configuration of common SSL parameters.
Optional. If true, then any attempt to make a request that does not use SSL
(either by calling "request", or as a result of a redirection) will
immediately fail.
Since version 0.42.
Additionally, any parameters whose names start with "SOCKS_" will be
stored and used by Net::Async::SOCKS to establish connections via a configured
proxy.
The following methods documented with a trailing call to "->get"
return Future instances.
When returning a Future, the following methods all indicate HTTP-level errors
using the Future failure name of "http". If the error relates to a
specific response it will be included. The original request is also included.
$f->fail( $message, "http", $response, $request )
Send an HTTP request to a server, returning a Future that will yield the
response. The request may be represented by an HTTP::Request object, or a URI
object, depending on the arguments passed.
The following named arguments are used for "HTTP::Request"s:
- request => HTTP::Request
- A reference to an "HTTP::Request" object
- host => STRING
- Hostname of the server to connect to
- port => INT or STRING
- Optional. Port number or service of the server to connect to. If not
defined, will default to "http" or "https" depending
on whether SSL is being used.
- family => INT
- Optional. Restricts the socket family for connecting. If not defined, will
default to the globally-configured value in the object.
- SSL => BOOL
- Optional. If true, an SSL connection will be used.
The following named arguments are used for "URI" requests:
- uri => URI or STRING
- A reference to a "URI" object, or a plain string giving the
request URI. If the scheme is "https" then an SSL connection
will be used.
- method => STRING
- Optional. The HTTP method name. If missing, "GET" is used.
- content => STRING or ARRAY ref
- Optional. The body content to use for "PUT" or "POST"
requests.
If this is a plain scalar it will be used directly, and a
"content_type" field must also be supplied to describe it.
If this is an ARRAY ref and the request method is "POST", it will
be form encoded. It should contain an even-sized list of field names and
values. For more detail see "POST" in
HTTP::Request::Common.
- content_type => STRING
- The type of non-form data "content".
- user => STRING
- pass => STRING
- Optional. If both are given, the HTTP Basic Authorization header will be
sent with these details.
- headers => ARRAY|HASH
- Optional. If provided, contains additional HTTP headers to set on the
constructed request object. If provided as an ARRAY reference, it should
contain an even-sized list of name/value pairs.
- proxy_host => STRING
- proxy_port => INT
- Optional. Override the hostname or port number implied by the URI.
For either request type, it takes the following arguments:
- request_body => STRING | CODE | Future
- Optional. Allows request body content to be generated by a future or
callback, rather than being provided as part of the "request"
object. This can either be a plain string, a "CODE" reference to
a generator function, or a future.
As this is passed to the underlying IO::Async::Stream "write"
method, the usual semantics apply here. If passed a "CODE"
reference, it will be called repeatedly whenever it's safe to write. The
code should should return "undef" to indicate completion. If
passed a "Future" it is expected to eventually yield the body
value.
As with the "content" parameter, the "content_type"
field should be specified explicitly in the request header, as should the
content length (typically via the HTTP::Request "content_length"
method). See also examples/PUT.pl.
- expect_continue => BOOL
- Optional. If true, sets the "Expect" request header to the value
"100-continue" and does not send the "request_body"
parameter until a "100 Continue" response is received from the
server. If an error response is received then the "request_body"
code, if present, will not be invoked.
- on_ready => CODE
- Optional. A callback that is invoked once a socket connection is
established with the HTTP server, but before the request is actually sent
over it. This may be used by the client code to inspect the socket, or
perform any other operations on it. This code is expected to return a
"Future"; only once that has completed will the request cycle
continue. If it fails, that failure is propagated to the caller.
$f = $on_ready->( $connection )
- on_redirect => CODE
- Optional. A callback that is invoked if a redirect response is received,
before the new location is fetched. It will be passed the response and the
new URL.
$on_redirect->( $response, $location )
- on_body_write => CODE
- Optional. A callback that is invoked after each successful
"syswrite" of the body content. This may be used to implement an
upload progress indicator or similar. It will be passed the total number
of bytes of body content written so far (i.e. excluding bytes consumed in
the header).
$on_body_write->( $written )
- max_redirects => INT
- Optional. How many levels of redirection to follow. If not supplied, will
default to the value given in the constructor.
- timeout => NUM
- stall_timeout => NUM
- Optional. Overrides the object's configured timeout values for this one
request. If not specified, will use the configured defaults.
On a timeout, the returned future will fail with either "timeout"
or "stall_timeout" as the operation name.
( $message, "timeout" ) = $f->failure
When not returning a future, the following extra arguments are used as callbacks
instead:
- on_response => CODE
- A callback that is invoked when a response to this request has been
received. It will be passed an HTTP::Response object containing the
response the server sent.
$on_response->( $response )
- on_header => CODE
- Alternative to "on_response". A callback that is invoked when
the header of a response has been received. It is expected to return a
"CODE" reference for handling chunks of body content. This
"CODE" reference will be invoked with no arguments once the end
of the request has been reached, and whatever it returns will be used as
the result of the returned "Future", if there is one.
$on_body_chunk = $on_header->( $header )
$on_body_chunk->( $data )
$response = $on_body_chunk->()
- on_error => CODE
- A callback that is invoked if an error occurs while trying to send the
request or obtain the response. It will be passed an error message.
$on_error->( $message )
If this is invoked because of a received "4xx" or "5xx"
error code in an HTTP response, it will be invoked with the response and
request objects as well.
$on_error->( $message, $response, $request )
Convenient wrappers for performing "GET", "HEAD",
"PUT" or "POST" requests with a "URI" object and
few if any other arguments, returning a "Future".
Remember that "POST" with non-form data (as indicated by a plain
scalar instead of an "ARRAY" reference of form data name/value
pairs) needs a "content_type" key in %args.
The following methods are intended as points for subclasses to override, to add
extra functionallity.
Called just before the "HTTP::Request" object is sent to the server.
Called after a non-redirect "HTTP::Response" has been received from a
server. The originating request will be set in the object.
If the required decompression modules are installed and available, compressed
content can be decoded. If the received "Content-Encoding" is
recognised and the required module is available, the content is transparently
decoded and the decoded content is returned in the resulting response object,
or passed to the data chunk handler. In this case, the original
"Content-Encoding" header will be deleted from the response, and its
value will be available instead as "X-Original-Content-Encoding".
The following content encoding types are recognised by these modules:
- •
- gzip (q=0.7) and deflate (q=0.5)
Recognised if Compress::Raw::Zlib version 2.057 or newer is installed.
- •
- bzip2 (q=0.8)
Recognised if Compress::Bzip2 version 2.10 or newer is installed.
Other content encoding types can be registered by calling the following method
Registers an encoding type called $name, at the quality value $q. In order to
decode this encoding type, $make_decoder will be invoked with no paramters,
and expected to return a CODE reference to perform one instance of decoding.
$decoder = $make_decoder->()
This decoder will be invoked on string buffers to decode them until the end of
stream is reached, when it will be invoked with no arguments.
$content = $decoder->( $encoded_content )
$content = $decoder->() # EOS
The "Future"-returning "GET" method makes it easy to await
multiple URLs at once, by using the Future::Utils "fmap_void"
utility
my @URLs = ( ... );
my $http = Net::Async::HTTP->new( ... );
$loop->add( $http );
my $future = fmap_void {
my ( $url ) = @_;
$http->GET( $url )
->on_done( sub {
my $response = shift;
say "$url succeeded: ", $response->code;
say " Content-Type":", $response->content_type;
} )
->on_fail( sub {
my $failure = shift;
say "$url failed: $failure";
} );
} foreach => \@URLs;
$loop->await( $future );
- •
- <http://tools.ietf.org/html/rfc2616> - Hypertext Transfer Protocol
-- HTTP/1.1
Parts of this code, or bugfixes to it were paid for by
- •
- SocialFlow <http://www.socialflow.com>
- •
- Shadowcat Systems <http://www.shadow.cat>
- •
- NET-A-PORTER <http://www.net-a-porter.com>
- •
- Cisco <http://www.cisco.com>
Paul Evans <leonerd@leonerd.org.uk>