There are a lot of cases when you'd want to have some code accessing a web site. It might be to automate some human interaction or it can be because you would like to talk to an API. Let's see a few simple examples in Perl 6.
httpbin.org is an excellent web site that allows us to try all kinds of web client code. We are going to use that in our examples.
Before you use it, you'll need to install LWP::Simple. Just run:
zef install LWP::Simple
Probably the most simple request is the HTTP GET request.
We just need to create an instance of LWP::Simple and call the get method of it.
examples/http_get.pl6
use v6;
use LWP::Simple;
my $html = LWP::Simple.new.get("http://httpbin.org/ip");
say $html;
The response will look something like this, containing your IP address or at least the IP address of the gateway you are using to access the Internet.
{
"origin": "17.19.208.37"
}
Using plain HTTP is fine for public data, but once you start sending and receiving private information, such as passwords, most of the web sites will work only on top of HTTPS. Luckily httpbin.org is available over https as well, so we can try out code switching http to https in our request:
examples/https_get.pl6
use v6;
use LWP::Simple;
my $html = LWP::Simple.new.get("https://httpbin.org/ip");
say $html;
If you run this you might get an error message:
501 Protocol scheme 'https' is only supported if IO::Socket::SSL is installed
You will then need to install IO::Scoket::SSL for this to work.
zef install IO::Socket::SSL
After it is successfully installed, running the above program again will get you the same response as before.
A slightly more interesting HTTP request would be another GET request, but this time with some parameters.
examples/http_get_params.pl6
use v6;
use LWP::Simple;
my $html = LWP::Simple.new.get("http://httpbin.org/get?name=Larry%20Wall&language;=Perl&math;=19%2B23%3D42");
say $html;
In this we pass three parameters with their respective values:
name => "Larry Wall" language => "Perl" math => "19+23=42"
In the URL it looks like this:
?name=Larry%20Wall&language;=Perl&math;=19%2B23%3D42
The ? separates the web address from the parameters. & separates the key-value pairs. Each pair is has the key followed by =, followed by the value.
The most problematic part though is that we need to escape the special characters passed in the URL. Before we take a look at how we can generate that, let's see the result of the above Perl script:
{
"args": {
"language": "Perl",
"math": "19+23=42",
"name": "Larry Wall"
},
"headers": {
"Connection": "close",
"Host": "httpbin.org",
"User-Agent": "LWP::Simple/0.090 Perl6/rakudo"
},
"origin": "17.19.208.37",
"url": "http://httpbin.org/get?name=Larry Wall&language;=Perl&math;=19%2B23%3D42"
}
In the args field we see the key-value pairs as we intended. So the server saw them correctly and could return them to us. Then we see the header our request generated. You can see the User-Agent field was set by the LWP::Simple library to reveal itself.
Finally we see our IP address and the original request.
BTW You could also try the above URL in your own browser. Just follow this link. The output will be similar, but the User-Agent field will contain information about your browser and there will be probably a lot more fields in the header.
The URI::Escape module is part of the URI library. If you don't have it, install it using
zef install URI
Given a hash of key-value pairs we can use the following code snippet to generate the parameter part of the request:
examples/uri_escape.pl6
use v6;
use URI::Escape;
my %params =
name => "Larry Wall",
language => "Perl",
math => "19+23=42",
;
say %params.kv.map( -> $k, $v { "$k=&uri-escape;($v)" }).join('&');
Combining the two we get the following script:
examples/http_get_uri_escape.pl6
use v6;
use URI::Escape;
use LWP::Simple;
my %params =
name => "Larry Wall",
language => "Perl",
math => "19+23=42",
;
my $url-params = %params.kv.map( -> $k, $v { "$k=&uri-escape;($v)" }).join('&');
my $html = LWP::Simple.new.get("http://httpbin.org/get?$url-params");
say $html;
It looks much more readable than the previous solution.
HTTP POST requests can be sent and tested in a similar way.
examples/http_post.pl6
use v6;
use URI::Escape;
use LWP::Simple;
my %params =
name => "Larry Wall",
language => "Perl",
math => "19+23=42",
;
my $url-params = (map { "{.key}={uri-escape(.value)}" }, %params).join('&');
my $html = LWP::Simple.new.post("http://httpbin.org/post", {}, $url-params);
say $html;
Here we call the post method, pass an empty hash, we'll get to it later, and then we pass a URI-Escaped string of the parameters.
The result of the request is quite similar to the previous one, but instead of having the request data parsed in the args field, we get it back in the data field.
{
"args": {},
"data": "name=Larry%20Wall&language;=Perl&math;=19%2B23%3D42",
"files": {},
"form": {},
"headers": {
"Connection": "close",
"Content-Length": "49",
"Host": "httpbin.org",
"User-Agent": "LWP::Simple/0.090 Perl6/rakudo"
},
"json": null,
"origin": "37.26.148.213",
"url": "http://httpbin.org/post"
}
Earlier we have noted that the User-Agent identifying our "browser" was set to LWP::Simple/0.090 Perl6/rakudo by LWP::Simple. We can however change that. The empty hash in the POST request we have seen can be used to set any header values, including the User-Agent.
So in the next example we do just that. Set the User-Agent:
examples/http_post_set_user_agent.pl6
use v6;
use URI::Escape;
use LWP::Simple;
my %params =
name => "Larry Wall",
language => "Perl",
math => "19+23=42",
;
my $url-params = %params.kv.map( -> $k, $v { "$k=&uri-escape;($v)" }).join('&');
my $html = LWP::Simple.new.post("http://httpbin.org/post",
{
"User-Agent" => "Perl 6 Maven"
}, $url-params);
say $html;
The result looks like this:
{
"args": {},
"data": "name=Larry%20Wall&language;=Perl&math;=19%2B23%3D42",
"files": {},
"form": {},
"headers": {
"Connection": "close",
"Content-Length": "49",
"Host": "httpbin.org",
"User-Agent": "Perl 6 Maven"
},
"json": null,
"origin": "37.26.148.213",
"url": "http://httpbin.org/post"
}
Recent versions of LWP::Simple also allow us to set the header in GET requests. We just need to pass a hash of the header fields:
examples/http_get_change_header.pl6
use v6;
use LWP::Simple;
my $html = LWP::Simple.new.get("http://httpbin.org/headers", {
"User-Agent" => "Perl 6 Maven articles",
"Zone" => "q" }
);
say $html;
In the response we can see that the server saw the new User-Agent and also the arbitrry field "Zone".
{
"headers": {
"Connection": "close",
"Host": "httpbin.org",
"User-Agent": "Perl 6 Maven articles",
"Zone": "q"
}
}
Published on 2017-07-15