Camelia

LWP::Simple - a simple web client in Perl 6

There are a lot of cases when you'd want to have some code accessing a web site. It might be to automate some human interaction or it can be because you would like to talk to an API. Let's see a few simple examples in Perl 6.

httpbin.org is an excellent web site that allows us to try all kinds of web client code. We are going to use that in our examples.

Note! This site is about Perl 6.
If you are looking for a solution for Perl 5, please check out the Perl 5 tutorial.

Install LWP::Simple

Before you use it, you'll need to install LWP::Simple. Just run:

zef install LWP::Simple

GET http

Probably the most simple request is the HTTP GET request.

We just need to create an instance of LWP::Simple and call the get method of it.

examples/http_get.pl6

use v6;
use LWP::Simple;

my $html = LWP::Simple.new.get("http://httpbin.org/ip");
say $html;

The response will look something like this, containing your IP address or at least the IP address of the gateway you are using to access the Internet.

{
  "origin": "17.19.208.37"
}

GET https

Using plain HTTP is fine for public data, but once you start sending and receiving private information, such as passwords, most of the web sites will work only on top of HTTPS. Luckily httpbin.org is available over https as well, so we can try out code switching http to https in our request:

examples/https_get.pl6

use v6;
use LWP::Simple;

my $html = LWP::Simple.new.get("https://httpbin.org/ip");
say $html;

If you run this you might get an error message:

501 Protocol scheme 'https' is only supported if IO::Socket::SSL is installed

You will then need to install IO::Scoket::SSL for this to work.

zef install IO::Socket::SSL

After it is successfully installed, running the above program again will get you the same response as before.

HTTP GET with parameters in the URL

A slightly more interesting HTTP request would be another GET request, but this time with some parameters.

examples/http_get_params.pl6

use v6;
use LWP::Simple;

my $html = LWP::Simple.new.get("http://httpbin.org/get?name=Larry%20Wall&language=Perl&math=19%2B23%3D42");
say $html;

In this we pass three parameters with their respective values:

  name     => "Larry Wall"
  language => "Perl"
  math     => "19+23=42"

In the URL it looks like this:

?name=Larry%20Wall&language=Perl&math=19%2B23%3D42

The ? separates the web address from the parameters. & separates the key-value pairs. Each pair is has the key followed by =, followed by the value.

The most problematic part though is that we need to escape the special characters passed in the URL. Before we take a look at how we can generate that, let's see the result of the above Perl script:

{
  "args": {
    "language": "Perl",
    "math": "19+23=42",
    "name": "Larry Wall"
  },
  "headers": {
    "Connection": "close",
    "Host": "httpbin.org",
    "User-Agent": "LWP::Simple/0.090 Perl6/rakudo"
  },
  "origin": "17.19.208.37",
  "url": "http://httpbin.org/get?name=Larry Wall&language=Perl&math=19%2B23%3D42"
}

In the args field we see the key-value pairs as we intended. So the server saw them correctly and could return them to us. Then we see the header our request generated. You can see the User-Agent field was set by the LWP::Simple library to reveal itself.

Finally we see our IP address and the original request.

BTW You could also try the above URL in your own browser. Just follow this link. The output will be similar, but the User-Agent field will contain information about your browser and there will be probably a lot more fields in the header.

URI::Escape

The URI::Escape module is part of the URI library. If you don't have it, install it using

zef install URI

Given a hash of key-value pairs we can use the following code snippet to generate the parameter part of the request:

examples/uri_escape.pl6

use v6;
use URI::Escape;

my %params = 
   name     => "Larry Wall",
   language => "Perl",
   math     => "19+23=42",
;

say %params.kv.map( -> $k, $v { "$k=&uri-escape($v)" }).join('&');

HTTP GET with URI::Escape

Combining the two we get the following script:

examples/http_get_uri_escape.pl6

use v6;
use URI::Escape;
use LWP::Simple;

my %params = 
   name     => "Larry Wall",
   language => "Perl",
   math     => "19+23=42",
;

my $url-params = %params.kv.map( -> $k, $v { "$k=&uri-escape($v)" }).join('&');

my $html = LWP::Simple.new.get("http://httpbin.org/get?$url-params");
say $html;

It looks much more readable than the previous solution.

HTTP POST requests

HTTP POST requests can be sent and tested in a similar way.

examples/http_post.pl6

use v6;
use URI::Escape;
use LWP::Simple;

my %params = 
   name     => "Larry Wall",
   language => "Perl",
   math     => "19+23=42",
;

my $url-params = (map { "{.key}={uri-escape(.value)}" }, %params).join('&');

my $html = LWP::Simple.new.post("http://httpbin.org/post", {}, $url-params);
say $html;

Here we call the post method, pass an empty hash, we'll get to it later, and then we pass a URI-Escaped string of the parameters.

The result of the request is quite similar to the previous one, but instead of having the request data parsed in the args field, we get it back in the data field.

{
  "args": {},
  "data": "name=Larry%20Wall&language=Perl&math=19%2B23%3D42",
  "files": {},
  "form": {},
  "headers": {
    "Connection": "close",
    "Content-Length": "49",
    "Host": "httpbin.org",
    "User-Agent": "LWP::Simple/0.090 Perl6/rakudo"
  },
  "json": null,
  "origin": "37.26.148.213",
  "url": "http://httpbin.org/post"
}

Set the User-Agent

Earlier we have noted that the User-Agent identifying our "browser" was set to LWP::Simple/0.090 Perl6/rakudo by LWP::Simple. We can however change that. The empty hash in the POST request we have seen can be used to set any header values, including the User-Agent.

So in the next example we do just that. Set the User-Agent:

examples/http_post_set_user_agent.pl6

use v6;
use URI::Escape;
use LWP::Simple;

my %params = 
   name     => "Larry Wall",
   language => "Perl",
   math     => "19+23=42",
;

my $url-params = %params.kv.map( -> $k, $v { "$k=&uri-escape($v)" }).join('&');

my $html = LWP::Simple.new.post("http://httpbin.org/post",
    {
        "User-Agent" => "Perl 6 Maven"
    }, $url-params);
say $html;


The result looks like this:

{
  "args": {},
  "data": "name=Larry%20Wall&language=Perl&math=19%2B23%3D42",
  "files": {},
  "form": {},
  "headers": {
    "Connection": "close",
    "Content-Length": "49",
    "Host": "httpbin.org",
    "User-Agent": "Perl 6 Maven"
  },
  "json": null,
  "origin": "37.26.148.213",
  "url": "http://httpbin.org/post"
}

Setting User-Agent and other header fields in GET

Recent versions of LWP::Simple also allow us to set the header in GET requests. We just need to pass a hash of the header fields:

examples/http_get_change_header.pl6

use v6;
use LWP::Simple;

my $html = LWP::Simple.new.get("http://httpbin.org/headers", {
    "User-Agent" => "Perl 6 Maven articles",
    "Zone" => "q" }
);
say $html;

In the response we can see that the server saw the new User-Agent and also the arbitrry field "Zone".

{
  "headers": {
    "Connection": "close",
    "Host": "httpbin.org",
    "User-Agent": "Perl 6 Maven articles",
    "Zone": "q"
  }
}

The Perl 6 Tricks and Treats newsletter has been around for a while. If you are interested to get special notification when there is new content on this site, it is the best way to keep track:
Email:
Full name:
This is a newsletter temporarily running on my personal site (szabgab.com) using Mailman, till I implement an alternative system in Perl 6.
Gabor Szabo
Written by Gabor Szabo

Published on 2017-07-15



Comments

In the comments, please wrap your code snippets within <pre> </pre> tags and use spaces for indentation.
comments powered by Disqus
Suggest a change
Elapsed time: 2.34170328

Perl 6 Tricks and Treats newsletter

Register to the free newsletter now, and get updates and news.
Email:
Name: