On Perl :: Index

There are a few CPAN modules I consider "must-haves" and as of today WWW::Mechanize is on that list. If you've ever had to do any screen scraping, parsing HTML on web pages for information, you'll want to try this module too.

In the example below, I've decided to use it to grab the most recent email subject, from my web-based email account. This would be simple except my account requires a web-based login, and this happens via a SSL connection and uses cookies. Could be a nightmare, but rest easy, WWW::Mechanize makes it seem almost trivial.

Before you get started make sure you have the required modules installed...

$ perl -MCPAN -eshell
cpan> install IO::Socket::SSL
cpan> install WWW::Mechanize
cpan> q

Now you can write something as simple as this...

use WWW::Mechanize;

my $url = "https://mail.example.com/login";
my $username = "joe_user";
my $password = "secret";

my $mech = WWW::Mechanize->new(
    agent => "Linux Mozilla",
    cookie_jar => {}
);

$mech->get($url);
unless ($mech->success) {
    die "Can't get login page $url: ",
    $mech->response->status_line;
}

$mech->field(Email => $username);
$mech->field(Passwd => $password);
$mech->click();

# scrape it...
my $content = $mech->content();
my ($latest) =
    $content =~ m{<td>(.+?)</td>}i;

print "Latest email: \"$latest\"\n";

Secure Scraping