Secure Scraping

There are a few CPAN modules I consider "must-haves" and as of today WWW::Mechanize is on that list. If you've ever had to do any screen scraping, parsing HTML on web pages for information, you'll want to try this module too.

In recent post I discussed the creation of a stopwords database. The next step was to create a more reusable perl module to access that database. This isn't terribly complex: essentially we just want to know if a particular word is in the database or not. We are, however, going to use an object oriented interface.

I gave myself a challenge when I started this blog: 100% of the software used to run it would be written in Perl. That seemed fairly obvious given the title, but I immediately wanted something more suited to formatting HTML. Yes, there are more than enough Perl HTML-template tools available for such a job, but I'm picky, and I didn't want to have to learn some invented mini-language to format text, when I already knew the best language ever invented for formatting text. What I came up with has worked very well during it's usage here, and now you can try it yourself. My Markup::Perl module is now available on the CPAN.

Before long every line of code that is used to run the On Perl blog will be publicly available here, but this is the first step in that direction: the data back-end. This site is actually more than a blog--to me anyway--it is also a chance to experiment, so in every reasonable way I've tried to create better and newer ways of doing things. I've made some decisions I soon regretted, and then, a little wiser, made improvements. When considering how I would handle the back-end I was somewhere between the almost obvious MySQL choice, and the seemingly more elegant, sleeker file-based choice of Bloxsom.

If you're sharing a server, the first time you use CPAN you may find that it's already configured for the root account, in which case you'll get a load of errors about incorrect permissions to install anything. You can either try and get the root password, or (more likely) set up your own CPAN install. It's straightforward, if just a bit tedius.

mkdir -p ~/.cpan/CPAN
cp /usr/lib/perl5/5.8.5/CPAN/Config.pm ~/.cpan/CPAN/MyConfig.pm
perl -p -i -e 's!/root/!/home/michael/!g' ~/.cpan/CPAN/MyConfig.pm
perl -MCPAN -eshell
cpan> o conf makepl_arg "LIB=/home/michael/perl
INSTALLMAN1DIR=/home/michael/perl/man1
INSTALLMAN3DIR=/home/michael/perl/man3"
cpan> o conf commit
cpan> install Some::Thing

I've had a couple jobs recently that required scripts to automatically manipulate images. I wrote one in PHP, using ImageMagick, and the other was in Perl, also using Image::Magick. I decided I wanted to do some of this sort of stuff on my Mac OS X iBook, but for various reasons wanted to give the GD module a try instead. I set aside an entire afternoon of free time, warned the family to stay clear, and prepared for the coming hours of thrashing in installation hell. Surprisingly, before I could even finish brewing the first cup of coffee, I was done.

While testing a recently released module I came across a mysterious and generally useless error message from the CPAN installer. Well, technically it came from the makeutility, but the CPAN installer was the one supposedly in charge.

The point of a comment is to do nothing, so what good are they? Well, you might be surprised. Firstly they are an invaluable form of documentation. If you're trying to work through a complicated process start with comments.

If you're working on a system that has multiple versions of perl installed with perhaps multiple users and a few different library locations, you might be confused about where to look when you get a problem related to a module. Where is the code for that module actually located? Got more than one of those modules installed in different places, which one is being used? Which version?

Randal Schwartz said it was like "training wheels without the bike" -- so why would anyone want to turn Perl into PHP? Dag Ågren explains it as a "travesty", a "joke" and something to "make Baby Jesus cry." We know it as PerlHP, and it comes pretty close to putting training wheels on the carbon-alloy LeMond racing bike of Perl. But is that a good thing?