Upload huge files with Perl’s LWP::UserAgent

Everybody who work with Perl knows LWP::UserAgent, the most used library when you need to work with HTTP connections.

The library has some methods that cover the most common usage cases, such as GET and POST request.
If you need something more particular you have to set up a HTTP::Request object and pass it to LWP::UserAgent’s request method. I don’t think I’m saying something unexpected.

I recently got some problems by sending a bug file. It wasn’t really big because I’m talking about a 100 MB file, but it was bit enough to send my small VPS server out of memory. This was because I needed to pass it as raw POST payload for Google Drive API and to do that I was slurping it into memory. A bad idea.

Continua la lettura di Upload huge files with Perl’s LWP::UserAgent

Extending PDOStatement for errors checking and other tasks

PDO is a great OO library for interacting with your database from PHP. It has a very consistent interface and your code will be cleaner than ever. And your PHP script will be more portable because PDO is not related to any database, but is a generic library to work with MySQL, PostgreSQL, SQLite and a lot others.

The only thing I miss in PDO is a better query error management. When I execute a query I have to check every time the PDOStatement->errorCode property to get sure everything is working well. That’s very annoying.

Fortunately there is a way to extend the PDOStatement class to get more comfortable in writing our script. In the official docs there is not a lot about this, but here is an explanation.

The official documentation talks only about a PDO::ATTR_STATEMENT_CLASS attribute for the PDO instance to be set through the setAttribute method. That’s not much, but here is an example:

try {
  $dbh = new PDO(
    "mysql:dbname=my_database;host=localhost;charset=utf8",
    $user,
    $pass
  );
  $dbh->setAttribute(PDO::ATTR_STATEMENT_CLASS, array('MyPDOStatement'));
} catch (PDOException $e) {
  die('Connection failed: ' . $e->getMessage());
}

You can set the attribute after the connection, not a problem. MyPDOStatement is a custom class that extends PDOStatement. A minimal version will look like this:

class MyPDOStatement extends PDOStatement
{
  private function __construct() {}

  function execute($input = array()) {
    parent::execute($input);

    if ($this->errorCode() !== '00000') {
      $err = $this->errorInfo();
      error_log(SQLException($err[2], $err[1]));
      return false;
    }

    return true;
  }
}

The private constructor is mandatory. You can use it to pass some variable to every MyPDOStatement instance, very useful in many situations.

For example we could use an array to track query execution times. So the code will look like this:

$info = array('queries' => 0, 'query_time' => 0);

try {
  $dbh = new PDO(
    "mysql:dbname=my_database;host=localhost;charset=utf8",
    $user,
    $pass
  );
  $dbh->setAttribute(PDO::ATTR_STATEMENT_CLASS, array('MyPDOStatement', array($info)));
} catch (PDOException $e) {
  die('Connection failed: ' . $e->getMessage());
}

class MyPDOStatement extends PDOStatement
{
  private $info;

  private function __construct(&info) {
    $this->info = &$info;
  }

  function execute($input = array()) {
    $t = microtime(true);
    parent::execute($input);
    $t = microtime(true) - $t;

    if ($this->errorCode() !== '00000') {
      $err = $this->errorInfo();
      error_log(SQLException($err[2], $err[1]));
      return false;
    }

    $this->info['queries']++;
    $this->info['query_time'] += $t;

    return true;
  }
}

Display usage hints for your Perl script

Perl is an awesome language. I like it very much more than Python because you can put some imagination inside your code, doing things with your style. I like very much the loose typing because you don’t have to care about the content in your variables: numbers, strings, objects are almost the same. You have to care about what you are doing, not how.

Commenting your code is very important for you and for everybody else that will see it. And if you are writing a standalone script it’s very useful to give hints about the usage without reading the code. Perl has a integrated comment system called Pod that can be used to generated documentation files in a number of format, but how can you display something on the command line by using the common –help parameter?

The answer is the Pod::Usage package. It’s a very powerful package, but I will use it in a minimal way. The official documentation will tell you everything you need to know, but let’s take a taste.

#!/usr/bin/perl
=head1 Example script

This is an example script to taste the Pod::Usage package.

=head1 SYNOPSIS

example.pl [--help] --source=1.2.3.4 --target=5.6.7.8

Options:
--help Displays this help
--source A source
--target A target

=cut

use strict;
use Getopt::Long;
use Pod::Usage;

my ($help, $source, $target);

GetOptions(
'source=s' => $source,
'target=s' => $target,
'help' => $help
);

pod2usage(1) if $help;

[...]

So how does this work? I use the Getopt::Long package to manage command line parameters. I pass two string parameters (source, target) and a flag (help). If flag is set I call the pod2usage function from Pod::Usage with a ‘1’ as parameter. The function will print out everything contained inside the SYNOPSYS paragraph and will exit the script with code 1 (or whatever number you passed).  That’s all!

Send mail with Sendmail from Bash command line

eMails are still the most important communication channel on internet. Most of the web applications you can find on the web require an eMail address to register because they will use it for important communications, to recover you password and so on. It’s still a reliable way to identify the users.

Well, today I was to write a monitor script who checks a server status and send an eMail to the admin if something wrong happens. For some reason I wanted to write it in Bash script and put it directly into the /etc/cron.daily/ folder. The check was pretty easy to write but then I had to send the eMail.

After some searching and trying I finally found out how to send emails directly from Bash with Sendmail and no need to write any file:

echo -e "To: "admin" <admin@example.com>nFrom: "The Server" <server@example.com>rnContent_Type: text/plainnSubject: [Server] Unexpected responsennResponse was wrongn" | sendmail -t

Now a little bit of explanation.

Basically I send all the needed headers and content to Sendmail with a pipe. Sendmail’s -t option forces to search the recipients addresses inside the header. Normally Sendmail expects this kind of addresses as a parameter, but this way everything has the same source, I mean the pipe.

The echo statement sends all the needed data to Sendmail. Remember the -e option to replace the backslashed special characters. This is very important because without it Sendmail will do not understand anything about you want to send.

You can add any kind of headers you need, such as the Cc and the Bcc fields or others, and check that after the last header and before the mail’s content there are two new line characters.

Remove “compatibility view” button in Internet Explorer

Old Internet Explorer version are often a big problem for web developers. Fortunately everyday the people using IE 7, IE6 or even IE5.5 became lesser and lesser. So usually we don’t have to spend to much time to fix everything in our CSS to make every page look perfect in all this old browsers because people using them are not so relevant.

Passing from IE7 to IE8 Microsoft  had the “brilliant” idea to help developers with this huge step forward, in particular enterprise sites was in past often strictly related to a browser version and the change could be a big problem for them. The solution was the “compatibility view” button, the button with a broken page icon, which switches your modern browser to an old IE7 mode. Holy crap! :-O

In real life the button is almost useless because Internet sites are already prepared to offer the content to the browser in a best way using different solutions (conditional comments, server side browser detection, javascript, CSS hacks, ignoring…) BUT sometimes users could press the stupid button while trying to press the “Reload Page” button. It’s like a nuke because instantly your browser start to render pages like the old IE7 and everything will look broken or wrong.

The “Compatibility View” is domain scoped and remembers the setting even after the browser was closed. So what can we do?

We can say to Internet Explorer: hey don’t worry, everything is okay! It’s pretty easy, we have just to use the X-UA-Compatible header and set this value: IE=edge. In a HTML page you can use a meta tag:

<meta http-equiv="X-UA-Compatible" content="IE=edge">

Or you can choose to add it to your web server configuration. What does it mean? It says to Explorer to use his last standard mode. So IE8 will render the pageas IE8, IE9 like IE9 and so one and the “Compatibility Button” will disappear. Gotcha!