Say Perl

I've just uploaded DBD::ODBC 1.33 to CPAN.

This release contains no new changes since the 1.32_5 development release but is the official release for all the 1.32 dev series. The complete changes can be found below. The main thrust has been Unicode improvements.

The most significant enhancements are:

• unicode support for catalog/schema/table/column names in table_info and column_info
• now using SQLGetTypeInfoW on unicode builds
• new odbc_driver_complete attribute to allow interaction with ODBC dialogs on Windows.
  I'd really like to hear from anyone who uses this as I've had no feedback so far - just drop me an email if you have

• and bug fixes:

  • remove a debugging printf - sorry, most embarrassing.

read more

If you're fortunate enough to have a test suite which allows parallel execution, one small prove feature can save you a lot of time.

prove, of course, is a relatively new utility included with Test::Harness and TAP::Harness. It handles many of the little details of running test programs and collecting and reporting the output; it's one of those utilities that looks really silly before you use it, and then becomes indispensible within a week.

prove of course has an option to run parallel tests. The -j# option allows you to specify how many test files to run at once. I've had good success with -j9 on my desktop machine; the right number depends on your tasks, the number of cores available, the amount of memory used by each process, and the runtime characteristics of each process.

prove's -l option adds the relative lib/ directory to Perl's include path, so that you can test pure-Perl code without running it through a build cycle or without having to add use lib '...'; lines to your test files.

The -r option searches a given directory recursively for .t files.

Thus, the command prove -lr -j9 t/ runs all of the .t files found under t/, up to nine at a time, and prefers modules found under lib/. This is useful.

Of course I have a shell alias with one more feature:

alias proveall='prove -j9 --state=slow,save -lr t'

prove's state flag saves information about the tests run. If you save state, subsequent runs can use that information to determine how to run tests again.

I often have several types of tests, especially for code with user interfaces and data models. The data model tests exercise business logic, and the UI tests exercise control flow and error handling. Usually the business tests take the longest to run—and usually only one or two test files take the most time. When prove saves the state of the test run, it can schedule those slow tests first so that the fast tests can run in the spots where the slow test blocks.

Again, this all depends on your workload. Much of my code is more IO bound than CPU bound. I've seen slow tests take 20% or more of total suite execution time after everything else has finished just because they have so many points where they have to wait.

I regularly have test suite times under 30 seconds (often closer to 10 or 12 seconds) on moderately large projects because I can exploit easy opportunities for parallelism. Certainly the right tweaking and scheduling could get me more benefit, but running proveall and making sure that parallelism is possible from the start gets me most of that benefit with almost no additional work.

(This isn't solely an academic obsession; in my measured personal experience, the more often I can run the entire test suite, the easier it is to find and fix bugs. I won't go as far as to say that continuous integration is a crutch, but if you're using CI and can't run the most important tests covering most of your code in 30 seconds, you're shortchanging yourself.)

Es ist wieder so weit - Perl Adventskalender 2011 sind online:

• Dancer Advent Calendar - the Dancer web framework's calendar
• Perl Advent Calendar (perladvent.org)
• Catalyst Advent Calendar - the Catalyst web framework's calendar
• Perl6 Advent Calendar

Via:

PreshBlog: Perl Advent Calendars for 2011

Hey everyone! I am starting a new article series called node.js modules you should know about. I have been using node for over 2 years now and I built Browserling startup using node so I know just about everything about it.

In this series I will go through a few dozen of node.js modules, give examples and explain where it's useful.

The first module in the series is dnode. Dnode is freestyle rpc library and it's written by James Halliday (SubStack) ― co-founder of Browserling and Testling.

Here is what it is. This is the server.js:

var dnode = require('dnode');

var server = dnode({
    mul : function (n, m, cb) { cb(n * m) }
});
server.listen(5050);

And here is the client.js:

var dnode = require('dnode');

dnode.connect(5050, function (remote) {
    remote.mul(10, 20, function (n) {
        console.log('10 * 20 = ' + n);
    });
});

Now when you run client.js, you get the output:

$ node client.js
200

See what it did? It called the mul function at server side from the client side and passed it arguments 10 and 20. They got multiplied at server side and the result got sent back to the client by calling cb.

It's important to stress that no code was passed along, all this happened purely through references. You can see the implementation dnode protocol in dnode-protocol github repo.

Here is a more complex example, where client calls server, which calls client again, which passes the result back to server, which then calls client and prints the result.

server.js:

In a rare moment (I hope) of stupidity last weekend when I was a little bored (bad cold stopped me doing what I really wanted to do) I looked through the DBD::ODBC TO_DO list and saw the "implement execute_array/execute_for_fetch" item which has been in that file for years. I know for sure implementing it will be a lot faster than using DBI's default methods (which effectively do a row at a time) mostly because I wrote an ODBC tutorial years ago showing how much faster binding arrays of parameters is (Easysoft ODBC-ODBC Bridge Performance White Paper) but I also knew it would be a PITA to write. The main problem is that DBD::ODBC has loads of workarounds for broken drivers and special cases.

read more

Mojolicious isn’t just useful for perl coders, it also includes a command line tool that can be quite handy for anybody who wants to get info from the web:

usage: /Users/marcus/perl5/perlbrew/perls/perl-5.14.2/bin/mojo get [OPTIONS] \
URL [SELECTOR] [COMMANDS]
  mojo get /
  mojo get mojolicio.us
  mojo get -v -r google.com
  mojo get --method POST --content 'trololo' mojolicio.us
  mojo get --header 'X-Bender: Bite my shiny metal ass!' mojolicio.us
  mojo get mojolicio.us 'head > title' text
  mojo get mojolicio.us .footer all
  mojo get mojolicio.us a attr href
  mojo get mojolicio.us '*' attr id
  mojo get mojolicio.us 'h1, h2, h3' 3 text

These options are available:
  --charset <charset>     Charset of HTML5/XML content, defaults to auto
                          detection or "UTF-8".
  --content <content>     Content to send with request.
  --header <name:value>   Additional HTTP header.
  --method <method>       HTTP method to use, defaults to "GET".
  --redirect              Follow up to 5 redirects.
  --verbose               Print verbose debug information to STDERR.

First, the name can be a bit awkward when you use it often. I tend to shorten it to ‘mg’:

    $ alias mg='mojo get'

mg is a command line utility similar to curl, but with some really neat tricks up it’s sleeve.

As you can see from the examples above, mg allows you to use familiar CSS selectors to process the response body. This turns out to be very useful. For instance, to get the links to the frontpaged apps on my site iusethis.com:

$ mg -r iusethis.com 'h2 a' attr href 

http://osx.iusethis.com/app/corebreach

http://osx.iusethis.com/app/terraray

http://osx.iusethis.com/app/preferencemanager

http://osx.iusethis.com/app/panoedit

http://osx.iusethis.com/app/findanyfile

http://osx.iusethis.com/app/webkit

http://osx.iusethis.com/app/nulanaslauncher

http://osx.iusethis.com/app/pagelayers

http://osx.iusethis.com/app/arrivalsampdepartures

http://osx.iusethis.com/app/iawriter

Check the top one for link tags:

$ mg -r http://osx.iusethis.com/app/corebreach link

<link href="http://osx.iusethis.com/appcast/corebreach" rel="alternate" title="Sparkle AppCast" type="application/rss+xml" />
<link href="http://osx.iusethis.com/comment/app.rss/corebreach" rel="alternate" title="Recent Comments" type="application/rss+xml" />
<link href="http://osx.iusethis.com/static/opensearch.xml" rel="search" title="iusethis" type="application/opensearchdescription+xml" />

Neat, an appcast, let’s take a look at the version history:

 $ mg http://osx.iusethis.com/appcast/corebreach title text
Appcast for CoreBreach
CoreBreach 1.1
CoreBreach 1.0.2
CoreBreach 1.0.1
CoreBreach 1.0

Mojo understands much more complex queries than these tho. Pretty much anything you can use in jQuery or CSS 3 works.
A good way to find these defintions is to open up your web page with the Chrome debugger:

Just right-click the element you want to know more about and choose ‘Inspect Element’ from the menu. This gives you a really simple view of the DOM, which shows you which element on the web page you are highlighting, as well as the parent nodes to your element for use in selectors.

As for the last argument, you can call anything that you can call on a Mojo DOM Node.
If you exclude it you just get the markup as you saw above.

Because the client uses the Mojo UserAgent, it already supports features like HTTPS and basic authentication credentials in the URL.
mojo getalso supports features like doing posts and setting headers, as well as setting the method and the request body, but I think it’s this DOM queries that puts it apart from the other HTTP clients like curl, wget or lwp-download. Here’s a final trick:

Set MOJO_USERAGENT_DEBUG=1 in your environment to get full traces of your HTTP requests.

Like it? Installation is really simple and fast. Go get it!

Per my experiments with parallelism in Perl test suites, I've adopted several patterns. One such pattern allows me to manipulate the filesystem in a parallel-friendly way.

A lot of my code handles batch processing: fetch some data, sort it into various logical buckets, manipulate the contents of each bucket, then produce some sort of output for everything that made it that far. While most of these steps only manipulate active data in the queue, some steps require me to read and write files in the filesystem—see Pod::PseudoPod::Book (soon to be on the CPAN) for example.

As with any parallelism, multiple units of execution contending over the same single shared resource is an exercise in conflict, or at least complicated locking.

For simple needs, File::Tempdir is great. You can create a temporary directory with a lifespan tied to the object representing it. When that object gets destroyed, its destructor removes the temporary directory.

I needed something more. I wrote the very silly, very simple Tempdir solely for one project's test suite:

package Tempdir;

use Cwd;
use autodie;
use File::Temp;
use File::Path;

use base 'File::Temp::Dir';

sub new
{
    my ($class, %options)       = @_;
    my $self                    = File::Temp->newdir;
    @{ $self }{ keys %options } = values %options;
    $self->{original_dir}      = cwd();

    chdir $self->dirname;
    File::Path::make_path( @{ $self->{mkdirs} } );

    bless $self, $class;
}

sub write_file
{
    my ($self, $name, $contents) = @_;
    open my $outfh, '>', $name;
    print {$outfh} $contents;
}

sub DESTROY
{
    my $self = shift;
    chdir delete $self->{original_dir};
    $self->SUPER::DESTROY( @_ );
}

1;

Like File::Tempdir, creating a new Tempdir object creates a temporary directory. In addition, it saves the current working directory and chdirs to the new temporary directory. Because I'm careful to use only relative paths within my code (business requirement: prefer running multiple related instances of a project on a single machine to separate virtual machines), as long as the relative necessary files and directories are present, everything continues to work correctly. (Also because this temporary directory manipulation happens at runtime, the test file's connection to the work queue is already in place, so chdir works just fine.)

If you provide the constructor a mkdirs key with an array reference as its values, the object will create, relative to the temporary directory, additional subdirectories of arbitrary depth. I also added a very simple convenience feature to write a file. I haven't needed more than this yet:

# create the storage directories for topics/2
my $tempdir = Tempdir->new(mkdirs => [ 'sites/Bravo', 'sites/Bravo/css' ]);

...

$tempdir->write_file( $css->filepath, $css->contents );

When $tempdir goes out of scope, all of these files and directories go away. Even if I were to run a hundred instances of the same test file simultaneously, they would all run successfully because they do not interfere with each other.

Though I chose an OO interface for this behavior, I prefer a higher-order interface in some ways. I'd like to be able to write:

within_tempdir mkdirs => [qw( some list of directories )]
{
    # do something
    ...
};

... but I haven't convinced myself quite yet that it's an improvement. Certainly it has the potential to be more correct, as nested lexical scoping has a better chance of applying and unapplying chdir calls in the correct order (it behaves more like properly associated pushd/popd calls in bash), but Perl 5's limited abilities for parameterization of these thunks is clunkier than it ought to be. I could experiment with an interface where you specify parameters to import which produces and exports a partially applied function, but the OO version is good enough for me for now and continues to stay out of my way.

Unfortunately O’Reilly’s Josette Garcia couldn’t be at the London Perl Workshop, so she asked if I could write something about it for her blog.

It took me longer than it should have done, but my post has just been published over at Josetteorama.

Hopefully Josette will be back at next year’s event. She was much missed (although, of course, Alice did a fine job of making up for Josette’s absence).

For the full article visit What is the Perl Maven?

 dancerctl 0.04 released. 

https://github.com/knutov/dancerctl

Main changes:

* Command line parameters order changed
  dancerctl action [appname [environment]]

* help updated

* added init - creates config file for app
  dancerctl init appname path [force]

* autoreload simplification: autoreload_inc removed
  use "autoreload: "path1,path2" instead

* change default config filename .dancer -> .dancerctl
  and dancer.conf -> dancerctl.conf

* env_alias feature
  env_alias:
    dev: development
   prod: production
  dancerctl start app dev # will start development
  
* autodetect positions of action and app
  dancerctl start app
  dancerctl app start 

This is the ten most rated questions at Stack Overflow last week.
Beetwen brackets: question score & answers count
Built date: 2011/11/27 13:30:32 GMT

1. What is the '@(' doing in this Perl code? - [9/4]
2. Understanding pushing to a hash and ||=[] construct. (generated in boilerplate.t) - [6/2]
3. Is there such a thing as a list in scalar context? - [6/6]
4. How to untaint system call in CGI.pm - [4/4]
5. Regexp doesn't work for specific special characters in Perl - [4/2]
6. Most efficient way to search very large text file and output lines matching any of a very large number of terms to another file - [3/2]
7. Using both ARGV and CGI in a Perl script - [3/4]
8. Perl, comparing scalar to array (with regex?) - [3/6]
9. Perl CGI gets parameters from a different request to the current URL - [3/1]
10. Capture variable assignments in a Perl eval - [3/3]

At the end of January I decided that Perl 6 is currently impractical for my purposes, but other people have different ideas. In any serious discussion of Perl 6, you'll find people claiming that Perl 6 is already usable.

Discovering the minimum viable utility of a programming language is a difficult exercise. Certainly in 1994 few people would have expected that the obvious successor to Perl 4 would eventually need multi-megabytes of extensions to add a full object system rivaling CLOS plus Smalltalk, an asynchronous event system, a high-powered object-relational mapper with pervasive laziness, and an abstraction mechanism for HTTP? Yet that's the state of the art in Perl in 2011, and we're all the better for it.

If you read carefully between the lines of my criticism of the current state of every Perl 6 implementation I've used as well as what the people who claim that "Perl 6 is usable right now" write, you may find that we approach the problems we're trying to solve from two different angles. We even agree on one of those vectors.

When someone like Moritz Lenz writes "You can use Rakudo or Niecza for real programs now!", I think it's fair to rephrase that as "Any of the leading Perl 6 implementations implements enough structure of a complete and useful programming language that you can implement your own code."

I can agree with this sentiment. I even go a step further; I don't mean to water down his assertion or to put words in his mouth. If you analyze Perl 6 right now in terms of the Perl 6 RFCs, it's clear that Perl 6 even as currently implemented has advantages over Perl 5 when you compare features on a matrix.

(I've since come to believe that you're dooming yourself to potential technical excellence and popular obscurity by focusing on features instead of Getting Stuff Done.)

To rephrase, Perl 6 may certainly be a better programming language than Perl 5 in that Perl 6 includes features Perl 5 should have had years ago. If design patterns are signposts of missing language features, Perl 6 is the Christopher Alexander of programming.

If that's sufficient for your purposes, great. (You still have to deal with performance issues and regressions, and licensing concerns with the Mono dependency of Niecza (Niecza fans, be honest. How is my business supposed to pay Novell anymore for indemnification per their patent licensing arrangement with Microsoft—you remember, the Microsoft that sued TomTom for patent infringement?), and the schism between Parrot and Rakudo on the Rakudo side, as well as the history of Rakudo undergoing lengthy and repeated and compatibility-busting rewrites of core components, but after eleven and a half years, you ought to know you're an early adopter by now. (Rakudo fans, be honest. How long after nom became the main development branch did it take to get all of the Panda modules passing tests again? Oh, they're still not? QED.)

The other vector from which to approach problem solving in Perl says something like "You know, if I'm already leaning heavily on the CPAN to download Plack and an IMAP server and an OAuth implementation and AnyEvent and several testing distributions, Moose really isn't that big a deal anymore." In other words, maybe it is rather silly that in 2011 Perl 5 still doesn't have much of an object system in the core and that it could really use real function signatures, but the Perl community is awfully good at making the important stuff on the CPAN just work, and Perl 5 without the CPAN is pretty anemic for solving big important problems anyway, so grab perlbrew and fire up cpanm and after you've finished your cup of tea, you're all set anyway.

The Python community has dealt with the same schism between Python 2.x and Python 3.x, where Python 3 is arguably a better language, but only superficial polyglot magpies and Usenet personas care solely and forever about language qua language,

Suppose the only thing Perl 5 ever really needed were a method keyword. If it were implemented correctly—with the associated tests and documentation and a discussion period to see how it fits in with existing code—the cost of adding this feature would be relatively small: it's not much code, it's a simple feature, and it has very few possibilities to interfere with existing code.

That's not the only thing Perl 5 needs, for some definition of need.

(I believe Perl 5 lacks a unified and coherent development vision, which makes this entire discussion both more interesting and less useful. Yet even an idealist has to deal with the world as it is sometimes.)

If you accept that Perl 5 needs (or "could use" or "would benefit from" or "really ought to have") a few more features, you have to answer at least two procedural questions in addition to the technical questions of "What is it?", "How should it work?", "How do we build it?", and "How do we know it works correctly?":

• Who will build it?
• Who will maintain it?

By way of analogy, consider the case of an appealing albatross in Parrot, specifically the compiler named IMCC. In Parrot's earliest days, the VM only ran bytecode. An assembler written in Perl 5 turned Parrot's assembly source code PASM into bytecode for Parrot to run.

Melvin Smith wrote an experimental intermediate compiler which added some syntactic sugar to PASM and produced Parrot bytecode either as output or directly in memory for Parrot to run. Parrot needed something like this.

It wasn't long before Melvin's IMCC found itself grafted onto the side of Parrot and used as the primary invocation and compilation mechanism. I hope I'm not mischaracterizing Melvin's opinion as including dismay that his code was a proof of concept which made assumptions and took shortcuts and wasn't exactly what he would have submitted for a shipping product. (I don't blame him one bit for writing prototype code—that's exactly what I would have done.)

Melvin stopped contributing to Parrot shortly after that point, but IMCC lives on to this day. (Again, this was prototype code which needed at least another round or two of severe refactoring before it was suitable for the sort of duties Parrot expected. As an example, I found a register use analysis algorithm which looked to have O(n12) complexity. That exponent of 12 is not a typo. Let me type it again: twelve. One dozen. Ten plus two. I managed to get it down to four in most cases.)

The problem is simple: someone (not Melvin!) dumped a big wad of code into the wrong directory in version control and now everyone has to maintain that code.

Perhaps marking this code as experimental would have helped, but other people (including me) added features and built onto it. (Dan's apologia for leaving Parrot mentions that his strategy of checking in messy code and hoping that would lure new people to help clean it up didn't work out as well in practice as he had hoped.)

In terms of Perl 5, it's important (as many people point out) to consider the question "Who will maintain this code?" That task shouldn't have to fall to the pumpking or a Nick Clark or Dave Mitchell by default, but it does.

The deeper question that Perl 5 needs to answer is "How can p5p make the whole of Perl 5 easier to maintain?"—not just to make it easier to add and support new features but to reduce the maintenance burden of existing features and to attract new contributors to help maintain code.

You can see this for yourself; open up almost any core module and see if you want to do the work to find and fix a bug. If that's not technical friction, I don't know what is.

For the full article visit The for loop in Perl

For the full article visit Perl Editor

$ ack -l 'IO::Socket::INET' | xargs perl -E 's/IO::Socket::INET/IO::Socket::IP/g'

• Cosmic Antimatter Excess Confirmed - ScienceNOW
  Terran tech up. / “Cosmic Antimatter Excess Confirmed - ScienceNOW” http://t.co/rOZWoypY
• IMG_3115 | Flickr - Photo Sharing!
  IMG_3115 [Flickr] http://t.co/U1n4cGRT
• Delicious.com - Discover Yourself!
  Links for 2011-11-22 [http://t.co/fE5pyUGx] http://t.co/JiAtsNQB
• 為了「...」而設計的的 Perl
  為了「...」而設計的的 Perl http://t.co/AUzdxfdQ
• Official Google Blog: More spring cleaning out of season
  Google 發表新產品：Google Shutdown / “Official Google Blog: More spring cleaning out of season” http://t.co/tmZ0uPC9
• Embedded Times: TP-LINK TL-WR703N - a tiny Linux-capable device for under $23 delivered
  世間巧合多。周二才在碼農聚會上聊 MIPS，就又看到用 MIPS 架構的玩具 / “Embedded Times: TP-LINK TL-WR703N - a tiny Linux-capable device for under $…” http://t.co/Ii2VPUF6
• Unihan data for U+3402
  在unihan 中找到「㐂」字，音義同「喜」，不知與打帕青哥中大獎有無關聯 http://t.co/7s6fAt1j

cat ~/.dancer/bill
===================
path: bill
autostart: 1
socket_path: ~/
env: production
production:
  method: socket
  workers: 4

Part of `dancerctl status`
===========================
Application  : bill
Running since: Thu Nov 24 08:18:47 2011
Environment  : production
Listen at    : /home/kola//bill.sock

`dancerctl restartall`
=======================
Stopped bill
Application 'bill' started in 'production' mode.
Listen on /home/kola//bill.sock 

 * 6af93ca 2011-11-24 09:54:56 | small fix (HEAD, 0.02, origin/master, origin/dev, master, dev) 
* f752321 2011-11-24 09:49:57 | added installation instructions 
* cd48d93 2011-11-24 09:37:13 | fixes in readme 
* 68c47e0 2011-11-24 09:30:02 | added plackup -r -R support 
* c483545 2011-11-24 09:29:10 | Revert "added plackup -r -R support" 
* 81d3b91 2011-11-24 09:22:44 | added plackup -r -R support 
* c0e3e62 2011-11-24 08:02:35 | fix in chmod 
* 9e38d28 2011-11-24 08:01:49 | added chmod for app.sock 
* 99d5340 2011-11-24 07:49:20 | some cleanups 
* 931884d 2011-11-24 07:47:26 | fix in help and config parser 
* 32c95d7 2011-11-24 07:38:58 | fixes in status 
* adba838 2011-11-24 07:08:27 | fixes in readme 
* 1771d59 2011-11-24 07:05:25 | fixes in readme 
* c1d4062 2011-11-24 07:02:40 | fixes in status and small fixes 
* 92d3f85 2011-11-24 06:32:53 | small fixes 
* c216a6b 2011-11-24 06:29:08 | small fixes 
* cb128f6 2011-11-24 06:21:47 | added readme in pod format 
* 6950824 2011-11-24 06:17:31 | added restartall, statusall and config sample in app.yml 
* e3b97bb 2011-11-24 04:54:27 | added start_all, stop_all, autostart param and minor fixes 
* cf327e9 2011-11-24 02:42:23 | added user-side default environment 

I've uploaded the 4th development release of DBD::ODBC 1.32. This version improves Unicode support in metadata functions even further. My intention is to release this as a full version in the next week unless I get a load of bug reports. The changes since 1.31 are:

read more

Designing monitoring systems can be thrillseeking. I have seen all kinds of homegrown monitoring systems built on top of nagios or mrtg. Usually all those solutions lack one thing: decent configuration possibilities.

In the old administration days one sysadmin was responsible for the installation of a server and the configuration system. Right now one sysadmin uses a central management server and fires up several virtual boxes or cloud instances with a few commands. But what about the monitoring? Is the monitoring included with the setup of your cloud instances? Lucky you. Usually those kind of tasks are only for brave sysadmins – and they are getting fewer every day.

Graph drawn by rrd2graph from collectd data

I gave a talk on Tuesday about a more flexible toolchain for monitoring than the common used nagios setup. I had some good experiences with this toolchain at the company I am working for and it seems to scale reasonably well.

collectd is small daemon which collects the data on every instance/server and stores the information in rrd databases using librrd or via rrdcached. The perfomance data is being pushed to central servers, too. I like to call them collectors. They gather all the data from the different servers/instances. The advantage is clear – nagios can get all the values from those databases from every server at one place. This is not only an advantage for the firewall, it also saves the nagios server alot of work. The graphs are being drawn using rrd2graph (see screenshot) which is part of the n2rrd project.

The best thing about using collectd is it’s way of configuration. I suggest to setup three different configuration levels. A base configuration which is exactly the same on every host. It just has the most basic configuration in. A second level called environment configuration deals with networking stuff like who is the collector server in a certain zone. The third layer is called service configuration. This layer deals with services running on the server such as mysql, httpd, nginx, etc.

If you are using a configuration management tool such as puppet or RedHat Satellite it’s an easy task to build classes or system service groups which care for the configuration layers. Use RPMs during the kickstart process to install the collectd binaries and provide them with a sane first /etc/collectd.conf.

collectd provides a nice plugin structure with about 90 pre-made plugins and it’s easy to extend by using it’s Perl, Java, Python apis.

Oh, did I mention? This works for Windows based servers, too! Please check out the slides for more information. The first link is a mindmap in xmind format with a small management summary. If you are interested in the slides of the presentation, please use the second link which is in PDF format.

Monitoring 2.0 – Presentation Summary (xmind – Mindmap)

Presentation about monitoring Unix environments with modern tools (pdf)

For the full article visit Call for translators, transcriptors and mentors for them!

• Delicious.com - Discover Yourself!
  Links for 2011-11-21 [http://t.co/fE5pyUGx] http://t.co/pIAvKHbv
• 用 AnyEvent 控制輸出量
  用 AnyEvent 控制輸出量 http://t.co/cwB7CUiS
• Untitled
  我還以為一般會說 NMP 可以預防癌症 / “深度烘焙咖啡 較不傷胃｜生活萬象｜中時樂活｜中時電子報” http://t.co/tE7q0rJc
• FrePAN - CPAN recent changes
  @tokuhirom Report. http://t.co/fwasNz4L rss feed should simply use http://t.co/5Ub3Fvnw instead of http://t.co/vPQnGcDz :)

This article was published yesterday. It shows a way to extract data about a film from IMDB and put it into a local database. Actually, it doesn’t even do that. It produces SQL that you can then run to insert the data.

It’s all rather nasty stuff and indicative of the fact that most people using Perl are still using idioms that would have made us shudder ten years ago.

There were a few things that I didn’t like about the code. The use of curl to grab data from the web site, the indirect object syntax when creating the XML::Simple object and, in particular, the huge amount of repetitive code used to create the SQL statements.

So I did what anyone would do in my situation. I rewrote it. Here’s my version.

#!/usr/bin/perl

use strict;
use warnings;

use XML::Simple;
use LWP::Simple;

@ARGV or die "Please provide movie title in quotes\n";

my $movie = shift;
$movie =~ s/\s/+/g;

my $movieData = get "http://www.imdbapi.com/?r=XML&t=$movie";
my $data = XMLin( $movieData );

my @fields = qw[released rating director genre writer runtime plot id
                title votes poster year rated actors];

my %film = %{$data->{movie}};

foreach (@fields) {
  $film{$_} =~ s/'/\\'/g;
}

my $tstamp = time();

my $sql = 'INSERT INTO movie_collection (';
$sql .= join ', ', @fields;
$sql .= ') VALUES (';
$sql .= join ', ', map { qq['$film{$_}'] } @fields;
$sql .= ",'" . time . "');\n";

print $sql;

I haven’t actually changed that much. I’ve tidied up a bit. Switched to using LWP::Simple, removed some unnecessary escaping, things like that. I have made two pretty big changes. I’ve got rid of all of the nasty variables containing data about a film. A film is a single object and therefore should be stored in a single variable. And, happily enough, the $data you get back from XMLin contains a hash that does the trick perfectly.

The second change I made was to rejig the way that the SQL is created. By using an array that contains the names of all of the columns in the table, I can generate the SQL programmatically without all of that repetitive code. I’ve even made the SQL a little safer by explicitly listing the columns that we are inserting data into (this has the side effect of no longer needing to insert a NULL into the id column).

Of course, this would just be a first step. The whole idea of generating SQL to run against the database is ugly. You’d really want to use DBIx::Class (or, at the very least, DBI) to insert the data directly into the database. And why mess around with raw XML when you can us something like IMDB::Film to do it?

At that point in my thought process I had an epiphany. You don’t need the database at all. The IMDB data changes all the time. Why take a local copy? Why not just use the web service directly with IMDB::Film (or perhaps WebService::IMDB – I haven’t used either of them so I have no strong opinions on this).

In general, I think that the original code was too complicated. Which made it hard to maintain. My version is better (but I am, of course, biased) but it can be made even better by using more from CPAN.

CPAN is Perl’s killer app. If you’re not using CPAN then you’re not using half the power of Perl.

What do you think? How would you write this program?

Yesterday, Monday 21st November, saw the official start at 08:00 UTC of the Google Code-in Student participation. What this meant was that students could sign up to the program and start taking tasks from that point. I thought I would take this opportunity to bring you all up to date on how the efforts are going and to supply you with further information.

Firstly due to a problem with Melange which is the software powering the initiative provided by Google we have been given an extra seven days to submit tasks. So if you still have tasks you would like to submit you can get them into the first round of tasks before Monday 28th November. We would particularly like to see tasks that focused on fixing bugs in a module or library, or perhaps writing or adding tests. We also seem to have very few Perl6 focused tasks so would appreciate any that addressed this. If you miss this date don’t worry as the next round is due on the 16th December 2011 where we can add more tasks. We currently have 335 tasks listed for our students to attempt, but the more choice we have the more students we can attract and many of the tasks can be completed in a few hours. We already saw success on the very first day of the event when a student completed a task and it was verified (before 08:00 UTC Tuesday which was within the PST that Google uses), two others were awaiting verification from mentors.

We would like to encourage all those people who have decided to sign up as a mentor, who think that they have the time to try mentoring (we will give help and each mentor has a back-up and support from the other mentors), or who have added their name to the list on the wiki but have not yet filled out the form, to go online at: http://www.google-melange.com/gci/profile/mentor/google/gci2011 as soon as possible and sign up. We have so far had twenty-eight people sign up as mentors and would dearly love to have more so that we can cover all the tasks and students . The role is a valued one and is enormously rewarding when you consider you are involved in perhaps the next generation of programmers who will help shape our world, and who knows we may encourage more people into the glorious world of Perl.

So far we have had twenty-five students sign up for the tasks that are on offer. Students will often complete more than one task and many of them will do a handful of tasks. the most successful applicants are invited to attend a special conference at Google’s headquarters in San Francisco and all students are rewarded for successfully taking part. If you know of anyone who would like to attend, or have the ability to pin up a flyer at a school, social club or other venue attended by student, particularly if you know a teacher or head of IT at a local school or young person’s college than we would be very grateful if you would do so. the flyers are available as pdf files online at:

Full colour – http://www.perlfoundation.org/attachment/press_releases/GCi-2011-basic.pdf

Reduced colour (prints in B&W) – http://www.perlfoundation.org/attachment/press_releases/GCi-2011-basic-home-small-office-printer.pdf

TPF press releases – http://www.perlfoundation.org/press_releases (simply scroll to the bottom of the page)

Students and Mentors can sign up at any point during the initiative but the sooner they do so the better it will be.

Thanks for all your help thus far, Rafl, Paul and I will be sure to keep you all updated as we progress through this year’s initiative.

- mdk

ORDB::CPANTS version 0.05 required--this is only version 0.04 at /Users/c9s/perl5/perlbrew/perls/perl-5.14.2/lib/site_perl/5.14.2/CPANTS/Weight.pm line 33.

For the full article visit Variable declaration in Perl

The relentless pursuit of user efficiency exposes drawbacks now and then. I added HARNESS_OPTIONS=j9 to my .bashrc a while ago, and then noticed that my regular CPAN updates (cpan-update -p | cpanm) had a lot more failures than usual.

Test::Harness (and its internals, TAP::Harness) use the environment variable HARNESS_OPTIONS to customize some of its behavior. This is very useful when running Perl tests through make test or ./Build test or any other mechanism where you don't launch the harness directly.

The j flag allows you to request that the harness attempt to run multiple test files in parallel. If you have multiple .t files and multiple cores in your computer, chances are that parallelism will speed up the test run. (I notice that a lot of my tests are IO bound, not CPU bound, so I can run more tests than I have cores.) My use of j9 works well on my four core machine; your numbers will vary based on your workloads and hardware.

Unfortunately, it's easy to write simple tests which just don't work in a parallel world. Consider the TestServer.pm module used to test Test::WWW::Mechanize. (I chose this as an example because Andy's a good sport, and because I've already opened a pull request for it.) This module starts a server for each test to control the responses returned to the Mech object. That's all well and good; it tests network communication in a mostly real way (yes, the loopback interface isn't exactly the same as a remote server, but it's real enough for most testing uses).

The TestServer constructor in 1.38 is:

our $pid;

sub new {
    my $class = shift;

    die 'An instance of TestServer has already been started.' if $pid;

    # XXX This should really be a random port.
    return $class->SUPER::new(13432, @_);
}

You can probably see the problem already from the comment. If multiple .t files use this module (and they do), and if these files each run in separate processes (and they do), then if these files run simultaneously (as they do i a parallel testing environment), only one file will be able to bind to this port and the others will all abort and cause test failures.

In fact, this is what happens.

I submitted a silly little patch which changes the port to:

    return $class->SUPER::new(13432 + $$, @_);

... which should reduce the likelihood of collisions. (For more safety, the code should check that the given port number is available, but then you have to deal with race conditions and so forth, and there's a point at which adding more complexity to your test just isn't worth it.)

The principle is this:

Manipulating external state in a test file reduces the possible parallelism of your test suite.

You can see the same thing when you write to hard-coded directories in certain tests. (Use File::Temp to create temporary directories—which can clean themselves up!). You can also see the problem when you use a single database for testing (use something like DBICx::TestDatabase to create and populate a database in memory).

Anti-parallelism bugs in test suites are unnecessary and in most cases are easy to fix, once you know what to look for. As the CPAN continues to grow and as our applications rely on more and more great dependencies, the mechanisms we use to manage our code become ever more important. It's easy to avoid these problems—and it's even easier to understand why parallel testing is valuable when you can cut your test run wallclock time in half.

Does Saint Pierre and Miquelon mean anything to you? It’s a small French-owned territory just off the coast of Newfoundland.

Why would this be of any interest on a Perl blog? Well, it’s a French territory with it’s own ccTLD. And that ccTLD is .pm.

Ever since Perl Mongers started we’ve looked longingly at that TLD, thinking how cool it would be to own a .pm domain. But domain registration in .pm is run by the French registry, AFNIC and for at least the last thirteen years they have refused all registrations under that domain. This made many Perl Mongers very sad.

But that is about to change. It appears that from 6th December, AFNIC are going to open registrations under a number of their previously suspended domains – including .pm. I think you’ll need to be in the EU in order to register a .pm domain, but I don’t think that will be a huge problem.

And it’s not just for Perl Monger groups. You’ll also be able to have domains for your favourite Perl modules too (or, at least, the ones without ‘::’ in their names).

Which .pm domains do you have your eye on? And what are you going to do with it.

Maybe one year we should have YAPC::NA in Saint Pierre and Miquelon and YAPC::EU in Poland.

WWW::Google::PageRank von Yuri Karaban ermittelt den Google Pagerank für Websites.

Pagerank ermitteln: get()

Die Methode get() liefert im skalaren Kontext den Pagerank, im Listenkontext den Pagerank und zusätzlich ein HTTP::Response Objekt zum Debuggen zurück.

Beispiel: get() im skalaren Kontext

#!/usr/bin/perl
use strict;
use warnings;

use WWW::Google::PageRank;

my $pr = WWW::Google::PageRank->new( host => 'toolbarqueries.google.de' );

my @urls = qw!
  http://perl-howto.de
  http://yahoo.de
  http://web.de
  !;

foreach my $url (@urls) {
    print "URL: $url ", scalar( $pr->get($url) ), "\n";
}

Das Programm liefert folgende Ausgabe:

URL: http://perl-howto.de 4
URL: http://yahoo.de 8
URL: http://web.de 7

Beispiel: get() im Listenkontext

#!/usr/bin/perl
use strict;
use warnings;

use WWW::Google::PageRank;

my $pr = WWW::Google::PageRank->new( host => 'toolbarqueries.google.de' )
  or die $!;

my @urls = qw!
  

This is the ten most rated questions at Stack Overflow last week.
Beetwen brackets: question score & answers count
Built date: 2011/11/20 15:05:08 GMT

1. In Perl, how can I test whether a sequence is of the form n, n + 1, n + 2, ..., n + k? - [14/7]
2. What is @$ref->{@keys} trying to access? - [9/2]
3. align string to a pattern in perl? - [7/3]
4. How to share an object which contains a filehandle? - [7/3]
5. Why does Perl allow invoking coderefs on unblessed data structures? - [6/3]
6. How to pass optional parameters to a Perl function? - [5/3]
7. Perl: Hash ref accessing array of keys - [5/1]
8. Is there a way to check for admin rights in Windows with Perl? - [5/1]
9. How do I convert a full path to a relative path using perl? - [5/2]
10. creating csv file with perl - [5/1]