/home/bricas/blog

Finding missing tracks in your Banshee library

Mon, 26 Mar 2012 12:17:00 -0430

I really love the MusicBrainz project. I use their Picard tagger all the time.

I also have an unfortunately complex setup due to the fact that I want to sync all of my music to my iPhone. The current incarnation of this is a Windows 7 VM and a shared folder so I can use Banshee in Ubuntu, and iTunes on Windows with the same folder as the source of my files.

In order for this to work properly, all of my music, for one, needs to be something other than ogg (what a PITA, seriously, Apple), and all of the filenames need to be Windows-safe. Picard makes the latter very easy with its renaming features.

One of the unintended side-effects of doing this is that it will rename all of my files to the name of the track in the MusicBrainz database (NB: directories were left unchanged). This means any differences from the original filename (spelling, capitalization, etc) will mess up my Banshee library.

Luckily the Banshee library is just an SQLite database. I wrote a little helper script to help me go through my files and select a new one when a missing one was encountered.

#!/usr/bin/perl

use strict;
use warnings;

use DBI;
use URI;
use URI::file;
use File::Basename;

my $dbh = DBI->connect( 'dbi:SQLite:dbname=' . shift );
my $sth = $dbh->prepare("SELECT trackid, uri from CoreTracks order by uri;");
$sth->execute();

my $sth_upd = $dbh->prepare( "UPDATE CoreTracks set uri = ? WHERE trackid = ?" );

while ( my $row = $sth->fetchrow_hashref ) {
    my $u = $row->{ Uri };
    next if $u =~ m{\.gvfs};
    next if $u =~ m{/Videos/};
    next unless $u =~ m{file://};

    my $url = URI->new( $u );
    my $file = $url->file;
    
    next if -e $file;

    print "OLD: $file\n";

    my $basename = esc_chars( basename( $file ) );
    my $dirname  = dirname( $file );

    my $new = `zenity --file-selection --filename="$dirname/" --title="$basename"`;

    unless( $new ) {
        print "CANCELLED\n";
        exit;
    }

    print "NEW: $new";

    my $newuri = URI::file->new( $new );

    $sth_upd->execute( $newuri, $row->{ TrackID } );
}

sub esc_chars {
    my $t = shift;
    $t =~ s/([;<>\*\|`&\$!#\(\)\[\]\{\}:'"])/\\$1/g;
    return $t;
}

The process is still a bit mind-numbing, but it’s slightly more palatable than any other idea I had.

Fixing Trac 0.11 for Subversion 1.7

Fri, 16 Dec 2011 14:38:00 -0430

This is a short note, which will serve as a reminder, and hopefully help other people who arrive here via the oracle.

If you happen to be running Ubuntu 10.04 LTS, with Trac 0.11 (which can be installed from the canonical repositories), and you decide you want to upgrade Subversion to 1.7 — you will encounter a problem.

One way to fix it, besides not upgrading things that weren’t broken in the first place, is to patch Trac.

I found this changeset, 10833, which is a commit against Trac 0.12, and adapted it for use with 0.11.

--- svn_fs.orig.py        2009-11-23 18:29:48.000000000 -0400
+++ svn_fs.py        2011-12-16 11:30:20.000000000 -0400
@@ -894,21 +894,31 @@
             path = _from_svn(path_utf8)
             base_path = _from_svn(base_path_utf8)
             base_rev = change.base_rev
-
+# SVN 1.7 FIX
+            action = getattr(change, 'action', None)
+# EOFIX
             # Ensure `base_path` is within the scope
             if not (_is_path_within_scope(self.scope, base_path) and
                     self.authz.has_permission(base_path)):
                 base_path, base_rev = None, -1
 
             # Determine the action
-            if not path:                # deletion
-                if base_path:
+# SVN 1.7 FIX
+#            if not path:                # deletion
+#                if base_path:
+            if not path and self.scope == '/': 
+                action = Changeset.EDIT # root property change 
+            elif not path or action == repos.CHANGE_ACTION_DELETE: 
+                if path:            # deletion 
+# EOFIX
                     if base_path in deletions:
                         continue # duplicates on base_path are possible (#3778)
                     action = Changeset.DELETE
                     deletions[base_path] = idx
-                elif self.scope == '/': # root property change
-                    action = Changeset.EDIT
+# SVN 1.7 FIX
+#                elif self.scope == '/': # root property change
+#                    action = Changeset.EDIT
+# EOFIX
                 else:                   # deletion outside of scope, ignore
                     continue
             elif change.added or not base_path: # add or copy

NB: The file on my system was located at /usr/share/pyshared/trac/versioncontrol/svn_fs.py

I’m not entirely unsure if this is 100% correct, but it works for me. I hope it helps you too.

Kwalitee Control and Relaxing the Rules

Wed, 30 Mar 2011 13:16:00 -0430

It’s been online for a couple of weeks now, but if you haven’t already seen it — check out the CPAN::Changes Kwalitee Service. Special thanks to Franck Cuny for hosting and feeding this beast!

The site tracks the overall CPAN::Changes spec conformance as well as a break down by author and distribution. For ease of tracking, we offer feeds for both of these facets as well as a global “recent releases” feed.

If you take a closer look at that feed, any non-conforming releases will indicate that fact with an error message. All of the releases with properly formatted changelogs will have their matching release information extracted and used in the body of the feed entry.

Any suggestions for the service are welcome. The source for the project can, of course, be found on github.

If you visited the site early on, you would have seen that the CPAN had about a 10% conformance rate. These days we’re hovering around 30%. Unfortunately, there hasn’t been a mass acceptance of the spec; I’ve just relaxed the date parsing rules.

I had a number of people question the use of W3CDTF dates when a number of existing distributions use “scalar localtime”-style dates. Since I can easily discover those dates, I’ve changed CPAN::Changes (release 0.09) to parse them and convert them to W3CDTF dates. I will continue to promote the use of W3CDTF dates and the CPAN::Changes tools will continue to output dates in that format only.

At the same time I’ve tightened up the validation of those W3CDTF dates (release 0.10) to get rid of any false positives I was seeing.

Happy changelogging!

Re: Changes; filenames

Tue, 08 Feb 2011 08:49:00 -0430

In talking about changelogs, I thought it might be good to take a snapshot of what is out there. I whipped up a quick script to parse through my minicpan and pull out filenames that looked like “change*” (case-insensitive) from the root directory.

21545 distributions were parsed. *

Out of those, 2305 (10.7%) had no files matching that regex. This is an unfortunate stat, but one could posit a couple of reasons: 1) the author simply neglected to include it 2) the author chose to use a different naming convention; both are plausible. For the latter, I’ve seen other files such as NEWS or even a pod file (generally within the lib/ structure) in a few distributions.

The most common filename was, of course, “Changes” with 17138 (79.5%) distributions falling under that naming. Second to that was “CHANGES” — 1016 (4.7%) distributions use that. In third place was “ChangeLog” at 847 (3.9%) distributions.

As you can see, the drop off after the top spot is significant. There were a few other variations worth mentioning: “CHANGELOG” (136), “Changelog” (70), “Changelog.ini” (64), “Changes.ttl” (58), “Changes.xml” (48) and “Changes.txt” (30).

It seems fair to say that to achieve consistency, we should adopt “Changes” as the standard name for CPAN distribution changelogs. This is certainly the path of least resistance as nearly 80% already follow this notion. Please feel free to voice your opinion in the comments below!

In my next post, I will do some further analysis on the actual content of these changelogs.

* Note: I pulled out all files that matched, thus some distributions returned more than one file. The total of the filenames comes to 21888, which gives us a small margin of error.

Revisiting CPAN Changes files

Thu, 03 Feb 2011 13:53:00 -0430

The topic of Changes files has once again come into focus — see Dave Rolsky’s posts here and here.

I posed a question a few years back about machine parsable Changes files. A number of people picked up on this and some further conversations ensued:

So, what happened? Well, not much.

I don’t think there’s a single person that could agree on what should be in a Changes files. Some people wanted to argue about the serialization format (YAML vs Text vs XML vs …). Some people thought that things were “too structured”, some “not structured enough.” This kind of back-and-forth was inevitable — everyone has an opinion.

At my own peril, I’ve decided to re-tread this old ground.

To that end, I’ve uploaded CPAN::Changes to CPAN (source here). What’s inside:

CPAN::Changes::Spec - A specification document. I’ve attempted to codify a number of standard practices seen in changelogs included in CPAN distributions. It’s a light read with only a few rules, but it gives some consistency to the file’s structure.
CPAN::Changes (and ::Release) - An API to read and write Changes files. Once you have a predictable structure, you should be able to play around with it in a programmatic fashion. I can envision integration with distribution building tools, SCM tools, and CPAN clients.
Test::CPAN::Changes - A simple module to help automate the validation of Changes files. This is probably most useful during release testing to make sure you’re sending a spec-conforming changelog to CPAN.

I’ve deliberately kept the spec simple, taking what’s already out there and applying some basic rules. It’s open for comments — let me know what you think!

File::HomeDir 0.91 data directory changes

Thu, 03 Jun 2010 18:02:00 -0430

After performing a routine upgrade of my CPAN modules on my $work machine, I noticed that I was prompted to configure CPAN for the first time after I attempted to load the CPAN shell again.

Obviously I had already done all of that so I had to figure out what had changed. I remembered having installed only a couple new modules, including File::HomeDir.

Browsing its Changes file shows:

Moving the FreeDesktop driver to prod

My operating system is Ubuntu 10.04, which apparently means that File::HomeDir will use these new FreeDesktop rules. The big change is that the default data dir is no longer your home directory, but ~/.local/share/

The simple fix was to move my .cpan dir to ~/.local/share/. From then on, everything worked as per usual. This will also affect any other apps that use File::HomeDir to locate your data dir, including Padre.

Project Stepping Stones

Thu, 04 Feb 2010 00:05:00 -0430

Ever looked back at something you’ve worked on and thought: “Gee, it’s too bad that project didn’t get to it’s ultimate goal, but, I’ve learned a lot from it.” I have one of those projects. Such is the world of technology, your toolset is constantly evolving and shape-shifting; even “The Next Big Thing” can become obsolete. We move on to the next “Next Big Thing.”

One such area in the Web Developer’s toolset is “Search.” I’m sure we can all relate to the experience of our first textbox with some behind-the-scenes code doing SQL “SELECT … LIKE” statements. Perhaps at first it was raw DBI calls; maybe moving on to an abstraction layer (ORM and whatnot) shortly thereafter. Here’s where things get interesting.

What happens when this is no longer Good Enough(tm). Google being essentially ubiquitous, people expect to plunk some words in the box and magically get what they want out the other side. I put in “Cat Hat” — why didn’t it give me “Cat in the Hat”? Okay, no problem. We can do some field and query normalization; removing stopwords, add term parsing … wait, wait wait. There has to be prior art for this.

In 2004, the options are somewhat limited as far as Free/Open Source search software goes. Especially in Perl land. Swish-e looks pretty neat. We actually did some prototyping with it. It was definitely a step ahead of plain old SQL. Plucene came on the scene. Unfortunately, it’s poor performance was a bit of a non-starter for us. The fact that it was modeled after the Lucene Java library, however, caught my eye.

I wanted to harness their project and its community, and bring it into our little Perl world. Luckily for me, someone else had already started down that road. The Lucene Web Service was a project by Robert Kaye, sponsored by CD Baby, which allowed users to talk to Lucene via an XML-based web service. After using it for a while, we developed some patches for bug fixes and enhancements. Because of our momentum with the project, we were eventually given total control over its development.

We attempted to strengthen the project by hooking into some existing standards. We leveraged the Atom Publishing Protocol as an analogy for dealing with indexes and documents. Search results were returned as an OpenSearch document. A document’s field-value pairs were listed in the XOXO microformat. Creating a client for this setup meant a bunch of glue between the existing components (XML::Atom::Client and WWW::OpenSearch).

Almost in parallel, the Solr project emerged. Similar idea, much more support behind it. In the end, our idea never got very far, and Solr has turned out to be a fabulous product — which we now use.

To this end, the Lucene WebService website will (finally) be shutting down in about a week’s time. I’ve moved the pertinent code and wiki data to github in case anyone wants it. I still think it has some niche applications, but without some serious revamping of the java code, it will likely just rot.

At least it’s a project that has led me to bigger and better things.

New Padre-Plugin-PerlTidy release handles project-specific setting

Fri, 15 Jan 2010 17:10:00 -0430

Padre 0.54 introduced a couple of project-specific settings, one for Perl::Critic and another for Perl::Tidy. As the maintainer of the Perl::Tidy plugin, it was only natural that I should implement support for this new feature.

Unfortunately, it wasn’t immediately obvious to me how I might get at this info. With Adam’s guidance, I was able to write the following:

my $tidyrc = $self->current->document->project->config->config_perltidy;

It’s a mess of chained method calls, but, it pretty clear that we’re getting the current document’s project-related config, if it exists.

Here’s a screenshot of the new version of the plugin in action.

Padre w/ Padre-Plugin-PerlTdiy 0.09

Grab version 0.09 from CPAN now.

The last five months

Fri, 18 Dec 2009 22:32:00 -0430

I haven’t bothered to post anything in the last five months. With Christmas just around the corner, I figure this is as good a time as any to play catch up.

Padre-Plugin-PerlTidy

A new release of this plugin — changes made mostly by other people.

Gedcom-FOAF

Rather than using a base url for the data, you can now specify a number of url templates. This makes the module actually useful. Thanks to Chris Prather for working through this with me.

Geo-IPfree

A couple of releases with various refactoring bits and bug fixes. The folks at software77.net now produce a data file specifically for this module. I will ship an updated copy with every release. Refactoring this code has been pretty satisfying, though there are some parts of the module API which I detest but I will be unable to modify them.

Image-Textmode/Image-Textmode-Reader-ANSI-XS

Various bug fixes thanks to some testing with a large dataset from Doug Moore and sixteencolors.net

Config-Any

Released version 0.18, which prefers YAML::XS over any other YAML parser. This created a number of issues with the HTML::FormFu crowd as existing parsers allowed this sort of syntax “auto_id: %n” — whereas YAML::XS complains about an exposed percent sign. The easy fix is to wrap the string in quotes “auto_id: ‘%n’”

GD-Image-Scale2x

Fixed a nasty bug due to a missing my() which randomly broke the module on some platforms.

CGI-Application-PhotoGallery

A tiny patch for max_height included in this release. This still has some pending issues in RT — though I have a hard time justifying spending any time on them as I don’t use this module at all.

Catalyst-Model-WebService-Solr

Apparently, this module was basically broken. Fixed thanks to a supplied patch.

Template-Provider

Another kind user supplied some patches/info to support mod_perl and fully qualified template names.

CQL-Parser/SRU

Removing use of UNIVERSAL->import from these module. Not even sure why it was there to begin with.

WebService-Solr

A couple of release of this module. Includes some bug fixes, feature additions and Solr 1.4 compatibility.

Remove auto_install from my dists

Although, as I understand it, auto_install now works in newer versions of Module::Install, I’ve decided to remove it from my dists to avoid any issues.

See you next year.

Adding a feature to Padre

Tue, 28 Jul 2009 14:53:00 -0430

I’ve been following (and even contributing to) the Padre IDE project from very early on. I’ve watched it grow from very modest beginings into something quite impressive — usable, even.

Its deep integration with Perl is such a killer feature. There are already a good two-dozen plugins, one of which I’ve been shepherding: Padre-Plugin-PerlTidy.

In light of Padre’s first birthday, I decided I wanted to give something back into the Padre core rather than just an ancillary project.

I tend to use gedit on Ubuntu, and I rather liked the “right margin” option. This option puts a gray vertical line on whichever column you specify. It’s an easy visual queue for long lines. It turns out that the Scintilla editor component supports this feature and all I had to do was enable the menus and dialogs to allow users to toggle the method.

Dear Module Author

Wed, 03 Jun 2009 22:11:00 -0430

Dear Module Author,

When preparing to upload a new release of your module to PAUSE could you please review your Changes file?

Did you remember to update it? Does it contain something meaningful? Here are a couple of examples of Changes entries which mean very little to me at a glance:

Bug Fixed
Foo::Bar Fixed
Fixed RT #12345

Also, your SCM revision log does not a good Changes file make.

Yes, this is old news. This is just a reminder.

Elsewhere

Tue, 26 May 2009 20:26:00 -0430

This is just a quick note to let y’all know that I now have a twitter account and an identi.ca account.

You have been warned.

Benchmark

Thu, 21 May 2009 20:00:00 -0430

As noted in my last post, I was able to get a bit of a speed boost based on observations made as a result of code profiling.

In general, if I want to see if one piece of code is faster than another, I use Benchmark. Benchmark is shipped as part of the core set of modules, so there’s no need to load up CPAN to get started. Its simplest usage, and the one i prefer looks something like this:

    use Benchmark ();

    Benchmark::cmpthese( $count, {
        Foo1 => sub {
            # code to do Foo1 here
        },
        Foo2 => sub {
            # code to do Foo1 here
        },
    } );

Of note is that $count can be negative, which will then signify how many seconds to run instead of the number of times. The result looks like this:

             Rate Foo1 Foo2
    Foo1 108665/s   — -38%
    Foo2 175460/s 61%   —

It’s pretty easy to see that Foo2 was faster. Using the above it was easy for me to test the XS-based ANSI parser vs the pure Perl version.

4k worth of ANSI over 10 seconds yields the following:

         Rate    PP    XS
    PP 15.7/s    — -96%
    XS 379/s 2316%    —

For giggles, i tested it against a 33k ANSI, giving:

         Rate    PP    XS
    PP 2.23/s    — -96%
    XS 58.7/s 2528%    —

Looks like a success to me!

Devel::NYTProf

Mon, 11 May 2009 18:07:00 -0430

Another week another QA tool.

This week I’m going to talk about Devel::NYTProf (aka NYTProf).

To start, if you’re interested in profilers, you should check out the brief history section of the pod, then take a glance at its features. Until recently, I hadn’t been very interested in profiling my code. I didn’t really have anything that needed the profiling, and the tools just seemed a bit awkward to me. This changed for me when I saw the output from nytprofhtml (1, 2).

While working on Image-TextMode, I noticed that parsing large (~75k) ANSI files was getting to be pretty slow. I decided to run NYTProf on the parsing code, and here’s what I got:

Profiling - Before

The putpixel(), width() and height() methods are called for every character/attribute combo stored for the image. This turns out to be a really big inefficiency. I’ve had some XS code in my back pocket for ANSI parsing, so I decided to whip up a replacement parser using that code and run the profiler again.

Profiling - After

Huge win! By moving _read() to XS (including putpixel, width, and height) I was able to shave over a second off of the total time (_read inclusive goes from 1.3 seconds to 0.03). Although working with XS was a bit of a pain, it was really great to see such a speed improvement.

I recommend everyone take a look at NYTProf if you’re looking find speed inefficients in your code.

Perl::Critic

Fri, 01 May 2009 22:11:00 -0430

Holy — this weekly thing goes by way too fast!

Anyway, as promised, I’m making my first QA tool post. This week, we’re chatting about Perl::Critic.

Perl::Critic has been around since late 2005. I was able to resist its icy gaze until last fall. So, why wouldn’t I want to jump right in with Perl::Critic early on? Mostly what I imagined was putting a significant amount of time in to bend Perl::Critic policies to my will so I wouldn’t have to change how I code. This is, of course, the wrong way to look at it.

There’s nothing wrong with having a tool that confirms you’re doing the right thing — but what I really wanted was a tool that showed me the bad habits I’ve learned and gave me a slap on the wrist every time I tried to use them. The easiest way to get started was to copy someone else’s polcy file. RJBS was nice enough to comply.

For the Image::TextMode project, after adding my own tweaks to the policies, this is the result. A simple automated test integrates it into my development cycle.

After running it against my code, it found some issues — most of my which were pretty tame: 2-arg open, lack of pod, plus a few regex and character matching niggles.

In my policy file, I have two sections: Things I don’t agree with and things I’ve had to disable temporarily. I hope to eventually go back and clean up my code so I can remove the remainder of the temporarily disabled policies. The policies I don’t agree with may change over time, but this is my current list of preferences.

I have yet to use this setup in any other project, but I think the tool is useful enough that I could put it into place from the very beginning of a project or go back and run it against all of my old projects over time.

Until next time…

Projects Updates and QA Tools

Wed, 22 Apr 2009 20:34:00 -0430

NB: This is my first post in the EPO Iron Man challenge. (Warning: contains some expletives)

First and foremost, the long awaited Catalyst 5.8 is out! My mind-share has primarily been with the 5.7x series so I’ve been pretty much out of the loop on everything that’s going on. Luckily enough, everything is basically backwards compatible (less any necessary module upgrades).

Besides the usual round of bug fixes, this release is built on top of Moose. I’m a big fan of Moose and the ease with which it lets me code, so I’m very excited to see this feature. Be sure to check the Changelog for all of the details.

As far as my personal projects go, I’ve finally been able to deprecate two of my oldest modules (Image::ANSI and Image::XBin) with the latest release of Image::TextMode. This release now mirrors all features provided by the two before it (and then some). It can now write each format (not 100% complete, but release-worthy) — I’ve even included a little bit of naive RLE compression.

Personal projects let me explore some new/different technologies which may not fit in do my daily $work. One of those would be Moose, as mentioned above (which is now part of our standard “toolkit”). Another would be XS. Writing Image::TextMode::Reader::ANSI::XS was very eye opening as far as hooking Perl and C code together and illuminating the Perl internals for me.

Recently, I’ve been in tune with adding new QA tools to my repertoire, such as: Benchmark (high time I learned more about it), Perl::Critic (again, about time) and Devel::NYTProf.

If I’m going to keep up with this “Iron Man Challenge,” then maybe I will save my favorite QA tools for their own post. Stay tuned!

github

Tue, 24 Mar 2009 17:19:00 -0430

Like so many before me, I have now joined the github cabal.

Check out my projects.

One of the more interesting parts of github is their breakdown of projects by language. At the time of this post, Perl has 6% of the projects — tied with C, 1% more than PHP, 3% less than Python and most surprisingly 1% less than “Shell”.

Obviously, given the origin of github, Ruby is way in the lead with 30%. But, I have a feeling that as people get more savvy about SCM tools, especially distributed SCM tools, Perl will make a significant dent in that.

Catalyst 5.71000

Tue, 20 Jan 2009 00:34:00 -0430

Like i said back in July, Catalyst 5.71000 would happen before the moose-ified 5.80 gets shipped. That day is here. Here’s the basics on what’s new since 5.7015:

Relatively chained actions
PathPrefix (I only mentioned that, oh, 2 years ago :)
$c->go and $c->visit (they do a full dispatch to the action; this should kill the SubRequest plugin)
Refactored component resolution (which is something else I worked on)
Documentation improvements
Misc bug fixes

There’s a full changelog available. You might also browse the search.cpan diff from 5.7015 to 5.7100.

Enjoy!

Series of articles featuring WebService::Solr

Fri, 09 Jan 2009 00:24:00 -0430

I’ve had a number of people show interest in WebService::Solr since its release last fall. This is really encouraging.

Especially encouraging is seeing Eric Lease Morgan’s first article in a series of three dealing with indexing and searching using WebService::Solr. The initial post deals with some basic definitions, setting up Solr and writing a basic set of indexing and searching scripts.

It’ll be great to see how the module gets used in a truly practical way.

The hazards of winter driving

Wed, 07 Jan 2009 23:53:00 -0430

I’m not exactly sure how it happened, but these photos were taken from the 3rd floor of our office building about an hour ago:

The Accident

The Aftermath

Note the fire hydrant in the second picture — apparently she was on top of it.