Programming

ProggIt had this amusing gem today, but it’s pretty obscure, so I put it here instead of the Homestarmy

So where can this argument possibly go?
I don’t have to tell you; I’m sure you must know.
By reductio, there cannot possibly be
a procedure that acts like the mythical P.

Great stuff.

Something came up in IRC today, I thought I found a pretty nice solution to it, thought I’d share it here. A guy has a table in his database that contains ‘prefixes’. It looks like this:

Given the phone number “18005551234”, he’d like to find the longest valid prefix. He had suggested to me that he was considering a stored procedure which loops, character-by-character, through the input looking for the longest match. Instead, I recommended he use ‘replace’ to trim the ‘prefixes’ off of the input, and let an ‘order by’ tell him which of those actually resulted in the shortest remaining string (and thus, was the longest match).

The query I ended up with goes something like this:
select id from prefixes order by length(replace('18005558355', number, '')) asc limit 1;
Given a better understanding of mysql’s ‘replace()’, we ended up using this instead – note the identities are renamed to reflect his actual sytem:
SELECT id, prefix from routes where locate(prefix, '18005558355') = 1 order by length(prefix) desc limit 1;

Is there a better way? Below the fold is a quick (postgresql) “dump” of the database in question so you can come up with your own proposals.

The always brilliant author of the `from future import *` blog brings us an article discussing a potential JavaScript Hijacking [pdf] scenario – the simple answer is “Don’t send JSON data as a list, send it as an object instead”. That is, if you wanted to send:

["foo", "bar"]

you can avoid the security hole by sending something like this instead:

{"data": ["foo", "bar"]}

and then stripping the object on the other end.

Drag this Make Page Editable link to your “Bookmarks Toolbar” in recent versions of Firefox, and click on it to enable the built-in “Page Composer”, allowing you to edit any page you want!

As a Code Kata, I’ve been working my way through the Project Euler stuff.

Here’s the problem.

Here’s my solution:

	
# A generator to derive the Fibonacci sequence

def fib():
    x, y = 0, 1
    yield x
    yield y
    while 1:
        x, y = y, x+y
        yield y

	
if name == ‘main‘:
    idx = 0
    sum = 0
    gen = fib()

    while idx <= 4000000:
        idx = gen.next()
        if (g % 2 == 0): h = h+g
    print h

Note: You want to be careful how you use infinite generators like the ‘fib()’ I created above.

One guy on the forums noted:

Now, replacing an odd number with O and an even with E, we get: O, O, E, O, O, E, O, O, E, O, O, E, O, O, E…And so each third number is even. We don’t need to calculate the odd numbers. Starting from an two odd terms x, y, the series is: x, y, x + y, x + 2y, 2x + 3y, 3x + 5y

Continuing on with yesterday’s theme of revision control, I decided to post a little bit about how I’m using VCS these days. I’ve moved to using git as a front-end for SVN, using a workflow that follows along the paths laid out here and here. Eventually, I’ll probably look into using partial clones as outlined here.

Since I recently moved to using “svn:externals”: in my main repo, I spent some time this morning looking for information on how to get git-svn to pull these external trees, and here are a few resources I found:

Life and Hacking: svn:externals for noobs!
SVN’s svn:externals to GIT’s Submodule for Rails Plugins
Phly, boy, phly: svn:externals

None of these have actually solved the problem for me yet (I think I may be fighting against the fact that I’m using debian’s packaged git instead of the newest upstream), so I have begun looking into alternatives. This article suggested I look into braid and/or piston, as well as pointing to svnmerge.py, which can assist with automatic branch management. I’ll be trying a few of these out over the coming weeks and will post my findings here.

The following is an excerpt from the 3rd Edition of the O’Reilly and Associates classic Learning GNU Emacs, regarding Emacs’ “vc-mode”, which I’ve been using more often of late:

12.5. VC Command Summary

To give you the flavor of the other things VC can do for you, Table 12-1 provides a summary of VC commands. Each one will be explained in detail, but you can probably guess some of their actions from the command names.

Table 12-1. VC commands

Keystrokes	Command name	Action
C-x v v	vc-next-action	Go to the next logical version control state.
C-x v =	vc-diff	Generate a version difference report.
C-x v u	vc-revert-buffer	Throw away changes since the last checked-in revision.
C-x v ~	vc-version-other-window	Retrieve a given revision in another window.
C-x v l	vc-print-log	Display a file’s change comments and history.
C-x v i	vc-register	Register a file for version control.
C-x v h	vc-insert-headers	Insert version control headers in a file.
C-x v r	vc-retrieve-snapshot	Check out a named project snapshot.
C-x v s	vc-create-snapshot	Create a named project snapshot.
C-x v c	vc-cancel-version	Throw away a saved revision.
C-x v a	vc-update-change-log	Update a GNU-style ChangeLog file.

From this article regarding the Mono 2.0 release, I bring you the following quotes from Miguel de Icaza:

“Thanks to statistical data from MoMA, we’re able to prioritize what people are actually using,” De Icaza said. “What is important about this is there are some API’s that we’ve definitely postponed. For example, [Windows Presentation Foundation] is not something that we’re working on right now.”

WPF is part of Microsoft .NET 3.0 specifications. At one time, it was best known by its code name, “Avalon.” WPF is included in Windows Vista and Windows Server 2008 and provides a graphical subsystem for developers. De Icaza claimed that WPF is not yet widely used by developers at this point.

“The IT developer cycles are not in lockstep with vendor release dates,” he added. “There are definitely the early adopters and they’ll be using the latest APIs and those users won’t be served by Mono. But anyone else on the tech adoption curve should be fine with Mono.”

Am I the only one that thinks this guy can’t open his mouth without sticking his foot in it? I doubt the bulk of the Mono developers really want to broadcast the message “We don’t care about early adopters” as the “party line”.

The always entertaining Paper Trail recently did a write up deciphering the Google paper describing the Google File System – a great read if you’re at all interested in file systems and “How Google Works”

proggit led me to this post by what appears to be a member of Google’s Chrome team, regarding the Squirrelfish Extreme engine’s performance on regex. There’s some fascinating stuff here, including the following gem:

WREC doesn’t save the intermediate matches, it just keeps track of the length of the text matched against /a*/ (repeatCount in my code) and decrements the position and count in order to backtrack. This is great, because it uses only two words of memory for state (instead of N million) but can find the next longest match very quickly (just 4 instructions).

If you’ve never read Mastering Regular Expressions, by Jeff Friedl, you might have trouble following some of the underlying concepts, but you should go out and pick up a copy today

[Note: This is an “Executive Summary” I wrote for some co-workers regarding Steve Yegge’s Business Requirements are Bullshit – thanks to Johnny Elbows for reviewing the summary and giving great feedback.]

Big Design Up Front (BDUF) is the methodology in software development that a program’s design should be completed and perfected before that program’s implementation is started. It is often associated with the waterfall model of software development. The argument between the proponents and critics of BDUF has somewhat degenerated into a “holy war”, with most people believing that a compromise between BDUF and the more extreme variants of agile software development is the best solution to most software development problems.

The most important phase of any BDUF project is “requirements gathering” – a project that does not gather the correct requirements will fail, although it will often take an extended amount of time for the effects of failed requirements gathering to actually impact the project. This creates an environment where either a lot of money is spent chasing poor requirements, or projects fail to start because they cannot sufficiently capture the business requirements.

Steve Yegge, a Google Engineer, has written the following article that suggests that this failure is endemic to projects that even need to consider “requirements gathering”. Taking a page from Peter Lynch and Warren Buffet, he suggests that any development project be approached as an investment, and that we should only “invest” in products we would be interested in owning ourselves.

“You can look at any phenomenally successful company,” he suggests, “and it’s pretty obvious that their success was founded on building on something they personally wanted. The extent that any company begins to deviate from this course is the extent to which their ship starts taking on water.” This means that projects which need to go through a “requirements gathering” phase are inherently doomed because “if it’s something you want, then you already know what the requirements are. You don’t need to “gather” them. You think about it all the time. You can list the requirements from memory.”

As we consider taking on projects, I think there is some sound advice contained here about what can make a project fail or succeed, as well as some tips about how simplicity (good) and imagination (bad) can affect the product itself.

Release Notes in the usual spot, but I wanted to hit a few highlights:

1. ADO.NET 2.0 API for accessing databases
2. ASP.NET 2.0 API for developing Web-based applications
3. Windows.Forms 2.0
4. System.Core – C# 3.0, including full support for LINQ
5. Runtime monkeypatching of compiled assemblies via Mono.Cecil
6. Mono.Cairo – cross-platform rendering: “Currently supported output targets include the X Window System, Quartz, Win32, image buffers, PostScript, PDF, and SVG file output. Experimental backends include OpenGL (through glitz), XCB, BeOS, OS/2, and DirectFB.”
7. Includes client libraries out of the box for: SQLite (presumably version 3), PostgresSQL, DB2, Oracle, Sybase, SQL server, and Firebird

I know of at least one project that has been put a bit on hold because of doubts about developing in C# for platforms other than Win32 – perhaps this will allow us to break the stalemate a bit?

Spent the day poking at Pylons 0.9.7rc2 as well as the new documentation site that came out with the release of Python 2.6 – some sweet stuff in there.

WebOb: The New Middleware Layer on 0.9.7, its Reference page, How WebOb is different and Ian Bicking’s Blog Post about it – Ian is one of those guys who is so smart it fills me with a jealous rage. I’ve already found a lot of useful stuff in WebOb that was missing or hard to do with the older Pylons middleware.

Base Pylons Documentation, AuthKit Cookbook, Roadmap to Pylons 1.0 – just some good links to pylonshq

Python modules I need to brush up on, or just links I used in the course of today’s work:
2.6 Documentation Index, shutil, xdrlib, functools – a lot of functional paradigm stuff moved here?, StringIO, tempfile – probably the best tempfile implementation by a programming language, ever, simpleJSON revved to 2.0.1, 2.6 ‘Standard Library’ Index

Mako – I’m not sure why there are so dang many Pythons template layers. In fact, Preppy is another one that I might use more in the future due to its integration with ReportLab’s PDF Toolkit

The Pylons Book – I’ve already pre-purchased, but wish it had more “nuts and bolts” based on what I’ve seen so far.

Sorry for the break – I spent the past week or so frantically preparing for, and then attending, my Director’s annual off-site planning meeting. As always, it was a valuable experience – the opportunity to interact with Managers in a more relaxed setting, and to see how each of us attacks a given problem from so many different angles, is incredibly cool. Plus, I got to play golf again.

Today I’m bringing you a quick example of how to use jQuery to check with a Service to see whether the user should be allowed to toggle the state of a given checkbox. In my example, the “Service” only allows you to check boxes labelled with a prime number. I’m sure there’s a more jQuery-approved method, but this one was simpler to figure out than .val() – I couldn’t figure out how to make that *un*check a box.

I had a friend who couldn’t mysqldump a customer’s database — the best guess freenode’s #mysql and I could come up with was that the username provided didn’t have the ability to LOCK tables — so I came up with an alternate method that I thought I’d abstract it and post it here for others to use. All this assumes you can connect to the database from your host…

1) Get database schema

for i in `mysql —host=$MYSQLDUMP_HOST —user=$MYSQLDUMP_USER —password=$MYSQLDUMP_PWD $MYSQLDUMP_DATABASENAME -e "SHOW TABLES" | tail -10`; do
    mysql —host=$MYSQLDUMP_HOST —user=$MYSQLDUMP_USER —password=$MYSQLDUMP_PWD $MYSQLDUMP_DATABASENAME -e "SHOW CREATE TABLE $i\G";

done > schema.sql

2) Dump the table contents as XML files to the local disk. I went with XML files because I had trouble getting this particular data set to dump and then import using the ‘batch’ format MySQL more natively supports.

for i in `mysql —host=$MYSQLDUMP_HOST —user=$MYSQLDUMP_USER —password=$MYSQLDUMP_PWD $MYSQLDUMP_DATABASENAME -e "SHOW TABLES" | tail -10`; do
    mysql —host=$MYSQLDUMP_HOST —user=$MYSQLDUMP_USER —password=$MYSQLDUMP_PWD $MYSQLDUMP_DATABASENAME —xml -e "SELECT * from $i;"  | tee $i.xml;

done

3) Turn the XML files into INSERT statements. I did this with Python and lxml – you could use just about anything that has a decent XML API.

import.py
python import.py > inserts.sql

4) Create the database, run the INSERTs.

mysql -e "CREATE DATABASE $MYSQLDUMP_DATABASENAME;"

mysql $MYSQLDUMP_DATABASENAME < schema.sql

mysql $MYSQLDUMP_DATABASENAME < inserts.sql

Hope that can help someone else.