A River of Words - Programming

The Halting Problem in the Style of Ted Geisel

nospam@example.com (TML) — Tue, 03 Mar 2009 04:40:57 -0700

ProggIt had this amusing gem today, but it’s pretty obscure, so I put it here instead of the Homestarmy

So where can this argument possibly go?
I don’t have to tell you; I’m sure you must know.
By reductio, there cannot possibly be
a procedure that acts like the mythical P.

Great stuff.

A "Longest-Prefix" Solution for SQL

nospam@example.com (TML) — Thu, 26 Feb 2009 14:32:00 -0700

Something came up in IRC today, I thought I found a pretty nice solution to it, thought I’d share it here. A guy has a table in his database that contains ‘prefixes’. It looks like this:

id	number
1	1
2	1800
3	18
4	180
5	1801

Given the phone number “18005551234”, he’d like to find the longest valid prefix. He had suggested to me that he was considering a stored procedure which loops, character-by-character, through the input looking for the longest match. Instead, I recommended he use ‘replace’ to trim the ‘prefixes’ off of the input, and let an ‘order by’ tell him which of those actually resulted in the shortest remaining string (and thus, was the longest match).

The query I ended up with goes something like this:
select id from prefixes order by length(replace('18005558355', number, '')) asc limit 1;
Given a better understanding of mysql’s ‘replace()’, we ended up using this instead – note the identities are renamed to reflect his actual sytem:
SELECT id, prefix from routes where locate(prefix, '18005558355') = 1 order by length(prefix) desc limit 1;

Is there a better way? Below the fold is a quick (postgresql) “dump” of the database in question so you can come up with your own proposals.

Continue reading "A "Longest-Prefix" Solution for SQL"

Bob Ippolito on JSON Hijacking

nospam@example.com (TML) — Tue, 04 Nov 2008 13:14:00 -0700

The always brilliant author of the `from future import *` blog brings us an article discussing a potential JavaScript Hijacking [pdf] scenario – the simple answer is “Don’t send JSON data as a list, send it as an object instead”. That is, if you wanted to send:

["foo", "bar"]

you can avoid the security hole by sending something like this instead:

{"data": ["foo", "bar"]}

and then stripping the object on the other end.

Stupid JavaScript Tricks: Make Page Editable

nospam@example.com (TML) — Thu, 16 Oct 2008 15:17:45 -0600

Drag this Make Page Editable link to your “Bookmarks Toolbar” in recent versions of Firefox, and click on it to enable the built-in “Page Composer”, allowing you to edit any page you want!

Project Euler, Problem #2 in Python

nospam@example.com (TML) — Tue, 14 Oct 2008 02:10:00 -0600

As a Code Kata, I’ve been working my way through the Project Euler stuff.

Here’s the problem.

Here’s my solution:



	# A generator to derive the Fibonacci sequence

def fib():
    x, y = 0, 1
    yield x
    yield y
    while 1:
        x, y = y, x+y
        yield y

	if name == ‘main‘:
    idx = 0
    sum = 0
    gen = fib()

    while idx <= 4000000:
        idx = gen.next()
        if (g % 2 == 0): h = h+g
    print h

Note: You want to be careful how you use infinite generators like the ‘fib()’ I created above.

One guy on the forums noted:

Now, replacing an odd number with O and an even with E, we get: O, O, E, O, O, E, O, O, E, O, O, E, O, O, E…And so each third number is even. We don’t need to calculate the odd numbers. Starting from an two odd terms x, y, the series is: x, y, x + y, x + 2y, 2x + 3y, 3x + 5y

git-svn and svn:externals

nospam@example.com (TML) — Fri, 10 Oct 2008 14:16:49 -0600

Continuing on with yesterday’s theme of revision control, I decided to post a little bit about how I’m using VCS these days. I’ve moved to using git as a front-end for SVN, using a workflow that follows along the paths laid out here and here. Eventually, I’ll probably look into using partial clones as outlined here.

Since I recently moved to using “svn:externals”: in my main repo, I spent some time this morning looking for information on how to get git-svn to pull these external trees, and here are a few resources I found:

Life and Hacking: svn:externals for noobs!
SVN’s svn:externals to GIT’s Submodule for Rails Plugins
Phly, boy, phly: svn:externals

None of these have actually solved the problem for me yet (I think I may be fighting against the fact that I’m using debian’s packaged git instead of the newest upstream), so I have begun looking into alternatives. This article suggested I look into braid and/or piston, as well as pointing to svnmerge.py, which can assist with automatic branch management. I’ll be trying a few of these out over the coming weeks and will post my findings here.

Emacs VC mode cheat sheet

nospam@example.com (TML) — Thu, 09 Oct 2008 13:00:37 -0600

The following is an excerpt from the 3rd Edition of the O’Reilly and Associates classic Learning GNU Emacs, regarding Emacs’ “vc-mode”, which I’ve been using more often of late:

12.5. VC Command Summary

To give you the flavor of the other things VC can do for you, Table 12-1 provides a summary of VC commands. Each one will be explained in detail, but you can probably guess some of their actions from the command names.

Table 12-1. VC commands

Keystrokes Command name Action

C-x v v vc-next-action Go to the next logical version control state.

C-x v = vc-diff Generate a version difference report.

C-x v u vc-revert-buffer Throw away changes since the last checked-in revision.

C-x v ~ vc-version-other-window Retrieve a given revision in another window.

C-x v l vc-print-log Display a file’s change comments and history.

C-x v i vc-register Register a file for version control.

C-x v h vc-insert-headers Insert version control headers in a file.

C-x v r vc-retrieve-snapshot Check out a named project snapshot.

C-x v s vc-create-snapshot Create a named project snapshot.

C-x v c vc-cancel-version Throw away a saved revision.

C-x v a vc-update-change-log Update a GNU-style ChangeLog file.

Keystrokes	Command name	Action
C-x v v	vc-next-action	Go to the next logical version control state.
C-x v =	vc-diff	Generate a version difference report.
C-x v u	vc-revert-buffer	Throw away changes since the last checked-in revision.
C-x v ~	vc-version-other-window	Retrieve a given revision in another window.
C-x v l	vc-print-log	Display a file’s change comments and history.
C-x v i	vc-register	Register a file for version control.
C-x v h	vc-insert-headers	Insert version control headers in a file.
C-x v r	vc-retrieve-snapshot	Check out a named project snapshot.
C-x v s	vc-create-snapshot	Create a named project snapshot.
C-x v c	vc-cancel-version	Throw away a saved revision.
C-x v a	vc-update-change-log	Update a GNU-style ChangeLog file.

More on Mono 2.0

nospam@example.com (TML) — Wed, 08 Oct 2008 04:11:00 -0600

From this article regarding the Mono 2.0 release, I bring you the following quotes from Miguel de Icaza:

“Thanks to statistical data from MoMA, we’re able to prioritize what people are actually using,” De Icaza said. “What is important about this is there are some API’s that we’ve definitely postponed. For example, [Windows Presentation Foundation] is not something that we’re working on right now.”

WPF is part of Microsoft .NET 3.0 specifications. At one time, it was best known by its code name, “Avalon.” WPF is included in Windows Vista and Windows Server 2008 and provides a graphical subsystem for developers. De Icaza claimed that WPF is not yet widely used by developers at this point.

“The IT developer cycles are not in lockstep with vendor release dates,” he added. “There are definitely the early adopters and they’ll be using the latest APIs and those users won’t be served by Mono. But anyone else on the tech adoption curve should be fine with Mono.”

Am I the only one that thinks this guy can’t open his mouth without sticking his foot in it? I doubt the bulk of the Mono developers really want to broadcast the message “We don’t care about early adopters” as the “party line”.

Paper Trail on GFS

nospam@example.com (TML) — Tue, 07 Oct 2008 11:07:00 -0600

The always entertaining Paper Trail recently did a write up deciphering the Google paper describing the Google File System – a great read if you’re at all interested in file systems and “How Google Works”

Regex Engines in JavaScript

nospam@example.com (TML) — Tue, 07 Oct 2008 00:19:50 -0600

proggit led me to this post by what appears to be a member of Google’s Chrome team, regarding the Squirrelfish Extreme engine’s performance on regex. There’s some fascinating stuff here, including the following gem:

WREC doesn’t save the intermediate matches, it just keeps track of the length of the text matched against /a*/ (repeatCount in my code) and decrements the position and count in order to backtrack. This is great, because it uses only two words of memory for state (instead of N million) but can find the next longest match very quickly (just 4 instructions).

If you’ve never read Mastering Regular Expressions, by Jeff Friedl, you might have trouble following some of the underlying concepts, but you should go out and pick up a copy today

The Fallacy of Business Requirements

nospam@example.com (TML) — Mon, 06 Oct 2008 15:05:52 -0600

[Note: This is an “Executive Summary” I wrote for some co-workers regarding Steve Yegge’s Business Requirements are Bullshit – thanks to Johnny Elbows for reviewing the summary and giving great feedback.]

Big Design Up Front (BDUF) is the methodology in software development that a program’s design should be completed and perfected before that program’s implementation is started. It is often associated with the waterfall model of software development. The argument between the proponents and critics of BDUF has somewhat degenerated into a “holy war”, with most people believing that a compromise between BDUF and the more extreme variants of agile software development is the best solution to most software development problems.

The most important phase of any BDUF project is “requirements gathering” – a project that does not gather the correct requirements will fail, although it will often take an extended amount of time for the effects of failed requirements gathering to actually impact the project. This creates an environment where either a lot of money is spent chasing poor requirements, or projects fail to start because they cannot sufficiently capture the business requirements.

Steve Yegge, a Google Engineer, has written the following article that suggests that this failure is endemic to projects that even need to consider “requirements gathering”. Taking a page from Peter Lynch and Warren Buffet, he suggests that any development project be approached as an investment, and that we should only “invest” in products we would be interested in owning ourselves.

“You can look at any phenomenally successful company,” he suggests, “and it’s pretty obvious that their success was founded on building on something they personally wanted. The extent that any company begins to deviate from this course is the extent to which their ship starts taking on water.” This means that projects which need to go through a “requirements gathering” phase are inherently doomed because “if it’s something you want, then you already know what the requirements are. You don’t need to “gather” them. You think about it all the time. You can list the requirements from memory.”

As we consider taking on projects, I think there is some sound advice contained here about what can make a project fail or succeed, as well as some tips about how simplicity (good) and imagination (bad) can affect the product itself.

Mono 2.0 Released

nospam@example.com (TML) — Mon, 06 Oct 2008 02:20:48 -0600

Release Notes in the usual spot, but I wanted to hit a few highlights:

ADO.NET 2.0 API for accessing databases
ASP.NET 2.0 API for developing Web-based applications
Windows.Forms 2.0
System.Core – C# 3.0, including full support for LINQ
Runtime monkeypatching of compiled assemblies via Mono.Cecil
Mono.Cairo – cross-platform rendering: “Currently supported output targets include the X Window System, Quartz, Win32, image buffers, PostScript, PDF, and SVG file output. Experimental backends include OpenGL (through glitz), XCB, BeOS, OS/2, and DirectFB.”
Includes client libraries out of the box for: SQLite (presumably version 3), PostgresSQL, DB2, Oracle, Sybase, SQL server, and Firebird

I know of at least one project that has been put a bit on hold because of doubts about developing in C# for platforms other than Win32 – perhaps this will allow us to break the stalemate a bit?

Pylons References

nospam@example.com (TML) — Thu, 02 Oct 2008 15:02:05 -0600

Spent the day poking at Pylons 0.9.7rc2 as well as the new documentation site that came out with the release of Python 2.6 – some sweet stuff in there.

WebOb: The New Middleware Layer on 0.9.7, its Reference page, How WebOb is different and Ian Bicking’s Blog Post about it – Ian is one of those guys who is so smart it fills me with a jealous rage. I’ve already found a lot of useful stuff in WebOb that was missing or hard to do with the older Pylons middleware.

Base Pylons Documentation, AuthKit Cookbook, Roadmap to Pylons 1.0 – just some good links to pylonshq

Python modules I need to brush up on, or just links I used in the course of today’s work:
2.6 Documentation Index, shutil, xdrlib, functools – a lot of functional paradigm stuff moved here?, StringIO, tempfile – probably the best tempfile implementation by a programming language, ever, simpleJSON revved to 2.0.1, 2.6 ‘Standard Library’ Index

Mako – I’m not sure why there are so dang many Pythons template layers. In fact, Preppy is another one that I might use more in the future due to its integration with ReportLab’s PDF Toolkit

The Pylons Book – I’ve already pre-purchased, but wish it had more “nuts and bolts” based on what I’ve seen so far.

Return from hiatus, and a quick jQuery example

nospam@example.com (TML) — Thu, 21 Aug 2008 22:37:10 -0600

Sorry for the break – I spent the past week or so frantically preparing for, and then attending, my Director’s annual off-site planning meeting. As always, it was a valuable experience – the opportunity to interact with Managers in a more relaxed setting, and to see how each of us attacks a given problem from so many different angles, is incredibly cool. Plus, I got to play golf again.

Today I’m bringing you a quick example of how to use jQuery to check with a Service to see whether the user should be allowed to toggle the state of a given checkbox. In my example, the “Service” only allows you to check boxes labelled with a prime number. I’m sure there’s a more jQuery-approved method, but this one was simpler to figure out than .val() – I couldn’t figure out how to make that *un*check a box.

When mysqldump fails...

nospam@example.com (TML) — Wed, 13 Aug 2008 20:17:36 -0600

I had a friend who couldn’t mysqldump a customer’s database — the best guess freenode’s #mysql and I could come up with was that the username provided didn’t have the ability to LOCK tables — so I came up with an alternate method that I thought I’d abstract it and post it here for others to use. All this assumes you can connect to the database from your host…

1) Get database schema



for i in `mysql —host=$MYSQLDUMP_HOST —user=$MYSQLDUMP_USER —password=$MYSQLDUMP_PWD $MYSQLDUMP_DATABASENAME -e "SHOW TABLES" | tail -10`; do
    mysql —host=$MYSQLDUMP_HOST —user=$MYSQLDUMP_USER —password=$MYSQLDUMP_PWD $MYSQLDUMP_DATABASENAME -e "SHOW CREATE TABLE $i\G";

done > schema.sql

2) Dump the table contents as XML files to the local disk. I went with XML files because I had trouble getting this particular data set to dump and then import using the ‘batch’ format MySQL more natively supports.



for i in `mysql —host=$MYSQLDUMP_HOST —user=$MYSQLDUMP_USER —password=$MYSQLDUMP_PWD $MYSQLDUMP_DATABASENAME -e "SHOW TABLES" | tail -10`; do
    mysql —host=$MYSQLDUMP_HOST —user=$MYSQLDUMP_USER —password=$MYSQLDUMP_PWD $MYSQLDUMP_DATABASENAME —xml -e "SELECT * from $i;"  | tee $i.xml;

done

I used ‘tee’ so I could watch the tables being created, in an attempt to gauge how much work was left. You could just as easily replace “| tee” with “>”.

3) Turn the XML files into INSERT statements. I did this with Python and lxml – you could use just about anything that has a decent XML API.

import.py
python import.py > inserts.sql

4) Create the database, run the INSERTs.

mysql -e "CREATE DATABASE $MYSQLDUMP_DATABASENAME;"

mysql $MYSQLDUMP_DATABASENAME < schema.sql

mysql $MYSQLDUMP_DATABASENAME < inserts.sql

Hope that can help someone else.

XMLReader punts on clone

nospam@example.com (TML) — Tue, 12 Aug 2008 23:12:32 -0600

PHP5’s XMLReader – which I used for my JSON-Like XML post, doesn’t implement a ‘clone’ handler, which came as quite a shock when I attempted to implement a wrapper class that would allow me to push values back into the stream when I had to read-ahead to implement ‘nested data structures’ as suggested by Davey. I had a hard time finding a good example of how to internally implement clone (even Sara Golemon’s book only gives it a brief mention¹), but I finally came up with something – I’m building it now to see how well it works, or even if it works at all.

¹ zend_object_value clone_obj(zval *object TSRMLS_DC) – Used to generate a new object copied from an already instantiated one. The default behavior is to create a new object instance, associate the original’s handler table with it, copy the properties table and, if the class entry for the object in question defines a __clone() method, call that to allow the new object to perform additional duplication work.

JSON-like XML

nospam@example.com (TML) — Sun, 10 Aug 2008 22:01:40 -0600

#phpc on Freenode had a discussion tonight about how there’s nothing in XML that’s quite as simple as JSON for encoding types. I came up with this off-the-cuff – as opposed to WDDX, this is more JSON-like in that each ‘XML Document’ encodes only a single value – but that value might be mapped to any of the built-in types.
Of course, the <object> stuff would only work with pretty simple objects – but the fact that it tries at all to instance a class is a step beyond what JSON does, which is ALWAYS an instance of stdClass. You could drop this extra ‘feature’ by ~~remove~~ removing line 43 ($class = ...) and changing line 44 to:

$data = new stdClass;

Continue reading "JSON-like XML"

Refinement of ringBuffer

nospam@example.com (TML) — Sat, 09 Aug 2008 14:38:32 -0600

    public function offsetSet($key, $value) {
        if ($key === NULL) return $this->push($value);
        $this->throwEx(array(‘Do not directly set indices in a %s, use push() instead’, CLASS));
        return false;
    }

This allows the use of PHP’s ‘push’ syntax: $foo[] = ‘bar’;
Full class after the jump.

Continue reading "Refinement of ringBuffer"

Using SPL to implement a ringBuffer

nospam@example.com (TML) — Fri, 08 Aug 2008 15:39:18 -0600

I liked the SPL I used in my last post, so I cracked open my Python Cookbook to a random location and decided to try and implement whatever it landed on in PHP using SPL – that led to the following implementation of a ring buffer.
There certain may be better ways to do it, but this was my first pass, and I think it behaves much how you’d expect a ring buffer to.

Continue reading "Using SPL to implement a ringBuffer"

Case-insensitive Array access in PHP

nospam@example.com (TML) — Thu, 07 Aug 2008 21:21:48 -0600

Someone in ##PHP was looking for an array that had ‘case-insensitive’ keys, so I threw this together using SPL

Continue reading "Case-insensitive Array access in PHP"