Programming

It took me a long time to figure this out, I didn’t want it to end up locked in my head, so here’s a rough guide on how I managed to get a pylons 0.9.7 project to do LDAP authentication. All of the below is done in config/middleware.py.

The key points are:

1. “import ldap” (for the obvious reason) and “from paste.auth.basic import AuthBasicHandler”
2. Wrap the stacked WSGI ‘app’ object in the AuthBasicHandler you just imported: app = AuthBasicHandler(app, 'The value you want to appear on the browser dialog box', yourAuthFunctionHere)
3. Define an auth function “yourAuthFunctionHere(requestEnvironment, username, password)”. Some tricky bits:
   1. If this function returns “True”, the request will proceed and there will be a new key in the Request object named ‘REMOTE_USER’ that contains the passed username.
   2. If this function returns “False”, the user will be prompted again until it succeeds. (I should probably find a way to limit these so people cannot just keep guessing.)
   3. Python’s LDAP has some strange behaviours:
   • Do ldap.initialize outside the auth function or you will swamp the LDAP server with bind requests
   • Use synchronous bind (‘bind_s’ as opposed to ‘bind’) – asynchronous bind in the WSGI middleware layer caused some really bizarre behaviour here, including segfaults
   • Trap ldap.INVALID_CREDENTIALS and return False when it is raised, or watch all invalid logins crash the server process
   • bind (and bind_s) return a tuple, if the first item in the tuple is int(97), the bind worked – otherwise, they might have bound but as an anonymous user, and we need to fail
   • the second item in that tuple is a list of messages from the server – I’m not handling those at all today, becuase in the sole case we’re interested in (97, or “auth”), the messages appear to be blank in our environment – there’s no reason to believe this is consistent, and I should probably research this further at some point to provide meaningful feedback to the user on failed auth.
4. You can’t touch the session from within the WSGI middleware layer

All of this was done because a new server in our datacenter doesn’t have packages for the old build of Apache that we used to configure our LDAP auth back in the day. I’m actually quite pleased at how the new system works, and am glad to be rid of that Apache+PHP millstone that’s been lurking as a dependancy in all of our Pylons projects simply for the LDAP authentication solution.

I have a project at work where we are using Pylons/paster as the web service provider. One of the “clients” of this service is written in Flex/Flash, and had as a component the AlivePDF AS3 library for generating a static PDF of the Flash content.

Unfortunately, because Flash cannot save content locally, in order to actually GET this PDF content back to the user, AlivePDF posts a byte array to the service and expects the service to bundle that as a PDF and send it back. They provide a “content.php” file as an example – which, frankly, is some pretty inscrutable code until you manage to figure out what it’s working around. So when one of my employees (the one who selected AlivePDF in the first place) sent me the PHP file, it took as long to understand what he wanted as it did to come up with a Pylons solution. You can find the latter bit below – replace ${service} and ${controller} with your actual values through-out, and trim the ellipses (they are there to indicate there may be additional context on either side of the line I’m giving you):


${service}/config/routing.py:
def make_map():
... map.connect(’/${controller}/create.php’, controller=’${controller}’, action=‘pdf’) # put this before the default routes, if you have any
...

${service}/controllers/${controller}.py:
Class ${controller}(BaseController):
... def pdf(self): response.headers[‘Content-Type’] = request.environ[‘CONTENT_TYPE’] response.headers[‘Content-Disposition’] = ‘inline; filename=”%s”’ % request.GET[‘name’] return request.environ[‘wsgi.input’].read(int(request.environ[‘CONTENT_LENGTH’]))
...


I’ve recently been reading through SQL and Relational Theory by C.J. Date, who was deeply involved in the original formulation of relational model. Working my way through the book, I discovered that O’Reilly and Associates have also produced a series of videos where Date lectures on the content of the book, and I’ve really been enjoying it. You can purchase them outright for about $300 USD, or you can subscribe to one month of Safari via their “Library” plan for ~$43 and stream them from there. All together, the videos run just over 16.5 hours – here’s a breakdown of the topics and the durations.

1:19:07 Setting the Scene
1:18:08 Types and domains
0:57:28 Tuples and Relations, Rows and Tables
0:50:47 No Duplicates, No Nulls
1:04:56 Base Relvars, Base Tables
3:15:15 SQL and Relational Algebra I: The Original Operators
1:19:06 SQL and Constraints
0:56:26 SQL and Views
2:04:04 SQL and logic I: Relational Calculus Part 1
0:46:39 SQL and Logic II: Using Logic to Write SQL Code
0:20:56 Further SQL Topics
0:38:53 The Relational Model
0:32:21 A Relational Approach to Missing Information
1:08:31 Database Design Theory

If you work with databases for a living, I really think it would be worth your time to sit through these.

ProggIt had this amusing gem today, but it’s pretty obscure, so I put it here instead of the Homestarmy

So where can this argument possibly go?
I don’t have to tell you; I’m sure you must know.
By reductio, there cannot possibly be
a procedure that acts like the mythical P.

Great stuff.

Something came up in IRC today, I thought I found a pretty nice solution to it, thought I’d share it here. A guy has a table in his database that contains ‘prefixes’. It looks like this:

Given the phone number “18005551234”, he’d like to find the longest valid prefix. He had suggested to me that he was considering a stored procedure which loops, character-by-character, through the input looking for the longest match. Instead, I recommended he use ‘replace’ to trim the ‘prefixes’ off of the input, and let an ‘order by’ tell him which of those actually resulted in the shortest remaining string (and thus, was the longest match).

The query I ended up with goes something like this:
select id from prefixes order by length(replace('18005558355', number, '')) asc limit 1;
Given a better understanding of mysql’s ‘replace()’, we ended up using this instead – note the identities are renamed to reflect his actual sytem:
SELECT id, prefix from routes where locate(prefix, '18005558355') = 1 order by length(prefix) desc limit 1;

Is there a better way? Below the fold is a quick (postgresql) “dump” of the database in question so you can come up with your own proposals.

The always brilliant author of the `from future import *` blog brings us an article discussing a potential JavaScript Hijacking [pdf] scenario – the simple answer is “Don’t send JSON data as a list, send it as an object instead”. That is, if you wanted to send:

["foo", "bar"]

you can avoid the security hole by sending something like this instead:

{"data": ["foo", "bar"]}

and then stripping the object on the other end.

Drag this Make Page Editable link to your “Bookmarks Toolbar” in recent versions of Firefox, and click on it to enable the built-in “Page Composer”, allowing you to edit any page you want!

As a Code Kata, I’ve been working my way through the Project Euler stuff.

Here’s the problem.

Here’s my solution:

	
# A generator to derive the Fibonacci sequence


def fib():
    x, y = 0, 1
    yield x
    yield y
    while 1:
        x, y = y, x+y
        yield y

	
if name == ‘main‘:
    idx = 0
    sum = 0
    gen = fib()

    while idx <= 4000000:
        idx = gen.next()
        if (g % 2 == 0): h = h+g
    print h

Note: You want to be careful how you use infinite generators like the ‘fib()’ I created above.

One guy on the forums noted:

Now, replacing an odd number with O and an even with E, we get: O, O, E, O, O, E, O, O, E, O, O, E, O, O, E…And so each third number is even. We don’t need to calculate the odd numbers. Starting from an two odd terms x, y, the series is: x, y, x + y, x + 2y, 2x + 3y, 3x + 5y

Continuing on with yesterday’s theme of revision control, I decided to post a little bit about how I’m using VCS these days. I’ve moved to using git as a front-end for SVN, using a workflow that follows along the paths laid out here and here. Eventually, I’ll probably look into using partial clones as outlined here.

Since I recently moved to using “svn:externals”: in my main repo, I spent some time this morning looking for information on how to get git-svn to pull these external trees, and here are a few resources I found:

Life and Hacking: svn:externals for noobs!
SVN’s svn:externals to GIT’s Submodule for Rails Plugins
Phly, boy, phly: svn:externals

None of these have actually solved the problem for me yet (I think I may be fighting against the fact that I’m using debian’s packaged git instead of the newest upstream), so I have begun looking into alternatives. This article suggested I look into braid and/or piston, as well as pointing to svnmerge.py, which can assist with automatic branch management. I’ll be trying a few of these out over the coming weeks and will post my findings here.

The following is an excerpt from the 3rd Edition of the O’Reilly and Associates classic Learning GNU Emacs, regarding Emacs’ “vc-mode”, which I’ve been using more often of late:

12.5. VC Command Summary

To give you the flavor of the other things VC can do for you, Table 12-1 provides a summary of VC commands. Each one will be explained in detail, but you can probably guess some of their actions from the command names.

Table 12-1. VC commands

Keystrokes	Command name	Action
C-x v v	vc-next-action	Go to the next logical version control state.
C-x v =	vc-diff	Generate a version difference report.
C-x v u	vc-revert-buffer	Throw away changes since the last checked-in revision.
C-x v ~	vc-version-other-window	Retrieve a given revision in another window.
C-x v l	vc-print-log	Display a file’s change comments and history.
C-x v i	vc-register	Register a file for version control.
C-x v h	vc-insert-headers	Insert version control headers in a file.
C-x v r	vc-retrieve-snapshot	Check out a named project snapshot.
C-x v s	vc-create-snapshot	Create a named project snapshot.
C-x v c	vc-cancel-version	Throw away a saved revision.
C-x v a	vc-update-change-log	Update a GNU-style ChangeLog file.

From this article regarding the Mono 2.0 release, I bring you the following quotes from Miguel de Icaza:

“Thanks to statistical data from MoMA, we’re able to prioritize what people are actually using,” De Icaza said. “What is important about this is there are some API’s that we’ve definitely postponed. For example, [Windows Presentation Foundation] is not something that we’re working on right now.”

WPF is part of Microsoft .NET 3.0 specifications. At one time, it was best known by its code name, “Avalon.” WPF is included in Windows Vista and Windows Server 2008 and provides a graphical subsystem for developers. De Icaza claimed that WPF is not yet widely used by developers at this point.

“The IT developer cycles are not in lockstep with vendor release dates,” he added. “There are definitely the early adopters and they’ll be using the latest APIs and those users won’t be served by Mono. But anyone else on the tech adoption curve should be fine with Mono.”

Am I the only one that thinks this guy can’t open his mouth without sticking his foot in it? I doubt the bulk of the Mono developers really want to broadcast the message “We don’t care about early adopters” as the “party line”.

The always entertaining Paper Trail recently did a write up deciphering the Google paper describing the Google File System – a great read if you’re at all interested in file systems and “How Google Works”

proggit led me to this post by what appears to be a member of Google’s Chrome team, regarding the Squirrelfish Extreme engine’s performance on regex. There’s some fascinating stuff here, including the following gem:

WREC doesn’t save the intermediate matches, it just keeps track of the length of the text matched against /a*/ (repeatCount in my code) and decrements the position and count in order to backtrack. This is great, because it uses only two words of memory for state (instead of N million) but can find the next longest match very quickly (just 4 instructions).

If you’ve never read Mastering Regular Expressions, by Jeff Friedl, you might have trouble following some of the underlying concepts, but you should go out and pick up a copy today

[Note: This is an “Executive Summary” I wrote for some co-workers regarding Steve Yegge’s Business Requirements are Bullshit – thanks to Johnny Elbows for reviewing the summary and giving great feedback.]

Big Design Up Front (BDUF) is the methodology in software development that a program’s design should be completed and perfected before that program’s implementation is started. It is often associated with the waterfall model of software development. The argument between the proponents and critics of BDUF has somewhat degenerated into a “holy war”, with most people believing that a compromise between BDUF and the more extreme variants of agile software development is the best solution to most software development problems.

The most important phase of any BDUF project is “requirements gathering” – a project that does not gather the correct requirements will fail, although it will often take an extended amount of time for the effects of failed requirements gathering to actually impact the project. This creates an environment where either a lot of money is spent chasing poor requirements, or projects fail to start because they cannot sufficiently capture the business requirements.

Steve Yegge, a Google Engineer, has written the following article that suggests that this failure is endemic to projects that even need to consider “requirements gathering”. Taking a page from Peter Lynch and Warren Buffet, he suggests that any development project be approached as an investment, and that we should only “invest” in products we would be interested in owning ourselves.

“You can look at any phenomenally successful company,” he suggests, “and it’s pretty obvious that their success was founded on building on something they personally wanted. The extent that any company begins to deviate from this course is the extent to which their ship starts taking on water.” This means that projects which need to go through a “requirements gathering” phase are inherently doomed because “if it’s something you want, then you already know what the requirements are. You don’t need to “gather” them. You think about it all the time. You can list the requirements from memory.”

As we consider taking on projects, I think there is some sound advice contained here about what can make a project fail or succeed, as well as some tips about how simplicity (good) and imagination (bad) can affect the product itself.

Release Notes in the usual spot, but I wanted to hit a few highlights:

1. ADO.NET 2.0 API for accessing databases
2. ASP.NET 2.0 API for developing Web-based applications
3. Windows.Forms 2.0
4. System.Core – C# 3.0, including full support for LINQ
5. Runtime monkeypatching of compiled assemblies via Mono.Cecil
6. Mono.Cairo – cross-platform rendering: “Currently supported output targets include the X Window System, Quartz, Win32, image buffers, PostScript, PDF, and SVG file output. Experimental backends include OpenGL (through glitz), XCB, BeOS, OS/2, and DirectFB.”
7. Includes client libraries out of the box for: SQLite (presumably version 3), PostgresSQL, DB2, Oracle, Sybase, SQL server, and Firebird

I know of at least one project that has been put a bit on hold because of doubts about developing in C# for platforms other than Win32 – perhaps this will allow us to break the stalemate a bit?