brian's blog

Entries for December 2005

Python: The Language That Gave Birth to REST

Quick silliness before I get into the real content: lately my posts have been about Python web frameworks, where I usually mention how divided the community around them has become. For those of you who don't know just how "bad" we have it, you may be surprised to learn that it's no exaggeration — here's an incomplete list: Myghty, Bricks, Webware, Django, web.py, Quixote, Aquarium, TurboGears, Subway, SkunkWeb, Spyce, Snakelets, Zope, Twisted, Albatross, jonpy... and those are just the multi-faceted toolkits (not including single components like CherryPy, mod_python, etc.).

Anyway, while reading an "old" post by Ben Bangert about URL mapping schemes (one aspect of Controllers) in various frameworks, I noticed that the discussion turned out to be even more fascinating.

We all know that the term REST was coined back in 2000 by Roy Fielding in his dissertation. But what were people calling it before that? Certainly Fielding didn't invent the concept (however, if you didn't know, he did author the HTTP specification). Well apparently, you can look to Python for the answer!

Phillip J. Eby kicked things off with this great post, which I simply can't put any better, so here it is:

"""If you really want to trace back this style of Controller, you'd most likely find yourself looking at Java code."""

I seriously doubt that. I have in front of me the March/April 1997 issue of “Web Apps Magazine”, which has an outside headline “Publishing Objects Without Using CGI…”

Inside, on page 50, is an article entitled “Using the Python Object Publisher”, by Paul Everitt, whom you may now know as the former president of Zope Corporation, then called Digital Creations. The article described “Bobo”, the great-grandpa of what eventually became Zope.

One of the reasons I still have this magazine is because that article was the reason I’m now a Python developer. At the time I read the article, I seized on the concept as being just what I needed for developing web applications, and so I tried implementing something like Bobo in Perl, and was beset by obstacle after obstacle. Not long after, Python became my language of choice.

At the time, I don’t believe Java had much introspective capability, though I could be wrong. However, the idea that it would have been mature enough at that time for somebody to come up with the idea for Java? Seems pretty darn unlikely to me. Note also that a March ‘97 issue of a magazine would’ve had the articles written months ahead of time, so that means that this type of controller has been out there since 1996, using docstrings instead of ‘expose’ attributes to mark methods suitable for publishing. The article mentions a lot of features that would’ve taken some time to develop, and the package being open source, and used with a variety of back-ends like FastCGI, so it’s likely that development was over a good part of 1996.

The article, by the way, either gives examples of, or explains how to do, pretty much all the styles of dispatch and response that you’ve listed here. If you’re going to give props, they rightfully belong to the Zope Pope, Jim Fulton, who singlehandedly invented this entire breed of dynamic language-based, RESTful web MVC, back before “REST” was even coined as a term, back when version 4 browsers were the latest-and-greatest.

Legend has it, by the way, that Jim was pressed into service to give a talk on CGI at the IPC (International Python Conference), and therefore had to learn CGI on the plane in order to give the talk. He was so horrified by what he learned that he invented a completely different paradigm on the flight back. It’s taken this many years for his radical invention to now be considered mainstream and even “cutting edge”.

Anyway, my point is that the frameworks you mention are similar because it’s a natural design, if you aren’t trapped by the CGI mindset. Jim Fulton just happened to do it first, not by a couple months or years, but by over half a decade ahead of even the oldest of the frameworks you mention.

And I’m only here as a Python programmer because of it. Before I tried duplicating Bobo in Perl, I was (believe it or not) perfectly happy with Perl. But once I saw what Bobo could do, I knew it was the right tool for the job. I never could have dreamed that it would take almost a decade for industry to suddenly “discover” this, and still be lagging behind it. No wonder Zope does so well with people who aren’t programmers; they don’t have as much to unlearn! (I personally hadn’t been doing CGI long when I read the article, not enough to stunt my brain anyway. :)

If I knew then what I knew now, Bobo would’ve had a screencast as soon as I could’ve figured out how to make one. There was no blogosphere and no Google, then, but I’d have found some way to spread the word. Bobo could have been for Python what Rails is to Ruby – the killer app you change languages for. It certainly did the trick for me.

Wow! Although I can't find the article Phillip mentions in Google's cache or on Archive.org, there are some similar pieces still floating around — The Python Object Publisher by the apparently under-credited Jim Fulton, dated August 1996, and Scripting the Web with Python by Python author Guido van Rossum, dated October 1996. If anyone can find the original Web Apps Magazine article, please link to it!

What we can take away from all this history is that the REST-MVC model that is so popular today was actually invented as a result of Jim Fulton's experiences at IPC 5.

Paul Everitt, author of the article Phillip cites, then added more interesting history:

At IPC 3, I gave a CGI tutorial. There were some guys from a company called eShop in the tutorial. One of them was this neat guy named Greg Stein. I showed something called the ILU Requestor (props for this go to Rob Head and Brian Lloyd), which eShop subsequently added to their e-commerce product. A product which subsequently became Microsoft’s SiteServer when eShop was bought. (I wrote a W3C tech note afterwards on the ILU Requestor.) When Jim arrived and immediately produced the idea of transparently traversing and calling Python objects, we immediately dropped the ILU Requestor and became devoteees of “object publishing”.

I remember thinking last year, after all this time, most packages haven’t caught up to what Jim had on his laptop after the plane ride back. So many systems seem more like the “Gateway” in CGI, than systems that adopt the object model of the web. I suppose the rise of dynamic languages in the mainstream has now made this idea more realistic.

In the same thread, Stephan Diehl brings up WebObjects by NeXT, which was released earlier in 1996. Jeff Shell notes that while the goal of WebObjects may have been the same, the implementation was quite different. You can still see this today in the URLs at the Apple Store, which certainly does not adhere to REST.

I don't know about you, but when we all started paying attention to this REST business not long ago, I certainly looked around the proverbial room for technologies we could point at as already being "there." Looks like Bobo (and hence Zope) had been there since 1996!

Python Web Framework Experiments, Day 1

Last night (aka this morning) I wrote that I would venture into the realm of making my own Python web framework, without any restrictions on making a complete (as in you-should-use-this-for-your-business) or Pythonic solution. As promised, here's my first attempt at some URL mapping mechanism. But first...

Thanks to Ian Bicking for pointing out Routes. It gave me that slight flutter of "ooh, what if someone's already done that?" For better or worse, it is quite different (and quite nice).

Adrian also corrected me on the origins of Django's components. I actually have spent some time looking at Django, and even read Adrian's presentation, which must be even better with sound & video. So either I just didn't comprehend the history of Django, or it isn't immediately obvious from looking at it that it is a single unit (I think a little of both, yes that is a slight jab, sorry).

Onto some code...

Some things to keep in mind. This is only my first attempt at this particular problem domain; as I said, I'm prepared to experiment. This was the first thing that came to mind and I was pleased when I got a working prototype in about 20 minutes.

Common Problem 1: URL Mapping

Some problems I listed with current URL mapping solutions the other day are regular expression usage, big jumps in complexity for deep URLs, and lots of space between where URLs are defined and where their respective methods are defined.

Hacky Solution 1: Division Overloading for Path Components (a la path), Which Can be Types!

This offers simple validation, keeps the URL and its corresponding method right next to each other (it's silly, but that's what I like the most), makes complex situations a bit easier, and you can call your methods and classes whatever you want.

Simple examples, assuming SQLObject classes for our model:

Edit: This may be painful to read without syntax highlighting, so here are some similar examples in a .py file.

class Blog:

    # Example: /index

    @url('index')
    def homePage(self):
        return Entry.select(orderBy='-created')[:10]


    entries = Path('entries')

    # Example: /entries?query=test&format=rss

    @url(entries, query=str, format=oneOf('html', 'rss'))
    def findEntries(self, query=None, format='html'):
        if query:
            return Entry.search(body=query)
        else:
            return Entry.select()


    # Example: /entries/5

    @url(entries/Entry.id)
    def showEntry(self, entry):
       print "Retrieving entry", entry.id
       return entry

Okay, there's something neat. That entry argument in showEntry won't be the entry ID from the URL, it will be the entry itself with that ID! When a URL component is an SQLObject alternateID field, the corresponding row is automatically retrieved. Let's go back to that findEntries example now that we know it plays nice with SQLObject...

    @url(entries, query=Entry.body)
    def findEntries(self, query):
        for entry in query:
            print "Found entry: %s." % entry.title

What's going on here? Entry.body isn't an alternateID, so it just uses selectBy instead, so query will be a SelectResults sequence. However, this is NOT the same as before, since it will only return exact matches, and there's no way to get the actual string parameter that was requested. Still, you can imagine this working well for, say, a phone book, where you only want to find exact first or last name matches. Continuing on...

    # Example: /entries/5/comments?format=rss

    @url(entries/Entry.id/'comments', format=oneOf('html', 'rss'))
    def listComments(self, entry, format='html'):
        return entry.comments

    # Example: /entries/5/comments/10

    @url(entries/Entry.id/'comments'/int)
    def showComment(self, entry, commentIndex):
        # Show an individual comment, maybe for thread view?
        return entry.comments[commentIndex]

As you can see, unknowns in the URL's path are mapped to positional arguments, while parameters are mapped to keyword arguments. Check out the use of the actual int type as a URL component in that last example. Any type will work, as long as you trust its conversion from string to be an accurate representation (bool('False') is a good example of where this fails).

Now for something a bit more complex, like file browsing in arbitrarily deep folders.

    static = Path('static')

    @url(static/[str]/str)
    def staticFile(self, subdirs, filename):
        import os
        fullPath = os.abspath('/'.join(subdirs + [filename]))
        if not os.path.isdir(fullPath):
            return file(fullPath).read()

    # Example: /static/images/anim/glow.gif
    # -> self.staticFile(['images', 'anim'], 'glow.gif')

Archives are often browsed by date, where the URL components can specify a varying date by year, month, and day. So we want to support anywhere from 0 to 3 consecutive integer arguments (0 showing the whole archive, 1 being the year, etc.)

    @url(entries/'archive'/[int])
    def listByDate(self, date):
        date = date[:3]
        date.extend([None] * (3 - len(date))) # Pad to length 3 with None
        year, month, day = date
        return Entry.selectBy(year=year, month=month, day=day)

Enclosing a component in a list means that it will match any number of that component, and the positional argument will be returned as a list. This is starting to look a bit like regular expressions, except prettier (to me, at least), and the values are converted to their respective types.

There are more things, like using tuples for enumerated values (oneOf is actually just for readability), and possibly using | like in regular expressions, but you get the idea.

This fits my brain nicely: "at this URL, return this function..."

Issues, besides the obvious: there's some repeating going on, but I think this happens in other systems with similar features. For example, you obviously have to retype the path prefixes every time if you're using regular expressions, and if you're not, there's the (typing) overhead of nesting classes (or at least making a bunch of them). And if you include validation, that's more re-typing of parameters (telling the validator which parameter should use which validator).

What about using multiple URLs for a function and vice versa? Well, the neat thing is that the url decorator sets a url property on the method itself, with all the relevant information. Redirecting is made easier with a custom Redirect exception (like in CherryPy), just give it the method, it can determine what URL to redirect to from there:

    @url(entries/'new'):
    def newEntryForm(self):
        return "<form>...</form>"

    @url(entries/'save', title=str, body=str)
    def makeEntry(self, title, body):
        entry = Entry(title=title, body=body)
        raise Redirect(showEntry, entry)

Right now this "simple" solution would actually work for all my current projects. I'm guessing anyone interested in this can probably make a good guess at how it's implemented, so I'll only post about that if someone asks. URL dispatching works fine (this isn't just a make-it-work-like-this proposal); I have a very simple function that takes a URL just like what a web server would receive and calls the appropriate method.

So, web people: could this technique actually survive? What would need to be changed?

Rails and Why Are Hacks & "Pythonic" Code Mutually Exclusive?

If you've just been admiring Ruby from a distance like me, check out this beautiful presentation given by the creator of Rails at the Snakes and Rubies event last weekend. It is simply delicious.

Along with the announcement of Reddit's switch to Python came the announcement of yet another Python web framework called web.py. I am a fan of its author, so I'm inclined to think that it does actually have something new to offer. And while it does some things differently, it still doesn't do things exactly how I'd want them. And that's why there's such fragmentation in the Python web area — because everyone seems to think that about some framework or another.

Here are a list of downsides that currently popular Python web frameworks have:

• Building URL schemes from classes/methods (TurboGears, anything CherryPy based, and many others) may result in lots of nested objects.


• Mapping URL schemes to classes/methods using regular expressions (web.py, Django) sucks because regular expressions suck. Yes they do (and they are overkill for most RESTful URLs).


• Forcing the programmer into specific design patterns due to the above two.


• It's easy to lose track of which URLs go to which methods.


• In almost every method, I'm converting URL parameters from strings (some frameworks automatically convert them if you specify a validator) and setting up try/except blocks to retrieve things with SQLObject, which gets very repetitive.


• Arbitrary resource nesting can get messy to handle if there are non-arbitrary elements in between (consider the path: /files/(any number of subdirectories)/images/(any number of subdirectories)/filename).


• Some mini-frameworks used to be templating-agnostic, but now all these "real solutions" force you into theirs. Is someone going to have to make the templating equivalent of what SQLObject did for SQL?

I think Python programmers are buying into these solutions (and I'm guilty of this as well) because they're "good enough." Even though TurboGears is more cohesive than the DIY glue I was using (Cheetah, CherryPy, SQLObject), it still needs some polish. This is obviously because all the Python frameworks out there are gluing together existing projects and therefore making sacrifices in how well these pieces mesh. Rails has a lot to show for being developed in-house.

So, I was inspired to start a new web project. That's right, you can call me part of the problem. The difference is that I have a simple goal in mind, which has nothing to do with making the "best" solution: use every conceivable Python hack out there. Many Python frameworks claim to be "Pythonic," which usually means they don't do a lot of wizardry and black magic, or at least don't expose that to the programmer.

Well, I want to do things a completely different way and hope to inspire some new solutions to existing problems. And if that means using crazy metaclasses, custom infix operators, constructs not meant for Python, and excessive operator overloading, then so be it. I'm going to experiment with possibilities others feared due to "Pythonic"-imposed limitations, and I'm going to use hacks that others would never dream of putting in "real" code.

My first task was to design a clever and easy-to-understand URL mapping scheme. Tomorrow I'll post the results of my effort and why I think what I came up with beats anything else out there for the use cases I most commonly encounter.

Friday Fun Facts! by Brian

Here are some random thoughts:

Every time I write a frivolous blog entry (that's all of them), I owe someone something. You can rest assured that each entry, including this one, carries with it a burden of guilt.

Case needs a web programming class, and I want need to teach it.

Girls must be really dirty, because I counted thirteen assorted bottles of stuff in a shower shared by two. What would happen if they just used soap and shampoo like normal humans?

Paul Graham's hackers-make-good-hires essay(s?) might be controversial, but I've come up with the same conclusion independently in the form "if you graduate with a CS degree and still don't consider yourself a hacker, there's something wrong." And if I were in a position to hire people, I'd be thinking the same thing.

Python happenings: TurboGears is YAPATROR (Yet Another "Python Answer to Ruby On Rails") and is almost identical to the suite of tools I already develop with, so I'm giving it a try.

I still haven't paid this semester's tuition. Someone please pay me some money (job from last semester, I'm looking at you).