Peterbe.com

uwsgi is the latest and greatest WSGI server and promising to be the fastest possible way to run Nginx + Django. Proof here But! Is it that simple? Especially if you're involving Django herself.

So I set out to benchmark good old threaded fcgi and gunicorn and then with a source compiled nginx with the uwsgi module baked in I also benchmarked uwsgi. The first mistake I did was testing a Django view that was using sessions and other crap. I profiled the view to make sure it wouldn't be the bottleneck as it appeared to take only 0.02 seconds each. However, with fcgi, gunicorn and uwsgi I kept being stuck on about 50 requests per second. Why? 1/0.02 = 50.0!!! Clearly the slowness of the Django view was thee bottleneck (for the curious, what took all of 0.02 was the need to create new session keys and putting them into the database).

So I wrote a really dumb Django view with no sessions middleware enabled. Now we're getting some interesting numbers:

fcgi (threaded)              640 r/s
fcgi (prefork 4 processors)  240 r/s (*)
gunicorn (2 workers)         1100 r/s
gunicorn (5 workers)         1300 r/s
gunicorn (10 workers)        1200 r/s (?!?)
uwsgi (2 workers)            1800 r/s
uwsgi (5 workers)            2100 r/s
uwsgi (10 workers)           2300 r/s

(* this made my computer exceptionally sluggish as CPU when through the roof)

Following on from yesterday's blog about How and why to use django-mongokit I extended the exampleproject which is inside the django-mongokit project with another app called exampleapp_sql which does the same thing as the exampleapp but does it with SQL instead. Then I added a very simple benchmarker app in the same project and wrote three functions:

1. One to create 10/100/500/1000 instances of my class
2. One to edit one field of all 10/100/500/1000 instances
3. One to delete each of the 10/100/500/1000 instances

Here I'm going to explain how to combine Django and MongoDB using MongoKit and django-mongokit.

MongoDB is a document store built for high speed and high concurrency with a very good redundancy story. It's an alternative to relational databases (e.g. MySQL) that is what Django is tightly coupled with in it's ORM (Object Relation Mapping) and what it's called now is ODM (Object Document Mapping) in lack of a better acronym. That's where MongoKit comes in. It's written in Python and it connects to the MongoDB database using a library called pymongo and it turns data from the MongoDB and turns it into instances of classes you have defined. MongoKit has nothing to do with Django. That's where django-mongokit comes in. Written by yours truly.

This cost me some hair-pulling today as I was trying to write a custom test runner for a Django project I'm working on that creates a test Xapian database just for running the tests. Basically, you can't do this:

os.mkdir(database_file_path)

Because if you do you end up getting these strange DatabaseOpeningError exceptions. So, here's how you do it:

import xapian
xapian.WritableDatabase(database_file_path,
                        xapian.DB_CREATE_OR_OPEN)

Hopefully by blogging about this some other poor coder will save some time.

YUI Compressor apparently supports compressing CSS. Cool! I had to try it and what's even more cool is that it's the only other CSS minifier/compressor that doesn't choke on CSS hacks (the ones I tried). The only other CSS minifier/compressor is my very own slimmer. So, let's see what the difference is.

Running the YUI Compressor 10 times on a relatively small file takes 0.3 seconds on average. Running the same with python 2.6 and slimmer.css_slimmer takes 0.1 seconds on average. I think most of this time is spent loading the jar file than the actual time of running the compression.

I have some code that checks that a file is treated differently if some time has passed. In other words, it reacts if the modification time is more than it was before. The code uses os.stat(filename)[stat.ST_MTIME] to do this. The challenge was to mock os.stat so that I can pretend some time has passed without having to wait. Here was my first solution which made running many tests sloooow:

def test_file_changed(self):
    filename = 'foo.txt'
    expect_timestamp = int(time.time())
    ...run the unit test with 'expect_timestamp'...
    time.sleep(1)
    expect_timestamp += 1
    ...run the unit test with 'expect_timestamp'...

So, here's how I mock os.stat to avoid having to do the time.sleep(1) in the test:

def test_file_changed(self):
    filename = 'foo.txt'
    expect_timestamp = int(time.time())
    ...run the unit test with 'expect_timestamp'...
    import os
    from posix import stat_result
    def fake_stat(arg):
        if arg == filename:
            faked = list(orig_os_stat(arg))
            faked[stat.ST_MTIME] = faked[stat.ST_MTIME] + 1
            return stat_result(faked)
        else:
            return orig_os_stat(arg)
    orig_os_stat = os.stat
    os.stat = fake_stat

    expect_timestamp += 1
    ...run the unit test with 'expect_timestamp'...

I hope this helps someone else who's trying to do the same. It took me some time to figure out that os.stat is used by lots of various sub routines in the os module so it's important to only mock the relevant argument otherwise you might get unexpected problems.

One really annoying thing about PIL is that it's importable as Image and PIL. It leads me and other newbies to think if it's different. I don't want choices:

Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41)
[GCC 4.3.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import PIL
>>> import Image
>>> ?

When PIL/Image is put into standard lib, can we call the module: imaging?

I've just finished my first Twitter app. It's basically a just a about using OAuth to allow people to sign up to KungFuPeople.com without having to pick yet another password.

I simply took the oauth.py module by Leah Culver and wrapped it with some useful functions taken from a similar Twitter app we've done at work.

Unlike other Twitter apps for this one I'm using Twitter solely for handling authorization and authentication. That means that it has to work with the existing user + profile functionality but just side-step the sign up and login.

Next goal: Google OAuth

JSMIN - The Javascript Minifier is written in C and it does an excellent job of minifying Javascript code. After all, Douglas Crockford wrote it. I noticed there's a Python implementation of it so I wanted to see how it stacks up against my slimmer which is also written in Python.

For sake of argument I compiled the C version and ran that in my little benchmark and did so by using the subprocess module. Also, for the sake of comparison I threw in a run with YUI Compressor. Here are some quick results:

On js/signup-core.js
--------------------
js_slimmer
from 9708 to 6905 in 0.0245039463043 seconds
jsmin
from 9708 to 6720 in 0.0850019454956 seconds
jsmin.c
from 9708 to 6721 in 0.0026159286499 seconds
yuicompressor
from 9708 to 6102 in 0.914173126221 seconds

On js/zoom.js 
-------------
js_slimmer
from 5920 to 3712 in 0.0106379985809 seconds
jsmin
from 5920 to 3582 in 0.0582370758057 seconds
jsmin.c
from 5920 to 3583 in 0.00282216072083 seconds
yuicompressor
from 5920 to 2771 in 0.839382171631 seconds

On js/diypack.js
----------------
js_slimmer
from 21559 to 14059 in 0.0409741401672 seconds
jsmin
from 21559 to 13655 in 0.177556037903 seconds
jsmin.c
from 21559 to 13656 in 0.00346994400024 seconds
yuicompressor
from 21559 to 11638 in 0.891603946686 seconds

So, roughly, slimmer is 4 times faster than jsmin.py but fails to minify a couple of bytes. jsmin.c is about 6 times faster than slimmer.py but is awkward since it's in C. I guess jsmin.c is the way forward when you want speed and the best result. slimmer has the advantage of being all in python and PyPi and contains functions for CSS, HTML and XHTML as well.

It's clear the YUI Compressor does a wicked job at minifying but by running a .jar file every time in a subprocess is crazily slow if that matters for you.

If you're on the python-uk mailing list you will have already seen this but if you're not, here we go.

Fry-IT is hosting a Code Dojo in our offices. It's free and open to anyone. My colleague Nicholas has written up a little bit about what a Code Dojo is which should get you excited.

Details are available on this page which is also the place to go to secure your place. Currently there are 12 people who say they're coming and we've decided to cap the geek influx to 30 people.

Cheers,
Peter-san