Peterbe.com

In 2001 I started my first and perhaps most successful Open Source project I've ever made: IssueTrackerProduct. After nearly a decade of maintaining it I have now officially abandoned it.

It all started when I needed a way to track feedback on my personal website. That's why it was originally called "SiteTrackerProduct". I needed something where I could collect bug reports and any other pieces of feedback and then process it in some structured fashion. It was therefore very important that it would be possible to run the application open for anonymous access. People should be able to submit bugs and issues without having to create an account. You see, kids, back in that day it was actually very common that sites would force users to register and create accounts even just because the content owner wanted it. These days, it's common knowledge that to get people to open up and share anything for others to benefit you make it absolutely trivial to jump straight in without having to see a registration page that looks like a tax return form.

Now, since I long ago abandoned the Zope2 application server technology stack and I no longer use IssueTrackerProduct for anything real it's no longer feasible to maintain this project. In the last five years or so we were actually using it actively to track all projects at Fry-IT where I used to work. I have to say, even though we did grow out of it, it was actually successful. It handled the load (after some much needed patches towards optimization) and it was easy for people to actually use since unlike many other bug trackers, it focused on the non-technical end user first and foremost. As much as possible was done to make it trivial to type in your bug or issue and it automatically took care of all notifications and access rights.

Being a personal Open Source project, over the years, it became a melting pot for experimenting and perfecting various new ideas. Many of them we take for granted today but back then it was quite novel if I may say so. This includes:

• ability to auto-save unfinished form inputs (added before Gmail had it)
• automatic updates of the content without reload made it possible to see other people participating as you're typing
• ability to reply directly to email notifications without having to open a web browser
• an advanced via-the-web programmable interface for adding and modifying custom fields (e.g. "Customer reference code")
• full-text search combined with ability to search on specific fields by key
• file attachments that are images automatically appear as little thumbnails
• file attachments that have text become searchable (e.g Word documents)
• advanced filtering where you can easily decide to search inclusive or exclusive on certain fields
• persistent filtering automatically saved and share'able between different users
• programmable search filters that are coded in Python which made it possible to create very specific reports
• ability to export and import bugs from and to Excel for offline processing

Writing all of this, I can not resist to get a bit nostalgic. I did sink A LOT of time into this project. Today when I look back at the code and almost feel sick seeing all the mistakes that I made. Much of the ugliness of the code can be attributed partially to the fact that I often used and abused the code to add new features. Also, because we often needed some features (since it was used to manage all of our projects) "yesterday" and then it was hard to justify doing things "properly". For example, the main .py file is over 14,000 lines of code!

I did called it "perhaps most successful Open Source project I've ever made" in the first sentence. The reason for that is that over the years many many people have downloaded it and installed and let it be used by thousands of users. That's something to be proud of.

Anyway! It's time to move on. So long and thank you for all the fish!

The code is still available at github.com/peterbe/IssueTrackerProduct

I was reading this article about linkfluence moving from CouchDB to Riak

Does that sound familiar, fellow Zope developer? I know a lot about ZODB but little about CouchDB. One thing that a lot of people don't know about ZODB is that it's very fast and I think this is true about CouchDB too. Speed isn't the same as a raw speed of inserts/queries because with the concurrency variable added the story gets a lot more complex.

It's the exact same perspectives I've always had on ZODB:

1) It's really convenient and powerful

2) It being a single HUGE file makes it hard to scale

3) Versioning can be nifty but it's often not needed and causes headache with the packing

4) It works great but when it cracks it cracks hard and cryptically

This cost me a good hour of debugging so I thought I'd share it in case anybody else stumbles across the same problem. In the end, to solve my problem I had to add debug statements to StringIO.py to be able to find out where in my Zope Page Template a non-unicode string with non-ascii characters appeared and messed things up.

The error I was getting was this, which I suspect several Zope developers have encountered before:

UnicodeDecodeError: \
'ascii' codec can't decode byte 0xc3 in position 46: ordinal not in range(128)

The traceback only mentions files in the innards of ZPT of which none you can really do anything about. We all know that the key to avoid Unicode error is to be consistent. You can do this:

>>> '\xc3' + 'string'
'\xc3string'
>>> u'\xc3' + u'string'
u'\xc3string'

But you can't do this:

>>> '\xc3' + u'string'
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 0: 
ordinal not in range(128)

So, how did I get these non-unicode strings into my application in first place. Simple, I have a search parameter q and end up with a URL like this:

/bla/bla?q=régime

And the template had this little innocent piece of code:

<input tal:attributes="value request/q"/>

That's what f'ed everything up. So, I ended up having to add this:

<input tal:attributes="value python:context.unicodify(request['q'])"/> 

With this little helper function in the base class:

def unicodify(self, s):
   if isinstance(s, str):
       return unicode(s, 'utf8')
   return s

So, hopefully by writing this it will help someone else making the same trivial mistake and not wasting their evening with sporadic print statements all over their frameworks code.

Doing some on-site consulting on an old Zope CMS that has been developed by many different developers over many years. It's pretty good and has lots of powerful features but over the years certain things have been allowed to slip. One problem was that you couldn't click the "Properties" tab. The reason was that it was trying to fetch properties that didn't exist anymore. What had happened was that the class attribute _properties (which is used by the "Properties" tab in the ZMI) had been stored as a persistent attribute. Here's how to solve that:

def manage_fixPropertiesProblem(self):
    """ fix so _properties becomes a class attribute instead """
    if '_properties' in self.__dict__.keys():
        del self._properties

    return "Awesome!"

I've just released a new little project in Python for tracking memory usage in Zope2 applications with the added benefit that you can hopefully see what URL causes which memory usage "jumps". Hopefully this can help Zope2 developers find out what causes RAM bloat but can also help in helping you optimize your application by early in the development process find out what uses too much RAM. I wouldn't be surprised that there is already a program that does something like this. I've just never seen one. Also by putting this out as an Open Source project and blogging about it hopefully more clever people than me will come forward and point out the right way to do things.

I've also used Google Code this time to manage the project. I've used it before but only for hosting a public SVN for the IssueTrackerProduct SVN. I have to say that I was quite impressed with Google Code this second time. I think it's still fundamentally wrong to confuse people with by offering both download and SVN checkout. I did both this time but I think I might give up on the downloads because who out there, who understands that he/she needs to debug RAM usage, doesn't know how to use SVN?

Finally a little disclaimer: By writing about this here, preparing it on Google Code and writing a README.txt file I've now spent more time "managing" the project than I have on coding it. It's an early test release which hopefully will stir up some ideas for genuine important improvements. I had fun coding it as well since this is my first attempt with Flot which has been great to work with. You get very quick and powerful results. Lastly, I haven't tested this in anything but 32-bit Ubuntu Linux and Firefox.

Here is a sample report: 2008-05-30_16.47.32__3.8_minutes

Martijn Faasen is my hero. Not only is an absolutely brilliant coder he's also able talk so that mortals understand.

This is why I like Grok

What he's replying about is mainly the question "What does Grok give me that, say, django does not?"

And, this is why I dislike Grok

Yes, you clever people. It's the same link. For some reason all the great documentation goes into replies on the mailing list rather than into a concise web page with cookbook, book and styled and funny tutorials. Why is that? They've actually made it quite easy now to enter documentation on grok.zope.org with the new Plone site.

An equally important question is: Why don't I do something about it rather than to complain? Well, I've written one how-to at least. My other "excuse" is that I'm not yet an expert enough and hence writing good documentation takes a very long time.

I think there's an important philosophical and political issue at hand. The Grok community is filled with really clever people who are very senior in the web development industry who like using mailing lists and perhaps more importantly, don't need documentation since they can study source code and unit tests to answer their questions. I know this is a sensitive statement but I'll take my chances since it implies that these guys are smarter (or perhaps just more time on their hands).

My internal battle of which new web framework to put my energy into continues. Today (thanks to Martijn's post) Grok earned one more point.

Don't ask why I'm developing products for Zope 2.7 but I had to and I should have been more careful with these oldtimers.

I kept getting this error:

TypeError:  expected 2 arguments, got 1

(notice the strange double space after the : colon) This is different from the standard python TypeError when you get the parameters wrong which looks like this TypeError: __init__() takes exactly 2 arguments (1 given).

The line it complained this happened looked like this:

class MyTool(SimpleItem, UniqueObject, OtherClass):
   id = 'some_tool'
   meta_type = 'some meta type'
   def __init__(self, id='some_tool'):
       self.id = id  # <--- THIS WAS THE CULPRIT LINE APPARENTLY!!

I couldn't understand what the hell was wrong on that line! Clearly it wasn't a normal Python error. Here's the explaination: That OtherClass was a new-style class inheriting from object. It looked like this:

class OtherClass(object):
   ...

When I changed that to:

class OtherClass:
   ...

The whole thing started to work. Long lesson learnt, don't use new-style classes mixed in into Zope 2.7.

Since I've seen so many poor solutions to this problem I thought I'd share mine. Here's how I make printer friendly pages.

1. Add a method in your base class that looks like this:

 def getMainTemplate(self):
      """ return the suitable METAL header object """
      # assuming zpt/main_template.zpt
      template_obj = self.main_template

      # assuming the "first" line of main_template.zpt to
      # look like this:
      # <metal:block metal:define-macro="master">
      return template_obj.macros['master']

2. Change all your Page Templates to refer to a method rather than a macro directly so that pages like index_html.zpt start like this:

<html metal:use-macro="here/getMainTemplate">

I've seen the hard coded way too many times where people do something like this <html metal:use-macro="here/main_template/macros/master"> which gives you no flexibility.

3. Now make a copy of main_template.zpt called print_main_template.zpt and the most important change is to make print.css load by default. Here's what it should look like this somewhere inside the <head> tag:

<link rel="stylesheet" type="text/css" href="/screen.css" media="screen" />
<link rel="stylesheet" type="text/css" href="/print.css" />

Note how the print.css link tag is now not conditional. Before in main_template.zpt it should have looked like this:

<link rel="stylesheet" type="text/css" href="/print.css" media="print" /> 
<link rel="stylesheet" type="text/css" href="/screen.css" media="screen" />

And note how the order is stacked just to be extra safe to weird browsers that don't understand the media condition.

As a last optional feature you should add is to add these lines at the bottom of the template 'print_main_template.zpt':

<script type="text/javascript">
window.print();
</script>
</body>
</html>

Another tip is to add something like this to the footer because it becomes useful when you look at a printed copy:

<div id="footer">
   Printed from <span tal:replace="here/absolute_url"></span> on 
   <span tal:replace="python:here.ZopeTime().strftime('%Y/%m/%d')"></span>

4. Now rewrite the method getMainTemplate() to become usefully intelligent:

  def getMainTemplate(self):
      """ return the suitable METAL header object """
      if self.REQUEST.get('print-version'):
          # assuming zpt/print_main_template.zpt
          template_obj = self.print_main_template
      else:
          # assuming zpt/main_template.zpt
          template_obj = self.main_template

      # assuming the "first" line of main_template.zpt to
      # look like this:
      # <metal:block metal:define-macro="master">
      return template_obj.macros['master']

5. Prepare the interface now for the printer friendly page. This can be done in two different ways. One way is to put a link in the footer or byline like this:

<a href="?print-version=1">Print this page</a>

Or if you want to force a particular page to always be printer friendly, for example print_invoice.zpt then write it like this:

<tal:item define="dummy python:request.set('print-version',1)"
          replace="nothing" 
 /><html metal:use-macro="here/getMainTemplate">

As a final point; how you solve your web design with screen.css and print.css varies. One way is to define multiple css files each suitable for individual things like this example shows:

<link rel="stylesheet" type="text/css" href="/typography.css" /> 
<link rel="stylesheet" type="text/css" href="/print.css" media="print" /> 
<link rel="stylesheet" type="text/css" href="/screen.css" media="screen" />

An alternative solution is to don't expect print.css to stand on its own two legs but only be a supplement of the general css file. When doing this you're probably just going to want to override some things and hide some other things like this example from a 'print.css':

body {
   width:100% !important
}

form#login, #navigation, .also-online {
  display:none
}

To conclude

This gives you a robust framework for enabling printer friendly pages that are quite different from the main template and doing it like this means that you don't have add conditional hacks to your main template that displays certain things if in printer friendly mode or not.

Most importantly, this gives you the framework for adding other versions of main template. For example these:

• mobile_main_template.zpt (guess what for)
• minimal_main_template.zpt (for things like Help page popups)

A healthy and fair use of METAL macros is also key to asserting that you don't have to repeat yourself too much in the copies of main_template.pt.

Good luck!

Here's one of many things I've learnt today at PyCon. Inspired by code that Grig Gheorghiu showed in his slides on automated testing, you can monkey patch a standard library that your application is using in your unit tests to, in my case, mock a remote service without having to run a server. I've done lots of monkey-patching in Zope but then I've only been monkey patching individual methods or attributes of imported classes. This is very similar to that. Here's what my application does:

from poplib import POP3
class MyZopeApp(...):
   def check4mail(self, hostname, port, user, pwd):
       connection = POP3(hostname, port=port)
       ...download emails and process them...

Adjacent to this I have a unit/integration test that looks like this:

class TestCase(ZopeTestCase):
   def test_check4mail(self):
       # monkey patch!
       # note that this imports a module, not a class
       from Products.IssueTrackerProduct import IssueTracker 
       FakePOP3.files = ('test1.email',)
       IssueTracker.POP3 = FakePOP3

       # now check what happens when check4mail() is run
       result = self.folder.tracker.check4mail()
       assert ...

I've installed a lot of Zope instances on my laptop since version 2.7.3 and out of curiosity and desperate need for more hard drive space I thought I'd log rotate them all with the standard Linux logrotate program.

Before doing the log rotate, the total size of all my event.log files came to about 290Mb! After running logrotate (twice of course to go from event.log.1 to event.log.2.gz) the total size become 20Mb. Not a huge significance in the world of gigabyte hard drives but at least something.