Optimization of __len__() in cgi.py

[Bob Kline]

I have a suggestion for speeding up the performance of code like this:

fields = cgi.FieldStorage()
if fields: ...

which, as it turns out, invokes FieldStorage.__len__(), which in turn
calls FieldStorage.keys(), which builds a list of keys by hand, taking
several minutes for large forms. This implementation of keys() reduces
the amount of time taken by several orders of magnitude:

def keys(self):
return {}.fromkeys([i.name for i in self.list]).keys()

Is there a better place for submitting suggestions like this?

Bob Kline

 

[Marc 'BlackJack' Rintsch]

I have a suggestion for speeding up the performance of code like this:

fields = cgi.FieldStorage()
if fields: ...

which, as it turns out, invokes FieldStorage.__len__(), which in turn
calls FieldStorage.keys(), which builds a list of keys by hand, taking
several minutes for large forms. This implementation of keys() reduces
the amount of time taken by several orders of magnitude:

def keys(self):
return {}.fromkeys([i.name for i in self.list]).keys()

This does not maintain the order of `self.list`. Don't know if there's
code relying on this.

In a recent Python version one could use `set()` and a generator
expression::

return list(set(item.name for item in self.list))

But maybe it's even faster to change `FieldStorage.__len__()` to return
the length of `self.list` directly without the need to create an
intermediate list that's thrown away immediately.

Ciao,
Marc 'BlackJack' Rintsch

 

[Sybren Stuvel]

Bob Kline enlightened us with:

I have a suggestion for speeding up the performance of code like
this:

fields = cgi.FieldStorage()
if fields: ...

which, as it turns out, invokes FieldStorage.__len__()

FieldStorage.__nonzero__ tried first if it exists. You might want to
use that for more optimization.

Sybren

 

[Georg Brandl]

I have a suggestion for speeding up the performance of code like this:

fields = cgi.FieldStorage()
if fields: ...

which, as it turns out, invokes FieldStorage.__len__(), which in turn
calls FieldStorage.keys(), which builds a list of keys by hand, taking
several minutes for large forms. This implementation of keys() reduces
the amount of time taken by several orders of magnitude:

def keys(self):
return {}.fromkeys([i.name for i in self.list]).keys()

Is there a better place for submitting suggestions like this?

Post a RFE to the Python Tracker at
http://sourceforge.net/tracker/?group_id=5470&atid=355470

If you want, assign it to me (gbrandl).

Georg

 

[Bob Kline]

def keys(self):
return {}.fromkeys([i.name for i in self.list]).keys()

This does not maintain the order of `self.list`. Don't know if there's
code relying on this.

Such code would be flying in the face of an implication that the order
of the keys is *not* preserved, as the keys() method is documented as
"Dictionary style keys() method" (and the documentation for dictionary
keys() says "... listed in an arbitrary order which is non-random ...").

If it were decided that the behavior of keys() should be preserved even
to the extent of preserving the order of the first occurrence of each
field name, then the optimization could just be moved to __len__():

def __len__(self):
return len({}.fromkeys([i.name for i in self.list]))

But maybe it's even faster to change `FieldStorage.__len__()` to return
the length of `self.list` directly without the need to create an
intermediate list that's thrown away immediately.

This approach also risks breaking code that assumes it's getting the
number of unique field names (an assumption which in this case would be
justified, given the documentation of the keys() method quoted above).

Bob

 

[Bob Kline]

FieldStorage.__nonzero__ tried first if it exists. You might want to
use that for more optimization.

Excellent suggestion! It would be nice if this were adopted to
supplement the original optimization, rather than replace it.

Bob

 

[Bob Kline]

Post a RFE to the Python Tracker at
http://sourceforge.net/tracker/?group_id=5470&atid=355470

If you want, assign it to me (gbrandl).

Done, thanks.

Bob