list() coercion

I

Ian Bicking

I have an iterable object. It supports many list-like methods,
specifically __len__. These methods are rather expensive (they result
in database calls, COUNT(*) to be specific), but cheaper than iterating
over the object. Sometimes it is useful to create a list from the
iterator, using list(). However, list() seems to call the object's
__len__, I imagine to pre-allocate space. This is a problem, as
pre-allocation saves much less than is spent doing __len__.

Is there a way I can keep this from happening? Maybe something list()
tries first that I can make fail. (I notice list() catches any
exceptions in __len__ and then will just skip that step)

Ian
 
G

Greg Ewing (using news.cis.dfn.de)

Ian said:
However, list() seems to call the object's
__len__,
>
Is there a way I can keep this from happening?

Give your object an __iter__ method that returns
itself.
 
I

Ian Bicking

Give your object an __iter__ method that returns
itself.

I'm not clear on how that will help...?

It already does have an __iter__ method, but it returns a separate
iterator.

Ian
 
A

Alan Kennedy

Greg Ewing:

Ian said:
I'm not clear on how that will help...?

It already does have an __iter__ method, but it returns a separate
iterator.

Just to be explicitly clear

1. You're retrieving record sets from a database.
2. You want to build a list from the database results, one list entry
for each result row.
3. You don't want to have the list preallocated (thus requiring
invocation of __len__()), because the time saving is not worth it
(compared to the cost of the SQL count() operation), since you're
going to be iterating over the record set anyway.

Therefore, use an iterator to build up the list. This will not call
__len__(). Instead, the list will continually be appended to, until
the iterator raise StopIteration. The list may be re-allocated
multiple times, as the number of records retrieved exceeds the
allocation unit size of lists. But this will likely still be lower
cost than your SQL count().

The object returned from the __iter__() method should have a .next()
method which returns the next row in your result set. So if you have
implemented a .next() on your result set object, define your
__.iter__() method as follows

class ResultSet:

def next(self):
"Pseudocode"
result = databasecursor.fetchone()
if result:
return result
else:
raise StopIteration

def __iter__(self):
return self

Have I understood the problem correctly?
 
R

Raymond Hettinger

Ian Bicking said:
I have an iterable object. It supports many list-like methods,
specifically __len__. These methods are rather expensive (they result
in database calls, COUNT(*) to be specific), but cheaper than iterating
over the object. Sometimes it is useful to create a list from the
iterator, using list(). However, list() seems to call the object's
__len__, I imagine to pre-allocate space. This is a problem, as
pre-allocation saves much less than is spent doing __len__.

Is there a way I can keep this from happening? Maybe something list()
tries first that I can make fail. (I notice list() catches any
exceptions in __len__ and then will just skip that step)

Instead of:

list(yourobj)

use:

list(iter(yourobj))

If that doesn't help, create your own wrapper:

def myiter(it):
for elem in it:
yield it

list(myiter(yourobj))


This idea is to provide 'list' with a wrapper that only supplies
the iter methods and not the len method.


Raymond Hettinger
 
I

Ian Bicking

Instead of:

list(yourobj)

use:

list(iter(yourobj))

Sigh... that's too bad. I'll probably just get rid of the __len__
method instead, as list(yourobj) (list(myobj)?) is a more appealing
idiom than len(somebodysobj).

Ian
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,744
Messages
2,569,483
Members
44,901
Latest member
Noble71S45

Latest Threads

Top