list() coercion

Discussion in 'Python' started by Ian Bicking, Jul 17, 2003.

  1. Ian Bicking

    Ian Bicking Guest

    I have an iterable object. It supports many list-like methods,
    specifically __len__. These methods are rather expensive (they result
    in database calls, COUNT(*) to be specific), but cheaper than iterating
    over the object. Sometimes it is useful to create a list from the
    iterator, using list(). However, list() seems to call the object's
    __len__, I imagine to pre-allocate space. This is a problem, as
    pre-allocation saves much less than is spent doing __len__.

    Is there a way I can keep this from happening? Maybe something list()
    tries first that I can make fail. (I notice list() catches any
    exceptions in __len__ and then will just skip that step)

    Ian
     
    Ian Bicking, Jul 17, 2003
    #1
    1. Advertising

  2. Ian Bicking wrote:
    > However, list() seems to call the object's
    > __len__,
    >
    > Is there a way I can keep this from happening?


    Give your object an __iter__ method that returns
    itself.

    --
    Greg Ewing, Computer Science Dept,
    University of Canterbury,
    Christchurch, New Zealand
    http://www.cosc.canterbury.ac.nz/~greg
     
    Greg Ewing (using news.cis.dfn.de), Jul 17, 2003
    #2
    1. Advertising

  3. Ian Bicking

    Ian Bicking Guest

    On Wed, 2003-07-16 at 22:04, Greg Ewing (using news.cis.dfn.de) wrote:
    > Ian Bicking wrote:
    > > However, list() seems to call the object's
    > > __len__,
    > >
    > > Is there a way I can keep this from happening?

    >
    > Give your object an __iter__ method that returns
    > itself.


    I'm not clear on how that will help...?

    It already does have an __iter__ method, but it returns a separate
    iterator.

    Ian
     
    Ian Bicking, Jul 17, 2003
    #3
  4. Ian Bicking

    Alan Kennedy Guest

    Ian Bicking wrote:

    >>> However, list() seems to call the object's
    >>> __len__,
    >>>
    >>> Is there a way I can keep this from happening?


    Greg Ewing:

    >> Give your object an __iter__ method that returns
    >> itself.


    Ian Bicking wrote:

    > I'm not clear on how that will help...?
    >
    > It already does have an __iter__ method, but it returns a separate
    > iterator.


    Just to be explicitly clear

    1. You're retrieving record sets from a database.
    2. You want to build a list from the database results, one list entry
    for each result row.
    3. You don't want to have the list preallocated (thus requiring
    invocation of __len__()), because the time saving is not worth it
    (compared to the cost of the SQL count() operation), since you're
    going to be iterating over the record set anyway.

    Therefore, use an iterator to build up the list. This will not call
    __len__(). Instead, the list will continually be appended to, until
    the iterator raise StopIteration. The list may be re-allocated
    multiple times, as the number of records retrieved exceeds the
    allocation unit size of lists. But this will likely still be lower
    cost than your SQL count().

    The object returned from the __iter__() method should have a .next()
    method which returns the next row in your result set. So if you have
    implemented a .next() on your result set object, define your
    __.iter__() method as follows

    class ResultSet:

    def next(self):
    "Pseudocode"
    result = databasecursor.fetchone()
    if result:
    return result
    else:
    raise StopIteration

    def __iter__(self):
    return self

    Have I understood the problem correctly?

    --
    alan kennedy
    -----------------------------------------------------
    check http headers here: http://xhaus.com/headers
    email alan: http://xhaus.com/mailto/alan
     
    Alan Kennedy, Jul 17, 2003
    #4
  5. "Ian Bicking" <> wrote in message
    news:...
    > I have an iterable object. It supports many list-like methods,
    > specifically __len__. These methods are rather expensive (they result
    > in database calls, COUNT(*) to be specific), but cheaper than iterating
    > over the object. Sometimes it is useful to create a list from the
    > iterator, using list(). However, list() seems to call the object's
    > __len__, I imagine to pre-allocate space. This is a problem, as
    > pre-allocation saves much less than is spent doing __len__.
    >
    > Is there a way I can keep this from happening? Maybe something list()
    > tries first that I can make fail. (I notice list() catches any
    > exceptions in __len__ and then will just skip that step)


    Instead of:

    list(yourobj)

    use:

    list(iter(yourobj))

    If that doesn't help, create your own wrapper:

    def myiter(it):
    for elem in it:
    yield it

    list(myiter(yourobj))


    This idea is to provide 'list' with a wrapper that only supplies
    the iter methods and not the len method.


    Raymond Hettinger
     
    Raymond Hettinger, Jul 17, 2003
    #5
  6. Ian Bicking

    Ian Bicking Guest

    On Thu, 2003-07-17 at 10:08, Raymond Hettinger wrote:
    > Instead of:
    >
    > list(yourobj)
    >
    > use:
    >
    > list(iter(yourobj))


    Sigh... that's too bad. I'll probably just get rid of the __len__
    method instead, as list(yourobj) (list(myobj)?) is a more appealing
    idiom than len(somebodysobj).

    Ian
     
    Ian Bicking, Jul 18, 2003
    #6
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Harald Kirsch

    jython type coercion to interface type

    Harald Kirsch, Jun 25, 2003, in forum: Python
    Replies:
    1
    Views:
    1,707
    Harald Kirsch
    Jun 26, 2003
  2. Bob Gailer

    RE: list() coercion

    Bob Gailer, Jul 17, 2003, in forum: Python
    Replies:
    1
    Views:
    314
    Raymond Hettinger
    Jul 17, 2003
  3. Holger Joukl

    python 2 coercion

    Holger Joukl, Aug 31, 2004, in forum: Python
    Replies:
    0
    Views:
    380
    Holger Joukl
    Aug 31, 2004
  4. Neil Schemenauer

    PEP: Generalised String Coercion

    Neil Schemenauer, Aug 6, 2005, in forum: Python
    Replies:
    1
    Views:
    283
  5. Replies:
    1
    Views:
    254
Loading...

Share This Page