Destruction of generator objects

Discussion in 'Python' started by Stefan Bellon, Aug 7, 2007.

  1. Hi all,

    I'm generating a binding from Python to C using SWIG. On the C side I
    have iterators over some data structures. On the Python side I
    currently use code like the following:

    def get_data(obj):
    result = []
    iter = make_iter(obj)
    while more(iter):
    item = next(iter)
    result.append(item)
    destroy(iter)
    return result

    Now I'd like to transform it to a generator function like the following
    in order to make it more memory and time efficient:

    def get_data(obj):
    iter = make_iter(obj)
    while more(iter):
    yield next(iter)
    destroy(iter)

    But in the generator case, I have a problem if the generator object is
    not iterated till the StopIteration occurs, but if iteration is stopped
    earlier. In that case, the C iterator's destroy is not called, thus the
    resource is not freed.

    Is there a way around this? Can I add some sort of __del__() to the
    generator object so that in case of an early destruction of the
    generator object, the external resource is freed as well?

    I'm looking forward to hearing your hints!

    --
    Stefan Bellon
    Stefan Bellon, Aug 7, 2007
    #1
    1. Advertising

  2. Stefan Bellon

    MRAB Guest

    On Aug 7, 11:28 pm, Stefan Bellon <> wrote:
    > Hi all,
    >
    > I'm generating a binding from Python to C using SWIG. On the C side I
    > have iterators over some data structures. On the Python side I
    > currently use code like the following:
    >
    > def get_data(obj):
    > result = []
    > iter = make_iter(obj)
    > while more(iter):
    > item = next(iter)
    > result.append(item)
    > destroy(iter)
    > return result
    >
    > Now I'd like to transform it to a generator function like the following
    > in order to make it more memory and time efficient:
    >
    > def get_data(obj):
    > iter = make_iter(obj)
    > while more(iter):
    > yield next(iter)
    > destroy(iter)
    >
    > But in the generator case, I have a problem if the generator object is
    > not iterated till the StopIteration occurs, but if iteration is stopped
    > earlier. In that case, the C iterator's destroy is not called, thus the
    > resource is not freed.
    >
    > Is there a way around this? Can I add some sort of __del__() to the
    > generator object so that in case of an early destruction of the
    > generator object, the external resource is freed as well?
    >
    > I'm looking forward to hearing your hints!
    >
    > --
    > Stefan Bellon


    Simple! :)

    def get_data(obj):
    iter = make_iter(obj)
    try:
    while more(iter):
    yield next(iter)
    finally:
    destroy(iter)
    MRAB, Aug 8, 2007
    #2
    1. Advertising

  3. On Wed, 08 Aug, MRAB wrote:

    > Simple! :)


    Sorry, I forgot to mention that I am forced to using Python 2.4.

    --
    Stefan Bellon
    Stefan Bellon, Aug 9, 2007
    #3
  4. On Aug 8, 8:28 am, Stefan Bellon <> wrote:
    > Hi all,
    >
    > I'm generating a binding from Python to C using SWIG. On the C side I
    > have iterators over some data structures. On the Python side I
    > currently use code like the following:
    >
    > def get_data(obj):
    > result = []
    > iter = make_iter(obj)
    > while more(iter):
    > item = next(iter)
    > result.append(item)
    > destroy(iter)
    > return result
    >
    > Now I'd like to transform it to a generator function like the following
    > in order to make it more memory and time efficient:
    >
    > def get_data(obj):
    > iter = make_iter(obj)
    > while more(iter):
    > yield next(iter)
    > destroy(iter)
    >
    > But in the generator case, I have a problem if the generator object is
    > not iterated till the StopIteration occurs, but if iteration is stopped
    > earlier. In that case, the C iterator's destroy is not called, thus the
    > resource is not freed.
    >
    > Is there a way around this? Can I add some sort of __del__() to the
    > generator object so that in case of an early destruction of the
    > generator object, the external resource is freed as well?
    >
    > I'm looking forward to hearing your hints!
    >
    > --
    > Stefan Bellon


    Perhaps read through:

    http://www.python.org/dev/peps/pep-0325/
    http://www.python.org/dev/peps/pep-0342/

    The latter superseding the first.

    Based on these what the WSGI specification:

    http://www.python.org/dev/peps/pep-0333/

    did was mandate that the consumer of data from the generator
    explicitly call the close() method on the generator no matter whether
    all data was consumed or not, or whether an exception occurred.

    result = application(environ, start_response)
    try:
    for data in result:
    if data: # don't send headers until body appears
    write(data)
    if not headers_sent:
    write('') # send headers now if body was empty
    finally:
    if hasattr(result,'close'):
    result.close()

    That the consumer called close() allowed generators to provide a
    close() method to cleanup resources even though older version of
    Python was being used which didn't support automatic means of close()
    being called. In other words, it allowed the required cleanup and was
    forward compatible with newer versions of Python.

    Graham
    Graham Dumpleton, Aug 9, 2007
    #4
  5. On Thu, 09 Aug, Graham Dumpleton wrote:

    > result = application(environ, start_response)
    > try:
    > for data in result:
    > if data: # don't send headers until body appears
    > write(data)
    > if not headers_sent:
    > write('') # send headers now if body was empty
    > finally:
    > if hasattr(result,'close'):
    > result.close()


    Hm, not what I hoped for ...

    Isn't it possible to add some __del__ method to the generator object
    via some decorator or somehow else in a way that works even with Python
    2.4 and can then be nicely written without cluttering up the logic
    between consumer and producer?

    --
    Stefan Bellon
    Stefan Bellon, Aug 9, 2007
    #5
  6. Stefan Bellon <> wrote:

    > On Thu, 09 Aug, Graham Dumpleton wrote:
    >
    > > result = application(environ, start_response)
    > > try:
    > > for data in result:
    > > if data: # don't send headers until body appears
    > > write(data)
    > > if not headers_sent:
    > > write('') # send headers now if body was empty
    > > finally:
    > > if hasattr(result,'close'):
    > > result.close()

    >
    > Hm, not what I hoped for ...
    >
    > Isn't it possible to add some __del__ method to the generator object
    > via some decorator or somehow else in a way that works even with Python
    > 2.4 and can then be nicely written without cluttering up the logic
    > between consumer and producer?


    No, you cannot do what you want in Python 2.4. If you can't upgrade to
    2.5 or better, whatever the reason may be, you will have to live with
    2.4's limitations (there ARE reasons we keep making new releases, after
    all...:).


    Alex
    Alex Martelli, Aug 11, 2007
    #6
  7. Stefan Bellon

    Kay Schluehr Guest

    On Aug 9, 1:14 am, Stefan Bellon <> wrote:
    > On Wed, 08 Aug, MRAB wrote:
    > > Simple! :)

    >
    > Sorry, I forgot to mention that I am forced to using Python 2.4.
    >
    > --
    > Stefan Bellon


    It doesn't matter. You can use try...finally as well in Python 2.4.
    It's just not possible to use except and finally clauses in one
    statement such as:

    try:
    1/0
    except ZeroDivisionError:
    print "incident!"
    finally:
    print "cleanup"

    However in theory you don't really need finally but you can simulate
    it using nested try...except statements:

    -----

    try:
    try:
    1/0
    try:
    print "cleanup"
    except Exception:
    raise FinallyException( sys.exc_info() )
    except ZeroDivisionError:
    print "incident!"
    try:
    print "cleanup"
    except Exception:
    raise FinallyException( sys.exc_info() )
    except Exception:
    exc_cls, exc_value, exc_tb = sys.exc_info()
    if exc_cls == FinallyException:
    fin_exc_cls, fin_exc_value, fin_exc_tb = exc_value[0]
    raise fin_exc_cls, fin_exc_value, fin_exc_tb
    else:
    print "cleanup"
    except Exception:
    raise FinallyException( sys.exc_info() )
    raise exc_cls, exc_value, exc_tb

    -------

    Note that this expression is regenerated from the above
    try...except..finally statement using Py25Lite ( see [1],[2] ) which
    is a tool used to provide some Python 2.5 constructs for programmers
    working with Python 2.4.

    [1] http://www.fiber-space.de/EasyExtend/doc/EE.html
    [2] http://www.fiber-space.de/EasyExtend/doc/Py25Lite/Py25Lite.html
    Kay Schluehr, Aug 11, 2007
    #7
  8. On Sat, 11 Aug, Kay Schluehr wrote:
    > On Aug 9, 1:14 am, Stefan Bellon <> wrote:


    > > Sorry, I forgot to mention that I am forced to using Python 2.4.


    > It doesn't matter. You can use try...finally as well in Python 2.4.
    > It's just not possible to use except and finally clauses in one
    > statement such as:


    [snip]

    The problem is yield inside try-finally which is not possible in 2.4.

    --
    Stefan Bellon
    Stefan Bellon, Aug 11, 2007
    #8
  9. Stefan Bellon

    Kay Schluehr Guest

    On Aug 11, 12:16 pm, Stefan Bellon <> wrote:
    > On Sat, 11 Aug, Kay Schluehr wrote:
    > > On Aug 9, 1:14 am, Stefan Bellon <> wrote:
    > > > Sorry, I forgot to mention that I am forced to using Python 2.4.

    > > It doesn't matter. You can use try...finally as well in Python 2.4.
    > > It's just not possible to use except and finally clauses in one
    > > statement such as:

    >
    > [snip]
    >
    > The problem is yield inside try-finally which is not possible in 2.4.
    >
    > --
    > Stefan Bellon


    Oh, yeah... i overlooked this.

    Honestly, I'd recommend wrapping the generator into a function object,
    create the resource on construction ( or pass it ) and destroy it
    implementing __del__.

    def gen_value(self):
    while True:
    yield self.iter.next()

    class GeneratorObj(object):
    def __init__(self, obj, gen):
    self.iter = make_iter(obj)
    self.gen = gen(self)

    def __del__(self):
    destroy(self.iter)

    def next(self):
    return self.gen.next()
    Kay Schluehr, Aug 11, 2007
    #9
  10. On Sat, 11 Aug, Kay Schluehr wrote:

    > Honestly, I'd recommend wrapping the generator into a function object,
    > create the resource on construction ( or pass it ) and destroy it
    > implementing __del__.
    >
    > def gen_value(self):
    > while True:
    > yield self.iter.next()
    >
    > class GeneratorObj(object):
    > def __init__(self, obj, gen):
    > self.iter = make_iter(obj)
    > self.gen = gen(self)
    >
    > def __del__(self):
    > destroy(self.iter)
    >
    > def next(self):
    > return self.gen.next()


    Ok, I think there is an __iter__ missing in order to implement the
    iterator protocol, and I don't see why the generator cannot be inside
    the class itself.

    Anyway, I came up with this code now:

    class ListGenerator(object):
    def __init__(self, iter):
    print "gen init"
    self.iter = iter
    self.gen = self._value()

    def __del__(self):
    print "gen del"
    destroy(self.iter)

    def _value(self):
    print "gen value"
    while more(self.iter):
    yield next(self.iter)

    def __iter__(self):
    print "gen iter"
    return self

    def next(self):
    print "gen next"
    return self.gen.next()

    When iterating over such a generator, I see the following output:

    >>> list(obj)

    gen init
    gen iter
    gen next
    gen value
    gen next
    gen next
    gen next
    gen next
    ['Item1', 'Item2', 'Item3', 'Item4']

    But I do not see an output of "gen del" which makes me think that the
    destructor is not called, thus not releasing the resource. It seems I
    have not completely understood how generators work ...

    --
    Stefan Bellon
    Stefan Bellon, Aug 11, 2007
    #10
  11. Stefan Bellon

    Kay Schluehr Guest

    On Aug 11, 2:00 pm, Stefan Bellon <> wrote:
    > On Sat, 11 Aug, Kay Schluehr wrote:
    > > Honestly, I'd recommend wrapping the generator into a function object,
    > > create the resource on construction ( or pass it ) and destroy it
    > > implementing __del__.

    >
    > > def gen_value(self):
    > > while True:
    > > yield self.iter.next()

    >
    > > class GeneratorObj(object):
    > > def __init__(self, obj, gen):
    > > self.iter = make_iter(obj)
    > > self.gen = gen(self)

    >
    > > def __del__(self):
    > > destroy(self.iter)

    >
    > > def next(self):
    > > return self.gen.next()

    >
    > Ok, I think there is an __iter__ missing in order to implement the
    > iterator protocol, and I don't see why the generator cannot be inside
    > the class itself.


    Sure.

    [...]

    > But I do not see an output of "gen del" which makes me think that the
    > destructor is not called, thus not releasing the resource. It seems I
    > have not completely understood how generators work ...


    But why shall the destructor be called? Your example does not indicate
    that a ListGenerator object is somewhere destroyed neither explicitely
    using del nor implicitely by destroying the scope it is living in.
    Kay Schluehr, Aug 11, 2007
    #11
  12. On Sat, 11 Aug, Kay Schluehr wrote:

    > But why shall the destructor be called? Your example does not indicate
    > that a ListGenerator object is somewhere destroyed neither explicitely
    > using del nor implicitely by destroying the scope it is living in.


    After having constructed the list itself, the generator is exhausted
    and not iterated or referenced anymore, so the generator should be
    destroyed, shouldn't it?

    Ok, let's make the example easier and take out the external iterator
    resource and just concentrate on the Python part:

    class ListGenerator(object):
    def __init__(self):
    print "gen init"
    self.gen = self._value()

    def __del__(self):
    print "gen del"

    def _value(self):
    print "gen value"
    for i in xrange(4):
    yield i

    def __iter__(self):
    print "gen iter"
    return self

    def next(self):
    print "gen next"
    return self.gen.next()

    Now, doing the following:

    >>> a = ListGenerator()

    gen init
    >>> a

    <__main__.ListGenerator object at 0x4020c3ec>
    >>> for i in a: print i

    ....
    gen iter
    gen next
    gen value
    0
    gen next
    1
    gen next
    2
    gen next
    3
    gen next
    >>> del a
    >>>


    So why is the destructor not called when the generator is even
    explicitly 'del'ed? Does somebody else still hold a reference on it?
    But then, even when terminating the interpreter, __del__ is not called.
    When taking out the yield statement, __del__ is called again. It looks
    to me that as soon as a generator function is involved in the class,
    the __del__ is not called anymore.

    --
    Stefan Bellon
    Stefan Bellon, Aug 11, 2007
    #12
  13. On Sat, 11 Aug 2007 14:50:33 +0200, Stefan Bellon wrote:

    > On Sat, 11 Aug, Kay Schluehr wrote:
    >
    >> But why shall the destructor be called? Your example does not indicate
    >> that a ListGenerator object is somewhere destroyed neither explicitely
    >> using del nor implicitely by destroying the scope it is living in.

    >
    > After having constructed the list itself, the generator is exhausted
    > and not iterated or referenced anymore, so the generator should be
    > destroyed, shouldn't it?
    >
    > Ok, let's make the example easier and take out the external iterator
    > resource and just concentrate on the Python part:
    >
    > class ListGenerator(object):
    > def __init__(self):
    > print "gen init"
    > self.gen = self._value()
    >
    > def __del__(self):
    > print "gen del"
    >
    > def _value(self):
    > print "gen value"
    > for i in xrange(4):
    > yield i
    >
    > def __iter__(self):
    > print "gen iter"
    > return self
    >
    > def next(self):
    > print "gen next"
    > return self.gen.next()
    >
    > Now, doing the following:
    >
    >>>> a = ListGenerator()

    > gen init
    >>>> a

    > <__main__.ListGenerator object at 0x4020c3ec>
    >>>> for i in a: print i

    > ...
    > gen iter
    > gen next
    > gen value
    > 0
    > gen next
    > 1
    > gen next
    > 2
    > gen next
    > 3
    > gen next
    >>>> del a
    >>>>

    >
    > So why is the destructor not called when the generator is even
    > explicitly 'del'ed?


    The generator is not ``del``\ed, just the name `a` is removed.

    > Does somebody else still hold a reference on it?


    Yes, the interactive Python shell holds the last non-`None` result in `_`:

    >>> from forum import ListGenerator
    >>> a = ListGenerator()

    gen init
    >>> a

    <forum.ListGenerator object at 0xb7d12c4c>
    >>> for i in a: print i

    ....
    gen iter
    gen next
    gen value
    0
    gen next
    1
    gen next
    2
    gen next
    3
    gen next
    >>> del a
    >>> _

    <forum.ListGenerator object at 0xb7d12c4c>
    >>> 42

    gen del
    42

    > But then, even when terminating the interpreter, __del__ is not called.


    Because that is not guaranteed by the language reference. The reason why
    it is a bad idea to depend on `__del__` for important resource management.

    Ciao,
    Marc 'BlackJack' Rintsch
    Marc 'BlackJack' Rintsch, Aug 11, 2007
    #13
  14. On Sat, 11 Aug, Marc 'BlackJack' Rintsch wrote:

    > On Sat, 11 Aug 2007 14:50:33 +0200, Stefan Bellon wrote:


    > > But then, even when terminating the interpreter, __del__ is not
    > > called.

    >
    > Because that is not guaranteed by the language reference. The reason
    > why it is a bad idea to depend on `__del__` for important resource
    > management.


    Ok, but then we are back to my initial question of whether the destroy
    of

    def get_data(obj):
    iter = make_iter(obj)
    while more(iter):
    yield next(iter)
    destroy(iter)

    can be guaranteed somehow in Python 2.4 while it can be done in Python
    2.5 like follows:

    def get_data(obj):
    iter = make_iter(obj)
    try:
    while more(iter):
    yield next(iter)
    finally:
    destroy(iter)

    And then the answer is: no, it cannot be done in 2.4 (like Alex
    Martelli already said ;-)

    --
    Stefan Bellon
    Stefan Bellon, Aug 11, 2007
    #14
  15. Stefan Bellon

    Kay Schluehr Guest

    On Aug 11, 2:50 pm, Stefan Bellon <> wrote:

    > So why is the destructor not called when the generator is even
    > explicitly 'del'ed? Does somebody else still hold a reference on it?


    You ( we ) have produced a reference cycle. In that case __del__
    doesn't work properly ( according to the docs ). The cycle is caused
    by the following assignment in your code:

    self.gen = self._value()

    So we have to refactor the solution to eliminate the cycle. I spent
    some time to create a generic decorator solution and a protocol for
    handling / releasing the resources.

    1) Create a FinalizerGenerator class that is cycle free ( of course
    you can find a trick to shoot yourself in your foot ). Pass a
    generator function together with its arguments into the constructor.
    An additional callable attribute close is used together with __del__.

    class FinalizerGenerator(object):
    def __init__(self, gen, *args, **kwd):
    print "gen init"
    self._gen = gen # no cycle when passing the _value function
    self.close = lambda: sys.stdout.write("gen del") # assign
    cleanup function

    def __del__(self):
    self.close()

    def __iter__(self):
    print "gen iter"
    return self

    def next(self):
    print "gen next"
    return self.gen.next()

    2) Define generators s.t. they yield the resource to be destroyed as
    their first value.

    def producer(resource):
    print "gen value"
    yield resource # yield resource before start iteration
    for item in resource:
    yield item

    3) Define the destructor for the resource. The resource must be passed
    as a first argument. The return value is a callable without arguments
    that serves as a close() function within FinalizerGenerator.__del__
    method.

    def close_resource(resource):
    return lambda: sys.stdout.write("close resource: %s"%resource)

    4) The finalize_with decorator

    def finalize_with(close_resource):
    def closing(func_gen):
    def fn(*args, **kwd):
    # fg serves as a replacement for the original generator
    func_def
    fg = FinalizerGenerator(func_gen)(*args, **kwd)
    # apply the closing protocol
    resource = fg.next()
    fg.close = close_resource(resource)
    return fg
    fn.__name__ = func_gen.__name__
    return fn
    return closing


    5) decorate the generator and check it out

    @finalize_with(close_resource)
    def producer(resource):
    print "gen value"
    yield resource
    for item in resource:
    yield item

    def test():
    pg = producer([1,2,3])
    pg.next()

    >>> test()

    gen init
    gen next # request for resource in finalize_with
    gen value
    gen next
    close resource: [1, 2, 3] # yep
    Kay Schluehr, Aug 12, 2007
    #15
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Martin Maurer
    Replies:
    3
    Views:
    4,837
    Peter
    Apr 19, 2006
  2. TheDustbustr
    Replies:
    1
    Views:
    448
    Sami Hangaslammi
    Jul 25, 2003
  3. Replies:
    4
    Views:
    266
    Victor Bazarov
    Feb 20, 2007
  4. Nathan Phillips
    Replies:
    5
    Views:
    264
    Chetan
    Mar 14, 2009
  5. Victor Bazarov
    Replies:
    11
    Views:
    748
    James Kanze
    Dec 9, 2009
Loading...

Share This Page