contextlib.nested()

[brasse]

Hello!

I have been running in to some problems when using
contextlib.nested(). My problem arises when using code similar to
this:

from __future__ import with_statement

from contextlib import nested

class Foo(object):

def __init__(self, tag, fail=False):
print 'ctor', tag
self.tag = tag
if fail:
raise Exception()

def __enter__(self):
print '__enter__', self.tag
return self

def __exit__(self, *args):
print '__exit__', self.tag

with nested(Foo('a'), Foo('b', True)) as (a, b):
print a.tag
print b.tag

Here the construction of b fails which in turn means that the
contextmanager fails to be created leaving me a constructed object (a)
that needs to be deconstructed in some way. I realize that nested() is
in a tight spot here to do anything about it since it doesn't exist.
This behavior makes it hard for me to use the with statement (using
nested()) the way I want.

Has anyone else been running in to this? Any tips on how to handle
multiple resources?

Regards,
Mattias

 

[Diez B. Roggisch]

Hello!

I have been running in to some problems when using
contextlib.nested(). My problem arises when using code similar to
this:

from __future__ import with_statement

from contextlib import nested

class Foo(object):

def __init__(self, tag, fail=False):
print 'ctor', tag
self.tag = tag
if fail:
raise Exception()

def __enter__(self):
print '__enter__', self.tag
return self

def __exit__(self, *args):
print '__exit__', self.tag

with nested(Foo('a'), Foo('b', True)) as (a, b):
print a.tag
print b.tag

Here the construction of b fails which in turn means that the
contextmanager fails to be created leaving me a constructed object (a)
that needs to be deconstructed in some way. I realize that nested() is
in a tight spot here to do anything about it since it doesn't exist.
This behavior makes it hard for me to use the with statement (using
nested()) the way I want.

Has anyone else been running in to this? Any tips on how to handle
multiple resources?

I don't fully understand this. Why is in need to be deconstructed? Sure, the
object is created, but whatever is actually done on initialization which is
non-trivial should of course to the __enter__-method - which isn't called.

So, a falls out of a scope & gets GC'ed. What else do you expect to happen?

Diez

 

[Robert Lehmann]

Hello!

I have been running in to some problems when using contextlib.nested().
My problem arises when using code similar to this:

from __future__ import with_statement

from contextlib import nested

class Foo(object):

def __init__(self, tag, fail=False):
print 'ctor', tag
self.tag = tag
if fail:
raise Exception()

def __enter__(self):
print '__enter__', self.tag
return self

def __exit__(self, *args):
print '__exit__', self.tag

with nested(Foo('a'), Foo('b', True)) as (a, b):
print a.tag
print b.tag

Here the construction of b fails which in turn means that the
contextmanager fails to be created leaving me a constructed object (a)
that needs to be deconstructed in some way. I realize that nested() is
in a tight spot here to do anything about it since it doesn't exist.
This behavior makes it hard for me to use the with statement (using
nested()) the way I want.

Has anyone else been running in to this? Any tips on how to handle
multiple resources?

Your problem does not seem to be connected to context managers. The error
occurs before calling `contextlib.nested` at all::

>>> foo = [Foo('a')] ctor a
>>> with nested(*foo) as a: print a

...
__enter__ a

>>> foo = [Foo('a'), Foo('b', True)]

ctor a
ctor b
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 7, in __init__
raise Exception()
Exception

If you need to deconstruct object `a` from your example, your staging is
probably broken. Allocate the resource in `__init__` but only go live
just in `__enter__`. If you do not enter the context, then, you won't
need to deconstruct it as well.

HTH,

 

[brasse]

Hello!

I have been running in to some problems when using contextlib.nested().
My problem arises when using code similar to this:

from __future__ import with_statement

from contextlib import nested

class Foo(object):

    def __init__(self, tag, fail=False):
        print 'ctor', tag
        self.tag = tag
        if fail:
            raise Exception()

    def __enter__(self):
        print '__enter__', self.tag
        return self

    def __exit__(self, *args):
        print '__exit__', self.tag

with nested(Foo('a'), Foo('b', True)) as (a, b):
    print a.tag
    print b.tag

Here the construction of b fails which in turn means that the
contextmanager fails to be created leaving me a constructed object (a)
that needs to be deconstructed in some way. I realize that nested() is
in a tight spot here to do anything about it since it doesn't exist.
This behavior makes it hard for me to use the with statement (using
nested()) the way I want.

Has anyone else been running in to this? Any tips on how to handle
multiple resources?

Your problem does not seem to be connected to context managers. The error
occurs before calling `contextlib.nested` at all::

   >>> foo = [Foo('a')]
   ctor a
   >>> with nested(*foo) as a: print a
   ...
   __enter__ a
   [<__main__.Foo object at 0x7fbc29408b90>]
   __exit__ a
   >>> foo = [Foo('a'), Foo('b', True)]
   ctor a
   ctor b
   Traceback (most recent call last):
     File "<stdin>", line 1, in <module>
     File "<stdin>", line 7, in __init__
       raise Exception()
   Exception

If you need to deconstruct object `a` from your example, your staging is
probably broken. Allocate the resource in `__init__` but only go live
just in `__enter__`. If you do not enter the context, then, you won't
need to deconstruct it as well.

HTH,

Diez, Robert,

OK. The practice of "going live" or doing non-trivial initialization
in __enter__ is new to me. I'm new to Python with a C++ background, so
that shouldn't be a surprise.

Ideally I would like to put all initialization in __init__ since then
I would be able to use my object right after constructing it, without
having to use it in a with statement. The reason I'm struggling with
this is probably my C++ background. I'm rally accustomed to design
with RAII in mind. Acquiring all resources in the ctor and releasing
all resources in the dtor is *really* handy.

If you had a class that wanted to acquire some external resources that
must be released at some point, how would you rewrite the code from my
example?

:.:: mattias

 

[Diez B. Roggisch]

Diez, Robert,

OK. The practice of "going live" or doing non-trivial initialization
in __enter__ is new to me. I'm new to Python with a C++ background, so
that shouldn't be a surprise.

Ideally I would like to put all initialization in __init__ since then
I would be able to use my object right after constructing it, without
having to use it in a with statement. The reason I'm struggling with
this is probably my C++ background. I'm rally accustomed to design
with RAII in mind. Acquiring all resources in the ctor and releasing
all resources in the dtor is *really* handy.

Yes, but this is a C++ idiom that does not translate well to python's
GC-based approach. Which is the *exact* reason why contexts have been
created in the first place.

If you had a class that wanted to acquire some external resources that
must be released at some point, how would you rewrite the code from my
example?

If you *can*, use a context. Use __enter__ and __exit__. Try really hard to
use it that way.

If not - create a specific finalize-method or some such, and try not to
forget to call that. Potentially with an atexit-handler or some such.

the problem is that python can't guarantee that a __del__-method is invoked
at all, and *if* it is, it might find other stuff being released already
that it relies upon - e.g. imported modules being freed & not known
anymore.

Diez

 

[brasse]

If you *can*, use a context. Use __enter__ and __exit__. Try really hard to
use it that way.

My case becomes something like this:

from __future__ import with_statement

from contextlib import nested

class Foo(object):

def __init__(self, tag, fail=False):
print 'ctor', tag
self.tag = tag
self.fail = fail

def __enter__(self):
if self.fail:
print 'fail', self.tag
raise Exception()
print '__enter__ acquire resource', self.tag
return self

def __exit__(self, *args):
print '__exit__ release resource', self.tag

with nested(Foo('a'), Foo('b', True)) as (a, b):
print a.tag
print b.tag

When using Foo objects in a with statement this works good for me. But
what if I want to use Foo objects as members in a class for example?
Since we now must contruct an instance of Foo in two stages the code
becomes less than ideal.

def __init__(self):
self.x = Foo()
self.x.__enter__()

Perhaps there is no way to write classes that fits neatly into all (or
even these two) usage scenarios?

If not - create a specific finalize-method or some such, and try not to
forget to call that. Potentially with an atexit-handler or some such.

It seems to me that I have to use the with statement (or some try-
finally construct) to be able to release all resources when my code
throws exceptions(). Just remembering to call close/finalize/destroy
will not be enough.

:.:: mattias

 

[Peter Otten]

with nested(Foo('a'), Foo('b', True)) as (a, b):
    print a.tag
    print b.tag

If been watching this thread for a while, and I think that your problems
will go away if you write actual nested with-blocks:

with Foo("a") as a:
with Foo("b") as b:
print a.tag
print b.tag

Why look for a complex solution if there is a simple one?

Peter

 

[brasse]

If been watching this thread for a while, and I think that your problems
will go away if you write actual nested with-blocks:

with Foo("a") as a:
    with Foo("b") as b:
        print a.tag
        print b.tag

Why look for a complex solution if there is a simple one?

That works great if you are only working with two objects. It gets a
bit uglier when you need to use three or more objects. I'm just trying
to figure out if there is some kind of best practice in the Python
community that works well (even with more than two objects) for the
two usage scenarios I have described.

:.:: mattias