contextlib.contextmanager and try/finally

[johannh]

I'm trying to write a context manager to handle database connections, under the principle that I should not rely on CPython's reference-counting semantics to clean up scarce resources, like connections.

I wrote:

@contexlib.contextmanager
def ensure_connection(con=None):
con_created = False
if con is None:
con_created, con = True, make_connection()
try:
yield con
finally:
if con_created:
con.close()

However, then I read the following paragraph from PEP-343:

Note that we're not guaranteeing that the finally-clause is
executed immediately after the generator object becomes unused,
even though this is how it will work in CPython. This is similar
to auto-closing files: while a reference-counting implementation
like CPython deallocates an object as soon as the last reference
to it goes away, implementations that use other GC algorithms do
not make the same guarantee. This applies to Jython, IronPython,
and probably to Python running on Parrot.

That suggests that I cannot rely on the contextlib.contextmanager decorator to ensure that the connection is closed and would have to write my own object with __enter__ and __exit__ methods to guarantee this.

Is this understanding accurate? If so, could someone illustrate why this is so?

Thanks,
Johann

 

[Neil Cerutti]

That suggests that I cannot rely on the
contextlib.contextmanager decorator to ensure that the
connection is closed and would have to write my own object with
__enter__ and __exit__ methods to guarantee this.

contextmanager wraps your generator in an object with __enter__
and __exit__ methods for you.

 

[Robert Kern]

I'm trying to write a context manager to handle database connections, under the principle that I should not rely on CPython's reference-counting semantics to clean up scarce resources, like connections.

I wrote:

@contexlib.contextmanager
def ensure_connection(con=None):
con_created = False
if con is None:
con_created, con = True, make_connection()
try:
yield con
finally:
if con_created:
con.close()

However, then I read the following paragraph from PEP-343:

Note that we're not guaranteeing that the finally-clause is
executed immediately after the generator object becomes unused,
even though this is how it will work in CPython. This is similar
to auto-closing files: while a reference-counting implementation
like CPython deallocates an object as soon as the last reference
to it goes away, implementations that use other GC algorithms do
not make the same guarantee. This applies to Jython, IronPython,
and probably to Python running on Parrot.

That suggests that I cannot rely on the contextlib.contextmanager decorator to ensure that the connection is closed and would have to write my own object with __enter__ and __exit__ methods to guarantee this.

Is this understanding accurate? If so, could someone illustrate why this is so?

Looking at the paragraph before this one, it appears that the PEP is talking
about the .close() method on generators, which is really just a general purpose
API for closing generators that might not be exhausted yet. It's not really
related to the context manager stuff except that it came up during the design
process of the context manager along with the related .send() and .throw() methods.

__enter__() will call .next() once to execute the code up to the yield
statement. Then __exit__() will call .next() once again to execute the code
after the yield statement, including the finally: clause. That's the only thing
you need to rely on. Your connection-closing code will be called if __exit__()
gets called. That will exhaust your generator, so the .close() method will not
really do anything helpful or hurtful in such a case.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 

[Ian Kelly]

However, then I read the following paragraph from PEP-343:

   Note that we're not guaranteeing that the finally-clause is
   executed immediately after the generator object becomes unused,
   even though this is how it will work in CPython.  This is similar
   to auto-closing files: while a reference-counting implementation
   like CPython deallocates an object as soon as the last reference
   to it goes away, implementations that use other GC algorithms do
   not make the same guarantee.  This applies to Jython, IronPython,
   and probably to Python running on Parrot.

That suggests that I cannot rely on the contextlib.contextmanager decorator to ensure that the connection is closed and would have to write my own object with __enter__ and __exit__ methods to guarantee this.

Is this understanding accurate?  If so, could someone illustrate why this is so?

First, this is just a timing issue. There is no guarantee that the
finally clause will be executed immediately, but it is guaranteed to
be executed at some point. In the other implementations, it would
happen whenever the GC algorithm runs.

Second, I believe that passage is not referring to the contextmanager
decorator specifically, but more generally to the changes that were
made to allow generators to yield from within a try-finally construct
(previously this would have been illegal syntax, since there was no
way to guarantee the finally block would be performed).

Like Neil mentioned, a contextmanager generator is wrapped with an
__exit__ method that is guaranteed to be called and that explicitly
resumes or closes the generator. So as long as your contextmanager
generator is properly written (i.e. it yields exactly once), the
finally block will execute in a timely fashion.

 

[johannh]

Second, I believe that passage is not referring to the contextmanager
decorator specifically, but more generally to the changes that were
made to allow generators to yield from within a try-finally construct
(previously this would have been illegal syntax, since there was no
way to guarantee the finally block would be performed).

Thanks. You and Robert have shown me the light. That paragraph was talking about the finalization that happens to a suspended generator when it is GC'd. Meanwhile, the contextmanager decorator ensures that its generator isrun all the way through, so it would never be left in a suspended state, so its GC/finalization semantics are irrelevant. (Please correct, if wrong.)

(Neil: I understood that it created a context manager object, but I had been reading the quoted paragraph to mean that by going through a generator the decorator was re-introducing a dependency on GC time. But I was wrong.)

Regards,
Johann

 

[johannh]

Second, I believe that passage is not referring to the contextmanager
decorator specifically, but more generally to the changes that were
made to allow generators to yield from within a try-finally construct
(previously this would have been illegal syntax, since there was no
way to guarantee the finally block would be performed).

Thanks. You and Robert have shown me the light. That paragraph was talking about the finalization that happens to a suspended generator when it is GC'd. Meanwhile, the contextmanager decorator ensures that its generator isrun all the way through, so it would never be left in a suspended state, so its GC/finalization semantics are irrelevant. (Please correct, if wrong.)

(Neil: I understood that it created a context manager object, but I had been reading the quoted paragraph to mean that by going through a generator the decorator was re-introducing a dependency on GC time. But I was wrong.)

Regards,
Johann

 

[Prasad, Ramit]

Like Neil mentioned, a contextmanager generator is wrapped with an

__exit__ method that is guaranteed to be called and that explicitly
resumes or closes the generator. So as long as your contextmanager
generator is properly written (i.e. it yields exactly once), the
finally block will execute in a timely fashion.

Is that true even in the face of something like sys.exit()?
What happens if 1) sys.exit is called while in the same thread
2) sys.exit is called from another thread but while this thread
is in context manager?

Ramit

Ramit Prasad | JPMorgan Chase Investment Bank | Currencies Technology
712 Main Street | Houston, TX 77002
work phone: 713 - 216 - 5423

--

This email is confidential and subject to important disclaimers and
conditions including on offers for the purchase or sale of
securities, accuracy and completeness of information, viruses,
confidentiality, legal privilege, and legal entity disclaimers,
available at http://www.jpmorgan.com/pages/disclosures/email.

 

[Peter Otten]

Is that true even in the face of something like sys.exit()?
What happens if 1) sys.exit is called while in the same thread
2) sys.exit is called from another thread but while this thread
is in context manager?

sys.exit() uses the normal exception mechanism to terminate a program:
.... sys.exit()
.... except SystemExit:
.... print "I'd rather not"
....
I'd rather not
It won't derail a context manager:
.... def f():
.... try:
.... yield
.... finally: print "bye"
........ sys.exit()
....
bye
$

 

[Ian Kelly]

Is that true even in the face of something like sys.exit()?
Yes.

What happens if 1) sys.exit is called while in the same thread

Why don't you try it and find out? To answer the question, though,
sys.exit() raises a SystemExit exception, which propagates out of the
with block and calls the __exit__ method, which then throws the
exception to the generator, which executes its finally clause and
exits. The __exit__ method returns false, so the SystemExit exception
continues to propagate, and if it is not caught, then the process
exits.

2) sys.exit is called from another thread but while this thread
is in context manager?

Then the other thread raises SystemExit, and the current thread is
unaffected. sys.exit only affects the thread it is called in.

You can certainly come up with scenarios in which the finally clause
does not execute, e.g. killing the interpreter with "kill -9" or
yanking out the power cord. Within the confines of the Python
interpreter, though, it is guaranteed that the finally block will
execute.

 

[Steven D'Aprano]

Is that true even in the face of something like sys.exit()?
Yes.

[...]
You can certainly come up with scenarios in which the finally clause
does not execute, e.g. killing the interpreter with "kill -9" or yanking
out the power cord. Within the confines of the Python interpreter,
though, it is guaranteed that the finally block will execute.

Almost, but not quite.

The finally block is not guaranteed to execute if the try block never
exits -- infinite loops are still infinite loops.

Also, unlike sys.exit, os._exit doesn't work through the exception
mechanism, can't be caught, and simply exits immediately.

.... os._exit(1)
.... finally:
.... print "exiting"
....
steve@runes:~$