File not closed on exception

A

arve.knudsen

Hi

I thought that file objects were supposed to be garbage-collected and
automatically closed once they go out of scope, at least that's what
I've been told by more merited Python programmers. I'm also quite sure
that this is quite a common assumption in various programs, at least
given what opensource code I've seen in my time. However, the
following script doesn't work on Windows, since the file is still open
when I try to remove it:

import os.path

def create():
f = file("tmp", "w")
raise Exception

try: create()
finally:
os.remove("tmp")


So, what's the deal exactly, is the file supposed to be garbage-
collected (and closed) at the end of create?

Thanks!
Arve
 
E

Ethan Furman

Hi

I thought that file objects were supposed to be garbage-collected and
automatically closed once they go out of scope, at least that's what
I've been told by more merited Python programmers. I'm also quite sure
that this is quite a common assumption in various programs, at least
given what opensource code I've seen in my time. However, the
following script doesn't work on Windows, since the file is still open
when I try to remove it:

import os.path

def create():
f = file("tmp", "w")
raise Exception

try: create()
finally:
os.remove("tmp")


So, what's the deal exactly, is the file supposed to be garbage-
collected (and closed) at the end of create?

Thanks!
Arve

When an exception is raised, the entire stack frame at that location
(which includes local vars) is saved in the exception traceback. Since
the objects are still alive, they are not GC'ed. That is why this is
better:

def create():
f = file("tmp", "w")
try:
do_stuff_that_raises_exception
finally:
os.remove("tmp")

~Ethan~
 
G

Grant Edwards

I thought that file objects were supposed to be
garbage-collected and automatically closed once they go out of
scope,

At some point after they go out of scope, they will be.
Eventually. Exactly when is an implementation detail.
at least that's what I've been told by more merited Python
programmers. I'm also quite sure that this is quite a common
assumption in various programs,

If your program relies on the assumption that some particular
object will be garbage-collected between points A and B, then
that's a bug in your program. If you depend on the fact that
some object has been delted, then "del" it. If you depend on
the fact that a file is closed, then close it.
at least given what opensource code I've seen in my time.
However, the following script doesn't work on Windows, since
the file is still open when I try to remove it:

import os.path

def create():
f = file("tmp", "w")
raise Exception

try: create()
finally:
os.remove("tmp")

So, what's the deal exactly, is the file supposed to be garbage-
collected (and closed) at the end of create?

Nothing is "supposed" to be garbage-collected. An object _may_
be garbage collected after some point.
 
G

Gabriel Genellina

En Mon, 19 Oct 2009 09:45:49 -0200, (e-mail address removed)
I thought that file objects were supposed to be garbage-collected and
automatically closed once they go out of scope, at least that's what
I've been told by more merited Python programmers.

An object (any object) is destroyed as soon as the last reference to the
it is removed. A local variable holds a reference to the file object; it
that is the ONLY reference, the file object will be destroyed when the
variable goes out of scope, yes.
Note that:
- there might be more references to the object
- garbage collection is a separate subject; objects are reference-counted,
zero=>kaputt, the GC has no say on this. GC is only used to break cycles
(a->b, b->a) that would prevent the objects to reach 0 references.
- this behavior is specific of CPython
I'm also quite sure
that this is quite a common assumption in various programs, at least
given what opensource code I've seen in my time.

When an object holds references to external resources that must be freed,
this is not a good idea. Being explicit with the resource deallocation is
much better than relying on object destruction sometime in the future...
However, the
following script doesn't work on Windows, since the file is still open
when I try to remove it:

import os.path

def create():
f = file("tmp", "w")
raise Exception

try: create()
finally:
os.remove("tmp")


So, what's the deal exactly, is the file supposed to be garbage-
collected (and closed) at the end of create?

The object does not go out of scope because there is an additional
reference: the exception traceback holds a reference to all execution
frames, and each frame holds a reference to its local variables. So "f"
is still alive. This is quite good for a debugger, or for logging
purposes, as one can inspect the values of each and every variable along
the frame chain. But until exception processing is finished, the "f"
variable is alive and the "tmp" file is open.

How to deal with this depends on your use case. I don't know what can I
modify on your small example and still being representative of your actual
problem. The reccomended way to process a file uses a with statement:

def create():
with open("tmp", "w") as f:
# do something with the file
raise Exception

This way the file is closed when leaving the with statement (either
normally or because of an exception). And, if the file doesn't exist or
access is denied, the open() call doesn't success either. In any case, it
never remains open.
 
A

arve.knudsen

When an exception is raised, the entire stack frame at that location
(which includes local vars) is saved in the exception traceback.  Since
the objects are still alive, they are not GC'ed.  That is why this is
better:

def create():
     f = file("tmp", "w")
     try:
         do_stuff_that_raises_exception
     finally:
         os.remove("tmp")

~Ethan~

Why should this work? If I replace "do_stuff_that_raises_exception"
with "raise Exception", it fails in the same way, since the file is
open. Maybe you forgot "f.close()"? In any case, thanks for explaining
that the traceback keeps the object alive, that explains the issue.

Arve
 
A

arve.knudsen

At some point after they go out of scope, they will be.
Eventually.  Exactly when is an implementation detail.


If your program relies on the assumption that some particular
object will be garbage-collected between points A and B, then
that's a bug in your program.  If you depend on the fact that
some object has been delted, then "del" it.  If you depend on
the fact that a file is closed, then close it.

Personally I am against that assumption, and prefer a utility function
which reads the file and automatically closes it in a "finally" block
(in lieu of the "with" keyword). However, when providing a patch for a
high-profile opensource Python project I was scolded for going to such
lengths, as the prescribed style was to just open files and let them
be closed implicitly.

Also, the problem may arise when I call a function in a 3rd party
library, that it opens files which I then can't delete upon an
exception from within said function. Actually, something like that did
happen and spurred my original question, but fortunately a reference
to the file was kept in the 3rd party object I was operating on, so I
was able to free it in a "finally" block.

Arve
 
A

arve.knudsen

En Mon, 19 Oct 2009 09:45:49 -0200, (e-mail address removed)  


An object (any object) is destroyed as soon as the last reference to the  
it is removed. A local variable holds a reference to the file object; it  
that is the ONLY reference, the file object will be destroyed when the  
variable goes out of scope, yes.
Note that:
- there might be more references to the object
- garbage collection is a separate subject; objects are reference-counted,  
zero=>kaputt, the GC has no say on this. GC is only used to break cycles  
(a->b, b->a) that would prevent the objects to reach 0 references.
- this behavior is specific of CPython


When an object holds references to external resources that must be freed,  
this is not a good idea. Being explicit with the resource deallocation is  
much better than relying on object destruction sometime in the future...

I agree, but like I said, I've been told that this (implicit closing
of files) is the correct style by more merited Python developers, so
that made me think I was probably wrong ..

Arve
 
G

Gabriel Genellina

En Tue, 20 Oct 2009 03:23:49 -0300, (e-mail address removed)
I agree, but like I said, I've been told that this (implicit closing
of files) is the correct style by more merited Python developers, so
that made me think I was probably wrong ..

Then tell those "more merited Python developers" that they're wrong, and
that the right way to ensure a file is closed when you're done with it is
to use a `with` statement (or a try/finally block in old Python releases)
 
A

arve.knudsen

En Tue, 20 Oct 2009 03:23:49 -0300, (e-mail address removed)  
<[email protected]> escribió:







Then tell those "more merited Python developers" that they're wrong, and  
that the right way to ensure a file is closed when you're done with it is  
to use a `with` statement (or a try/finally block in old Python releases)

Easier said than done :) In any case, I now have this discussion as a
useful reference in the future. Thanks!

Arve
 
U

Ulrich Eckhardt

(e-mail address removed) wrote: [...]
def create():
f = file("tmp", "w")
raise Exception

try:
create()
finally:
os.remove("tmp")
[...]
When an exception is raised, the entire stack frame at that location
(which includes local vars) is saved in the exception traceback.  [...]
this is better:

def create():
f = file("tmp", "w")
try:
do_stuff_that_raises_exception
finally:
os.remove("tmp")
[...]
Why should this work? If I replace "do_stuff_that_raises_exception"
with "raise Exception", it fails in the same way, since the file is
open. Maybe you forgot "f.close()"?

I was puzzled by the same, but too lazy to try or ask. Anyhow, I think that
if you replaced the 'os.remove("tmp")' with 'f.close()', then the calling
function can remain the same as you wrote. This is basically the same as
when using the new-style "with", as mentioned by Gabriel.

Uli
 
M

Mel

I agree, but like I said, I've been told that this (implicit closing
of files) is the correct style by more merited Python developers, so
that made me think I was probably wrong ..

It would be nice. The trouble is that CPython is not the only Python.
Jython, for example, uses the Java Runtime Environment for its virtual
machine. JRE doesn't have reference-counts, so Jython can't close files
immediately after the last reference ends. It seems guaranteed object
cleanup would lock Python out of too many possible platforms.

Mel.
 
E

Ethan Furman

Why should this work? If I replace "do_stuff_that_raises_exception"
with "raise Exception", it fails in the same way, since the file is
open. Maybe you forgot "f.close()"? In any case, thanks for explaining
that the traceback keeps the object alive, that explains the issue.

Arve

Indeed. That should have been f.close() in the finally block. My
apologies.

~Ethan~
 
B

Bruno Desthuilliers

(e-mail address removed) a écrit :
AFAICT, the file descriptor associated to the file object will be freed
when the CPython process will finish - which implies you're using the
CPython interpreter. This doesn't mean the file is garanteed to be
closed _at the point where you're trying to os.remove() it_. And as I
said, this is a CPython implementation detail - not a language
specification. Jython or IronPython (or any other implementation) may
not work that way.
Personally I am against that assumption, and prefer a utility function
which reads the file and automatically closes it in a "finally" block
(in lieu of the "with" keyword).

What's your problem with the with ???

But anyway : explicitely releasing resources such as files, network
connections etc is of course the RightThing(tm), except eventually in
one-shot throwaway scripts.
However, when providing a patch for a
high-profile opensource Python project I was scolded for going to such
lengths, as the prescribed style was to just open files and let them
be closed implicitly.

Err... Care to name the project ? I hope it's not one I ever advertized :(
 
A

arve.knudsen

What's your problem with the with ???

No problem whatsoever, but I believe I wrote this utility function
before the keyword was available, and it might be good to support
older Python versions.
But anyway : explicitely releasing resources such as files, network
connections etc is of course the RightThing(tm), except eventually in
one-shot throwaway scripts.


Err... Care to name the project ? I hope it's not one I ever advertized :(

I'm not going to name the project, but it is incidentally used by the
Python project itself :)

Arve
 
G

Gabriel Genellina

En Tue, 20 Oct 2009 04:47:02 -0300, (e-mail address removed)
Easier said than done :) In any case, I now have this discussion as a
useful reference in the future. Thanks!

If this thread is not enough, you can ask them to read the official Python
tutorial:

"It is good practice to use the with keyword when dealing with file
objects. This has the advantage that the file is properly closed after its
suite finishes, even if an exception is raised on the way. It is also much
shorter than writing equivalent try-finally blocks."

http://docs.python.org/tutorial/inputoutput.html#methods-of-file-objects
 
A

arve.knudsen

En Tue, 20 Oct 2009 04:47:02 -0300, (e-mail address removed)  



If this thread is not enough, you can ask them to read the official Python  
tutorial:

"It is good practice to use the with keyword when dealing with file  
objects. This has the advantage that the file is properly closed after its  
suite finishes, even if an exception is raised on the way. It is also much  
shorter than writing equivalent try-finally blocks."

http://docs.python.org/tutorial/inputoutput.html#methods-of-file-objects

Perhaps the general attitude has changed now that the "with" keyword
makes it so easy anyway (unless one needs to support older Pythons of
course).

Arve
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,482
Members
44,901
Latest member
Noble71S45

Latest Threads

Top