File not closed on exception

Discussion in 'Python' started by arve.knudsen@gmail.com, Oct 19, 2009.

  1. Guest

    Hi

    I thought that file objects were supposed to be garbage-collected and
    automatically closed once they go out of scope, at least that's what
    I've been told by more merited Python programmers. I'm also quite sure
    that this is quite a common assumption in various programs, at least
    given what opensource code I've seen in my time. However, the
    following script doesn't work on Windows, since the file is still open
    when I try to remove it:

    import os.path

    def create():
    f = file("tmp", "w")
    raise Exception

    try: create()
    finally:
    os.remove("tmp")


    So, what's the deal exactly, is the file supposed to be garbage-
    collected (and closed) at the end of create?

    Thanks!
    Arve
     
    , Oct 19, 2009
    #1
    1. Advertising

  2. Ethan Furman Guest

    wrote:
    > Hi
    >
    > I thought that file objects were supposed to be garbage-collected and
    > automatically closed once they go out of scope, at least that's what
    > I've been told by more merited Python programmers. I'm also quite sure
    > that this is quite a common assumption in various programs, at least
    > given what opensource code I've seen in my time. However, the
    > following script doesn't work on Windows, since the file is still open
    > when I try to remove it:
    >
    > import os.path
    >
    > def create():
    > f = file("tmp", "w")
    > raise Exception
    >
    > try: create()
    > finally:
    > os.remove("tmp")
    >
    >
    > So, what's the deal exactly, is the file supposed to be garbage-
    > collected (and closed) at the end of create?
    >
    > Thanks!
    > Arve


    When an exception is raised, the entire stack frame at that location
    (which includes local vars) is saved in the exception traceback. Since
    the objects are still alive, they are not GC'ed. That is why this is
    better:

    def create():
    f = file("tmp", "w")
    try:
    do_stuff_that_raises_exception
    finally:
    os.remove("tmp")

    ~Ethan~
     
    Ethan Furman, Oct 19, 2009
    #2
    1. Advertising

  3. On 2009-10-19, <> wrote:

    > I thought that file objects were supposed to be
    > garbage-collected and automatically closed once they go out of
    > scope,


    At some point after they go out of scope, they will be.
    Eventually. Exactly when is an implementation detail.

    > at least that's what I've been told by more merited Python
    > programmers. I'm also quite sure that this is quite a common
    > assumption in various programs,


    If your program relies on the assumption that some particular
    object will be garbage-collected between points A and B, then
    that's a bug in your program. If you depend on the fact that
    some object has been delted, then "del" it. If you depend on
    the fact that a file is closed, then close it.

    > at least given what opensource code I've seen in my time.
    > However, the following script doesn't work on Windows, since
    > the file is still open when I try to remove it:
    >
    > import os.path
    >
    > def create():
    > f = file("tmp", "w")
    > raise Exception
    >
    > try: create()
    > finally:
    > os.remove("tmp")
    >
    > So, what's the deal exactly, is the file supposed to be garbage-
    > collected (and closed) at the end of create?


    Nothing is "supposed" to be garbage-collected. An object _may_
    be garbage collected after some point.

    --
    Grant
     
    Grant Edwards, Oct 19, 2009
    #3
  4. En Mon, 19 Oct 2009 09:45:49 -0200,
    <> escribió:

    > I thought that file objects were supposed to be garbage-collected and
    > automatically closed once they go out of scope, at least that's what
    > I've been told by more merited Python programmers.


    An object (any object) is destroyed as soon as the last reference to the
    it is removed. A local variable holds a reference to the file object; it
    that is the ONLY reference, the file object will be destroyed when the
    variable goes out of scope, yes.
    Note that:
    - there might be more references to the object
    - garbage collection is a separate subject; objects are reference-counted,
    zero=>kaputt, the GC has no say on this. GC is only used to break cycles
    (a->b, b->a) that would prevent the objects to reach 0 references.
    - this behavior is specific of CPython

    > I'm also quite sure
    > that this is quite a common assumption in various programs, at least
    > given what opensource code I've seen in my time.


    When an object holds references to external resources that must be freed,
    this is not a good idea. Being explicit with the resource deallocation is
    much better than relying on object destruction sometime in the future...

    > However, the
    > following script doesn't work on Windows, since the file is still open
    > when I try to remove it:
    >
    > import os.path
    >
    > def create():
    > f = file("tmp", "w")
    > raise Exception
    >
    > try: create()
    > finally:
    > os.remove("tmp")
    >
    >
    > So, what's the deal exactly, is the file supposed to be garbage-
    > collected (and closed) at the end of create?


    The object does not go out of scope because there is an additional
    reference: the exception traceback holds a reference to all execution
    frames, and each frame holds a reference to its local variables. So "f"
    is still alive. This is quite good for a debugger, or for logging
    purposes, as one can inspect the values of each and every variable along
    the frame chain. But until exception processing is finished, the "f"
    variable is alive and the "tmp" file is open.

    How to deal with this depends on your use case. I don't know what can I
    modify on your small example and still being representative of your actual
    problem. The reccomended way to process a file uses a with statement:

    def create():
    with open("tmp", "w") as f:
    # do something with the file
    raise Exception

    This way the file is closed when leaving the with statement (either
    normally or because of an exception). And, if the file doesn't exist or
    access is denied, the open() call doesn't success either. In any case, it
    never remains open.

    --
    Gabriel Genellina
     
    Gabriel Genellina, Oct 19, 2009
    #4
  5. Guest

    On Oct 19, 3:48 pm, Ethan Furman <> wrote:
    > wrote:
    > > Hi

    >
    > > I thought that file objects were supposed to be garbage-collected and
    > > automatically closed once they go out of scope, at least that's what
    > > I've been told by more merited Python programmers. I'm also quite sure
    > > that this is quite a common assumption in various programs, at least
    > > given what opensource code I've seen in my time. However, the
    > > following script doesn't work on Windows, since the file is still open
    > > when I try to remove it:

    >
    > > import os.path

    >
    > > def create():
    > >     f = file("tmp", "w")
    > >     raise Exception

    >
    > > try: create()
    > > finally:
    > >     os.remove("tmp")

    >
    > > So, what's the deal exactly, is the file supposed to be garbage-
    > > collected (and closed) at the end of create?

    >
    > > Thanks!
    > > Arve

    >
    > When an exception is raised, the entire stack frame at that location
    > (which includes local vars) is saved in the exception traceback.  Since
    > the objects are still alive, they are not GC'ed.  That is why this is
    > better:
    >
    > def create():
    >      f = file("tmp", "w")
    >      try:
    >          do_stuff_that_raises_exception
    >      finally:
    >          os.remove("tmp")
    >
    > ~Ethan~


    Why should this work? If I replace "do_stuff_that_raises_exception"
    with "raise Exception", it fails in the same way, since the file is
    open. Maybe you forgot "f.close()"? In any case, thanks for explaining
    that the traceback keeps the object alive, that explains the issue.

    Arve
     
    , Oct 20, 2009
    #5
  6. Guest

    On Oct 19, 4:14 pm, Grant Edwards <> wrote:
    > On 2009-10-19, <> wrote:
    >
    > > I thought that file objects were supposed to be
    > > garbage-collected and automatically closed once they go out of
    > > scope,

    >
    > At some point after they go out of scope, they will be.
    > Eventually.  Exactly when is an implementation detail.
    >
    > > at least that's what I've been told by more merited Python
    > > programmers. I'm also quite sure that this is quite a common
    > > assumption in various programs,

    >
    > If your program relies on the assumption that some particular
    > object will be garbage-collected between points A and B, then
    > that's a bug in your program.  If you depend on the fact that
    > some object has been delted, then "del" it.  If you depend on
    > the fact that a file is closed, then close it.


    Personally I am against that assumption, and prefer a utility function
    which reads the file and automatically closes it in a "finally" block
    (in lieu of the "with" keyword). However, when providing a patch for a
    high-profile opensource Python project I was scolded for going to such
    lengths, as the prescribed style was to just open files and let them
    be closed implicitly.

    Also, the problem may arise when I call a function in a 3rd party
    library, that it opens files which I then can't delete upon an
    exception from within said function. Actually, something like that did
    happen and spurred my original question, but fortunately a reference
    to the file was kept in the 3rd party object I was operating on, so I
    was able to free it in a "finally" block.

    Arve
     
    , Oct 20, 2009
    #6
  7. Guest

    On Oct 19, 5:56 pm, "Gabriel Genellina" <>
    wrote:
    > En Mon, 19 Oct 2009 09:45:49 -0200,  
    > <> escribió:
    >
    > > I thought that file objects were supposed to be garbage-collected and
    > > automatically closed once they go out of scope, at least that's what
    > > I've been told by more merited Python programmers.

    >
    > An object (any object) is destroyed as soon as the last reference to the  
    > it is removed. A local variable holds a reference to the file object; it  
    > that is the ONLY reference, the file object will be destroyed when the  
    > variable goes out of scope, yes.
    > Note that:
    > - there might be more references to the object
    > - garbage collection is a separate subject; objects are reference-counted,  
    > zero=>kaputt, the GC has no say on this. GC is only used to break cycles  
    > (a->b, b->a) that would prevent the objects to reach 0 references.
    > - this behavior is specific of CPython
    >
    > > I'm also quite sure
    > > that this is quite a common assumption in various programs, at least
    > > given what opensource code I've seen in my time.

    >
    > When an object holds references to external resources that must be freed,  
    > this is not a good idea. Being explicit with the resource deallocation is  
    > much better than relying on object destruction sometime in the future...


    I agree, but like I said, I've been told that this (implicit closing
    of files) is the correct style by more merited Python developers, so
    that made me think I was probably wrong ..

    Arve
     
    , Oct 20, 2009
    #7
  8. En Tue, 20 Oct 2009 03:23:49 -0300,
    <> escribió:
    > On Oct 19, 5:56 pm, "Gabriel Genellina" <>
    > wrote:
    >> En Mon, 19 Oct 2009 09:45:49 -0200,  
    >> <> escribió:
    >>
    >> > I thought that file objects were supposed to be garbage-collected and
    >> > automatically closed once they go out of scope, at least that's what
    >> > I've been told by more merited Python programmers.

    >>
    >> When an object holds references to external resources that must be
    >> freed,  
    >> this is not a good idea. Being explicit with the resource deallocation
    >> is  
    >> much better than relying on object destruction sometime in the future...

    >
    > I agree, but like I said, I've been told that this (implicit closing
    > of files) is the correct style by more merited Python developers, so
    > that made me think I was probably wrong ..


    Then tell those "more merited Python developers" that they're wrong, and
    that the right way to ensure a file is closed when you're done with it is
    to use a `with` statement (or a try/finally block in old Python releases)

    --
    Gabriel Genellina
     
    Gabriel Genellina, Oct 20, 2009
    #8
  9. Guest

    On 20 Okt, 09:40, "Gabriel Genellina" <> wrote:
    > En Tue, 20 Oct 2009 03:23:49 -0300,  
    > <> escribió:
    >
    >
    >
    >
    >
    > > On Oct 19, 5:56 pm, "Gabriel Genellina" <>
    > > wrote:
    > >> En Mon, 19 Oct 2009 09:45:49 -0200,  
    > >> <> escribió:

    >
    > >> > I thought that file objects were supposed to be garbage-collected and
    > >> > automatically closed once they go out of scope, at least that's what
    > >> > I've been told by more merited Python programmers.

    >
    > >> When an object holds references to external resources that must be  
    > >> freed,  
    > >> this is not a good idea. Being explicit with the resource deallocation  
    > >> is  
    > >> much better than relying on object destruction sometime in the future....

    >
    > > I agree, but like I said, I've been told that this (implicit closing
    > > of files) is the correct style by more merited Python developers, so
    > > that made me think I was probably wrong ..

    >
    > Then tell those "more merited Python developers" that they're wrong, and  
    > that the right way to ensure a file is closed when you're done with it is  
    > to use a `with` statement (or a try/finally block in old Python releases)


    Easier said than done :) In any case, I now have this discussion as a
    useful reference in the future. Thanks!

    Arve
     
    , Oct 20, 2009
    #9
  10. wrote:
    > On Oct 19, 3:48 pm, Ethan Furman <> wrote:
    >> wrote:

    [...]
    >>> def create():
    >>> f = file("tmp", "w")
    >>> raise Exception
    >>>
    >>> try:
    >>> create()
    >>> finally:
    >>> os.remove("tmp")
    >>>

    [...]
    >> When an exception is raised, the entire stack frame at that location
    >> (which includes local vars) is saved in the exception traceback.  [...]
    >> this is better:
    >>
    >> def create():
    >> f = file("tmp", "w")
    >> try:
    >> do_stuff_that_raises_exception
    >> finally:
    >> os.remove("tmp")

    [...]
    > Why should this work? If I replace "do_stuff_that_raises_exception"
    > with "raise Exception", it fails in the same way, since the file is
    > open. Maybe you forgot "f.close()"?


    I was puzzled by the same, but too lazy to try or ask. Anyhow, I think that
    if you replaced the 'os.remove("tmp")' with 'f.close()', then the calling
    function can remain the same as you wrote. This is basically the same as
    when using the new-style "with", as mentioned by Gabriel.

    Uli

    --
    Sator Laser GmbH
    Geschäftsführer: Thorsten Föcking, Amtsgericht Hamburg HR B62 932
     
    Ulrich Eckhardt, Oct 20, 2009
    #10
  11. Mel Guest

    wrote:

    > I agree, but like I said, I've been told that this (implicit closing
    > of files) is the correct style by more merited Python developers, so
    > that made me think I was probably wrong ..


    It would be nice. The trouble is that CPython is not the only Python.
    Jython, for example, uses the Java Runtime Environment for its virtual
    machine. JRE doesn't have reference-counts, so Jython can't close files
    immediately after the last reference ends. It seems guaranteed object
    cleanup would lock Python out of too many possible platforms.

    Mel.
     
    Mel, Oct 20, 2009
    #11
  12. Ethan Furman Guest

    wrote:
    > On Oct 19, 3:48 pm, Ethan Furman <> wrote:
    >
    >> wrote:
    >>
    >>>Hi

    >>
    >>>I thought that file objects were supposed to be garbage-collected and
    >>>automatically closed once they go out of scope, at least that's what
    >>>I've been told by more merited Python programmers. I'm also quite sure
    >>>that this is quite a common assumption in various programs, at least
    >>>given what opensource code I've seen in my time. However, the
    >>>following script doesn't work on Windows, since the file is still open
    >>>when I try to remove it:

    >>
    >>>import os.path

    >>
    >>>def create():
    >>> f = file("tmp", "w")
    >>> raise Exception

    >>
    >>>try: create()
    >>>finally:
    >>> os.remove("tmp")

    >>
    >>>So, what's the deal exactly, is the file supposed to be garbage-
    >>>collected (and closed) at the end of create?

    >>
    >>>Thanks!
    >>>Arve

    >>
    >>When an exception is raised, the entire stack frame at that location
    >>(which includes local vars) is saved in the exception traceback. Since
    >>the objects are still alive, they are not GC'ed. That is why this is
    >>better:
    >>
    >>def create():
    >> f = file("tmp", "w")
    >> try:
    >> do_stuff_that_raises_exception
    >> finally:
    >> os.remove("tmp")
    >>
    >>~Ethan~

    >
    >
    > Why should this work? If I replace "do_stuff_that_raises_exception"
    > with "raise Exception", it fails in the same way, since the file is
    > open. Maybe you forgot "f.close()"? In any case, thanks for explaining
    > that the traceback keeps the object alive, that explains the issue.
    >
    > Arve


    Indeed. That should have been f.close() in the finally block. My
    apologies.

    ~Ethan~
     
    Ethan Furman, Oct 20, 2009
    #12
  13. a écrit :
    > On Oct 19, 4:14 pm, Grant Edwards <> wrote:
    >> On 2009-10-19, <> wrote:
    >>
    >>> I thought that file objects were supposed to be
    >>> garbage-collected and automatically closed once they go out of
    >>> scope,

    >> At some point after they go out of scope, they will be.
    >> Eventually. Exactly when is an implementation detail.
    >>
    >>> at least that's what I've been told by more merited Python
    >>> programmers. I'm also quite sure that this is quite a common
    >>> assumption in various programs,


    AFAICT, the file descriptor associated to the file object will be freed
    when the CPython process will finish - which implies you're using the
    CPython interpreter. This doesn't mean the file is garanteed to be
    closed _at the point where you're trying to os.remove() it_. And as I
    said, this is a CPython implementation detail - not a language
    specification. Jython or IronPython (or any other implementation) may
    not work that way.

    >> If your program relies on the assumption that some particular
    >> object will be garbage-collected between points A and B, then
    >> that's a bug in your program. If you depend on the fact that
    >> some object has been delted, then "del" it. If you depend on
    >> the fact that a file is closed, then close it.

    >
    > Personally I am against that assumption, and prefer a utility function
    > which reads the file and automatically closes it in a "finally" block
    > (in lieu of the "with" keyword).


    What's your problem with the with ???

    But anyway : explicitely releasing resources such as files, network
    connections etc is of course the RightThing(tm), except eventually in
    one-shot throwaway scripts.

    > However, when providing a patch for a
    > high-profile opensource Python project I was scolded for going to such
    > lengths, as the prescribed style was to just open files and let them
    > be closed implicitly.


    Err... Care to name the project ? I hope it's not one I ever advertized :(
     
    Bruno Desthuilliers, Oct 20, 2009
    #13
  14. Guest

    On 20 Okt, 16:00, Bruno Desthuilliers <bruno.
    > wrote:
    > What's your problem with the with ???


    No problem whatsoever, but I believe I wrote this utility function
    before the keyword was available, and it might be good to support
    older Python versions.

    > But anyway : explicitely releasing resources such as files, network
    > connections etc is of course the RightThing(tm), except eventually in
    > one-shot throwaway scripts.
    >
    > > However, when providing a patch for a
    > > high-profile opensource Python project I was scolded for going to such
    > > lengths, as the prescribed style was to just open files and let them
    > > be closed implicitly.

    >
    > Err... Care to name the project ? I hope it's not one I ever advertized :(


    I'm not going to name the project, but it is incidentally used by the
    Python project itself :)

    Arve
     
    , Oct 20, 2009
    #14
  15. En Tue, 20 Oct 2009 04:47:02 -0300,
    <> escribió:
    > On 20 Okt, 09:40, "Gabriel Genellina" <> wrote:
    >> En Tue, 20 Oct 2009 03:23:49 -0300,
    >> <> escribió:


    >> > I agree, but like I said, I've been told that this (implicit closing
    >> > of files) is the correct style by more merited Python developers, so
    >> > that made me think I was probably wrong ..

    >>
    >> Then tell those "more merited Python developers" that they're wrong,
    >> and that the right way to ensure a file is closed when you're done with
    >> it is to use a `with` statement (or a try/finally block in old Python
    >> releases)

    >
    > Easier said than done :) In any case, I now have this discussion as a
    > useful reference in the future. Thanks!


    If this thread is not enough, you can ask them to read the official Python
    tutorial:

    "It is good practice to use the with keyword when dealing with file
    objects. This has the advantage that the file is properly closed after its
    suite finishes, even if an exception is raised on the way. It is also much
    shorter than writing equivalent try-finally blocks."

    http://docs.python.org/tutorial/inputoutput.html#methods-of-file-objects

    --
    Gabriel Genellina
     
    Gabriel Genellina, Oct 20, 2009
    #15
  16. Guest

    On 20 Okt, 21:13, "Gabriel Genellina" <> wrote:
    > En Tue, 20 Oct 2009 04:47:02 -0300,  
    > <> escribió:
    >
    > > On 20 Okt, 09:40, "Gabriel Genellina" <> wrote:
    > >> En Tue, 20 Oct 2009 03:23:49 -0300,  
    > >> <> escribió:
    > >> > I agree, but like I said, I've been told that this (implicit closing
    > >> > of files) is the correct style by more merited Python developers, so
    > >> > that made me think I was probably wrong ..

    >
    > >> Then tell those "more merited Python developers" that they're wrong,  
    > >> and that the right way to ensure a file is closed when you're done with  
    > >> it is to use a `with` statement (or a try/finally block in old Python  
    > >> releases)

    >
    > > Easier said than done :) In any case, I now have this discussion as a
    > > useful reference in the future. Thanks!

    >
    > If this thread is not enough, you can ask them to read the official Python  
    > tutorial:
    >
    > "It is good practice to use the with keyword when dealing with file  
    > objects. This has the advantage that the file is properly closed after its  
    > suite finishes, even if an exception is raised on the way. It is also much  
    > shorter than writing equivalent try-finally blocks."
    >
    > http://docs.python.org/tutorial/inputoutput.html#methods-of-file-objects


    Perhaps the general attitude has changed now that the "with" keyword
    makes it so easy anyway (unless one needs to support older Pythons of
    course).

    Arve
     
    , Oct 20, 2009
    #16
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Mark
    Replies:
    0
    Views:
    696
  2. Replies:
    0
    Views:
    3,594
  3. Simon

    Connection closed exception

    Simon, Dec 24, 2004, in forum: ASP .Net Web Services
    Replies:
    1
    Views:
    110
    Simon
    Jan 18, 2005
  4. Brent Heinz
    Replies:
    0
    Views:
    107
    Brent Heinz
    Dec 7, 2005
  5. Matt Kruse
    Replies:
    5
    Views:
    322
    Richard Cornford
    Sep 9, 2003
Loading...

Share This Page