when should I explicitely close a file?

Discussion in 'Python' started by gelonida, Apr 13, 2010.

  1. gelonida

    gelonida Guest

    Hi,


    I've been told, that following code snippet is not good.


    open("myfile","w").write(astring) , because I'm neither explicitely
    closing nor using the new 'with' syntax.

    What exactly is the impact of not closing the file explicitely
    (implicitley with a 'with' block)?


    Even with my example
    I'd expected to get an exception raised if not all data could have
    been written.

    I'd also expected, that all write data is flushed as soon as the
    filehandle is out of scope (meaning in the next line of my source
    code).


    Thanks for explaining me exactly what kind of evil I could encounter
    with not explicitely closing.
    gelonida, Apr 13, 2010
    #1
    1. Advertising

  2. gelonida

    Chris Rebert Guest

    On Tue, Apr 13, 2010 at 3:01 PM, gelonida <> wrote:
    > Hi,
    >
    > I've been told, that following code snippet is not good.
    >
    > open("myfile","w").write(astring) , because I'm neither explicitely
    > closing nor using the new 'with' syntax.
    >
    > What exactly is the impact of not closing the file explicitely
    > (implicitley with a 'with' block)?
    >
    > Even with my example
    > I'd expected to get an exception raised if not all data could have
    > been written.
    >
    > I'd also expected, that all write data is flushed as soon as the
    > filehandle is out of scope (meaning in the next line of my source
    > code).


    That extremely-quick responsiveness of the garbage-collection
    machinery is only guaranteed by CPython, not the language
    specification itself, and indeed some of the other implementations
    *explicitly don't* make that guarantee (and hence the data may not get
    flushed in a timely manner on those implementations). And portability
    of code is encouraged, hence the admonishment you encountered.

    Cheers,
    Chris
    --
    http://blog.rebertia.com
    Chris Rebert, Apr 13, 2010
    #2
    1. Advertising

  3. On Tue, 13 Apr 2010 15:01:25 -0700, gelonida wrote:

    > Hi,
    >
    >
    > I've been told, that following code snippet is not good.
    >
    >
    > open("myfile","w").write(astring) , because I'm neither explicitely
    > closing nor using the new 'with' syntax.
    >
    > What exactly is the impact of not closing the file explicitely
    > (implicitley with a 'with' block)?



    Your data may not be actually written to disk until the file closes. If
    you have a reference loop, and Python crashes badly enough, the garbage
    collector may never run and the file will never be closed, hence you will
    get data loss.

    If you are running something other than CPython (e.g. IronPython or
    Jython) then the file might not be closed until your program exits. If
    you have a long-running program that opens many, many files, it is
    possible for you to run out of system file handles and be unable to open
    any more.

    Best practice is to explicitly close the file when you are done with it,
    but for short scripts, I generally don't bother. Laziness is a virtue :)

    But for library code and larger applications, I always explicitly close
    the file, because I want to control exactly when the file is closed
    rather than leave it up to the interpreter. I don't know if my code might
    one day end up in a long-running Jython application so I try to code
    defensively and avoid even the possibility of a problem.


    > Even with my example
    > I'd expected to get an exception raised if not all data could have been
    > written.


    Generally if you get an exception while trying to *close* a file, you're
    pretty much stuffed. What are you going to do? How do you recover?

    My feeling is that you're probably safe with something as simple as

    file("myfile", "w").write("my data\n")

    but if you do something like

    some_data_structure.filehandle = file("myfile", "w")
    some_data_structure.filehandle.write("my data\n")
    # ... lots more code here

    and some_data_structure keeps the file open until the interpreter shuts
    down, there *might* be rare circumstances where you won't be notified of
    an exception, depending on the exact circumstances of timing of when the
    file gets closed. In the worst case, the file might not be closed until
    the interpreter is shutting down *and* has already dismantled the
    exception infrastructure, and so you can't get an exception. I don't know
    enough about the Python runtime (particularly about how it works during
    shutdown) to know how real this danger is, but if it is a danger, I bet
    it involves __del__ methods.



    > I'd also expected, that all write data is flushed as soon as the
    > filehandle is out of scope (meaning in the next line of my source code).


    This is only guaranteed with CPython, not other implementations.

    My feeling is that explicit closing is pedantic and careful, implicit
    closing is lazy and easy. You make your choice and take your chance :)




    --
    Steven
    Steven D'Aprano, Apr 14, 2010
    #3
  4. Giampaolo Rodola', Apr 14, 2010
    #4
  5. gelonida

    Chris Rebert Guest

    On Tue, Apr 13, 2010 at 5:45 PM, Giampaolo Rodola' <> wrote:
    > What about open('foo', 'w').close().
    > Does it have the same problems?


    Well, no, but that's only because it's a pointless no-op that doesn't
    really do anything besides possibly throwing an exception (e.g. if the
    script didn't have write access to the current directory).

    Cheers,
    Chris
    --
    http://blog.rebertia.com
    Chris Rebert, Apr 14, 2010
    #5
  6. gelonida

    Dave Angel Guest

    gelonida wrote:
    > Hi,
    >
    >
    > I've been told, that following code snippet is not good.
    >
    >
    > open("myfile","w").write(astring) , because I'm neither explicitely
    > closing nor using the new 'with' syntax.
    >
    > What exactly is the impact of not closing the file explicitely
    > (implicitley with a 'with' block)?
    >
    >
    > Even with my example
    > I'd expected to get an exception raised if not all data could have
    > been written.
    >
    > I'd also expected, that all write data is flushed as soon as the
    > filehandle is out of scope (meaning in the next line of my source
    > code).
    >
    >
    > Thanks for explaining me exactly what kind of evil I could encounter
    > with not explicitely closing.
    >
    >
    >

    Evil? No. Just undefined behavior.

    The language does NOT guarantee that a close or even a flush will occur
    when an object "goes out of scope." This is the same in Python as it is
    in Java. There's also no exception for data not being flushed.

    In one particular implementation of Python, called CPython, there are
    some things that tend to help. So if you're sure you're always going to
    be using this particular implementation, and understand what the
    restrictions are, then go ahead and be sloppy. Similarly, on some OS
    systems, files are flushed when a process ends. So if you know your
    application is only going to run on those environments, you might not
    bother closing files at the end of execution.

    It all depends on how restrictive your execution environment is going to be.

    DaveA
    Dave Angel, Apr 14, 2010
    #6
  7. gelonida

    Ryan Kelly Guest

    On Tue, 2010-04-13 at 18:19 -0700, Chris Rebert wrote:
    > On Tue, Apr 13, 2010 at 5:45 PM, Giampaolo Rodola' <> wrote:
    > > What about open('foo', 'w').close().
    > > Does it have the same problems?

    >
    > Well, no, but that's only because it's a pointless no-op that doesn't
    > really do anything besides possibly throwing an exception (e.g. if the
    > script didn't have write access to the current directory).


    Actually, it will create the file if it doesn't exist, and truncate it
    to zero length if it does.


    Ryan


    --
    Ryan Kelly
    http://www.rfk.id.au | This message is digitally signed. Please visit
    | http://www.rfk.id.au/ramblings/gpg/ for details
    Ryan Kelly, Apr 14, 2010
    #7
  8. Re: when should I explicitly close a file?

    In message
    <>, gelonida
    wrote:

    > I've been told, that following code snippet is not good.
    >
    > open("myfile","w").write(astring) ...


    I do that for reads, but never for writes.

    For writes, you want to give a chance for write errors to raise an exception
    and alert the user, instead of failing silently, to avoid inadvertent data
    loss. Hence the explicit close.
    Lawrence D'Oliveiro, Apr 17, 2010
    #8
  9. gelonida

    Lie Ryan Guest

    Re: when should I explicitly close a file?

    On 04/17/10 21:23, Lawrence D'Oliveiro wrote:
    > In message
    > <>, gelonida
    > wrote:
    >
    >> I've been told, that following code snippet is not good.
    >>
    >> open("myfile","w").write(astring) ...

    >
    > I do that for reads, but never for writes.
    >
    > For writes, you want to give a chance for write errors to raise an exception
    > and alert the user, instead of failing silently, to avoid inadvertent data
    > loss. Hence the explicit close.


    In short, in case of doubt, just be explicit.

    Since in python nothing is guaranteed about implicit file close, you
    must always explicitly close it.
    Lie Ryan, Apr 17, 2010
    #9
  10. Re: when should I explicitly close a file?

    In message <4bc9aadb$>, Lie Ryan wrote:

    > Since in python nothing is guaranteed about implicit file close ...


    It is guaranteed that objects with a reference count of zero will be
    disposed. In my experiments, this happens immediately.
    Lawrence D'Oliveiro, Apr 22, 2010
    #10
  11. gelonida

    Chris Rebert Guest

    Re: when should I explicitly close a file?

    On Wed, Apr 21, 2010 at 5:53 PM, Lawrence D'Oliveiro wrote:
    > In message <4bc9aadb$>, Lie Ryan wrote:
    >
    >> Since in python nothing is guaranteed about implicit file close ...

    >
    > It is guaranteed that objects with a reference count of zero will be
    > disposed.


    > In my experiments, this happens immediately.


    Experiment with an implementation other than CPython and prepare to be
    surprised.

    Cheers,
    Chris
    --
    http://blog.rebertia.com
    Chris Rebert, Apr 22, 2010
    #11
  12. Re: when should I explicitly close a file?

    On Thu, 22 Apr 2010 12:53:51 +1200, Lawrence D'Oliveiro wrote:

    > In message <4bc9aadb$>, Lie Ryan wrote:
    >
    >> Since in python nothing is guaranteed about implicit file close ...

    >
    > It is guaranteed that objects with a reference count of zero will be
    > disposed.


    Not all Python implementations have reference counts at all, e.g. Jython
    and IronPython. Neither of those close files immediately.


    > In my experiments, this happens immediately.


    Are your experiments done under PyPy, CLPython, or Pynie?



    --
    Steven
    Steven D'Aprano, Apr 22, 2010
    #12
  13. Re: when should I explicitly close a file?

    * Lawrence D'Oliveiro:
    > In message <4bc9aadb$>, Lie Ryan wrote:
    >
    >> Since in python nothing is guaranteed about implicit file close ...

    >
    > It is guaranteed that objects with a reference count of zero will be
    > disposed.


    Only in current CPython.


    > In my experiments, this happens immediately.


    Depends what you mean, but even in current CPython destruction of a local can be
    postponed indefinitely if a reference to the stack frame is kept somewhere.

    And that happens, for example, when an exception is raised (until the handler
    completes, but it doesn't necessarily complete for a Very Long Time).


    Cheers & hth.,

    - Alf
    Alf P. Steinbach, Apr 22, 2010
    #13
  14. Re: when should I explicitly close a file?

    On Thu, 2010-04-22 at 12:53 +1200, Lawrence D'Oliveiro wrote:
    > In message <4bc9aadb$>, Lie Ryan wrote:
    > > Since in python nothing is guaranteed about implicit file close ...

    > It is guaranteed that objects with a reference count of zero will be
    > disposed. In my experiments, this happens immediately.


    A current implementation specific detail. Always close files.
    Otherwise, in the future, or on a different run-time, your code will
    break.
    Adam Tauno Williams, Apr 22, 2010
    #14
  15. Re: when should I explicitly close a file?

    In message <>, Chris
    Rebert wrote:

    > On Wed, Apr 21, 2010 at 5:53 PM, Lawrence D'Oliveiro wrote:
    >
    >> In message <4bc9aadb$>, Lie Ryan wrote:
    >>
    >>> Since in python nothing is guaranteed about implicit file close ...

    >>
    >> It is guaranteed that objects with a reference count of zero will be
    >> disposed.

    >
    >> In my experiments, this happens immediately.

    >
    > Experiment with an implementation other than CPython and prepare to be
    > surprised.


    Any implementation that doesn’t do reference-counting is brain-damaged.
    Lawrence D'Oliveiro, Apr 23, 2010
    #15
  16. Re: when should I explicitly close a file?

    On Fri, 23 Apr 2010 16:29:46 +1200, Lawrence D'Oliveiro wrote:

    > Any implementation that doesn’t do reference-counting is brain-damaged.


    Funny, that's exactly what other people say about implementations that
    *do* use reference counting.

    --
    Steven
    Steven D'Aprano, Apr 23, 2010
    #16
  17. Re: when should I explicitly close a file?

    On Fri, 2010-04-23 at 16:29 +1200, Lawrence D'Oliveiro wrote:
    > In message <>, Chris
    > Rebert wrote:
    > > On Wed, Apr 21, 2010 at 5:53 PM, Lawrence D'Oliveiro wrote:
    > >> In message <4bc9aadb$>, Lie Ryan wrote:
    > >>> Since in python nothing is guaranteed about implicit file close ...
    > >> It is guaranteed that objects with a reference count of zero will be
    > >> disposed.
    > >> In my experiments, this happens immediately.

    > > Experiment with an implementation other than CPython and prepare to be
    > > surprised.

    > Any implementation that doesn’t do reference-counting is brain-damaged.


    Why? There are much better ways to do memory management / garbage
    collection; especially when dealing with large applications.
    Adam Tauno Williams, Apr 23, 2010
    #17
  18. Re: when should I explicitly close a file?

    * Adam Tauno Williams:
    > On Fri, 2010-04-23 at 16:29 +1200, Lawrence D'Oliveiro wrote:
    >> In message <>, Chris
    >> Rebert wrote:
    >>> On Wed, Apr 21, 2010 at 5:53 PM, Lawrence D'Oliveiro wrote:
    >>>> In message <4bc9aadb$>, Lie Ryan wrote:
    >>>>> Since in python nothing is guaranteed about implicit file close ...
    >>>> It is guaranteed that objects with a reference count of zero will be
    >>>> disposed.
    >>>> In my experiments, this happens immediately.
    >>> Experiment with an implementation other than CPython and prepare to be
    >>> surprised.

    >> Any implementation that doesn’t do reference-counting is brain-damaged.

    >
    > Why?


    Depends on what the statement was meant to mean.

    But for a literal context-free interpretation e.g. the 'sys.getrefcount'
    function is not documented as CPython only and thus an implementation that
    didn't do reference counting would not be a conforming Python implementation.

    Whether it uses reference counting to destroy objects at earliest opportunity is
    another matter.


    > There are much better ways to do memory management / garbage
    > collection; especially when dealing with large applications.


    Depends on whether you're talking about Python implementations or as a matter of
    general principle, and depends on how you define "better", "large" and so on.

    On its own it's a pretty meaningless statement.

    But although a small flame war erupted the last time I mentioned this, I think a
    case can be made that Python is not designed for programming-in-the-large. And
    that the current CPython scheme is eminently suitable for small scripts. But it
    has its drawbacks, especially considering the various ways that stack frames can
    be retained, and considering the documentation of 'gc.garbage', ...

    "Objects that have __del__() methods and are part of a reference cycle cause
    the entire reference cycle to be uncollectable, including objects not
    necessarily in the cycle but reachable only from it."

    .... which means that a programming style assuming current CPython semantics and
    employing RAII can be detrimental in a sufficiently large system.


    Cheers & hth.,

    - Alf
    Alf P. Steinbach, Apr 23, 2010
    #18
  19. Re: when should I explicitly close a file?

    On Fri, 23 Apr 2010 13:19:41 +0200, Alf P. Steinbach wrote:

    > But for a literal context-free interpretation e.g. the 'sys.getrefcount'
    > function is not documented as CPython only and thus an implementation
    > that didn't do reference counting would not be a conforming Python
    > implementation.


    Since Jython and IronPython are conforming Python implementations, and
    Guido has started making policy decisions specifically to support these
    other implementations (e.g. the language feature moratorium, PEP 3003), I
    think we can assume that this is a documentation bug.

    However, a Python implementation that always returned 0 for
    sys.getrefcount would technically satisfy the word of the documentation,
    if not the spirit.



    --
    Steven
    Steven D'Aprano, Apr 23, 2010
    #19
  20. Re: when should I explicitly close a file?

    * Steven D'Aprano:
    > On Fri, 23 Apr 2010 13:19:41 +0200, Alf P. Steinbach wrote:
    >
    >> But for a literal context-free interpretation e.g. the 'sys.getrefcount'
    >> function is not documented as CPython only and thus an implementation
    >> that didn't do reference counting would not be a conforming Python
    >> implementation.

    >
    > Since Jython and IronPython are conforming Python implementations, and
    > Guido has started making policy decisions specifically to support these
    > other implementations (e.g. the language feature moratorium, PEP 3003), I
    > think we can assume that this is a documentation bug.


    The documentation for Jython specifies the same for 'sys.getrefcount'.

    However, testing:

    <output>
    *sys-package-mgr*: processing new jar, 'C:\Program Files\jython2.5.1\jython.jar'
    *sys-package-mgr*: processing new jar, 'C:\Program
    Files\Java\jre6\lib\resources.jar'
    *sys-package-mgr*: processing new jar, 'C:\Program Files\Java\jre6\lib\rt.jar'
    *sys-package-mgr*: processing new jar, 'C:\Program Files\Java\jre6\lib\jsse.jar'
    *sys-package-mgr*: processing new jar, 'C:\Program Files\Java\jre6\lib\jce.jar'
    *sys-package-mgr*: processing new jar, 'C:\Program Files\Java\jre6\lib\charsets.jar'
    *sys-package-mgr*: processing new jar, 'C:\Program
    Files\Java\jre6\lib\ext\dnsns.jar'
    *sys-package-mgr*: processing new jar, 'C:\Program
    Files\Java\jre6\lib\ext\localedata.jar'
    *sys-package-mgr*: processing new jar, 'C:\Program
    Files\Java\jre6\lib\ext\sunjce_provider.jar'
    *sys-package-mgr*: processing new jar, 'C:\Program
    Files\Java\jre6\lib\ext\sunmscapi.jar'
    *sys-package-mgr*: processing new jar, 'C:\Program
    Files\Java\jre6\lib\ext\sunpkcs11.jar'
    A created
    Traceback (most recent call last):
    File "c:\test\refcount.py", line 17, in <module>
    writeln( str( sys.getrefcount( a ) - 1 ) )
    AttributeError: 'systemstate' object has no attribute 'getrefcount'
    </output>


    > However, a Python implementation that always returned 0 for
    > sys.getrefcount would technically satisfy the word of the documentation,
    > if not the spirit.


    Yes.

    OK, learned something new: I though Jython actually implemented getrefcount.

    The Jython docs says it does...


    Cheers,

    - Alf
    Alf P. Steinbach, Apr 24, 2010
    #20
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Amelyan
    Replies:
    0
    Views:
    384
    Amelyan
    May 24, 2005
  2. Phaedrus

    Explicitely calling Thread.run

    Phaedrus, Jul 5, 2003, in forum: Java
    Replies:
    3
    Views:
    468
    Roedy Green
    Jul 6, 2003
  3. Victor Bazarov

    Calling constructor explicitely

    Victor Bazarov, Feb 23, 2005, in forum: C++
    Replies:
    9
    Views:
    4,464
    Ron Natalie
    Feb 24, 2005
  4. Xiao Ma
    Replies:
    16
    Views:
    1,424
    =?ISO-8859-15?Q?Arne_Vajh=F8j?=
    Oct 27, 2007
  5. Iñaki Baz Castillo
    Replies:
    7
    Views:
    839
    Iñaki Baz Castillo
    Jan 12, 2010
Loading...

Share This Page