tempfile.NamedTemporaryFile wont work

Discussion in 'Python' started by Imbaud Pierre, Nov 19, 2006.

  1. On suse 9.3, tempfile.NamedTemporaryFile() doesnt work as expected.
    (I found a permanent workaround, so I dont ask for help)
    I expected to write to a file, and access it thru a shell command.
    This code, in a loop:
    tf = tempfile.NamedTemporaryFile()
    tfName = tf.name
    #tf.seek(0) # rewind the file
    tf.write(chunk); tf.flush()
    print >> sys.stderr, '%s: %s' % (tfName, ['no',
    'yes'][os.path.exists(tfName)])
    subprocess.Popen(['strings', tfName])

    Symptom: the file does not always exist, after the call to
    NamedTemporaryFile(). Or at least its not seen by the strings command,
    or by os.path.exists.

    I guess the bug is pretty much os dependent, or even filesystem
    dependent (Im on reiserfs). Maybe the os is buggy, maybe, somehow, the
    python interface. Or did I miss something?
    Shame, I didnt even try to check for a python bug tracker.
    Imbaud Pierre, Nov 19, 2006
    #1
    1. Advertising

  2. Imbaud Pierre wrote:

    > tf = tempfile.NamedTemporaryFile()
    > tfName = tf.name
    > [...]
    > print >> sys.stderr, '%s: %s' % (tfName, ['no',
    > 'yes'][os.path.exists(tfName)])
    > subprocess.Popen(['strings', tfName])


    Just out of curiosity: Why did you assign tf.name to tfname?

    Hypothetically, if tf.name changed, tfname wouldn't follow since
    strings are immutable.

    Regards,


    Björn

    --
    BOFH excuse #149:

    Dew on the telephone lines.
    Bjoern Schliessmann, Nov 19, 2006
    #2
    1. Advertising

  3. On Sun, 19 Nov 2006 13:11:13 +0100, Imbaud Pierre wrote:

    > On suse 9.3, tempfile.NamedTemporaryFile() doesnt work as expected.

    [snip]

    > Symptom: the file does not always exist, after the call to
    > NamedTemporaryFile(). Or at least its not seen by the strings command,
    > or by os.path.exists.
    >
    > I guess the bug is pretty much os dependent, or even filesystem
    > dependent (Im on reiserfs). Maybe the os is buggy, maybe, somehow, the
    > python interface. Or did I miss something?
    > Shame, I didnt even try to check for a python bug tracker.


    I can verify this problem occurs on Fedora Core 5 too:

    import os
    import sys
    import tempfile
    import subprocess
    def test(n):
    chunk = ': +++ abcd +++'
    for i in xrange(n):
    tf = tempfile.NamedTemporaryFile()
    tfName = tf.name
    tf.seek(0)
    tf.write(str(i) + chunk)
    tf.flush()
    if not os.path.exists(tfName):
    print 'pre-check: %s not there' % tfName
    subprocess.Popen(['strings', tfName])
    if not os.path.exists(tfName):
    print 'post-check: %s not there' % tfName


    And here is a typical run, with the boring bits removed for ease of
    reading:

    >>> test(30)

    0: +++ abcd +++
    1: +++ abcd +++
    [ more of the same ]
    14: +++ abcd +++
    strings: '/tmp/tmpOALbx9': No such file
    16: +++ abcd +++
    17: +++ abcd +++
    18: +++ abcd +++
    [ more of the same ]
    27: +++ abcd +++
    strings: /tmp/tmpdc52Nz: No such file or directory
    29: +++ abcd +++


    Curiouser and curiouser... not only does os.path.exist always report the
    temp file as existing (at least in my tests), even when strings can't find
    it, but strings returns different error messages.

    Is it possible this is a bug in strings?


    --
    Steven.
    Steven D'Aprano, Nov 19, 2006
    #3
  4. On Sun, 19 Nov 2006 13:18:39 +0100, Bjoern Schliessmann wrote:

    > Imbaud Pierre wrote:
    >
    >> tf = tempfile.NamedTemporaryFile()
    >> tfName = tf.name
    >> [...]
    >> print >> sys.stderr, '%s: %s' % (tfName, ['no',
    >> 'yes'][os.path.exists(tfName)])
    >> subprocess.Popen(['strings', tfName])

    >
    > Just out of curiosity: Why did you assign tf.name to tfname?
    >
    > Hypothetically, if tf.name changed, tfname wouldn't follow since
    > strings are immutable.


    Well, yes, but if tf.name changed, that won't change the file name on disk
    either:

    >>> tf = tempfile.NamedTemporaryFile()
    >>> tf.name

    '/tmp/tmpYVV1Ij'
    >>> os.path.exists(tf.name)

    True
    >>> oldname = tf.name
    >>> tf.name = "/tmp/something"
    >>> os.path.exists(tf.name)

    False
    >>> os.path.exists(oldname)

    True


    I'm guessing that binding tf.name to tfName is a micro-optimization. In a
    very tight loop, name lookups can take considerable time, and one
    optimization can be to reduce the number of lookups:

    method = something.method
    while 1:
    something.method # needs at least two lookups
    method # needs a single lookup


    --
    Steve.
    Steven D'Aprano, Nov 19, 2006
    #4
  5. Imbaud Pierre

    Peter Otten Guest

    Steven D'Aprano wrote:

    > On Sun, 19 Nov 2006 13:11:13 +0100, Imbaud Pierre wrote:
    >
    >> On suse 9.3, tempfile.NamedTemporaryFile() doesnt work as expected.

    > [snip]
    >
    >> Symptom: the file does not always exist, after the call to
    >> NamedTemporaryFile(). Or at least its not seen by the strings command,
    >> or by os.path.exists.
    >>
    >> I guess the bug is pretty much os dependent, or even filesystem
    >> dependent (Im on reiserfs). Maybe the os is buggy, maybe, somehow, the
    >> python interface. Or did I miss something?
    >> Shame, I didnt even try to check for a python bug tracker.

    >
    > I can verify this problem occurs on Fedora Core 5 too:
    >
    > import os
    > import sys
    > import tempfile
    > import subprocess
    > def test(n):
    > chunk = ': +++ abcd +++'
    > for i in xrange(n):
    > tf = tempfile.NamedTemporaryFile()
    > tfName = tf.name
    > tf.seek(0)
    > tf.write(str(i) + chunk)
    > tf.flush()
    > if not os.path.exists(tfName):
    > print 'pre-check: %s not there' % tfName
    > subprocess.Popen(['strings', tfName])
    > if not os.path.exists(tfName):
    > print 'post-check: %s not there' % tfName
    >
    >
    > And here is a typical run, with the boring bits removed for ease of
    > reading:
    >
    >>>> test(30)

    > 0: +++ abcd +++
    > 1: +++ abcd +++
    > [ more of the same ]
    > 14: +++ abcd +++
    > strings: '/tmp/tmpOALbx9': No such file
    > 16: +++ abcd +++
    > 17: +++ abcd +++
    > 18: +++ abcd +++
    > [ more of the same ]
    > 27: +++ abcd +++
    > strings: /tmp/tmpdc52Nz: No such file or directory
    > 29: +++ abcd +++
    >
    >
    > Curiouser and curiouser... not only does os.path.exist always report the
    > temp file as existing (at least in my tests), even when strings can't find
    > it, but strings returns different error messages.
    >
    > Is it possible this is a bug in strings?


    What /you/ are seeing is not a bug, I think. Popen() is asynchronous,
    therefore you may enter the second iteration -- which implicitly closes the
    temporary file -- before strings actually tries to access it. Use call()
    and everything should be fine.

    Peter
    Peter Otten, Nov 19, 2006
    #5
  6. Steven D'Aprano a écrit :
    > On Sun, 19 Nov 2006 13:18:39 +0100, Bjoern Schliessmann wrote:
    >
    >
    >>Imbaud Pierre wrote:
    >>
    >>
    >>> tf = tempfile.NamedTemporaryFile()
    >>> tfName = tf.name
    >>>[...]
    >>> print >> sys.stderr, '%s: %s' % (tfName, ['no',
    >>>'yes'][os.path.exists(tfName)])
    >>> subprocess.Popen(['strings', tfName])

    >>
    >>Just out of curiosity: Why did you assign tf.name to tfname?
    >>
    >>Hypothetically, if tf.name changed, tfname wouldn't follow since
    >>strings are immutable.

    >
    >
    > Well, yes, but if tf.name changed, that won't change the file name on disk
    > either:
    >
    >
    >>>>tf = tempfile.NamedTemporaryFile()
    >>>>tf.name

    >
    > '/tmp/tmpYVV1Ij'
    >
    >>>>os.path.exists(tf.name)

    >
    > True
    >
    >>>>oldname = tf.name
    >>>>tf.name = "/tmp/something"
    >>>>os.path.exists(tf.name)

    >
    > False
    >
    >>>>os.path.exists(oldname)

    >
    > True
    >
    >
    > I'm guessing that binding tf.name to tfName is a micro-optimization.

    indeed. And I dont see why tf.name would change.
    In a
    > very tight loop, name lookups can take considerable time, and one
    > optimization can be to reduce the number of lookups:
    >
    > method = something.method
    > while 1:
    > something.method # needs at least two lookups
    > method # needs a single lookup
    >
    >
    Imbaud Pierre, Nov 19, 2006
    #6
  7. Peter Otten a écrit :
    > Steven D'Aprano wrote:
    >
    >
    >>On Sun, 19 Nov 2006 13:11:13 +0100, Imbaud Pierre wrote:
    >>
    >>
    >>>On suse 9.3, tempfile.NamedTemporaryFile() doesnt work as expected.

    >>
    >>[snip]
    >>
    >>
    >>>Symptom: the file does not always exist, after the call to
    >>>NamedTemporaryFile(). Or at least its not seen by the strings command,
    >>>or by os.path.exists.
    >>>
    >>>I guess the bug is pretty much os dependent, or even filesystem
    >>>dependent (Im on reiserfs). Maybe the os is buggy, maybe, somehow, the
    >>>python interface. Or did I miss something?
    >>>Shame, I didnt even try to check for a python bug tracker.

    >>
    >>I can verify this problem occurs on Fedora Core 5 too:
    >>
    >>import os
    >>import sys
    >>import tempfile
    >>import subprocess
    >>def test(n):
    >>chunk = ': +++ abcd +++'
    >>for i in xrange(n):
    >>tf = tempfile.NamedTemporaryFile()
    >>tfName = tf.name
    >>tf.seek(0)
    >>tf.write(str(i) + chunk)
    >> tf.flush()
    >>if not os.path.exists(tfName):
    >>print 'pre-check: %s not there' % tfName
    >>subprocess.Popen(['strings', tfName])
    >>if not os.path.exists(tfName):
    >>print 'post-check: %s not there' % tfName
    >>
    >>
    >>And here is a typical run, with the boring bits removed for ease of
    >>reading:
    >>
    >>
    >>>>>test(30)

    >>
    >>0: +++ abcd +++
    >>1: +++ abcd +++
    >> [ more of the same ]
    >>14: +++ abcd +++
    >>strings: '/tmp/tmpOALbx9': No such file
    >>16: +++ abcd +++
    >>17: +++ abcd +++
    >>18: +++ abcd +++
    >> [ more of the same ]
    >>27: +++ abcd +++
    >>strings: /tmp/tmpdc52Nz: No such file or directory
    >>29: +++ abcd +++
    >>
    >>
    >>Curiouser and curiouser... not only does os.path.exist always report the
    >>temp file as existing (at least in my tests), even when strings can't find
    >>it, but strings returns different error messages.
    >>
    >>Is it possible this is a bug in strings?

    >
    >
    > What /you/ are seeing is not a bug, I think. Popen() is asynchronous,
    > therefore you may enter the second iteration -- which implicitly closes the
    > temporary file -- before strings actually tries to access it. Use call()
    > and everything should be fine.

    Thanks A LOT, works fine, I feel kind of silly; your diagnostic is
    pretty obvious, afterward... I felt uneasy not closing the Popen, but
    it worked, so why bother? Its so easy to make ugly code!
    Imbaud Pierre, Nov 19, 2006
    #7
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Replies:
    0
    Views:
    530
  2. Lee Harr
    Replies:
    5
    Views:
    696
    Tim Peters
    Dec 30, 2005
  3. Jason Lunz

    monkeypatching NamedTemporaryFile

    Jason Lunz, May 26, 2006, in forum: Python
    Replies:
    2
    Views:
    288
    Jason Lunz
    May 27, 2006
  4. Roy Smith
    Replies:
    1
    Views:
    594
    Alice Bevan–McGregor
    Dec 31, 2010
  5. Alex van der Spek

    Difference between tempfile and spooled tempfile?

    Alex van der Spek, Apr 5, 2012, in forum: Python
    Replies:
    2
    Views:
    357
    Steve Howell
    Apr 5, 2012
Loading...

Share This Page