tempfile.NamedTemporaryFile wont work

[Imbaud Pierre]

On suse 9.3, tempfile.NamedTemporaryFile() doesnt work as expected.
(I found a permanent workaround, so I dont ask for help)
I expected to write to a file, and access it thru a shell command.
This code, in a loop:
tf = tempfile.NamedTemporaryFile()
tfName = tf.name
#tf.seek(0) # rewind the file
tf.write(chunk); tf.flush()
print >> sys.stderr, '%s: %s' % (tfName, ['no',
'yes'][os.path.exists(tfName)])
subprocess.Popen(['strings', tfName])

Symptom: the file does not always exist, after the call to
NamedTemporaryFile(). Or at least its not seen by the strings command,
or by os.path.exists.

I guess the bug is pretty much os dependent, or even filesystem
dependent (Im on reiserfs). Maybe the os is buggy, maybe, somehow, the
python interface. Or did I miss something?
Shame, I didnt even try to check for a python bug tracker.

 

[Bjoern Schliessmann]

tf = tempfile.NamedTemporaryFile()
tfName = tf.name
[...]
print >> sys.stderr, '%s: %s' % (tfName, ['no',
'yes'][os.path.exists(tfName)])
subprocess.Popen(['strings', tfName])

Just out of curiosity: Why did you assign tf.name to tfname?

Hypothetically, if tf.name changed, tfname wouldn't follow since
strings are immutable.

Regards,


Björn

 

[Steven D'Aprano]

On suse 9.3, tempfile.NamedTemporaryFile() doesnt work as expected. [snip]

Symptom: the file does not always exist, after the call to
NamedTemporaryFile(). Or at least its not seen by the strings command,
or by os.path.exists.

I guess the bug is pretty much os dependent, or even filesystem
dependent (Im on reiserfs). Maybe the os is buggy, maybe, somehow, the
python interface. Or did I miss something?
Shame, I didnt even try to check for a python bug tracker.

I can verify this problem occurs on Fedora Core 5 too:

import os
import sys
import tempfile
import subprocess
def test(n):
chunk = ': +++ abcd +++'
for i in xrange(n):
tf = tempfile.NamedTemporaryFile()
tfName = tf.name
tf.seek(0)
tf.write(str(i) + chunk)
tf.flush()
if not os.path.exists(tfName):
print 'pre-check: %s not there' % tfName
subprocess.Popen(['strings', tfName])
if not os.path.exists(tfName):
print 'post-check: %s not there' % tfName


And here is a typical run, with the boring bits removed for ease of
reading:
0: +++ abcd +++
1: +++ abcd +++
[ more of the same ]
14: +++ abcd +++
strings: '/tmp/tmpOALbx9': No such file
16: +++ abcd +++
17: +++ abcd +++
18: +++ abcd +++
[ more of the same ]
27: +++ abcd +++
strings: /tmp/tmpdc52Nz: No such file or directory
29: +++ abcd +++


Curiouser and curiouser... not only does os.path.exist always report the
temp file as existing (at least in my tests), even when strings can't find
it, but strings returns different error messages.

Is it possible this is a bug in strings?

 

[Steven D'Aprano]

tf = tempfile.NamedTemporaryFile()
tfName = tf.name
[...]
print >> sys.stderr, '%s: %s' % (tfName, ['no',
'yes'][os.path.exists(tfName)])
subprocess.Popen(['strings', tfName])

Just out of curiosity: Why did you assign tf.name to tfname?

Hypothetically, if tf.name changed, tfname wouldn't follow since
strings are immutable.

Well, yes, but if tf.name changed, that won't change the file name on disk
either:
True


I'm guessing that binding tf.name to tfName is a micro-optimization. In a
very tight loop, name lookups can take considerable time, and one
optimization can be to reduce the number of lookups:

method = something.method
while 1:
something.method # needs at least two lookups
method # needs a single lookup

 

[Peter Otten]

On suse 9.3, tempfile.NamedTemporaryFile() doesnt work as expected. [snip]

Symptom: the file does not always exist, after the call to
NamedTemporaryFile(). Or at least its not seen by the strings command,
or by os.path.exists.

I guess the bug is pretty much os dependent, or even filesystem
dependent (Im on reiserfs). Maybe the os is buggy, maybe, somehow, the
python interface. Or did I miss something?
Shame, I didnt even try to check for a python bug tracker.

I can verify this problem occurs on Fedora Core 5 too:

import os
import sys
import tempfile
import subprocess
def test(n):
chunk = ': +++ abcd +++'
for i in xrange(n):
tf = tempfile.NamedTemporaryFile()
tfName = tf.name
tf.seek(0)
tf.write(str(i) + chunk)
tf.flush()
if not os.path.exists(tfName):
print 'pre-check: %s not there' % tfName
subprocess.Popen(['strings', tfName])
if not os.path.exists(tfName):
print 'post-check: %s not there' % tfName


And here is a typical run, with the boring bits removed for ease of
reading:
0: +++ abcd +++
1: +++ abcd +++
[ more of the same ]
14: +++ abcd +++
strings: '/tmp/tmpOALbx9': No such file
16: +++ abcd +++
17: +++ abcd +++
18: +++ abcd +++
[ more of the same ]
27: +++ abcd +++
strings: /tmp/tmpdc52Nz: No such file or directory
29: +++ abcd +++


Curiouser and curiouser... not only does os.path.exist always report the
temp file as existing (at least in my tests), even when strings can't find
it, but strings returns different error messages.

Is it possible this is a bug in strings?

What /you/ are seeing is not a bug, I think. Popen() is asynchronous,
therefore you may enter the second iteration -- which implicitly closes the
temporary file -- before strings actually tries to access it. Use call()
and everything should be fine.

Peter

 

[Imbaud Pierre]

Steven D'Aprano a écrit :

Imbaud Pierre wrote:

tf = tempfile.NamedTemporaryFile()
tfName = tf.name
[...]
print >> sys.stderr, '%s: %s' % (tfName, ['no',
'yes'][os.path.exists(tfName)])
subprocess.Popen(['strings', tfName])

Just out of curiosity: Why did you assign tf.name to tfname?

Hypothetically, if tf.name changed, tfname wouldn't follow since
strings are immutable.

Well, yes, but if tf.name changed, that won't change the file name on disk
either:

'/tmp/tmpYVV1Ij'


True

False

True


I'm guessing that binding tf.name to tfName is a micro-optimization.

indeed. And I dont see why tf.name would change.
In a

 

[Imbaud Pierre]

Peter Otten a écrit :

Steven D'Aprano wrote:

On suse 9.3, tempfile.NamedTemporaryFile() doesnt work as expected.
[snip]


Symptom: the file does not always exist, after the call to
NamedTemporaryFile(). Or at least its not seen by the strings command,
or by os.path.exists.

I guess the bug is pretty much os dependent, or even filesystem
dependent (Im on reiserfs). Maybe the os is buggy, maybe, somehow, the
python interface. Or did I miss something?
Shame, I didnt even try to check for a python bug tracker.

I can verify this problem occurs on Fedora Core 5 too:

import os
import sys
import tempfile
import subprocess
def test(n):
chunk = ': +++ abcd +++'
for i in xrange(n):
tf = tempfile.NamedTemporaryFile()
tfName = tf.name
tf.seek(0)
tf.write(str(i) + chunk)
tf.flush()
if not os.path.exists(tfName):
print 'pre-check: %s not there' % tfName
subprocess.Popen(['strings', tfName])
if not os.path.exists(tfName):
print 'post-check: %s not there' % tfName


And here is a typical run, with the boring bits removed for ease of
reading:

test(30)

0: +++ abcd +++
1: +++ abcd +++
[ more of the same ]
14: +++ abcd +++
strings: '/tmp/tmpOALbx9': No such file
16: +++ abcd +++
17: +++ abcd +++
18: +++ abcd +++
[ more of the same ]
27: +++ abcd +++
strings: /tmp/tmpdc52Nz: No such file or directory
29: +++ abcd +++


Curiouser and curiouser... not only does os.path.exist always report the
temp file as existing (at least in my tests), even when strings can't find
it, but strings returns different error messages.

Is it possible this is a bug in strings?

What /you/ are seeing is not a bug, I think. Popen() is asynchronous,
therefore you may enter the second iteration -- which implicitly closes the
temporary file -- before strings actually tries to access it. Use call()
and everything should be fine.

Thanks A LOT, works fine, I feel kind of silly; your diagnostic is
pretty obvious, afterward... I felt uneasy not closing the Popen, but
it worked, so why bother? Its so easy to make ugly code!