stdout funniness from os.system() calls when redirecting output

[Birch]

I have a python script that uses the print function throughout, and as
well uses calls to os.system() to spawn DOS commandline executables.

My issue is that I redirect all of the output from this script to a
file (including that which is printed by the spawned programs) via the
redirect (">") function on a Win2K Command Prompt. In the captured
output however, the output from the os.system() calls ALWAYS comes
before the output from the print calls in the python script.

This does NOT happen if I run the python script without redirecting
the output to a file. (everything prints out properly in the Command
Prompt window)

Does anyone have any experience with this, and/or know the fix?

Much Appreciated!

 

[Dennis Lee Bieber]

Birch fed this fish to the penguins on Tuesday 14 October 2003 16:11 pm:

My issue is that I redirect all of the output from this script to a
file (including that which is printed by the spawned programs) via the
redirect (">") function on a Win2K Command Prompt. In the captured
output however, the output from the os.system() calls ALWAYS comes
before the output from the print calls in the python script.

This does NOT happen if I run the python script without redirecting
the output to a file. (everything prints out properly in the Command
Prompt window)

Does anyone have any experience with this, and/or know the fix?

Hypothesis:

Once redirected, the OS no longer sees an "interactive" file (terminal
window -- I/O is flushed on EOL), but sees a disk file.

Disk file I/O is buffered.

The main program uses a buffer separate from that used by each
os.system() process.

The buffer is flushed when the process finishes.

Hence, os.system() process begins shutdown processing, flushing
redirected buffer, THEN main process concludes and flushes its buffer.

--

 

[Paul Watson]

I have a python script that uses the print function throughout, and as
well uses calls to os.system() to spawn DOS commandline executables.

My issue is that I redirect all of the output from this script to a
file (including that which is printed by the spawned programs) via the
redirect (">") function on a Win2K Command Prompt. In the captured
output however, the output from the os.system() calls ALWAYS comes
before the output from the print calls in the python script.

This does NOT happen if I run the python script without redirecting
the output to a file. (everything prints out properly in the Command
Prompt window)

You probably need to flush what is in the buffers of the parent process
before calling os.system(). Then, do not attempt redirection on the
os.system() command. The stdout file handle should be passed to the child.
This worked on AIX under Python 2.1.

#! /usr/bin/env python
import os
import sys
print "python is started"
sys.stdout.flush()
os.system('ls -al')
print "python is done"
sys.stdout.flush()

However, it appears that Win32 Python 2.3 may not be passing the the file
descriptor or something else is awry. Is this a bug?

C:\src\t>type pt.py
#! /usr/bin/env python
import os
print "python is started"
os.system('dir')
print "python is done"

C:\src\t>pt.py
python is started
Volume in drive C is dir
Volume Serial Number is 146D-04D8

Directory of C:\src\t

2003-10-19 20:24 <DIR> .
2003-10-19 20:24 <DIR> ..
2003-10-19 20:25 35 jj
2003-10-19 20:20 104 pt.py
2 File(s) 139 bytes
2 Dir(s) 4,800,147,456 bytes free
python is done

C:\src\t>pt.py >jj
There is not enough space on the disk.

 

[Birch]

You probably need to flush what is in the buffers of the parent process
before calling os.system(). Then, do not attempt redirection on the
os.system() command. The stdout file handle should be passed to the child.
This worked on AIX under Python 2.1.

#! /usr/bin/env python
import os
import sys
print "python is started"
sys.stdout.flush()
os.system('ls -al')
print "python is done"
sys.stdout.flush()

However, it appears that Win32 Python 2.3 may not be passing the the file
descriptor or something else is awry. Is this a bug?

C:\src\t>type pt.py
#! /usr/bin/env python
import os
print "python is started"
os.system('dir')
print "python is done"

C:\src\t>pt.py
python is started
Volume in drive C is dir
Volume Serial Number is 146D-04D8

Directory of C:\src\t

2003-10-19 20:24 <DIR> .
2003-10-19 20:24 <DIR> ..
2003-10-19 20:25 35 jj
2003-10-19 20:20 104 pt.py
2 File(s) 139 bytes
2 Dir(s) 4,800,147,456 bytes free
python is done

C:\src\t>pt.py >jj
There is not enough space on the disk.

sys.stdout.flush() seems to do it... now I can organize my output log
files to mean something to people other than myself!

Thanks so much!