os.popen and lengthy operations

[Dmitry Teslenko]

Hello!
I'm using os.popen to perform lengthy operation such as building some
project from source.
It looks like this:
def execute_and_save_output( command, out_file, err_file):

import os

def execute_and_save_output( command, out_file, err_file):
(i,o,e) = os.popen3( command )
try:
for line in o:
out_file.write( line )

for line in e:
err_file.write( line )
finally:
i.close()
o.close()
e.close()

....
execute_and_save_output( '<some long to run command>', out_file, err_file)

Problem is that script hangs on operations that take long to execute
and have lots of output such as building scripts.

 

[Marc 'BlackJack' Rintsch]

I'm using os.popen to perform lengthy operation such as building some
project from source.
It looks like this:
def execute_and_save_output( command, out_file, err_file):

import os

def execute_and_save_output( command, out_file, err_file):
(i,o,e) = os.popen3( command )
try:
for line in o:
out_file.write( line )

for line in e:
err_file.write( line )
finally:
i.close()
o.close()
e.close()

...
execute_and_save_output( '<some long to run command>', out_file, err_file)

Problem is that script hangs on operations that take long to execute
and have lots of output such as building scripts.

Your code reads from the process' stdout until there is nothin to read
anymore and then from stderr. The process might output something to both.
The output is buffered. And when the stderr buffer is full the process
blocks until your application reads something from `e`. That's where the
whole thing hangs. You wait for something on `o` and the process waits
for you to read from `e` → deadlock.

You have to use threads to read both `o` and `e` or the `select` module to
look which file has something to read.

Ciao,
Marc 'BlackJack' Rintsch

 

[tokland]

I'm using os.popen to perform lengthy operation such as building some
project from source.

def execute_and_save_output( command, out_file, err_file):
(i,o,e) = os.popen3( command )

You should consider using the higher-level "subprocess" module:

import subprocess

def run(command, output, error, bufsize=None):
popen = subprocess.Popen(command.split(), bufsize=bufsize,
stdout=output, stderr=error)
popen.communicate()
return popen.returncode

example:

create = lambda path: open(path, "w")
run("ls /etc/services abc", create("/tmp/output"), create("/tmp/
error"))

Check how to use the "bufsize" parameter in the docs:

http://docs.python.org/lib/node529.html

arnau