Popen3 and capturestderr

[Kenneth Pronovici]

Hi,

I have a problem with a popen2 pipe hanging partway through execution of
a command. This happens in my function executeCommand() that is used
for every "shell" command execution in my program. Source code for this
function is below. My development environment is Debian unstable with
Python 2.3.5.

Within executeCommand(), output of every executed command is always
logged using a Python logger. If outputFile is passed in, then output
will also be written into the outputFile, which might have been created
using open(), gzip.GzipFile() or bz2.BZ2File(), etc. This functionality
exists so I have an equivalent of command-line redirection of stdout,
i.e. command > file.

If ignoreStderr=False, I use popen2.Popen4 so that stderr and stdout are
intermingled. If ignoreStderr=True, I use popen2.Popen3 with
capturestderr=True so stderr doesn't appear in the output. This
functionality exists so I have an equivalent of command-line redirection
of stderr, i.e. command 2>/dev/null.

I am using popen2 rather than popen because I want to avoid all of the
issues around shell interpolation of arguments, etc.

This whole thing has been working pretty well for a while. Then lately,
I started using it on commands that produce a large amount of output,
and it's begun to hang partway through execution. A good way to
reproduce this behavior on my system is to use command "ls -laR /". The
function hangs whever the output file reaches a certain size (around 20
MB). When I interrupt the program, the stack trace shows that it's
stuck at pipe.fromchild.readline().

After some digging, I've decided that this behavior probably occurs
because I am ignoring the pipe.childerr file object. Indeed, if I call
pipe.childerr.close() right after opening the pipe, my "ls" command that
had been hanging completes normally. However, other commands which
actually attempt to write to stderr don't seem to like this very much.

What is the right way to discard stderr when working with a pipe? I
want to consistently throw it away, and I don't see a good way to do
this with the popen2 implementation.

Thanks for the help,

KEN

------------------------------- snip -------------------------------

def executeCommand(command, args, returnOutput=False,
ignoreStderr=False, outputFile=None):
"""
@param command: Shell command to execute in a list like [ "ls", ]
@param args: List of arguments to the command, like [ "-laR", "/", ]
@param returnOutput: Indicates whether to return the output of the command
@param outputFile: File object that all output should be written to.
@return: Tuple of C{(result, output)}.
"""
output = []
fields = command[:]
fields.extend(args)
if ignoreStderr:
pipe = popen2.Popen3(fields, capturestderr=True)
else:
pipe = popen2.Popen4(fields)
pipe.tochild.close()
while True:
line = pipe.fromchild.readline()
if not line: break
if returnOutput: output.append(line)
if outputFile is not None: outputFile.write(line)
outputLogger.info(line[:-1])
if returnOutput:
return (pipe.wait(), output)
else:
return (pipe.wait(), None)

------------------------------- snip -------------------------------

 

[Kenneth Pronovici]

Quoth Kenneth Pronovici <[email protected]>:
...
| If ignoreStderr=False, I use popen2.Popen4 so that stderr and stdout are
| intermingled. If ignoreStderr=True, I use popen2.Popen3 with
| capturestderr=True so stderr doesn't appear in the output. This
| functionality exists so I have an equivalent of command-line redirection
| of stderr, i.e. command 2>/dev/null.
...
| After some digging, I've decided that this behavior probably occurs
| because I am ignoring the pipe.childerr file object. Indeed, if I call
| pipe.childerr.close() right after opening the pipe, my "ls" command that
| had been hanging completes normally. However, other commands which
| actually attempt to write to stderr don't seem to like this very much.
|
| What is the right way to discard stderr when working with a pipe? I
| want to consistently throw it away, and I don't see a good way to do
| this with the popen2 implementation.

Right, popen2 gives you about 3 options, out of probably dozens that
you could get with shell redirections. On the other hand, the source
is available, and Python is an OOP language, so I assume there is no
reason you can't make a derived class that does just what you want.
In the present case I guess that would mean something like
null = os.open('/dev/null', os.O_RDWR)
os.dup2(null, 0)
os.dup2(null, 2) (depending)
os.close(null)
along with other stuff you can just copy from Popen4.

Ah... ok, subclassing is an option I hadn't considered. I'll give that
a whirl and see whether I can make it work.

KEN

--
Kenneth J. Pronovici <[email protected]>
Personal Homepage: http://www.skyjammer.com/~pronovic/
"They that can give up essential liberty to obtain a little
temporary safety deserve neither liberty nor safety."
- Benjamin Franklin, Historical Review of Pennsylvania, 1759