python script hangs when run from subprocess

[Larry.Martell]

I have a python script and when I run it directly from the command line it runs to completion. But I need to run it from another script. I do that like this:

p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
rv = p.wait()
out_buf = p.stdout.read()

When I do this, wait never returns. If I trace the underlying script it's always in the same write to stderr that never seems to complete:

write(2, "/KA22/05Feb12/Images/12063LBO003"..., 24457

I run many other scripts and commands in the same manner, and they all complete, it's just this one. Anyone have any ideas why this is happening, and how I can further debug or fix this?

TIA!
-larry

 

[Peter Otten]

I have a python script and when I run it directly from the command line it
runs to completion. But I need to run it from another script. I do that
like this:

p = subprocess.Popen(cmd, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT) rv = p.wait()
out_buf = p.stdout.read()

When I do this, wait never returns. If I trace the underlying script it's
always in the same write to stderr that never seems to complete:

write(2, "/KA22/05Feb12/Images/12063LBO003"..., 24457

I run many other scripts and commands in the same manner, and they all
complete, it's just this one. Anyone have any ideas why this is happening,
and how I can further debug or fix this?

The script writes to an OS buffer, and when that buffer is full it blocks
forever. p.wait() in turn then waits forever for the script to terminate...

As a fix try

out_buf = subprocess.Popen(...).communicate()[0]


This uses threads or select (depending on the OS) to avoid the problem --
and is prominently mentionend in the documentation:

http://docs.python.org/2/library/subprocess.html#subprocess.Popen.wait

 

[Nobody]

I have a python script and when I run it directly from the command line
it runs to completion. But I need to run it from another script. I do
that like this:

p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
rv = p.wait()
out_buf = p.stdout.read()

When I do this, wait never returns.

The last two statements are the wrong way around. If you're reading a
process' output via a pipe, you shouldn't wait() for it until it has
closed its end of the pipe.

As it stands, you have a potential deadlock. If the subprocess tries to
write more data than will fit into the pipe, it will block until the
parent reads from the pipe. But the parent won't read from the pipe until
after the subprocess has terminated, which won't happen because the
subprocess is blocked waiting for the parent to read from the pipe ...

 

[Larry.Martell]

(e-mail address removed) wrote:

I have a python script and when I run it directly from the command line it

runs to completion. But I need to run it from another script. I do that

like this:

p = subprocess.Popen(cmd, stdout=subprocess.PIPE,

stderr=subprocess.STDOUT) rv = p.wait()

out_buf = p.stdout.read()

When I do this, wait never returns. If I trace the underlying script it's

always in the same write to stderr that never seems to complete:

write(2, "/KA22/05Feb12/Images/12063LBO003"..., 24457

I run many other scripts and commands in the same manner, and they all

complete, it's just this one. Anyone have any ideas why this is happening,

and how I can further debug or fix this?

The script writes to an OS buffer, and when that buffer is full it blocks

forever. p.wait() in turn then waits forever for the script to terminate...



As a fix try



out_buf = subprocess.Popen(...).communicate()[0]





This uses threads or select (depending on the OS) to avoid the problem --

and is prominently mentionend in the documentation:



http://docs.python.org/2/library/subprocess.html#subprocess.Popen.wait

Thanks. I hadn't seen this. I'll check it out.

 

[Larry.Martell]

The last two statements are the wrong way around. If you're reading a

process' output via a pipe, you shouldn't wait() for it until it has

closed its end of the pipe.



As it stands, you have a potential deadlock. If the subprocess tries to

write more data than will fit into the pipe, it will block until the

parent reads from the pipe. But the parent won't read from the pipe until

after the subprocess has terminated, which won't happen because the

subprocess is blocked waiting for the parent to read from the pipe ...

Thanks. I reversed the order of the wait and read calls, and it no longer hangs.