Problems with background processes on Windows

[geoff.bache]

Hi all,

The following code behaves differently on Windows and Linux using
Python 2.5.2. The Linux behaviour is what I expect in both places
Perhaps somebody could help explain this. Or maybe it is a Python bug.
Or a Windows feature...

---- communicate.py ---

import subprocess
p = subprocess.Popen([ "python", "sleep.py" ], stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
p.communicate()
-----------------------
---- sleep.py ----

import subprocess, os
subprocess.Popen([ 'python', '-c', 'import time; time.sleep(10)' ],
stdin=open(os.devnull), stdout=open(os.devnull, "w"),
stderr=subprocess.STDOUT)
--------------------

In short, we start a subprocess which in turn starts a different
subprocess and then returns without waiting for it. We then try to
"communicate" with the first subprocess.

On Windows if I run "communicate.py" it does not return for 10
seconds, i.e. until the "grandchild" background process terminates. On
Linux it returns immediately as I would expect.

If I replace "p.communicate()" with "p.wait()" it returns immediately
on both. If I don't point stdin, stdout and stderr of the child
process to os.devnull then it will wait 10 seconds on Linux also,
which I'd also expect as we can't collect info from these places if
they've been forwarded elsewhere. It seems like Windows somehow
doesn't notice this.

Any help gratefully appreciated.

Regards,
Geoff Bache

 

[geoffbache]

Hi Tim,

If you trace through this:
    python -m trace --trace communicate.py

you'll see that it hangs in subprocess in the stdout_thread waiting for
stdout to close.

Thanks for this tip, haven't used this before.

I'm not sure I expect this to work as you expect.  When you open a null
device, it's just another file handle.  Why should the behavior be
different?

Well yes, but the point is surely that the standard output of the
background sleeping process is pointed to a different location? (you
can replace the null device with a file name of your choice and the
point is the same.) This process should not have any connection to the
standard output of sleep.py, and hence we shouldn't need to wait for
it to finish when collecting the standard output of sleep.py, surely?
(Even explicitly calling sys.stdout.close() in sleep.py doesn't seem
to help)

Also, the code behaves differently on Linux and Windows, and I haven't
called anything that is documented as behaving differently on
different platforms. If it's a genuine OS difference I'd be interested
in hearing why.

To put this another way: what can I do in sleep.py that allows me to
start a time-consuming background process and exit before it's
complete, while not forcing a process that is "communicating" with me
to also wait for this background process before proceeding?

Regards,
Geoff

 

[Gabriel Genellina]

Hi Tim,


Thanks for this tip, haven't used this before.


Well yes, but the point is surely that the standard output of the
background sleeping process is pointed to a different location? (you
can replace the null device with a file name of your choice and the
point is the same.) This process should not have any connection to the
standard output of sleep.py, and hence we shouldn't need to wait for
it to finish when collecting the standard output of sleep.py, surely?
(Even explicitly calling sys.stdout.close() in sleep.py doesn't seem
to help)

Thesis: When the subprocess module creates the child process, it inherits
the stdin/stdout/stderr handles from its parent (even if its own
stdin/stdout/stderr are redirected; they're different). Until the
grandchild process finishes, the grandparent stdout.read() won't return,
because the pipe isn't closed until the last handle to it is closed.

Another option is that subprocess.py leaks file handles (I remember it
uses DuplicateHandle).

Unfortunately I won't be able to test either of these until next week.

Also, the code behaves differently on Linux and Windows, and I haven't
called anything that is documented as behaving differently on
different platforms. If it's a genuine OS difference I'd be interested
in hearing why.

Well, subprocess has completely separate implementations for Windows and
Linux - it tries hard to hide the differences, but they're not the same
thing.

 

[Gabriel Genellina]

En Sat, 28 Mar 2009 06:03:33 -0300, geoffbache <geoff.bache <at> jeppesen.com>
escribió:

Thesis: When the subprocess module creates the child process, it inherits
the stdin/stdout/stderr handles from its parent (even if its own
stdin/stdout/stderr are redirected; they're different). Until the
grandchild process finishes, the grandparent stdout.read() won't return,
because the pipe isn't closed until the last handle to it is closed.

I've confirmed the above description.

--- begin p0.py ---
import subprocess,os

p1 = subprocess.Popen(["python", "p1.py"],
stdout=subprocess.PIPE,
stderr=open(os.devnull, "wt"),
stdin=open(os.devnull))
print p1.communicate()
--- end p0.py ---

--- begin p1.py ---
import subprocess,sys,os,msvcrt

subprocess.Popen(
["python", "p2.py", str(msvcrt.get_osfhandle(sys.stdout.fileno()))],
stdout=open(os.devnull, "wt"),
stderr=open(os.devnull, "wt"),
stdin=open(os.devnull, "rt"))
print "exit p1.py"
--- end p1.py ---

--- begin p2.py ---
import sys, win32api, time, os

with open("p2.pid","wt") as f: f.write("%d" % os.getpid())
win32api.CloseHandle(int(sys.argv[1]))
time.sleep(30)
--- end p2.py ---

p2 has to close the inherited file handle corresponding to p1's stdout. Then,
when p1 itself finishes, the writing end of the pipe is actually closed and p0
can continue.

C:\TEMP\subp>python p0.py
('exit p1.py\r\n', None)

C:\TEMP\subp>type p2.pid
3018
C:\TEMP\subp>tasklist | find "python.exe"
python.exe 3018 0 4.304 KB

I'm unsure this could be considered a bug in subprocess - the documentation
says that parameter close_fds=True is not supported on Windows (so, the child
process inherits all open files, and this includes the parent's
stdin/stdout/stderr).

At the end, this is due to the fact that file handles 0, 1, 2 have no special
significance on Windows - it's the C runtime library which makes such things
special, not the OS.

 

[geoffbache]

Gabriel Genellina <gagsl-py2 <at> yahoo.com.ar> writes:

En Sat, 28 Mar 2009 06:03:33 -0300, geoffbache <geoff.bache <at> jeppesen.com>  
escribió:

Thesis: When the subprocess module creates the child process, it inherits  
the stdin/stdout/stderr handles from its parent (even if its own  
stdin/stdout/stderr are redirected; they're different). Until the  
grandchild process finishes, the grandparent stdout.read() won't return,  
because the pipe isn't closed until the last handle to it is closed.

I've confirmed the above description.

--- begin p0.py ---
import subprocess,os

p1 = subprocess.Popen(["python", "p1.py"],
      stdout=subprocess.PIPE,
      stderr=open(os.devnull, "wt"),
      stdin=open(os.devnull))
print p1.communicate()
--- end p0.py ---

--- begin p1.py ---
import subprocess,sys,os,msvcrt

subprocess.Popen(
    ["python", "p2.py", str(msvcrt.get_osfhandle(sys.stdout.fileno()))],
    stdout=open(os.devnull, "wt"),
    stderr=open(os.devnull, "wt"),
    stdin=open(os.devnull, "rt"))
print "exit p1.py"
--- end p1.py ---

--- begin p2.py ---
import sys, win32api, time, os

with open("p2.pid","wt") as f: f.write("%d" % os.getpid())
win32api.CloseHandle(int(sys.argv[1]))
time.sleep(30)
--- end p2.py ---

p2 has to close the inherited file handle corresponding to p1's stdout. Then,
when p1 itself finishes, the writing end of the pipe is actually closed and p0
can continue.

C:\TEMP\subp>python p0.py
('exit p1.py\r\n', None)

C:\TEMP\subp>type p2.pid
3018
C:\TEMP\subp>tasklist | find "python.exe"
python.exe                  3018                         0     4.304 KB

I'm unsure this could be considered a bug in subprocess - the documentation
says that parameter close_fds=True is not supported on Windows (so, the child
process inherits all open files, and this includes the parent's
stdin/stdout/stderr).

At the end, this is due to the fact that file handles 0, 1, 2 have no special
significance on Windows - it's the C runtime library which makes such things
special, not the OS.

Thanks for this investigation.

I'm not the right person to pronounce on whether this is a Python bug
or not but I certainly found the behaviour surprising and unhelpful.
When you redirect output you expect it to be redirected, not for some
connection to be invisibly maintained to the original
location. I still don't really understand why sys.stdout.close() in p1
doesn't help either, is there some other way I can just ditch the
standard output from within p1?

Actually, I'm quite happy if somebody can tell me how to work around
it, this is just a test I wrote to simulate some behaviour in my
system (which is the thing calling communicate(), i.e. p0). So both
the child process and grandchild process are just part of the test
(the real versions aren't python programs and p2 runs remotely).

Is there anything I can do to either of these processes to fix this
issue without importing non-standard libraries (i.e. import win32api
is out)?

/Geoff