subprocess.Popen inheriting

[Aaron Brady]

Hi,

I have a file handle I want to inherit in a child process. I am
looking at '_make_inheritable' in 'Popen', but it needs an instance,
and by the time I have one, the subprocess is already running.

Can't I call 'Popen._make_inheritable( None, handle )'? The method
does not use 'self'.

 

[Gabriel Genellina]

I have a file handle I want to inherit in a child process. I am
looking at '_make_inheritable' in 'Popen', but it needs an instance,
and by the time I have one, the subprocess is already running.

Can't I call 'Popen._make_inheritable( None, handle )'? The method
does not use 'self'.

File handles are inherited by default, I think. What's your specific
problem?

 

[Aaron Brady]

File handles are inherited by default, I think. What's your specific  
problem?

I want a dual-console application for some multi-threaded output. The
main process spawns a second process in a second window, and directs
its readout through a pipe. That is, the second process is just a
dummy, print pipe directly to console.

It worked when I used the 'stdin= PIPE' keyword in Popen, but passing
my own pipe handle on the command line isn't working. The keyword is
a workaround.

File handles are inherited by default, I think.

I thought so too. The web seems to say that on Linux they are, and on
Windows, you need to call DuplicateHandle for it.

By the way, there are a few solutions to the inheritance problem.

1. Override __init__ and __del__ in a subclass.
2. Assign DummyClass= Popen._make_inheritable
3. Override Popen._execute_child to call _make_inheritable, then call
the super method.

 

[Jeremy Sanders]

I thought so too. The web seems to say that on Linux they are, and on
Windows, you need to call DuplicateHandle for it.

I hit this problem - it looks like pipes aren't very versatile on Windows.
There's also the complicating factor that the handles in windows aren't the
same as the file numbers that Python uses, so you have to convert between
them.

It would be nice if Python created pipes that are properly inheritable by
default by child processes, as they're mostly used for IPC.

It was so painful that I converted my code to use sockets instead, which
seem much more portable between Windows and Unix (though you don't get to
use socketpair and AF_UNIX in Windows).

Jeremy

 

[Gabriel Genellina]

En Wed, 17 Dec 2008 12:21:38 -0200, Jeremy Sanders

Or set bInheritHandle=True when creating the pipe initially. os.pipe()
doesn't do that.

I hit this problem - it looks like pipes aren't very versatile on
Windows.
There's also the complicating factor that the handles in windows aren't
the
same as the file numbers that Python uses, so you have to convert between
them.

It would be nice if Python created pipes that are properly inheritable by
default by child processes, as they're mostly used for IPC.

I'd say it is a bug in os.pipe implementation; they should be inheritable
by default, as in posix (after all, the code is in "posixmodule.c").

 

[Aaron Brady]

En Wed, 17 Dec 2008 12:21:38 -0200, Jeremy Sanders  


Or set bInheritHandle=True when creating the pipe initially. os.pipe()  
doesn't do that.



I'd say it is a bug in os.pipe implementation; they should be inheritable  
by default, as in posix (after all, the code is in "posixmodule.c").

The code looks like this:

ok = CreatePipe(&read, &write, NULL, 0);
Py_END_ALLOW_THREADS
if (!ok)
return win32_error("CreatePipe", NULL);
read_fd = _open_osfhandle((Py_intptr_t)read, 0);
write_fd = _open_osfhandle((Py_intptr_t)write, 1);

'If lpPipeAttributes is NULL, the handle cannot be inherited.' You
could populate a 'SECURITY_ATTRIBUTES' structure, or call
DuplicateHandle on both of them.

A patch would look like this:

SECURITY_ATTRIBUTES sattribs;
sattribs.nLength = sizeof(sattribs);
sattribs.lpSecurityDescriptor = NULL;
sattribs.bInheritHandle = TRUE;
ok = CreatePipe(&read, &write, &sattribs, 0);

This still doesn't answer whether the file descriptor return by
'_open_osfhandle' can be inherited too.

 

[Gabriel Genellina]

The code looks like this:

ok = CreatePipe(&read, &write, NULL, 0);
Py_END_ALLOW_THREADS
if (!ok)
return win32_error("CreatePipe", NULL);
read_fd = _open_osfhandle((Py_intptr_t)read, 0);
write_fd = _open_osfhandle((Py_intptr_t)write, 1);

'If lpPipeAttributes is NULL, the handle cannot be inherited.' You
could populate a 'SECURITY_ATTRIBUTES' structure, or call
DuplicateHandle on both of them.

A patch would look like this:

SECURITY_ATTRIBUTES sattribs;
sattribs.nLength = sizeof(sattribs);
sattribs.lpSecurityDescriptor = NULL;
sattribs.bInheritHandle = TRUE;
ok = CreatePipe(&read, &write, &sattribs, 0);

Yes, that's exactly how os.popen does it (in posixmodule.c)

This still doesn't answer whether the file descriptor return by
'_open_osfhandle' can be inherited too.

It doesn't matter. The OS only cares about file handles, not C RTL
structures.

 

[Aaron Brady]

En Wed, 17 Dec 2008 22:46:32 -0200, Aaron Brady <[email protected]>  
escribió:









Yes, that's exactly how os.popen does it (in posixmodule.c)


It doesn't matter. The OS only cares about file handles, not C RTL  
structures.

Ah, I see. Was it an executive decision about what is Pythonic, or
just a bug? Do you think the patch would be accepted? I probably
ought to mimic a small Python embedding to see if it needs anything
else.

 

[Aaron Brady]

En Wed, 17 Dec 2008 22:46:32 -0200, Aaron Brady <[email protected]>  
escribió:









Yes, that's exactly how os.popen does it (in posixmodule.c)


It doesn't matter. The OS only cares about file handles, not C RTL  
structures.

Sorry for the multiple posts. File handles are inheritable by child
processes, if the permissions are right. File descriptors are not.
Is there a way that we can get the handles of a pipe into code, so
that we can pass them to a subprocess? Will it take calling
'CreatePipe' from ctypes directly if on Windows? Or can 'os.pipe' be
made to abstract that? If Windows can't inherit descriptors,
'os.pipe' should return handles, and 'os.read' &co. should accept
them.

It is a fairly large patch.

 

[Gabriel Genellina]

Ah, I see. Was it an executive decision about what is Pythonic, or
just a bug? Do you think the patch would be accepted? I probably
ought to mimic a small Python embedding to see if it needs anything
else.

I don't know - I guess someone (years ago) blindly just replaced the
pipe() system call by a CreatePipe call without further analysis.

This is how I would summarize the issue:

Pros (of changing os.pipe() to return inheritable pipes):

- it isn't explicitely documented whether os.pipe() returns inheritable
pipes or not, so both versions are "right" according to the documentation.
- if someone relies on pipes being non-inheritable on Windows, that is
undocumented behaviour, and Python has the right to change it.
- the change would improve POSIX compatibility, it mimics what os.pipe()
does on those OS.
- inheritable pipes are less surprising for guys coming from other OS
- inheritable pipes are a lot more useful than non-inheritable ones when
doing IPC (probably its main usage).

Cons:

- os.pipe has behaved that way since long time ago.
- some programs *might* break, if they relied on pipes being
non-inheritable on Windows, even if that was undocumented behaviour.

 

[Gabriel Genellina]

Sorry for the multiple posts. File handles are inheritable by child
processes, if the permissions are right. File descriptors are not.
Is there a way that we can get the handles of a pipe into code, so
that we can pass them to a subprocess?

On Windows, file handles are the real OS stuff, the "true" reference to an
open file. File descriptors are not, they exist only to please the C
runtime library. Programs not written in C (directly, or indirectly like
Python) don't care at all about file descriptors. And in case one actually
cares, there is _open_osfhandle in the C RTL (available as
msvcrt.open_osfhandle from Python).
A subprocess may inherit handles from its parent [there are two filters:
the parameter "bInheritHandles" in the CreateProcess call provides global
control, and individual handles can be made inheritable or not, before
creating the new subprocess].
"Anonymous" pipes are good to replace stdin/stdout/stderr, because there
is no need to explicitely communicate the handle value to the subprocess:
one just replaces the corresponding handle with the desired pipe, and the
subprocess might not even notice it.
In case this is not enough, one might pass the handle (as a number) in the
command line, but probably a "named pipe" would be better. As this is not
transparent for the child process, one must explicitely code such things.

Will it take calling
'CreatePipe' from ctypes directly if on Windows? Or can 'os.pipe' be
made to abstract that? If Windows can't inherit descriptors,
'os.pipe' should return handles, and 'os.read' &co. should accept
them.

I think the best way would be to modify os.pipe so it returns inheritable
pipes, as it should have been from the beginning.

It is a fairly large patch.

Not at all, you have already posted most of it.

 

[Aaron Brady]

En Thu, 18 Dec 2008 08:35:58 -0200, Aaron Brady <[email protected]>
escribió:





I don't know - I guess someone (years ago) blindly just replaced the
pipe() system call by a CreatePipe call without further analysis.

This is how I would summarize the issue:

Pros (of changing os.pipe() to return inheritable pipes):

- it isn't explicitely documented whether os.pipe() returns inheritable
pipes or not, so both versions are "right" according to the documentation..
- if someone relies on pipes being non-inheritable on Windows, that is
undocumented behaviour, and Python has the right to change it.
- the change would improve POSIX compatibility, it mimics what os.pipe()
does on those OS.
- inheritable pipes are less surprising for guys coming from other OS
- inheritable pipes are a lot more useful than non-inheritable ones when
doing IPC (probably its main usage).

Cons:

- os.pipe has behaved that way since long time ago.
- some programs *might* break, if they relied on pipes being
non-inheritable on Windows, even if that was undocumented behaviour.

Hi,

Microsoft has this example:

http://msdn.microsoft.com/en-us/library/aa298531.aspx

It creates two descriptors (not handles) with '_pipe'. Then it spawns
a subprocess using 'spawnl', which correctly inherits a descriptor.
So, if 'Popen' could mimic 'spawnl', then 'os.pipe', and consequently,
'os.read', 'os.write', &c. could stay as is.

 

[Aaron Brady]

En Thu, 18 Dec 2008 19:46:45 -0200, Aaron Brady <[email protected]>  
escribió: snip
On Windows, file handles are the real OS stuff, the "true" reference to an  
open file. File descriptors are not, they exist only to please the C  
runtime library. Programs not written in C (directly, or indirectly like  
Python) don't care at all about file descriptors. And in case one actually  
cares, there is _open_osfhandle in the C RTL (available as  
msvcrt.open_osfhandle from Python).
A subprocess may inherit handles from its parent [there are two filters:  
the parameter "bInheritHandles" in the CreateProcess call provides global  
control, and individual handles can be made inheritable or not, before  
creating the new subprocess].
"Anonymous" pipes are good to replace stdin/stdout/stderr, because there  
is no need to explicitely communicate the handle value to the subprocess:  
one just replaces the corresponding handle with the desired pipe, and the  
subprocess might not even notice it.
In case this is not enough, one might pass the handle (as a number) in the  
command line, but probably a "named pipe" would be better. As this is not  
transparent for the child process, one must explicitely code such things.

Will it take calling
'CreatePipe' from ctypes directly if on Windows?  Or can 'os.pipe' be
made to abstract that?  If Windows can't inherit descriptors,
'os.pipe' should return handles, and 'os.read' &co. should accept
them.

I think the best way would be to modify os.pipe so it returns inheritable  
pipes, as it should have been from the beginning.

It is a fairly large patch.

Not at all, you have already posted most of it.

I have marginally tested the patch on a custom build. It works, but
there is a catch. The descriptor can't be passed directly to the
child on the command line. You need to call 'msvcrt.get_osfhandle' on
the descriptor, pass the result, then call 'msvcrt.open_osfhandle' in
the child.

 

[Aaron Brady]

I have marginally tested the patch on a custom build.  It works, but
there is a catch.  The descriptor can't be passed directly to the
child on the command line.  You need to call 'msvcrt.get_osfhandle' on
the descriptor, pass the result, then call 'msvcrt.open_osfhandle' in
the child.

Issue is at:
http://bugs.python.org/issue4708