spawn process with bidirectional pipe (std/stdout)

[Gernot Frisch]

Hi,

I want to create a process and get a (or 2) FILE* to fwrite() to its stdin and fread() its stdout, but cross platform.

I found a windows only way:
http://support.microsoft.com/kb/190351/en-us

But all unix version seem to rely on fork(), which is total overkill if you fork() a big process (or am I wrong? - I'm a windows
guy).

So, is there a way to get a read/write handle to a process without fork?

Thank you,

 

[red floyd]

Hi,

I want to create a process and get a (or 2) FILE* to fwrite() to its stdin and fread() its stdout, but cross platform.

1. By definition that's OS dependent. You're kind of OT here. But
for half-duplex
(read only or write only), look at popen() on unix.

I found a windows only way:http://support.microsoft.com/kb/190351/en-us

But all unix version seem to rely on fork(), which is total overkill if you fork() a big process (or am I wrong? - I'm a windows
guy).

2. And how, pray tell, do you plan on creating a new process, other
than fork()?

 

[Jens Thoms Toerring]

I want to create a process and get a (or 2) FILE* to fwrite() to its stdin and fread() its stdout, but cross platform.

I found a windows only way:
http://support.microsoft.com/kb/190351/en-us

But all unix version seem to rely on fork(), which is total overkill if
you fork() a big process (or am I wrong? - I'm a windows guy).

fork() nowadays (and already for quite some time) uses copy-
on-write, i.e. only those pages in memory that are being writ-
ten to by one of the processes are copied, so fork() does not
copy the whole process memory but only sucessively and as far
as necessary. So it's not the overkill you may suspect it to
be.

So, is there a way to get a read/write handle to a process without fork?

You can't. If you need a new process you need to call fork(),
that's how processes are created (well, there's also vfork(),
but that you definitely don't want to use).

For the details of the redirections of the file descriptors
and then the FILE pointers etc. involved I would think it
would be wiser to ask in e.g. comp.unix.programmer where it
is as on-topic as you can get;-)
Regards, Jens

 

[Joshua Maurice]

Hi,

I want to create a process and get a (or 2) FILE* to fwrite() to its stdin and fread() its stdout, but cross platform.

I found a windows only way:http://support.microsoft.com/kb/190351/en-us

But all unix version seem to rely on fork(), which is total overkill if you fork() a big process (or am I wrong? - I'm a windows
guy).

So, is there a way to get a read/write handle to a process without fork?

Thank you,

For POSIX systems, your only real option is fork. (There's a couple
more obscure ones which you don't want to use, or which are basically
wrappers on top of fork, such as popen and posix_spawn.)

IMHO, this is a broken state of affairs for a variety of reasons.

My biggest annoyance is that it's incredibly hard to ensure no
resource is "leaked" across a fork in a quick and portable way, and
the extra memory required for forking. Almost the entire point of
processes is fault isolation. The entire OS and the hardware chip
itself is built with this goal in mind, e.g. kernel mode and user
mode. The default usage of fork makes it incredibly easy, standard
practice even, to leak resources across a fork, which is quite
contradictory. I think the most notable thing is that it's a huge
gaping security hole when files, mmaps, etc., can be leaked from one
process to another so easily.

As mentioned else-thread, most windows / mac / unix systems implement
fork so that its virtual pages are copy on write, and this helps a
lot, but I would mention that still has some negative side effects.
Either you have overcommit on, in which case if your system is loaded,
random programs will be killed with great prejudice (aka very bad), or
you have overcommit off, which means that while a fork will not copy
all of the contents of the virtual pages, it will commit those virtual
pages, which can fail if your system is loaded and near the commit
limit. At least with overcommit off, the process gets an error when it
tries to get a new virtual page as opposed to a random process just
being instantly arbitrarily killed. I suppose the problem with
overcommit off can be solved with a larger swap partition size if you
know the size of your workload, so I guess that it's not that big of a
deal. I just wanted to make it clear.

To finish this rant, I don't mind fork existing. I just want another
primitive which spawns a process from an executable file in a clean
environment, so I can choose to be easily leak-free if so desired. I'd
prefer this new spawn would be the default for developers as well. In
the end, what fork has over this new method is simply ease of use.
With my spawn primitive, you would have to put the code for the child
process which sets up the environment into its own cpp file and
compile it into its own executable, whereas with the fork you can put
the code inline. Saves some keystrokes only.

Off topic, I know. Sorry. Rant done.

PS: Is windows broken in the same way with regards to file handles? I
can't parse the msdn website very well in this regard, but from what I
can gather, /I think that/ windows has the same resource leak problem
as unix style forks - all file handles are inherited by default into
the child, and there's no easy way to say "Only inherit these specific
X file handles".

 

[Marc]

For POSIX systems, your only real option is fork. (There's a couple
more obscure ones which you don't want to use, or which are basically
wrappers on top of fork, such as popen and posix_spawn.)

Why discard popen and posix_spawn so fast? Properly implemented, they
address some of your concerns, like the one about overcommit.

 

[Miles Bader]

But all unix version seem to rely on fork(), which is total overkill
if you fork() a big process (or am I wrong? - I'm a windows guy).

Fork+exec is quite efficient. You shouldn't worry about it unless you
actually have specific data showing a problem for your usage.

So, is there a way to get a read/write handle to a process without fork?

You can use "sendmsg" to send an open file descriptor to
an existing process, but of course you've got to create the receiving
process first...

-Miles

--
Mad, adj. Affected with a high degree of intellectual independence; not
conforming to standards of thought, speech, and action derived by the
conformants [sic] from study of themselves; at odds with the majority; in
short, unusual. It is noteworthy that persons are pronounced mad by officials
destitute of evidence that they themselves are sane.

 

[Joshua Maurice]

Joshua Maurice  wrote:

Why discard popen and posix_spawn so fast? Properly implemented, they
address some of your concerns, like the one about overcommit.

True, but they do nothing for my resource "leaking" / "bleeding"
problem. I was straightforward about my concerns, I think.

 

[Jorgen Grahn]

Hi,

I want to create a process and get a (or 2) FILE* to
fwrite() to its stdin and fread() its stdout, but cross platform.

I found a windows only way:
http://support.microsoft.com/kb/190351/en-us

But all unix version seem to rely on fork(), which is total overkill
if you fork() a big process (or am I wrong? - I'm a windows
guy).

There is some misunderstanding here. The way to create a true process
on Unix /is/ fork(); it cannot by definition be overkill to create a
process by that method. Did you mean a thread, of do you confuse
fork() with exec()?

Also note that it's very easy to run into a deadlock when you set up
double pipes like this -- especially with the buffering done by FILE*.
You may end up with process A being blocked when writing to B, because
B is blocked trying to write to A.

And finally ... all this is offtopic here.

/Jorgen