IPC::Open2 and sort

[grocery_stocker]

I'm sort of drawing a blank on the following statement found in the in
the documentation for IPC::Open2

"Programs like sort that read their entire input stream first, however,
are quite apt to cause deadlock."

The only think I can possibly think of is that sort is using the
standard I/O default buffering as opposed to the write() function. Am I
close

Chad

 

[xhoster]

I'm sort of drawing a blank on the following statement found in the in
the documentation for IPC::Open2

"Programs like sort that read their entire input stream first, however,
are quite apt to cause deadlock."

The only think I can possibly think of is that sort is using the
standard I/O default buffering as opposed to the write() function. Am I
close

I don't think so. I think the docs are just wrong. What leads to
deadlocks is when the other processes generates output at unpredictable
intervals. "one line in, one line out" (like bc) and "read all the input
first, then produce all the output after" (like sort) are both predictable
and fairly robust against deadlocks (Provided, of course, that you don't do
something stupid).

Xho

 

[Barry Margolin]

I don't think so. I think the docs are just wrong. What leads to
deadlocks is when the other processes generates output at unpredictable
intervals. "one line in, one line out" (like bc) and "read all the input
first, then produce all the output after" (like sort) are both predictable
and fairly robust against deadlocks (Provided, of course, that you don't do
something stupid).

However, you can still run into problems due to stdio buffering. Even
if the application works in "one line in, one line out" mode, stdio may
buffer the output lines. If the driving application tries to read this,
it will block because the buffer hasn't been flushed. But the buffer
won't be flushed until the application produces more output, which won't
happen until the driving application sends more input, but this won't
happen because the application is blocked in read().

I recently wrote an application that uses IPC::Open2, and I resolved
this by forking a child process to do the writing, while the main
process went into a read loop.

 

[zentara]

I'm sort of drawing a blank on the following statement found in the in
the documentation for IPC::Open2

"Programs like sort that read their entire input stream first, however,
are quite apt to cause deadlock."

The only think I can possibly think of is that sort is using the
standard I/O default buffering as opposed to the write() function. Am I
close
Chad

Hi, here is a possible route to a solution.

I was toying around with running bc thru IPC3 (similar to IPC2),
and ran into a problem with huge output.

If I entered a calculation which produced huge output > 4k,
the last semi-full buffer would not get flushed out, leading
to incomplete answers, and problems with the next entry
getting the remnants of the previous query.

Anyways, if you want, see:
http://perlmonks.org?node_id=300420

 

[zentara]

"Programs like sort that read their entire input stream first, however,
are quite apt to cause deadlock."

The only think I can possibly think of is that sort is using the
standard I/O default buffering as opposed to the write() function. Am I
close
Chad

Here is something else to look at. When you run sort from the
command line, it allows you to enter strings on separate lines,
and it only returns when you hit control-d ( send an EOF). Well
sending an EOF from perl, is just closing the input filenandle.

So in IPC, you have to distinquish between pipes which you
must open and close in the loop (i.e. send EOF ) or ones which
you can keep open.

So look at this example:
#!/usr/bin/perl
use warnings;
use strict;
use IPC::Open2;

while(1){
my ($rd, $wr);
open2($rd, $wr, "sort");
print "Hit Enter to sort, control-c to exit\n";

my @Arr = (1..1000000);
my $prompt= <STDIN>;
print $wr "@Arr";
close $wr; # here we send EOF to signal sort to output

my $x = <$rd>;
print $x;
close($rd);
}
__END__

 

[K-mart Cashier]

However, you can still run into problems due to stdio buffering. Even
if the application works in "one line in, one line out" mode, stdio may
buffer the output lines. If the driving application tries to read this,
it will block because the buffer hasn't been flushed. But the buffer
won't be flushed until the application produces more output, which won't
happen until the driving application sends more input, but this won't
happen because the application is blocked in read().

Would this happen if the buffer wasn't full? I was always under the
impression a fully buffered output was flushed when it was filled.
Hmm.... actually as I type this, I'm getting this funny feeling I'm not
drawing a clear distinction between fully and line buffered.

Chad

 

[Uri Guttman]

BM> However, you can still run into problems due to stdio buffering. Even
BM> if the application works in "one line in, one line out" mode, stdio may
BM> buffer the output lines. If the driving application tries to read this,
BM> it will block because the buffer hasn't been flushed. But the buffer
BM> won't be flushed until the application produces more output, which won't
BM> happen until the driving application sends more input, but this won't
BM> happen because the application is blocked in read().

BM> I recently wrote an application that uses IPC::Open2, and I resolved
BM> this by forking a child process to do the writing, while the main
BM> process went into a read loop.

i have two words for you: 'event loop'. i would never run long running
subprocesses and try to deal with their stdio without an event
loop. then you never have to worry about deadlock.

uri

 

[anno4000]

I don't think so. I think the docs are just wrong.

It seems to me the doc supposes (but doesn't explicitly say so) the
user is trying to perform a line-by-line dialog with the program.

What leads to
deadlocks is when the other processes generates output at unpredictable
intervals. "one line in, one line out" (like bc) and "read all the input
first, then produce all the output after" (like sort) are both predictable
and fairly robust against deadlocks (Provided, of course, that you don't do
something stupid).

That's what the doc assumes you're doing.

Anno

 

[xhoster]

It seems to me the doc supposes (but doesn't explicitly say so) the
user is trying to perform a line-by-line dialog with the program.

Well, if they are trying to have a line by line dialog with sort, then
I would say that deadlocks are the least of their problems, so I wouldn't
think it serves as a good example of a deadlock problem.

I'd rather have no example than a misleading one. Of course they already
have the cat -v example, I don't know that they need to have another one.

I think we should just take out the "sort" sentence and combine the rest of
that paragraph with the next paragraph.

Xho

 

[Barry Margolin]

BM> However, you can still run into problems due to stdio buffering. Even
BM> if the application works in "one line in, one line out" mode, stdio may
BM> buffer the output lines. If the driving application tries to read
this,
BM> it will block because the buffer hasn't been flushed. But the buffer
BM> won't be flushed until the application produces more output, which
won't
BM> happen until the driving application sends more input, but this won't
BM> happen because the application is blocked in read().

BM> I recently wrote an application that uses IPC::Open2, and I resolved
BM> this by forking a child process to do the writing, while the main
BM> process went into a read loop.

i have two words for you: 'event loop'. i would never run long running
subprocesses and try to deal with their stdio without an event
loop. then you never have to worry about deadlock.[/QUOTE]

While I suppose you could do that, it makes it hard to make use of many
of the convenience features of Perl. It's nice to be able to write a
simple 'while (<FILEHANDLE>) ...' loop to process the output of the
subprocess.

 

[Uri Guttman]

BM> In article <[email protected]>,

BM> While I suppose you could do that, it makes it hard to make use of many
BM> of the convenience features of Perl. It's nice to be able to write a
BM> simple 'while (<FILEHANDLE>) ...' loop to process the output of the
BM> subprocess.

i have no problems with that as i have done event loops forever and i
like to control my own buffering. not everyone groks them so easily (too
often i see how event loops boggle some) and they want threads or
procedural ways to do things. the answer is to not write low level event
loop code but to use a framework (such as Stem.pm) that does all the
i/o work for you. running a subprocess that way is much cleaner than
using IPC::Open[23] IMNSHO.

and something anonymously irc'ed me about sullying the
comp.unix.programmer group with to my previous post in this thread. all
i can say is that he wasn't kind to certain sexual pursuasions and
accused me of such. too bad for that group that it has such a police dog
guarding it. i can't wait to see its (sic) response to this but i don't
give much credit to anon slander.

uri

 

[Uri Guttman]

UG> and something anonymously irc'ed me about sullying the
UG> comp.unix.programmer group with to my previous post in this thread. all
UG> i can say is that he wasn't kind to certain sexual pursuasions and
UG> accused me of such. too bad for that group that it has such a police dog
UG> guarding it. i can't wait to see its (sic) response to this but i don't
UG> give much credit to anon slander.

this is fun. i have had no problems with usenet or irc trolls but this
one is silly:

23:30 my_boner=quit acting like a little girl on comp.unix.programmer. Gawd.
Must you carry your homosexual wanker tendecies from the IRC to the
newsgroups?

23:33 * my_boner H 5 [email protected] [no]

23:41 my_boner=continue to fag up my USENET experience

is he one of yours? he claims to be a regular but then all the trolls in
comp.lang.perl.misc claim that too.

please police your group better than this. we have had our share of
trolls and keep them locked up in the dungeons most of the time.

uri

 

[Joe Smith]

Would this happen if the buffer wasn't full? I was always under the
impression a fully buffered output was flushed when it was filled.
Hmm.... actually as I type this, I'm getting this funny feeling I'm not
drawing a clear distinction between fully and line buffered.

That's right. STDIO operates a line at at time (flush buffer after "\n")
when output is going to a tty. It waits until the buffer is full when
output is directed to a file or a pipe.
-Joe