Input from subprocess called using open() buffered?

[g r a e m e b [at] c a d e n c e [dot] c o m]

Hi,

I have a perl wrapper which runs a command (in this case, cvs) and formats the
output. It presently runs the command something like this:

$| = 1;

open(COMMAND, "$cmd 2>&1 |") || die "Couldn't run command";
while (<COMMAND>) {
print;
}

The problem is that the output from the command appears to arrive in chunks (a
bit like watching 'tail' on a file which is constantly updating). It looks as
though it's getting buffered until it exceeds a certain threshold, then all
coming out at once.

If I run it like this, however, the output comes out 'real time', as it does if
I run it on the command-line manually:

system("$cmd 2>&1");

Is there some way I can read from COMMAND and actually get the output when it
appears, rather than when there is a sizable chunk?

Thanks,
Graeme.

 

[Scott W Gifford]

"g r a e m e b [at] c a d e n c e [dot] c o m" <"g r a e m e b [at] c a d e n c e [dot] c o m"> writes:

[...]

Is there some way I can read from COMMAND and actually get the output
when it appears, rather than when there is a sizable chunk?

You can either look for a way to tell COMMAND not to buffer its
output, perhaps with a command-line flag or environment variable
mentioned in its documentation, or else use a pseudo-tty to make it
think it's talking to a terminal. IIRC, the easiest way to use a
pseudo-tty is to use the Expect or Expect::Simple module.

Good luck!

----ScottG.

 

[Brian McCauley]

open(COMMAND, "$cmd 2>&1 |") || die "Couldn't run command";
while (<COMMAND>) {
print;
}

The problem is that the output from the command appears to arrive in
chunks (a bit like watching 'tail' on a file which is constantly
updating).

It's not that it's arriving in chunks, it's being sent in chunks.

It looks as though it's getting buffered until it exceeds a
certain threshold, then all coming out at once.

That is probably correct. But this is happening in the other process
not Perl.

If I run it like this, however, the output comes out 'real time', as it
does if I run it on the command-line manually:

system("$cmd 2>&1");

Actually that's not entirely true. If Perl processes STDOUT was
anything other than a tty device then I would expect the other process
once again to send its output chunked.

Is there some way I can read from COMMAND and actually get the output
when it appears, rather than when there is a sizable chunk?

You are reading it when it appears. It's just that it's appearing in
chunks. There may be a way to tell the command in question not to chunk
its output but that is a question about the specific command, not Perl.

Failing that, I Expect (hint, hint) there's something on CPAN you could
use to allow Perl to fool the other process into thinking its output was
going to a terminal.

 

[510046470588-0001]

Failing that, I Expect (hint, hint) there's something on CPAN you
could use to allow Perl to fool the other process into thinking its
output was going to a terminal.

IO:ty, which comes with IO::Tty

Klaus Schilling

 

[Joe Smith]

g r a e m e b [at] c a d e n c e [dot] c o m wrote:

open(COMMAND, "$cmd 2>&1 |") || die "Couldn't run command";

while (<COMMAND>) {
print;
}

The problem is that the output from the command appears to arrive in
chunks (a bit like watching 'tail' on a file which is constantly
updating). It looks as though it's getting buffered until it exceeds a
certain threshold, then all coming out at once.

You will see the same symptoms if when running it from the command line.
bash% cmd 2>&1 | cat

Is there some way I can read from COMMAND and actually get the output
when it appears, rather than when there is a sizable chunk?

Most implementations of stdio buffer output whenever it is directed
to anything other than a tty (or a pty). Using a pty instead of a pipe
will avoid this behavior.
-Joe

 

[g r a e m e b [at] c a d e n c e [dot] c o m]

Success!

Thanks for all the replies, I managed to get two solutions which do what I
want, neither of which are probably correct! ;D

The first one, using raw IO:ty, looks a little like this:

my $pty = new IO:ty;
my $ttyname = $pty->ttyname();
print "Spawned tty $ttyname\n";
open(COMMAND, "$cmd > $ttyname 2>&1|") || die "Couldn't run command";
while(<$pty>) {
print ":: $_";
}
close($pty);

I take it that I'm supposed to run the command and direct the IO manually to
the pty I've opened (eg. "cvs update > /dev/pty56 2>&1")? This seems to work
fine, I just want to check I'm using it correctly.

I am a bit baffled as to why I need to have COMMAND as a filehandle - is there
a better way of doing this? And do I need to close both $pty and COMMAND?
Presumably all the output is going to $pty, so I don't even need COMMAND, or do I?


The other method, using Expect, looked a little like this (note that I hacked
this up BEFORE looking at the Expect docs!):

my $process = Expect->spawn("$cmd 2>&1");
while (<$process>) {
print "|| $_";
}
close($process);

It's more succinct than the one above, but when I looked at the docs, it didn't
seem to even mention being able to loop over the filehandle returned (in the
question about "what if I only want to get output without 'expecting'
anything?". Not sure if this works by accident or design!

Any code criticism welcome...

Thanks,
Graeme.

PS. I was running 'cvs update' (and checkout) - it does indeed have the same
effect if I do 'cvs update 2>&1 | cat'. Thanks, I thought this was an obscure
perl thing, didn't realise other programs actually did different things
depending upon the output file...

 

[g r a e m e b [at] c a d e n c e [dot] c o m]

Actually, I tell a lie - none of these work...

Although they receive the output at the right time, neither one of them ever
terminates. I tried checking for EOF on COMMAND, but either this hangs until
EOF is actually received, or if I use fcntl to set it to nonblocking, it always
returns EOF!

There must be a better way of doing this... I presume CVS is block-buffering
because it's usually used across a network, but redirecting the output to a pty
is proving somewhat troublesome! Any help would be appreciated!

Thanks,
Graeme.