Input from subprocess called using open() buffered?

Discussion in 'Perl Misc' started by g r a e m e b [at] c a d e n c e [dot] c o m, Aug 6, 2004.

  1. Hi,

    I have a perl wrapper which runs a command (in this case, cvs) and formats the
    output. It presently runs the command something like this:

    $| = 1;

    open(COMMAND, "$cmd 2>&1 |") || die "Couldn't run command";
    while (<COMMAND>) {
    print;
    }

    The problem is that the output from the command appears to arrive in chunks (a
    bit like watching 'tail' on a file which is constantly updating). It looks as
    though it's getting buffered until it exceeds a certain threshold, then all
    coming out at once.

    If I run it like this, however, the output comes out 'real time', as it does if
    I run it on the command-line manually:

    system("$cmd 2>&1");

    Is there some way I can read from COMMAND and actually get the output when it
    appears, rather than when there is a sizable chunk?

    Thanks,
    Graeme.
     
    g r a e m e b [at] c a d e n c e [dot] c o m, Aug 6, 2004
    #1
    1. Advertising

  2. "g r a e m e b [at] c a d e n c e [dot] c o m" <"g r a e m e b [at] c a d e n c e [dot] c o m"> writes:

    [...]

    > Is there some way I can read from COMMAND and actually get the output
    > when it appears, rather than when there is a sizable chunk?


    You can either look for a way to tell COMMAND not to buffer its
    output, perhaps with a command-line flag or environment variable
    mentioned in its documentation, or else use a pseudo-tty to make it
    think it's talking to a terminal. IIRC, the easiest way to use a
    pseudo-tty is to use the Expect or Expect::Simple module.

    Good luck!

    ----ScottG.
     
    Scott W Gifford, Aug 6, 2004
    #2
    1. Advertising

  3. g r a e m e b [at] c a d e n c e [dot] c o m wrote:

    > open(COMMAND, "$cmd 2>&1 |") || die "Couldn't run command";
    > while (<COMMAND>) {
    > print;
    > }
    >
    > The problem is that the output from the command appears to arrive in
    > chunks (a bit like watching 'tail' on a file which is constantly
    > updating).


    It's not that it's arriving in chunks, it's being sent in chunks.

    > It looks as though it's getting buffered until it exceeds a
    > certain threshold, then all coming out at once.


    That is probably correct. But this is happening in the other process
    not Perl.

    > If I run it like this, however, the output comes out 'real time', as it
    > does if I run it on the command-line manually:
    >
    > system("$cmd 2>&1");


    Actually that's not entirely true. If Perl processes STDOUT was
    anything other than a tty device then I would expect the other process
    once again to send its output chunked.

    > Is there some way I can read from COMMAND and actually get the output
    > when it appears, rather than when there is a sizable chunk?


    You are reading it when it appears. It's just that it's appearing in
    chunks. There may be a way to tell the command in question not to chunk
    its output but that is a question about the specific command, not Perl.

    Failing that, I Expect (hint, hint) there's something on CPAN you could
    use to allow Perl to fool the other process into thinking its output was
    going to a terminal.
     
    Brian McCauley, Aug 6, 2004
    #3
  4. g r a e m e b [at] c a d e n c e [dot] c o m

    Guest

    Brian McCauley <> writes:
    >
    > Failing that, I Expect (hint, hint) there's something on CPAN you
    > could use to allow Perl to fool the other process into thinking its
    > output was going to a terminal.



    IO::pty, which comes with IO::Tty

    Klaus Schilling
     
    , Aug 6, 2004
    #4
  5. g r a e m e b [at] c a d e n c e [dot] c o m

    Joe Smith Guest

    g r a e m e b [at] c a d e n c e [dot] c o m wrote:

    open(COMMAND, "$cmd 2>&1 |") || die "Couldn't run command";
    > while (<COMMAND>) {
    > print;
    > }
    >
    > The problem is that the output from the command appears to arrive in
    > chunks (a bit like watching 'tail' on a file which is constantly
    > updating). It looks as though it's getting buffered until it exceeds a
    > certain threshold, then all coming out at once.


    You will see the same symptoms if when running it from the command line.
    bash% cmd 2>&1 | cat

    > Is there some way I can read from COMMAND and actually get the output
    > when it appears, rather than when there is a sizable chunk?


    Most implementations of stdio buffer output whenever it is directed
    to anything other than a tty (or a pty). Using a pty instead of a pipe
    will avoid this behavior.
    -Joe
     
    Joe Smith, Aug 7, 2004
    #5
  6. Success! :)

    Thanks for all the replies, I managed to get two solutions which do what I
    want, neither of which are probably correct! ;D

    The first one, using raw IO::pty, looks a little like this:

    my $pty = new IO::pty;
    my $ttyname = $pty->ttyname();
    print "Spawned tty $ttyname\n";
    open(COMMAND, "$cmd > $ttyname 2>&1|") || die "Couldn't run command";
    while(<$pty>) {
    print ":: $_";
    }
    close($pty);

    I take it that I'm supposed to run the command and direct the IO manually to
    the pty I've opened (eg. "cvs update > /dev/pty56 2>&1")? This seems to work
    fine, I just want to check I'm using it correctly.

    I am a bit baffled as to why I need to have COMMAND as a filehandle - is there
    a better way of doing this? And do I need to close both $pty and COMMAND?
    Presumably all the output is going to $pty, so I don't even need COMMAND, or do I?


    The other method, using Expect, looked a little like this (note that I hacked
    this up BEFORE looking at the Expect docs!):

    my $process = Expect->spawn("$cmd 2>&1");
    while (<$process>) {
    print "|| $_";
    }
    close($process);

    It's more succinct than the one above, but when I looked at the docs, it didn't
    seem to even mention being able to loop over the filehandle returned (in the
    question about "what if I only want to get output without 'expecting'
    anything?". Not sure if this works by accident or design!

    Any code criticism welcome...

    Thanks,
    Graeme.

    PS. I was running 'cvs update' (and checkout) - it does indeed have the same
    effect if I do 'cvs update 2>&1 | cat'. Thanks, I thought this was an obscure
    perl thing, didn't realise other programs actually did different things
    depending upon the output file...


    wrote:
    > Brian McCauley <> writes:
    >
    >>Failing that, I Expect (hint, hint) there's something on CPAN you
    >>could use to allow Perl to fool the other process into thinking its
    >>output was going to a terminal.

    >
    >
    >
    > IO::pty, which comes with IO::Tty
    >
    > Klaus Schilling
     
    g r a e m e b [at] c a d e n c e [dot] c o m, Aug 9, 2004
    #6
  7. Actually, I tell a lie - none of these work... :(

    Although they receive the output at the right time, neither one of them ever
    terminates. I tried checking for EOF on COMMAND, but either this hangs until
    EOF is actually received, or if I use fcntl to set it to nonblocking, it always
    returns EOF!

    There must be a better way of doing this... I presume CVS is block-buffering
    because it's usually used across a network, but redirecting the output to a pty
    is proving somewhat troublesome! Any help would be appreciated!

    Thanks,
    Graeme.

    g r a e m e b [at] c a d e n c e [dot] c o m wrote:
    > Success! :)
    >
    > Thanks for all the replies, I managed to get two solutions which do what
    > I want, neither of which are probably correct! ;D
    >
    > The first one, using raw IO::pty, looks a little like this:
    >
    > my $pty = new IO::pty;
    > my $ttyname = $pty->ttyname();
    > print "Spawned tty $ttyname\n";
    > open(COMMAND, "$cmd > $ttyname 2>&1|") || die "Couldn't run command";
    > while(<$pty>) {
    > print ":: $_";
    > }
    > close($pty);
    >
    > I take it that I'm supposed to run the command and direct the IO
    > manually to the pty I've opened (eg. "cvs update > /dev/pty56 2>&1")?
    > This seems to work fine, I just want to check I'm using it correctly.
    >
    > I am a bit baffled as to why I need to have COMMAND as a filehandle - is
    > there a better way of doing this? And do I need to close both $pty and
    > COMMAND? Presumably all the output is going to $pty, so I don't even
    > need COMMAND, or do I?
    >
    >
    > The other method, using Expect, looked a little like this (note that I
    > hacked this up BEFORE looking at the Expect docs!):
    >
    > my $process = Expect->spawn("$cmd 2>&1");
    > while (<$process>) {
    > print "|| $_";
    > }
    > close($process);
    >
    > It's more succinct than the one above, but when I looked at the docs, it
    > didn't seem to even mention being able to loop over the filehandle
    > returned (in the question about "what if I only want to get output
    > without 'expecting' anything?". Not sure if this works by accident or
    > design!
    >
    > Any code criticism welcome...
    >
    > Thanks,
    > Graeme.
    >
    > PS. I was running 'cvs update' (and checkout) - it does indeed have the
    > same effect if I do 'cvs update 2>&1 | cat'. Thanks, I thought this was
    > an obscure perl thing, didn't realise other programs actually did
    > different things depending upon the output file...
    >
    >
    > wrote:
    >
    >> Brian McCauley <> writes:
    >>
    >>> Failing that, I Expect (hint, hint) there's something on CPAN you
    >>> could use to allow Perl to fool the other process into thinking its
    >>> output was going to a terminal.

    >>
    >>
    >>
    >>
    >> IO::pty, which comes with IO::Tty
    >>
    >> Klaus Schilling

    >
    >
     
    g r a e m e b [at] c a d e n c e [dot] c o m, Aug 10, 2004
    #7
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. dhaval dalal via JavaKB.com
    Replies:
    0
    Views:
    457
    dhaval dalal via JavaKB.com
    Feb 24, 2005
  2. Peter Ashford

    Buffered image using float data

    Peter Ashford, May 2, 2005, in forum: Java
    Replies:
    5
    Views:
    1,831
    Peter Ashford
    May 2, 2005
  3. Replies:
    9
    Views:
    686
    Michael Wojcik
    Aug 23, 2005
  4. hiral
    Replies:
    2
    Views:
    605
    Jean-Michel Pichavant
    May 5, 2010
  5. Kingsley Turner
    Replies:
    1
    Views:
    389
    Diez B. Roggisch
    Oct 14, 2010
Loading...

Share This Page