IPC::Open2::open2() -- How to pass strings stdin, stdout?

Discussion in 'Perl Misc' started by Jerry Krinock, Sep 8, 2009.

  1. Markdown.pl is a Perl text-processing filter which takes input from
    stdin and outputs to stdout. I'd like to call it as a child process
    from within my script -- pass it a string and get a string back. I've
    come to believe that this requires something like IPC::Open2:eek:pen2()
    and have written the following code. However, the child process never
    exits, as though maybe it's waiting for the stdin pipe to close.
    What's wrong with this code?...

    #!/usr/bin/perl

    require IPC::Open2 ;
    use strict ;

    my $markdownIn = "*Hello* **World**\n" ;
    print ("markdownIn = $markdownIn") ;
    my $cmd = '/Users/jk/Downloads/Markdown_1.0.1/Markdown.pl';
    my $markdownOut ;
    my $childpid = IPC::Open2::eek:pen2($markdownOut, $markdownIn, $cmd)
    or die "can't open pipe to $cmd: $!";
    waitpid( $childpid, 0 );
    my $cmdExitStatus = $? >> 8;
    print ("cmdExitStatus = $cmdExitStatus\n") ;
    print ("markdownOut = $markdownOut\n") ;


    If I comment out the waitpid(), then it runs but does not give the
    desired result:

    markdownIn = *Hello* **World**
    cmdExitStatus = 0
    markdownOut = GLOB(0x816b3c)

    Markdown.pl definitely works fine if I give it that string from stdin
    in a bash shell:
    Jerrys-Mac-Mini:~ jk$ echo "*Hello* **World**\n" | /Users/jk/Downloads/
    Markdown_1.0.1/Markdown.pl
    <p><em>Hello</em> <strong>World</strong>\n</p>

    Thank you!

    Jerry Krinock
    Jerry Krinock, Sep 8, 2009
    #1
    1. Advertising

  2. Jerry Krinock

    Uri Guttman Guest

    >>>>> "JK" == Jerry Krinock <> writes:

    JK> require IPC::Open2 ;

    generally 'use' is best for loading modules. require works but it is a
    runtime thing vs compile time for use. also by using use you will get
    the open2 sub imported into your namespace.

    JK> use strict ;

    JK> my $markdownIn = "*Hello* **World**\n" ;
    JK> print ("markdownIn = $markdownIn") ;
    JK> my $cmd = '/Users/jk/Downloads/Markdown_1.0.1/Markdown.pl';
    JK> my $markdownOut ;
    JK> my $childpid = IPC::Open2::eek:pen2($markdownOut, $markdownIn, $cmd)

    those first two args to open2 are not data but file handles. you need to
    first call the open2 with 2 handles and then send data to the input
    handle and read data from the output handle. open2 will autovivify
    handles for you so you can just do this (untested):

    use IPC::Open2 ;

    my $cmd = '/Users/jk/Downloads/Markdown_1.0.1/Markdown.pl';
    my($chld_out, $chld_in);
    my $pid = $pid = open2($chld_out, $chld_in, $cmd ) ;

    print $chld_in "*Hello* **World**\n" ;
    my $out = <$chld_out> ;

    print "read [$out]\n" ;

    this doesn't reap the process or handle repeated i/o which should be
    done carefully or it may block. or you would need an event loop.

    read perldoc perlipc for examples of how to use open2.

    uri

    --
    Uri Guttman ------ -------- http://www.sysarch.com --
    ----- Perl Code Review , Architecture, Development, Training, Support ------
    --------- Gourmet Hot Cocoa Mix ---- http://bestfriendscocoa.com ---------
    Uri Guttman, Sep 8, 2009
    #2
    1. Advertising

  3. First, I tried Uri's code, but it has the same problem as mine. That
    is, it waits forever, on the line
    my $out = <$chld_out> ;

    Apparently my first problem was believing that something so simple
    could be so hard. Having crossed that hurdle, as Uri suggested, I re-
    read the section on open2() in perldoc perlipc.

    Indeed, the single example that they give in there works, but, as
    explained there, only because the 'cat' command they invoke is one of
    the few which supports a 'unbuffered' or '-u' parameter. In that
    example, if I change
    $pid = open2(*Reader, *Writer, "cat -u -n" );
    to simply
    $pid = open2(*Reader, *Writer, "cat -n" );
    then, again, it waits forever, on $got = <Reader>;. Arghhhh.

    So, reading down further, I see that they recommend using the 'Expect'
    module. Fortunately, 'Expect' and all of its prerequisites seem to
    already be on my system for some reason. So, I try this code:

    #!/usr/bin/perl

    use Expect ;

    my $stdin = "*Hello* **World**\n" ;
    my $cmd = '/Users/jk/Downloads/Markdown_1.0.1/Markdown.pl';
    my @parameters = () ; # No arguments to Markdown.pl

    my $exp = new Expect;
    $exp->raw_pty(1);
    $exp->spawn($cmd, @parameters)
    or die "Cannot spawn $command: $!\n";
    $exp->send($stdin);
    print "Waiting for output.\n" ;
    my $stdout = $exp->expect(undef);
    print "stdout: $stdout\n" ;

    Result: Still the same problem -- "Waiting for output" -- forever. If
    I give the expect() function a timeout parameter, then it times out
    after the given time, but no returns no data.

    Does anyone know how to use the Expect module in the simple case of
    spawning a command with stdin to get stdout? The following quote from
    Expect documentation poo-poos this usage:
    Question: "I just want to read the output of a process without
    expect()ing anything. How can I do this?"
    Answer: "[ Are you sure you need Expect for this? How about qx() or
    open("prog|")? ]

    My answer: Yes, I've already been there!

    Had I known this was going to be so hard, I would have written my
    input to a temporary file instead. I know that Markdown.pl can read
    from a file. Is that the only way to do this?
    Jerry Krinock, Sep 8, 2009
    #3
  4. On Sep 8, 7:31 am, Ben Morrow <> wrote:

    > ...if you don't need true interaction (that is, if you know what
    > input you're going to send before you start) then it's often easier.
    > It's also likely to be *much* more portable to non-Unix systems, if
    > that's a concern.


    Thank you, Ben. That's what I'm doing now -- writing my stdin to a
    temporary file and getting stdout using backticks. Works great.
    Here's a little function to help archive readers get started...

    =com
    Execute a given external program (command). The program must
    take input from a file specified as the last parameter on
    its command line. This function takes input you provide and passes
    it to the program in a temporary file. The program's stdout is
    returned.

    Provide the command, with any command-line options, as the
    first parameter, and the input data as the second parameter.

    Use this in lieu of open2() and the methods of Expect.pm, both
    of which will hang indefinitely unless the program supports
    unbuffered I/O, typically a -u option.
    =cut
    sub getStdoutWithInputFromCmd {
    my $cmd = shift ;
    my $stdin = shift ;

    # Write stdin to temp file
    my $tempFilePath = File::Temp->tmpnam() ;
    my $didWriteOK = open (TEMP,">$tempFilePath") ;
    print TEMP $stdin ;
    close (TEMP) ;

    # Execute command
    my $stdout = `$cmd \"$tempFilePath\"` ;

    # Clean up temporary file
    unlink($tempFilePath) ;

    return $stdout ;
    }
    Jerry Krinock, Sep 8, 2009
    #4
  5. Someone named æŽå°ä¼Ÿ sent me an even easier way. Hardly even worth a sub
    now, but just so I don't ever forget this....

    =com
    Execute a given external program (command). The program must
    take input from stdin. The program's stdout is returned.

    Provide the command, with any command-line options, as the
    first parameter, and the input data as the second parameter.

    Use this in lieu of open2() and the methods of Expect.pm, both
    of which will hang indefinitely unless the program supports
    unbuffered I/O, typically a -u option.
    =cut
    sub getStdoutWithStdinFromCmd {
    my $cmd = shift ;
    my $stdin = shift ;

    return `echo \"$stdin\" | \"$cmd\"` ;

    =com

    Hand hits forehead!!!! Duh!!!!!!!!
    Jerry Krinock, Sep 8, 2009
    #5
  6. Please quote relevant parts from the posting you are replying to.

    On 2009-09-08 12:04, Jerry Krinock <> wrote:
    > First, I tried Uri's code, but it has the same problem as mine. That
    > is, it waits forever, on the line
    > my $out = <$chld_out> ;


    You already mentioned to solution to this problem in your first posting:
    You need to close $chld_in after you are done sending data to it.
    Otherwise, how does your filter know you are done?

    hp
    Peter J. Holzer, Sep 9, 2009
    #6
  7. On 2009-09-08 20:47, Ben Morrow <> wrote:
    > Quoth Jerry Krinock <>:
    >> Use this in lieu of open2() and the methods of Expect.pm, both
    >> of which will hang indefinitely unless the program supports
    >> unbuffered I/O, typically a -u option.

    >
    > This comment is inaccurate. It is perfectly possible to drive a program
    > that uses buffered IO using open2, you just need to be ready to feed
    > input as needed and get output as it arrives.


    Yes. Using select (or IO::Select) is a bit tricky, though. There are a
    few frameworks for handling async Events and I/O, but I've never used
    them.


    >> sub getStdoutWithStdinFromCmd {
    >> my $cmd = shift ;
    >> my $stdin = shift ;
    >>
    >> return `echo \"$stdin\" | \"$cmd\"` ;

    >
    > This is a bad idea. For one thing, if $stdin contains a '$' you will not
    > get the input you expect; for another, if $stding is longer than the
    > (system-specific) maximum command-line length, this will fail.
    >
    > Forking a child to feed the input pipe might, however, be a sensible
    > solution.


    Yes. That is frequently the simplest solution, especially if you could
    write it as a pipe sequence in the shell.

    > Something like
    >
    > use POSIX qw/_exit/;
    >
    > my $kid = fork;
    > defined $kid or die "can't fork: $!";
    >
    > unless ($kid) {
    > print *Writer $stdin;


    I assume Writer comes from open2 in your snippet?

    In this case I wouldn't use open2. You don't need a bidirectional
    communication, so just use plain open:


    #!/usr/bin/perl
    use warnings;
    use strict;


    my $pid1 = open(my $filter_out, '-|');
    if ($pid1 == 0) {
    # kid process
    # STDOUT is connected to $filter_out in the parent.
    # now we start the filter
    my $pid2 = open(my $filter_in, '|-', 'tr', 'a-z', 'A-Z');
    # $filter_in is now connected to stdin of the filter,
    # stdout of the filter is still connected to $filter_out
    # of the parent.
    # So we when we print something to $filter_in, it is passed through
    # the filter to the parent.
    for ('a' .. 'z', 'a' .. 'z', 'a' .. 'z') {
    print STDERR "kid: sending $_\n";
    print $filter_in $_ x 80, "\n";
    sleep 1;
    }
    # exit will close the pipe to the filter which will cause the
    # filter to terminate which will close the pipe to the parent.
    # so everything is cleaned up after that.
    exit 0;
    } else {
    while (<$filter_out>) {
    print STDERR "parent: received $_";
    }
    }

    # child is reaped when $filter_out goes out of scope
    __END__

    (error handling is left as an exercise for the reader)

    On my system this prints:

    kid: sending a
    kid: sending b
    ....
    kid: sending y
    kid: sending z
    kid: sending a
    kid: sending b
    ....
    kid: sending x
    kid: sending y
    parent: received AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
    parent: received YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
    parent: received ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ
    parent: received AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
    parent: received BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB
    ....
    parent: received XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
    kid: sending z
    kid: sending a
    ....
    kid: sending y
    kid: sending z
    parent: received YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
    parent: received ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ
    parent: received AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
    parent: received BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB
    ....
    parent: received YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
    parent: received ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ

    Note the asynchronous output caused by buffering.

    hp
    Peter J. Holzer, Sep 9, 2009
    #7
  8. On Sep 8, 1:47 pm, Ben Morrow <> wrote:

    > =com is not a pod directive. If you just want un-sectioned pod, ...


    Sorry for abusing Perl's comment syntax. Actually, I'm a big fan of
    doxygen when I write C code and one of these days I'm going to take 15
    minutes and read about how to properly author Perl documentation using
    POD.

    > > Use this in lieu of open2() and the methods of Expect.pm, ...

    >
    > This comment is inaccurate.


    I agree; Indeed, I short-cutted the truth.

    > > return `echo \"$stdin\" | \"$cmd\"` ;

    >
    > This is a bad idea. For one thing, if $stdin contains a '$' you will not
    > get the input you expect; for another, if $stding is longer than the


    And, for another thing, if there are double-quotes in $stdin ...
    ditto. It is a very bad idea.

    >
    > Forking a child to feed the input pipe might, however, be a sensible
    > solution. Something like..


    Ben, this looks like fun and thanks, but I'm filing it away. Writing
    to a temporary file works with as many lines of code, and I could
    explain that to my mother :)

    On Sep 9, 3:02 am, "Peter J. Holzer" <> wrote:
    > In this case I wouldn't use open2. You don't need a bidirectional
    > communication, so just use plain open:
    >

    Well, this is quite an educational little script. Since we have Perl
    5.8, I used usleep() instead of sleep and cut down on some of the
    iterations so it doesn't take so long.

    #!/usr/bin/perl
    > use warnings;
    > use strict;

    use Time::HiRes; # for usleep. Requires Perl 5.8 or higher
    print STDERR "Entering Peter's Script\n" ;
    >
    > my $pid1 = open(my $filter_out, '-|');
    > if ($pid1 == 0) {
    > my $pid2 = open(my $filter_in, '|-', 'tr', 'a-z', 'A-Z');
    > for ('a' .. 'z', 'a' .. 'z', 'a' .. 'z') {
    > print STDERR "kid: sending $_\n";
    > print $filter_in $_ x 80, "\n";
    > Time::HiRes::usleep(1000) ;
    > }
    > exit 0;} else {


    How does this script jump from the if($pid1 == 0) branch to the else
    branch at this point? The "Entering Peter's Script..." only logs
    once...

    > while (<$filter_out>) {
    > print STDERR "parent: received $_";
    > }
    > }


    ....here...
    > kid: sending y
    > parent: received AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAA
    >
    > Note the asynchronous output caused by buffering.


    I believe you mean that the output does not get read until all the
    input is written.

    You guys are amazing, as usual.
    Jerry Krinock, Sep 10, 2009
    #8
  9. Jerry Krinock

    C.DeRykus Guest

    On Sep 8, 12:09 pm, Jerry Krinock <> wrote:
    > Someone named æŽå°ä¼Ÿ sent me an even easier way.  Hardly even worth a sub
    > now, but just so I don't ever forget this....
    >
    > =com
    > Execute a given external program (command).  The program must
    > take input from stdin.  The program's stdout is returned.
    >
    > Provide the command, with any command-line options, as the
    > first parameter, and the input data as the second parameter.
    >
    > Use this in lieu of open2() and the methods of Expect.pm, both
    > of which will hang indefinitely unless the program supports
    > unbuffered I/O, typically a -u option.
    > =cut
    > sub getStdoutWithStdinFromCmd {
    >         my $cmd = shift ;
    >         my $stdin = shift ;
    >
    >         return `echo \"$stdin\" | \"$cmd\"` ;
    >
    > =com
    >
    > Hand hits forehead!!!!  Duh!!!!!!!!


    An even easier way using the suggested IPC::Run:

    use IPC::Run qw( run timeout );
    use strict ;
    use warnings;

    my $in = "*Hello* **World**\n" ;

    run ['/path/to/Markdown.pl'], \my $in, \my $out,
    \my $err,timeout( 10 )
    or die "Markdown.pl failed: $?";

    print "out=$out\n";

    --
    Charles DeRykus
    C.DeRykus, Sep 11, 2009
    #9
  10. Eventually I got into utf8 warnings with my temporary file, so rather
    than look up my Perl UTF8 Incantations Cheat Sheet, I waited a few
    minutes for CPAN to install IPC::Run and used Charles' one-liner.

    Thanks, Charles ... I would not have made it through the IPC::Run
    documentation without that!
    Jerry Krinock, Sep 19, 2009
    #10
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Antonio
    Replies:
    0
    Views:
    653
    Antonio
    May 17, 2004
  2. Ben
    Replies:
    2
    Views:
    1,337
    jacob navia
    Aug 29, 2009
  3. grocery_stocker

    IPC::Open2 and sort

    grocery_stocker, Dec 9, 2006, in forum: Perl Misc
    Replies:
    12
    Views:
    179
    Joe Smith
    Dec 15, 2006
  4. Bernard Chan

    IPC::Open2 - Bad File Descriptor

    Bernard Chan, Mar 6, 2007, in forum: Perl Misc
    Replies:
    9
    Views:
    225
    Ben Morrow
    Mar 8, 2007
  5. Justin C
    Replies:
    4
    Views:
    665
    Rainer Weikusat
    Jul 3, 2012
Loading...

Share This Page