IPC::Open2::open2() -- How to pass strings stdin, stdout?

J

Jerry Krinock

Markdown.pl is a Perl text-processing filter which takes input from
stdin and outputs to stdout. I'd like to call it as a child process
from within my script -- pass it a string and get a string back. I've
come to believe that this requires something like IPC::Open2:eek:pen2()
and have written the following code. However, the child process never
exits, as though maybe it's waiting for the stdin pipe to close.
What's wrong with this code?...

#!/usr/bin/perl

require IPC::Open2 ;
use strict ;

my $markdownIn = "*Hello* **World**\n" ;
print ("markdownIn = $markdownIn") ;
my $cmd = '/Users/jk/Downloads/Markdown_1.0.1/Markdown.pl';
my $markdownOut ;
my $childpid = IPC::Open2::eek:pen2($markdownOut, $markdownIn, $cmd)
or die "can't open pipe to $cmd: $!";
waitpid( $childpid, 0 );
my $cmdExitStatus = $? >> 8;
print ("cmdExitStatus = $cmdExitStatus\n") ;
print ("markdownOut = $markdownOut\n") ;


If I comment out the waitpid(), then it runs but does not give the
desired result:

markdownIn = *Hello* **World**
cmdExitStatus = 0
markdownOut = GLOB(0x816b3c)

Markdown.pl definitely works fine if I give it that string from stdin
in a bash shell:
Jerrys-Mac-Mini:~ jk$ echo "*Hello* **World**\n" | /Users/jk/Downloads/
Markdown_1.0.1/Markdown.pl
<p><em>Hello</em> <strong>World</strong>\n</p>

Thank you!

Jerry Krinock
 
U

Uri Guttman

JK> require IPC::Open2 ;

generally 'use' is best for loading modules. require works but it is a
runtime thing vs compile time for use. also by using use you will get
the open2 sub imported into your namespace.

JK> use strict ;

JK> my $markdownIn = "*Hello* **World**\n" ;
JK> print ("markdownIn = $markdownIn") ;
JK> my $cmd = '/Users/jk/Downloads/Markdown_1.0.1/Markdown.pl';
JK> my $markdownOut ;
JK> my $childpid = IPC::Open2::eek:pen2($markdownOut, $markdownIn, $cmd)

those first two args to open2 are not data but file handles. you need to
first call the open2 with 2 handles and then send data to the input
handle and read data from the output handle. open2 will autovivify
handles for you so you can just do this (untested):

use IPC::Open2 ;

my $cmd = '/Users/jk/Downloads/Markdown_1.0.1/Markdown.pl';
my($chld_out, $chld_in);
my $pid = $pid = open2($chld_out, $chld_in, $cmd ) ;

print $chld_in "*Hello* **World**\n" ;
my $out = <$chld_out> ;

print "read [$out]\n" ;

this doesn't reap the process or handle repeated i/o which should be
done carefully or it may block. or you would need an event loop.

read perldoc perlipc for examples of how to use open2.

uri
 
J

Jerry Krinock

First, I tried Uri's code, but it has the same problem as mine. That
is, it waits forever, on the line
my $out = <$chld_out> ;

Apparently my first problem was believing that something so simple
could be so hard. Having crossed that hurdle, as Uri suggested, I re-
read the section on open2() in perldoc perlipc.

Indeed, the single example that they give in there works, but, as
explained there, only because the 'cat' command they invoke is one of
the few which supports a 'unbuffered' or '-u' parameter. In that
example, if I change
$pid = open2(*Reader, *Writer, "cat -u -n" );
to simply
$pid = open2(*Reader, *Writer, "cat -n" );
then, again, it waits forever, on $got = <Reader>;. Arghhhh.

So, reading down further, I see that they recommend using the 'Expect'
module. Fortunately, 'Expect' and all of its prerequisites seem to
already be on my system for some reason. So, I try this code:

#!/usr/bin/perl

use Expect ;

my $stdin = "*Hello* **World**\n" ;
my $cmd = '/Users/jk/Downloads/Markdown_1.0.1/Markdown.pl';
my @parameters = () ; # No arguments to Markdown.pl

my $exp = new Expect;
$exp->raw_pty(1);
$exp->spawn($cmd, @parameters)
or die "Cannot spawn $command: $!\n";
$exp->send($stdin);
print "Waiting for output.\n" ;
my $stdout = $exp->expect(undef);
print "stdout: $stdout\n" ;

Result: Still the same problem -- "Waiting for output" -- forever. If
I give the expect() function a timeout parameter, then it times out
after the given time, but no returns no data.

Does anyone know how to use the Expect module in the simple case of
spawning a command with stdin to get stdout? The following quote from
Expect documentation poo-poos this usage:
Question: "I just want to read the output of a process without
expect()ing anything. How can I do this?"
Answer: "[ Are you sure you need Expect for this? How about qx() or
open("prog|")? ]

My answer: Yes, I've already been there!

Had I known this was going to be so hard, I would have written my
input to a temporary file instead. I know that Markdown.pl can read
from a file. Is that the only way to do this?
 
J

Jerry Krinock

...if you don't need true interaction (that is, if you know what
input you're going to send before you start) then it's often easier.
It's also likely to be *much* more portable to non-Unix systems, if
that's a concern.

Thank you, Ben. That's what I'm doing now -- writing my stdin to a
temporary file and getting stdout using backticks. Works great.
Here's a little function to help archive readers get started...

=com
Execute a given external program (command). The program must
take input from a file specified as the last parameter on
its command line. This function takes input you provide and passes
it to the program in a temporary file. The program's stdout is
returned.

Provide the command, with any command-line options, as the
first parameter, and the input data as the second parameter.

Use this in lieu of open2() and the methods of Expect.pm, both
of which will hang indefinitely unless the program supports
unbuffered I/O, typically a -u option.
=cut
sub getStdoutWithInputFromCmd {
my $cmd = shift ;
my $stdin = shift ;

# Write stdin to temp file
my $tempFilePath = File::Temp->tmpnam() ;
my $didWriteOK = open (TEMP,">$tempFilePath") ;
print TEMP $stdin ;
close (TEMP) ;

# Execute command
my $stdout = `$cmd \"$tempFilePath\"` ;

# Clean up temporary file
unlink($tempFilePath) ;

return $stdout ;
}
 
J

Jerry Krinock

Someone named æŽå°ä¼Ÿ sent me an even easier way. Hardly even worth a sub
now, but just so I don't ever forget this....

=com
Execute a given external program (command). The program must
take input from stdin. The program's stdout is returned.

Provide the command, with any command-line options, as the
first parameter, and the input data as the second parameter.

Use this in lieu of open2() and the methods of Expect.pm, both
of which will hang indefinitely unless the program supports
unbuffered I/O, typically a -u option.
=cut
sub getStdoutWithStdinFromCmd {
my $cmd = shift ;
my $stdin = shift ;

return `echo \"$stdin\" | \"$cmd\"` ;

=com

Hand hits forehead!!!! Duh!!!!!!!!
 
P

Peter J. Holzer

Please quote relevant parts from the posting you are replying to.

First, I tried Uri's code, but it has the same problem as mine. That
is, it waits forever, on the line
my $out = <$chld_out> ;

You already mentioned to solution to this problem in your first posting:
You need to close $chld_in after you are done sending data to it.
Otherwise, how does your filter know you are done?

hp
 
P

Peter J. Holzer

This comment is inaccurate. It is perfectly possible to drive a program
that uses buffered IO using open2, you just need to be ready to feed
input as needed and get output as it arrives.

Yes. Using select (or IO::Select) is a bit tricky, though. There are a
few frameworks for handling async Events and I/O, but I've never used
them.

This is a bad idea. For one thing, if $stdin contains a '$' you will not
get the input you expect; for another, if $stding is longer than the
(system-specific) maximum command-line length, this will fail.

Forking a child to feed the input pipe might, however, be a sensible
solution.

Yes. That is frequently the simplest solution, especially if you could
write it as a pipe sequence in the shell.
Something like

use POSIX qw/_exit/;

my $kid = fork;
defined $kid or die "can't fork: $!";

unless ($kid) {
print *Writer $stdin;

I assume Writer comes from open2 in your snippet?

In this case I wouldn't use open2. You don't need a bidirectional
communication, so just use plain open:


#!/usr/bin/perl
use warnings;
use strict;


my $pid1 = open(my $filter_out, '-|');
if ($pid1 == 0) {
# kid process
# STDOUT is connected to $filter_out in the parent.
# now we start the filter
my $pid2 = open(my $filter_in, '|-', 'tr', 'a-z', 'A-Z');
# $filter_in is now connected to stdin of the filter,
# stdout of the filter is still connected to $filter_out
# of the parent.
# So we when we print something to $filter_in, it is passed through
# the filter to the parent.
for ('a' .. 'z', 'a' .. 'z', 'a' .. 'z') {
print STDERR "kid: sending $_\n";
print $filter_in $_ x 80, "\n";
sleep 1;
}
# exit will close the pipe to the filter which will cause the
# filter to terminate which will close the pipe to the parent.
# so everything is cleaned up after that.
exit 0;
} else {
while (<$filter_out>) {
print STDERR "parent: received $_";
}
}

# child is reaped when $filter_out goes out of scope
__END__

(error handling is left as an exercise for the reader)

On my system this prints:

kid: sending a
kid: sending b
....
kid: sending y
kid: sending z
kid: sending a
kid: sending b
....
kid: sending x
kid: sending y
parent: received AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
parent: received YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
parent: received ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ
parent: received AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
parent: received BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB
....
parent: received XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
kid: sending z
kid: sending a
....
kid: sending y
kid: sending z
parent: received YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
parent: received ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ
parent: received AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
parent: received BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB
....
parent: received YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
parent: received ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ

Note the asynchronous output caused by buffering.

hp
 
J

Jerry Krinock

=com is not a pod directive. If you just want un-sectioned pod, ...

Sorry for abusing Perl's comment syntax. Actually, I'm a big fan of
doxygen when I write C code and one of these days I'm going to take 15
minutes and read about how to properly author Perl documentation using
POD.
This comment is inaccurate.

I agree; Indeed, I short-cutted the truth.
This is a bad idea. For one thing, if $stdin contains a '$' you will not
get the input you expect; for another, if $stding is longer than the

And, for another thing, if there are double-quotes in $stdin ...
ditto. It is a very bad idea.
Forking a child to feed the input pipe might, however, be a sensible
solution. Something like..

Ben, this looks like fun and thanks, but I'm filing it away. Writing
to a temporary file works with as many lines of code, and I could
explain that to my mother :)

In this case I wouldn't use open2. You don't need a bidirectional
communication, so just use plain open:
Well, this is quite an educational little script. Since we have Perl
5.8, I used usleep() instead of sleep and cut down on some of the
iterations so it doesn't take so long.

#!/usr/bin/perl
use warnings;
use strict;
use Time::HiRes; # for usleep. Requires Perl 5.8 or higher
print STDERR "Entering Peter's Script\n" ;
my $pid1 = open(my $filter_out, '-|');
if ($pid1 == 0) {
my $pid2 = open(my $filter_in, '|-', 'tr', 'a-z', 'A-Z');
for ('a' .. 'z', 'a' .. 'z', 'a' .. 'z') {
print STDERR "kid: sending $_\n";
print $filter_in $_ x 80, "\n";
Time::HiRes::usleep(1000) ;
}
exit 0;} else {

How does this script jump from the if($pid1 == 0) branch to the else
branch at this point? The "Entering Peter's Script..." only logs
once...
while (<$filter_out>) {
print STDERR "parent: received $_";
}
}
....here...
kid: sending y
parent: received AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAA

Note the asynchronous output caused by buffering.

I believe you mean that the output does not get read until all the
input is written.

You guys are amazing, as usual.
 
C

C.DeRykus

Someone named æŽå°ä¼Ÿ sent me an even easier way.  Hardly even worth a sub
now, but just so I don't ever forget this....

=com
Execute a given external program (command).  The program must
take input from stdin.  The program's stdout is returned.

Provide the command, with any command-line options, as the
first parameter, and the input data as the second parameter.

Use this in lieu of open2() and the methods of Expect.pm, both
of which will hang indefinitely unless the program supports
unbuffered I/O, typically a -u option.
=cut
sub getStdoutWithStdinFromCmd {
        my $cmd = shift ;
        my $stdin = shift ;

        return `echo \"$stdin\" | \"$cmd\"` ;

=com

Hand hits forehead!!!!  Duh!!!!!!!!

An even easier way using the suggested IPC::Run:

use IPC::Run qw( run timeout );
use strict ;
use warnings;

my $in = "*Hello* **World**\n" ;

run ['/path/to/Markdown.pl'], \my $in, \my $out,
\my $err,timeout( 10 )
or die "Markdown.pl failed: $?";

print "out=$out\n";
 
J

Jerry Krinock

Eventually I got into utf8 warnings with my temporary file, so rather
than look up my Perl UTF8 Incantations Cheat Sheet, I waited a few
minutes for CPAN to install IPC::Run and used Charles' one-liner.

Thanks, Charles ... I would not have made it through the IPC::Run
documentation without that!
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,903
Latest member
orderPeak8CBDGummies

Latest Threads

Top