Understanding tempfile and open3

[usenet]

hello, world\n

I've read the perlfaq#8 up and down, can spell File::Temp backwards,
but still, I'm puzzled because no matter what I do, I get an empty
tempfile when it clearly should not be empty. Here's what I want:
Run "gcc -E -v -dM -x -c /dev/null" and write stderr to a temp file
while reading stderr from some handle (exactly so. Suggestions like
"write two temp files; see perlfaq8" are inelegant). The way I try to
combine tempfile and open3 is not correct and I want to know my
mistake. Here's a complete program to illustrate the issue:

#!/usr/local/bin/perl -w

use strict;
use File::Temp qw(tempfile);
use IPC::Open3;
use Symbol qw(gensym);


# A command producing output on both stdout and stderr.
my @cmd = ('gcc', '-E', '-v', '-dM', '-x', 'c', '/dev/null');

# Temp file where @cmd's stdout should go.
my ($filehandle, $filename) = tempfile ('predef-XXXXXX', DIR => ".",
SUFFIX => '.h', UNLINK => 0);

# What I want: write stdout to $filehandle and end up in $filename;
# read stderr from <ERR>.
my $pid = open3 (gensym, $filehandle, \*ERR, @cmd);
while (<ERR>) {
print $_; # This works fine.
}
waitpid ($pid, 0) or die "$!";
close $filehandle or die "$!";

# What I get: 0 bytes even though it should be many lines.
# I would have expected $filename to be more than a zero-length
file...
# What exactly is the relation between $filehandle and $filename?
system ('wc', $filename);


The output is:
Using built-in specs.
Target: i386-undermydesk-freebsd
Configured with: FreeBSD/i386 system compiler
Thread model: posix
gcc version 4.2.1 20070719 [FreeBSD]
/usr/libexec/cc1 -E -quiet -v -D_LONGLONG /dev/null -dM
ignoring duplicate directory "/usr/include"
#include "..." search starts here:
#include <...> search starts here:
/usr/include
End of search list.
0 0 0 ./predef-Om1rvd.h


The part form "Using built-in specs." to "End of search list." is
gcc's stderr.
Stdout has about 50 "#define" directives that you can see when running
the gcc command from the shell command line.

Thanks for any clue. Merry Christmas!

Regards,
Jens

 

[Uri Guttman]

u> use File::Temp qw(tempfile);
u> use IPC::Open3;

u> # A command producing output on both stdout and stderr.
u> my @cmd = ('gcc', '-E', '-v', '-dM', '-x', 'c', '/dev/null');

u> # Temp file where @cmd's stdout should go.
u> my ($filehandle, $filename) = tempfile ('predef-XXXXXX', DIR => ".",
u> SUFFIX => '.h', UNLINK => 0);

u> # What I want: write stdout to $filehandle and end up in $filename;
u> # read stderr from <ERR>.
u> my $pid = open3 (gensym, $filehandle, \*ERR, @cmd);


your problem is in that line. open3 doesn't connect a process to file
handles that are open with files. it is meant to give you an open handle
to talk to the subprocess. once open3 is called you have to do all the
i/o afterwards - all it does is fork/exec the command while attaching
the handles (which are not pre-opened to anything) to the stdio/err of
the process.

u> while (<ERR>) {
u> print $_; # This works fine.
u> }

so just print that to the temp handle:

print $filehandle $_ ;

or since you don't care about the other handles, just slurp it and
print:

print $filehandle <ERR> ;

you can even use File::Slurp and not need to create the tempfile and its
handle:

use File::Slurp
write_file( 'pick_a_file_name', <ERR> ) ;

uri

 

[usenet]

  u> use File::Temp qw(tempfile);
  u> use IPC::Open3;

  u> # A command producing output on both stdout and stderr.
  u> my @cmd = ('gcc', '-E', '-v', '-dM', '-x', 'c', '/dev/null');

  u> # Temp file where @cmd's stdout should go.
  u> my ($filehandle, $filename) = tempfile ('predef-XXXXXX', DIR => ".",
  u>   SUFFIX => '.h', UNLINK => 0);

  u> # What I want: write stdout to $filehandle and end up in $filename;
  u> #              read stderr from <ERR>.
  u> my $pid = open3 (gensym, $filehandle, \*ERR, @cmd);

your problem is in that line. open3 doesn't connect a process to file
handles that are open with files. it is meant to give you an open handle
to talk to the subprocess. once open3 is called you have to do all the
i/o afterwards - all it does is fork/exec the command while attaching
the handles (which are not pre-opened to anything) to the stdio/err of
the process.

Thanks Uri, for pointing me to this. In fact, open3 has a way
to use an existing file handle when it is prefixed with ">&".
This works as expected:

#!/usr/local/bin/perl -w

use strict;
use File::Temp qw(tempfile);
use IPC::Open3;
use Symbol qw(gensym);

# A command producing output on both stdout and stderr.
my @cmd = ('gcc', '-E', '-v', '-dM', '-x', 'c', '/dev/null');

# Temp file where @cmd's stdout should go.
my ($filehandle, $filename) = tempfile ('predef-XXXXXX', DIR => ".",
SUFFIX => '.h', UNLINK => 0);

# What I want: write stdout to $filehandle and end up in $filename;
# read stderr from <ERR>.
open FH, ">$filename" or die "$!";
my $pid = open3 (gensym, ">&FH", \*ERR, @cmd);
while (<ERR>) {
print $_ if /^\s\S+$/; # This works fine.
# Do more things with $_.
}
waitpid ($pid, 0) or die "$!";
close FH;

system ('wc', $filename);


Now the only ugly part here is that I seem unable to
use $filehandle directly with open3 and ">&". Instead
I go the detour via open FH, ">$filename". Hasn't
tempfile() done something very similar (given me a
handle and name)? Is there a way to make open3 accept
some combination of ">&" and $filehandle without the
FH detour? (I want tempfile for its race avoidance and
autoremoval of temp file features).

Thanks and merry Christmas!

Jens

 

[sln]

Thanks Uri, for pointing me to this. In fact, open3 has a way
to use an existing file handle when it is prefixed with ">&".
This works as expected:

#!/usr/local/bin/perl -w

use strict;
use File::Temp qw(tempfile);
use IPC::Open3;
use Symbol qw(gensym);

# A command producing output on both stdout and stderr.
my @cmd = ('gcc', '-E', '-v', '-dM', '-x', 'c', '/dev/null');

# Temp file where @cmd's stdout should go.
my ($filehandle, $filename) = tempfile ('predef-XXXXXX', DIR => ".",
SUFFIX => '.h', UNLINK => 0);

# What I want: write stdout to $filehandle and end up in $filename;
# read stderr from <ERR>.
open FH, ">$filename" or die "$!";
my $pid = open3 (gensym, ">&FH", \*ERR, @cmd);
while (<ERR>) {
print $_ if /^\s\S+$/; # This works fine.
# Do more things with $_.
}
waitpid ($pid, 0) or die "$!";
close FH;

system ('wc', $filename);


Now the only ugly part here is that I seem unable to
use $filehandle directly with open3 and ">&". Instead
I go the detour via open FH, ">$filename". Hasn't
tempfile() done something very similar (given me a
handle and name)? Is there a way to make open3 accept
some combination of ">&" and $filehandle without the
FH detour? (I want tempfile for its race avoidance and
autoremoval of temp file features).

Thanks and merry Christmas!

Jens

You could try something like this:

use strict;
use warnings;

use IPC::Open3;
my $wtr;

open my $stdoutfd, '>', 'tmpfile1.txt' or die "can't open tmpfile1.txt";
open STDFH, ">&=", $stdoutfd or die "can't fdopen \$stdoutfd";

my $pid = open3($wtr, ">&STDFH", \*ERR, "dir c:\\");

while (<ERR>) {
print "Error: $_";
}

waitpid ($pid, 0) or die "$!";

close $stdoutfd or die "$!";
close STDFH or die "$!";

 

[C.DeRykus]

...
# What I want: write stdout to $filehandle and end up in $filename;
# read stderr from <ERR>.
open FH, ">$filename" or die "$!";
my $pid = open3 (gensym, ">&FH", \*ERR, @cmd);
while (<ERR>) {
print $_ if /^\s\S+$/; # This works fine.
# Do more things with $_.}

waitpid ($pid, 0) or die "$!";
close FH;

system ('wc', $filename);

Now the only ugly part here is that I seem unable to
use $filehandle directly with open3 and ">&". Instead
I go the detour via open FH, ">$filename". Hasn't
tempfile() done something very similar (given me a
handle and name)? Is there a way to make open3 accept
some combination of ">&" and $filehandle without the
FH detour? (I want tempfile for its race avoidance and
autoremoval of temp file features).

I'm not sure why ">&$filehandle" doesn't work either. However, this
did work for me:

open3 (gensym, , '>&=' . fileno($filehandle) ,
'>&=' . fileno($error_fh),
@cmd ) ;

 

[Uri Guttman]

u> Thanks Uri, for pointing me to this. In fact, open3 has a way
u> to use an existing file handle when it is prefixed with ">&".
u> This works as expected:

you are missing the core point. open3 doesn't do any I/O! a handle from
a process can only be used in one direction at a time. so you can take
an existing or new handle and connect it to a subprocess. that doesn't
make a 'pipe' of i/o between the process and the file. it just reuses
the existing handle and connects it to the process. you still need to
read stderr from the process and write it to the file.

u> Now the only ugly part here is that I seem unable to
u> use $filehandle directly with open3 and ">&". Instead
u> I go the detour via open FH, ">$filename". Hasn't
u> tempfile() done something very similar (given me a
u> handle and name)? Is there a way to make open3 accept
u> some combination of ">&" and $filehandle without the
u> FH detour? (I want tempfile for its race avoidance and
u> autoremoval of temp file features).

you can't do that. the handle from tempfile is for writing to it. the
handle for stderr is for reading from the process. how could one handle
be both?

uri

 

[sln]

I'm not sure why ">&$filehandle" doesn't work either. However, this
did work for me:

open3 (gensym, , '>&=' . fileno($filehandle) ,
'>&=' . fileno($error_fh),
@cmd ) ;

Hmm, interresting, was thinking of that, wondered if that did fdopen like.
Apparently, it does.

sln

 

[sln]

u> Thanks Uri, for pointing me to this. In fact, open3 has a way
u> to use an existing file handle when it is prefixed with ">&".
u> This works as expected:

you are missing the core point. open3 doesn't do any I/O! a handle from
a process can only be used in one direction at a time. so you can take
an existing or new handle and connect it to a subprocess. that doesn't
make a 'pipe' of i/o between the process and the file. it just reuses
the existing handle and connects it to the process. you still need to
read stderr from the process and write it to the file.

u> Now the only ugly part here is that I seem unable to
u> use $filehandle directly with open3 and ">&". Instead
u> I go the detour via open FH, ">$filename". Hasn't
u> tempfile() done something very similar (given me a
u> handle and name)? Is there a way to make open3 accept
u> some combination of ">&" and $filehandle without the
u> FH detour? (I want tempfile for its race avoidance and
u> autoremoval of temp file features).

you can't do that. the handle from tempfile is for writing to it. the
handle for stderr is for reading from the process. how could one handle
be both?

uri

huh? think again

sln

 

[C.DeRykus]

...
u> Now the only ugly part here is that I seem unable to
u> use $filehandle directly with open3 and ">&". Instead
u> I go the detour via open FH, ">$filename". Hasn't
u> tempfile() done something very similar (given me a
u> handle and name)? Is there a way to make open3 accept
u> some combination of ">&" and $filehandle without the
u> FH detour? (I want tempfile for its race avoidance and
u> autoremoval of temp file features).

you can't do that. the handle from tempfile is for writing to it. the
handle for stderr is for reading from the process. how could one handle
be both?

Not to jump in the middle but his example seems to
suggest he just wanted to use the existing filehandle
returned by File::Temp for Open3's child output rather
than having to open the output himself as he ended
up doing:

Jens> open FH, ">$filename" or die "$!";
Jens> my $pid = open3 (gensym, ">&FH", \*ERR, @cmd);

But Open3 docs suggest you should be able to use the
filehandle returned by File::Temp:

If CHLD_OUT or CHLD_ERR begins with ">&", then the
child will send output directly to that filehandle. In both
cases, there will be a dup(2) instead of a pipe(2) made.

So, if File::Temp returns the handle $filehandle, you should
be able to use ">&$filehandle" ... but no output appeared
in my test with ">&$filehandle" even with autoflush set; but, if the
file descriptor is used '>&=' . fileno($filehandle) , it does.

 

[sln]

u> Thanks Uri, for pointing me to this. In fact, open3 has a way
u> to use an existing file handle when it is prefixed with ">&".
u> This works as expected:

you are missing the core point. open3 doesn't do any I/O! a handle from
a process can only be used in one direction at a time. so you can take
an existing or new handle and connect it to a subprocess. that doesn't
make a 'pipe' of i/o between the process and the file. it just reuses
the existing handle and connects it to the process. you still need to
read stderr from the process and write it to the file.

u> Now the only ugly part here is that I seem unable to
u> use $filehandle directly with open3 and ">&". Instead
u> I go the detour via open FH, ">$filename". Hasn't
u> tempfile() done something very similar (given me a
u> handle and name)? Is there a way to make open3 accept
u> some combination of ">&" and $filehandle without the
u> FH detour? (I want tempfile for its race avoidance and
u> autoremoval of temp file features).

you can't do that. the handle from tempfile is for writing to it. the
handle for stderr is for reading from the process. how could one handle
be both?

uri

The handle for tempfile (as he used it) and stderr globe ref are both
readers from the perspective of the caller, not writers.

sln

 

[sln]

The handle for tempfile (as he used it) and stderr globe ref are both
readers from the perspective of the caller, not writers.

sln

And btw, both readers can be the same handle.

sln

 

[usenet]

you are missing the core point. open3 doesn't do any I/O! a handle from
a process can only be used in one direction at a time. so you can take
an existing or new handle and connect it to a subprocess. that doesn't
make a 'pipe' of i/o between the process and the file. it just reuses
the existing handle and connects it to the process. you still need to
read stderr from the process and write it to the file.

AAARGH! <slaps forehead> Sorry Uri, I know where the confusion
comes from: in my original posting I have a nasty typo: I wrote

"write stderr to a temp file while reading stderr from some handle"

when I meant

"write stdout to a temp file while reading stderr from some handle"

Does that make more sense? In pseudo shell syntax

gcc args >tempfile 2>|while(<ERR>)

where "2>|while(<ERR>)" is supposed to mean "make stderr
available to read with <ERR>. I believe this can be done.

Regards,

Jens