Safe Pipe Open question

[Mike Paolucci]

I'm getting unexplained errors when I read from a log file. I'm using a
"Safe Pipe Open", and my reference for this is:

http://www.litespeed.net/perldocs/lib/Pod/perlipc.html#Using_open_for_IPC


Here's the code:

my $logFile = 'full_path_to_logfile'; (not specified for confidentially)
my $ipAddress = '12.34.56.78'; (not specified for confidentially)

my $fh;
open( $fh, "grep $ipAddress $logFile | tail 2>&1 |" ) or
die "Cannot open logfile!!\n";

# get the last lines
my $logData = '';
while ( <$fh> )
{
$logData .= $_;
}
close( $fh ) or die "Cannot close logfile!\n";

# display these lines
print $logData;


The code works perfectly and displays no errors. The probem is that
when I look in the system error file I see lines and lines of:

grep: writing output: Broken pipe
grep: writing output: Broken pipe
grep: writing output: Broken pipe
grep: writing output: Broken pipe
grep: writing output: Broken pipe
grep: writing output: Broken pipe
grep: writing output: Broken pipe
grep: writing output: Broken pipe
grep: writing output: Broken pipe
grep: writing output: Broken pipe


Why are these errors there?

What do they mean?

How do I prevent them?

 

[robic0]

I'm getting unexplained errors when I read from a log file. I'm using a
"Safe Pipe Open", and my reference for this is:

http://www.litespeed.net/perldocs/lib/Pod/perlipc.html#Using_open_for_IPC


Here's the code:

my $logFile = 'full_path_to_logfile'; (not specified for confidentially)
my $ipAddress = '12.34.56.78'; (not specified for confidentially)

my $fh;
open( $fh, "grep $ipAddress $logFile | tail 2>&1 |" ) or
die "Cannot open logfile!!\n";

# get the last lines
my $logData = '';
while ( <$fh> )
{
$logData .= $_;
}
close( $fh ) or die "Cannot close logfile!\n";

# display these lines
print $logData;


The code works perfectly and displays no errors. The probem is that
when I look in the system error file I see lines and lines of:

grep: writing output: Broken pipe
grep: writing output: Broken pipe
grep: writing output: Broken pipe
grep: writing output: Broken pipe
grep: writing output: Broken pipe
grep: writing output: Broken pipe
grep: writing output: Broken pipe
grep: writing output: Broken pipe
grep: writing output: Broken pipe
grep: writing output: Broken pipe


Why are these errors there?

What do they mean?

How do I prevent them?

Since the code works perfectly, I would say
"grep" has written a problem message out.
I could be wrong though...

robic0

 

[John Bokma]

my $fh;
open( $fh, "grep $ipAddress $logFile | tail 2>&1 |" ) or
die "Cannot open logfile!!\n";

It's $! not !!.

 

[robic0]

Your code works perfectly for me (Mac OS 10.4, perl 5.8.6), although I
did not try it on any really big files. I suggest if you just want to
get your program working, you read and process the log file within your
Perl program and save lines that contain a substring:

A relly big file can caue I/O errors? So nested system calls and pipe
handles should not be trusted then if i/o handle resolved, imbed system calls
in LUNIX produce *large* files?

Is this *really* the LUNIX version problem? Hey, hey...
what would happen if the equivalent were done from a 'C/C++' LARNIX
program?

Hey, why pick on Perl? The Perl writers acknowledge, without precedence,
extreme I/O bugs, crossplatform don't they?
I mean I've read so many crossplatform caveats its not funny...

gluck, switch to a LARNIX version os 'C' platform for your development.
*Don't* expect Perl to solve your problems. As far as I can see, you
don't know any other solution. I could write the equivalent 'C' in
2 minutes. So is Perl just a testbed for Perl specific I/O? Because if
it is, Perl loses every time, period !! Always did, always will!!

robic0

 

[A. Sinan Unur]

I'm getting unexplained errors when I read from a log file. I'm using
a "Safe Pipe Open", and my reference for this is:

http://www.litespeed.net/perldocs/lib/Pod/perlipc.html#Using_open_for_IPC

The same documentation is also available on your hard drive:

perldoc perlipc

my $logFile = 'full_path_to_logfile';
my $ipAddress = '12.34.56.78';

my $fh;
open( $fh, "grep $ipAddress $logFile | tail 2>&1 |" ) or
die "Cannot open logfile!!\n";

open my $fh, '-|', "grep $ipAddress $logFile | tail 2>&1"
or die "Cannot pipe: $!";

Always declare variables in the *smallest* applicable scope.

# get the last lines
my $logData = '';
while ( <$fh> )
{
$logData .= $_;
}

The code works perfectly and displays no errors. The probem is that
when I look in the system error file I see lines and lines of:

grep: writing output: Broken pipe

Exactly which log file are you referring to?

I cannot replicate this on FreeBSD 5.4 and Fedora Core.

Sorry can't be of any more help.

Sinan

 

[A. Sinan Unur]

....

my $fh;
open( $fh, '<', $logFile ) or
die "Cannot open logfile!!\n";

my $logData = '';
my $matches = 0;
while ( <$fh> )
{
if( index($_,$ipAddress) >= 0 ) {
$logData .= $_ ;
last if ++$matches >= 10;
}

That does not do the same thing as the OP's code though.

The OP's code finds the last ten occurences of the IP address
in the log file whereas your code finds the first ten.

For a pure Perl solution, you can use File::ReadBackwards:

#!/usr/bin/perl

use strict;
use warnings;

use File::ReadBackwards;

my $log = q{C:/opt/Apache2/logs/recex-error.log};
my $ip = '127.0.0.1';

tie *BW, 'File::ReadBackwards', $log
or die "Cannot read '$log': $!";

my ($n, @logData);

while ( <BW> ) {
next unless /$ip/o;
unshift @logData, substr $_, 0, 80;
++$n;
last if $n == 10;
}

print join q{}, @logData;
__END__

D:\Home\asu1\UseNet\clpmisc> log
[Tue Mar 21 09:45:32 2006] [error] [client 127.0.0.1] Global symbol "$child1_inc
[Tue Mar 21 09:46:27 2006] [error] [client 127.0.0.1] Global symbol "$child1_inc
[Tue Mar 21 09:47:07 2006] [error] [client 127.0.0.1] Global symbol "%data" requ
[Tue Mar 21 09:50:24 2006] [error] [client 127.0.0.1] Global symbol "$draw1" req
[Tue Mar 21 09:59:08 2006] [error] [client 127.0.0.1] syntax error at D:/Recex/l
[Tue Mar 21 10:27:31 2006] [error] [client 127.0.0.1] File does not exist: D:/Re
[Tue Mar 21 11:15:57 2006] [error] [client 127.0.0.1] Global symbol "$treatment"
[Tue Mar 21 13:59:12 2006] [error] [client 127.0.0.1] Bareword "LOCK_EX" not all
[Tue Mar 21 14:18:48 2006] [error] [client 127.0.0.1] syntax error at D:/Recex/l
[Tue Mar 21 14:40:34 2006] [error] [client 127.0.0.1] Bareword "Tie::IxHash" not


Sinan

 

[xhoster]

I'm getting unexplained errors when I read from a log file. I'm using a
"Safe Pipe Open", and my reference for this is:

http://www.litespeed.net/perldocs/lib/Pod/perlipc.html#Using_open_for_IPC

Your code bears little resemblence to what this reference discusses under
the heading of "Safe Pipe Opens". A key point of "Safe Pipe Open" is that
you bypass the shell by using
open my $fh, "-|" or die;
followed by a safe exec call. You clearly are not doing that.

(I'm not saying you need to be doing that, only that you should know
what it is you are doing, and safe pipe open is not what you are doing.)

my $fh;
open( $fh, "grep $ipAddress $logFile | tail 2>&1 |" ) or
die "Cannot open logfile!!\n";

Your problem may (or may not) be related to the 2>&1.
Why are you doing that?

....

The code works perfectly and displays no errors.

If it did emit error messages, would you know?

The probem is that
when I look in the system error file I see lines and lines of:

What is the system error file? Is that the same thing as perls STDERR?

grep: writing output: Broken pipe
grep: writing output: Broken pipe
grep: writing output: Broken pipe
grep: writing output: Broken pipe
grep: writing output: Broken pipe
grep: writing output: Broken pipe
grep: writing output: Broken pipe
grep: writing output: Broken pipe
grep: writing output: Broken pipe
grep: writing output: Broken pipe

Why are these errors there?

grep tried to write to tail, but tail was no longer listening. I've seen
this on some systems (SunOS, for example) when piping into head. It is
obvious why head stops listening early. Why tail would stop listening, I
don't know. I think you have a OS/shell problem, rather than a Perl
problem.

Xho