script hangs when run from command line and redirecting stdout and stderr to file

[it_says_BALLS_on_your forehead]

when i run this perl script from the command line directly, there is no
problem. when i run it like so:

script.pl > script.log 2>&1 &


....i noticed that script.log ceased being written to after a while--and
not always at the same point.

my script has print statements in it, which should be redirected to
script.log. i tried running the above line a couple of times, and some
of the lines that i printed were cut off in mid-string. i received no
error message though, and when i did a ps -ef | grep script.pl the
process was still running.

one pattern i noticed was that it always stopped after the first
&measure_url_performance was done. this may be because i only ran it
twice this way.

here is a short but (hopefully) complete working script:

#!/usr/bin/perl
use strict; use warnings;
use File::Find;
use Date::Manip;
use Mail::Sender;
use Statistics:escriptive;
require "tools.pl";


my (
$beginDate,
$numDays,
$chkCount,
$printDate,
$allRecs,
$logDate, # Log Date to look for in the format "YYDDMM", example
030509 for 5/9/03
$avgDuration,
$mailBody,
$maxDuration,
$minDuration,
$noDuration,
$pctile90,
$parseString, # String to look for in the logs. Enclose in
quotes if spaces are included.
$totFiles, # Total number of files reviewed.
$recCount, # Total records found in all files that match the
parseString
$stat, # Statistics:escriptive object
$yesterday
);

my @one_dirs = glob("/path1");
my @one_urls = ('this/is/a/string');

my @two_dirs = glob("/path2");
my @two_urls = ('this/is/another/string');

measure_url_performance( \@one_urls, \@one_dirs, 1 );
measure_url_performance( \@two_urls, \@two_dirs, 2 );

sub measure_url_performance {
my ($urls_ref, $dirs_ref, $type) = @_;

my @urls = $urls_ref ? @{$urls_ref} : ();
my @dirs = $dirs_ref ? @{$dirs_ref} : ();

my $wanted = get_parse_routine($type);

foreach (@urls) {
$parseString = $_;
print "parseString is -$parseString-\n";
$stat = Statistics:escriptive::Full->new();
for my $diff ( 0..$numDays ) {
$logDate = UnixDate(DateCalc($beginDate, " + $diff day"),
'%y%m%d');
$printDate = $logDate;
$printDate =~ s/(\d\d)(\d\d)(\d\d)/$2\/$3\/$1/;
$recCount = 0;
print "logDate is -$logDate-\n";
# find( \&process_file, @dirs );
find( $wanted, @dirs );
$allRecs += $recCount;
}
$pctile90 = sprintf( "%.3f", $stat->percentile(90) );
$avgDuration = sprintf( "%.3f",$stat->mean() );
$minDuration = sprintf( "%.3f", $stat->min() );
$maxDuration = sprintf( "%.3f", $stat->max() );
print LOGINCSV
"$parseString,$beginDate,$printDate,$allRecs,$pctile90,$avgDuration,$minDuration,$maxDuration\n";
print "Found $allRecs records with string $parseString in a
total of $totFiles files, $noDuration with no duration.\n";
$allRecs = $totFiles = 0;
}

}

sub get_parse_routine {
my ( $type ) = @_;
return \&parse_ewp if ( $type == 1 );
return \&parse_pslogin if ( $type == 2 );
return \&parse_login if ( $type == 3 );
}

sub parse_ewp {
my $fileName = $File::Find::name;
return unless ($fileName =~ /^.*$logDate\.gz$/);

print "Ready to process -$fileName-\n";
print "test scope, parseString set in calling context
-$parseString-\n";

my $chkCount;

$totFiles++;
open (ZIPFILE, "gzip -dc $fileName |") || die "Can't open $fileName
$!\n";
while (<ZIPFILE>) {
next unless /\Q$parseString\E/i;
$chkCount++;
chomp;

my $part2 = (split /"/,$_)[2];
my $duration = (split / /,$part2)[3];

# print "part2 is -$part2- duration is -$duration-\n";

if ($duration =~ /^\d[\d\.]*$/) {
$stat->add_data($duration);
$recCount++;
}
else {
$noDuration++;
}
next;
if ($chkCount > 10) {
$chkCount = 0;
last;
}
}
close ZIPFILE;
}

sub parse_pslogin {
my $fileName = $File::Find::name;
return unless ($fileName =~ /^.*$logDate\.gz$/);

print "Ready to process -$fileName-\n";
print "test scope, parseString set in calling context
-$parseString-\n";

my $chkCount;

$totFiles++;
open ( ZIPFILE, "gzip -dc $fileName |" ) || die "Can't open
$fileName $!\n";
while ( <ZIPFILE> ) {
next unless /\Q$parseString\E/i;
$chkCount++;
chomp;

my $part6 = (split /"/,$_)[6];
my $duration = (split /\ /,$part6)[5];

# print "part6 is -$part6- duration is -$duration-\n";

if ( $duration =~ /^\d[\d\.]*$/ ) {
$stat->add_data($duration);
$recCount++;
}
else {
$noDuration++;
}
next;
if ($chkCount > 10) {
$chkCount = 0;
last;
}
my $part2 = (split /"/,$_)[2];
my $duration = (split / /,$part2)[3];

# print "part2 is -$part2- duration is -$duration-\n";

if ($duration =~ /^\d[\d\.]*$/) {
$stat->add_data($duration);
$recCount++;
}
else {
$noDuration++;
}
next;
if ($chkCount > 10) {
$chkCount = 0;
last;
}
}
close ZIPFILE;
}



sub parse_login {
my $fileName = $File::Find::name;
return unless ($fileName =~ /^.*$logDate\.gz$/);

print "Ready to process -$fileName-\n";
print "test scope, parseString set in calling context
-$parseString-\n";

my $chkCount;

$totFiles++;
open ( ZIPFILE, "gzip -dc $fileName |" ) || die "Can't open
$fileName $!\n";
while ( <ZIPFILE> ) {
next unless /\Q$parseString\E/i;
$chkCount++;
chomp;

my $duration = (split /\t/,$_)[13];

# print "duration is -$duration-\n";

if ( $duration =~ /^\d[\d\.]*$/ ) {
$stat->add_data($duration);
$recCount++;
}
else {
$noDuration++;
}
next;
if ($chkCount > 10) {
$chkCount = 0;
last;
}
}
close ZIPFILE;
}

 

[Anno Siegel]

when i run this perl script from the command line directly, there is no
problem. when i run it like so:

script.pl > script.log 2>&1 &


...i noticed that script.log ceased being written to after a while--and
not always at the same point.

my script has print statements in it, which should be redirected to
script.log. i tried running the above line a couple of times, and some
of the lines that i printed were cut off in mid-string. i received no
error message though, and when i did a ps -ef | grep script.pl the
process was still running.

one pattern i noticed was that it always stopped after the first
&measure_url_performance was done. this may be because i only ran it
twice this way.

here is a short but (hopefully) complete working script:

Hopefully? Didn't you make a test run?

#!/usr/bin/perl
use strict; use warnings;
use File::Find;
use Date::Manip;
use Mail::Sender;

Where are you using Mail::Sender? You are making it harder than
to run your script.

use Statistics:escriptive;
require "tools.pl";

Where is tools.pl, what is in it. Do you need it?

my (
$beginDate,
$numDays,
$chkCount,
$printDate,
$allRecs,
$logDate, # Log Date to look for in the format "YYDDMM", example
030509 for 5/9/03
$avgDuration,
$mailBody,
$maxDuration,
$minDuration,
$noDuration,
$pctile90,
$parseString, # String to look for in the logs. Enclose in
quotes if spaces are included.
$totFiles, # Total number of files reviewed.
$recCount, # Total records found in all files that match the
parseString
$stat, # Statistics:escriptive object
$yesterday
);

Don't declare variables in one bulk. Declare each one close to its
first usage. Also, your comments wrap. Keep the line length below 72
characters on Usenet, in code as well as in other text.

my @one_dirs = glob("/path1");
my @one_urls = ('this/is/a/string');

my @two_dirs = glob("/path2");
my @two_urls = ('this/is/another/string');

measure_url_performance( \@one_urls, \@one_dirs, 1 );
measure_url_performance( \@two_urls, \@two_dirs, 2 );

sub measure_url_performance {
my ($urls_ref, $dirs_ref, $type) = @_;

my @urls = $urls_ref ? @{$urls_ref} : ();
my @dirs = $dirs_ref ? @{$dirs_ref} : ();

my $wanted = get_parse_routine($type);

foreach (@urls) {
$parseString = $_;
print "parseString is -$parseString-\n";
$stat = Statistics:escriptive::Full->new();
for my $diff ( 0..$numDays ) {

$numDays is a global variable (why?) that is nowhere set. Giving up
at this point. Too many errors, too much cruft. Post a minimal runnable
program.

Anno

 

[Joe Smith]

my script has print statements in it, which should be redirected to
script.log. i tried running the above line a couple of times, and some
of the lines that i printed were cut off in mid-string.

That's what happens when you neglect to set $| to a nonzero value.
Just add
$|++;
near the start of your program.
-Joe