Parallel::ForkManager and Net::FTP problem?

[it_says_BALLS_on_your forehead]

Hello, I am running the following code:
#------
#!/apps/webstats/bin/perl

use strict; use warnings;
use Net::FTP;
use File::Copy;
use Parallel::ForkManager;

require "tools.pl";

my $area = shift;

my $processDate;
$processDate = '051006';

my $command = "ssh -l freclogs eclog31
/export/home/freclogs/simon/1-perl/getFileSize.pl $area";

my @result = `$command`;
my %fetch;

for (@result) {
chomp;
my ($key, $value) = split('=>');
$fetch{$key} = $value;
}

my $toList = "simon.chao\@fmr.com";

&startCycle( $area, 16, \%fetch );

#--- subs ---
sub startCycle {
my ($area, $num_groups, $data_ref) = @_;
my %data = %{$data_ref};

my $ftp;
my $server = 'eclog31';
unless ($ftp = Net::FTP->new($server)) {
doMail("Problem with $0", "Can't connect to $server with ftp,
$@\n", $toList);
die;
}

my ($userName, $password) = ('user','password');

unless ($ftp->login($userName, $password)) {
doMail("Problem with $0", "Can't login to $server with ftp using
-$userName- and -$password- $@\n", $toList);
die;
}
$ftp->binary();

my %missing;

my $pm = Parallel::ForkManager->new($num_groups);

while( my ($file, $size) = each %fetch ) {
$pm->start and next;

if ($size) {
# fetch and parse
my ($server, $dir, $file_type, $slice, $controlNo, $isCritical) =
split(/\|/, $file);

my $remoteFile = "$dir/$processDate.gz";
my $localFile =
"$cycleType{$area}->{rawLogs}/$file_type.$processDate$slice";

next if (-e $localFile);
if ($ftp->get($remoteFile, $localFile)) {
print "got $remoteFile to $localFile\n";
my $doneFile = $localFile;
$doneFile =~ s/^.*\///g;
$doneFile =~ s/\.gz$//;
$doneFile = "$cycleType{$area}->{work}/$doneFile";
# Kick Off Parsing for this file:
my $command = "preParse.pl $processDate $localFile $doneFile
$hash{$area}->{parseMethod}";
system($command);
}
else {
print "FTP MESSAGE: $ftp->message\n";
}
}
else {
# Capture missing logs to deal with later
# can add to a hash, or write to a file, or maybe even use
Storable.
# maybe sleep 300, then kick off another startCycle?

print "failed to get $file\n";
$missing{$file} = localtime();
}

$pm->finish;
}
$ftp->quit();

my $mailBody = "";
while( my ($missingFile, $time) = each %missing ) {
$mailBody .= "File $missingFile missing at $time\n";
}

print "$mailBody\n";
doMail("Subject", "$mailBody", $toList);
$pm->wait_all_children;
}

#-----

and the output is:

[smro180 191] /webstatmmk1/pre/authlogs/bin > betaProcess.pl authlogs &
[1] 20047
[smro180 192] /webstatmmk1/pre/authlogs/bin >

*************************************************************************
* WARNING: This system is restricted to authorized users for
legitimate *
* business purposes and is subject to audit. The unauthorized access,
*
* use, or modification of this system or of the data contained
therein *
* or in transit to/from it is a criminal violation of Federal and
*
* state laws
*

*************************************************************************
Argument "" isn't numeric in numeric comparison (<=>) at
../betaProcess.pl line 101.
eclog31|/logs/raw/fswas103/FebSec.febsec.audit|fsaudit.fidelity|.fswas103.gz

39214498
eclog31|/logs/raw/fswas101/FebSec.febsec.audit|fsaudit.fidelity|.fswas101.gz

38977495
eclog31|/logs/raw/fswas102/FebSec.febsec.audit|fsaudit.fidelity|.fswas102.gz

38807337
eclog31|/logs/raw/fswas401/FebSec.febsec.audit|fsaudit.fidelity|.fswas401.gz

37928934
eclog31|/logs/raw/fswas402/FebSec.febsec.audit|fsaudit.fidelity|.fswas402.gz

37771574
eclog31|/logs/raw/fswas403/FebSec.febsec.audit|fsaudit.fidelity|.fswas403.gz

37770895
eclog31|/logs/raw/fswas301/FebSec.febsec.audit|fsaudit.fidelity|.fswas301.gz

37762543
eclog31|/logs/raw/fswas303/FebSec.febsec.audit|fsaudit.fidelity|.fswas303.gz

37707907
eclog31|/logs/raw/fswas302/FebSec.febsec.audit|fsaudit.fidelity|.fswas302.gz

37608154
eclog31|/logs/raw/fswas102/FebSec.ibg.audit|fsaudit.ibg|.fswas102.gz

4062110
eclog31|/logs/raw/fswas103/FebSec.ibg.audit|fsaudit.ibg|.fswas103.gz

4056833
eclog31|/logs/raw/blah/FebSec.ibg.audit|fsaudit.ibg|.fswas403.gz

# the above ^^^ is fake, to test

FTP MESSAGE: Net::FTP=GLOB(0x57b1e4)->message
FTP MESSAGE: Net::FTP=GLOB(0x57b1e4)->message
FTP MESSAGE: Net::FTP=GLOB(0x57b1e4)->message
FTP MESSAGE: Net::FTP=GLOB(0x57b1e4)->message
FTP MESSAGE: Net::FTP=GLOB(0x57b1e4)->message
FTP MESSAGE: Net::FTP=GLOB(0x57b1e4)->message
FTP MESSAGE: Net::FTP=GLOB(0x57b1e4)->message
FTP MESSAGE: Net::FTP=GLOB(0x57b1e4)->message
FTP MESSAGE: Net::FTP=GLOB(0x57b1e4)->message
FTP MESSAGE: Net::FTP=GLOB(0x57b1e4)->message
failed to get
eclog31|/logs/raw/blah/FebSec.ibg.audit|fsaudit.ibg|.fswas403.gz
FTP MESSAGE: Net::FTP=GLOB(0x57b1e4)->message
FTP MESSAGE: Net::FTP=GLOB(0x57b1e4)->message
FTP MESSAGE: Net::FTP=GLOB(0x57b1e4)->message
FTP MESSAGE: Net::FTP=GLOB(0x57b1e4)->message
FTP MESSAGE: Net::FTP=GLOB(0x57b1e4)->message
Unable to close datastream at ./betaProcess.pl line 174



....i apologize for the copious amount of text, but i wanted to present
the problem as fully as possible.

there were even times when i got this message:
Cannot start another process while you are in the child process at
/apps/webstats/lib/perl5/site_perl/5.6.1/Parallel/ForkManager.pm line
281.


....that may have been because i did not kill all of the processes?
i tried ps -ef | grep beta
and did a kill 9 <pid> on all of them, but i'm not sure if i have to
wait for them all to die. even after i ps -ef | grep beta again, and
see that there are no pids associated with betaProcess.pl, i have a
suspicion (perhaps unwarranted) that the processes are still stubbornly
clutching on to life.

i thought this was going to be relatively simple to implement--but
after doing some research online i could not discover how to resolve
these issues.

basically i'm trying to fork some processes and FTP get some files,
after i get them, i call a script preParse.pl via a system() call. this
may be causing some difficulty, but i'm too new with
Parallel::ForkManager to have experience with this. if you've even read
to this point, you've impressed me . i hope someone can give me some
hints.

 

[it_says_BALLS_on_your forehead]

i apologize: the content of @results is the following:
each element contains: <filepath>=><filesize>

and the reason the output has all of that stuff is because of this
code:
for my $key ( sort{ $fetch{$b} <=> $fetch{$a} } keys %fetch ) {
my $entry = sprintf( "%-150s%-20s", $key, $fetch{$key});
print "$entry\n";
# print "$key ----> $fetch{$key}\n";
}

....which appears right before the delcaration/initialization of the
$toList variable.

again, i'm sorry for the mess, i tried to trim down the code, but
realized belatedly that it badly obfuscated the connection between the
code and its output.

 

[xhoster]

....

my $ftp;
my $server = 'eclog31';
unless ($ftp = Net::FTP->new($server)) {
doMail("Problem with $0", "Can't connect to $server with ftp,
$@\n", $toList);
die;
}

my ($userName, $password) = ('user','password');

unless ($ftp->login($userName, $password)) {
doMail("Problem with $0", "Can't login to $server with ftp using
-$userName- and -$password- $@\n", $toList);
die;
}
$ftp->binary();
....

while( my ($file, $size) = each %fetch ) {
$pm->start and next;

You have all the children trying to use the same FTP connection
simultaneously. In general, assume that things which connect to the outside
world are neither thread safe nor fork safe, unless you verify that they
are. Each child must get its own FTP connection, which means that that
connection should not be made until after the $pm->start(). The cleanest
way to do that is to make a subroutine that returns the connection object.

if ($size) {

Why not weed out things with zero $size before you get into the forking
loop?


....

failed to get
eclog31|/logs/raw/blah/FebSec.ibg.audit|fsaudit.ibg|.fswas403.gz
FTP MESSAGE: Net::FTP=GLOB(0x57b1e4)->message
FTP MESSAGE: Net::FTP=GLOB(0x57b1e4)->message
FTP MESSAGE: Net::FTP=GLOB(0x57b1e4)->message
FTP MESSAGE: Net::FTP=GLOB(0x57b1e4)->message
FTP MESSAGE: Net::FTP=GLOB(0x57b1e4)->message
Unable to close datastream at ./betaProcess.pl line 174

This is because Net::FTP is not thread or fork safe.

there were even times when i got this message:
Cannot start another process while you are in the child process at
/apps/webstats/lib/perl5/site_perl/5.6.1/Parallel/ForkManager.pm line
281.

That is because of this line:

next if (-e $localFile);

The child process hits the "next", and jumps to the top of the loop
(where it tries to start a new process), bypassing the $pm->finish that
would normally cause it to exit before it had a chance to get back to the
top.

If you want the child to be able to "next" out of the loop, then put the
$pm->finish in a "continue" block. (Then the parent will also call
"finish" when *it* invokes "next", but "finish" is a silent no-op when
called by the parent.)

It seems to me that the "next if (-e $localFile)" should be applied when
you build up your todo list in the first place, rather than in the forking
loop.

As a general rule, everything that initiates stateful connections to the
outside world must not span the forking loop, they should either be
entirely within the loop, or be entirely before the loop.

Also, everthing that is neither a stateful connection nor part the of the
bottleneck you are trying to solve (like your various filters) should go
outside the forking loop. Make it as simple as possible.

Xho

 

[it_says_BALLS_on_your forehead]

...

You have all the children trying to use the same FTP connection
simultaneously. In general, assume that things which connect to the outside
world are neither thread safe nor fork safe, unless you verify that they
are. Each child must get its own FTP connection, which means that that
connection should not be made until after the $pm->start(). The cleanest
way to do that is to make a subroutine that returns the connection object.

ahh, i see. i didn't want to make a connection for each file, but i
didn't consider the possibility that FTP->get was not fork safe. i'll
try your suggestion.

Why not weed out things with zero $size before you get into the forking
loop?

i'm operating under the assumption that the files will eventually be >
0, so i wanted to stick them in the queue for later processing. i
suppose i could create another hash of all the files of size 0 or
empty, but wouldn't i be doing the same amount of work? i don't think
i'd be saving myself any work--just shifting it around. i have a
pre-defined list of files i need to fetch, and i want to get a status
of each of them (here, the metric i'm using for status is size in
bytes). i could go through the hash again and remove the key/value
pairs where the size is zero or null, so that i wouldn't have to deal
with them in the forking loop, but now i did the work before the loop
anyway. although maybe grabbing the hash entries where the value is 0
or null is more efficient than looping through them in the fork loop
and doing an if ($value) check...

...



This is because Net::FTP is not thread or fork safe.
gotcha.


That is because of this line:


The child process hits the "next", and jumps to the top of the loop
(where it tries to start a new process), bypassing the $pm->finish that
would normally cause it to exit before it had a chance to get back to the
top.

If you want the child to be able to "next" out of the loop, then put the
$pm->finish in a "continue" block. (Then the parent will also call
"finish" when *it* invokes "next", but "finish" is a silent no-op when
called by the parent.)

It seems to me that the "next if (-e $localFile)" should be applied when
you build up your todo list in the first place, rather than in the forking
loop.

i see now. this may have taken me a while to figure out. you certainly
saved me a lot of headache here! thx!

As a general rule, everything that initiates stateful connections to the
outside world must not span the forking loop, they should either be
entirely within the loop, or be entirely before the loop.

Also, everthing that is neither a stateful connection nor part the of the
bottleneck you are trying to solve (like your various filters) should go
outside the forking loop. Make it as simple as possible.

great advice, thx again!