Parallel::ForkManager and Net::FTP problem?

Discussion in 'Perl Misc' started by it_says_BALLS_on_your forehead, Oct 7, 2005.

  1. Hello, I am running the following code:
    #------
    #!/apps/webstats/bin/perl

    use strict; use warnings;
    use Net::FTP;
    use File::Copy;
    use Parallel::ForkManager;

    require "tools.pl";

    my $area = shift;

    my $processDate;
    $processDate = '051006';

    my $command = "ssh -l freclogs eclog31
    /export/home/freclogs/simon/1-perl/getFileSize.pl $area";

    my @result = `$command`;
    my %fetch;

    for (@result) {
    chomp;
    my ($key, $value) = split('=>');
    $fetch{$key} = $value;
    }

    my $toList = "simon.chao\@fmr.com";

    &startCycle( $area, 16, \%fetch );

    #--- subs ---
    sub startCycle {
    my ($area, $num_groups, $data_ref) = @_;
    my %data = %{$data_ref};

    my $ftp;
    my $server = 'eclog31';
    unless ($ftp = Net::FTP->new($server)) {
    doMail("Problem with $0", "Can't connect to $server with ftp,
    $@\n", $toList);
    die;
    }

    my ($userName, $password) = ('user','password');

    unless ($ftp->login($userName, $password)) {
    doMail("Problem with $0", "Can't login to $server with ftp using
    -$userName- and -$password- $@\n", $toList);
    die;
    }
    $ftp->binary();

    my %missing;

    my $pm = Parallel::ForkManager->new($num_groups);

    while( my ($file, $size) = each %fetch ) {
    $pm->start and next;

    if ($size) {
    # fetch and parse
    my ($server, $dir, $file_type, $slice, $controlNo, $isCritical) =
    split(/\|/, $file);

    my $remoteFile = "$dir/$processDate.gz";
    my $localFile =
    "$cycleType{$area}->{rawLogs}/$file_type.$processDate$slice";

    next if (-e $localFile);
    if ($ftp->get($remoteFile, $localFile)) {
    print "got $remoteFile to $localFile\n";
    my $doneFile = $localFile;
    $doneFile =~ s/^.*\///g;
    $doneFile =~ s/\.gz$//;
    $doneFile = "$cycleType{$area}->{work}/$doneFile";
    # Kick Off Parsing for this file:
    my $command = "preParse.pl $processDate $localFile $doneFile
    $hash{$area}->{parseMethod}";
    system($command);
    }
    else {
    print "FTP MESSAGE: $ftp->message\n";
    }
    }
    else {
    # Capture missing logs to deal with later
    # can add to a hash, or write to a file, or maybe even use
    Storable.
    # maybe sleep 300, then kick off another startCycle?

    print "failed to get $file\n";
    $missing{$file} = localtime();
    }

    $pm->finish;
    }
    $ftp->quit();

    my $mailBody = "";
    while( my ($missingFile, $time) = each %missing ) {
    $mailBody .= "File $missingFile missing at $time\n";
    }

    print "$mailBody\n";
    doMail("Subject", "$mailBody", $toList);
    $pm->wait_all_children;
    }

    #-----

    and the output is:

    [smro180 191] /webstatmmk1/pre/authlogs/bin > betaProcess.pl authlogs &
    [1] 20047
    [smro180 192] /webstatmmk1/pre/authlogs/bin >

    *************************************************************************
    * WARNING: This system is restricted to authorized users for
    legitimate *
    * business purposes and is subject to audit. The unauthorized access,
    *
    * use, or modification of this system or of the data contained
    therein *
    * or in transit to/from it is a criminal violation of Federal and
    *
    * state laws
    *

    *************************************************************************
    Argument "" isn't numeric in numeric comparison (<=>) at
    ../betaProcess.pl line 101.
    eclog31|/logs/raw/fswas103/FebSec.febsec.audit|fsaudit.fidelity|.fswas103.gz

    39214498
    eclog31|/logs/raw/fswas101/FebSec.febsec.audit|fsaudit.fidelity|.fswas101.gz

    38977495
    eclog31|/logs/raw/fswas102/FebSec.febsec.audit|fsaudit.fidelity|.fswas102.gz

    38807337
    eclog31|/logs/raw/fswas401/FebSec.febsec.audit|fsaudit.fidelity|.fswas401.gz

    37928934
    eclog31|/logs/raw/fswas402/FebSec.febsec.audit|fsaudit.fidelity|.fswas402.gz

    37771574
    eclog31|/logs/raw/fswas403/FebSec.febsec.audit|fsaudit.fidelity|.fswas403.gz

    37770895
    eclog31|/logs/raw/fswas301/FebSec.febsec.audit|fsaudit.fidelity|.fswas301.gz

    37762543
    eclog31|/logs/raw/fswas303/FebSec.febsec.audit|fsaudit.fidelity|.fswas303.gz

    37707907
    eclog31|/logs/raw/fswas302/FebSec.febsec.audit|fsaudit.fidelity|.fswas302.gz

    37608154
    eclog31|/logs/raw/fswas102/FebSec.ibg.audit|fsaudit.ibg|.fswas102.gz

    4062110
    eclog31|/logs/raw/fswas103/FebSec.ibg.audit|fsaudit.ibg|.fswas103.gz

    4056833
    eclog31|/logs/raw/blah/FebSec.ibg.audit|fsaudit.ibg|.fswas403.gz

    # the above ^^^ is fake, to test

    FTP MESSAGE: Net::FTP=GLOB(0x57b1e4)->message
    FTP MESSAGE: Net::FTP=GLOB(0x57b1e4)->message
    FTP MESSAGE: Net::FTP=GLOB(0x57b1e4)->message
    FTP MESSAGE: Net::FTP=GLOB(0x57b1e4)->message
    FTP MESSAGE: Net::FTP=GLOB(0x57b1e4)->message
    FTP MESSAGE: Net::FTP=GLOB(0x57b1e4)->message
    FTP MESSAGE: Net::FTP=GLOB(0x57b1e4)->message
    FTP MESSAGE: Net::FTP=GLOB(0x57b1e4)->message
    FTP MESSAGE: Net::FTP=GLOB(0x57b1e4)->message
    FTP MESSAGE: Net::FTP=GLOB(0x57b1e4)->message
    failed to get
    eclog31|/logs/raw/blah/FebSec.ibg.audit|fsaudit.ibg|.fswas403.gz
    FTP MESSAGE: Net::FTP=GLOB(0x57b1e4)->message
    FTP MESSAGE: Net::FTP=GLOB(0x57b1e4)->message
    FTP MESSAGE: Net::FTP=GLOB(0x57b1e4)->message
    FTP MESSAGE: Net::FTP=GLOB(0x57b1e4)->message
    FTP MESSAGE: Net::FTP=GLOB(0x57b1e4)->message
    Unable to close datastream at ./betaProcess.pl line 174



    ....i apologize for the copious amount of text, but i wanted to present
    the problem as fully as possible.

    there were even times when i got this message:
    Cannot start another process while you are in the child process at
    /apps/webstats/lib/perl5/site_perl/5.6.1/Parallel/ForkManager.pm line
    281.


    ....that may have been because i did not kill all of the processes?
    i tried ps -ef | grep beta
    and did a kill 9 <pid> on all of them, but i'm not sure if i have to
    wait for them all to die. even after i ps -ef | grep beta again, and
    see that there are no pids associated with betaProcess.pl, i have a
    suspicion (perhaps unwarranted) that the processes are still stubbornly
    clutching on to life.

    i thought this was going to be relatively simple to implement--but
    after doing some research online i could not discover how to resolve
    these issues.

    basically i'm trying to fork some processes and FTP get some files,
    after i get them, i call a script preParse.pl via a system() call. this
    may be causing some difficulty, but i'm too new with
    Parallel::ForkManager to have experience with this. if you've even read
    to this point, you've impressed me :). i hope someone can give me some
    hints.
     
    it_says_BALLS_on_your forehead, Oct 7, 2005
    #1
    1. Advertising

  2. i apologize: the content of @results is the following:
    each element contains: <filepath>=><filesize>

    and the reason the output has all of that stuff is because of this
    code:
    for my $key ( sort{ $fetch{$b} <=> $fetch{$a} } keys %fetch ) {
    my $entry = sprintf( "%-150s%-20s", $key, $fetch{$key});
    print "$entry\n";
    # print "$key ----> $fetch{$key}\n";
    }

    ....which appears right before the delcaration/initialization of the
    $toList variable.

    again, i'm sorry for the mess, i tried to trim down the code, but
    realized belatedly that it badly obfuscated the connection between the
    code and its output.
     
    it_says_BALLS_on_your forehead, Oct 7, 2005
    #2
    1. Advertising

  3. it_says_BALLS_on_your forehead

    Guest

    "it_says_BALLS_on_your forehead" <> wrote:


    ....
    > my $ftp;
    > my $server = 'eclog31';
    > unless ($ftp = Net::FTP->new($server)) {
    > doMail("Problem with $0", "Can't connect to $server with ftp,
    > $@\n", $toList);
    > die;
    > }
    >
    > my ($userName, $password) = ('user','password');
    >
    > unless ($ftp->login($userName, $password)) {
    > doMail("Problem with $0", "Can't login to $server with ftp using
    > -$userName- and -$password- $@\n", $toList);
    > die;
    > }
    > $ftp->binary();
    >

    ....
    >
    > while( my ($file, $size) = each %fetch ) {
    > $pm->start and next;


    You have all the children trying to use the same FTP connection
    simultaneously. In general, assume that things which connect to the outside
    world are neither thread safe nor fork safe, unless you verify that they
    are. Each child must get its own FTP connection, which means that that
    connection should not be made until after the $pm->start(). The cleanest
    way to do that is to make a subroutine that returns the connection object.

    > if ($size) {


    Why not weed out things with zero $size before you get into the forking
    loop?


    ....


    > failed to get
    > eclog31|/logs/raw/blah/FebSec.ibg.audit|fsaudit.ibg|.fswas403.gz
    > FTP MESSAGE: Net::FTP=GLOB(0x57b1e4)->message
    > FTP MESSAGE: Net::FTP=GLOB(0x57b1e4)->message
    > FTP MESSAGE: Net::FTP=GLOB(0x57b1e4)->message
    > FTP MESSAGE: Net::FTP=GLOB(0x57b1e4)->message
    > FTP MESSAGE: Net::FTP=GLOB(0x57b1e4)->message
    > Unable to close datastream at ./betaProcess.pl line 174


    This is because Net::FTP is not thread or fork safe.

    >
    > there were even times when i got this message:
    > Cannot start another process while you are in the child process at
    > /apps/webstats/lib/perl5/site_perl/5.6.1/Parallel/ForkManager.pm line
    > 281.


    That is because of this line:

    > next if (-e $localFile);


    The child process hits the "next", and jumps to the top of the loop
    (where it tries to start a new process), bypassing the $pm->finish that
    would normally cause it to exit before it had a chance to get back to the
    top.

    If you want the child to be able to "next" out of the loop, then put the
    $pm->finish in a "continue" block. (Then the parent will also call
    "finish" when *it* invokes "next", but "finish" is a silent no-op when
    called by the parent.)

    It seems to me that the "next if (-e $localFile)" should be applied when
    you build up your todo list in the first place, rather than in the forking
    loop.

    As a general rule, everything that initiates stateful connections to the
    outside world must not span the forking loop, they should either be
    entirely within the loop, or be entirely before the loop.

    Also, everthing that is neither a stateful connection nor part the of the
    bottleneck you are trying to solve (like your various filters) should go
    outside the forking loop. Make it as simple as possible.

    Xho

    --
    -------------------- http://NewsReader.Com/ --------------------
    Usenet Newsgroup Service $9.95/Month 30GB
     
    , Oct 7, 2005
    #3
  4. wrote:
    > "it_says_BALLS_on_your forehead" <> wrote:
    >
    >
    > ...
    > > my $ftp;
    > > my $server = 'eclog31';
    > > unless ($ftp = Net::FTP->new($server)) {
    > > doMail("Problem with $0", "Can't connect to $server with ftp,
    > > $@\n", $toList);
    > > die;
    > > }
    > >
    > > my ($userName, $password) = ('user','password');
    > >
    > > unless ($ftp->login($userName, $password)) {
    > > doMail("Problem with $0", "Can't login to $server with ftp using
    > > -$userName- and -$password- $@\n", $toList);
    > > die;
    > > }
    > > $ftp->binary();
    > >

    > ...
    > >
    > > while( my ($file, $size) = each %fetch ) {
    > > $pm->start and next;

    >
    > You have all the children trying to use the same FTP connection
    > simultaneously. In general, assume that things which connect to the outside
    > world are neither thread safe nor fork safe, unless you verify that they
    > are. Each child must get its own FTP connection, which means that that
    > connection should not be made until after the $pm->start(). The cleanest
    > way to do that is to make a subroutine that returns the connection object.


    ahh, i see. i didn't want to make a connection for each file, but i
    didn't consider the possibility that FTP->get was not fork safe. i'll
    try your suggestion.

    >
    > > if ($size) {

    >
    > Why not weed out things with zero $size before you get into the forking
    > loop?
    >


    i'm operating under the assumption that the files will eventually be >
    0, so i wanted to stick them in the queue for later processing. i
    suppose i could create another hash of all the files of size 0 or
    empty, but wouldn't i be doing the same amount of work? i don't think
    i'd be saving myself any work--just shifting it around. i have a
    pre-defined list of files i need to fetch, and i want to get a status
    of each of them (here, the metric i'm using for status is size in
    bytes). i could go through the hash again and remove the key/value
    pairs where the size is zero or null, so that i wouldn't have to deal
    with them in the forking loop, but now i did the work before the loop
    anyway. although maybe grabbing the hash entries where the value is 0
    or null is more efficient than looping through them in the fork loop
    and doing an if ($value) check...

    >
    > ...
    >
    >
    > > failed to get
    > > eclog31|/logs/raw/blah/FebSec.ibg.audit|fsaudit.ibg|.fswas403.gz
    > > FTP MESSAGE: Net::FTP=GLOB(0x57b1e4)->message
    > > FTP MESSAGE: Net::FTP=GLOB(0x57b1e4)->message
    > > FTP MESSAGE: Net::FTP=GLOB(0x57b1e4)->message
    > > FTP MESSAGE: Net::FTP=GLOB(0x57b1e4)->message
    > > FTP MESSAGE: Net::FTP=GLOB(0x57b1e4)->message
    > > Unable to close datastream at ./betaProcess.pl line 174

    >
    > This is because Net::FTP is not thread or fork safe.


    gotcha.

    >
    > >
    > > there were even times when i got this message:
    > > Cannot start another process while you are in the child process at
    > > /apps/webstats/lib/perl5/site_perl/5.6.1/Parallel/ForkManager.pm line
    > > 281.

    >
    > That is because of this line:
    >
    > > next if (-e $localFile);

    >
    > The child process hits the "next", and jumps to the top of the loop
    > (where it tries to start a new process), bypassing the $pm->finish that
    > would normally cause it to exit before it had a chance to get back to the
    > top.
    >
    > If you want the child to be able to "next" out of the loop, then put the
    > $pm->finish in a "continue" block. (Then the parent will also call
    > "finish" when *it* invokes "next", but "finish" is a silent no-op when
    > called by the parent.)
    >
    > It seems to me that the "next if (-e $localFile)" should be applied when
    > you build up your todo list in the first place, rather than in the forking
    > loop.


    i see now. this may have taken me a while to figure out. you certainly
    saved me a lot of headache here! thx! :)

    >
    > As a general rule, everything that initiates stateful connections to the
    > outside world must not span the forking loop, they should either be
    > entirely within the loop, or be entirely before the loop.
    >
    > Also, everthing that is neither a stateful connection nor part the of the
    > bottleneck you are trying to solve (like your various filters) should go
    > outside the forking loop. Make it as simple as possible.
    >


    great advice, thx again!

    > Xho
    >
    > --
    > -------------------- http://NewsReader.Com/ --------------------
    > Usenet Newsgroup Service $9.95/Month 30GB
     
    it_says_BALLS_on_your forehead, Oct 10, 2005
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. nvp
    Replies:
    0
    Views:
    125
  2. nvp
    Replies:
    2
    Views:
    184
    Luc Heinrich
    Dec 31, 2009
  3. Replies:
    0
    Views:
    274
  4. chaitu
    Replies:
    2
    Views:
    166
    chaitu
    Sep 21, 2006
  5. Jacob JKW
    Replies:
    1
    Views:
    162
    Jacob JKW
    Jan 24, 2008
Loading...

Share This Page