Win32::Process - Limiting the number of processes launched from within a script

Discussion in 'Perl Misc' started by Patrick Paquet, Jan 27, 2004.

  1. Hello everyone,

    I've been working on a script that fetches information from all
    workstations on our NT domain, the problem I've now run into at this
    point is that it takes a very long time to execute completely. The
    reason for this is probably simply because I'm checking each machine
    in sequence (I check machine A, and only when I'm done do I move on to
    B etc).

    I've been looking into running these checks in parallel, via the
    Win32::process module, but I haven't found a way to control the
    number of simultaneous processes adequately. I can launch a certain
    number of them, but I can't figure out how to check the status of a
    process and to lower a counter when it's done, so that I can launch
    another one to replace it. What I'm looking to do is lauch no more
    then 10 (for example) instances of the script, and to stay as close to
    that without going over as long as there are machines to check.
    Making sense?

    I have been looking thru this group and various ressources on the net
    (thank you Dave Roth!) on and off for well over 2 weeks now, but
    haven't made much headway (although as usual I've picked up a couple
    of things along the way).

    I am open to alternatives as to how I can achieve this as well, I'm
    not dead set on Win32::process, so if you have ideas or suggestions,
    I'm interested. My current method generates several text files with
    listings of machines to check, and launches the script via the system
    command attaching the file as an argument, but I'm sure there's a more
    elegant way to do this, and I'm also pretty sure that it's not as
    difficult as it seems, but for now the solution escapes me.

    My current script is fully functionnal as far as the workstation
    checks are concerned. It is also a bit lengthy (+750 lines) so I
    won't post it here for brevity's sake. At this point it doesn't even
    include the Win32::process module as I just haven't been able to do
    much with it at all, besides launching or (way too) many processes.

    What I'm really asking for (after all this...) is some pointers and
    simple examples (if possible) as to how I can check the status of the
    processes and launch a new process on exit from the previous one.

    I'm running ActiveState Perl 5.005_02, and cannot upgrade it. Many
    scripts are run centrally and I don't have authority to do the upgrade
    on the script server.

    Thanks in advance for any assistance, suggestions, pearls of wisdow
    and all the fish!


    Patrick PAQUET
     
    Patrick Paquet, Jan 27, 2004
    #1
    1. Advertising

  2. Patrick Paquet

    Bill Guest

    Re: Win32::process - Limiting the number of processes launched fromwithin a script

    Patrick Paquet wrote:
    ....
    >
    > I've been looking into running these checks in parallel, via the
    > Win32::process module, but I haven't found a way to control the
    > number of simultaneous processes adequately. I can launch a certain
    >

    ....
    > My current script is fully functionnal as far as the workstation
    > checks are concerned. It is also a bit lengthy (+750 lines) so I
    > won't post it here for brevity's sake. At this point it doesn't even
    > include the Win32::process module as I just haven't been able to do
    > much with it at all, besides launching or (way too) many processes.
    >
    >
    > I'm running ActiveState Perl 5.005_02, and cannot upgrade it. Many
    > scripts are run centrally and I don't have authority to do the upgrade
    > on the script server.


    If I were you, I'd get 5.8 on the server and use threads and queues.

    If you really cannt do this, I suggest a more primitive solution--create
    a perl script (script1) that will stuff server names or IP's into a
    queue. Create a second script (script2) that will read the queue and
    check the server listed in the queue and output the results to a file
    (use flock). The scripts should be run from a cmd file like this:
    === cmd or .bat file ===
    perl script1.pl
    exit
    =============

    With a third script,
    set up a perl script like this:
    ===
    system("start script1.cmd");
    for(1..10) { system("start script2.cmd"; sleep 1 }

    ===

    ....and let the OS do the process management for you. Perl 5.005 process
    management is too broken for Win32 to bother agonizing over IMO. (It
    works okay in 5.8 though).
     
    Bill, Jan 27, 2004
    #2
    1. Advertising

  3. Patrick Paquet

    Chris Guest

    Re: Win32::process - Limiting the number of processes launched fromwithin a script

    Patrick Paquet wrote:
    > Hello everyone,
    >
    > I've been working on a script that fetches information from all
    > workstations on our NT domain, the problem I've now run into at this
    > point is that it takes a very long time to execute completely. The
    > reason for this is probably simply because I'm checking each machine
    > in sequence (I check machine A, and only when I'm done do I move on to
    > B etc).
    >
    > I've been looking into running these checks in parallel, via the
    > Win32::process module, but I haven't found a way to control the
    > number of simultaneous processes adequately. I can launch a certain
    > number of them, but I can't figure out how to check the status of a
    > process and to lower a counter when it's done, so that I can launch
    > another one to replace it. What I'm looking to do is lauch no more
    > then 10 (for example) instances of the script, and to stay as close to
    > that without going over as long as there are machines to check.
    > Making sense?
    >
    > [snipage]
    > I'm running ActiveState Perl 5.005_02, and cannot upgrade it. Many
    > scripts are run centrally and I don't have authority to do the upgrade
    > on the script server.
    >


    Ewww. I tried using Perl 5.005_03 once (from Hip Communications - the
    predecessor name to AS) and it was awful. I'd say as someone else said
    that your options are limited with this version of Perl - very limited.
    Not even David Roth can help you with that and he's king of Perl Win32...

    If it were me, I might try (roughly) writing a listener that I could put
    on every end point (node) that listened on say port 9010 (or some other
    port). Have a central listener on a central machine on port 9011 (or
    some other port) to handle replies to requests sent to the listener port
    of the end nodes. Have the central box run a request out to all the (or
    a batch of) listeners on port 9010 and wait for the replies to come back
    in on port 9011. You can batch your requests based on the capacity of
    the central listener (on port 9011) to handle replies. When the central
    node receives all the replies (or gives up trying/waiting due to end
    node down, etc), then you are done.

    The simplest way to write a listener in Perl on Windows (with the
    version you have) IMO is to combine your listener code with SRVANY.EXE
    (available on Windows RK CD) and create Windows service daemons.

    Chris
    -----
    Chris Olive
    chris +at+ technologEase +dot+ com
    http://www.technologEase.com
    (pronounced "technlogies")
     
    Chris, Jan 28, 2004
    #3
  4. (Patrick Paquet) wrote in message news:<>...
    <Original posting snipped>

    Hello everyone,

    I'm replying to my original post to clarify a bit the problem I was
    having, and also to post the solution that I am using, in the hopes
    that this will be helpful to someone else.

    Apart from the replies I've gotten in the newsgroup, I've also gotten
    2 suggestions directly to my e-mail. I've used one of those to fix my
    problem, I will post it at the end.

    First, some clarifications. The script is launched from a central
    server, and polls all the workstations, so nothing is actually sent
    from the PCs themselves, everything is fetched from the server. Most
    of the traffic is netbios stuff, read from registry or from the HD,
    nothing very fancy involved. Once the information is fetched, it's
    pumped in an SQL database. After that, various tools are used to
    generate batch jobs, or just reporting purposes.

    The code below was provided by Brian Helterline, and works perfectly
    for me once integrated and modified for my script. I launch 20 (still
    scaling at this point though) processes with it, when one exits,
    another is launched, it's exactly what I was looking for. I'm kinda
    new at this Perl thing and do not have any formal programming
    training, I just don't think of using some commands (never occured to
    me to use 'while' like this, or 'shift' even), so this is all very
    educationnal.

    A big thank you to all who took the time to reply, it is very much
    appreciated!

    Hope everybody has a nice day!


    Patrick


    ######

    use strict;
    use warnings;

    use Win32::process;

    my @machines = qw/ 1 2 3 4 5 6 7 8 9 10 11 12/; # list of machines to
    test
    my %running; # hash of processes running
    my $done = 0;
    while ( ! $done ) {
    if ( keys %running < 10 && @machines ) {
    my $obj;
    Win32::process::Create($obj,'C:\winnt\system32\notepad.exe',
    "notepad c:\\" . shift( @machines ) .
    '.txt',
    0,
    NORMAL_PRIORITY_CLASS,
    ".") || die Win32::GetLastError();
    $running{$obj} = $obj;
    print "started object $obj\n";

    }
    foreach my $obj ( keys %running ) {
    my $code;
    my $rc = $running{$obj}->GetExitCode( $code );
    delete $running{$obj} unless $code == 259;
    print "removed object $obj\n" unless $code == 259;
    }
    $done = not %running;
    }
    print "all done.\n";
     
    Patrick Paquet, Jan 29, 2004
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Tim Mackey
    Replies:
    0
    Views:
    302
    Tim Mackey
    Jul 27, 2004
  2. MARTIN Herve \(EXT\)
    Replies:
    1
    Views:
    1,462
    Anthony Borla
    Nov 27, 2003
  3. Charles Fox (Sheffield)
    Replies:
    9
    Views:
    736
    Adam Skutt
    Feb 12, 2011
  4. Pierre Morel
    Replies:
    5
    Views:
    123
    Pierre Morel
    Feb 17, 2010
  5. Replies:
    1
    Views:
    420
    Sisyphus
    Aug 31, 2005
Loading...

Share This Page