Creating a 'load simulator' by calling Perl Programs - or Forking?

Discussion in 'Perl Misc' started by pgodfrin, Apr 3, 2008.

  1. pgodfrin

    pgodfrin Guest

    Greetings,

    I would like to create a 'driver' (simulation) program that calls
    existing perl programs, waits for them to complete and provides some
    feedback. Ideally I would like to have the driver echo to the screen
    that the simulation is still running and perhaps provide some
    information. That may be too ambitious, but that is the basic idea.

    I have a handful of perl programs that do things - read a database
    (kudos to the DBI modules) and output a report, from a few lines to a
    few thousand.

    Question #1: What is the proper way to execute the perl programs?

    I am familiar with exec() and system() and have done this:

    system("sallmr1 -n50000 >/dev/null");
    system("srngmr1 -sr -es -n10000 >/dev/null");
    system("srngmr2 -sr -es -n10000 >/dev/null");

    Which executes these three programs sequentially - or by adding '&' at
    the end I can execute them at the same time. Of course the 'calling'
    program cannot wait for the background tasks because of the shell
    return.

    Which leads me down the path of fork(). If I were to code the
    following:

    for(1..2)
    {
    if($pid=fork)
    {
    # parent
    } elsif (defined $pid)
    {
    # child
    exec("sallmr1 -n50000 >/dev/null");
    } # end child (elsif)
    } # end for loop
    do { $x=wait; print "$x died\n";} until $x==-1;

    Then I would get a pair of runs, and wait until each child dies.

    Question #2: Is there a way to echo to the screen that the wait() is
    happening and provide some indication of progress?

    To clarify this question, I had originally tried:
    do {$x=wait; print ".";} until $x==-1;

    Thinking I would get a period printed during the loop. (silly me -
    wait only comes back when a child process dies...).

    Of course if this is much to complicated a way to so a simulated load,
    then so be it - just a plain old fork and wait works fine... And I
    could always remove the /dev/null and actually see the called programs
    output, which will tell me that it's still running... But I was
    curious if there is a more elegant or at least interesting way of
    doing this?

    thanks,
    phil
     
    pgodfrin, Apr 3, 2008
    #1
    1. Advertising

  2. pgodfrin <> writes:

    > Greetings,
    >
    > I would like to create a 'driver' (simulation) program that calls
    > existing perl programs, waits for them to complete and provides some
    > feedback. Ideally I would like to have the driver echo to the screen
    > that the simulation is still running and perhaps provide some
    > information. That may be too ambitious, but that is the basic idea.


    It may take a bit of getting used to, but POE::Wheel::Run can do this
    very nicely, including monitoring of the fork'd procesesses' STDOUT and
    STDERR, catching signals etc. And knowing POE will be useful in many
    event-based processes.

    See:
    http://search.cpan.org/~rcaputo/POE-1.0000/lib/POE/Wheel/Run.pm

    and the section "Process Management" in the POE cookbook:
    http://poe.perl.org/?POE_Cookbook


    --
    Joost Diepenmaat | blog: http://joost.zeekat.nl/ | work: http://zeekat.nl/
     
    Joost Diepenmaat, Apr 3, 2008
    #2
    1. Advertising

  3. pgodfrin

    pgodfrin Guest

    Re: Creating a 'load simulator' by calling Perl Programs - orForking?

    On Apr 3, 3:30 pm, Joost Diepenmaat <> wrote:
    > pgodfrin <> writes:
    > > Greetings,

    >
    > > I would like to create a 'driver' (simulation) program that calls
    > > existing perl programs, waits for them to complete and provides some
    > > feedback. Ideally I would like to have the driver echo to the screen
    > > that the simulation is still running and perhaps provide some
    > > information. That may be too ambitious, but that is the basic idea.

    >
    > It may take a bit of getting used to, but POE::Wheel::Run can do this
    > very nicely, including monitoring of the fork'd procesesses' STDOUT and
    > STDERR, catching signals etc. And knowing POE will be useful in many
    > event-based processes.
    >
    > See:http://search.cpan.org/~rcaputo/POE-1.0000/lib/POE/Wheel/Run.pm
    >
    > and the section "Process Management" in the POE cookbook:http://poe.perl.org/?POE_Cookbook
    >
    > --
    > Joost Diepenmaat | blog:http://joost.zeekat.nl/| work:http://zeekat.nl/


    I was looking at IPC::Run as well. What do you think of it?
    pg
     
    pgodfrin, Apr 3, 2008
    #3
  4. Re: Creating a 'load simulator' by calling Perl Programs - or Forking?

    pgodfrin <> writes:

    > I was looking at IPC::Run as well. What do you think of it?


    Haven't used it all. And to be honest, from skimming the docs I can't
    really see if it will even do what you want it to do if you have more
    than one child process running, which I can guarantee you POE will.

    But as I implied, POE has a bit of a learning curve, and it requires you
    to (re)structure your code in a event handling way. YMMV, probably :)

    --
    Joost Diepenmaat | blog: http://joost.zeekat.nl/ | work: http://zeekat.nl/
     
    Joost Diepenmaat, Apr 3, 2008
    #4
  5. pgodfrin

    pgodfrin Guest

    Re: Creating a 'load simulator' by calling Perl Programs - orForking?

    On Apr 3, 3:49 pm, Joost Diepenmaat <> wrote:
    > pgodfrin <> writes:
    > > I was looking at IPC::Run as well. What do you think of it?

    >
    > Haven't used it all. And to be honest, from skimming the docs I can't
    > really see if it will even do what you want it to do if you have more
    > than one child process running, which I can guarantee you POE will.
    >
    > But as I implied, POE has a bit of a learning curve, and it requires you
    > to (re)structure your code in a event handling way. YMMV, probably :)
    >
    > --
    > Joost Diepenmaat | blog:http://joost.zeekat.nl/| work:http://zeekat.nl/


    Yeah - I agree. As I've mentioned in the past, I'm just a lowly DBA
    (out of work btw) who's trying to not only learn Perl real good, but
    use it simply to drive my databases so I can do DBA stuff - you know
    like look at CPU util, I/O, funky SQL, etc.

    So - I'm certain the POE will do the trick, on your say-so, but I'm
    not sure if it is worth the learning curve at this juncture. But I
    have bookmarked it and I will at least peruse it to see what it's all
    about.

    You never know...

    smiles,
    pg
     
    pgodfrin, Apr 3, 2008
    #5
  6. pgodfrin

    Guest

    pgodfrin <> wrote:

    ....
    > Question #1: What is the proper way to execute the perl programs?


    You gave two examples, sequentially and in parallel. Which one is
    appropriate depends on what kind of load you want to simulate, which in
    turn depends on why are you are doing the simulation.


    > Question #2: Is there a way to echo to the screen that the wait() is
    > happening and provide some indication of progress?
    >
    > To clarify this question, I had originally tried:
    > do {$x=wait; print ".";} until $x==-1;
    >
    > Thinking I would get a period printed during the loop. (silly me -
    > wait only comes back when a child process dies...).


    So you do get a period printed, once per child-exit. This is what I would
    want to do, anyway, but I seem to be in a philosophical minority in this
    regard (I hate progress indicators that merely indicate that the system
    clock is ticking, and nothing about the actual progress of the actual
    program.) I'm not sure what you originally thought it might do instead.
    Periods to be printed as fast as Perl possible can print them?



    > Of course if this is much to complicated a way to so a simulated load,
    > then so be it - just a plain old fork and wait works fine... And I
    > could always remove the /dev/null and actually see the called programs
    > output, which will tell me that it's still running... But I was
    > curious if there is a more elegant or at least interesting way of
    > doing this?


    If the waiting program is still running, and is still waiting, then the
    program it is waiting on must still be running. What other possibilities
    are there?

    If you really want the "time has not come to a halt" thing, then
    something like this:

    perl -le '$|=1; fork or do {sleep 4*$_; exit} foreach (1..3); \
    $SIG{ALRM}=sub{print "."; alarm 1}; alarm 1; \
    do {$x=wait; print "$x";} until $x==-1; \
    alarm 0'
    ..
    ..
    ..
    20287
    ..
    ..
    ..
    ..
    20288
    ..
    ..
    ..
    ..
    20289
    -1


    Xho

    --
    -------------------- http://NewsReader.Com/ --------------------
    The costs of publication of this article were defrayed in part by the
    payment of page charges. This article must therefore be hereby marked
    advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
    this fact.
     
    , Apr 3, 2008
    #6
  7. pgodfrin

    pgodfrin Guest

    Re: Creating a 'load simulator' by calling Perl Programs - orForking?

    On Apr 3, 4:22 pm, wrote:
    > pgodfrin <> wrote:
    >
    > ...
    >
    > > Question #1: What is the proper way to execute the perl programs?

    >
    > You gave two examples, sequentially and in parallel. Which one is
    > appropriate depends on what kind of load you want to simulate, which in
    > turn depends on why are you are doing the simulation.
    >
    > > Question #2: Is there a way to echo to the screen that the wait() is
    > > happening and provide some indication of progress?

    >
    > > To clarify this question, I had originally tried:
    > > do {$x=wait; print ".";} until $x==-1;

    >
    > > Thinking I would get a period printed during the loop. (silly me -
    > > wait only comes back when a child process dies...).

    >
    > So you do get a period printed, once per child-exit. This is what I would
    > want to do, anyway, but I seem to be in a philosophical minority in this
    > regard (I hate progress indicators that merely indicate that the system
    > clock is ticking, and nothing about the actual progress of the actual
    > program.) I'm not sure what you originally thought it might do instead.
    > Periods to be printed as fast as Perl possible can print them?
    >
    > > Of course if this is much to complicated a way to so a simulated load,
    > > then so be it - just a plain old fork and wait works fine... And I
    > > could always remove the /dev/null and actually see the called programs
    > > output, which will tell me that it's still running... But I was
    > > curious if there is a more elegant or at least interesting way of
    > > doing this?

    >
    > If the waiting program is still running, and is still waiting, then the
    > program it is waiting on must still be running. What other possibilities
    > are there?
    >
    > If you really want the "time has not come to a halt" thing, then
    > something like this:
    >
    > perl -le '$|=1; fork or do {sleep 4*$_; exit} foreach (1..3); \
    > $SIG{ALRM}=sub{print "."; alarm 1}; alarm 1; \
    > do {$x=wait; print "$x";} until $x==-1; \
    > alarm 0'
    > .
    > .
    > .
    > 20287
    > .
    > .
    > .
    > .
    > 20288
    > .
    > .
    > .
    > .
    > 20289
    > -1
    >
    > Xho
    >
    > --
    > --------------------http://NewsReader.Com/--------------------
    > The costs of publication of this article were defrayed in part by the
    > payment of page charges. This article must therefore be hereby marked
    > advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
    > this fact.


    Hi Xho,
    As always - good points. I am looking to load up the system, so a
    parallel simulation seems to be the best approach. And frankly, the
    period printing was just a way to see that indeed something was
    happening, but clearly and wholly unnecessary - although I did get a
    chuckle over the time halting comment...

    And anyway 'watch ps u' in anotehr session celarly shows me the
    processes still running.

    But - the question still remains - what's the best way to call other
    perl programs? Exec, system or something else?

    pg
     
    pgodfrin, Apr 4, 2008
    #7
  8. pgodfrin

    Guest

    pgodfrin <> wrote:
    >
    > But - the question still remains - what's the best way to call other
    > perl programs? Exec, system or something else?


    I'd do fork and exec. If you want to do something in the child after
    the other thing was done, then I'd do fork and system.

    Xho

    --
    -------------------- http://NewsReader.Com/ --------------------
    The costs of publication of this article were defrayed in part by the
    payment of page charges. This article must therefore be hereby marked
    advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
    this fact.
     
    , Apr 4, 2008
    #8
  9. pgodfrin

    Guest

    Re: Creating a 'load simulator' by calling Perl Programs - orForking?

    On Apr 4, 6:12 am, pgodfrin <> wrote:
    > On Apr 3, 4:22 pm, wrote:
    >
    >
    >
    > > pgodfrin <> wrote:

    >
    > > ...

    >
    > > > Question #1: What is the proper way to execute the perl programs?

    >
    > > You gave two examples, sequentially and in parallel. Which one is
    > > appropriate depends on what kind of load you want to simulate, which in
    > > turn depends on why are you are doing the simulation.

    >
    > > > Question #2: Is there a way to echo to the screen that the wait() is
    > > > happening and provide some indication of progress?

    >
    > > > To clarify this question, I had originally tried:
    > > > do {$x=wait; print ".";} until $x==-1;

    >
    > > > Thinking I would get a period printed during the loop. (silly me -
    > > > wait only comes back when a child process dies...).

    >
    > > So you do get a period printed, once per child-exit. This is what I would
    > > want to do, anyway, but I seem to be in a philosophical minority in this
    > > regard (I hate progress indicators that merely indicate that the system
    > > clock is ticking, and nothing about the actual progress of the actual
    > > program.) I'm not sure what you originally thought it might do instead.
    > > Periods to be printed as fast as Perl possible can print them?

    >
    > > > Of course if this is much to complicated a way to so a simulated load,
    > > > then so be it - just a plain old fork and wait works fine... And I
    > > > could always remove the /dev/null and actually see the called programs
    > > > output, which will tell me that it's still running... But I was
    > > > curious if there is a more elegant or at least interesting way of
    > > > doing this?

    >
    > > If the waiting program is still running, and is still waiting, then the
    > > program it is waiting on must still be running. What other possibilities
    > > are there?

    >
    > > If you really want the "time has not come to a halt" thing, then
    > > something like this:

    >
    > > perl -le '$|=1; fork or do {sleep 4*$_; exit} foreach (1..3); \
    > > $SIG{ALRM}=sub{print "."; alarm 1}; alarm 1; \
    > > do {$x=wait; print "$x";} until $x==-1; \
    > > alarm 0'
    > > .
    > > .
    > > .
    > > 20287
    > > .
    > > .
    > > .
    > > .
    > > 20288
    > > .
    > > .
    > > .
    > > .
    > > 20289
    > > -1

    >
    > > Xho

    >
    > > --
    > > --------------------http://NewsReader.Com/--------------------
    > > The costs of publication of this article were defrayed in part by the
    > > payment of page charges. This article must therefore be hereby marked
    > > advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
    > > this fact.

    >
    > Hi Xho,
    > As always - good points. I am looking to load up the system, so a
    > parallel simulation seems to be the best approach. And frankly, the
    > period printing was just a way to see that indeed something was
    > happening, but clearly and wholly unnecessary - although I did get a
    > chuckle over the time halting comment...
    >
    > And anyway 'watch ps u' in anotehr session celarly shows me the
    > processes still running.
    >
    > But - the question still remains - what's the best way to call other
    > perl programs? Exec, system or something else?
    >
    > pg


    Hi pg,

    a module that I've written couple of months ago was for that very
    purpose.
    Basically your script will have to instantiate an PreforkAgent, tell
    it how many children to beget (degree of parallelization) and provide
    it with a couple of callbacks as means of communication.
    These callback routines may provide any task, execute it as you like
    and perform evaluation of results and logging.
    The module's initial purpose was it to generate as much load on a
    remote system as the local server would allow. You can however, setup
    a small number of children and even delay the task assignments without
    interfering with the former tasks.
    Children will live and be re-used until the job-queue is depleted.

    You might download the module from http://www.atablis.com/temp/PreforkAgent.pm
    and give it a try.

    Cheers,
    Steffen



    It is rather easy to integrate
     
    , Apr 4, 2008
    #9
  10. pgodfrin

    pgodfrin Guest

    Re: Creating a 'load simulator' by calling Perl Programs - orForking?

    On Apr 4, 1:44 am, wrote:
    > On Apr 4, 6:12 am, pgodfrin <> wrote:
    >
    >
    >
    > > On Apr 3, 4:22 pm, wrote:

    >
    > > > pgodfrin <> wrote:

    >
    > > > ...

    >
    > > > > Question #1: What is the proper way to execute the perl programs?

    >
    > > > You gave two examples, sequentially and in parallel. Which one is
    > > > appropriate depends on what kind of load you want to simulate, which in
    > > > turn depends on why are you are doing the simulation.

    >
    > > > > Question #2: Is there a way to echo to the screen that the wait() is
    > > > > happening and provide some indication of progress?

    >
    > > > > To clarify this question, I had originally tried:
    > > > > do {$x=wait; print ".";} until $x==-1;

    >
    > > > > Thinking I would get a period printed during the loop. (silly me -
    > > > > wait only comes back when a child process dies...).

    >
    > > > So you do get a period printed, once per child-exit. This is what I would
    > > > want to do, anyway, but I seem to be in a philosophical minority in this
    > > > regard (I hate progress indicators that merely indicate that the system
    > > > clock is ticking, and nothing about the actual progress of the actual
    > > > program.) I'm not sure what you originally thought it might do instead.
    > > > Periods to be printed as fast as Perl possible can print them?

    >
    > > > > Of course if this is much to complicated a way to so a simulated load,
    > > > > then so be it - just a plain old fork and wait works fine... And I
    > > > > could always remove the /dev/null and actually see the called programs
    > > > > output, which will tell me that it's still running... But I was
    > > > > curious if there is a more elegant or at least interesting way of
    > > > > doing this?

    >
    > > > If the waiting program is still running, and is still waiting, then the
    > > > program it is waiting on must still be running. What other possibilities
    > > > are there?

    >
    > > > If you really want the "time has not come to a halt" thing, then
    > > > something like this:

    >
    > > > perl -le '$|=1; fork or do {sleep 4*$_; exit} foreach (1..3); \
    > > > $SIG{ALRM}=sub{print "."; alarm 1}; alarm 1; \
    > > > do {$x=wait; print "$x";} until $x==-1; \
    > > > alarm 0'
    > > > .
    > > > .
    > > > .
    > > > 20287
    > > > .
    > > > .
    > > > .
    > > > .
    > > > 20288
    > > > .
    > > > .
    > > > .
    > > > .
    > > > 20289
    > > > -1

    >
    > > > Xho

    >
    > > > --
    > > > --------------------http://NewsReader.Com/--------------------
    > > > The costs of publication of this article were defrayed in part by the
    > > > payment of page charges. This article must therefore be hereby marked
    > > > advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
    > > > this fact.

    >
    > > Hi Xho,
    > > As always - good points. I am looking to load up the system, so a
    > > parallel simulation seems to be the best approach. And frankly, the
    > > period printing was just a way to see that indeed something was
    > > happening, but clearly and wholly unnecessary - although I did get a
    > > chuckle over the time halting comment...

    >
    > > And anyway 'watch ps u' in anotehr session celarly shows me the
    > > processes still running.

    >
    > > But - the question still remains - what's the best way to call other
    > > perl programs? Exec, system or something else?

    >
    > > pg

    >
    > Hi pg,
    >
    > a module that I've written couple of months ago was for that very
    > purpose.
    > Basically your script will have to instantiate an PreforkAgent, tell
    > it how many children to beget (degree of parallelization) and provide
    > it with a couple of callbacks as means of communication.
    > These callback routines may provide any task, execute it as you like
    > and perform evaluation of results and logging.
    > The module's initial purpose was it to generate as much load on a
    > remote system as the local server would allow. You can however, setup
    > a small number of children and even delay the task assignments without
    > interfering with the former tasks.
    > Children will live and be re-used until the job-queue is depleted.
    >
    > You might download the module fromhttp://www.atablis.com/temp/PreforkAgent.pm
    > and give it a try.
    >
    > Cheers,
    > Steffen
    >
    > It is rather easy to integrate


    cool! I'll take a look...
    phil
     
    pgodfrin, Apr 4, 2008
    #10
  11. pgodfrin

    J. Gleixner Guest

    Re: Creating a 'load simulator' by calling Perl Programs - or Forking?

    wrote:
    [...]
    > a module that I've written couple of months ago was for that very
    > purpose.
    > Basically your script will have to instantiate an PreforkAgent, tell
    > it how many children to beget (degree of parallelization) and provide
    > it with a couple of callbacks as means of communication.


    Hu.. sounds pretty much like Parallel::ForkManager.
     
    J. Gleixner, Apr 4, 2008
    #11
  12. pgodfrin

    Guest

    Re: Creating a 'load simulator' by calling Perl Programs - orForking?

    On Apr 4, 6:19 pm, "J. Gleixner" <>
    wrote:
    > wrote:
    >
    > [...]
    >
    > Hu.. sounds pretty much like Parallel::ForkManager.


    That might well be.
    PreforkAgent came into being after I'd 'inherited' a customized
    loadtesting script to which more and more functionality was to add. So
    I was looking for a means of abstraction. Only after that was
    finished, did I peak into the Parallel namespace.
    Yet I didn't bother to explore how much of my work, which I had
    enjoyed, has been gratuitous.
    :)

    steffen
     
    , Apr 4, 2008
    #12
  13. pgodfrin

    Guest

    Re: Creating a 'load simulator' by calling Perl Programs - orForking?

    On Apr 4, 6:19 pm, "J. Gleixner" <>
    wrote:
    > wrote:
    >
    > [...]
    >
    > > a module that I've written couple of months ago was for that very
    > > purpose.
    > > Basically your script will have to instantiate anPreforkAgent, tell
    > > it how many children to beget (degree of parallelization) and provide
    > > it with a couple of callbacks as means of communication.

    >
    > Hu.. sounds pretty much like Parallel::ForkManager.


    Now I got the chance to look at Parallel::ForkManager and get back on
    this.

    Well, the obvious difference is that PreforkAgent preforks while
    Parallel::ForkManager does very much the same as fork(), only limiting
    the child number and abstracting waitpid.
    And as I wrote above
    > > Children will live and be re-used until the job-queue is depleted.


    While the Parallel::ForkManager is creating a new process on every job
    (with all the performance overhead that involves), PreforkAgent does
    no forking once the work 's been taken up and its children do exit
    only after the last job was assigned.

    The basic incentive for PreforkAgent.pm was making the most efficient
    use of the host's ressources. (E. g., to wreak havoc with real big
    application servers for the sake of load simulation.)

    steffen
     
    , Apr 24, 2008
    #13
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. R Evans
    Replies:
    3
    Views:
    1,015
    R Evans
    Sep 3, 2004
  2. Replies:
    12
    Views:
    1,673
    Dave Thompson
    Jan 10, 2005
  3. Coca
    Replies:
    7
    Views:
    757
    Aidan Grey
    Aug 24, 2004
  4. Casey Hawthorne
    Replies:
    14
    Views:
    456
  5. yezi

    Simulator using perl ?

    yezi, Sep 3, 2005, in forum: Perl Misc
    Replies:
    0
    Views:
    89
Loading...

Share This Page