Creating a 'load simulator' by calling Perl Programs - or Forking?

P

pgodfrin

Greetings,

I would like to create a 'driver' (simulation) program that calls
existing perl programs, waits for them to complete and provides some
feedback. Ideally I would like to have the driver echo to the screen
that the simulation is still running and perhaps provide some
information. That may be too ambitious, but that is the basic idea.

I have a handful of perl programs that do things - read a database
(kudos to the DBI modules) and output a report, from a few lines to a
few thousand.

Question #1: What is the proper way to execute the perl programs?

I am familiar with exec() and system() and have done this:

system("sallmr1 -n50000 >/dev/null");
system("srngmr1 -sr -es -n10000 >/dev/null");
system("srngmr2 -sr -es -n10000 >/dev/null");

Which executes these three programs sequentially - or by adding '&' at
the end I can execute them at the same time. Of course the 'calling'
program cannot wait for the background tasks because of the shell
return.

Which leads me down the path of fork(). If I were to code the
following:

for(1..2)
{
if($pid=fork)
{
# parent
} elsif (defined $pid)
{
# child
exec("sallmr1 -n50000 >/dev/null");
} # end child (elsif)
} # end for loop
do { $x=wait; print "$x died\n";} until $x==-1;

Then I would get a pair of runs, and wait until each child dies.

Question #2: Is there a way to echo to the screen that the wait() is
happening and provide some indication of progress?

To clarify this question, I had originally tried:
do {$x=wait; print ".";} until $x==-1;

Thinking I would get a period printed during the loop. (silly me -
wait only comes back when a child process dies...).

Of course if this is much to complicated a way to so a simulated load,
then so be it - just a plain old fork and wait works fine... And I
could always remove the /dev/null and actually see the called programs
output, which will tell me that it's still running... But I was
curious if there is a more elegant or at least interesting way of
doing this?

thanks,
phil
 
J

Joost Diepenmaat

pgodfrin said:
Greetings,

I would like to create a 'driver' (simulation) program that calls
existing perl programs, waits for them to complete and provides some
feedback. Ideally I would like to have the driver echo to the screen
that the simulation is still running and perhaps provide some
information. That may be too ambitious, but that is the basic idea.

It may take a bit of getting used to, but POE::Wheel::Run can do this
very nicely, including monitoring of the fork'd procesesses' STDOUT and
STDERR, catching signals etc. And knowing POE will be useful in many
event-based processes.

See:
http://search.cpan.org/~rcaputo/POE-1.0000/lib/POE/Wheel/Run.pm

and the section "Process Management" in the POE cookbook:
http://poe.perl.org/?POE_Cookbook
 
P

pgodfrin

J

Joost Diepenmaat

pgodfrin said:
I was looking at IPC::Run as well. What do you think of it?

Haven't used it all. And to be honest, from skimming the docs I can't
really see if it will even do what you want it to do if you have more
than one child process running, which I can guarantee you POE will.

But as I implied, POE has a bit of a learning curve, and it requires you
to (re)structure your code in a event handling way. YMMV, probably :)
 
P

pgodfrin

Haven't used it all. And to be honest, from skimming the docs I can't
really see if it will even do what you want it to do if you have more
than one child process running, which I can guarantee you POE will.

But as I implied, POE has a bit of a learning curve, and it requires you
to (re)structure your code in a event handling way. YMMV, probably :)

Yeah - I agree. As I've mentioned in the past, I'm just a lowly DBA
(out of work btw) who's trying to not only learn Perl real good, but
use it simply to drive my databases so I can do DBA stuff - you know
like look at CPU util, I/O, funky SQL, etc.

So - I'm certain the POE will do the trick, on your say-so, but I'm
not sure if it is worth the learning curve at this juncture. But I
have bookmarked it and I will at least peruse it to see what it's all
about.

You never know...

smiles,
pg
 
X

xhoster

....
Question #1: What is the proper way to execute the perl programs?

You gave two examples, sequentially and in parallel. Which one is
appropriate depends on what kind of load you want to simulate, which in
turn depends on why are you are doing the simulation.

Question #2: Is there a way to echo to the screen that the wait() is
happening and provide some indication of progress?

To clarify this question, I had originally tried:
do {$x=wait; print ".";} until $x==-1;

Thinking I would get a period printed during the loop. (silly me -
wait only comes back when a child process dies...).

So you do get a period printed, once per child-exit. This is what I would
want to do, anyway, but I seem to be in a philosophical minority in this
regard (I hate progress indicators that merely indicate that the system
clock is ticking, and nothing about the actual progress of the actual
program.) I'm not sure what you originally thought it might do instead.
Periods to be printed as fast as Perl possible can print them?


Of course if this is much to complicated a way to so a simulated load,
then so be it - just a plain old fork and wait works fine... And I
could always remove the /dev/null and actually see the called programs
output, which will tell me that it's still running... But I was
curious if there is a more elegant or at least interesting way of
doing this?

If the waiting program is still running, and is still waiting, then the
program it is waiting on must still be running. What other possibilities
are there?

If you really want the "time has not come to a halt" thing, then
something like this:

perl -le '$|=1; fork or do {sleep 4*$_; exit} foreach (1..3); \
$SIG{ALRM}=sub{print "."; alarm 1}; alarm 1; \
do {$x=wait; print "$x";} until $x==-1; \
alarm 0'
..
..
..
20287
..
..
..
..
20288
..
..
..
..
20289
-1


Xho

--
-------------------- http://NewsReader.Com/ --------------------
The costs of publication of this article were defrayed in part by the
payment of page charges. This article must therefore be hereby marked
advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
this fact.
 
P

pgodfrin

...


You gave two examples, sequentially and in parallel. Which one is
appropriate depends on what kind of load you want to simulate, which in
turn depends on why are you are doing the simulation.




So you do get a period printed, once per child-exit. This is what I would
want to do, anyway, but I seem to be in a philosophical minority in this
regard (I hate progress indicators that merely indicate that the system
clock is ticking, and nothing about the actual progress of the actual
program.) I'm not sure what you originally thought it might do instead.
Periods to be printed as fast as Perl possible can print them?


If the waiting program is still running, and is still waiting, then the
program it is waiting on must still be running. What other possibilities
are there?

If you really want the "time has not come to a halt" thing, then
something like this:

perl -le '$|=1; fork or do {sleep 4*$_; exit} foreach (1..3); \
$SIG{ALRM}=sub{print "."; alarm 1}; alarm 1; \
do {$x=wait; print "$x";} until $x==-1; \
alarm 0'
.
.
.
20287
.
.
.
.
20288
.
.
.
.
20289
-1

Xho

--
--------------------http://NewsReader.Com/--------------------
The costs of publication of this article were defrayed in part by the
payment of page charges. This article must therefore be hereby marked
advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
this fact.

Hi Xho,
As always - good points. I am looking to load up the system, so a
parallel simulation seems to be the best approach. And frankly, the
period printing was just a way to see that indeed something was
happening, but clearly and wholly unnecessary - although I did get a
chuckle over the time halting comment...

And anyway 'watch ps u' in anotehr session celarly shows me the
processes still running.

But - the question still remains - what's the best way to call other
perl programs? Exec, system or something else?

pg
 
X

xhoster

pgodfrin said:
But - the question still remains - what's the best way to call other
perl programs? Exec, system or something else?

I'd do fork and exec. If you want to do something in the child after
the other thing was done, then I'd do fork and system.

Xho

--
-------------------- http://NewsReader.Com/ --------------------
The costs of publication of this article were defrayed in part by the
payment of page charges. This article must therefore be hereby marked
advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
this fact.
 
S

sheinrich

Hi Xho,
As always - good points. I am looking to load up the system, so a
parallel simulation seems to be the best approach. And frankly, the
period printing was just a way to see that indeed something was
happening, but clearly and wholly unnecessary - although I did get a
chuckle over the time halting comment...

And anyway 'watch ps u' in anotehr session celarly shows me the
processes still running.

But - the question still remains - what's the best way to call other
perl programs? Exec, system or something else?

pg

Hi pg,

a module that I've written couple of months ago was for that very
purpose.
Basically your script will have to instantiate an PreforkAgent, tell
it how many children to beget (degree of parallelization) and provide
it with a couple of callbacks as means of communication.
These callback routines may provide any task, execute it as you like
and perform evaluation of results and logging.
The module's initial purpose was it to generate as much load on a
remote system as the local server would allow. You can however, setup
a small number of children and even delay the task assignments without
interfering with the former tasks.
Children will live and be re-used until the job-queue is depleted.

You might download the module from http://www.atablis.com/temp/PreforkAgent.pm
and give it a try.

Cheers,
Steffen



It is rather easy to integrate
 
P

pgodfrin

Hi pg,

a module that I've written couple of months ago was for that very
purpose.
Basically your script will have to instantiate an PreforkAgent, tell
it how many children to beget (degree of parallelization) and provide
it with a couple of callbacks as means of communication.
These callback routines may provide any task, execute it as you like
and perform evaluation of results and logging.
The module's initial purpose was it to generate as much load on a
remote system as the local server would allow. You can however, setup
a small number of children and even delay the task assignments without
interfering with the former tasks.
Children will live and be re-used until the job-queue is depleted.

You might download the module fromhttp://www.atablis.com/temp/PreforkAgent.pm
and give it a try.

Cheers,
Steffen

It is rather easy to integrate

cool! I'll take a look...
phil
 
J

J. Gleixner

a module that I've written couple of months ago was for that very
purpose.
Basically your script will have to instantiate an PreforkAgent, tell
it how many children to beget (degree of parallelization) and provide
it with a couple of callbacks as means of communication.

Hu.. sounds pretty much like Parallel::ForkManager.
 
S

sheinrich

(e-mail address removed) wrote:

[...]

Hu.. sounds pretty much like Parallel::ForkManager.

That might well be.
PreforkAgent came into being after I'd 'inherited' a customized
loadtesting script to which more and more functionality was to add. So
I was looking for a means of abstraction. Only after that was
finished, did I peak into the Parallel namespace.
Yet I didn't bother to explore how much of my work, which I had
enjoyed, has been gratuitous.
:)

steffen
 
S

sheinrich

(e-mail address removed) wrote:

[...]
a module that I've written couple of months ago was for that very
purpose.
Basically your script will have to instantiate anPreforkAgent, tell
it how many children to beget (degree of parallelization) and provide
it with a couple of callbacks as means of communication.

Hu.. sounds pretty much like Parallel::ForkManager.

Now I got the chance to look at Parallel::ForkManager and get back on
this.

Well, the obvious difference is that PreforkAgent preforks while
Parallel::ForkManager does very much the same as fork(), only limiting
the child number and abstracting waitpid.
And as I wrote above
While the Parallel::ForkManager is creating a new process on every job
(with all the performance overhead that involves), PreforkAgent does
no forking once the work 's been taken up and its children do exit
only after the last job was assigned.

The basic incentive for PreforkAgent.pm was making the most efficient
use of the host's ressources. (E. g., to wreak havoc with real big
application servers for the sake of load simulation.)

steffen
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,743
Messages
2,569,478
Members
44,899
Latest member
RodneyMcAu

Latest Threads

Top