What's happening with the last child process here????

[trxrse]

Hi, everyone

can you tell me please what's happening here?

This piece of code is supposed to launch 3 children processes and to
keep track of each one when they finish.
THE PROBLEM: SOMETIMES (50% OF THE TIME) THE LAST PROCESS NEVER RETURNS
THE USR1 MESSAGE TO THE PARENT.
For example:

../fork3
Started Wed Jan 10 09:23:36 2007
Forked by Wed Jan 10 09:23:36 2007
I am finished... Process 1
Look:3 $nfound:0
Look:2 $nfound:0
Look:1 $nfound:1
Wed Jan 10 09:23:37 2007 - Completed 1
I am finished... Process 2
Look:3 $nfound:0
Look:2 $nfound:0
Look:1 $nfound:0
I am finished... Process 3
Look:3 $nfound:0
Look:2 $nfound:2
Wed Jan 10 09:23:39 2007 - Completed 2
Look:1 $nfound:0

And now the code:


#! /bin/perl
# Forking out a series of processes which may take
# a while to run. System waits for any signal to be
# received back so that it can get the faster ones
# finalised quickly! Once a signal is received back
# it finds the appropriate pipe and reads from it

@waitlist = (3,2,1);
$SIG{USR1} = "doneit";
$start = localtime();
print "Started $start\n";
$parent = $$;

# Use a "typeglob" to store, in effect, a list of
# file handles.

foreach $item (@waitlist) {
pipe *{$item},FH;
unless ($pid = fork()) {
# Child process
sleep $item;
print FH "Completed $item\n";
kill "USR1",$parent;
print "I am finished... Process ",$item,"\n";
exit();
}
# Parent process - loop to start others
$kids++;
}
$mid = localtime();
print "Forked by $mid\n";

# Parent - wait for returned data
# To add - uses select to check channels!

while ($kids >0) {
while (! $gotone) {
sleep 1;
}
$gotone = 0;
foreach $item (@waitlist) {
$rin = $win = "";
vec($rin, fileno(*{$item}), 1) = 1;
$ein = $rin ;
$nfound=select($rin,$win,$ein,0);
print "Look:",$item," \$nfound:",$nfound,"\n";
if ($nfound) {
sysread($item,$response,40);
$now = localtime();
print "$now - $response";
close $item;
$kids--;
}
}
}

$end = localtime();
print "Completed $end\n";

sub doneit {
$gotone = 1;
}

 

[anno4000]

Hi, everyone

can you tell me please what's happening here?

This piece of code is supposed to launch 3 children processes and to
keep track of each one when they finish.
THE PROBLEM: SOMETIMES (50% OF THE TIME) THE LAST PROCESS NEVER RETURNS
THE USR1 MESSAGE TO THE PARENT.

No need to shout.

[output sample snipped]

And now the code:

#! /bin/perl

You are missing "use strict" and "use warnings" here. That makes
your code hard to debug.

# Forking out a series of processes which may take
# a while to run. System waits for any signal to be
# received back so that it can get the faster ones
# finalised quickly! Once a signal is received back
# it finds the appropriate pipe and reads from it

@waitlist = (3,2,1);
$SIG{USR1} = "doneit";
$start = localtime();
print "Started $start\n";
$parent = $$;

# Use a "typeglob" to store, in effect, a list of
# file handles.

foreach $item (@waitlist) {
pipe *{$item},FH;

The last statement only works without strictures. You are creating
filehandles with non-standard names "3", "2" and "1". Is that what
you intend? I haven't followed your code any further.

Please make your code strict-compliant. If the use of numeric
filehandles is somehow essential to the program revoke strict 'refs'
locally, but I think you should be able to work with standard
filehandles.

Repost your code with strict and warnings in place. You'll have
better chances of someone finding the error or the explanation.

Anno

 

[anno4000]

Hi, everyone

can you tell me please what's happening here?

This piece of code is supposed to launch 3 children processes and to
keep track of each one when they finish.
THE PROBLEM: SOMETIMES (50% OF THE TIME) THE LAST PROCESS NEVER RETURNS
THE USR1 MESSAGE TO THE PARENT.

No need to shout.

[output sample snipped]

And now the code:

#! /bin/perl

You are missing "use strict" and "use warnings" here. That makes
your code hard to debug.

# Forking out a series of processes which may take
# a while to run. System waits for any signal to be
# received back so that it can get the faster ones
# finalised quickly! Once a signal is received back
# it finds the appropriate pipe and reads from it

@waitlist = (3,2,1);
$SIG{USR1} = "doneit";
$start = localtime();
print "Started $start\n";
$parent = $$;

# Use a "typeglob" to store, in effect, a list of
# file handles.

foreach $item (@waitlist) {
pipe *{$item},FH;

The last statement only works without strictures. You are creating
filehandles with non-standard names "3", "2" and "1". Is that what
you intend? I haven't followed your code any further.

Please make your code strict-compliant. If the use of numeric
filehandles is somehow essential to the program revoke strict 'refs'
locally, but I think you should be able to work with standard
filehandles.

Repost your code with strict and warnings in place. You'll have
better chances of someone finding the error or the explanation.

Anno

 

[xhoster]

Hi, everyone

can you tell me please what's happening here?

This piece of code is supposed to launch 3 children processes and to
keep track of each one when they finish.
THE PROBLEM: SOMETIMES (50% OF THE TIME) THE LAST PROCESS NEVER RETURNS
THE USR1 MESSAGE TO THE PARENT.

USR1 isn't a message.

$SIG{USR1} = "doneit";

Do you need to reinstall signals after they've been fired on your system?

foreach $item (@waitlist) {
pipe *{$item},FH;
unless ($pid = fork()) {
# Child process
sleep $item;
print FH "Completed $item\n";
kill "USR1",$parent;
print "I am finished... Process ",$item,"\n";
exit();
}

Why mix signals and select? When the child is done writing on it's
handle, then it is done. No need for signals at all.

Xho

 

[Uri Guttman]

t> can you tell me please what's happening here?

not really. it is one of the most convoluted and bizarre forking
examples i have seen. why are you doing it the hard way?

t> THE PROBLEM: SOMETIMES (50% OF THE TIME) THE LAST PROCESS NEVER RETURNS
t> THE USR1 MESSAGE TO THE PARENT.

signals can be merged into one actual call to your signal handler. this
is an OS dependent thing and perl can't fix it. when you get a signal,
you have to make sure it was only one or possible more than one that
were merged. the way to do this with subprocesses is to call
wait/waitpid in nonblcoking mode and loop until no more processes are
reaped. use the sigchild signal which is better as it will be delivered
after the child exits unlike yours which could be delivered before it
really exited.



t> @waitlist = (3,2,1);

as anno said, this is a very bad way to track the processes. save the
pids and use those to track what is alive and dead.

t> $SIG{USR1} = "doneit";

you can use a code ref there since your callback is so small.

$SIG{USR1} = sub { $doneit = 1 } ;

but as i said use sigchild.

and use strict and warnings also as anno said.

t> # Use a "typeglob" to store, in effect, a list of
t> # file handles.

that is not a good way to do this. use lexical file handles and make one
for each call to pipe. use Symbol::gensym or IPC::Open2 but not
typeglobs which are symbolic refs.

t> foreach $item (@waitlist) {
t> pipe *{$item},FH;

the FH will be the overwritten in the parent for all the children which
is not smart. one issue with pipes to children is detecting closed
pipes with select as you seem to be doing below. you have to close the
other side in the parent or child since those will be dupped and stay
open.

t> unless ($pid = fork()) {
t> # Child process
t> sleep $item;

reusing the item number for sleep time is very nutso. i can't say
anything more as i am just shocked by that code.

t> print FH "Completed $item\n";
t> kill "USR1",$parent;

that signal could be delivered before this process exits so you won't be
able to immediately detect it. but my guess is that the signals were
merged and you don't use wait/waitpid to really check for childred to
reap. actually you aren't even reaping the children (only wait/waitpid
does that) so you create zombie children which will get reaped by the
init process when your parent exits. as i said, this is all done totally
the wrong way.

t> $mid = localtime();
t> print "Forked by $mid\n";

forking will happen much faster than the granulaity of localtime (which
is normally 1 second).

t> # Parent - wait for returned data
t> # To add - uses select to check channels!

t> while ($kids >0) {
t> while (! $gotone) {
t> sleep 1;
t> }
t> $gotone = 0;
t> foreach $item (@waitlist) {
t> $rin = $win = "";
t> vec($rin, fileno(*{$item}), 1) = 1;
t> $ein = $rin ;
t> $nfound=select($rin,$win,$ein,0);

coding select directly is not for the faint of heart. use IO::Select or
event.pm. event.pm can also handle signals which makes it very good for
process management.

enough for now. my brane hertz.

uri

 

[trxrse]

Thanks very much for your replies. I appologize, I am not a
professional perl developer... even if I'd like to... - probably I
need some years for this. Anyway, I appreciate your answers.

To be honest, I am trying to write a very simple "scheduler" that gets
an amount of work as input (eg process a file having 1 million lines),
splits this work in chunks and launches a limited number of parallel
processes, each one doing some small chunk of work. When a child
process is finished OK, the "scheduler" has to launch another child
that does the next chunk. Finally, I want to embed some elegant
behavior in the master/parent - aka re-launch the chunk of work for the
subprocess that collapsed, etc...

Can you give me a hint, please? Of course, I am not asking you people
to solve my problem, but I would like to find your professional views.

Thanks again
R


PS I will rewrite the code as Anno said earlier.

 

[Uri Guttman]

t> To be honest, I am trying to write a very simple "scheduler" that
t> gets an amount of work as input (eg process a file having 1 million
t> lines), splits this work in chunks and launches a limited number of
t> parallel processes, each one doing some small chunk of work. When a
t> child process is finished OK, the "scheduler" has to launch another
t> child that does the next chunk. Finally, I want to embed some
t> elegant behavior in the master/parent - aka re-launch the chunk of
t> work for the subprocess that collapsed, etc...

this is not a trivial project. your best bet is to hire someone to do it
for you or use some framework to do much of the heavy lifting. but since
you are not that experienced, even a framework will be tricky to learn
and use.

t> Can you give me a hint, please? Of course, I am not asking you people
t> to solve my problem, but I would like to find your professional views.

there are too many issues in such a project to just learn from hints. if
you are up for learning a framework, you can check out stem (which i
created) on cpan which does subprocess management and IPC for you. but
as i said, even though it will do much of the work, you will need to
learn a different paradigm and also better coding skills. my suggestion
again would be to hire someone (possibly me to either do this or help
you develop it.

uri

 

[trxrse]

I find amazing that there is no open source project on this. I will try
to do it myself - I have no other chance.

Thanks
Remus

 

[Uri Guttman]

t> I find amazing that there is no open source project on this. I will try
t> to do it myself - I have no other chance.

don't top post! read the list guidelines.

what do you mean project? you haven't specified this at all. and stem
(which is open source on cpan) can do most of the work as i said. you
still need to code at a higher level than what you have shown to do this
in any 'open source project' you could find. they don't do your real
work for you and you have to use their apis and such. also as i said
they all involve learning curves so you are stuck with that no matter
what you do. process farms are not difficult but they are not beginner
projects.

why do developers paint themselves into corners like this? someone is
paying for this in either time and/or money and they should know the
work and skill level required. it is not my fault that i have to tell
you the bad news that you will need help with this.

uri