What's happening with the last child process here????

Discussion in 'Perl Misc' started by trxrse, Jan 10, 2007.

  1. trxrse

    trxrse Guest

    Hi, everyone

    can you tell me please what's happening here?

    This piece of code is supposed to launch 3 children processes and to
    keep track of each one when they finish.
    THE PROBLEM: SOMETIMES (50% OF THE TIME) THE LAST PROCESS NEVER RETURNS
    THE USR1 MESSAGE TO THE PARENT.
    For example:

    ../fork3
    Started Wed Jan 10 09:23:36 2007
    Forked by Wed Jan 10 09:23:36 2007
    I am finished... Process 1
    Look:3 $nfound:0
    Look:2 $nfound:0
    Look:1 $nfound:1
    Wed Jan 10 09:23:37 2007 - Completed 1
    I am finished... Process 2
    Look:3 $nfound:0
    Look:2 $nfound:0
    Look:1 $nfound:0
    I am finished... Process 3
    Look:3 $nfound:0
    Look:2 $nfound:2
    Wed Jan 10 09:23:39 2007 - Completed 2
    Look:1 $nfound:0
    >>>>>>>>>>>>>>>>>>>>>>>> And here it gets stuck forever!!!!!!!!!!!!!!!!!



    And now the code:


    #! /bin/perl
    # Forking out a series of processes which may take
    # a while to run. System waits for any signal to be
    # received back so that it can get the faster ones
    # finalised quickly! Once a signal is received back
    # it finds the appropriate pipe and reads from it

    @waitlist = (3,2,1);
    $SIG{USR1} = "doneit";
    $start = localtime();
    print "Started $start\n";
    $parent = $$;

    # Use a "typeglob" to store, in effect, a list of
    # file handles.

    foreach $item (@waitlist) {
    pipe *{$item},FH;
    unless ($pid = fork()) {
    # Child process
    sleep $item;
    print FH "Completed $item\n";
    kill "USR1",$parent;
    print "I am finished... Process ",$item,"\n";
    exit();
    }
    # Parent process - loop to start others
    $kids++;
    }
    $mid = localtime();
    print "Forked by $mid\n";

    # Parent - wait for returned data
    # To add - uses select to check channels!

    while ($kids >0) {
    while (! $gotone) {
    sleep 1;
    }
    $gotone = 0;
    foreach $item (@waitlist) {
    $rin = $win = "";
    vec($rin, fileno(*{$item}), 1) = 1;
    $ein = $rin ;
    $nfound=select($rin,$win,$ein,0);
    print "Look:",$item," \$nfound:",$nfound,"\n";
    if ($nfound) {
    sysread($item,$response,40);
    $now = localtime();
    print "$now - $response";
    close $item;
    $kids--;
    }
    }
    }

    $end = localtime();
    print "Completed $end\n";

    sub doneit {
    $gotone = 1;
    }
    trxrse, Jan 10, 2007
    #1
    1. Advertising

  2. trxrse

    -berlin.de Guest

    trxrse <> wrote in comp.lang.perl.misc:
    > Hi, everyone
    >
    > can you tell me please what's happening here?
    >
    > This piece of code is supposed to launch 3 children processes and to
    > keep track of each one when they finish.
    > THE PROBLEM: SOMETIMES (50% OF THE TIME) THE LAST PROCESS NEVER RETURNS
    > THE USR1 MESSAGE TO THE PARENT.


    No need to shout.

    [output sample snipped]

    > And now the code:
    >
    > #! /bin/perl


    You are missing "use strict" and "use warnings" here. That makes
    your code hard to debug.

    > # Forking out a series of processes which may take
    > # a while to run. System waits for any signal to be
    > # received back so that it can get the faster ones
    > # finalised quickly! Once a signal is received back
    > # it finds the appropriate pipe and reads from it
    >
    > @waitlist = (3,2,1);
    > $SIG{USR1} = "doneit";
    > $start = localtime();
    > print "Started $start\n";
    > $parent = $$;
    >
    > # Use a "typeglob" to store, in effect, a list of
    > # file handles.
    >
    > foreach $item (@waitlist) {
    > pipe *{$item},FH;


    The last statement only works without strictures. You are creating
    filehandles with non-standard names "3", "2" and "1". Is that what
    you intend? I haven't followed your code any further.

    Please make your code strict-compliant. If the use of numeric
    filehandles is somehow essential to the program revoke strict 'refs'
    locally, but I think you should be able to work with standard
    filehandles.

    Repost your code with strict and warnings in place. You'll have
    better chances of someone finding the error or the explanation.

    Anno
    --
    # make a japh from an arbitrary anagram of the famous signoff
    # less PC version:
    # $anagram = 'Trojan Slut? Heck, rape her!'; # as Achilles said to Odysseus
    -berlin.de, Jan 10, 2007
    #2
    1. Advertising

  3. trxrse

    -berlin.de Guest

    trxrse <> wrote in comp.lang.perl.misc:
    > Hi, everyone
    >
    > can you tell me please what's happening here?
    >
    > This piece of code is supposed to launch 3 children processes and to
    > keep track of each one when they finish.
    > THE PROBLEM: SOMETIMES (50% OF THE TIME) THE LAST PROCESS NEVER RETURNS
    > THE USR1 MESSAGE TO THE PARENT.


    No need to shout.

    [output sample snipped]

    > And now the code:
    >
    > #! /bin/perl


    You are missing "use strict" and "use warnings" here. That makes
    your code hard to debug.

    > # Forking out a series of processes which may take
    > # a while to run. System waits for any signal to be
    > # received back so that it can get the faster ones
    > # finalised quickly! Once a signal is received back
    > # it finds the appropriate pipe and reads from it
    >
    > @waitlist = (3,2,1);
    > $SIG{USR1} = "doneit";
    > $start = localtime();
    > print "Started $start\n";
    > $parent = $$;
    >
    > # Use a "typeglob" to store, in effect, a list of
    > # file handles.
    >
    > foreach $item (@waitlist) {
    > pipe *{$item},FH;


    The last statement only works without strictures. You are creating
    filehandles with non-standard names "3", "2" and "1". Is that what
    you intend? I haven't followed your code any further.

    Please make your code strict-compliant. If the use of numeric
    filehandles is somehow essential to the program revoke strict 'refs'
    locally, but I think you should be able to work with standard
    filehandles.

    Repost your code with strict and warnings in place. You'll have
    better chances of someone finding the error or the explanation.

    Anno
    --
    $anagram = 'Knuth heals rare project'; # by Abigail
    push @{ $pos{ $_}}, $pos ++ for split //, lc $anagram;
    print "print +(split //, '$anagram')[ $_]\n" for
    join ', ', map shift @$_, @pos{ split //, lc "Just another Perl hacker"};
    -berlin.de, Jan 10, 2007
    #3
  4. trxrse

    Guest

    "trxrse" <> wrote:
    > Hi, everyone
    >
    > can you tell me please what's happening here?
    >
    > This piece of code is supposed to launch 3 children processes and to
    > keep track of each one when they finish.
    > THE PROBLEM: SOMETIMES (50% OF THE TIME) THE LAST PROCESS NEVER RETURNS
    > THE USR1 MESSAGE TO THE PARENT.


    USR1 isn't a message.


    > $SIG{USR1} = "doneit";


    Do you need to reinstall signals after they've been fired on your system?


    > foreach $item (@waitlist) {
    > pipe *{$item},FH;
    > unless ($pid = fork()) {
    > # Child process
    > sleep $item;
    > print FH "Completed $item\n";
    > kill "USR1",$parent;
    > print "I am finished... Process ",$item,"\n";
    > exit();
    > }


    Why mix signals and select? When the child is done writing on it's
    handle, then it is done. No need for signals at all.

    Xho

    --
    -------------------- http://NewsReader.Com/ --------------------
    Usenet Newsgroup Service $9.95/Month 30GB
    , Jan 10, 2007
    #4
  5. trxrse

    Uri Guttman Guest

    >>>>> "t" == trxrse <> writes:

    t> can you tell me please what's happening here?

    not really. it is one of the most convoluted and bizarre forking
    examples i have seen. why are you doing it the hard way?

    t> THE PROBLEM: SOMETIMES (50% OF THE TIME) THE LAST PROCESS NEVER RETURNS
    t> THE USR1 MESSAGE TO THE PARENT.

    signals can be merged into one actual call to your signal handler. this
    is an OS dependent thing and perl can't fix it. when you get a signal,
    you have to make sure it was only one or possible more than one that
    were merged. the way to do this with subprocesses is to call
    wait/waitpid in nonblcoking mode and loop until no more processes are
    reaped. use the sigchild signal which is better as it will be delivered
    after the child exits unlike yours which could be delivered before it
    really exited.



    t> @waitlist = (3,2,1);

    as anno said, this is a very bad way to track the processes. save the
    pids and use those to track what is alive and dead.

    t> $SIG{USR1} = "doneit";

    you can use a code ref there since your callback is so small.

    $SIG{USR1} = sub { $doneit = 1 } ;

    but as i said use sigchild.

    and use strict and warnings also as anno said.

    t> # Use a "typeglob" to store, in effect, a list of
    t> # file handles.

    that is not a good way to do this. use lexical file handles and make one
    for each call to pipe. use Symbol::gensym or IPC::Open2 but not
    typeglobs which are symbolic refs.

    t> foreach $item (@waitlist) {
    t> pipe *{$item},FH;

    the FH will be the overwritten in the parent for all the children which
    is not smart. one issue with pipes to children is detecting closed
    pipes with select as you seem to be doing below. you have to close the
    other side in the parent or child since those will be dupped and stay
    open.

    t> unless ($pid = fork()) {
    t> # Child process
    t> sleep $item;

    reusing the item number for sleep time is very nutso. i can't say
    anything more as i am just shocked by that code.

    t> print FH "Completed $item\n";
    t> kill "USR1",$parent;

    that signal could be delivered before this process exits so you won't be
    able to immediately detect it. but my guess is that the signals were
    merged and you don't use wait/waitpid to really check for childred to
    reap. actually you aren't even reaping the children (only wait/waitpid
    does that) so you create zombie children which will get reaped by the
    init process when your parent exits. as i said, this is all done totally
    the wrong way.

    t> $mid = localtime();
    t> print "Forked by $mid\n";

    forking will happen much faster than the granulaity of localtime (which
    is normally 1 second).

    t> # Parent - wait for returned data
    t> # To add - uses select to check channels!

    t> while ($kids >0) {
    t> while (! $gotone) {
    t> sleep 1;
    t> }
    t> $gotone = 0;
    t> foreach $item (@waitlist) {
    t> $rin = $win = "";
    t> vec($rin, fileno(*{$item}), 1) = 1;
    t> $ein = $rin ;
    t> $nfound=select($rin,$win,$ein,0);

    coding select directly is not for the faint of heart. use IO::Select or
    event.pm. event.pm can also handle signals which makes it very good for
    process management.

    enough for now. my brane hertz.

    uri

    --
    Uri Guttman ------ -------- http://www.stemsystems.com
    --Perl Consulting, Stem Development, Systems Architecture, Design and Coding-
    Search or Offer Perl Jobs ---------------------------- http://jobs.perl.org
    Uri Guttman, Jan 10, 2007
    #5
  6. trxrse

    trxrse Guest

    Thanks very much for your replies. I appologize, I am not a
    professional perl developer... even if I'd like to... :( - probably I
    need some years for this. Anyway, I appreciate your answers.

    To be honest, I am trying to write a very simple "scheduler" that gets
    an amount of work as input (eg process a file having 1 million lines),
    splits this work in chunks and launches a limited number of parallel
    processes, each one doing some small chunk of work. When a child
    process is finished OK, the "scheduler" has to launch another child
    that does the next chunk. Finally, I want to embed some elegant
    behavior in the master/parent - aka re-launch the chunk of work for the
    subprocess that collapsed, etc...

    Can you give me a hint, please? Of course, I am not asking you people
    to solve my problem, but I would like to find your professional views.

    Thanks again
    R


    PS I will rewrite the code as Anno said earlier.
    trxrse, Jan 10, 2007
    #6
  7. trxrse

    Uri Guttman Guest

    >>>>> "t" == trxrse <> writes:

    t> To be honest, I am trying to write a very simple "scheduler" that
    t> gets an amount of work as input (eg process a file having 1 million
    t> lines), splits this work in chunks and launches a limited number of
    t> parallel processes, each one doing some small chunk of work. When a
    t> child process is finished OK, the "scheduler" has to launch another
    t> child that does the next chunk. Finally, I want to embed some
    t> elegant behavior in the master/parent - aka re-launch the chunk of
    t> work for the subprocess that collapsed, etc...

    this is not a trivial project. your best bet is to hire someone to do it
    for you or use some framework to do much of the heavy lifting. but since
    you are not that experienced, even a framework will be tricky to learn
    and use.

    t> Can you give me a hint, please? Of course, I am not asking you people
    t> to solve my problem, but I would like to find your professional views.

    there are too many issues in such a project to just learn from hints. if
    you are up for learning a framework, you can check out stem (which i
    created) on cpan which does subprocess management and IPC for you. but
    as i said, even though it will do much of the work, you will need to
    learn a different paradigm and also better coding skills. my suggestion
    again would be to hire someone (possibly me :) to either do this or help
    you develop it.

    uri

    --
    Uri Guttman ------ -------- http://www.stemsystems.com
    --Perl Consulting, Stem Development, Systems Architecture, Design and Coding-
    Search or Offer Perl Jobs ---------------------------- http://jobs.perl.org
    Uri Guttman, Jan 10, 2007
    #7
  8. trxrse

    trxrse Guest

    I find amazing that there is no open source project on this. I will try
    to do it myself - I have no other chance.

    Thanks
    Remus


    Uri Guttman wrote:
    > >>>>> "t" == trxrse <> writes:

    >
    > t> To be honest, I am trying to write a very simple "scheduler" that
    > t> gets an amount of work as input (eg process a file having 1 million
    > t> lines), splits this work in chunks and launches a limited number of
    > t> parallel processes, each one doing some small chunk of work. When a
    > t> child process is finished OK, the "scheduler" has to launch another
    > t> child that does the next chunk. Finally, I want to embed some
    > t> elegant behavior in the master/parent - aka re-launch the chunk of
    > t> work for the subprocess that collapsed, etc...
    >
    > this is not a trivial project. your best bet is to hire someone to do it
    > for you or use some framework to do much of the heavy lifting. but since
    > you are not that experienced, even a framework will be tricky to learn
    > and use.
    >
    > t> Can you give me a hint, please? Of course, I am not asking you people
    > t> to solve my problem, but I would like to find your professional views.
    >
    > there are too many issues in such a project to just learn from hints. if
    > you are up for learning a framework, you can check out stem (which i
    > created) on cpan which does subprocess management and IPC for you. but
    > as i said, even though it will do much of the work, you will need to
    > learn a different paradigm and also better coding skills. my suggestion
    > again would be to hire someone (possibly me :) to either do this or help
    > you develop it.
    >
    > uri
    >
    > --
    > Uri Guttman ------ -------- http://www.stemsystems.com
    > --Perl Consulting, Stem Development, Systems Architecture, Design and Coding-
    > Search or Offer Perl Jobs ---------------------------- http://jobs.perl.org
    trxrse, Jan 10, 2007
    #8
  9. trxrse

    Uri Guttman Guest

    >>>>> "t" == trxrse <> writes:

    t> I find amazing that there is no open source project on this. I will try
    t> to do it myself - I have no other chance.

    don't top post! read the list guidelines.

    what do you mean project? you haven't specified this at all. and stem
    (which is open source on cpan) can do most of the work as i said. you
    still need to code at a higher level than what you have shown to do this
    in any 'open source project' you could find. they don't do your real
    work for you and you have to use their apis and such. also as i said
    they all involve learning curves so you are stuck with that no matter
    what you do. process farms are not difficult but they are not beginner
    projects.

    why do developers paint themselves into corners like this? someone is
    paying for this in either time and/or money and they should know the
    work and skill level required. it is not my fault that i have to tell
    you the bad news that you will need help with this.

    uri

    --
    Uri Guttman ------ -------- http://www.stemsystems.com
    --Perl Consulting, Stem Development, Systems Architecture, Design and Coding-
    Search or Offer Perl Jobs ---------------------------- http://jobs.perl.org
    Uri Guttman, Jan 10, 2007
    #9
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Replies:
    3
    Views:
    407
    Howard
    Feb 22, 2005
  2. dan miller (moderator, s.p.d)

    what is happening here?

    dan miller (moderator, s.p.d), Apr 6, 2004, in forum: Python
    Replies:
    3
    Views:
    354
  3. whats happening here

    , Mar 25, 2008, in forum: C Programming
    Replies:
    4
    Views:
    2,572
    Eric Sosman
    Mar 25, 2008
  4. vikramtheone

    Basics of VHDL. Whats happening here?

    vikramtheone, Jun 1, 2009, in forum: VHDL
    Replies:
    1
    Views:
    495
    jeppe
    Jun 1, 2009
  5. Z W
    Replies:
    0
    Views:
    171
Loading...

Share This Page