Re: multiprocessing signal defect

Discussion in 'Python' started by Adam Tauno Williams, Oct 29, 2010.

  1. On Fri, 2010-10-29 at 08:39 -0400, Neal Becker wrote:
    > Adam Tauno Williams wrote:
    >
    > > On Fri, 2010-10-29 at 08:12 -0400, Neal Becker wrote:
    > >> Seems multiprocessing doesn't behave well with signals:
    > >> ---------
    > >> from multiprocessing import Pool
    > >> import time
    > >> def sleep (dummy):
    > >> time.sleep (10)
    > >> if __name__ == '__main__':
    > >> pool = Pool (processes=2)
    > >> result = pool.map (sleep, range (4))
    > >> -------------
    > >> start it up
    > >> $ python test_multip.py
    > >> ----------------------
    > >> ps auxf | grep python
    > >> nbecker 6605 1.6 0.1 338192 6952 pts/1 Sl+ 08:03 0:00 |
    > >> \_ python test_multip.py
    > >> nbecker 6606 0.0 0.1 186368 4760 pts/1 S+ 08:03 0:00 |
    > >> \_ python test_multip.py
    > >> nbecker 6607 0.0 0.1 186372 4740 pts/1 S+ 08:03 0:00 |
    > >> \_ python test_multip.py
    > >> kill 6607
    > >> ps auxf | grep python
    > >> nbecker 6605 0.5 0.1 338192 6952 pts/1 Sl+ 08:03 0:00 |
    > >> \_ python test_multip.py
    > >> nbecker 6606 0.0 0.1 186368 4760 pts/1 S+ 08:03 0:00 |
    > >> \_ python test_multip.py
    > >> nbecker 6607 0.0 0.0 0 0 pts/1 Z+ 08:03 0:00 |
    > >> \_ [python] <defunct>
    > >> kill 6606
    > >> ps auxf | grep python
    > >> nbecker 6605 0.3 0.1 338192 6952 pts/1 Sl+ 08:03 0:00 |
    > >> \_ python test_multip.py
    > >> nbecker 6606 0.0 0.0 0 0 pts/1 Z+ 08:03 0:00 |
    > >> \_ [python] <defunct>
    > >> nbecker 6607 0.0 0.0 0 0 pts/1 Z+ 08:03 0:00 |
    > >> \_ [python] <defunct>
    > >> Now we have 2 dead children and the parent is hung forever.
    > >> Isn't this a serious defect?

    > > No, I think this is just POSIX/UNIX process behavior. If the parent
    > > never joins on the child the child can never exit [which is what a
    > > Zombie process is].
    > > For example, see the do_verify_workers method in

    > <http://coils.hg.sourceforge.net/hgweb/coils/coils/file/6ab5ade3e488/src/coils/logic/workflow/services/executor.py>
    > > A parent process needs to make some effort to reap its children.

    > Yes, and isn't this a defect in mulitprocessing module that the parent
    > process does not reap its children in response to signals like show above?


    No, I don't think so. You're asking the module to over generalize
    behavior. Reaping of the child is important, and that the child needs
    to be reaped may matter to the master child (why? did something go
    wrong?). Silently reaping them [which would reduce the size of the
    Pool? Or would it dynamically create a new worker?] might have
    unintended side effects. Maybe since Pool specifically generalizes
    child management you could make an argument it should reap, but I'm not
    sure. Personally I'd recommend that your worker processes include a
    signal handler to do something smart in the case of a "-15" [for which
    there isn't really a thread equivalent - can you sent a SystemV style
    signal to an individual thread in a process? I don't think so.]

    How would a 'traditional' thread pool behave if a thread abended? [of
    course, that depends on the thread-pool implementation] The correct
    behavior in case of an exception in a thread is a topic of some debate.
     
    Adam Tauno Williams, Oct 29, 2010
    #1
    1. Advertising

  2. On Oct 29, 10:08 am, Adam Tauno Williams <>
    wrote:
    > signal handler to do something smart in the case of a "-15" [for which
    > there isn't really a thread equivalent - can you sent a SystemV style
    > signal to an individual thread in a process?  I don't think so.]


    Yes. pthread_kill(P)

    Jean-Paul
     
    Jean-Paul Calderone, Oct 29, 2010
    #2
    1. Advertising

  3. On Fri, 2010-10-29 at 07:31 -0700, Jean-Paul Calderone wrote:
    > On Oct 29, 10:08 am, Adam Tauno Williams <>
    > wrote:
    > > signal handler to do something smart in the case of a "-15" [for which
    > > there isn't really a thread equivalent - can you sent a SystemV style
    > > signal to an individual thread in a process? I don't think so.]

    > Yes. pthread_kill(P)


    Eh, only sort of.

    <quote>
    Signal dispositions are process-wide: if a signal handler is installed,
    the handler will be invoked in the thread thread, but if the disposition
    of the signal is "stop", "continue", or "terminate", this action will
    affect the whole process.
    </quote>

    Also phthread_kill can only be called for a thread in the same process.
    Which is quite different that SystemV signals.

    So there can't really be an exact 1:1 match of how thread-signaling vs.
    multiprocessing-process-signaling works; which was my point.
     
    Adam Tauno Williams, Oct 29, 2010
    #3
  4. Adam Tauno Williams

    Adam Skutt Guest

    On Oct 29, 10:08 am, Adam Tauno Williams <>
    wrote:
    > No, I don't think so.  You're asking the module to over generalize
    > behavior.  Reaping of the child is important, and that the child needs
    > to be reaped may matter to the master child (why? did something go
    > wrong?).

    And such information is made available when you reap the child
    process. There are a ton of ways for the module to provide sane and
    proper default behavior and still provide hooks for people who need to
    modify the behavior or retrieve information. However, I can't imagine
    a use case where the end-developer cares about the process' exit
    status from a multiprocess.Pool.

    > Silently reaping them [which would reduce the size of the
    > Pool? Or would it dynamically create a new worker?] might have
    > unintended side effects.

    Such as? Personally, even if you do enumerate one, I'm not sure how
    much it matters, as an unintended side effect has already occurred.
    It's too late to avoid side effects, they've already happened.
    Moreover, merely blithely continuing on invites all sorts of fun side
    effects, like the lovely "undefined behavior" side effect.

    > How would a 'traditional' thread pool behave if a thread abended?  [of
    > course, that depends on the thread-pool implementation]  The correct
    > behavior in case of an exception in a thread is a topic of some debate.

    Yes, but we're not talking about an exception in a thread, we're
    talking about a thread being terminated by the OS. Which should be
    treated like every other OS termination: as a fatal error. There's
    not way for the multiprocessing module to fully recover from such an
    event to a known state, so the only sensible thing left is
    termination. Anything else runs the risk of undefined behavior.

    Adam
     
    Adam Skutt, Oct 30, 2010
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. =?Utf-8?B?U2hhdw==?=

    Good defect tracking system for software development

    =?Utf-8?B?U2hhdw==?=, Oct 7, 2004, in forum: ASP .Net
    Replies:
    2
    Views:
    451
    ESPN Lover
    Oct 7, 2004
  2. Azumanga
    Replies:
    3
    Views:
    284
    Azumanga
    Nov 10, 2006
  3. user923005

    Visual C++ : Defect report or not?

    user923005, Nov 1, 2007, in forum: C++
    Replies:
    6
    Views:
    495
    Dann Corbit
    Nov 1, 2007
  4. Keld Jørn Simonsen

    Defect report log v1.25 available

    Keld Jørn Simonsen, Sep 15, 2008, in forum: C Programming
    Replies:
    0
    Views:
    327
    Keld Jørn Simonsen
    Sep 15, 2008
  5. Neal Becker

    multiprocessing signal defect

    Neal Becker, Oct 29, 2010, in forum: Python
    Replies:
    0
    Views:
    302
    Neal Becker
    Oct 29, 2010
Loading...

Share This Page