Re: Finding the source of an exception in a python multiprocessingprogram

Discussion in 'Python' started by Neil Cerutti, Apr 24, 2013.

  1. Neil Cerutti

    Neil Cerutti Guest

    On 2013-04-24, William Ray Wing <> wrote:
    > When I look at the pool module, the error is occurring in
    > get(self, timeout=None) on the line after the final else:
    >
    > def get(self, timeout=None):
    > self.wait(timeout)
    > if not self._ready:
    > raise TimeoutError
    > if self._success:
    > return self._value
    > else:
    > raise self._value


    The code that's failing is in self.wait. Somewhere in there you
    must be masking an exception and storing it in self._value
    instead of letting it propogate and crash your program. This is
    hiding the actual context.

    --
    Neil Cerutti
     
    Neil Cerutti, Apr 24, 2013
    #1
    1. Advertising

  2. Neil Cerutti

    Dave Angel Guest

    On 04/24/2013 05:09 PM, William Ray Wing wrote:
    > On Apr 24, 2013, at 4:31 PM, Neil Cerutti <> wrote:
    >
    >> On 2013-04-24, William Ray Wing <> wrote:
    >>> When I look at the pool module, the error is occurring in
    >>> get(self, timeout=None) on the line after the final else:
    >>>
    >>> def get(self, timeout=None):
    >>> self.wait(timeout)
    >>> if not self._ready:
    >>> raise TimeoutError
    >>> if self._success:
    >>> return self._value
    >>> else:
    >>> raise self._value

    >>
    >> The code that's failing is in self.wait. Somewhere in there you
    >> must be masking an exception and storing it in self._value
    >> instead of letting it propogate and crash your program. This is
    >> hiding the actual context.
    >>
    >> --
    >> Neil Cerutti
    >> --
    >> http://mail.python.org/mailman/listinfo/python-list

    >
    > I'm sorry, I'm not following you. The "get" routine (and thus self.wait) is part of the "pool" module in the Python multiprocessing library.
    > None of my code has a class or function named "get".
    >
    > -Bill
    >


    My question is why bother with multithreading? Why not just do these as
    separate processes? You said "they in no way interact with each other"
    and that's a clear clue that separate processes would be cleaner.

    Without knowing anything about those libraries, I'd guess that somewhere
    they do store state in a global attribute or equivalent, and when that
    is accessed by both threads, it can crash.

    Separate processes will find it much more difficult to interact, which
    is a good thing most of the time. Further, they seem to be scheduled
    more efficiently because of the GIL, though that may not make that much
    difference when you're time-limited by network data.

    --
    DaveA
     
    Dave Angel, Apr 25, 2013
    #2
    1. Advertising

  3. On 25 April 2013 00:26, Dave Angel <> wrote:
    > On 04/24/2013 05:09 PM, William Ray Wing wrote:
    >>
    >> On Apr 24, 2013, at 4:31 PM, Neil Cerutti <> wrote:
    >>
    >>> On 2013-04-24, William Ray Wing <> wrote:
    >>>>
    >>>> When I look at the pool module, the error is occurring in
    >>>> get(self, timeout=None) on the line after the final else:
    >>>>
    >>>> def get(self, timeout=None):
    >>>> self.wait(timeout)
    >>>> if not self._ready:
    >>>> raise TimeoutError
    >>>> if self._success:
    >>>> return self._value
    >>>> else:
    >>>> raise self._value
    >>>
    >>>
    >>> The code that's failing is in self.wait. Somewhere in there you
    >>> must be masking an exception and storing it in self._value
    >>> instead of letting it propogate and crash your program. This is
    >>> hiding the actual context.
    >>>
    >>> --
    >>> Neil Cerutti
    >>> --
    >>> http://mail.python.org/mailman/listinfo/python-list

    >>
    >>
    >> I'm sorry, I'm not following you. The "get" routine (and thus self.wait)
    >> is part of the "pool" module in the Python multiprocessing library.
    >> None of my code has a class or function named "get".
    >>
    >> -Bill
    >>

    >
    > My question is why bother with multithreading? Why not just do these as
    > separate processes? You said "they in no way interact with each other" and
    > that's a clear clue that separate processes would be cleaner.


    It's using multiprocessing rather than threads: they are separate processes.

    >
    > Without knowing anything about those libraries, I'd guess that somewhere
    > they do store state in a global attribute or equivalent, and when that is
    > accessed by both threads, it can crash.


    It's state that is passed to it by the subprocess and should only be
    accessed by the top-level process after the subprocess completes (I
    think!).

    >
    > Separate processes will find it much more difficult to interact, which is a
    > good thing most of the time. Further, they seem to be scheduled more
    > efficiently because of the GIL, though that may not make that much
    > difference when you're time-limited by network data.


    They are separate processes and do not share the GIL (unless I'm very
    much mistaken). Also I think the underlying program is limited by the
    call to sleep for 15 seconds.


    Oscar
     
    Oscar Benjamin, Apr 25, 2013
    #3
  4. Neil Cerutti

    Dave Angel Guest

    On 04/24/2013 08:00 PM, Oscar Benjamin wrote:
    > On 25 April 2013 00:26, Dave Angel <> wrote:
    >> On 04/24/2013 05:09 PM, William Ray Wing wrote:
    >>>
    >>>
    >>> <SNIP>
    >>>

    >>
    >> My question is why bother with multithreading? Why not just do these as
    >> separate processes? You said "they in no way interact with each other" and
    >> that's a clear clue that separate processes would be cleaner.

    >
    > It's using multiprocessing rather than threads: they are separate processes.
    >


    You're right; I was completely off base. brain-freeze.

    >>
    >> <SNIP>

    >
    > It's state that is passed to it by the subprocess and should only be
    > accessed by the top-level process after the subprocess completes (I
    > think!).
    >
    >>
    >> Separate processes will find it much more difficult to interact, which is a
    >> good thing most of the time. Further, they seem to be scheduled more
    >> efficiently because of the GIL, though that may not make that much
    >> difference when you're time-limited by network data.

    >
    > They are separate processes and do not share the GIL (unless I'm very
    > much mistaken).


    No, you're not mistaken. Somehow I interpreted the original as saying
    multi-thread, and everything else was wrong as a result. Now it sounds
    like a bug in, or misuse of, the Pool class.



    --
    DaveA
     
    Dave Angel, Apr 25, 2013
    #4
  5. Neil Cerutti

    Neil Cerutti Guest

    On 2013-04-24, William Ray Wing <> wrote:
    > On Apr 24, 2013, at 4:31 PM, Neil Cerutti <> wrote:
    >
    >> On 2013-04-24, William Ray Wing <> wrote:
    >>> When I look at the pool module, the error is occurring in
    >>> get(self, timeout=None) on the line after the final else:
    >>>
    >>> def get(self, timeout=None):
    >>> self.wait(timeout)
    >>> if not self._ready:
    >>> raise TimeoutError
    >>> if self._success:
    >>> return self._value
    >>> else:
    >>> raise self._value

    >>
    >> The code that's failing is in self.wait. Somewhere in there you
    >> must be masking an exception and storing it in self._value
    >> instead of letting it propogate and crash your program. This is
    >> hiding the actual context.

    >
    > I'm sorry, I'm not following you. The "get" routine (and thus
    > self.wait) is part of the "pool" module in the Python
    > multiprocessing library. None of my code has a class or
    > function named "get".


    Oops! I failed to notice it was part of the pool module and not
    your own code.

    --
    Neil Cerutti
     
    Neil Cerutti, Apr 25, 2013
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. news.amnet.net.au

    finding source jsp from web page

    news.amnet.net.au, Oct 1, 2003, in forum: Java
    Replies:
    2
    Views:
    396
    Gerbrand van Dieijen
    Oct 1, 2003
  2. Replies:
    1
    Views:
    339
    Dave Kelly
    Sep 8, 2006
  3. Subra

    Finding the source files in the binary

    Subra, Aug 7, 2007, in forum: C Programming
    Replies:
    17
    Views:
    470
    Kenny McCormack
    Aug 9, 2007
  4. Oscar Benjamin
    Replies:
    0
    Views:
    131
    Oscar Benjamin
    Apr 24, 2013
  5. MRAB
    Replies:
    0
    Views:
    155
Loading...

Share This Page