race/deadlock when creating a multiprocessing.manager instance whileimporting a module ?

Discussion in 'Python' started by Sebastien Binet, Dec 14, 2009.

  1. hi there,

    say I have this module

    ## farnsworth ##
    __all__ = [
    'mgr',
    'maths',
    ]

    from multiprocessing.managers import BaseManager

    class MathsClass(object):
    def add(self, x, y):
    return x + y
    def mul(self, x, y):
    return x * y

    class MyManager(BaseManager):
    pass

    MyManager.register('Maths', MathsClass)

    def _setup():
    print "creating a manager..."
    mgr = MyManager()
    print "starting the manager..."
    mgr.start()
    print "sciencing faster..."
    maths = mgr.Maths()
    print maths.add(4,3)
    print maths.mul(7,8)
    print "done with sciencing."
    return (mgr, maths)

    # exec at module import
    mgr, maths = _setup()

    # prevent hysteresis + clean-up
    del _setup
    ## EOF ##

    if I use it like so:

    $ python -m farnsworth
    creating a manager...
    starting the manager...
    sciencing faster...
    7
    56
    done with sciencing.

    all is fine, but if I try to use it thru an import:
    $ python
    py> import farnsworth.mgr as mgr
    creating a manager...
    starting the manager...
    sciencing faster...

    [stuck for some time... hitting ^C]
    Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "farnsworth.py", line 32, in <module>
    mgr, maths = _setup()
    File "farnsworth.py", line 25, in _setup
    maths = mgr.Maths()
    File "/usr/lib/python2.6/multiprocessing/managers.py", line 634, in
    temp
    token, exp = self._create(typeid, *args, **kwds)
    File "/usr/lib/python2.6/multiprocessing/managers.py", line 532, in
    _create
    conn = self._Client(self._address, authkey=self._authkey)
    File "/usr/lib/python2.6/multiprocessing/connection.py", line 140,
    in Client
    answer_challenge(c, authkey)
    File "/usr/lib/python2.6/multiprocessing/connection.py", line 372,
    in answer_challenge
    message = connection.recv_bytes(256) # reject large
    message
    KeyboardInterrupt

    is this a known limitation/feature of the multiprocessing module ?
    is there a workaround (acquiring some import lock maybe) ?

    cheers,
    sebastien.

    PS:
    $ python
    Python 2.6.4 (r264:75706, Oct 27 2009, 06:25:13)
    [GCC 4.4.1] on linux2
    Type "help", "copyright", "credits" or "license" for more information.

    $ python -c 'import sys; print sys.version_info'
    (2, 6, 4, 'final', 0)

    $ uname -a
    Linux farnsworth 2.6.31-ARCH #1 SMP PREEMPT Tue Nov 10 19:01:40 CET
    2009 x86_64 Intel(R) Core(TM)2 Duo CPU T9400 @ 2.53GHz GenuineIntel
    GNU/Linux
     
    Sebastien Binet, Dec 14, 2009
    #1
    1. Advertising

  2. Sebastien Binet

    Dave Angel Guest

    Re: race/deadlock when creating a multiprocessing.manager instancewhile importing a module ?

    Sebastien Binet wrote:
    > hi there,
    >
    > say I have this module
    >
    > ## farnsworth ##
    > __all__ = [
    > 'mgr',
    > 'maths',
    > ]
    >
    > from multiprocessing.managers import BaseManager
    >
    > class MathsClass(object):
    > def add(self, x, y):
    > return x + y
    > def mul(self, x, y):
    > return x * y
    >
    > class MyManager(BaseManager):
    > pass
    >
    > MyManager.register('Maths', MathsClass)
    >
    > def _setup():
    > print "creating a manager..."
    > mgr = MyManager()
    > print "starting the manager..."
    > mgr.start()
    > print "sciencing faster..."
    > maths = mgr.Maths()
    > print maths.add(4,3)
    > print maths.mul(7,8)
    > print "done with sciencing."
    > return (mgr, maths)
    >
    > # exec at module import
    > mgr, maths = _setup()
    >
    > # prevent hysteresis + clean-up
    > del _setup
    > ## EOF ##
    >
    > if I use it like so:
    >
    > $ python -m farnsworth
    > creating a manager...
    > starting the manager...
    > sciencing faster...
    > 7
    > 56
    > done with sciencing.
    >
    > all is fine, but if I try to use it thru an import:
    > $ python
    > py> import farnsworth.mgr as mgr
    > creating a manager...
    > starting the manager...
    > sciencing faster...
    >
    > [stuck for some time... hitting ^C]
    > Traceback (most recent call last):
    > File "<stdin>", line 1, in <module>
    > File "farnsworth.py", line 32, in <module>
    > mgr, maths = _setup()
    > File "farnsworth.py", line 25, in _setup
    > maths = mgr.Maths()
    > File "/usr/lib/python2.6/multiprocessing/managers.py", line 634, in
    > temp
    > token, exp = self._create(typeid, *args, **kwds)
    > File "/usr/lib/python2.6/multiprocessing/managers.py", line 532, in
    > _create
    > conn = self._Client(self._address, authkey=self._authkey)
    > File "/usr/lib/python2.6/multiprocessing/connection.py", line 140,
    > in Client
    > answer_challenge(c, authkey)
    > File "/usr/lib/python2.6/multiprocessing/connection.py", line 372,
    > in answer_challenge
    > message = connection.recv_bytes(256) # reject large
    > message
    > KeyboardInterrupt
    >
    > is this a known limitation/feature of the multiprocessing module ?
    > is there a workaround (acquiring some import lock maybe) ?
    >
    > cheers,
    > sebastien.
    >
    > PS:
    > $ python
    > Python 2.6.4 (r264:75706, Oct 27 2009, 06:25:13)
    > [GCC 4.4.1] on linux2
    > Type "help", "copyright", "credits" or "license" for more information.
    >
    > $ python -c 'import sys; print sys.version_info'
    > (2, 6, 4, 'final', 0)
    >
    > $ uname -a
    > Linux farnsworth 2.6.31-ARCH #1 SMP PREEMPT Tue Nov 10 19:01:40 CET
    > 2009 x86_64 Intel(R) Core(TM)2 Duo CPU T9400 @ 2.53GHz GenuineIntel
    > GNU/Linux
    >
    >

    Since I don't see any other responses, I'll give my guess, even though
    I'm not very experienced with the multiprocessing module.

    It's my understanding that threads may not be created or destroyed
    during an import. So you need to find a way to defer the creation till
    the imports are done.

    DaveA
     
    Dave Angel, Dec 15, 2009
    #2
    1. Advertising

  3. Re: race/deadlock when creating a multiprocessing.manager instancewhile importing a module ?

    Dave,

    [..snip..]
    On Tuesday 15 December 2009 02:49:00 Dave Angel wrote:
    > Since I don't see any other responses, I'll give my guess, even though
    > I'm not very experienced with the multiprocessing module.
    >
    > It's my understanding that threads may not be created or destroyed
    > during an import. So you need to find a way to defer the creation till
    > the imports are done.


    yes, I figured this much and I already wrapped my module with a ModuleFacade
    object which has lazy properties attributes [1].
    this solves the problem of import my module, but if the module importing my
    module is itself being imported, I am back to square one.

    and my module is being used by many people.

    I guess I'd need at least a way to detect that somebody is trying to use the
    lazy properties of my module facade while an import is "still active".
    is there such a mechanism ?

    cheers,
    sebastien.

    [1]
    https://svnweb.cern.ch/trac/atlasoff/browser/Tools/PyUtils/trunk/python/AthFile/__init__.py
    --
    #########################################
    # Dr. Sebastien Binet
    # Laboratoire de l'Accelerateur Lineaire
    # Universite Paris-Sud XI
    # Batiment 200
    # 91898 Orsay
    #########################################
     
    Sebastien Binet, Dec 15, 2009
    #3
  4. Sebastien Binet

    pograph Guest

    Re: race/deadlock when creating a multiprocessing.manager instancewhile importing a module ?

    On Dec 15, 2:57 am, Sebastien Binet <> wrote:
    > Dave,
    >
    > [..snip..]
    > On Tuesday 15 December 2009 02:49:00 Dave Angel wrote:
    >
    > > Since I don't see any other responses, I'll give my guess, even though
    > > I'm not very experienced with the multiprocessing module.

    >
    > > It's my understanding that threads may not be created or destroyed
    > > during an import. So you need to find a way to defer the creation till
    > > the imports are done.

    >
    > yes, I figured this much and I already wrapped my module with a ModuleFacade
    > object which has lazy properties attributes [1].
    > this solves the problem of import my module, but if the module importing my
    > module is itself being imported, I am back to square one.
    >
    > and my module is being used by many people.
    >
    > I guess I'd need at least a way to detect that somebody is trying to use the
    > lazy properties of my module facade while an import is "still active".
    > is there such a mechanism ?
    >
    > cheers,
    > sebastien.
    >
    > [1]https://svnweb.cern.ch/trac/atlasoff/browser/Tools/PyUtils/trunk/pyth...
    > --
    > #########################################
    > # Dr. Sebastien Binet
    > # Laboratoire de l'Accelerateur Lineaire
    > # Universite Paris-Sud XI
    > # Batiment 200
    > # 91898 Orsay
    > #########################################


    The document of multiprocessing says:
    Functionality within this package requires that the __main__ method be
    importable by the children. This is covered in Programming guidelines
    however it is worth pointing out here. This means that some examples,
    such as the multiprocessing.Pool examples will not work in the
    interactive interpreter.
     
    pograph, Dec 16, 2009
    #4
  5. Re: race/deadlock when creating a multiprocessing.manager instancewhile importing a module ?

    hi there,

    > On Tuesday 15 December 2009 02:49:00 Dave Angel wrote:
    > > Since I don't see any other responses, I'll give my guess, even though
    > > I'm not very experienced with the multiprocessing module.
    > >
    > > It's my understanding that threads may not be created or destroyed
    > > during an import. So you need to find a way to defer the creation till
    > > the imports are done.

    >
    > yes, I figured this much and I already wrapped my module with a
    > ModuleFacade object which has lazy properties attributes [1].
    > this solves the problem of import my module, but if the module importing my
    > module is itself being imported, I am back to square one.
    >
    > and my module is being used by many people.
    >
    > I guess I'd need at least a way to detect that somebody is trying to use
    > the lazy properties of my module facade while an import is "still active".
    > is there such a mechanism ?

    digging thru the imp.py module, I devised this simple snippet:
    import imp
    [...]
    if imp.lock_held():
    raise RuntimeError('some crystal clear error message')

    thanks for the help :)

    cheers,
    sebastien.
    --
    #########################################
    # Dr. Sebastien Binet
    # Laboratoire de l'Accelerateur Lineaire
    # Universite Paris-Sud XI
    # Batiment 200
    # 91898 Orsay
    #########################################
     
    Sebastien Binet, Dec 16, 2009
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Felix

    Multiprocessing.Queue deadlock

    Felix, Oct 7, 2009, in forum: Python
    Replies:
    4
    Views:
    1,917
    Felix Schlesinger
    Oct 8, 2009
  2. Brian Quinlan

    multiprocessing deadlock

    Brian Quinlan, Oct 23, 2009, in forum: Python
    Replies:
    5
    Views:
    551
    larudwer
    Oct 24, 2009
  3. Metalone
    Replies:
    0
    Views:
    350
    Metalone
    Jan 6, 2010
  4. Atherun
    Replies:
    9
    Views:
    637
    Wong Wah Meng-R32813
    Sep 28, 2011
  5. smhall05
    Replies:
    4
    Views:
    126
Loading...

Share This Page