race/deadlock when creating a multiprocessing.manager instance whileimporting a module ?

[Sebastien Binet]

hi there,

say I have this module

## farnsworth ##
__all__ = [
'mgr',
'maths',
]

from multiprocessing.managers import BaseManager

class MathsClass(object):
def add(self, x, y):
return x + y
def mul(self, x, y):
return x * y

class MyManager(BaseManager):
pass

MyManager.register('Maths', MathsClass)

def _setup():
print "creating a manager..."
mgr = MyManager()
print "starting the manager..."
mgr.start()
print "sciencing faster..."
maths = mgr.Maths()
print maths.add(4,3)
print maths.mul(7,8)
print "done with sciencing."
return (mgr, maths)

# exec at module import
mgr, maths = _setup()

# prevent hysteresis + clean-up
del _setup
## EOF ##

if I use it like so:

$ python -m farnsworth
creating a manager...
starting the manager...
sciencing faster...
7
56
done with sciencing.

all is fine, but if I try to use it thru an import:
$ python
py> import farnsworth.mgr as mgr
creating a manager...
starting the manager...
sciencing faster...

[stuck for some time... hitting ^C]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "farnsworth.py", line 32, in <module>
mgr, maths = _setup()
File "farnsworth.py", line 25, in _setup
maths = mgr.Maths()
File "/usr/lib/python2.6/multiprocessing/managers.py", line 634, in
temp
token, exp = self._create(typeid, *args, **kwds)
File "/usr/lib/python2.6/multiprocessing/managers.py", line 532, in
_create
conn = self._Client(self._address, authkey=self._authkey)
File "/usr/lib/python2.6/multiprocessing/connection.py", line 140,
in Client
answer_challenge(c, authkey)
File "/usr/lib/python2.6/multiprocessing/connection.py", line 372,
in answer_challenge
message = connection.recv_bytes(256) # reject large
message
KeyboardInterrupt

is this a known limitation/feature of the multiprocessing module ?
is there a workaround (acquiring some import lock maybe) ?

cheers,
sebastien.

PS:
$ python
Python 2.6.4 (r264:75706, Oct 27 2009, 06:25:13)
[GCC 4.4.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.

$ python -c 'import sys; print sys.version_info'
(2, 6, 4, 'final', 0)

$ uname -a
Linux farnsworth 2.6.31-ARCH #1 SMP PREEMPT Tue Nov 10 19:01:40 CET
2009 x86_64 Intel(R) Core(TM)2 Duo CPU T9400 @ 2.53GHz GenuineIntel
GNU/Linux

 

[Dave Angel]

hi there,

say I have this module

## farnsworth ##
__all__ = [
'mgr',
'maths',
]

from multiprocessing.managers import BaseManager

class MathsClass(object):
def add(self, x, y):
return x + y
def mul(self, x, y):
return x * y

class MyManager(BaseManager):
pass

MyManager.register('Maths', MathsClass)

def _setup():
print "creating a manager..."
mgr = MyManager()
print "starting the manager..."
mgr.start()
print "sciencing faster..."
maths = mgr.Maths()
print maths.add(4,3)
print maths.mul(7,8)
print "done with sciencing."
return (mgr, maths)

# exec at module import
mgr, maths = _setup()

# prevent hysteresis + clean-up
del _setup
## EOF ##

if I use it like so:

$ python -m farnsworth
creating a manager...
starting the manager...
sciencing faster...
7
56
done with sciencing.

all is fine, but if I try to use it thru an import:
$ python
py> import farnsworth.mgr as mgr
creating a manager...
starting the manager...
sciencing faster...

[stuck for some time... hitting ^C]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "farnsworth.py", line 32, in <module>
mgr, maths = _setup()
File "farnsworth.py", line 25, in _setup
maths = mgr.Maths()
File "/usr/lib/python2.6/multiprocessing/managers.py", line 634, in
temp
token, exp = self._create(typeid, *args, **kwds)
File "/usr/lib/python2.6/multiprocessing/managers.py", line 532, in
_create
conn = self._Client(self._address, authkey=self._authkey)
File "/usr/lib/python2.6/multiprocessing/connection.py", line 140,
in Client
answer_challenge(c, authkey)
File "/usr/lib/python2.6/multiprocessing/connection.py", line 372,
in answer_challenge
message = connection.recv_bytes(256) # reject large
message
KeyboardInterrupt

is this a known limitation/feature of the multiprocessing module ?
is there a workaround (acquiring some import lock maybe) ?

cheers,
sebastien.

PS:
$ python
Python 2.6.4 (r264:75706, Oct 27 2009, 06:25:13)
[GCC 4.4.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.

$ python -c 'import sys; print sys.version_info'
(2, 6, 4, 'final', 0)

$ uname -a
Linux farnsworth 2.6.31-ARCH #1 SMP PREEMPT Tue Nov 10 19:01:40 CET
2009 x86_64 Intel(R) Core(TM)2 Duo CPU T9400 @ 2.53GHz GenuineIntel
GNU/Linux

Since I don't see any other responses, I'll give my guess, even though
I'm not very experienced with the multiprocessing module.

It's my understanding that threads may not be created or destroyed
during an import. So you need to find a way to defer the creation till
the imports are done.

DaveA

 

[Sebastien Binet]

Dave,

[..snip..]

Since I don't see any other responses, I'll give my guess, even though
I'm not very experienced with the multiprocessing module.

It's my understanding that threads may not be created or destroyed
during an import. So you need to find a way to defer the creation till
the imports are done.

yes, I figured this much and I already wrapped my module with a ModuleFacade
object which has lazy properties attributes [1].
this solves the problem of import my module, but if the module importing my
module is itself being imported, I am back to square one.

and my module is being used by many people.

I guess I'd need at least a way to detect that somebody is trying to use the
lazy properties of my module facade while an import is "still active".
is there such a mechanism ?

cheers,
sebastien.

[1]
https://svnweb.cern.ch/trac/atlasoff/browser/Tools/PyUtils/trunk/python/AthFile/__init__.py
--
#########################################
# Dr. Sebastien Binet
# Laboratoire de l'Accelerateur Lineaire
# Universite Paris-Sud XI
# Batiment 200
# 91898 Orsay
#########################################

 

[pograph]

Dave,

[..snip..]

Since I don't see any other responses, I'll give my guess, even though
I'm not very experienced with the multiprocessing module.

It's my understanding that threads may not be created or destroyed
during an import. So you need to find a way to defer the creation till
the imports are done.

yes, I figured this much and I already wrapped my module with a ModuleFacade
object which has lazy properties attributes [1].
this solves the problem of import my module, but if the module importing my
module is itself being imported, I am back to square one.

and my module is being used by many people.

I guess I'd need at least a way to detect that somebody is trying to use the
lazy properties of my module facade while an import is "still active".
is there such a mechanism ?

cheers,
sebastien.

[1]https://svnweb.cern.ch/trac/atlasoff/browser/Tools/PyUtils/trunk/pyth...
--
#########################################
# Dr. Sebastien Binet
# Laboratoire de l'Accelerateur Lineaire
# Universite Paris-Sud XI
# Batiment 200
# 91898 Orsay
#########################################

The document of multiprocessing says:
Functionality within this package requires that the __main__ method be
importable by the children. This is covered in Programming guidelines
however it is worth pointing out here. This means that some examples,
such as the multiprocessing.Pool examples will not work in the
interactive interpreter.

 

[Sebastien Binet]

hi there,

Since I don't see any other responses, I'll give my guess, even though
I'm not very experienced with the multiprocessing module.

It's my understanding that threads may not be created or destroyed
during an import. So you need to find a way to defer the creation till
the imports are done.

yes, I figured this much and I already wrapped my module with a
ModuleFacade object which has lazy properties attributes [1].
this solves the problem of import my module, but if the module importing my
module is itself being imported, I am back to square one.

and my module is being used by many people.

I guess I'd need at least a way to detect that somebody is trying to use
the lazy properties of my module facade while an import is "still active".
is there such a mechanism ?

digging thru the imp.py module, I devised this simple snippet:
import imp
[...]
if imp.lock_held():
raise RuntimeError('some crystal clear error message')

thanks for the help

cheers,
sebastien.
--
#########################################
# Dr. Sebastien Binet
# Laboratoire de l'Accelerateur Lineaire
# Universite Paris-Sud XI
# Batiment 200
# 91898 Orsay
#########################################