Problem with multiprocessing

[Pietro Abate]

Hi all,

I'm a bit struggling to understand a KeyError raised by the multiprocessing library.

My idea is pretty simple. I want to create a server that will spawn a number of
workers that will share the same socket and handle requests independently. The
goal is to build a 3-tier structure where all requests are handled via an http
server and then dispatched to nodes sitting in a cluster and from nodes to
workers via the multiprocessing managers...

There is one public server, one node per machine and x number of workers on
each machine depending on the number of cores... I know I can use a more
sophisticated library, but for such a simple task (I'm just prototyping here) I
would just use the multiprocessing library... Is this possible or I should
explore directly other solutions ? I feel I'm very close to have something
working here ...

The problem with the code below is that if I run the server as
`python server.py 1` , that is, using only one process, it works as expected.

However if I spawn two processes (`python server.py 2`) listening for
connections, I get a nasty error :

$python client.py ping
Traceback (most recent call last):
File "client.py", line 24, in <module>
sys.exit(main(sys.argv))
File "client.py", line 21, in main
print m.solver(args[1])._getvalue()
File "/usr/lib/python2.6/multiprocessing/managers.py", line 637, in temp
authkey=self._authkey, exposed=exp
File "/usr/lib/python2.6/multiprocessing/managers.py", line 894, in AutoProxy
incref=incref)
File "/usr/lib/python2.6/multiprocessing/managers.py", line 700, in __init__
self._incref()
File "/usr/lib/python2.6/multiprocessing/managers.py", line 750, in _incref
dispatch(conn, None, 'incref', (self._id,))
File "/usr/lib/python2.6/multiprocessing/managers.py", line 79, in dispatch
raise convert_to_error(kind, result)
multiprocessing.managers.RemoteError:
---------------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/lib/python2.6/multiprocessing/managers.py", line 181, in handle_request
result = func(c, *args, **kwds)
File "/usr/lib/python2.6/multiprocessing/managers.py", line 402, in incref
self.id_to_refcount[ident] += 1
KeyError: '7fb51084c518'
---------------------------------------------------------------------------

My understanding is that all processes share the same socket (from the
Manager). When a client wants to connect, a new connection is created and
server independently by that process. If you look at the server trace (using
logging), it actually receives the connection, handles it, but fails to
communicate back to the client.

Can anybody shed some light for me and maybe propose a solution ?

thanks
pietro

----------------------------------------

Server :

import sys
from multiprocessing.managers import BaseManager, BaseProxy, Process

def baz(aa) :
l = []
for i in range(3) :
l.append(aa)
return l

class SolverManager(BaseManager): pass

class MyProxy(BaseProxy): pass

manager = SolverManager(address=('127.0.0.1', 50000), authkey='mpm')
manager.register('solver', callable=baz, proxytype=MyProxy)

def serve_forever(server):
try :
server.serve_forever()
except KeyboardInterrupt:
pass

def runpool(n):
server = manager.get_server()
workers = []

for i in range(int(n)):
Process(target=serve_forever, args=(server,)).start()

if __name__ == '__main__':
runpool(sys.argv[1])


Client :

import sys
from multiprocessing.managers import BaseManager, BaseProxy

import multiprocessing, logging

class SolverManager(BaseManager): pass

class MyProxy(BaseProxy): pass

def main(args) :
SolverManager.register('solver')
m = SolverManager(address=('127.0.0.1', 50000), authkey='mpm')
m.connect()

print m.solver(args[1])._getvalue()

if __name__ == '__main__':
sys.exit(main(sys.argv))


also tried on stack overflow and the python list, but I didn't manage to come up
with a working solution yet...