mutiprocessing, manager, list & threading ?

Discussion in 'Python' started by mika.saari@wipsl.com, Dec 28, 2010.

  1. Guest

    Hi,

    Testing if I could be able to use multiprocessing BaseManager to manage
    list of instance pointers between processes. If my intance inherits
    Thread, I get pickling error about _thread.lock.

    I found Steven Bethard's recipe for this kind of problem in
    http://bytes.com/topic/python/answers/552476-why-cant-you-pickle-instancemethods,
    but I do not to get the idea how to copyreg the _thread.lock.

    Is it even possible to save instance pointer to list which is
    synchronized between separate processes ?

    Thanks a lot,
    -Mika


    Client side code:

    from multiprocessing.managers import BaseManager
    from threading import Thread, Lock

    class ListManager(BaseManager):
    pass

    class Testing(Thread):
    def __init__(self):
    Thread.__init__(self)

    def nothing(self):
    print("Nothing")

    ListManager.register('get_list')
    ListManager.register('Testing',Testing)
    m = ListManager(address=('', 50000), authkey=b'abc')
    m.connect()

    list1 = m.get_list()
    test = Testing()
    print("TEST:",test)
    list1.append(test)
    print("TEST:",test)


    Error:
    -----------------------------------------------------------------------
    Traceback (most recent call last):
    File "mpclit.py", line 23, in <module>
    list1.append(test)
    File "<string>", line 2, in append
    File "/usr/local/lib/python3.1/multiprocessing/managers.py", line 735,
    in _callmethod

    conn.send((self._id, methodname, args, kwds))
    File "/usr/local/lib/python3.1/pickle.py", line 1358, in dumps
    Pickler(f, protocol, fix_imports=fix_imports).dump(obj)
    _pickle.PicklingError: Can't pickle <built-in method release of
    _thread.lock object at 0x1af710f0>: attribute lookup __main__.release
    failed


    References:
    ----------------------------------------------------------------------
    http://bytes.com/topic/python/answers/552476-why-cant-you-pickle-instancemethods
     
    , Dec 28, 2010
    #1
    1. Advertising

  2. John Nagle Guest

    On 12/28/2010 1:38 AM, wrote:
    > Hi,
    >
    > Testing if I could be able to use multiprocessing BaseManager to manage
    > list of instance pointers between processes. If my intance inherits
    > Thread, I get pickling error about _thread.lock.


    The "multiprocessing" module works by running completely separate
    programs which communicate by copying data back and forth through
    pipes or sockets. You can't share thread locks. (There's a gimmick
    that lets you put an array of fixed type and size in shared memory,
    but that's very limited, because Python's locking doesn't really
    understand multiprocessing.)

    If you need to access a list from several processes, have
    one process own the list, and use a Manager from the multiprocessing
    module to create a proxy to access the list from the other processes.
    Note that this is slower than regular list access. But if you just
    need to have a to-do list that feeds multiple processes, that's a way
    to do it.

    John Nagle
     
    John Nagle, Dec 28, 2010
    #2
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. padma
    Replies:
    0
    Views:
    354
    padma
    Oct 3, 2007
  2. Replies:
    9
    Views:
    1,048
    Mark Space
    Dec 29, 2007
  3. Steven Woody
    Replies:
    0
    Views:
    425
    Steven Woody
    Jan 9, 2009
  4. Steven Woody
    Replies:
    0
    Views:
    458
    Steven Woody
    Jan 9, 2009
  5. Metalone
    Replies:
    0
    Views:
    346
    Metalone
    Jan 6, 2010
Loading...

Share This Page