mutiprocessing, manager, list & threading ?

Discussion in 'Python' started by mika.saari@wipsl.com, Dec 28, 2010.

  1. Guest

    Hi,

    Testing if I could be able to use multiprocessing BaseManager to manage
    list of instance pointers between processes. If my intance inherits
    Thread, I get pickling error about _thread.lock.

    I found Steven Bethard's recipe for this kind of problem in
    http://bytes.com/topic/python/answers/552476-why-cant-you-pickle-instancemethods,
    but I do not to get the idea how to copyreg the _thread.lock.

    Is it even possible to save instance pointer to list which is
    synchronized between separate processes ?

    Thanks a lot,
    -Mika


    Client side code:

    from multiprocessing.managers import BaseManager
    from threading import Thread, Lock

    class ListManager(BaseManager):
    pass

    class Testing(Thread):
    def __init__(self):
    Thread.__init__(self)

    def nothing(self):
    print("Nothing")

    ListManager.register('get_list')
    ListManager.register('Testing',Testing)
    m = ListManager(address=('', 50000), authkey=b'abc')
    m.connect()

    list1 = m.get_list()
    test = Testing()
    print("TEST:",test)
    list1.append(test)
    print("TEST:",test)


    Error:
    -----------------------------------------------------------------------
    Traceback (most recent call last):
    File "mpclit.py", line 23, in <module>
    list1.append(test)
    File "<string>", line 2, in append
    File "/usr/local/lib/python3.1/multiprocessing/managers.py", line 735,
    in _callmethod

    conn.send((self._id, methodname, args, kwds))
    File "/usr/local/lib/python3.1/pickle.py", line 1358, in dumps
    Pickler(f, protocol, fix_imports=fix_imports).dump(obj)
    _pickle.PicklingError: Can't pickle <built-in method release of
    _thread.lock object at 0x1af710f0>: attribute lookup __main__.release
    failed


    References:
    ----------------------------------------------------------------------
    http://bytes.com/topic/python/answers/552476-why-cant-you-pickle-instancemethods
     
    , Dec 28, 2010
    #1
    1. Advertisements

  2. John Nagle Guest

    On 12/28/2010 1:38 AM, wrote:
    > Hi,
    >
    > Testing if I could be able to use multiprocessing BaseManager to manage
    > list of instance pointers between processes. If my intance inherits
    > Thread, I get pickling error about _thread.lock.


    The "multiprocessing" module works by running completely separate
    programs which communicate by copying data back and forth through
    pipes or sockets. You can't share thread locks. (There's a gimmick
    that lets you put an array of fixed type and size in shared memory,
    but that's very limited, because Python's locking doesn't really
    understand multiprocessing.)

    If you need to access a list from several processes, have
    one process own the list, and use a Manager from the multiprocessing
    module to create a proxy to access the list from the other processes.
    Note that this is slower than regular list access. But if you just
    need to have a to-do list that feeds multiple processes, that's a way
    to do it.

    John Nagle
     
    John Nagle, Dec 28, 2010
    #2
    1. Advertisements

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. S P Arif Sahari Wibowo

    Mailing list manager with VERP

    S P Arif Sahari Wibowo, Apr 3, 2004, in forum: Perl
    Replies:
    0
    Views:
    607
    S P Arif Sahari Wibowo
    Apr 3, 2004
  2. roopa
    Replies:
    6
    Views:
    1,039
    Jerry Coffin
    Aug 27, 2004
  3. List Manager response

    , Apr 18, 2004, in forum: Python
    Replies:
    0
    Views:
    318
  4. padma
    Replies:
    0
    Views:
    583
    padma
    Oct 3, 2007
  5. Replies:
    9
    Views:
    1,652
    Mark Space
    Dec 29, 2007
  6. Steven Woody
    Replies:
    0
    Views:
    656
    Steven Woody
    Jan 9, 2009
  7. Metalone
    Replies:
    0
    Views:
    554
    Metalone
    Jan 6, 2010
  8. La Sonda

    Mailing list manager

    La Sonda, Nov 15, 2004, in forum: Javascript
    Replies:
    0
    Views:
    175
    La Sonda
    Nov 15, 2004
Loading...