multiprocessing module - isn't it a bug?

Discussion in 'Python' started by dmitrey, Mar 14, 2009.

  1. dmitrey

    dmitrey Guest

    # THIS WORKS OK
    from multiprocessing import Pool
    N = 400
    K = 800
    processes = 2

    def costlyFunction2(z):
    r = 0
    for k in xrange(1, K+2):
    r += z ** (1 / k**1.5)
    return r

    class ABC:
    def __init__(self): pass
    def testParallel(self):
    po = Pool(processes=processes)
    r = po.map(costlyFunction2, xrange(N), chunksize = N/
    (10*processes))
    A=ABC()
    A.testParallel()
    print 'done'

    # But when I define costlyFunction2 inside of class, it doesn't work:
    from multiprocessing import Pool
    N = 400
    K = 800
    processes = 2
    class ABC:
    def __init__(self): pass
    def testParallel(self):
    po = Pool(processes=processes)
    def costlyFunction2(z):
    r = 0
    for k in xrange(1, K+2):
    r += z ** (1 / k**1.5)
    return r
    r = po.map(costlyFunction2, xrange(N), chunksize = N/
    (10*processes))
    A=ABC()
    A.testParallel()
    print 'done'

    Exception in thread Thread-1:
    Traceback (most recent call last):
    File "/usr/lib/python2.6/threading.py", line 522, in
    __bootstrap_inner
    self.run()
    File "/usr/lib/python2.6/threading.py", line 477, in run
    self.__target(*self.__args, **self.__kwargs)
    File "/usr/lib/python2.6/multiprocessing/pool.py", line 225, in
    _handle_tasks
    put(task)
    PicklingError: Can't pickle <type 'function'>: attribute lookup
    __builtin__.function failed

    This doesn't work for
    costlyFunction2 = lambda x: 11
    as well; and it doesn't work for imap, apply_async as well (same
    error).
    So, isn't it a bug, or it can be somehow fixed?
    Thank you in advance, D.
    dmitrey, Mar 14, 2009
    #1
    1. Advertising

  2. dmitrey

    Terry Reedy Guest

    dmitrey wrote:
    > # THIS WORKS OK
    > from multiprocessing import Pool
    > N = 400
    > K = 800
    > processes = 2
    >
    > def costlyFunction2(z):
    > r = 0
    > for k in xrange(1, K+2):
    > r += z ** (1 / k**1.5)
    > return r
    >
    > class ABC:
    > def __init__(self): pass
    > def testParallel(self):
    > po = Pool(processes=processes)
    > r = po.map(costlyFunction2, xrange(N), chunksize = N/
    > (10*processes))
    > A=ABC()
    > A.testParallel()
    > print 'done'
    >
    > # But when I define costlyFunction2 inside of class, it doesn't work:
    > from multiprocessing import Pool
    > N = 400
    > K = 800
    > processes = 2
    > class ABC:
    > def __init__(self): pass
    > def testParallel(self):
    > po = Pool(processes=processes)
    > def costlyFunction2(z):
    > r = 0
    > for k in xrange(1, K+2):
    > r += z ** (1 / k**1.5)
    > return r
    > r = po.map(costlyFunction2, xrange(N), chunksize = N/
    > (10*processes))
    > A=ABC()
    > A.testParallel()
    > print 'done'
    >
    > Exception in thread Thread-1:
    > Traceback (most recent call last):
    > File "/usr/lib/python2.6/threading.py", line 522, in
    > __bootstrap_inner
    > self.run()
    > File "/usr/lib/python2.6/threading.py", line 477, in run
    > self.__target(*self.__args, **self.__kwargs)
    > File "/usr/lib/python2.6/multiprocessing/pool.py", line 225, in
    > _handle_tasks
    > put(task)
    > PicklingError: Can't pickle <type 'function'>: attribute lookup
    > __builtin__.function failed
    >
    > This doesn't work for
    > costlyFunction2 = lambda x: 11
    > as well; and it doesn't work for imap, apply_async as well (same
    > error).
    > So, isn't it a bug, or it can be somehow fixed?


    Since the multiproccessing Programming Guidelines say
    "Picklability
    Ensure that the arguments to the methods of proxies are picklable."
    and you got a PicklingError, perhaps 'no' and 'no', but I would research
    the pickle module and 'picklability' to be sure.

    tjr
    Terry Reedy, Mar 14, 2009
    #2
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. erikcw
    Replies:
    10
    Views:
    2,001
    Jesse Noller
    Aug 5, 2009
  2. Michael Riedel
    Replies:
    2
    Views:
    1,328
    Michael
    Aug 27, 2009
  3. Felix
    Replies:
    1
    Views:
    854
    Robert Kern
    Oct 8, 2009
  4. Frank Millman

    Minor bug in multiprocessing?

    Frank Millman, Dec 19, 2009, in forum: Python
    Replies:
    2
    Views:
    250
    Frank Millman
    Jan 6, 2010
  5. Jerrad Genson
    Replies:
    0
    Views:
    472
    Jerrad Genson
    Nov 4, 2010
Loading...

Share This Page