multiprocessing question/error

Discussion in 'Python' started by Eduardo Lenz, Jan 17, 2009.

  1. Eduardo Lenz

    Eduardo Lenz Guest

    Hi,

    I was using the former processing package with python 2.5 with no problems.
    After switching to python 2.6.1 I am having some problems with the same code.
    The problem seems to be related to the fact that I am using Pool.map
    with a bounded method, since it is inside a class. To clarify a little bit,
    let me show some parts of the code ....

    class Pygen3(self)....
    ....
    ....
    ....
    def calcula(self,indiv):
    ....
    ....
    ....

    def evaluate(self):
    ....
    ....
    indiv = range(mult*self.popsize,(mult+1)*self.popsize)
    pool = Pool(processes=nproc)
    results = pool.map(self.calcula,indiv)
    ...
    ...

    the error is the following

    Exception in thread Thread-1:
    Traceback (most recent call last):
    File "/usr/lib/python2.6/threading.py", line 522, in __bootstrap_inner
    self.run()
    File "/usr/lib/python2.6/threading.py", line 477, in run
    self.__target(*self.__args, **self.__kwargs)
    File "/usr/lib/python2.6/multiprocessing/pool.py", line 225, in
    _handle_tasks
    put(task)
    PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup
    __builtin__.instancemethod failed


    Thanks for your help.

    --

    Eduardo Lenz Cardoso
    Dr. Eng.
    Associate Professor

    State University of Santa Catarina
    Department of Mechanical Engineering
    89223-100 - Joinville-SC - Brasil

    Tel: +55 47 4009-7971 - Fax: +55 47 4009-7940
    E-mail:
    ---------------------------------------------


    --
    Esta mensagem foi verificada pelo sistema de antivírus e
    acredita-se estar livre de perigo.
     
    Eduardo Lenz, Jan 17, 2009
    #1
    1. Advertising

  2. Eduardo Lenz

    Aaron Brady Guest

    On Jan 16, 11:39 pm, Eduardo Lenz <> wrote:
    > Hi,
    >
    > I was using the former processing package with python 2.5 with no problems.
    > After switching to python 2.6.1 I am having some problems with the same code.
    > The problem seems to be related to the fact that I am using Pool.map
    > with a bounded method, since it is inside a class. To clarify a little bit,
    > let me show some parts of the code ....


    Hello,

    > class Pygen3(self)....


    I infer this was a typo.

    >     ....
    >     ....
    >     ....
    >     def calcula(self,indiv):
    >         ....
    >         ....
    >         ....
    >
    >    def evaluate(self):
    >      ....
    >      ....
    >     indiv = range(mult*self.popsize,(mult+1)*self.popsize)
    >    pool = Pool(processes=nproc)
    >    results = pool.map(self.calcula,indiv)


    I infer the indentation drifted.

    >    ...
    >    ...
    >
    > the error is the following
    >
    > Exception in thread Thread-1:
    > Traceback (most recent call last):
    >   File "/usr/lib/python2.6/threading.py", line 522, in __bootstrap_inner
    >     self.run()
    >   File "/usr/lib/python2.6/threading.py", line 477, in run
    >     self.__target(*self.__args, **self.__kwargs)
    >   File "/usr/lib/python2.6/multiprocessing/pool.py", line 225, in
    > _handle_tasks
    >     put(task)
    > PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup
    > __builtin__.instancemethod failed
    >
    > Thanks for your help.


    Checking on two typos above. I have this working:

    from multiprocessing import Process

    class C(object):
    def m1( self ):
    p1 = Process(target=self.m2, args=('bob1',))
    p2 = Process(target=self.m2, args=('bob2',))
    p1.start()
    p2.start()
    p1.join()
    p2.join()
    def m2( self, name ):
    print 'hello', name

    if __name__ == '__main__':
    c= C()
    c.m1()

    But I get the same error as you with this:

    from multiprocessing import Process, Pool

    class C(object):
    def m1( self ):
    arg= range( 10 )
    pool= Pool( 5 )
    results = pool.map(self.m2,arg)
    print results
    def m2( self, name ):
    print 'hello', name

    if __name__ == '__main__':
    c= C()
    c.m1()

    I build a workaround with this:

    from multiprocessing import Process, Pool

    class C(object):
    def m1( self ):
    arg= zip( [ self ]* 10, range( 10 ) )
    pool= Pool( 5 )
    results = pool.map(C_m2,arg)
    print results
    def m2( self, arg ):
    print 'hello', arg
    return arg* 2
    def C_m2( ar, **kwar ):
    return C.m2( *ar, **kwar )

    if __name__ == '__main__':
    c= C()
    c.m1()

    Note that you have to explicitly include 'self' as a first parameter
    when packing the arguments, and then explicitly unpack it in the
    helper function. Also, wrapping 'm2' with '@staticmethod' also fails
    with this error:

    Exception in thread Thread-1:
    Traceback (most recent call last):
    File "c:\programs\python26\lib\threading.py", line 522, in
    __bootstrap_inner
    self.run()
    File "c:\programs\python26\lib\threading.py", line 477, in run
    self.__target(*self.__args, **self.__kwargs)
    File "c:\programs\python26\lib\multiprocessing\pool.py", line 225,
    in _handle_tasks
    put(task)
    PicklingError: Can't pickle <type 'function'>: attribute lookup
    __builtin__.function failed

    If this works for you, it's possible to make the helper function more
    dynamic, by accepting a class name and an attribute name as string
    arguments, then retrieving them in the subprocesses.
     
    Aaron Brady, Jan 17, 2009
    #2
    1. Advertising

  3. En Sat, 17 Jan 2009 14:14:37 -0200, Eduardo Lenz <>
    escribió:

    > Another question: is it possible to
    > change a bound method to a function on the fly ? I was wondering if
    > something
    > like class.__dict__["function_name"] could do the job.


    Exactly.
    Or, if you already have the method, use its im_func attribute to obtain
    the original function. BTW, im_self is the "self" argument.

    --
    Gabriel Genellina
     
    Gabriel Genellina, Jan 17, 2009
    #3
  4. Eduardo Lenz

    Eduardo Lenz Guest

    On Saturday 17 January 2009 00:43:35 Aaron Brady wrote:
    > On Jan 16, 11:39 pm, Eduardo Lenz <> wrote:
    > > Hi,
    > >
    > > I was using the former processing package with python 2.5 with no
    > > problems. After switching to python 2.6.1 I am having some problems with
    > > the same code. The problem seems to be related to the fact that I am
    > > using Pool.map with a bounded method, since it is inside a class. To
    > > clarify a little bit, let me show some parts of the code ....

    >
    > Hello,
    >
    > > class Pygen3(self)....

    >
    > I infer this was a typo.
    >
    > >     ....
    > >     ....
    > >     ....
    > >     def calcula(self,indiv):
    > >         ....
    > >         ....
    > >         ....
    > >
    > >    def evaluate(self):
    > >      ....
    > >      ....
    > >     indiv = range(mult*self.popsize,(mult+1)*self.popsize)
    > >    pool = Pool(processes=nproc)
    > >    results = pool.map(self.calcula,indiv)

    >
    > I infer the indentation drifted.
    >
    > >    ...
    > >    ...
    > >
    > > the error is the following
    > >
    > > Exception in thread Thread-1:
    > > Traceback (most recent call last):
    > >   File "/usr/lib/python2.6/threading.py", line 522, in __bootstrap_inner
    > >     self.run()
    > >   File "/usr/lib/python2.6/threading.py", line 477, in run
    > >     self.__target(*self.__args, **self.__kwargs)
    > >   File "/usr/lib/python2.6/multiprocessing/pool.py", line 225, in
    > > _handle_tasks
    > >     put(task)
    > > PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup
    > > __builtin__.instancemethod failed
    > >
    > > Thanks for your help.

    >
    > Checking on two typos above. I have this working:
    >
    > from multiprocessing import Process
    >
    > class C(object):
    > def m1( self ):
    > p1 = Process(target=self.m2, args=('bob1',))
    > p2 = Process(target=self.m2, args=('bob2',))
    > p1.start()
    > p2.start()
    > p1.join()
    > p2.join()
    > def m2( self, name ):
    > print 'hello', name
    >
    > if __name__ == '__main__':
    > c= C()
    > c.m1()
    >
    > But I get the same error as you with this:
    >
    > from multiprocessing import Process, Pool
    >
    > class C(object):
    > def m1( self ):
    > arg= range( 10 )
    > pool= Pool( 5 )
    > results = pool.map(self.m2,arg)
    > print results
    > def m2( self, name ):
    > print 'hello', name
    >
    > if __name__ == '__main__':
    > c= C()
    > c.m1()
    >
    > I build a workaround with this:
    >
    > from multiprocessing import Process, Pool
    >
    > class C(object):
    > def m1( self ):
    > arg= zip( [ self ]* 10, range( 10 ) )
    > pool= Pool( 5 )
    > results = pool.map(C_m2,arg)
    > print results
    > def m2( self, arg ):
    > print 'hello', arg
    > return arg* 2
    > def C_m2( ar, **kwar ):
    > return C.m2( *ar, **kwar )
    >
    > if __name__ == '__main__':
    > c= C()
    > c.m1()
    >
    > Note that you have to explicitly include 'self' as a first parameter
    > when packing the arguments, and then explicitly unpack it in the
    > helper function. Also, wrapping 'm2' with '@staticmethod' also fails
    > with this error:
    >
    > Exception in thread Thread-1:
    > Traceback (most recent call last):
    > File "c:\programs\python26\lib\threading.py", line 522, in
    > __bootstrap_inner
    > self.run()
    > File "c:\programs\python26\lib\threading.py", line 477, in run
    > self.__target(*self.__args, **self.__kwargs)
    > File "c:\programs\python26\lib\multiprocessing\pool.py", line 225,
    > in _handle_tasks
    > put(task)
    > PicklingError: Can't pickle <type 'function'>: attribute lookup
    > __builtin__.function failed
    >
    > If this works for you, it's possible to make the helper function more
    > dynamic, by accepting a class name and an attribute name as string
    > arguments, then retrieving them in the subprocesses.
    > --
    > http://mail.python.org/mailman/listinfo/python-list


    Thanks for your help ! The only question remaninig is why it iused to work
    with the former processing package. Another question: is it possible to
    change a bound method to a function on the fly ? I was wondering if something
    like class.__dict__["function_name"] could do the job. Actually, I was
    wondering with something like a casting, but a guess it is not possible.

    Thanks again,
    Eduardo.
    --

    Eduardo Lenz Cardoso
    Dr. Eng.
    Associate Professor

    State University of Santa Catarina
    Department of Mechanical Engineering
    89223-100 - Joinville-SC - Brasil

    Tel: +55 47 4009-7971 - Fax: +55 47 4009-7940
    E-mail:
    ---------------------------------------------

    --
    Esta mensagem foi verificada pelo sistema de antivírus e
    acredita-se estar livre de perigo.
     
    Eduardo Lenz, Jan 17, 2009
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Amit N
    Replies:
    4
    Views:
    523
    Paddy
    Sep 13, 2007
  2. Eduardo Lenz

    multiprocessing question/error

    Eduardo Lenz, Jan 16, 2009, in forum: Python
    Replies:
    0
    Views:
    330
    Eduardo Lenz
    Jan 16, 2009
  3. xera121
    Replies:
    0
    Views:
    282
    xera121
    Oct 3, 2009
  4. Replies:
    1
    Views:
    586
    Tennessee
    Oct 14, 2009
  5. Junfeng Hu
    Replies:
    12
    Views:
    1,138
    Junfeng Hu
    Nov 19, 2011
Loading...

Share This Page