Multiprocessing Pool and functions with many arguments

Discussion in 'Python' started by psaffrey@googlemail.com, Apr 29, 2009.

  1. Guest

    I'm trying to get to grips with the multiprocessing module, having
    only used ParallelPython before.

    based on this example:

    http://docs.python.org/library/multiprocessing.html#using-a-pool-of-workers

    what happens if I want my "f" to take more than one argument? I want
    to have a list of tuples of arguments and have these correspond the
    arguments in f, but it keeps complaining that I only have one argument
    (the tuple). Do I have to pass in a tuple and break it up inside f? I
    can't use multiple input lists, as I would with regular map.

    Thanks,

    Peter
     
    , Apr 29, 2009
    #1
    1. Advertising

  2. Jesse Noller Guest

    On Wed, Apr 29, 2009 at 2:01 PM,
    <> wrote:
    > I'm trying to get to grips with the multiprocessing module, having
    > only used ParallelPython before.
    >
    > based on this example:
    >
    > http://docs.python.org/library/multiprocessing.html#using-a-pool-of-workers
    >
    > what happens if I want my "f" to take more than one argument? I want
    > to have a list of tuples of arguments and have these correspond the
    > arguments in f, but it keeps complaining that I only have one argument
    > (the tuple). Do I have to pass in a tuple and break it up inside f? I
    > can't use multiple input lists, as I would with regular map.
    >
    > Thanks,
    >
    > Peter


    Perhaps these articles will help you:

    http://www.doughellmann.com/PyMOTW/multiprocessing/communication.html#pool-map
    http://www.doughellmann.com/PyMOTW/multiprocessing/mapreduce.html#simplemapreduce

    jesse
     
    Jesse Noller, Apr 30, 2009
    #2
    1. Advertising

  3. >>>>> "" <> (P) wrote:

    >P> I'm trying to get to grips with the multiprocessing module, having
    >P> only used ParallelPython before.


    >P> based on this example:


    >P> http://docs.python.org/library/multiprocessing.html#using-a-pool-of-workers


    >P> what happens if I want my "f" to take more than one argument? I want
    >P> to have a list of tuples of arguments and have these correspond the
    >P> arguments in f, but it keeps complaining that I only have one argument
    >P> (the tuple). Do I have to pass in a tuple and break it up inside f? I
    >P> can't use multiple input lists, as I would with regular map.


    You give the tuple of the arguments for the function:

    def f(a, b, c):
    return a + b * c

    pool = Pool(processes=4) # start 4 worker processes

    result = pool.apply(f, (2, 3, 4)) # evaluate "f(2, 3, 4)"

    Or if you have a list:

    args = [ (2, 3, 4), # arguments for call 1
    (5, 6, 7) # arguments for call 2
    ]

    print [pool.apply(f, a) for a in args]

    However, as each call to apply wait for its results, this will execute
    sequentially instead of parallel.

    You can't use map directly as it works only with single argument functions.
    >>> print pool.map(f, args)

    Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/multiprocessing/pool.py", line 148, in map
    return self.map_async(func, iterable, chunksize).get()
    File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/multiprocessing/pool.py", line 422, in get
    raise self._value
    TypeError: f() takes exactly 3 arguments (1 given)

    Is that what you mean?

    But you can use a wrapper function:

    def wrapf(abc):
    return f(*abc)

    [later...]

    print pool.map(wrapf, args)

    This is covered in the examples section of multiprocessing (see
    calculate and calculatestar for example).

    Or you can use apply_async and later wait for the results:

    results = [pool.apply_async(f, a) for a in args]
    print [r.get() for r in results]

    Now the calls to f are done in parallel, which you can check by putting
    a sleep inside f.

    --
    Piet van Oostrum <>
    URL: http://pietvanoostrum.com [PGP 8DAE142BE17999C4]
    Private email:
     
    Piet van Oostrum, May 1, 2009
    #3
  4. Aaron Brady Guest

    On Apr 29, 1:01 pm, ""
    <> wrote:
    > I'm trying to get to grips with the multiprocessing module, having
    > only used ParallelPython before.
    >
    > based on this example:
    >
    > http://docs.python.org/library/multiprocessing.html#using-a-pool-of-w...
    >
    > what happens if I want my "f" to take more than one argument? I want
    > to have a list of tuples of arguments and have these correspond the
    > arguments in f, but it keeps complaining that I only have one argument
    > (the tuple). Do I have to pass in a tuple and break it up inside f? I
    > can't use multiple input lists, as I would with regular map.
    >
    > Thanks,
    >
    > Peter


    The basic constructor should allow it.

    http://docs.python.org/library/multiprocessing.html#the-process-class

    p = Process(target=f, args=('bob',))

    I think you are imagining:

    result = pool.apply_async(f, [ *(1,2,3), *(4,5,6) ] )

    But I don't know of a way.
     
    Aaron Brady, May 1, 2009
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Replies:
    1
    Views:
    337
    Jesse Noller
    Apr 30, 2009
  2. masher
    Replies:
    4
    Views:
    907
    ryles
    Jul 3, 2009
  3. Rick Lawson
    Replies:
    8
    Views:
    826
    Graham Dumpleton
    Jul 17, 2009
  4. Allen Fowler
    Replies:
    1
    Views:
    1,261
  5. John Ladasky
    Replies:
    7
    Views:
    911
    John Ladasky
    Sep 17, 2011
Loading...

Share This Page