Problem with multiprocessing

Discussion in 'Python' started by tleeuwenburg@gmail.com, Sep 2, 2009.

  1. Guest

    I have a problem using multiprocessing in a simple way. I created a
    file, testmp.py, with the following contents:

    ---------------------------------------------------
    import multiprocessing as mp

    p = mp.Pool(5)

    def f(x):
    return x * x

    print map(f, [1,2,3,4,5])
    print p.map(f, [1,2,3,4,5])

    ----------------------------------------------------

    I'm using 2.6 r26:66713, so not quite bleeding edge.

    If I run 'python2.6 testmp.py' I get errors of exactly the kind
    mentioned on docs.python.org for when multiprocessing is run using the
    interactive interpreter. However, I'm obviously *not* running the
    interactive interpreter.

    Any suggestions?
    , Sep 2, 2009
    #1
    1. Advertising

  2. Peter Otten Guest

    wrote:

    > I have a problem using multiprocessing in a simple way. I created a
    > file, testmp.py, with the following contents:
    >
    > ---------------------------------------------------
    > import multiprocessing as mp
    >
    > p = mp.Pool(5)
    >
    > def f(x):
    > return x * x
    >
    > print map(f, [1,2,3,4,5])
    > print p.map(f, [1,2,3,4,5])
    >
    > ----------------------------------------------------
    >
    > I'm using 2.6 r26:66713, so not quite bleeding edge.
    >
    > If I run 'python2.6 testmp.py' I get errors of exactly the kind
    > mentioned on docs.python.org for when multiprocessing is run using the
    > interactive interpreter. However, I'm obviously *not* running the
    > interactive interpreter.
    >
    > Any suggestions?


    I'm too lazy to read the docs, so I just tinkered:

    import multiprocessing as mp
    import testmp

    p = mp.Pool(5)

    def f(x):
    return x * x

    if __name__ == "__main__":
    print map(f, [1,2,3,4,5])
    print p.map(testmp.f, [1,2,3,4,5])
    Peter Otten, Sep 2, 2009
    #2
    1. Advertising

  3. On 09/02/2009 04:51 AM, Peter Otten wrote:
    > wrote:
    >
    >> I have a problem using multiprocessing in a simple way. I created a
    >> file, testmp.py, with the following contents:
    >>
    >> ---------------------------------------------------
    >> import multiprocessing as mp
    >>
    >> p = mp.Pool(5)
    >>
    >> def f(x):
    >> return x * x
    >>
    >> print map(f, [1,2,3,4,5])
    >> print p.map(f, [1,2,3,4,5])

    >
    > I'm too lazy to read the docs, so I just tinkered:
    >
    > import multiprocessing as mp
    > import testmp
    >
    > p = mp.Pool(5)
    >
    > def f(x):
    > return x * x
    >
    > if __name__ == "__main__":
    > print map(f, [1,2,3,4,5])
    > print p.map(testmp.f, [1,2,3,4,5])
    >
    >

    Yes, to use the multiprocessing module, you must make your script
    importable, so runtime statements should go into a __main__
    conditional. This way, when multiprocessing imports the module for each
    of its threads, the actual runtime code only gets executed once in the
    parent thread, which has the lock. At least, that is what I think is
    happening.


    import multiprocessing as mp

    def f(x):
    return x * x

    if __name__ == '__main__':
    p = mp.Pool(5)
    print map(f, [1,2,3,4,5])
    print p.map(f, [1,2,3,4,5])


    http://docs.python.org/library/multiprocessing.html

    --
    http://invisibleroads.com
    Connecting Python developers with local businesses
    Roy Hyunjin Han, Sep 2, 2009
    #3
  4. Tennessee Guest

    > Yes, to use the multiprocessing module, you must make your script
    > importable, so runtime statements should go into a __main__
    > conditional. This way, when multiprocessing imports the module for each
    > of its threads, the actual runtime code only gets executed once in the
    > parent thread, which has the lock. At least, that is what I think is
    > happening.


    Oh, that makes total sense. Thanks to both of you!

    -T
    Tennessee, Sep 4, 2009
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. gopal mishra

    problem in implementing multiprocessing

    gopal mishra, Jan 16, 2009, in forum: Python
    Replies:
    0
    Views:
    244
    gopal mishra
    Jan 16, 2009
  2. James Mills
    Replies:
    2
    Views:
    302
    Aaron Brady
    Jan 19, 2009
  3. Wu Zhe
    Replies:
    2
    Views:
    453
    Piet van Oostrum
    May 27, 2009
  4. Metalone
    Replies:
    0
    Views:
    212
    Metalone
    Jan 6, 2010
  5. Robert Kern
    Replies:
    2
    Views:
    315
    Wolodja Wentland
    Jan 12, 2010
Loading...

Share This Page