multiprocessing

Discussion in 'Python' started by elsa, Apr 7, 2011.

  1. elsa

    elsa Guest

    Hi guys,

    I want to try out some pooling of processors, but I'm not sure if it
    is possible to do what I want to do. Basically, I want to have a
    global object, that is updated during the execution of a function, and
    I want to be able to run this function several times on parallel
    processors. The order in which the function runs doesn't matter, and
    the value of the object doesn't matter to the function, but I do want
    the processors to take turns 'nicely' when updating the object, so
    there are no collisions. Here is an extremely simplified and trivial
    example of what I have in mind:

    from multiprocessing import Pool
    import random

    p=Pool(4)
    myDict={}

    def update(value):
    global myDict
    index=random.random()
    myDict[index]+=value

    total=1000

    p.map(update,range(total))


    After, I would also like to be able to use several processors to
    access the global object (but not modify it). Again, order doesn't
    matter:

    p1=Pool(4)

    def getValues(index):
    global myDict
    print myDict[index]

    p1.map(getValues,keys.myDict)

    Is there a way to do this?

    Thanks,

    Elsa.
     
    elsa, Apr 7, 2011
    #1
    1. Advertising

  2. On Apr 7, 2011, at 8:57 PM, Kerensa McElroy wrote:

    >
    > Hi,
    >
    > thanks for your response.
    >
    > I checked out multiprocessing.value, however from what I can make out, it works with object of only a very limited type. Is there a way to do this for more complex objects? (In reality, my object is a large multi-dimensional numpy array).


    Elsa,
    Are you following the current thread in this list which is talking about sharing numpy arrays via multiprocessing?

    http://mail.python.org/pipermail/python-list/2011-April/1269173.html





    > Date: Wed, 6 Apr 2011 22:20:06 -0700
    > Subject: Re: multiprocessing
    > From:
    > To:
    > CC:
    >
    >
    > On Wed, Apr 6, 2011 at 9:06 PM, elsa <> wrote:
    >
    > Hi guys,
    >
    >
    >
    > I want to try out some pooling of processors, but I'm not sure if it
    >
    > is possible to do what I want to do. Basically, I want to have a
    >
    > global object, that is updated during the execution of a function, and
    >
    > I want to be able to run this function several times on parallel
    >
    > processors. The order in which the function runs doesn't matter, and
    >
    > the value of the object doesn't matter to the function, but I do want
    >
    > the processors to take turns 'nicely' when updating the object, so
    >
    > there are no collisions. Here is an extremely simplified and trivial
    >
    > example of what I have in mind:
    >
    >
    >
    > from multiprocessing import Pool
    >
    > import random
    >
    >
    >
    > p=Pool(4)
    >
    > myDict={}
    >
    >
    >
    > def update(value):
    >
    > global myDict
    >
    > index=random.random()
    >
    > myDict[index]+=value
    >
    >
    >
    > total=1000
    >
    >
    >
    > p.map(update,range(total))
    >
    >
    >
    >
    >
    > After, I would also like to be able to use several processors to
    >
    > access the global object (but not modify it). Again, order doesn't
    >
    > matter:
    >
    >
    >
    > p1=Pool(4)
    >
    >
    >
    > def getValues(index):
    >
    > global myDict
    >
    > print myDict[index]
    >
    >
    >
    > p1.map(getValues,keys.myDict)
    >
    >
    >
    > Is there a way to do this
    > This should give you a synchronized wrapper around an object in shared memory:
    >
    > http://docs.python.org/library/multiprocessing.html#multiprocessing.Value
    >
    >
    > --
    > http://mail.python.org/mailman/listinfo/python-list
     
    Philip Semanchuk, Apr 8, 2011
    #2
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Amit N
    Replies:
    4
    Views:
    553
    Paddy
    Sep 13, 2007
  2. sturlamolden

    multiprocessing module (PEP 371)

    sturlamolden, Jun 4, 2008, in forum: Python
    Replies:
    6
    Views:
    455
    John Nagle
    Jun 8, 2008
  3. Max Ivanov

    multiprocessing eats memory

    Max Ivanov, Sep 25, 2008, in forum: Python
    Replies:
    6
    Views:
    352
    redbaron
    Sep 27, 2008
  4. Aaron \Castironpi\ Brady

    2.6 multiprocessing and pdb

    Aaron \Castironpi\ Brady, Oct 2, 2008, in forum: Python
    Replies:
    1
    Views:
    705
    Gabriel Genellina
    Oct 3, 2008
  5. nhwarriors

    Using multiprocessing

    nhwarriors, Oct 10, 2008, in forum: Python
    Replies:
    4
    Views:
    292
    Aaron \Castironpi\ Brady
    Oct 11, 2008
Loading...

Share This Page