Using multiprocessing

Discussion in 'Python' started by nhwarriors, Oct 10, 2008.

  1. nhwarriors

    nhwarriors Guest

    I am attempting to use the (new in 2.6) multiprocessing package to
    process 2 items in a large queue of items simultaneously. I'd like to
    be able to print to the screen the results of each item before
    starting the next one. I'm having trouble with this so far.

    Here is some (useless) example code that shows how far I've gotten by
    reading the documentation:

    from multiprocessing import Process, Queue, current_process

    def main():
    facs = []
    for i in range(50000,50005):
    facs.append(i)

    tasks = [(fac, (i,)) for i in facs]
    task_queue = Queue()
    done_queue = Queue()

    for task in tasks:
    task_queue.put(task)

    for i in range(2):
    Process(target = worker, args = (task_queue, done_queue)).start()

    for i in range(len(tasks)):
    print done_queue.get()

    for i in range(2):
    task_queue.put('STOP')

    def worker(input, output):
    for func, args in iter(input.get, 'STOP'):
    result = func(*args)
    output.put(result)

    def fac(n):
    f = n
    for i in range(n-1,1,-1):
    f *= i
    return 'fac('+str(n)+') done on '+current_process().name

    if __name__ == '__main__':
    main()

    This works great, except that nothing can be output until everything
    in the queue is finished. I'd like to write out the result of fac(n)
    for each item in the queue as it happens.

    I'm probably approaching the problem all wrong - can anyone set me on
    the right track?
     
    nhwarriors, Oct 10, 2008
    #1
    1. Advertising

  2. nhwarriors

    Jesse Noller Guest

    On Fri, Oct 10, 2008 at 4:32 PM, nhwarriors <> wrote:
    > I am attempting to use the (new in 2.6) multiprocessing package to
    > process 2 items in a large queue of items simultaneously. I'd like to
    > be able to print to the screen the results of each item before
    > starting the next one. I'm having trouble with this so far.
    >
    > Here is some (useless) example code that shows how far I've gotten by
    > reading the documentation:
    >
    > from multiprocessing import Process, Queue, current_process
    >
    > def main():
    > facs = []
    > for i in range(50000,50005):
    > facs.append(i)
    >
    > tasks = [(fac, (i,)) for i in facs]
    > task_queue = Queue()
    > done_queue = Queue()
    >
    > for task in tasks:
    > task_queue.put(task)
    >
    > for i in range(2):
    > Process(target = worker, args = (task_queue, done_queue)).start()
    >
    > for i in range(len(tasks)):
    > print done_queue.get()
    >
    > for i in range(2):
    > task_queue.put('STOP')
    >
    > def worker(input, output):
    > for func, args in iter(input.get, 'STOP'):
    > result = func(*args)
    > output.put(result)
    >
    > def fac(n):
    > f = n
    > for i in range(n-1,1,-1):
    > f *= i
    > return 'fac('+str(n)+') done on '+current_process().name
    >
    > if __name__ == '__main__':
    > main()
    >
    > This works great, except that nothing can be output until everything
    > in the queue is finished. I'd like to write out the result of fac(n)
    > for each item in the queue as it happens.
    >
    > I'm probably approaching the problem all wrong - can anyone set me on
    > the right track?


    I'm not quite following: If you run this, the results are printed by
    the main thread, unordered, as they are put on the results queue -
    this works as intended (and the example this is based on works the
    same way) .

    For example:
    result put Process-2
    result put Process-1
    fac(50000) done on Process-1
    result put Process-2
    fac(50001) done on Process-2
    result put Process-1
    fac(50003) done on Process-1
    result put Process-2
    fac(50002) done on Process-2
    fac(50004) done on Process-2

    You can see this if you expand the range:

    result put Process-1
    result put Process-2
    result put Process-2
    fac(50001) done on Process-2
    result put Process-1
    fac(50000) done on Process-1
    fac(50003) done on Process-2
    result put Process-2
    result put Process-1
    fac(50004) done on Process-2
    result put Process-2
    fac(50006) done on Process-2
    result put Process-1
    fac(50002) done on Process-1
    fac(50005) done on Process-1
    fac(50007) done on Process-1
    result put Process-2
    result put Process-1
    fac(50008) done on Process-2
    result put Process-2
    result put Process-1
    fac(50010) done on Process-2
    result put Process-2
    result put Process-1
    fac(50009) done on Process-1
    fac(50011) done on Process-1
    fac(50013) done on Process-1
    result put Process-2
    fac(50012) done on Process-2
    fac(50014) done on Process-2

    One trick I use is when I have a results queue to manage, I spawn an
    addition process to read off of the results queue and deal with the
    results. This is mainly so I can process the results outside of the
    main thread, as they appear on the results queue

    -jesse
     
    Jesse Noller, Oct 11, 2008
    #2
    1. Advertising

  3. On Oct 10, 3:32 pm, nhwarriors <> wrote:
    > I am attempting to use the (new in 2.6) multiprocessing package to
    > process 2 items in a large queue of items simultaneously. I'd like to
    > be able to print to the screen the results of each item before
    > starting the next one. I'm having trouble with this so far.
    >
    > Here is some (useless) example code that shows how far I've gotten by
    > reading the documentation:
    >

    snip code
    >
    > This works great, except that nothing can be output until everything
    > in the queue is finished. I'd like to write out the result of fac(n)
    > for each item in the queue as it happens.
    >
    > I'm probably approaching the problem all wrong - can anyone set me on
    > the right track?


    Works fine for me. Formatting time.clock(), with a different range:

    0.00000 fac(25000) done on Process-2
    0.09334 fac(25001) done on Process-1
    2.25036 fac(25002) done on Process-2
    2.41227 fac(25003) done on Process-1
    3.57167 fac(25004) done on Process-2

    I'm on Win32.
     
    Aaron \Castironpi\ Brady, Oct 11, 2008
    #3
  4. nhwarriors

    nhwarriors Guest

    On Oct 10, 10:52 pm, "Aaron \"Castironpi\" Brady"
    <> wrote:
    > On Oct 10, 3:32 pm, nhwarriors <> wrote:
    >
    >
    >
    > > I am attempting to use the (new in 2.6) multiprocessing package to
    > > process 2 items in a large queue of items simultaneously. I'd like to
    > > be able to print to the screen the results of each item before
    > > starting the next one. I'm having trouble with this so far.

    >
    > > Here is some (useless) example code that shows how far I've gotten by
    > > reading the documentation:

    >
    > snip code
    >
    > > This works great, except that nothing can be output until everything
    > > in the queue is finished. I'd like to write out the result of fac(n)
    > > for each item in the queue as it happens.

    >
    > > I'm probably approaching the problem all wrong - can anyone set me on
    > > the right track?

    >
    > Works fine for me.  Formatting time.clock(), with a different range:
    >
    > 0.00000 fac(25000) done on Process-2
    > 0.09334 fac(25001) done on Process-1
    > 2.25036 fac(25002) done on Process-2
    > 2.41227 fac(25003) done on Process-1
    > 3.57167 fac(25004) done on Process-2
    >
    > I'm on Win32.


    Strange. I was on Win32 (Cygwin) earlier today and it was doing what I
    reported above. At home on Linux, it works as I wanted and you
    experienced. Must be something screwy with running it in Cygwin.

    Thanks guys!
     
    nhwarriors, Oct 11, 2008
    #4
  5. On Oct 10, 10:48 pm, nhwarriors <> wrote:
    > On Oct 10, 10:52 pm, "Aaron \"Castironpi\" Brady"
    >
    >
    >
    > <> wrote:
    > > On Oct 10, 3:32 pm, nhwarriors <> wrote:

    >
    > > > I am attempting to use the (new in 2.6) multiprocessing package to
    > > > process 2 items in a large queue of items simultaneously. I'd like to
    > > > be able to print to the screen the results of each item before
    > > > starting the next one. I'm having trouble with this so far.

    >
    > > > Here is some (useless) example code that shows how far I've gotten by
    > > > reading the documentation:

    >
    > > snip code

    >
    > > > This works great, except that nothing can be output until everything
    > > > in the queue is finished. I'd like to write out the result of fac(n)
    > > > for each item in the queue as it happens.

    >
    > > > I'm probably approaching the problem all wrong - can anyone set me on
    > > > the right track?

    >
    > > Works fine for me.  Formatting time.clock(), with a different range:

    >
    > > 0.00000 fac(25000) done on Process-2
    > > 0.09334 fac(25001) done on Process-1
    > > 2.25036 fac(25002) done on Process-2
    > > 2.41227 fac(25003) done on Process-1
    > > 3.57167 fac(25004) done on Process-2

    >
    > > I'm on Win32.

    >
    > Strange. I was on Win32 (Cygwin) earlier today and it was doing what I
    > reported above. At home on Linux, it works as I wanted and you
    > experienced. Must be something screwy with running it in Cygwin.
    >
    > Thanks guys!


    You might need to flush the standard out between calls. That gets
    suggested from time to time. Not sure what Python's req.s are.
     
    Aaron \Castironpi\ Brady, Oct 11, 2008
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Volodya
    Replies:
    6
    Views:
    720
    Mark Hammond
    Feb 6, 2009
  2. Rob Newman
    Replies:
    0
    Views:
    972
    Rob Newman
    Jun 16, 2009
  3. Matt
    Replies:
    2
    Views:
    412
    Mike Kazantsev
    Jun 17, 2009
  4. mheavner
    Replies:
    10
    Views:
    484
    mheavner
    Jul 20, 2009
  5. Adam Tauno Williams
    Replies:
    0
    Views:
    401
    Adam Tauno Williams
    Jan 15, 2010
Loading...

Share This Page