Using multiprocessing

N

nhwarriors

I am attempting to use the (new in 2.6) multiprocessing package to
process 2 items in a large queue of items simultaneously. I'd like to
be able to print to the screen the results of each item before
starting the next one. I'm having trouble with this so far.

Here is some (useless) example code that shows how far I've gotten by
reading the documentation:

from multiprocessing import Process, Queue, current_process

def main():
facs = []
for i in range(50000,50005):
facs.append(i)

tasks = [(fac, (i,)) for i in facs]
task_queue = Queue()
done_queue = Queue()

for task in tasks:
task_queue.put(task)

for i in range(2):
Process(target = worker, args = (task_queue, done_queue)).start()

for i in range(len(tasks)):
print done_queue.get()

for i in range(2):
task_queue.put('STOP')

def worker(input, output):
for func, args in iter(input.get, 'STOP'):
result = func(*args)
output.put(result)

def fac(n):
f = n
for i in range(n-1,1,-1):
f *= i
return 'fac('+str(n)+') done on '+current_process().name

if __name__ == '__main__':
main()

This works great, except that nothing can be output until everything
in the queue is finished. I'd like to write out the result of fac(n)
for each item in the queue as it happens.

I'm probably approaching the problem all wrong - can anyone set me on
the right track?
 
J

Jesse Noller

I am attempting to use the (new in 2.6) multiprocessing package to
process 2 items in a large queue of items simultaneously. I'd like to
be able to print to the screen the results of each item before
starting the next one. I'm having trouble with this so far.

Here is some (useless) example code that shows how far I've gotten by
reading the documentation:

from multiprocessing import Process, Queue, current_process

def main():
facs = []
for i in range(50000,50005):
facs.append(i)

tasks = [(fac, (i,)) for i in facs]
task_queue = Queue()
done_queue = Queue()

for task in tasks:
task_queue.put(task)

for i in range(2):
Process(target = worker, args = (task_queue, done_queue)).start()

for i in range(len(tasks)):
print done_queue.get()

for i in range(2):
task_queue.put('STOP')

def worker(input, output):
for func, args in iter(input.get, 'STOP'):
result = func(*args)
output.put(result)

def fac(n):
f = n
for i in range(n-1,1,-1):
f *= i
return 'fac('+str(n)+') done on '+current_process().name

if __name__ == '__main__':
main()

This works great, except that nothing can be output until everything
in the queue is finished. I'd like to write out the result of fac(n)
for each item in the queue as it happens.

I'm probably approaching the problem all wrong - can anyone set me on
the right track?

I'm not quite following: If you run this, the results are printed by
the main thread, unordered, as they are put on the results queue -
this works as intended (and the example this is based on works the
same way) .

For example:
result put Process-2
result put Process-1
fac(50000) done on Process-1
result put Process-2
fac(50001) done on Process-2
result put Process-1
fac(50003) done on Process-1
result put Process-2
fac(50002) done on Process-2
fac(50004) done on Process-2

You can see this if you expand the range:

result put Process-1
result put Process-2
result put Process-2
fac(50001) done on Process-2
result put Process-1
fac(50000) done on Process-1
fac(50003) done on Process-2
result put Process-2
result put Process-1
fac(50004) done on Process-2
result put Process-2
fac(50006) done on Process-2
result put Process-1
fac(50002) done on Process-1
fac(50005) done on Process-1
fac(50007) done on Process-1
result put Process-2
result put Process-1
fac(50008) done on Process-2
result put Process-2
result put Process-1
fac(50010) done on Process-2
result put Process-2
result put Process-1
fac(50009) done on Process-1
fac(50011) done on Process-1
fac(50013) done on Process-1
result put Process-2
fac(50012) done on Process-2
fac(50014) done on Process-2

One trick I use is when I have a results queue to manage, I spawn an
addition process to read off of the results queue and deal with the
results. This is mainly so I can process the results outside of the
main thread, as they appear on the results queue

-jesse
 
A

Aaron \Castironpi\ Brady

I am attempting to use the (new in 2.6) multiprocessing package to
process 2 items in a large queue of items simultaneously. I'd like to
be able to print to the screen the results of each item before
starting the next one. I'm having trouble with this so far.

Here is some (useless) example code that shows how far I've gotten by
reading the documentation:
snip code

This works great, except that nothing can be output until everything
in the queue is finished. I'd like to write out the result of fac(n)
for each item in the queue as it happens.

I'm probably approaching the problem all wrong - can anyone set me on
the right track?

Works fine for me. Formatting time.clock(), with a different range:

0.00000 fac(25000) done on Process-2
0.09334 fac(25001) done on Process-1
2.25036 fac(25002) done on Process-2
2.41227 fac(25003) done on Process-1
3.57167 fac(25004) done on Process-2

I'm on Win32.
 
N

nhwarriors

snip code



Works fine for me.  Formatting time.clock(), with a different range:

0.00000 fac(25000) done on Process-2
0.09334 fac(25001) done on Process-1
2.25036 fac(25002) done on Process-2
2.41227 fac(25003) done on Process-1
3.57167 fac(25004) done on Process-2

I'm on Win32.

Strange. I was on Win32 (Cygwin) earlier today and it was doing what I
reported above. At home on Linux, it works as I wanted and you
experienced. Must be something screwy with running it in Cygwin.

Thanks guys!
 
A

Aaron \Castironpi\ Brady

Strange. I was on Win32 (Cygwin) earlier today and it was doing what I
reported above. At home on Linux, it works as I wanted and you
experienced. Must be something screwy with running it in Cygwin.

Thanks guys!

You might need to flush the standard out between calls. That gets
suggested from time to time. Not sure what Python's req.s are.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,818
Messages
2,569,736
Members
45,709
Latest member
DRISenaida

Latest Threads

Top