Pool module -- does map pre-assign tasks to processes?

L

Luca

I would like to use the Pool module, but my tasks take sometimes
unpredictably different time to complete. The simplest way to write
the code would be to put all task descriptions in an array, then call

p = Pool(8)
p.map(f, a)

But I don't want to preassign process 0 with elements a[0], a[8], a
[16], process 1 with elements a[1], a[9], a[17], and so forth.
Rather, I would like all tasks to be put into a queue, and I would
like the processes to each time grab the next task to be done, and do
it. This would ensure fairly equal loading.

My question is: does the map() method of Pool pre-assign which task
gets done by each process, or is this done at runtime, on a get-first-
task-to-be-done basis?

Many thanks,

Luca
 
A

Aahz

I would like to use the Pool module, but my tasks take sometimes
unpredictably different time to complete. The simplest way to write
the code would be to put all task descriptions in an array, then call

p = Pool(8)
p.map(f, a)

But I don't want to preassign process 0 with elements a[0], a[8], a
[16], process 1 with elements a[1], a[9], a[17], and so forth.
Rather, I would like all tasks to be put into a queue, and I would
like the processes to each time grab the next task to be done, and do
it. This would ensure fairly equal loading.

My question is: does the map() method of Pool pre-assign which task
gets done by each process, or is this done at runtime, on a get-first-
task-to-be-done basis?

You can probably get what you want by setting chunksize to 1.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,903
Latest member
orderPeak8CBDGummies

Latest Threads

Top