D
DPalao
Hello,
I'm trying to use multiprocessing to parallelize a code. There is a number of
tasks (usually 12) that can be run independently. Each task produces a numpy
array, and at the end, those arrays must be combined.
I implemented this using Queues (multiprocessing.Queue): one for input and
another for output.
But the code blocks. And it must be related to the size of the item I put on
the Queue: if I put a small array, the code works well; if the array is
realistically large (in my case if can vary from 160kB to 1MB), the code
blocks apparently forever.
I have tried this:
http://www.bryceboe.com/2011/01/28/the-python-multiprocessing-queue-and-large-
objects/
but it didn't work (especifically I put a None sentinel at the end for each
worker).
Before I change the implementation,
is there a way to bypass this problem with multiprocessing.Queue?
Should I post the code (or a sketchy version of it)?
TIA,
David
I'm trying to use multiprocessing to parallelize a code. There is a number of
tasks (usually 12) that can be run independently. Each task produces a numpy
array, and at the end, those arrays must be combined.
I implemented this using Queues (multiprocessing.Queue): one for input and
another for output.
But the code blocks. And it must be related to the size of the item I put on
the Queue: if I put a small array, the code works well; if the array is
realistically large (in my case if can vary from 160kB to 1MB), the code
blocks apparently forever.
I have tried this:
http://www.bryceboe.com/2011/01/28/the-python-multiprocessing-queue-and-large-
objects/
but it didn't work (especifically I put a None sentinel at the end for each
worker).
Before I change the implementation,
is there a way to bypass this problem with multiprocessing.Queue?
Should I post the code (or a sketchy version of it)?
TIA,
David