How to queue each loop iteration into a thread pool, with a max size?

[David Karr]

I have a loop nested inside another loop. Each iteration of the inner loopis independent, so I could conceivably create a thread for each iteration.However, I would think that would create too many threads (couple hundred) and probably make my box thrash before it dies.

Is there a practical way to make it queue N loop iterations and then block on the N+1st until a slot is available? If I was doing this in Java I would use a ThreadPoolExecutor, but I'm not aware of a similar mechanism in Perl (not already written as a reusable library, that is).

 

[ccc31807]

I have a loop nested inside another loop.  Each iteration of the inner loop is independent, so I could conceivably create a thread for each iteration.

If the inner loop is independent of the outer loop, why loop them?
Obviously, you don't mean 'independent' in the usual sense, so what
I'm understanding you to say that there are no shared variables
between the two loops, but the environment of the inner loop depends
on the state of the outer loop.

One simple minded example that comes to me is averaging the grades of
students in a class -- you want to loop through the students one at a
time, but the students don't share state with each other. In such a
case, I could see what you might think of as an 'outer loop' given the
task of opening a file, reading the values line by line, packing each
line in an array (or a reference to an array), and calling a separate
Perl script as the 'inner loop', passing the array or reference to the
caller, or maybe writing independently to a file.

Is there a practical way to make it queue N loop iterations and then block on the N+1st until a slot is available?  If I was doing this in Java I would use a ThreadPoolExecutor, but I'm not aware of a similar mechanism inPerl (not already written as a reusable library, that is).

If the inner loop is independent of the outer loop, why would you want
the outer loop to block? Could you give a simple example of your
problem?

CC.

 

[Willem]

David Karr wrote:
) I have a loop nested inside another loop. Each iteration of the inner
) loop is independent, so I could conceivably create a thread for each
) iteration. However, I would think that would create too many threads
) (couple hundred) and probably make my box thrash before it dies.
)
) Is there a practical way to make it queue N loop iterations and then
) block on the N+1st until a slot is available? If I was doing this in
) Java I would use a ThreadPoolExecutor, but I'm not aware of a similar
) mechanism in Perl (not already written as a reusable library, that is).

You mean like Thread:ool ?


SaSW, Willem
--
Disclaimer: I am in no way responsible for any of the statements
made in the above text. For all I know I might be
drugged or something..
No I'm not paranoid. You all think I'm paranoid, don't you !
#EOT