How to queue each loop iteration into a thread pool, with a max size?

Discussion in 'Perl Misc' started by David Karr, Apr 6, 2011.

  1. David Karr

    David Karr Guest

    I have a loop nested inside another loop. Each iteration of the inner loopis independent, so I could conceivably create a thread for each iteration.However, I would think that would create too many threads (couple hundred) and probably make my box thrash before it dies.

    Is there a practical way to make it queue N loop iterations and then block on the N+1st until a slot is available? If I was doing this in Java I would use a ThreadPoolExecutor, but I'm not aware of a similar mechanism in Perl (not already written as a reusable library, that is).
    David Karr, Apr 6, 2011
    #1
    1. Advertising

  2. David Karr

    ccc31807 Guest

    On Apr 6, 3:47 pm, David Karr <> wrote:
    > I have a loop nested inside another loop.  Each iteration of the inner loop is independent, so I could conceivably create a thread for each iteration.


    If the inner loop is independent of the outer loop, why loop them?
    Obviously, you don't mean 'independent' in the usual sense, so what
    I'm understanding you to say that there are no shared variables
    between the two loops, but the environment of the inner loop depends
    on the state of the outer loop.

    One simple minded example that comes to me is averaging the grades of
    students in a class -- you want to loop through the students one at a
    time, but the students don't share state with each other. In such a
    case, I could see what you might think of as an 'outer loop' given the
    task of opening a file, reading the values line by line, packing each
    line in an array (or a reference to an array), and calling a separate
    Perl script as the 'inner loop', passing the array or reference to the
    caller, or maybe writing independently to a file.

    > Is there a practical way to make it queue N loop iterations and then block on the N+1st until a slot is available?  If I was doing this in Java I would use a ThreadPoolExecutor, but I'm not aware of a similar mechanism inPerl (not already written as a reusable library, that is).


    If the inner loop is independent of the outer loop, why would you want
    the outer loop to block? Could you give a simple example of your
    problem?

    CC.
    ccc31807, Apr 7, 2011
    #2
    1. Advertising

  3. David Karr

    Willem Guest

    Re: How to queue each loop iteration into a thread pool, with a maxsize?

    David Karr wrote:
    ) I have a loop nested inside another loop. Each iteration of the inner
    ) loop is independent, so I could conceivably create a thread for each
    ) iteration. However, I would think that would create too many threads
    ) (couple hundred) and probably make my box thrash before it dies.
    )
    ) Is there a practical way to make it queue N loop iterations and then
    ) block on the N+1st until a slot is available? If I was doing this in
    ) Java I would use a ThreadPoolExecutor, but I'm not aware of a similar
    ) mechanism in Perl (not already written as a reusable library, that is).

    You mean like Thread::pool ?


    SaSW, Willem
    --
    Disclaimer: I am in no way responsible for any of the statements
    made in the above text. For all I know I might be
    drugged or something..
    No I'm not paranoid. You all think I'm paranoid, don't you !
    #EOT
    Willem, Apr 7, 2011
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Vladimir Davidov

    "max pool size was reached" problem again!

    Vladimir Davidov, Nov 20, 2003, in forum: ASP .Net
    Replies:
    1
    Views:
    9,110
    +The_Taco+
    Nov 20, 2003
  2. Russell Warren

    Is Queue.Queue.queue.clear() thread-safe?

    Russell Warren, Jun 22, 2006, in forum: Python
    Replies:
    4
    Views:
    681
    Russell Warren
    Jun 27, 2006
  3. =?Utf-8?B?Sm9u?=

    Timeout Expired...max pool size was reached.

    =?Utf-8?B?Sm9u?=, Oct 3, 2006, in forum: ASP .Net
    Replies:
    0
    Views:
    662
    =?Utf-8?B?Sm9u?=
    Oct 3, 2006
  4. meselfo
    Replies:
    6
    Views:
    3,350
    Cupdoo
    Sep 11, 2011
  5. Kris
    Replies:
    0
    Views:
    480
Loading...

Share This Page