Good thread pool module

Discussion in 'Python' started by David Hirschfield, Mar 22, 2006.

  1. There isn't a thread pool module in the standard library, but I'm sure
    many have been written by people in the python community.
    Anyone have a favorite? Is there one particular implementation that's
    recommended?

    Not looking for anything fancy, just something that lets me queue up
    tasks to be performed by a pool of threads and then retrieve results
    when the tasks complete.

    Thanks,
    -Dave

    --
    Presenting:
    mediocre nebula.
    David Hirschfield, Mar 22, 2006
    #1
    1. Advertising

  2. David Hirschfield

    Rune Hansen Guest

    I think you want the Queue module in standard lib. Haven't started a
    thread yet without it :)

    regards
    /rune
    Rune Hansen, Mar 22, 2006
    #2
    1. Advertising

  3. David Hirschfield

    Steve M Guest

    I believe "Python in a Nutshell" has a couple of clear examples using
    Queue and Threading, including one with a pool of worker threads that
    wait for entries in one queue and place results in another.

    Also you should look at the Python Cookbook, which probably includes
    the same or similar examples from the Nutshell book, since the author
    of that is an editor of Cookbook.

    http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Threads
    Steve M, Mar 22, 2006
    #3
  4. David Hirschfield wrote:
    > There isn't a thread pool module in the standard library, but I'm sure
    > many have been written by people in the python community.
    > Anyone have a favorite? Is there one particular implementation that's
    > recommended?


    Because of the GIL, thread pools are not as useful in Python as you
    might expect -- they execute one at a time and do not take advantage of
    hyper-threading or multiple processors. If that kind of efficiency is
    what you are after, then take a look at PyLinda which can coordinate
    communication between multiple instances of Python running in separate
    threads:

    http://www-users.cs.york.ac.uk/~aw/pylinda/

    > Not looking for anything fancy, just something that lets me queue up
    > tasks to be performed by a pool of threads and then retrieve results
    > when the tasks complete.


    FWIW, I wrote a small enchancement to the Queue module that makes it a
    little bit easier to use a pool of worker threads and then be able to
    tell when they are all done with processing:

    http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/475160


    Raymond
    Raymond Hettinger, Mar 23, 2006
    #4
  5. On 22 Mar 2006 21:35:28 -0800, "Raymond Hettinger" <>
    declaimed the following in comp.lang.python:

    > Because of the GIL, thread pools are not as useful in Python as you
    > might expect -- they execute one at a time and do not take advantage of
    > hyper-threading or multiple processors. If that kind of efficiency is


    If the task is I/O bound (something like a web spider?), seems
    they'd still be useful...
    >
    > FWIW, I wrote a small enchancement to the Queue module that makes it a


    Pardon my levity, but was that an enhancement, or an enchantment <G>

    --
    > ============================================================== <
    > | Wulfraed Dennis Lee Bieber KD6MOG <
    > | Bestiaria Support Staff <
    > ============================================================== <
    > Home Page: <http://www.dm.net/~wulfraed/> <
    > Overflow Page: <http://wlfraed.home.netcom.com/> <
    Dennis Lee Bieber, Mar 23, 2006
    #5
  6. David Hirschfield

    Rene Pijlman Guest

    Dennis Lee Bieber:
    >Raymond Hettinger:
    >> Because of the GIL, thread pools are not as useful in Python as you
    >> might expect -- they execute one at a time and do not take advantage of
    >> hyper-threading or multiple processors. If that kind of efficiency is

    >
    > If the task is I/O bound (something like a web spider?), seems
    >they'd still be useful...


    Yes, I use a thread pool for that. But async I/O may be a more efficient
    solution (e.g. Twisted).

    --
    René Pijlman

    Wat wil jij leren? http://www.leren.nl
    Rene Pijlman, Mar 23, 2006
    #6
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Hugo
    Replies:
    4
    Views:
    1,711
    Logan Shaw
    Mar 27, 2008
  2. testisok
    Replies:
    0
    Views:
    299
    testisok
    Feb 17, 2009
  3. Rick Lawson
    Replies:
    8
    Views:
    774
    Graham Dumpleton
    Jul 17, 2009
  4. Glazner
    Replies:
    0
    Views:
    334
    Glazner
    Jan 6, 2010
  5. Navin Mishra
    Replies:
    0
    Views:
    181
    Navin Mishra
    Mar 22, 2005
Loading...

Share This Page