Good thread pool module

[David Hirschfield]

There isn't a thread pool module in the standard library, but I'm sure
many have been written by people in the python community.
Anyone have a favorite? Is there one particular implementation that's
recommended?

Not looking for anything fancy, just something that lets me queue up
tasks to be performed by a pool of threads and then retrieve results
when the tasks complete.

Thanks,
-Dave

 

[Rune Hansen]

I think you want the Queue module in standard lib. Haven't started a
thread yet without it

regards
/rune

 

[Steve M]

I believe "Python in a Nutshell" has a couple of clear examples using
Queue and Threading, including one with a pool of worker threads that
wait for entries in one queue and place results in another.

Also you should look at the Python Cookbook, which probably includes
the same or similar examples from the Nutshell book, since the author
of that is an editor of Cookbook.

http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Threads

 

[Raymond Hettinger]

There isn't a thread pool module in the standard library, but I'm sure
many have been written by people in the python community.
Anyone have a favorite? Is there one particular implementation that's
recommended?

Because of the GIL, thread pools are not as useful in Python as you
might expect -- they execute one at a time and do not take advantage of
hyper-threading or multiple processors. If that kind of efficiency is
what you are after, then take a look at PyLinda which can coordinate
communication between multiple instances of Python running in separate
threads:

http://www-users.cs.york.ac.uk/~aw/pylinda/

Not looking for anything fancy, just something that lets me queue up
tasks to be performed by a pool of threads and then retrieve results
when the tasks complete.

FWIW, I wrote a small enchancement to the Queue module that makes it a
little bit easier to use a pool of worker threads and then be able to
tell when they are all done with processing:

http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/475160


Raymond

 

[Dennis Lee Bieber]

Because of the GIL, thread pools are not as useful in Python as you
might expect -- they execute one at a time and do not take advantage of
hyper-threading or multiple processors. If that kind of efficiency is

If the task is I/O bound (something like a web spider?), seems
they'd still be useful...

FWIW, I wrote a small enchancement to the Queue module that makes it a

Pardon my levity, but was that an enhancement, or an enchantment <G>

--

 

[Rene Pijlman]

Dennis Lee Bieber:

Raymond Hettinger:

If the task is I/O bound (something like a web spider?), seems
they'd still be useful...

Yes, I use a thread pool for that. But async I/O may be a more efficient
solution (e.g. Twisted).