Too many threads

[mark.pelletier]

For some reason, the tasks I put into my thread pool occasionally get
run more than once.

Here's the code:

#
-------------------------------------------------------------------------------------------------------------------
from threading import Thread
from queue import Queue
import subprocess


class ThreadPool(object):
def __init__(self, thread_count):
'''
Argument thread_count is the maximum number of theads.
'''
self.thread_count = thread_count
self.queue = Queue()
# create and start the threads
for i in range(self.thread_count):
t = Thread(target=self._worker)
t.daemon = True
t.start()

def _worker(self):
'''
A "private" method that pulls tasks off the queue and does
something
with them.
'''
while True:
item = self.queue.get()
print(item)
self.queue.task_done()

def run(self, tasklist):
'''
Put tasks in the queue.
'''
for item in tasklist:
self.queue.put(item)
# block until all threads are done.
self.queue.join()
# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
-
tasklist = [ 'task1', 'task2', 'task3', 'task4', 'task5', 'task6',
'task7',
'task8', 'task9', 'task10', 'task11', 'task12', 'task13', 'task14',
'task15',
'task16']

if __name__ == '__main__':

t = ThreadPool(3)
t.run(tasklist)

#---------------------------------------------------------------------------------------------------

And here's some typical output:

task1
task2
task2
task3
task4
task5
task5
task6
task7
task8
task8
task9
task8
task9
task10
task11
task11
task12
task13
task13
task14
task15
task15
task16


I only want a task to get fired off once. What am I doing wrong?

Thanks!

 

[Cameron Simpson]

| For some reason, the tasks I put into my thread pool occasionally get
| run more than once.
|
| Here's the code:

You need to post your _exact_ code. I had to change:
from queue import Queue
into
from Queue import Queue
So: _do_ you have a "queue" (lowercase) module? Is it buggy?

Let's pretend not.

Using the standard Queue module I can't reproduce your problem and your
code looks correct to my eye.

However, it is possible that the output stream used by print() is not
thread safe. In which case your multiple outputs may stem from output
buffer mishandling and not from your thread pool.

Make another Queue to receice items, change _worker() to .put(item) onto
it and collect form the queue and print in the program's main code. That
way only one thread (the main one) will be trying to print(), removing the
possibility of mismanagement of the output if print() is not thread safe.

Cheers,
--
Cameron Simpson <[email protected]> DoD#743
http://www.cskk.ezoshosting.com/cs/

Rules of Optimization:
Rule 1: Don't do it.
Rule 2 (for experts only): Don't do it yet.
- M.A. Jackson

 

[Ned Deily]

On 16Sep2010 09:55, (e-mail address removed) <[email protected]>
wrote:
| For some reason, the tasks I put into my thread pool occasionally get
| run more than once.
|
| Here's the code:

You need to post your _exact_ code. I had to change:
from queue import Queue
into
from Queue import Queue
So: _do_ you have a "queue" (lowercase) module? Is it buggy?

The OP is probably using Python 3.