Too many threads

Discussion in 'Python' started by mark.pelletier, Sep 16, 2010.

  1. For some reason, the tasks I put into my thread pool occasionally get
    run more than once.

    Here's the code:

    #
    -------------------------------------------------------------------------------------------------------------------
    from threading import Thread
    from queue import Queue
    import subprocess


    class ThreadPool(object):
    def __init__(self, thread_count):
    '''
    Argument thread_count is the maximum number of theads.
    '''
    self.thread_count = thread_count
    self.queue = Queue()
    # create and start the threads
    for i in range(self.thread_count):
    t = Thread(target=self._worker)
    t.daemon = True
    t.start()

    def _worker(self):
    '''
    A "private" method that pulls tasks off the queue and does
    something
    with them.
    '''
    while True:
    item = self.queue.get()
    print(item)
    self.queue.task_done()

    def run(self, tasklist):
    '''
    Put tasks in the queue.
    '''
    for item in tasklist:
    self.queue.put(item)
    # block until all threads are done.
    self.queue.join()
    # - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    -
    tasklist = [ 'task1', 'task2', 'task3', 'task4', 'task5', 'task6',
    'task7',
    'task8', 'task9', 'task10', 'task11', 'task12', 'task13', 'task14',
    'task15',
    'task16']

    if __name__ == '__main__':

    t = ThreadPool(3)
    t.run(tasklist)

    #---------------------------------------------------------------------------------------------------

    And here's some typical output:

    task1
    task2
    task2
    task3
    task4
    task5
    task5
    task6
    task7
    task8
    task8
    task9
    task8
    task9
    task10
    task11
    task11
    task12
    task13
    task13
    task14
    task15
    task15
    task16


    I only want a task to get fired off once. What am I doing wrong?

    Thanks!
     
    mark.pelletier, Sep 16, 2010
    #1
    1. Advertisements

  2. | For some reason, the tasks I put into my thread pool occasionally get
    | run more than once.
    |
    | Here's the code:

    You need to post your _exact_ code. I had to change:
    from queue import Queue
    into
    from Queue import Queue
    So: _do_ you have a "queue" (lowercase) module? Is it buggy?

    Let's pretend not.

    Using the standard Queue module I can't reproduce your problem and your
    code looks correct to my eye.

    However, it is possible that the output stream used by print() is not
    thread safe. In which case your multiple outputs may stem from output
    buffer mishandling and not from your thread pool.

    Make another Queue to receice items, change _worker() to .put(item) onto
    it and collect form the queue and print in the program's main code. That
    way only one thread (the main one) will be trying to print(), removing the
    possibility of mismanagement of the output if print() is not thread safe.

    Cheers,
    --
    Cameron Simpson <> DoD#743
    http://www.cskk.ezoshosting.com/cs/

    Rules of Optimization:
    Rule 1: Don't do it.
    Rule 2 (for experts only): Don't do it yet.
    - M.A. Jackson
     
    Cameron Simpson, Sep 17, 2010
    #2
    1. Advertisements

  3. mark.pelletier

    Ned Deily Guest

    The OP is probably using Python 3.
     
    Ned Deily, Sep 17, 2010
    #3
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.