Not fully understanding the role of Queue.task_done()

Discussion in 'Python' started by Martin DeMello, Sep 4, 2008.

  1. I'm writing a cluster monitor, that collects information from a set of
    machines and logs it to a database

    In the interests of not hammering the db unnecessarily, I'm
    considering the following
    1. A series of independent "monitor" threads that collect information
    over TCP from the cluster of machines, and write it to a queue
    2. A "logger" thread that empties the queue every second or so and
    inserts the collected information to the db via a single insert
    statement

    Reading up on python's built in Queue class, though, it seems oriented
    towards "job queues", with a two-step dequeue operation (get() and
    task_done()). I'm worried that this would make it too heavyweight for
    my application. Is ther documentation somewhere on what exactly
    task_done() does, and whether I can disable the tracking of a job once
    it's removed from the queue? The python docs for the Queue module were
    a bit light.

    martin
     
    Martin DeMello, Sep 4, 2008
    #1
    1. Advertising

  2. Martin DeMello wrote:

    > Reading up on python's built in Queue class, though, it seems oriented
    > towards "job queues", with a two-step dequeue operation (get() and
    > task_done()). I'm worried that this would make it too heavyweight for
    > my application. Is ther documentation somewhere on what exactly
    > task_done() does, and whether I can disable the tracking of a job once
    > it's removed from the queue? The python docs for the Queue module were
    > a bit light.


    "task_done" just decrements a counter (incremented by "put"). when the
    counter reaches zero, the "join" call is unblocked.

    </F>
     
    Fredrik Lundh, Sep 4, 2008
    #2
    1. Advertising

  3. On Sep 4, 12:41 pm, Fredrik Lundh <> wrote:

    > "task_done" just decrements a counter (incremented by "put").  when the
    > counter reaches zero, the "join" call is unblocked.


    Thanks! Is there any standard python idiom to empty a queue into a
    list? Or do I just call get() repeatedly and catch the exception when
    it's done?

    martin
     
    Martin DeMello, Sep 4, 2008
    #3
  4. Martin DeMello

    castironpi Guest

    On Sep 4, 2:51 pm, Martin DeMello <> wrote:
    > On Sep 4, 12:41 pm, Fredrik Lundh <> wrote:
    >
    > > "task_done" just decrements a counter (incremented by "put").  when the
    > > counter reaches zero, the "join" call is unblocked.

    >
    > Thanks! Is there any standard python idiom to empty a queue into a
    > list? Or do I just call get() repeatedly and catch the exception when
    > it's done?
    >
    > martin


    Random access isn't supported by the defined interface. You can make
    it more convenient, though.

    import Queue

    class IterQueue( Queue.Queue ):
    def __iter__( self ):
    return self
    def next( self ):
    if self.empty():
    raise StopIteration
    return self.get()

    q= IterQueue()
    q.put( 'a' )
    q.put( 'b' )
    q.put( 'c' )

    print [ x for x in q ]

    /Output:
    ['a', 'b', 'c']
     
    castironpi, Sep 4, 2008
    #4
  5. On Sep 4, 1:04 pm, castironpi <> wrote:
    >
    > Random access isn't supported by the defined interface.  You can make
    > it more convenient, though.


    Thanks. I wasn't looking for random access, just wondering what the
    cleanest way to implement items = Queue.get_all() was. Your code
    should work nicely for that.

    martin
     
    Martin DeMello, Sep 4, 2008
    #5
  6. Martin DeMello wrote:

    > I'm writing a cluster monitor, that collects information from a set of
    > machines and logs it to a database
    >
    > In the interests of not hammering the db unnecessarily, I'm
    > considering the following
    > 1. A series of independent "monitor" threads that collect information
    > over TCP from the cluster of machines, and write it to a queue
    > 2. A "logger" thread that empties the queue every second or so and
    > inserts the collected information to the db via a single insert
    > statement


    why are you using a queue for this case, btw? why not just use a plain list

    L = []
    lock = threading.Lock()

    and add stuff using append in the monitor threads

    with lock:
    L.append(item)

    and regularily reset the list in the logger thread

    with lock:
    data = L[:]
    L[:] = [] # clear the list
    for item in data:
    ... insert into database ...

    (list append and assignments to global variables are atomic in CPython,
    so you can eliminate the lock by being a bit clever, but that's probably
    better left for a non-premature optimization pass).

    </F>
     
    Fredrik Lundh, Sep 4, 2008
    #6
  7. On Sep 4, 1:51 pm, Fredrik Lundh <> wrote:
    > Martin DeMello wrote:
    > > I'm writing a cluster monitor, that collects information from a set of
    > > machines and logs it to a database

    >
    > > In the interests of not hammering the db unnecessarily, I'm
    > > considering the following
    > > 1. A series of independent "monitor" threads that collect information
    > > over TCP from the cluster of machines, and write it to a queue
    > > 2. A "logger" thread that empties the queue every second or so and
    > > inserts the collected information to the db via a single insert
    > > statement

    >
    > why are you using a queue for this case, btw?  why not just use a plain list
    >
    >      L = []
    >      lock = threading.Lock()


    Good point - I thought of queue because it was self-locking, but
    you're right, I can as well use a simple list and lock it myself.

    martin
     
    Martin DeMello, Sep 4, 2008
    #7
  8. Martin DeMello

    Aahz Guest

    In article <>,
    Fredrik Lundh <> wrote:
    >Martin DeMello wrote:
    >>
    >> In the interests of not hammering the db unnecessarily, I'm
    >> considering the following
    >> 1. A series of independent "monitor" threads that collect information
    >> over TCP from the cluster of machines, and write it to a queue
    >> 2. A "logger" thread that empties the queue every second or so and
    >> inserts the collected information to the db via a single insert
    >> statement

    >
    >why are you using a queue for this case, btw? why not just use a plain list
    >
    > L = []
    > lock = threading.Lock()
    >
    >and add stuff using append in the monitor threads
    >
    > with lock:
    > L.append(item)


    Because using a queue requires less thinking. I certainly would use a
    queue in this case instead of rolling my own.
    --
    Aahz () <*> http://www.pythoncraft.com/

    "Argue for your limitations, and sure enough they're yours." --Richard Bach
     
    Aahz, Sep 7, 2008
    #8
  9. Aahz wrote:

    >> why are you using a queue for this case, btw? why not just use a plain list
    >>
    >> L = []
    >> lock = threading.Lock()
    >>
    >> and add stuff using append in the monitor threads
    >>
    >> with lock:
    >> L.append(item)

    >
    > Because using a queue requires less thinking.


    given that the whole reason for this thread was that Queue API didn't
    fit the OP:s problem, that's a rather dubious statement.

    (btw, I've always thought that Python was all about making it easy to
    express the solution to a given problem in code, not to let you write
    programs without using your brain. when did that change?)

    </F>
     
    Fredrik Lundh, Sep 7, 2008
    #9
  10. Martin DeMello

    alex23 Guest

    Fredrik Lundh <> wrote:
    > (btw, I've always thought that Python was all about making it easy to
    > express the solution to a given problem in code, not to let you write
    > programs without using your brain.  when did that change?)


    The day Google App Engine was opened up to developers, I believe.
     
    alex23, Sep 8, 2008
    #10
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Jesper Stocholm
    Replies:
    2
    Views:
    8,134
    John Saunders
    Aug 23, 2003
  2. Russell Warren

    Is Queue.Queue.queue.clear() thread-safe?

    Russell Warren, Jun 22, 2006, in forum: Python
    Replies:
    4
    Views:
    691
    Russell Warren
    Jun 27, 2006
  3. Gal Aviel
    Replies:
    8
    Views:
    915
    Dennis Lee Bieber
    Feb 29, 2008
  4. Gabriel Rossetti
    Replies:
    3
    Views:
    564
    Jerry Hill
    Apr 25, 2008
  5. Kris
    Replies:
    0
    Views:
    494
Loading...

Share This Page