Threading / Queue management

Discussion in 'Python' started by Power Button, Feb 2, 2009.

  1. Power Button

    Power Button Guest

    hi there,

    I wonder if anyone can help with the following. I have written a
    script which polls a server and if it finds and pending orders, it
    instantiates an new object (foo) - in a new thread and processes some
    data. In the new object (foo), there are also some long running
    processes so I am trying to create a thread pool/queue and to push
    items onto the queue in the instantiated object (foo). Currently I am
    creating the Queue in foo but this means I am starting up 5 new
    threads every time I instantiate foo. What I want to be able to do is
    create the Queue and start 5 (n) threads when my program starts and
    then push items onto the queue in foo.

    My question is, how can I create the Queue in my main object and set
    the target function for the Thread Constructor to be a function in
    foo?

    Below is a code snippet if it will help explain more - as I am fairly
    new to Python.

    Thanks.
    M


    import os, threading, time
    import Queue
    class Foo(threading.Thread):

    def __init__(self, record):
    self.mRecord = record
    self.mQueue = None
    threading.Thread.__init__(self)

    def run(self):
    files = self.getFiles()
    self.sendFiles(files)
    #snip

    def getFiles(self):
    #snip
    pass

    def sendFiles(self, files):
    self.mQueue = Queue.Queue()

    for i in range(5):
    t = threading.Thread(target=self.doSend)
    t.setDaemon(False)
    t.start()

    for f in files:
    self.mQueue.put(['ok', f])

    self.mQueue.join()

    def doSend(self):
    flag = 'ok'
    while flag != 'stop':
    flag,item = self.mQueue.get()
    if flag == 'ok':
    self.sendIt(item)

    self.mQueue.task_done()

    def sendIt(self, item):
    #snip
    pass


    import sys, traceback, time
    from modules.foo import Foo
    from modules.bar import Bar
    class Server(object):

    def startPolling(self):
    while True:
    bar = Bar(accessKey)
    pending = None
    pending = bar.getPending()

    if pending != None:
    self. processRecord(bar)
    time.sleep(10)


    def processRecord(self, record):
    Foo(record).start()



    def main():
    server = Server()
    server.startPolling()

    if __name__ == '__main__':
    main()
    Power Button, Feb 2, 2009
    #1
    1. Advertising

  2. Power Button

    Steve Holden Guest

    Power Button wrote:
    > hi there,
    >
    > I wonder if anyone can help with the following. I have written a
    > script which polls a server and if it finds and pending orders, it
    > instantiates an new object (foo) - in a new thread and processes some
    > data. In the new object (foo), there are also some long running
    > processes so I am trying to create a thread pool/queue and to push
    > items onto the queue in the instantiated object (foo). Currently I am
    > creating the Queue in foo but this means I am starting up 5 new
    > threads every time I instantiate foo. What I want to be able to do is
    > create the Queue and start 5 (n) threads when my program starts and
    > then push items onto the queue in foo.
    >
    > My question is, how can I create the Queue in my main object and set
    > the target function for the Thread Constructor to be a function in
    > foo?
    >

    This might help.

    http://www.chrisarndt.de/projects/threadpool/

    If not, Google around for "Python threadpool". If that doesn't help
    either, come back to the list.

    regards
    Steve
    --
    Steve Holden +1 571 484 6266 +1 800 494 3119
    Holden Web LLC http://www.holdenweb.com/
    Steve Holden, Feb 2, 2009
    #2
    1. Advertising

  3. "Power Button" <>

    > My question is, how can I create the Queue in my main object and set
    > the target function for the Thread Constructor to be a function in
    > foo?


    Just create it, giving it some name.
    Then start the static long running stuff,
    and pass the name you gave it.

    Pass the name to the thing that starts the
    per transaction thread, so that it can pass
    it to the new thread, and away you go.

    You may even need an output queue, so that
    the static long running stuff knows what to do
    with the results, and a thread to handle the
    final output.

    - Hendrik
    Hendrik van Rooyen, Feb 3, 2009
    #3
  4. Power Button

    Aahz Guest

    In article <>,
    Power Button <> wrote:
    >
    >I wonder if anyone can help with the following. I have written a
    >script which polls a server and if it finds and pending orders, it
    >instantiates an new object (foo) - in a new thread and processes some
    >data. In the new object (foo), there are also some long running
    >processes so I am trying to create a thread pool/queue and to push
    >items onto the queue in the instantiated object (foo). Currently I am
    >creating the Queue in foo but this means I am starting up 5 new
    >threads every time I instantiate foo. What I want to be able to do is
    >create the Queue and start 5 (n) threads when my program starts and
    >then push items onto the queue in foo.


    Not sure what you're trying to do, but you meay find this sample code
    useful:

    http://www.pythoncraft.com/OSCON2001/index.html
    --
    Aahz () <*> http://www.pythoncraft.com/

    Weinberg's Second Law: If builders built buildings the way programmers wrote
    programs, then the first woodpecker that came along would destroy civilization.
    Aahz, Feb 9, 2009
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. pouet
    Replies:
    2
    Views:
    738
    Will Hartung
    Jul 30, 2004
  2. Paul L. Du Bois

    Queue.Queue-like class without the busy-wait

    Paul L. Du Bois, Mar 24, 2005, in forum: Python
    Replies:
    29
    Views:
    1,039
    Antoon Pardon
    Apr 4, 2005
  3. Russell Warren

    Is Queue.Queue.queue.clear() thread-safe?

    Russell Warren, Jun 22, 2006, in forum: Python
    Replies:
    4
    Views:
    667
    Russell Warren
    Jun 27, 2006
  4. Kceiw
    Replies:
    3
    Views:
    982
    Jim Langston
    Mar 14, 2006
  5. Kris
    Replies:
    0
    Views:
    466
Loading...

Share This Page