Threading / Queue management

[Power Button]

hi there,

I wonder if anyone can help with the following. I have written a
script which polls a server and if it finds and pending orders, it
instantiates an new object (foo) - in a new thread and processes some
data. In the new object (foo), there are also some long running
processes so I am trying to create a thread pool/queue and to push
items onto the queue in the instantiated object (foo). Currently I am
creating the Queue in foo but this means I am starting up 5 new
threads every time I instantiate foo. What I want to be able to do is
create the Queue and start 5 (n) threads when my program starts and
then push items onto the queue in foo.

My question is, how can I create the Queue in my main object and set
the target function for the Thread Constructor to be a function in
foo?

Below is a code snippet if it will help explain more - as I am fairly
new to Python.

Thanks.
M


import os, threading, time
import Queue
class Foo(threading.Thread):

def __init__(self, record):
self.mRecord = record
self.mQueue = None
threading.Thread.__init__(self)

def run(self):
files = self.getFiles()
self.sendFiles(files)
#snip

def getFiles(self):
#snip
pass

def sendFiles(self, files):
self.mQueue = Queue.Queue()

for i in range(5):
t = threading.Thread(target=self.doSend)
t.setDaemon(False)
t.start()

for f in files:
self.mQueue.put(['ok', f])

self.mQueue.join()

def doSend(self):
flag = 'ok'
while flag != 'stop':
flag,item = self.mQueue.get()
if flag == 'ok':
self.sendIt(item)

self.mQueue.task_done()

def sendIt(self, item):
#snip
pass


import sys, traceback, time
from modules.foo import Foo
from modules.bar import Bar
class Server(object):

def startPolling(self):
while True:
bar = Bar(accessKey)
pending = None
pending = bar.getPending()

if pending != None:
self. processRecord(bar)
time.sleep(10)


def processRecord(self, record):
Foo(record).start()



def main():
server = Server()
server.startPolling()

if __name__ == '__main__':
main()

 

[Steve Holden]

hi there,

I wonder if anyone can help with the following. I have written a
script which polls a server and if it finds and pending orders, it
instantiates an new object (foo) - in a new thread and processes some
data. In the new object (foo), there are also some long running
processes so I am trying to create a thread pool/queue and to push
items onto the queue in the instantiated object (foo). Currently I am
creating the Queue in foo but this means I am starting up 5 new
threads every time I instantiate foo. What I want to be able to do is
create the Queue and start 5 (n) threads when my program starts and
then push items onto the queue in foo.

My question is, how can I create the Queue in my main object and set
the target function for the Thread Constructor to be a function in
foo?

This might help.

http://www.chrisarndt.de/projects/threadpool/

If not, Google around for "Python threadpool". If that doesn't help
either, come back to the list.

regards
Steve

 

[Hendrik van Rooyen]

My question is, how can I create the Queue in my main object and set
the target function for the Thread Constructor to be a function in
foo?

Just create it, giving it some name.
Then start the static long running stuff,
and pass the name you gave it.

Pass the name to the thing that starts the
per transaction thread, so that it can pass
it to the new thread, and away you go.

You may even need an output queue, so that
the static long running stuff knows what to do
with the results, and a thread to handle the
final output.

- Hendrik

 

[Aahz]

I wonder if anyone can help with the following. I have written a
script which polls a server and if it finds and pending orders, it
instantiates an new object (foo) - in a new thread and processes some
data. In the new object (foo), there are also some long running
processes so I am trying to create a thread pool/queue and to push
items onto the queue in the instantiated object (foo). Currently I am
creating the Queue in foo but this means I am starting up 5 new
threads every time I instantiate foo. What I want to be able to do is
create the Queue and start 5 (n) threads when my program starts and
then push items onto the queue in foo.

Not sure what you're trying to do, but you meay find this sample code
useful:

http://www.pythoncraft.com/OSCON2001/index.html