J
James R. Saker Jr.
I've got a: "start server thread > Queue object, start server thread <>
Queue object, start parsing client < Queue object" application that's
got me puzzled. Probably an easy threads issue, but after digging thru
Programming Python and Python Recipes sections on Threading class and
running thru the examples, I'm still missing something.
My Server/Server/Client app is a syslog collector app (syslog input in,
zope control web interface to manage ala start/stop/status, and BEEPy
out) - two servers and one client app that has got me somewhat puzzled
per how to handle with threads. The app:
1. read in config file
2. launch a syslog server to listen on port 514, taking input and
putting it into a database-persistent Queue (I'm using BDB's DB_QUEUE
implemented in a class Logfile which I've created to allow me to:
l = Logfile(db_filename, db_recsize) ## BSD DB's Queue has a fixed
record length limitation which gets set by db.set_re_len
and then
l.push("Syslog data.....")
l.push("More syslog data...")
as well as l.pop(), l.hasrec() and l.size() methods to control
the Queue.
3. runs a test message to see if I've launched the first server and am
ready to do more work (this is where the second server and client will
come in once I'm crawling along).
The problem I run into is that I'm apparently not threading the syslog
server and returning control to my app:
class Syslogd(ThreadingUDPServer, InterruptibleServer):
is based on:
http://www.drbeat.li/pycgi/webutil.py/html?py=py/syslogd.py.txt
with modifications to reference my Berkeley DB Queue, instead of
displaying to sys.stdout.write. So far, so good - I'm logging and
writing to the database (though it appears to be committing on the
db.close() rather than writes, but that's another issue I'll have to
deal with via the BSD DB C/Java docs since the Python docs on it's Queue
method are limited - I've looked at ZODB for this as well).
Here's where I get into trouble, and as mentioned at the beginning, is
probably just a lack of me getting threads:
if __name__ == '__main__':
try:
log = Logfile('syslog.db',255) # create BSD DB Queue object
syslogd = Syslogd(log, timefmt='%H:%M:%S') #create syslog object
syslogd.serve() #start serving syslog input on port
print "This is a test"
except KeyboardInterrupt:
print "Closing queue database..."
log.close()
print "Operation canceled by user."
I never get to the test print, as once I'm in syslog.serve(), I'm there
until I quit. Maybe I'm missing the logic here completely - if I want to
share access to Logfile between syslogd object and two other objects, am
I on the wrong track?
Eventually, after threading syslog.serve(), I need to do the same with
the Zope HTTP service (which allows control to the application, much
like HTTP to a Linksys router for status, configuration, start, stop
functions), and also launching the BEEPy parsing of data from the Queue
which gets passed onto a database upstream. Talking BEEP is a must due
to the firewalls involved, so no distributed object approach will help
here. A "syslogd in, dump to queue backed by database in case device
gets shut off before it can parse and send, and a parse & send via BEEP
engine" model.
Maybe there's a better example out there of:
1. start server dumping to a shared object
2. start another server accessing a shared object
3. start a client processing the shared object
model someone is aware of - ala Queue?
Thanks much...
Jamie
"So much to learn, so little caffeine!"
Queue object, start parsing client < Queue object" application that's
got me puzzled. Probably an easy threads issue, but after digging thru
Programming Python and Python Recipes sections on Threading class and
running thru the examples, I'm still missing something.
My Server/Server/Client app is a syslog collector app (syslog input in,
zope control web interface to manage ala start/stop/status, and BEEPy
out) - two servers and one client app that has got me somewhat puzzled
per how to handle with threads. The app:
1. read in config file
2. launch a syslog server to listen on port 514, taking input and
putting it into a database-persistent Queue (I'm using BDB's DB_QUEUE
implemented in a class Logfile which I've created to allow me to:
l = Logfile(db_filename, db_recsize) ## BSD DB's Queue has a fixed
record length limitation which gets set by db.set_re_len
and then
l.push("Syslog data.....")
l.push("More syslog data...")
as well as l.pop(), l.hasrec() and l.size() methods to control
the Queue.
3. runs a test message to see if I've launched the first server and am
ready to do more work (this is where the second server and client will
come in once I'm crawling along).
The problem I run into is that I'm apparently not threading the syslog
server and returning control to my app:
class Syslogd(ThreadingUDPServer, InterruptibleServer):
is based on:
http://www.drbeat.li/pycgi/webutil.py/html?py=py/syslogd.py.txt
with modifications to reference my Berkeley DB Queue, instead of
displaying to sys.stdout.write. So far, so good - I'm logging and
writing to the database (though it appears to be committing on the
db.close() rather than writes, but that's another issue I'll have to
deal with via the BSD DB C/Java docs since the Python docs on it's Queue
method are limited - I've looked at ZODB for this as well).
Here's where I get into trouble, and as mentioned at the beginning, is
probably just a lack of me getting threads:
if __name__ == '__main__':
try:
log = Logfile('syslog.db',255) # create BSD DB Queue object
syslogd = Syslogd(log, timefmt='%H:%M:%S') #create syslog object
syslogd.serve() #start serving syslog input on port
print "This is a test"
except KeyboardInterrupt:
print "Closing queue database..."
log.close()
print "Operation canceled by user."
I never get to the test print, as once I'm in syslog.serve(), I'm there
until I quit. Maybe I'm missing the logic here completely - if I want to
share access to Logfile between syslogd object and two other objects, am
I on the wrong track?
Eventually, after threading syslog.serve(), I need to do the same with
the Zope HTTP service (which allows control to the application, much
like HTTP to a Linksys router for status, configuration, start, stop
functions), and also launching the BEEPy parsing of data from the Queue
which gets passed onto a database upstream. Talking BEEP is a must due
to the firewalls involved, so no distributed object approach will help
here. A "syslogd in, dump to queue backed by database in case device
gets shut off before it can parse and send, and a parse & send via BEEP
engine" model.
Maybe there's a better example out there of:
1. start server dumping to a shared object
2. start another server accessing a shared object
3. start a client processing the shared object
model someone is aware of - ala Queue?
Thanks much...
Jamie
"So much to learn, so little caffeine!"