J
Justin Chiu
Hi all,
What is the best approach to writing a concurrent daemon that can
execute callbacks for different types of events (AMQP messages, parsed
output of a subprocess, HTTP requests)?
I am considering [twisted][1], the built-in [threading][2] module, and
[greenlet][3]. I must admit that I am very unfamiliar with concurrent
programming and Python programming in general (formerly a data analysis
driven procedural programmer). Any resources on threaded/concurrent
programming (specifically daemons...not just multi-threading a single
task) would be much appreciated.
Thank you.
Details:
1) Listens into AMQP messaging queues and executes callbacks when
messages arrive.
Example: Immediately after startup, the daemon continuously listens to
the [Openstack Notifications messaging queue][4]. When a virtual machine
is launched, a notification is generated by Openstack with the hostname,
IP address, etc. The daemon should read this message and write some info
to a log (or POST the info to a server, or notify the user...something
simple).
2) Parse the output of a subprocess and execute callbacks based on the
output.
Example: Every 30 seconds, a system command "[qstat][5]" is run to query
a job resource manager (e.g. TORQUE). Similar callbacks to 1).
3) Receive requests from a user and process them. I think this will be
via WSGI HTTP.
Example: User submits an XML template with virtual machine templates.
The daemon does some simple XML parsing and writes a job script for the
job resource manager. The job is submitted to the resource manager and
the daemon continually checks for the status of the job with "qstat" and
for messages from AMQP. It should return "live" feedback to the user and
write to a log.
[1]: https://twistedmatrix.com/trac/wiki/Documentation
[2]: http://docs.python.org/2/library/threading.html
[3]: http://greenlet.readthedocs.org/en/latest/
[4]:
https://wiki.openstack.org/wiki/NotificationEventExamples#Immediate_Notifications:
[5]: http://www.clusterresources.com/torquedocs21/commands/qstat.shtml
Justin Chiu TRIUMF
What is the best approach to writing a concurrent daemon that can
execute callbacks for different types of events (AMQP messages, parsed
output of a subprocess, HTTP requests)?
I am considering [twisted][1], the built-in [threading][2] module, and
[greenlet][3]. I must admit that I am very unfamiliar with concurrent
programming and Python programming in general (formerly a data analysis
driven procedural programmer). Any resources on threaded/concurrent
programming (specifically daemons...not just multi-threading a single
task) would be much appreciated.
Thank you.
Details:
1) Listens into AMQP messaging queues and executes callbacks when
messages arrive.
Example: Immediately after startup, the daemon continuously listens to
the [Openstack Notifications messaging queue][4]. When a virtual machine
is launched, a notification is generated by Openstack with the hostname,
IP address, etc. The daemon should read this message and write some info
to a log (or POST the info to a server, or notify the user...something
simple).
2) Parse the output of a subprocess and execute callbacks based on the
output.
Example: Every 30 seconds, a system command "[qstat][5]" is run to query
a job resource manager (e.g. TORQUE). Similar callbacks to 1).
3) Receive requests from a user and process them. I think this will be
via WSGI HTTP.
Example: User submits an XML template with virtual machine templates.
The daemon does some simple XML parsing and writes a job script for the
job resource manager. The job is submitted to the resource manager and
the daemon continually checks for the status of the job with "qstat" and
for messages from AMQP. It should return "live" feedback to the user and
write to a log.
[1]: https://twistedmatrix.com/trac/wiki/Documentation
[2]: http://docs.python.org/2/library/threading.html
[3]: http://greenlet.readthedocs.org/en/latest/
[4]:
https://wiki.openstack.org/wiki/NotificationEventExamples#Immediate_Notifications:
[5]: http://www.clusterresources.com/torquedocs21/commands/qstat.shtml
Justin Chiu TRIUMF