web status display for long running program

B

Brian Roberts

I have a command line Python program that sometimes takes a bit
(several minutes) to run. I want to provide an optional method for an
impatient user (me!) to check the status of the program. The type and
amount of status information doesn't fit nicely into a --verbose or
logger -- either too little or too much information at different
points.

I think an optional web page would be convenient interface. The
Python program would listen on some port, and if queried (by me
browsing to localhost:12345 for example) would return a pretty status
display. Hitting reload would update the status etc.

My problem is that I'm not sure how to do this:
- I don't want to embed a full web server into the application or
require any special PC setup.
- I think I know how to listen on a socket, but not sure how to send
stuff to to a web browser -- just start with <HTML>? Or like a CGI
script with the header stuff like text/html? (I don't care if I have
to write the HTML by hand or can use a toolkit -- not important).
- Do I need a separate thread to listen and send the HTML? The
application is currently single threaded. I'm confortable with
threads, but would prefer to avoid them if possible.

Or is there a better/different way of doing this? Any general advice
or pointers to some code that already does this would be very much
appreciated.

Python 2.3, under both Linux & Windows if that makes a difference.

Thanks,
Brian.
 
J

John Lenton

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Brian Roberts wrote:
| I have a command line Python program that sometimes takes a bit
| (several minutes) to run. I want to provide an optional method for an
| impatient user (me!) to check the status of the program. The type and
| amount of status information doesn't fit nicely into a --verbose or
| logger -- either too little or too much information at different
| points.

dd and pppd (and probably others) solve this problem by registering
a signal handler that either toggles debug or prints status
information to stderr.

- --
John Lenton ([email protected]) -- Random fortune:
Work consists of whatever a body is obliged to do.
Play consists of whatever a body is not obliged to do.
-- Mark Twain
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.0 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFCHpMZcxEeCh6qRm8RAg11AKCek0hV6QCHw6fm3TM3KAJIXPb1RQCg4qhy
HMr5y+w7fxESkJ2vQ0GEmm0=
=c4bs
-----END PGP SIGNATURE-----
 
P

Peter Hansen

Brian said:
I have a command line Python program ...
I think an optional web page would be convenient interface. The
Python program would listen on some port, and if queried (by me
browsing to localhost:12345 for example) would return a pretty status
display. Hitting reload would update the status etc.

My problem is that I'm not sure how to do this:
- I don't want to embed a full web server into the application or
require any special PC setup.
- I think I know how to listen on a socket, but not sure how to send
stuff to to a web browser -- just start with <HTML>?
Click to expand...

See below. I don't care if you use the code, it was fun to
whip up on the spur of the moment. I'm quite certain that
if you try searching for "smallest web server" or something you'll
find a bunch of Perl guys who've competed to produce something
that does the same in a tenth the space...
- Do I need a separate thread to listen and send the HTML? The
application is currently single threaded. I'm confortable with
threads, but would prefer to avoid them if possible.
Click to expand...

If you don't want a thread, you probably don't need one, but
this might be a little cleaner to put in a thread. I'm not
sure exactly what approach would be best to avoid a thread,
but using select() on the server socket, with a timeout of 0
to poll, instead of blocking as the following code does, would
probably be the simplest.

Try this out and maybe you can morph it into something like
what you want...

'''miniweb: microscopic web server'''

import socket
from select import select
from datetime import datetime

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(('', 80))
s.listen(5)
while True:
cs = s.accept()[0]
try:
header = ''
while True:
r, w, e = select([cs], [], [cs], 2)
if not r or e:
break

header += cs.recv(1024)
if '\r\n\r\n' in header:
break

msg = ('<html><head><title>Miniweb!</title></head>'
'<body><h3>Clock</h3><p>The time is %s</p></body></html>' %
datetime.now())

headers = '\r\n'.join([
'HTTP/1.0 200 OK',
'Server: Miniweb-0.1',
'Content-Type: text/html',
'Connection: close',
'Content-Length: %s' % len(msg),
])

cs.send(headers + '\r\n\r\n' + msg + '\n')

finally:
cs.close()


I make no claims that this will survive any particular kind
of use or abuse. I did test with telnet, Firefox, Opera, and Explorer,
however. A couple of key points found while testing:

1. Explorer needs you to read the header in its entirety, so I
had to add the part that actually scans the header for the
terminating sequence (\r\n\r\n).

2. There's a two-second timeout on reading the header,
in case something connects but doesn't send anything... not
really tested.

3. Some or most of the headers in the reply may not be
required. Perhaps the first line and the Content-Length
(or not even that?) are all that are required. Play if you want.
No doubt others will pick this apart and we'll all learn
something. ;-)

4. MIT license ;-) as in don't complain to me if this
doesn't do what you need, but use it any way you wish...

-Peter
 
P

Paul Rubin

- I don't want to embed a full web server into the application or
require any special PC setup.
Click to expand...

That's not a big deal, just use the standard library's http server class.
- I think I know how to listen on a socket, but not sure how to send
stuff to to a web browser -- just start with <HTML>? Or like a CGI
script with the header stuff like text/html? (I don't care if I have
to write the HTML by hand or can use a toolkit -- not important).
Click to expand...

You'd have to send a full http response, but the http server class
does some of it for you.
- Do I need a separate thread to listen and send the HTML? The
application is currently single threaded. I'm confortable with
threads, but would prefer to avoid them if possible.
Click to expand...

Separate thread is the obvious way to do it.
 
K

Kamilche

Cute! Thanks for posting that. I too like the 'web interface' concept,
it has made SmoothWall a pleasure to use, even on older machines.

I was inspired to enhance your code, and perform a critical bug-fix.
Your code would not have sent large files out to dialup users, because
it assumed all data was sent on the 'send' command. I added code to
check for the number of bytes sent, and loop until it's all gone. I
also turned it into a class, and made it 'command' based.

Have fun with it, Brian!

'''
miniweb: microscopic web server by Peter Hansen

This is a simple web server that handles one request at
a time. Therefore, it's useful mainly for administrative
tasks that have one person connected at a time.

It doesn't actually serve pages, it executes commands
based on the web page you request. The commands it
currently includes are:

time - Send the time
header - Echo back the original header
largefile - Test sending a large file back to the person
quit - Shut down the web server

To add more commands, add new functions prefixed with
'request_' similar to the sample functions below.
As you add functions, the menu choices displayed to
the user are automatically updated.

I fixed the 'send' feature so it could send large files.
Before, it would send the first X bytes, then stop
without checking to make sure all bytes were went.
I also made it a class, created the 'command based'
feature, and added the auto-updating menu choices.

Have fun with it!

--Kamilche

'''

import socket
import select
import datetime

_HEADER = '\r\n'.join([
'HTTP/1.0 %s',
'Server: Miniweb-0.1',
'Content-Type: text/html',
'Connection: close',
'Content-Length: %d',
'',
'',
])

_BODY = '\r\n'.join([
'<html><head><title>%s</title></head>',
'<body><h3>%s</h3>\r\n%s\r\n</body></html>',
])

_QUIT = False


class WebServer(object):

def __init__(self, port):
' Start listening on the specified port'
self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.socket.bind(('', port))
self.socket.listen(5)

' Build the list of menu choices'
self.menu = '<pre>\n'
for key, value in self.__class__.__dict__.items():
if key[:8] == 'request_':
self.menu += "%-10s- %s\r\n" % (key[8:], value.__doc__)
self.menu += '</pre>\n'

def Timer(self):
' Process new requests'
r, w, e = select.select([self.socket], [], [], 0.0)
if not r:
return
cs = self.socket.accept()[0]
header = ''
while not '\r\n\r\n' in header:
r, w, e = select.select([cs], [], [cs], 2.0)
if not r or e:
break
header += cs.recv(1024)
status = '200 OK'
title = "Miniweb!"
if not '\r\n\r\n' in header:
status = '408 Request Timeout'
body = ['Your request was not received in time.']
else:
lines = header.split('\r\n')
words = lines[0].split(' ')
request = ' '.join(words[1:-1])
request = request[1:]
if request == '':
fn = self.default
else:
fn = getattr(self, 'request_' + request, None)
if not fn:
status = '404 Not Found'
fn = self.notfound
body = fn(header, request)

body = _BODY % (title, title,
''.join([str(arg) for arg in body]))
header = _HEADER % (status, len(body))
data = header + body
while len(data) > 0:
cc = cs.send(data)
data = data[cc:]
cs.close()

def default(self, header, request):
' Print the available choices'
return ('Welcome to Miniweb!',
'Available commands are:<p>',
self.menu)

def notfound(self, header, request):
' Handle unknown requests'
return ('Unknown request <pre>', request, '</pre>',
'Your header was:<br><pre>', header, '</pre>',
'Available commands are:<p>', self.menu)

def request_time(self, header, request):
' Send the time'
return ('The time is ', datetime.datetime.now(), '<p>')

def request_header(self, header, request):
' Echo back the original header'
return ('Your header was:<br><pre>', header, '</pre>')

def request_largefile(self, header, request):
' Test sending a large file back to the person'
temp = ['<pre>\r\n']
for i in range(10000):
temp.append("This is line %d of the result.\r\n" % i)
temp.append('</pre>')
return temp

def request_quit(self, header, request):
' Shut down the web server'
global _QUIT
_QUIT = True
return ('Web server shut down at ', datetime.datetime.now())


def main():
' Main loop'
server = WebServer(80)
while not _QUIT:
server.Timer()
print "Done!"

main()
 
P

Paul Rubin

Kamilche said:
I was inspired to enhance your code, and perform a critical bug-fix.
Your code would not have sent large files out to dialup users, because
it assumed all data was sent on the 'send' command. I added code to
check for the number of bytes sent, and loop until it's all gone. I
also turned it into a class, and made it 'command' based.
Click to expand...

That server seems to do a cpu-busy loop polling the listening socket
for incoming requests! Ouch! It also assumes that once it's gotten
\r\n in the input, that the entire request has been received. If
someone is telnetting into the server typing one character at a time,
that could fool it.

I think it's better to just launch a separate thread and use
BaseHTTPServer which is already in the stdlib.
 
S

Simon Wittber

I was inspired to enhance your code, and perform a critical bug-fix.
Your code would not have sent large files out to dialup users, because
it assumed all data was sent on the 'send' command. I added code to
check for the number of bytes sent, and loop until it's all gone.
Click to expand...

Another solution is to simply use the socket.sendall method, which
will continue to send all data until all the bytes have been sent.

Sw.
 
M

Michele Simionato

This is fun, so I will give my solution too (of course,
the effort here is to give the shortest solution, not the
more robust solution ;).

This is the server program, which just counts forever:

from threading import Thread
from CGIHTTPServer import test
import os

class Counter(Thread):
def run(self):
n = 0
self.loop = True
while self.loop:
n += 1
os.environ["COUNTER"] = str(n)
def stop(self):
self.loop = False


if __name__ == "__main__":
counter = Counter()
counter.start()
try:
test()
finally: # for instance, if CTRL-C is called
counter.stop()

And this is the CGI viewer:

#!/usr/bin/python
import os
print "Content-type: text/plain\n"
print "Counter: %s" % os.environ["COUNTER"]

Pretty bare-bone ;)

Michele Simionato
 
L

Larry Bates

Not exactly on point, but this is what I use in many of my
programs to show progress on long running console apps.

Larry Bates


class progressbarClass:
def __init__(self, finalcount, progresschar=None):
import sys
self.finalcount=finalcount
self.blockcount=0
#
# See if caller passed me a character to use on the
# progress bar (like "*"). If not use the block
# character that makes it look like a real progress
# bar.
#
if not progresschar: self.block=chr(178)
else: self.block=progresschar
#
# Get pointer to sys.stdout so I can use the write/flush
# methods to display the progress bar.
#
self.f=sys.stdout
#
# If the final count is zero, don't start the progress gauge
#
if not self.finalcount : return
self.f.write('\n------------------ % Progress -------------------1\n')
self.f.write(' 1 2 3 4 5 6 7 8 9 0\n')
self.f.write('----0----0----0----0----0----0----0----0----0----0\n')
return

def progress(self, count):
#
# Make sure I don't try to go off the end (e.g. >100%)
#
count=min(count, self.finalcount)
#
# If finalcount is zero, I'm done
#
if self.finalcount:
percentcomplete=int(round(100*count/self.finalcount))
if percentcomplete < 1: percentcomplete=1
else:
percentcomplete=100

#print "percentcomplete=",percentcomplete
blockcount=int(percentcomplete/2)
#print "blockcount=",blockcount
if blockcount > self.blockcount:
for i in range(self.blockcount,blockcount):
self.f.write(self.block)
self.f.flush()

if percentcomplete == 100: self.f.write("\n")
self.blockcount=blockcount
return

if __name__ == "__main__":
from time import sleep
pb=progressbarClass(8,"*")
count=0
while count<9:
count+=1
pb.progress(count)
sleep(0.2)

pb=progressbarClass(100)
pb.progress(20)
sleep(0.2)
pb.progress(47)
sleep(0.2)
pb.progress(90)
sleep(0.2)
pb.progress(100)
print "testing 1:"
pb=progressbarClass(1)
pb.progress(1)
 
P

Peter Hansen

Paul said:
That server seems to do a cpu-busy loop polling the listening socket
for incoming requests! Ouch!
Click to expand...

Not likely, as the socket is blocking by default. It won't
use any CPU at all until a connection arrives.
It also assumes that once it's gotten
\r\n in the input, that the entire request has been received.
Click to expand...

Which is perfectly correct given the OP's specifications,
which were that he simply wanted to hit "Refresh" in his
browser to get a new page. Note also it's \r\n\r\n, not
just a single one.
If someone is telnetting into the server typing one character at a time,
that could fool it.
Click to expand...

Fool it how? We're not talking actual data transfer, just
a simple HTTP request which ends its header section with
the aforementioned sequence, guaranteed.
I think it's better to just launch a separate thread and use
BaseHTTPServer which is already in the stdlib.
Click to expand...

Very likely.... of course, that wouldn't be any fun. ;-)

-Peter
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Similar Threads

Bash scripts for web apps 1
<Button ...> display is fine, except for two things 1
Suggestions for new Website project 3
Tracking the status of python script execution 0
mailbox.mbox Status/X-Status flags RFC? 0
How to display Isometric tiles in pygame 0
How to ensure running option on terminal or the program? 6
How can I view / open / render / display a pdf file with c code? 0

Members online

No members online now.
Total: 31 (members: 1, guests: 30)
Robots: 130

Forum statistics

Threads
473,774
Messages
2,569,599
Members
45,175
Latest member
Vinay Kumar_ Nevatia

Latest Threads

Top