os.system(), HTTPServer, and finishing HTTP requests

Discussion in 'Python' started by Erik Johnson, Aug 16, 2004.

  1. Erik Johnson

    Erik Johnson Guest


    I am trying to spawn a daemon type program to go off on its own and do
    some work (asynchoronously) from within an HTTPServer, but I am running into
    a problem where the web browser (actually a Perl LWP program) still seems to
    be waiting for more input until the child that is forked in the server
    finishes (granchild, actually). I don't understand this. I need to be able
    to have the POST to the server complete and go on even though there is a
    (grand)child process still running.

    So... that's the short summary, more specific code follows..

    worker.py is a program that does the standard double-fork paradigm,
    allowing the original parent to return and leaving an orphaned child behind
    to do its own thing. If you call this program directly through the shell, it
    returns you immediately to your shell, which is just like what I want and
    would expect:

    #! /usr/bin/python

    import sys, os, time

    t = 30

    pid = os.fork()
    if (pid == 0):
    # child: fork() again and exit so that the original
    # parent is not left waiting on us.
    pid = os.fork()
    if (pid > 0):
    # child
    print "child (PID %d): forked grandchild %d. I'm exiting." %\
    (os.getpid(), pid)

    # grandchild continues here, drops out bottom of if-block

    elif (pid > 0):
    # parent: exit so caller can continue
    p_pid = os.getpid()
    print "parent (PID %d): forked child %d. ." % (p_pid, pid)
    (c_pid, status) = os.wait()
    print "parent (PID %d): child %d is done. I'm exiting." % (p_pid, c_pid)
    sys.exit() # normal exit

    # grandchild continues here - my worker process
    g_pid = os.getpid()
    s = "grandchild (PID %d): going to sleep for %d seconds..." % (g_pid, t)
    print s

    # pretend to do some work

    # done
    print "grandchild (PID %d) waking up... exiting." % g_pid

    Here is sample output from running worker.py:
    ej@sand:~/src/python/problem> worker.py
    grandchild (PID 9697): going to sleep for 30 seconds...
    child (PID 9696): forked grandchild 9697. I'm exiting.
    parent (PID 9695): forked child 9696. .
    parent (PID 9695): child 9696 is done. I'm exiting.

    and then about 30 sec later, after getting back to my shell, my console

    grandchild (PID 9697) waking up... exiting.

    This is all fine and dandy. One step removed from that is to call this
    program from another Python program via os.system(). It too exits
    immediately and returns me to the shell even though my (grand)child process
    is still running. Again, this is just what I want and expect. Here is the
    code for that program:

    #! /usr/bin/python

    import os, sys

    PROG_NAME = sys.argv[0]

    print PROG_NAME + ": calling worker.py..."
    print PROG_NAME + ": all done. exiting."

    Here is the output from running it (again, all fine and dandy):

    ej@sand:~/src/python/problem> call_worker.py
    ../call_worker.py: calling worker.py...
    grandchild (PID 9710): going to sleep for 30 seconds...
    child (PID 9709): forked grandchild 9710. I'm exiting.
    parent (PID 9708): forked child 9709. .
    parent (PID 9708): child 9709 is done. I'm exiting.
    ../call_worker.py: all done. exiting.

    and after 30 seconds, your again-active shell window should print:

    grandchild (PID 9710) waking up... exiting.

    But if I make this same os.system() call from within my own HTTPServer,
    then the browser that is making the request is left hanging until that
    sleeping grandchild is done. It's like there is still a socket connection
    between it and my browser?!? But the grandchild is not forked from the
    server - it's an os.system() call! It should not be inherting any file
    descriptors and such, right? Even though the do_GET() function should be
    over, and in fact, you can see in the shell that started the server that the
    server process has already exited, the browser is apparently still waiting
    for the web page to finish loading.

    I don't get it! I have a long task I need to instantiate in my Python
    server based on info in the HTTP POST, and the program that is making that
    POST needs to be able to go on and do other stuff before my task is over.
    Why am I not getting a clean exit in my POSTing process? What do I need to
    do to get my HTTP request to my server to finish essentially "immediately"?

    Thanks for taking the time to read my post. Any help greatly appreciated!
    :) -ej

    Below is the code for a simple server you can run. It is currently coded
    to listen on port 5238 and only serve one request before the server exits.
    It serves a little static chunk of HTML on a GET request regardless of what
    PATH you call (I talked about POST above, but the browser-hanging behaviour
    is the same).

    #! /usr/bin/python


    from BaseHTTPServer import HTTPServer, BaseHTTPRequestHandler
    import os, sys

    SERVER_PORT = 5238 # arbitrary port not in use (open on firewall?)
    WORKER = 'worker.py' # name of the program to call via os.system()
    PROG = sys.argv[0] # name of this program

    class RequestHandler(BaseHTTPRequestHandler):

    def do_GET(self):

    # call my "daemon" to get some stuff done in the background
    print "%s: about to call os.system(%s)" % (PROG, WORKER)
    print "%s: back from os.system(%s)" % (PROG, WORKER)

    # send HTTP headers
    self.send_header("Content-type", 'text/html');

    # start HTML output
    html = """\
    <body><h2>All done!</h2></body>


    # BEGIN main
    if (__name__ == '__main__'):

    # create a new HTTP server and handle a request
    server = HTTPServer(('', SERVER_PORT), RequestHandler)
    Erik Johnson, Aug 16, 2004
    1. Advertisements

  2. Erik Johnson

    Donn Cave Guest

    In article <41214c28$>,
    "Erik Johnson" <ej <at> wellkeeper <dot> com> wrote:
    > I'm not sure why those would all be open, but this little bit seems to
    > resolve the problem
    > (it throws OSError when it hits the first invalid file descriptor).
    > for fd in xrange(3, 256):
    > try:
    > os.close(fd)
    > except OSError:
    > break
    > If you know a better/smarter way to effect the same thing, I'd be glad to
    > hear about it.

    That's a variation on a common idiom. The difference is
    the break, where more commonly you'd write "pass". If this
    is happening in the context of a short-term programming effort
    that has to be very efficient, it's a good idea, because the
    exceptions make the close much more expensive than it would
    have been in C, where we usually see this loop. If it needs
    to work for the indefinite future, I would say change the break
    to pass. You can't count on that continuous series of open
    file descriptors.

    Donn Cave,
    Donn Cave, Aug 17, 2004
    1. Advertisements

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Fernando Arámburu

    web requests and mobile requests

    Fernando Arámburu, Apr 8, 2005, in forum: ASP .Net
    Joerg Jooss
    Apr 8, 2005
  2. Christian Schmidt

    finishing a thead

    Christian Schmidt, Aug 21, 2003, in forum: Java
    Adam Maass
    Aug 25, 2003
  3. Krusty
    Sep 5, 2003
  4. Will Stuyvesant

    HTTP Location: supported by HTTPServer?

    Will Stuyvesant, Dec 4, 2003, in forum: Python
    Will Stuyvesant
    Dec 4, 2003
  5. jkn
    Feb 27, 2006
  6. tshad

    File Download not finishing Sub

    tshad, Nov 7, 2006, in forum: ASP .Net
    Nov 8, 2006
  7. Dave Garcia
    Dave Garcia
    Jun 5, 2009
  8. Zhidian Du
    Zhidian Du
    Feb 21, 2004