Advice on long running processes

Discussion in 'Python' started by commander_coder@hotmail.com, Oct 11, 2007.

  1. Guest

    Hello,

    I write a lot of CGI scripts, in Python of course. Now I need to
    convert some to long-running processes. I'm having trouble finding
    resources about the best practices to do that.

    I've found a lot of email discussions that say something like, "You
    need to educate yourself about the differences when you have long-
    running processes" but I've not had a lot of luck with finding things
    that explain the differences. I've seen some talk about database
    timeouts, for instance, but I'm not sure I understand the problems.
    Can anyone here suggest some resources? I'd be happy with web sites,
    with buying a book, anything.

    I ask here because I write in Python and so if those resources used
    Python then that would be super.

    Thanks,
    Jim
     
    , Oct 11, 2007
    #1
    1. Advertising

  2. Roy Smith Guest

    wrote:

    > Hello,
    >
    > I write a lot of CGI scripts, in Python of course. Now I need to
    > convert some to long-running processes. I'm having trouble finding
    > resources about the best practices to do that.
    >
    > I've found a lot of email discussions that say something like, "You
    > need to educate yourself about the differences when you have long-
    > running processes" but I've not had a lot of luck with finding things
    > that explain the differences.


    The biggest differences between run-and-exit vs. long running processes are
    resource management and error recovery. Let's take them one at a time.

    Resource management. In a short-lived process, you really don't have to
    worry about this at all. Snarf as much memory as you need, open as many
    files as you want, and when you exit, the operating system cleans it all up
    for you. With a long running process, you have to worry about stuff like
    that.

    In Python, you're isolate from the low-level details of memory management,
    but still need to think about it a bit. Imagine you had code that looked
    like this in your main loop:

    for request in getNextRequest():
    requestList.append (request)
    processRequest(request)

    requestList is going to keep growing without bounds and eventually will eat
    up all available memory in the system and your process will crash.
    Everything you store, you also need to delete when you're done with it.

    Same with files. In a short-lived process, you can generally open as many
    files as you want and never worry about closing them. It unlikely you will
    ever run out of file descriptors. In a long running process, that's not
    the case. If you open a new file each time you get a request and never
    close it, after a few hundred requests (depending on the operating system,
    maybe even a few thousand), you'll run out of file descriptors.

    The other big thing is error recovery. In a short lived process, if
    something fails, you print an error message and exit. In a long running
    process, you need to somehow recover from the error and keep going as best
    you can. This can be tricky.
     
    Roy Smith, Oct 11, 2007
    #2
    1. Advertising

  3. Roy Smith wrote:
    > wrote:
    >
    >
    >>Hello,
    >>
    >>I write a lot of CGI scripts, in Python of course. Now I need to
    >>convert some to long-running processes. I'm having trouble finding
    >>resources about the best practices to do that.
    >>
    >>I've found a lot of email discussions that say something like, "You
    >>need to educate yourself about the differences when you have long-
    >>running processes" but I've not had a lot of luck with finding things
    >>that explain the differences.

    >
    >
    > The biggest differences between run-and-exit vs. long running processes are
    > resource management and error recovery. Let's take them one at a time.
    >
    > Resource management. In a short-lived process, you really don't have to
    > worry about this at all. Snarf as much memory as you need, open as many
    > files as you want, and when you exit, the operating system cleans it all up
    > for you. With a long running process, you have to worry about stuff like
    > that.
    >
    > In Python, you're isolate from the low-level details of memory management,
    > but still need to think about it a bit. Imagine you had code that looked
    > like this in your main loop:
    >
    > for request in getNextRequest():
    > requestList.append (request)
    > processRequest(request)
    >
    > requestList is going to keep growing without bounds and eventually will eat
    > up all available memory in the system and your process will crash.
    > Everything you store, you also need to delete when you're done with it.


    In particular, it is a good idea to call gc.collect() every now
    and then, specially if you are in such a loop. I don't know what is the
    gc policy in python, but an application of mine that seemed to eat as much
    memory as it was available was reduced to a constant small amount of
    memory after I started to call the gc directly.

    > The other big thing is error recovery. In a short lived process, if
    > something fails, you print an error message and exit. In a long running
    > process, you need to somehow recover from the error and keep going as best
    > you can. This can be tricky.


    You should have your main loop inside a try/except, to catch any
    exceptions that were not otherwise caught without exiting the application.
    Log the exceptions, of course, but in most long running applications
    work in a loop like Roy Smith's code above, so if one of them fails it
    won't disrupt the others to come.

    --
    Bruno Barberi Gnecco <brunobg_at_users.sourceforge.net>
    The earth is like a tiny grain of sand, only much, much heavier.
     
    Bruno Barberi Gnecco, Oct 11, 2007
    #3
  4. a écrit :
    > Hello,
    >
    > I write a lot of CGI scripts, in Python of course. Now I need to
    > convert some to long-running processes. I'm having trouble finding
    > resources about the best practices to do that.
    >
    > I've found a lot of email discussions that say something like, "You
    > need to educate yourself about the differences when you have long-
    > running processes" but I've not had a lot of luck with finding things
    > that explain the differences. I've seen some talk about database
    > timeouts, for instance, but I'm not sure I understand the problems.
    > Can anyone here suggest some resources? I'd be happy with web sites,
    > with buying a book, anything.
    >
    > I ask here because I write in Python and so if those resources used
    > Python then that would be super.


    As far as I'm concerned, I'd go for one of the available wsgi frameworks.
     
    Bruno Desthuilliers, Oct 11, 2007
    #4
  5. Guest

    Thank you to folks for the replies.

    Jim
     
    , Oct 12, 2007
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. George Marsaglia

    Assigning unsigned long to unsigned long long

    George Marsaglia, Jul 8, 2003, in forum: C Programming
    Replies:
    1
    Views:
    708
    Eric Sosman
    Jul 8, 2003
  2. Daniel Rudy

    unsigned long long int to long double

    Daniel Rudy, Sep 19, 2005, in forum: C Programming
    Replies:
    5
    Views:
    1,221
    Peter Shaggy Haywood
    Sep 20, 2005
  3. Mathieu Dutour

    long long and long

    Mathieu Dutour, Jul 17, 2007, in forum: C Programming
    Replies:
    4
    Views:
    497
    santosh
    Jul 24, 2007
  4. Bruno Desthuilliers

    What's with "long running processes" ?

    Bruno Desthuilliers, Sep 17, 2007, in forum: Python
    Replies:
    1
    Views:
    318
    walterbyrd
    Sep 18, 2007
  5. BillAtWork
    Replies:
    1
    Views:
    515
    Mr. Arnold
    Oct 12, 2009
Loading...

Share This Page