Speeding up Python's exit

Discussion in 'Python' started by Steven D'Aprano, Feb 28, 2013.

  1. I just quit an interactive session using Python 2.7 on Linux. It took in
    excess of twelve minutes to exit, with the load average going well past 9
    for much of that time.

    I think the reason it took so long was that Python was garbage-collecting
    a giant dict with 10 million entries, each one containing a list of the
    form [1, [2, 3], 4]. But still, that's terribly slow -- ironically, it
    took longer to dispose of the dict (12+ minutes) than it took to create
    it in the first place (approx 3 minutes, with a maximum load of 4).

    Can anyone explain why this was so painfully slow, and what (if anything)
    I can do to avoid it in the future?

    I know there is a function os._exit which effectively kills the Python
    interpreter dead immediately, without doing any cleanup. What are the
    consequences of doing this? I assume that the memory used by the Python
    process will be reclaimed by the operating system, but other resources
    such as opened files may not be.


    --
    Steven
     
    Steven D'Aprano, Feb 28, 2013
    #1
    1. Advertising

  2. Steven D'Aprano

    Neil Cerutti Guest

    On 2013-02-28, Steven D'Aprano <> wrote:
    > Can anyone explain why this was so painfully slow, and what (if
    > anything) I can do to avoid it in the future?


    I think your explanation makes sense. Maybe the nested nature of
    the strings was causing it to churn looking for circular
    references?

    Disabling gc before exiting might do the trick, assuming you're
    assiduously managing other resources with context managers.

    gc.disable()
    exit()


    --
    Neil Cerutti
     
    Neil Cerutti, Feb 28, 2013
    #2
    1. Advertising

  3. On Fri, Mar 1, 2013 at 3:49 AM, Steven D'Aprano
    <> wrote:
    > I think the reason it took so long was that Python was garbage-collecting
    > a giant dict with 10 million entries, each one containing a list of the
    > form [1, [2, 3], 4]. But still, that's terribly slow -- ironically, it
    > took longer to dispose of the dict (12+ minutes) than it took to create
    > it in the first place (approx 3 minutes, with a maximum load of 4).


    Leaving the question of just *why* you have so much in your dict.....
    but anyway.

    Is it any different if you create a deliberate reference loop and then
    stuff it into some module somewhere? That would force it to be kept
    until interpreter shutdown, and then a cyclic garbage collection after
    that, which quite probably would be never run. A stupid trick,
    perhaps, but it might work; I tested it with a dummy class with a
    __del__ method and it wasn't called. Putting it into some other module
    may not be necessary, but I don't know what happens with the
    interactive interpreter and what gets freed up when.

    ChrisA
     
    Chris Angelico, Feb 28, 2013
    #3
  4. On Thu, Feb 28, 2013 at 12:06 PM, Chris Angelico <> wrote:
    > Is it any different if you create a deliberate reference loop and then
    > stuff it into some module somewhere? That would force it to be kept
    > until interpreter shutdown, and then a cyclic garbage collection after
    > that, which quite probably would be never run. A stupid trick,
    > perhaps, but it might work; I tested it with a dummy class with a
    > __del__ method and it wasn't called. Putting it into some other module
    > may not be necessary, but I don't know what happens with the
    > interactive interpreter and what gets freed up when.


    __del__ is never called for cyclic references.

    -- Devin
     
    Devin Jeanpierre, Feb 28, 2013
    #4
  5. On Thu, Feb 28, 2013 at 12:31 PM, Devin Jeanpierre
    <> wrote:
    > On Thu, Feb 28, 2013 at 12:06 PM, Chris Angelico <> wrote:
    >> Is it any different if you create a deliberate reference loop and then
    >> stuff it into some module somewhere? That would force it to be kept
    >> until interpreter shutdown, and then a cyclic garbage collection after
    >> that, which quite probably would be never run. A stupid trick,
    >> perhaps, but it might work; I tested it with a dummy class with a
    >> __del__ method and it wasn't called. Putting it into some other module
    >> may not be necessary, but I don't know what happens with the
    >> interactive interpreter and what gets freed up when.

    >
    > __del__ is never called for cyclic references.


    Sorry, I posted too early. Not only is __del__ never called, but
    __del__ is the reason the cycles aren't collected. I don't know if
    your trick will work without __del__.

    -- Devin
     
    Devin Jeanpierre, Feb 28, 2013
    #5
  6. On Fri, Mar 1, 2013 at 4:31 AM, Devin Jeanpierre <> wrote:
    > On Thu, Feb 28, 2013 at 12:06 PM, Chris Angelico <> wrote:
    >> Is it any different if you create a deliberate reference loop and then
    >> stuff it into some module somewhere? That would force it to be kept
    >> until interpreter shutdown, and then a cyclic garbage collection after
    >> that, which quite probably would be never run. A stupid trick,
    >> perhaps, but it might work; I tested it with a dummy class with a
    >> __del__ method and it wasn't called. Putting it into some other module
    >> may not be necessary, but I don't know what happens with the
    >> interactive interpreter and what gets freed up when.

    >
    > __del__ is never called for cyclic references.


    D'oh. Test is flawed, then. But is the theory plausible? That the
    cycle detector won't be called on exit after other modules get freed?

    ChrisA
     
    Chris Angelico, Feb 28, 2013
    #6
  7. On 2013-02-28, Steven D'Aprano <> wrote:

    > I know there is a function os._exit which effectively kills the
    > Python interpreter dead immediately, without doing any cleanup. What
    > are the consequences of doing this?


    You loose any data you haven't saved to disk.

    > I assume that the memory used by the Python process will be reclaimed
    > by the operating system, but other resources such as opened files may
    > not be.


    All open files (including sockets, pipes, serial ports, etc) will be
    flushed (from an OS standpoint) and closed. If you've closed all the
    files you've written to, there should be no danger in just pulling the
    plug.

    --
    Grant
     
    Grant Edwards, Mar 1, 2013
    #7
  8. Steven D'Aprano <steve+comp.lang.python <at> pearwood.info> writes:
    >
    > I just quit an interactive session using Python 2.7 on Linux. It took in
    > excess of twelve minutes to exit, with the load average going well past 9
    > for much of that time.
    >
    > I think the reason it took so long was that Python was garbage-collecting
    > a giant dict with 10 million entries, each one containing a list of the
    > form [1, [2, 3], 4]. But still, that's terribly slow -- ironically, it
    > took longer to dispose of the dict (12+ minutes) than it took to create
    > it in the first place (approx 3 minutes, with a maximum load of 4).
    >
    > Can anyone explain why this was so painfully slow, and what (if anything)
    > I can do to avoid it in the future?


    You are basically asking people to guess where your performance problem
    comes from, without even providing a snippet so that people can reproduce ;)

    > I know there is a function os._exit which effectively kills the Python
    > interpreter dead immediately, without doing any cleanup. What are the
    > consequences of doing this? I assume that the memory used by the Python
    > process will be reclaimed by the operating system, but other resources
    > such as opened files may not be.


    The OS always disposes of per-process resources when the process terminates
    (except if the OS is buggy ;-)). However, file buffers will not be flushed,
    atexit handlers and other destructors will not be called, database
    transactions will be abandoned (rolled back), etc.

    Regards

    Antoine.
     
    Antoine Pitrou, Mar 1, 2013
    #8
  9. Grant Edwards <invalid <at> invalid.invalid> writes:
    >
    > > I assume that the memory used by the Python process will be reclaimed
    > > by the operating system, but other resources such as opened files may
    > > not be.

    >
    > All open files (including sockets, pipes, serial ports, etc) will be
    > flushed (from an OS standpoint) and closed.


    According to POSIX, no, open files will not be flushed:

    “The _Exit() and _exit() functions shall not call functions registered with
    atexit() nor any registered signal handlers. Open streams shall not be flushed.
    Whether open streams are closed (without flushing) is implementation-defined.â€

    http://pubs.opengroup.org/onlinepubs/9699919799/functions/_exit.html

    (under the hood, os._exit() calls C _exit())

    Regards

    Antoine.
     
    Antoine Pitrou, Mar 1, 2013
    #9
  10. Steven D'Aprano

    Dave Angel Guest

    On 03/01/2013 02:10 PM, Antoine Pitrou wrote:
    > Grant Edwards <invalid <at> invalid.invalid> writes:
    >> <snip>
    >> All open files (including sockets, pipes, serial ports, etc) will be
    >> flushed (from an OS standpoint) and closed.

    >
    > According to POSIX, no, open files will not be flushed:
    >
    > “The _Exit() and _exit() functions shall not call functions registered with
    > atexit() nor any registered signal handlers. Open streams shall not be flushed.
    > Whether open streams are closed (without flushing) is implementation-defined.â€
    >


    Note he didn't say the python buffers would be flushed. It's the OS
    buffers that are flushed.


    --
    DaveA
     
    Dave Angel, Mar 1, 2013
    #10
  11. Dave Angel <davea <at> davea.name> writes:
    >
    > Note he didn't say the python buffers would be flushed. It's the OS
    > buffers that are flushed.


    Now please read my message again. The OS buffers are *not* flushed according
    to POSIX.
     
    Antoine Pitrou, Mar 1, 2013
    #11
  12. Steven D'Aprano

    Jason Swails Guest

    On Fri, Mar 1, 2013 at 5:51 PM, Antoine Pitrou <> wrote:

    > Dave Angel <davea <at> davea.name> writes:
    > >
    > > Note he didn't say the python buffers would be flushed. It's the OS
    > > buffers that are flushed.

    >
    > Now please read my message again. The OS buffers are *not* flushed
    > according
    > to POSIX.
    >


    I have observed this behavior on some Linux systems with a Fortran program
    that terminated abnormally (via a kill signal). Other Linux systems I've
    used appear to flush their file buffers to disk in the event of a kill
    signal, it really depends on the system.

    If a file object's destructor is not called when the Python interpreter
    exits and it's up to the OS to flush the file buffers to disk, you can't be
    sure that it will do so. And as Antoine pointed out, POSIX standard
    doesn't require that they do.

    All the best,
    Jason
     
    Jason Swails, Mar 2, 2013
    #12
  13. Steven D'Aprano

    Ross Ridge Guest

    Antoine Pitrou <> wrote:
    >Now please read my message again. The OS buffers are *not* flushed according
    >to POSIX.


    POSIX says open *streams* might not be flushed. POSIX streams are C
    FILE * streams and generally aren't regarded as being part of the OS.

    When you call os._exit() in a Python program any unwritten data still
    in Python's own file buffers will be lost. Any unwritten data still
    in the C library's FILE * buffers will be lost. Any data successfuly
    written through a POSIX file descriptor (eg. using the write() function)
    will not be lost becasue os._exit() was used.

    Note that this doesn't mean that OS buffers will flushed when os._exit()
    is called. Data that hasn't yet been physically written to disk, hasn't
    be successfully transmitted over the network, or otherwise hasn't been
    fully comitted could still be lost. However, exiting Python normally
    doesn't change this. Only the Python process's own internal buffers are
    flushed, the OS doesn't change its handling of its buffers. If you want
    written data to be fully committed before exiting you need to use other
    OS services that guarantee this.

    Ross Ridge

    --
    l/ // Ross Ridge -- The Great HTMU
    [oo][oo]
    -()-/()/ http://www.csclub.uwaterloo.ca/~rridge/
    db //
     
    Ross Ridge, Mar 3, 2013
    #13
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. =?Utf-8?B?U2FuZHk=?=

    Code to Exit Web App and Exit Internet Explorer

    =?Utf-8?B?U2FuZHk=?=, Aug 3, 2005, in forum: ASP .Net
    Replies:
    7
    Views:
    7,926
    =?Utf-8?B?U2FuZHk=?=
    Aug 5, 2005
  2. Joe Smith
    Replies:
    4
    Views:
    65,957
    sandeep1976
    Nov 8, 2006
  3. Replies:
    2
    Views:
    468
    Jeff Epler
    May 31, 2005
  4. QQ
    Replies:
    5
    Views:
    532
    Jonathan Adams
    May 10, 2005
  5. Vicky

    Difference between exit(0) & exit (1)

    Vicky, Aug 8, 2006, in forum: C Programming
    Replies:
    6
    Views:
    714
    Kenneth Brody
    Aug 8, 2006
Loading...

Share This Page