embedding Python: how to avoid memory leaks?

Discussion in 'Python' started by Andrew Trevorrow, Mar 9, 2006.

  1. No response to my last message, so I'll try a different tack...

    Does anyone know of, or even better, has anyone here written a
    C++ application for Mac/Windows that allows users to run Python
    scripts from within the app? Not just once, but many times in
    a single session, and without leaking memory. Preferably an
    open source app so I can see how it's done.

    Our app (http://golly.sourceforge.net/) currently uses calls
    like these every time a user decides to run a script:

    Py_Initialize();
    PyRun_SimpleString("execfile('foo.py')");
    Py_Finalize();

    But even if foo.py is *empty* the above calls result in a memory
    leak of about 12K on Mac OS 10.3.9 (using Python 2.3) and about
    11K on Windows 2000 (using Python 2.4.2). I wouldn't mind if
    there was a one-off cost due to calling Py_Initialize the very
    first time, but we see leaks every time a script is executed.

    I've tried calling Py_Initialize just once (at app startup)
    and Py_Finalize once on exit, but that doesn't really solve
    anything. It avoids leaks when using trivial scripts (like
    an empty .py file!) but we want to run some rather complicated
    scripts that consume lots of memory, so we need a reliable way
    to release that memory. I was surprised to discover that
    Py_Finalize doesn't always do that.

    Is there some magic Python code that can restore memory usage
    to what it was before each execfile call? Something like
    PostScript's save and restore.

    I've been struggling with this problem for about a week now.
    Having been seduced by Python's power and beauty I'd hate to
    have to abandon it and switch to Perl or some other crappy
    scripting language! Please help...

    Andrew
    Andrew Trevorrow, Mar 9, 2006
    #1
    1. Advertising

  2. Hallöchen!

    Andrew Trevorrow <> writes:

    > [...]
    >
    > [...] Not just once, but many times in a single session, and
    > without leaking memory. Preferably an open source app so I can
    > see how it's done.
    >
    > Our app (http://golly.sourceforge.net/) currently uses calls
    > like these every time a user decides to run a script:
    >
    > Py_Initialize();
    > PyRun_SimpleString("execfile('foo.py')");


    Does PyRun_AnyFile show the same effect? That's the way I'm about
    to go.

    Tschö,
    Torsten.

    --
    Torsten Bronger, aquisgrana, europa vetus ICQ 264-296-646
    Torsten Bronger, Mar 9, 2006
    #2
    1. Advertising

  3. > > Our app (http://golly.sourceforge.net/) currently uses calls
    > > like these every time a user decides to run a script:
    > >
    > > Py_Initialize();
    > > PyRun_SimpleString("execfile('foo.py')");

    >
    > Does PyRun_AnyFile show the same effect? That's the way I'm about
    > to go.


    I couldn't get the PyRun_*File* calls to work on Windows, presumably
    because of the FILE* problem mentioned in the docs.

    I'll be very surprised if it makes any difference to the memory
    leak problem. Let me know how you get on!

    Andrew
    Andrew Trevorrow, Mar 9, 2006
    #3
  4. Andrew Trevorrow wrote:
    > Our app (http://golly.sourceforge.net/) currently uses calls
    > like these every time a user decides to run a script:
    >
    > Py_Initialize();
    > PyRun_SimpleString("execfile('foo.py')");
    > Py_Finalize();
    >
    > But even if foo.py is *empty* the above calls result in a memory
    > leak of about 12K on Mac OS 10.3.9 (using Python 2.3) and about
    > 11K on Windows 2000 (using Python 2.4.2).


    I could reproduce a memory leak with the code

    #include <Python.h>
    int main()
    {
    while(1){
    Py_Initialize();
    PyRun_SimpleString("execfile('foo.py')");
    Py_Finalize();
    }
    }

    However, I could not reproduce a memory leak with the code

    #include <Python.h>
    int main()
    {
    Py_Initialize();
    while(1){
    PyRun_SimpleString("execfile('foo.py')");
    }
    Py_Finalize();
    }

    So I recommend you do Py_Initialize only once. It is well-known
    that initializing the Python interpreter allocates memory that
    can never be freed, e.g. global variables in extension modules
    (there just isn't any API to tell all the modules to release their
    memory). So a cycle of Py_Initialize/Py_Finalize will certainly
    leak.

    OTOH, PyRun_SimpleString shouldn't leak, and didn't when I
    tried it.

    Regards,
    Martin
    =?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=, Mar 9, 2006
    #4
  5. Hallöchen!

    (Andrew Trevorrow) writes:

    > [...]
    >
    > I couldn't get the PyRun_*File* calls to work on Windows, presumably
    > because of the FILE* problem mentioned in the docs.


    Which compiler do you use?

    Tschö,
    Torsten.

    --
    Torsten Bronger, aquisgrana, europa vetus ICQ 264-296-646
    Torsten Bronger, Mar 9, 2006
    #5
  6. Torsten Bronger <-aachen.de> wrote:

    > (Andrew Trevorrow) writes:
    >
    > > [...]
    > >
    > > I couldn't get the PyRun_*File* calls to work on Windows, presumably
    > > because of the FILE* problem mentioned in the docs.

    >
    > Which compiler do you use?


    MSVC++ (version 6 from memory -- I do most of my development on the
    Mac and fire up Virtual PC occasionally to test Win builds).

    Andrew
    Andrew Trevorrow, Mar 10, 2006
    #6
  7. Hallöchen!

    (Andrew Trevorrow) writes:

    > Torsten Bronger <-aachen.de> wrote:
    >
    >> (Andrew Trevorrow) writes:
    >>
    >>> [...]
    >>>
    >>> I couldn't get the PyRun_*File* calls to work on Windows,
    >>> presumably because of the FILE* problem mentioned in the docs.

    >>
    >> Which compiler do you use?

    >
    > MSVC++ (version 6 from memory -- I do most of my development on
    > the Mac and fire up Virtual PC occasionally to test Win builds).


    Well, I don't really *know*, but it's hard to believe to me that the
    file descriptor format changed within the Microsoft product series.

    Tschö,
    Torsten.

    --
    Torsten Bronger, aquisgrana, europa vetus ICQ 264-296-646
    Torsten Bronger, Mar 10, 2006
    #7
  8. wrote:

    > I could reproduce a memory leak with the code
    >
    > #include <Python.h>
    > int main()
    > {
    > while(1){
    > Py_Initialize();
    > PyRun_SimpleString("execfile('foo.py')");
    > Py_Finalize();
    > }
    > }
    >
    > However, I could not reproduce a memory leak with the code
    >
    > #include <Python.h>
    > int main()
    > {
    > Py_Initialize();
    > while(1){
    > PyRun_SimpleString("execfile('foo.py')");
    > }
    > Py_Finalize();
    > }
    >
    > So I recommend you do Py_Initialize only once. It is well-known
    > that initializing the Python interpreter allocates memory that
    > can never be freed, e.g. global variables in extension modules
    > (there just isn't any API to tell all the modules to release their
    > memory). So a cycle of Py_Initialize/Py_Finalize will certainly
    > leak.


    Surely that's a bug that should be fixed. There should be some way
    to tell Python "release all the memory you've ever allocated and
    start again with a clean slate".

    > OTOH, PyRun_SimpleString shouldn't leak, and didn't when I
    > tried it.


    Ok, after reading http://evanjones.ca/python-memory.html I think I
    understand what's happening. Apparently the Python memory allocator
    never releases memory back to the OS! So if a complicated script
    happens to consume 100MB of interpreter memory then that amount is
    no longer available to the app in which Python is embededded.
    Even worse, if a script has a (Python) memory leak then there's
    nothing the app can do about it. It would be great if wrapping
    each script inside Py_Initialize/Py_Finalize could avoid all that.

    I've been told that the next version of Python will release memory,
    so that's good news. You can get it now if you're willing to build
    Python from the latest source code.

    Andrew
    Andrew Trevorrow, Mar 10, 2006
    #8
  9. Andrew Trevorrow wrote:
    > Surely that's a bug that should be fixed. There should be some way
    > to tell Python "release all the memory you've ever allocated and
    > start again with a clean slate".


    This bug cannot be fixed in any foreseeable future.

    > I've been told that the next version of Python will release memory,
    > so that's good news. You can get it now if you're willing to build
    > Python from the latest source code.


    That still won't release all memory - only the arenas that don't
    have live Python objects on them anymore.

    Regards,
    Martin
    =?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=, Mar 10, 2006
    #9
  10. Torsten Bronger wrote:
    >>>>I couldn't get the PyRun_*File* calls to work on Windows,
    >>>>presumably because of the FILE* problem mentioned in the docs.


    > Well, I don't really *know*, but it's hard to believe to me that the
    > file descriptor format changed within the Microsoft product series.


    The layout of the FILE type indeed didn't change. However, passing
    FILE* across different versions of msvcrt will still crash; google
    for details. In particular, if you build an application with VC6,
    it will be linked with msvcrt4.dll. If you combine this with Python
    2.4 (which is linked with msvcr71.dll), you cannot use PyRun_*File*.

    Regards,
    Martin
    =?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=, Mar 10, 2006
    #10
  11. Andrew Trevorrow

    Wolfgang Guest

    Hallo,

    > >>> I couldn't get the PyRun_*File* calls to work on Windows,
    > >>> presumably because of the FILE* problem mentioned in the docs.
    > >>
    > >> Which compiler do you use?

    > >
    > > MSVC++ (version 6 from memory -- I do most of my development on
    > > the Mac and fire up Virtual PC occasionally to test Win builds).

    >
    > Well, I don't really *know*, but it's hard to believe to me that the
    > file descriptor format changed within the Microsoft product series.


    Yes it is hard to believe but true.
    Even debug and release versions have incompatible file pointer.
    It's not possible to pass af FILE* from a debug app to the embedded
    release python.dll.

    A workaround is to create the FILE* with the python api functions and
    pass this pointer back to python.
    Wolfgang, Mar 10, 2006
    #11
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Alexander Malkis
    Replies:
    8
    Views:
    505
    Alexander Malkis
    Apr 14, 2004
  2. Robert

    Python 2.2.3 memory leaks?

    Robert, Dec 9, 2003, in forum: Python
    Replies:
    0
    Views:
    299
    Robert
    Dec 9, 2003
  3. Yash
    Replies:
    2
    Views:
    78
    Brian McCauley
    Apr 15, 2004
  4. Victor Stinner
    Replies:
    0
    Views:
    111
    Victor Stinner
    Mar 7, 2013
  5. Replies:
    4
    Views:
    107
Loading...

Share This Page