Tuple size and memory allocation for embedded Python

Discussion in 'Python' started by Jinming Xu, Jan 21, 2005.

  1. Jinming Xu

    Jinming Xu Guest

    Hi Folks,

    Python seems unstable, when allocating big memory. For example, the
    following C++ code creates a tuple of tuples:

    PyObject* arCoord = PyTuple_New(n);
    double d = 1.5;
    for(int i=0; i<n; i++)
    {
    PyObject* coord = PyTuple_New(2);
    PyTuple_SetItem(coord,0, PyFloat_FromDouble(d));//x
    PyTuple_SetItem(coord,1, PyFloat_FromDouble(d));//y
    PyTuple_SetItem(arCoord,i, coord);
    }

    When the n is small, say 100, the code works fine. when n is big, say
    10,000, Python has trouble allocating memory, saying:

    "Exception exceptions.IndexError: 'tuple index out of range' in 'garbage
    collection' ignored
    Fatal Python error: unexpected exception during garbage collection
    Aborted"

    Could anyone please give me some insight or a fix for this?

    Thanks in advance for your answer.

    Jinming Xu

    _________________________________________________________________
    Is your PC infected? Get a FREE online computer virus scan from McAfee®
    Security. http://clinic.mcafee.com/clinic/ibuy/campaign.asp?cid=3963
     
    Jinming Xu, Jan 21, 2005
    #1
    1. Advertising

  2. Jinming Xu

    Steve Holden Guest

    Jinming Xu wrote:

    > Hi Folks,
    >
    > Python seems unstable, when allocating big memory. For example, the
    > following C++ code creates a tuple of tuples:
    >
    > PyObject* arCoord = PyTuple_New(n);
    > double d = 1.5;
    > for(int i=0; i<n; i++)
    > {
    > PyObject* coord = PyTuple_New(2);
    > PyTuple_SetItem(coord,0, PyFloat_FromDouble(d));//x
    > PyTuple_SetItem(coord,1, PyFloat_FromDouble(d));//y
    > PyTuple_SetItem(arCoord,i, coord);
    > }
    >
    > When the n is small, say 100, the code works fine. when n is big, say
    > 10,000, Python has trouble allocating memory, saying:
    >
    > "Exception exceptions.IndexError: 'tuple index out of range' in 'garbage
    > collection' ignored
    > Fatal Python error: unexpected exception during garbage collection
    > Aborted"
    >
    > Could anyone please give me some insight or a fix for this?
    >
    > Thanks in advance for your answer.
    >

    I'm going to guess that the problem is related to incorrect reference
    counts. I don't see any IncRefs in there. It seems probable that the
    program will work until you make n high enough to trigger a garbage
    collection sweep, then memory your program still regards as allocated is
    garbage collected by Python and reused. Ugly :p

    Python is pretty stable, so it's usually best to suspect our own code
    unless you're heavily into using the C API (which I'm not, so feel free
    to ignore me).

    regards
    Steve
    --
    Steve Holden http://www.holdenweb.com/
    Python Web Programming http://pydish.holdenweb.com/
    Holden Web LLC +1 703 861 4237 +1 800 494 3119
     
    Steve Holden, Jan 21, 2005
    #2
    1. Advertising

  3. Jinming Xu

    Craig Ringer Guest

    On Fri, 2005-01-21 at 17:20 -0500, Steve Holden wrote:
    > Jinming Xu wrote:
    >
    > > Hi Folks,
    > >
    > > Python seems unstable, when allocating big memory. For example, the
    > > following C++ code creates a tuple of tuples:
    > >
    > > PyObject* arCoord = PyTuple_New(n);
    > > double d = 1.5;
    > > for(int i=0; i<n; i++)
    > > {
    > > PyObject* coord = PyTuple_New(2);
    > > PyTuple_SetItem(coord,0, PyFloat_FromDouble(d));//x
    > > PyTuple_SetItem(coord,1, PyFloat_FromDouble(d));//y
    > > PyTuple_SetItem(arCoord,i, coord);
    > > }
    > >
    > > When the n is small, say 100, the code works fine. when n is big, say
    > > 10,000, Python has trouble allocating memory, saying:
    > >
    > > "Exception exceptions.IndexError: 'tuple index out of range' in 'garbage
    > > collection' ignored
    > > Fatal Python error: unexpected exception during garbage collection
    > > Aborted"
    > >
    > > Could anyone please give me some insight or a fix for this?
    > >
    > > Thanks in advance for your answer.
    > >

    > I'm going to guess that the problem is related to incorrect reference
    > counts.


    It's usually a safe bet, after all. Another biggie is unchecked return
    codes leaving the exception state set, though... that can cause _really_
    _weird_ problems. ALWAYS check return values.

    > I don't see any IncRefs in there.


    In this case it looks OK. PyFloat_FromDouble() reuturns a new reference,
    as does PyTuple_New(), and PyTuple_SetItem() steals a reference to its
    PyObject* argument.

    Of course, there could be refcount errors outside the shown code
    segment, but in this case I'd say the immediate error will be because of
    an unhandled exception. As to why that exception is being thrown....

    Also, forget my comment in my last post about not resizing - I'd failed
    to notice the initial creation size of the tuple (the creation of which
    isn't checked, but would segfault the app on failure).

    > Python is pretty stable, so it's usually best to suspect our own code
    > unless you're heavily into using the C API (which I'm not, so feel free
    > to ignore me).


    That's been my experience - stability issues in my Python/C code have
    almost always come down to refcounting bugs and/or failing to detect and
    handle or propagate an exception.

    --
    Craig Ringer
     
    Craig Ringer, Jan 21, 2005
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Michal Mikolajczyk
    Replies:
    1
    Views:
    832
    Larry Bates
    Apr 20, 2004
  2. Jeff Epler
    Replies:
    0
    Views:
    1,029
    Jeff Epler
    Apr 20, 2004
  3. Bill Scherer
    Replies:
    0
    Views:
    633
    Bill Scherer
    Apr 20, 2004
  4. Ken
    Replies:
    24
    Views:
    3,930
    Ben Bacarisse
    Nov 30, 2006
  5. chris
    Replies:
    6
    Views:
    1,019
    chris
    Oct 28, 2005
Loading...

Share This Page