Cyclic GC rules for subtyped objects with tp_dictoffset

Discussion in 'Python' started by BChess, Mar 31, 2009.

  1. BChess

    BChess Guest

    Hi,

    I'm writing a new PyTypeObject that is base type, supports cyclic GC,
    and has a tp_dictoffset. If my type is sub-typed by a python class,
    what exactly are the rules for how I'm supposed to treat my PyDict
    object with regards to cyclic GC? Do I still visit it in my traverse
    () function if I'm subtyped? Do I decrement the refcount upon
    dealloc? By the documentation, I'm assuming I should always be using
    _PyObject_GetDictPtr() to be accessing the dictionary, which I do.
    But visiting the dictionary in traverse() in the case it's subtyped
    results in a crash in weakrefobject.c. I'm using Python 2.5.

    Thanks for any help anyone has!
    Ben
     
    BChess, Mar 31, 2009
    #1
    1. Advertising

  2. [ Questions such as this might be better suited for the capi-sig list,
    http://mail.python.org/mailman/listinfo/capi-sig ]

    BChess <> writes:

    > I'm writing a new PyTypeObject that is base type, supports cyclic
    > GC, and has a tp_dictoffset. If my type is sub-typed by a python
    > class, what exactly are the rules for how I'm supposed to treat my
    > PyDict object with regards to cyclic GC? Do I still visit it in my
    > traverse () function if I'm subtyped? Do I decrement the refcount
    > upon dealloc? By the documentation, I'm assuming I should always be
    > using _PyObject_GetDictPtr() to be accessing the dictionary, which I
    > do. But visiting the dictionary in traverse() in the case it's
    > subtyped results in a crash in weakrefobject.c. I'm using Python
    > 2.5.


    First off, if your class is intended only as a base class, are you
    aware that simply inheriting from a dictless class adds a dict
    automatically? For example, the base "object" type has no dict, but
    inheriting from it automatically adds one (unless you override that
    using __slots__). Having said that, I'll assume that the base class
    is usable on its own and its direct instances need to have a dict as
    well.

    I'm not sure if this kind of detail is explicitly documented, but as
    far as the implementation goes, the answer to your question is in
    Objects/typeobject.c:subtype_traverse. That function gets called to
    traverse instances of heap types (python subclasses of built-in
    classes such as yours). It contains code like this:

    if (type->tp_dictoffset != base->tp_dictoffset) {
    PyObject **dictptr = _PyObject_GetDictPtr(self);
    if (dictptr && *dictptr)
    Py_VISIT(*dictptr);
    }

    According to this, the base class is responsible for visiting its dict
    in its tp_traverse, and the subtype only visits the dict it added
    (which is why its location differs). Note that visiting an object
    twice still shouldn't cause a crash; objects may be and are visited an
    arbitrary number of times, and it's up to the GC to ignore those it
    has already seen. So it's possible that you have a bug elsewhere in
    the code.

    As far as the decrementing goes, the rule of thumb is: if you created
    it, you get to decref it. subtype_dealloc contains very similar
    logic:

    /* If we added a dict, DECREF it */
    if (type->tp_dictoffset && !base->tp_dictoffset) {
    PyObject **dictptr = _PyObject_GetDictPtr(self);
    if (dictptr != NULL) {
    PyObject *dict = *dictptr;
    if (dict != NULL) {
    Py_DECREF(dict);
    *dictptr = NULL;
    }
    }
    }

    So, if the subtype added a dict, it was responsible for creating it
    and it will decref it. If the dict was created by you, it's up to you
    to dispose of it.
     
    Hrvoje Niksic, Mar 31, 2009
    #2
    1. Advertising

  3. BChess

    BChess Guest

    On Mar 31, 12:27 am, Hrvoje Niksic <> wrote:
    > [ Questions such as this might be better suited for the capi-sig list,
    >  http://mail.python.org/mailman/listinfo/capi-sig]
    >
    > BChess<> writes:
    > > I'm writing a new PyTypeObject that is base type, supports cyclic
    > > GC, and has a tp_dictoffset.  If my type is sub-typed by a python
    > > class, what exactly are the rules for how I'm supposed to treat my
    > > PyDict object with regards to cyclic GC?  Do I still visit it in my
    > > traverse () function if I'm subtyped?  Do I decrement the refcount
    > > upon dealloc?  By the documentation, I'm assuming I should always be
    > > using _PyObject_GetDictPtr() to be accessing the dictionary, which I
    > > do.  But visiting the dictionary in traverse() in the case it's
    > > subtyped results in a crash in weakrefobject.c.  I'm using Python
    > > 2.5.

    >
    > First off, if your class is intended only as a base class, are you
    > aware that simply inheriting from a dictless class adds a dict
    > automatically?  For example, the base "object" type has no dict, but
    > inheriting from it automatically adds one (unless you override that
    > using __slots__).  Having said that, I'll assume that the base class
    > is usable on its own and its direct instances need to have a dict as
    > well.
    >
    > I'm not sure if this kind of detail is explicitly documented, but as
    > far as the implementation goes, the answer to your question is in
    > Objects/typeobject.c:subtype_traverse.  That function gets called to
    > traverse instances of heap types (python subclasses of built-in
    > classes such as yours).  It contains code like this:
    >
    >      if (type->tp_dictoffset != base->tp_dictoffset) {
    >          PyObject **dictptr = _PyObject_GetDictPtr(self);
    >              if (dictptr && *dictptr)
    >                  Py_VISIT(*dictptr);
    >      }
    >
    > According to this, the base class is responsible for visiting its dict
    > in its tp_traverse, and the subtype only visits the dict it added
    > (which is why its location differs).  Note that visiting an object
    > twice still shouldn't cause a crash; objects may be and are visited an
    > arbitrary number of times, and it's up to the GC to ignore those it
    > has already seen.  So it's possible that you have a bug elsewhere in
    > the code.
    >
    > As far as the decrementing goes, the rule of thumb is: if you created
    > it, you get to decref it.  subtype_dealloc contains very similar
    > logic:
    >
    >         /* If we added a dict, DECREF it */
    >         if (type->tp_dictoffset && !base->tp_dictoffset) {
    >                 PyObject **dictptr = _PyObject_GetDictPtr(self);
    >                 if (dictptr != NULL) {
    >                         PyObject *dict = *dictptr;
    >                         if (dict != NULL) {
    >                                 Py_DECREF(dict);
    >                                 *dictptr = NULL;
    >                         }
    >                 }
    >         }
    >
    > So, if the subtype added a dict, it was responsible for creating it
    > and it will decref it.  If the dict was created by you, it's up to you
    > to dispose of it.


    My confusion stemmed from the fact that I wasn't actually,
    technically, allocating a PyDict in this space.
    PyObject_GenericSetAttr() does that automatically when it finds that
    it's NULL. But that seems to be the same as if I had made it myself
    -- so I'm to dealloc either way.

    Thank you for the very in-depth answer, though. You're right: the
    problem was elsewhere. The crash stemmed from using a negative number
    for tp_dictoffset. This doesn't seem to do the right thing when
    subtyping -- the tp_dictoffset was pointing to the same memory as the
    offset specified in the subtype's tp_weaklistoffset. I missed the
    sentence in the documentation that negative tp_dictoffsets should only
    be used for variable-length objects. Using a positive offset instead
    worked like a charm.

    Thanks again,
    Ben
     
    BChess, Mar 31, 2009
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Laird Nelson

    Cyclic object graph question

    Laird Nelson, Oct 10, 2003, in forum: Java
    Replies:
    2
    Views:
    604
    Roedy Green
    Oct 10, 2003
  2. Hongzheng Wang
    Replies:
    1
    Views:
    411
    Michael Borgwardt
    Dec 2, 2003
  3. Gregory Bond
    Replies:
    1
    Views:
    330
    Denis S. Otkidach
    Apr 21, 2005
  4. 7stud
    Replies:
    11
    Views:
    701
    Dennis Lee Bieber
    Mar 20, 2007
  5. Ulrich Eckhardt

    C-API, tp_dictoffset vs tp_members

    Ulrich Eckhardt, Jul 21, 2009, in forum: Python
    Replies:
    5
    Views:
    530
    Martin v. Löwis
    Jul 26, 2009
Loading...

Share This Page