Accessing __slots__ from C

[Chris]

Hi,

I'd like to be able to access an attribute of a particular Python
object as fast as possible from some C code.

I wondered if using __slots__ to store the attribute would allow me to
do this in a faster way.

The reason I'd like to do this is because I need to access the
attribute inside a loop within some C code, and I find that the
attribute lookup using the 'PyObject_GetAttrString' call is far slower
than any of the subsequent calculations I perform in C.

Using the 'PyObject_GetAttrString' function to get the attribute, I
find it is slightly faster when the attribute is a slot than when it
isn't, but that the attribute lookup remains the performance-limiting
factor.

Additionally, if I instead build a list of the attributes first and
pass that to the C code, using the 'PyList_GetItem' function to access
each item, the performance improves significantly. I'd rather be able
to access the information from C, instead of building the list
externally.

As far as I can tell, nothing is documented about accessing __slot__
members in a special way from C, but perhaps someone knows how to do
it anyway?

To be more specific, below is an example of what I'm talking about. I
use SciPy's weave to inline C code, but I assume that doesn't make any
difference to my problem.

Thanks for any suggestions,
Chris



class MyObject(object):

__slots__ = ['attr_one']

def __init__(self,attr_one=1.0):
self.attr_one = attr_one


import weave
def test_loop(myobjects):

rows,cols = len(myobjects),len(myobjects[0])

code = """
for (int r=0; r<rows; ++r) {
PyObject *myobjects_row = PyList_GetItem(myobjects,r);
for (int l=0; l<cols; ++l) {
PyObject *myobject = PyList_GetItem(myobjects_row,l);
// any faster way when attr_one is a slot?
PyObject *attr_one_obj=PyObject_GetAttrString(myobject,"attr_one");

// more computations involving attr_one; just print for now...
double attr_one = PyFloat_AsDouble(attr_one_obj);
printf("%f\\n",attr_one);
}
}
"""
weave.inline(code,['myobjects','rows','cols'],local_dict=locals(),verbose=1)


test_list = [[MyObject(0.0),MyObject(1.0)],[MyObject(2.0),MyObject(3.0)]]

test_loop(test_list)

 

[Chris]

[ You can use the capi-sig for questions like this; see
http://mail.python.org/mailman/listinfo/capi-sig ]

Thanks, I had no idea about that.

PyObject_GetAttrString is convenient, but it creates a Python string
only so it can intern it (and in most cases throw away the freshly
created version). For maximum efficiency, pre-create the string
object using PyString_InternFromString, and use that with
PyObject_GetAttr.

Yes, we'd thought of that too, but it doesn't seem to be an important
factor compared to the actual attribute lookup.

Thanks for the advice,
Chris

 

[Carl Banks]

Hi,

I'd like to be able to access an attribute of a particular Python
object as fast as possible from some C code.

I wondered if using __slots__ to store the attribute would allow me to
do this in a faster way.

The reason I'd like to do this is because I need to access the
attribute inside a loop within some C code, and I find that the
attribute lookup using the 'PyObject_GetAttrString' call is far slower
than any of the subsequent calculations I perform in C.

Using the 'PyObject_GetAttrString' function to get the attribute, I
find it is slightly faster when the attribute is a slot than when it
isn't, but that the attribute lookup remains the performance-limiting
factor.

You can determine the offset the of the slot in the object structure
by
querying the member descriptor of the type object.

descr = GetAttrString(cls,"varname");
offset = descr->d_member->offset;
slotvar = (PyObject*)(((char*)obj)+offset)

There might be some macros to simplify this.

Use at your own risk.


Carl Banks

 

[Chris]

You can determine the offset the of the slot in the object structure
by
querying the member descriptor of the type object.

That sounds like just the kind of thing we were looking for - thanks!

descr = GetAttrString(cls,"varname");
offset = descr->d_member->offset;
slotvar = (PyObject*)(((char*)obj)+offset)

Unfortunately, I am inexperienced at this kind of thing, so I wasn't
able to get something working. Maybe someone could tell me what's
wrong with the code below (it gives the error "'struct _object' has no
member named 'd_member'")?

PyObject *descr = PyObject_GetAttrString(x,"attr_one");
int offset = descr->d_member->offset;
PyObject* slotvar = (PyObject*)(((char*)obj)+offset);

where x is the class and attr_one is a slot (the full example is
appended to this message). I guessed the type of offset; I'm not sure
what it should be.

I couldn't find any information about d_member on the web.

There might be some macros to simplify this.

Sorry to say that I also have no idea about where to find such macros!
Maybe I should continue this thread on capi-sig?


Thanks for your help,
Chris


class MyObject(object):

__slots__ = ['attr_one']

def __init__(self,attr_one=1.0):
self.attr_one = attr_one

import weave
def test():

x = MyObject

code = """
PyObject *descr = PyObject_GetAttrString(x,"attr_one");
int offset = descr->d_member->offset;
//PyObject* slotvar = (PyObject*)(((char*)obj)+offset);
"""
weave.inline(code,['x'],local_dict=locals(),verbose=1)

test()

 

[Chris]

I'd like to see your test code.

Thanks for your interest in this. My test code is a whole function
that's part of a big simulator, unfortunately! I need to use the data
structures created by the simulator as part of the testing. While the
source is freely available*, that doesn't make it easy for others to
run the tests...

In my experience, as long as you're
accessing simple slots, you should notice a difference.

(I'm not sure what you mean by a 'simple slot'. The slot we're
accessing is a numpy array.)

Sorry I wasn't clear before - we do notice a difference, but not as
big a difference as when we access the attributes (arrays) from a
pre-built list. Below are timings from running the simulator
(i.e. calling the function in question many times) using the three
approaches (GetAttrString, GetAttr, and instead using a list and
GetItem; times outside parentheses are from a stopwatch; times in
parentheses are from Python's cProfile module):


- GetAttrString: 55 seconds (58 seconds)
inside the loop:
PyObject *weights_obj = PyObject_GetAttrString(obj,"attr_one");


- GetAttr: 46 seconds (46 seconds)
outside the loop:
PyObject *name = PyString_FromString("attr_one");

inside the loop:
PyObject *obj = PyObject_GetAttr(obj,name);


- PyList_GetItem: 35 seconds (37 seconds)


So, of course, you are right to say that we should notice a
difference! But speed is critical for us here.

Incidentally, what we have is a simulator written entirely in Python,
but we also provide optimized C versions of some parts of it. These C
parts must be entirely optional.

Here is a test program that shows a 4.6 time speedup simply by
switching from PyObject_GetAttrString to PyObject_GetAttr:

Thanks for the illustration. While I didn't run your code myself,
I did try to study it. Illustrations like that are very helpful.


Chris


* from http://topographica.org/

 

[Carl Banks]

You are getting that error because Carl forgot to cast the descriptor
to the appropriate C type, in this case PyMemberDescrObject.  The last
line is also incorrect, I think.

Yep, was in too much of a hurry to waste too much time showing how to
do something slightly dubious.

The offsetof() macro would help for the third line (in fact, it's the
recommended standard way since there are some C implmentions,
somewhere, that the pointer artithmetic method fails).


Carl Banks

 

[Chris]

....

The code you gave was great, thanks! Now we have done exactly what we
wanted, and have fast access from C to two particular attributes of
instances of a particular class.

We use that fast access in optimized versions of pure-Python
components of our simulator.

Chris