extending Python - passing nested lists

[Christian Meesters]

Hi,

I would like to write a C-extension function for an application of mine. For
this I need to pass a nested list (like: [[a, b, c], [d, e, f], ...], where
all letters are floats) to the C-function. Now, with the code I have the
compiler is complaining: "subscripted value is neither array nor pointer".
Can somebody tell me what's wrong?

Here a code snippet to reproduce this problem:

static PyObject *_foo(PyObject *self, PyObject *args) {
int i;
long lenght;
float ax, ay, az;
PyObject *dummy_list;

if (!PyArg_ParseTuple(args, "O", &dummy_list))
return NULL;
dummy_list = PySequence_Fast(dummy_list, "argument must be iterable");

lenght = PyObject_Length(dummy_list);

for (i=0; i < lenght; i++) {
// part which does not work:
ax = dummy_list[0];
ay = dummy_list[1];
az = dummy_list[2];
}
return 0;
}

TIA
Christian

 

[Mark Dickinson]

Hi,

I would like to write a C-extension function for an application of mine. For
this I need to pass a nested list (like: [[a, b, c], [d, e, f], ...], where
all letters are floats) to the C-function. Now, with the code I have the
compiler is complaining: "subscripted value is neither array nor pointer".
Can somebody tell me what's wrong?

Well, it's pretty clear: you misspelt "length" as "lenght".

PySequence_Fast doesn't return an array: it returns a PyObject---in
this case, a PyObject corresponding to a Python tuple. As the
compiler says, a PyObject is neither an array nor a pointer, so when
you write

dummy_list

the compiler doesn't know what you mean. You probably want to use
PySequence_Fast_GET_ITEM. See the documentation at

http://docs.python.org/api/sequence.html#l2h-333

Mark

 

[Christian Meesters]

Well, it's pretty clear: you misspelt "length" as "lenght".

Well, that's not it ;-). (Damn copy & paste plague ...)

PySequence_Fast doesn't return an array: it returns a PyObject---in
this case, a PyObject corresponding to a Python tuple.

That's it. Thanks. Think I just need a different approach. I got completely
on the wrong track here.

Thanks
Christian

 

[Christian Meesters]

Think, that I'm still at the wrong track. Point is that I cannot find any
examples and don't know where to start here.
Perhaps my problem boils down to two questions:
I'd like to pass lists (in some cases nested ones) from Python to C and
convert those Python-lists to C-arrays (e. g. of doubles). My second wish
is to return a C-array of longs to a Python list.

My approach so far:

static PyObject *_foo(PyObject *self, PyObject *args) {
double *v;
if (!PyArg_Parse(args, "(d)", &v))
return NULL;
// then I can't access v like v[1] ...
<snip>
// then return *v
return with something like PyBuildValue (but didn't get so far)
}

Can somebody give me a hint here, please? Passing simple arguments to and
fro (e. g. single integer values) is no problem, but lists of unknown size?

TIA
Christian

 

[Arnaud Delobelle]

Think, that I'm still at the wrong track. Point is that I cannot find any
examples and don't know where to start here.
Perhaps my problem boils down to two questions:
I'd like to pass lists (in some cases nested ones) from Python to C and
convert those Python-lists to C-arrays (e. g. of doubles). My second wish
is to return a C-array of longs to a Python list.

My approach so far:

static PyObject *_foo(PyObject *self, PyObject *args) {
  double *v;
  if (!PyArg_Parse(args, "(d)", &v))
    return NULL;
  // then I can't access v like v[1] ...

I'm not a C API guru and I may not understand properly what info you
are looking for but from http://docs.python.org/api/sequence.html:

PyObject* PySequence_GetItem( PyObject *o, Py_ssize_t i)

Return value: New reference.
Return the ith element of o, or NULL on failure. This is the
equivalent of the Python expression "o".

  <snip>
  // then return *v
  return with something like PyBuildValue (but didn't get so far)

This allows you to create a list (from http://docs.python.org/api/listObjects.html):

PyObject* PyList_New( Py_ssize_t len)

Return value: New reference.
Return a new list of length len on success, or NULL on failure. Note:
If length is greater than zero, the returned list object's items are
set to NULL. Thus you cannot use abstract API functions such as
PySequence_SetItem() or expose the object to Python code before
setting all items to a real object with PyList_SetItem().

...and this allows you to populate it (from http://docs.python.org/api/listObjects.html):

int PyList_Append( PyObject *list, PyObject *item)

Append the object item at the end of list list. Return 0 if
successful; return -1 and set an exception if unsuccessful. Analogous
to list.append(item).

}

Can somebody give me a hint here, please? Passing simple arguments to and
fro (e. g. single integer values) is no problem, but lists of unknown size?

HTH

 

[Christian Meesters]

Thanks. Point is that all such approaches would require lots(!) of calls to
the Python API - a way by which I won't gain the desired speed.

I've tried pyrex and the corresponding C-file is so convoluted with dummy
variables, incrementing & decrementing references, and other stuff, that I
want to try to write a C-function myself. My goal is not to avoid
PyObjects* and the corresponding reference handling - apart from the head
and end of the function. (I could use ctypes instead, but that again would
obfuscate the API of my package a bit.)

Christian

 

[Arnaud Delobelle]

Thanks. Point is that all such approaches would require lots(!) of calls to
the Python API - a way by which I won't gain the desired speed.

You didn't mention speed in your original post. What about using
array.array? Unless I am mistaken, these are just a thin wrapper
around normal C arrays. Anyway you could always convert your list
into a c array, do lots and lots of fast calculations, then convert it
back again to a list.

 

[Christian Meesters]

You didn't mention speed in your original post.
Sorry, perhaps I considered this self-evident - which it is, of course, not.

What about using
array.array? Unless I am mistaken, these are just a thin wrapper
around normal C arrays.

The algorithm I want to implement requires several million floating point
operations. Neither the array-modules nor numpy's thin layer seem thin
enough for me. ;-)

Anyway you could always convert your list
into a c array, do lots and lots of fast calculations, then convert it
back again to a list.

I guess I am too blind to see, but I couldn't discover a method description
like "double* PyList_toDouble". So, yes, my question is a C-API-newbie
question: What is the way to say "myCarray = SomePyMethod(InputPyList)"? Or
better, what should be here instead
static PyObject *_foo(PyObject *self, PyObject *args) {
double *v;
if (!PyArg_Parse(args, "(d)", &v))
return NULL;
to get a list as an array of doubles into 'v' (and back to Python)?

I did read the API-description, but still am lost at this point. I presume,
once I get to know the answer I'll bang my head on the table ... ;-)

Cheers
Christian

 

[Mel]

Sorry, perhaps I considered this self-evident - which it is, of course, not.

The algorithm I want to implement requires several million floating point
operations. Neither the array-modules nor numpy's thin layer seem thin
enough for me. ;-)

I guess I am too blind to see, but I couldn't discover a method description
like "double* PyList_toDouble". So, yes, my question is a C-API-newbie
question: What is the way to say "myCarray = SomePyMethod(InputPyList)"? Or
better, what should be here instead
static PyObject *_foo(PyObject *self, PyObject *args) {
double *v;
if (!PyArg_Parse(args, "(d)", &v))
return NULL;
to get a list as an array of doubles into 'v' (and back to Python)?

I did read the API-description, but still am lost at this point. I presume,
once I get to know the answer I'll bang my head on the table ... ;-)

I haven't strictly tried this, but PyArg_ParseTuple and Py_BuildValue
seem to be the orthodox ways to do Python->C and C->Python conversions.
But if Numpy isn't fast enough, then any Python at all in the solution
might be too much. Perhaps keeping your values in a file and reading
them into the C programs will work.

Mel.

 

[Arnaud Delobelle]

Sorry, perhaps I considered this self-evident - which it is, of course, not.


The algorithm I want to implement requires several million floating point
operations. Neither the array-modules nor numpy's thin layer seem thin
enough for me. ;-)

I'm not sure I understand. Here, taken from Modules/arraymodule.c, is
the definition of an arrayobject:

typedef struct arrayobject {
PyObject_VAR_HEAD
char *ob_item;
Py_ssize_t allocated;
struct arraydescr *ob_descr;
PyObject *weakreflist; /* List of weak references */
} arrayobject;

Now here is the function that gets an item from an array of floats:

static PyObject *
f_getitem(arrayobject *ap, Py_ssize_t i)
{
return PyFloat_FromDouble((double) ((float *)ap->ob_item));
}

This means that if you define an array of floats, call the arrayobject
'ap', then (float *)ap->ob_item seems to meexactly what you want: a c
array of floats. In what way is this not suitable?

I guess I am too blind to see, but I couldn't discover a method description
like "double* PyList_toDouble". So, yes, my question is a C-API-newbie
question: What is the way to say "myCarray = SomePyMethod(InputPyList)"? Or
better, what should be here instead
static PyObject *_foo(PyObject *self, PyObject *args) {
  double *v;
  if (!PyArg_Parse(args, "(d)", &v))
    return NULL;
to get a list as an array of doubles into 'v' (and back to Python)?

You can always iterate over the elements of the list an fill your c
array with the results. Look at the array_fromlist() function in
arraymodule.c, this function does exactly this.

HTH