Type checking inside a C extension

[Jon Perez]

I have a C extension function into which I pass a list of lists
of tuples:

[
[ ("A",1), ("B",2) ("C",3) ],
[ ("A",1), ("B",2) ("C",3) ],
[ ("A",1), ("B",2) ("C",3) ],
]

I then unpack the values (down to the tuple elements) into their C
values using:

PyLong_AsLong(PyTuple_GetItem(tupl,0))

to extract the int from the first element of the tuple and

PyString_AsString(PyTuple_GetItem(tupl,1))[0]

to get the char from the second element.


The problem is that neither PyLong_AsLong() nor PyString_AsString()
does any type checking so the interpreter crashes when I try to use
the values returned by PyLong_AsLong() and PyString_AsString() if
they happen to be fed objects - the tuple elements in this case - of
the wrong type.

I could certainly do a type check using PyLong_Check() and
PyString_Check() on the tuple items and raise a TypeError exception
to avoid leading to a crash, but I am concerned about the speed hit
as this is the innermost loop of a routine which gets called
really often (I use it for text block transfers).

Is there a less expensive way to check the type, or somehow
avoid a crashing situation (i.e. an exception gets properly raised)
without calling PyLong_Check() and PyString_Check() the elements
of each and every tuple?

 

[Paul Prescod]

> The problem is that neither PyLong_AsLong() nor
> PyString_AsString() does any type checking so the
> interpreter crashes when I try to use the values
> returned by PyLong_AsLong() and PyString_AsString() if
> they happen to be fed objects - the tuple elements in
> this case - of the wrong type.

It isn't true that PyLong_AsLong does no type checking. It does. It
reports problems through the standard Python exception system.

I could certainly do a type check using PyLong_Check() and
PyString_Check() on the tuple items and raise a TypeError exception
to avoid leading to a crash, but I am concerned about the speed hit
as this is the innermost loop of a routine which gets called really
often (I use it for text block transfers).

PyLong_Check is pretty cheap:

#define PyObject_TypeCheck(ob, tp) \
((ob)->ob_type == (tp) || PyType_IsSubtype((ob)->ob_type, (tp)))
#define PyLong_Check(op) PyObject_TypeCheck(op, &PyLong_Type)

Note that it is basically just a dereference and comparison check. I
wouldn't worry about that even in an inner loop.

Is there a less expensive way to check the type, or somehow
avoid a crashing situation (i.e. an exception gets properly raised)
without calling PyLong_Check() and PyString_Check() the elements
of each and every tuple?

I would usually use Pyrex for a job like this. But barring that, I often
use its output to remember how to do little Python/C things. Given this
input program:

def foo(b):
cdef int a
a = b

it generates code like this:

/* "/private/tmp/foo.pyx":3 */
__pyx_1 = PyInt_AsLong(__pyx_v_b);
if (PyErr_Occurred()) {error handling and return}

By PyErr_Occurred() is a real function call and is probably slower than
PyLong_Check.

Paul Prescod

 

[Andrew MacIntyre]

Is there a less expensive way to check the type, or somehow
avoid a crashing situation (i.e. an exception gets properly raised)
without calling PyLong_Check() and PyString_Check() the elements
of each and every tuple?

It might be expensive, but perhaps PyArg_ParseTuple() might be able to do
the decoding of your tuples more neatly than the various PyXX_Check()
calls.

 

[Jon Perez]

It isn't true that PyLong_AsLong does no type checking. It does. It
reports problems through the standard Python exception system.

This illustrates a case where a crash occurs instead of an
exception being thrown. Feed spam.getstring() something besides a
string:

static PyObject *spam_getstring(PyObject *self, PyObject *args) {
PyObject *tup, *a, *b;

char strstr[100];

if (!PyArg_ParseTuple(args, "O", &strobj))
return NULL;

// PyString_AsString(strobj); // raises exception
strcpy(strstr,PyString_AsString(strobj)); /* crashes!! */

Py_INCREF(Py_None);
return Py_None;
}

What gives? Shouldn't the exception be raised within PyString_AsString()
before it even gets a chance to return a value to strcpy()? (Note:
I have been using IDLE - in the default separate subprocess mode - to observe
this behaviour)


Assuming I am resigned to calling PyString_Check() myself, can I
avoid the redundant PyString_Check() that PyString_AsString() is
probably calling?

PyLong_Check is pretty cheap:

#define PyObject_TypeCheck(ob, tp) \
((ob)->ob_type == (tp) || PyType_IsSubtype((ob)->ob_type, (tp)))
#define PyLong_Check(op) PyObject_TypeCheck(op, &PyLong_Type)
Kewl...

it generates code like this:

/* "/private/tmp/foo.pyx":3 */
__pyx_1 = PyInt_AsLong(__pyx_v_b);
if (PyErr_Occurred()) {error handling and return}

By PyErr_Occurred() is a real function call and is probably slower than
PyLong_Check.

Is PyType_IsSubtype((ob)->ob_Type,(tp))) a real function call as well though?

 

[Andrew MacIntyre]

// PyString_AsString(strobj); // raises exception
strcpy(strstr,PyString_AsString(strobj)); /* crashes!! */

Py_INCREF(Py_None);
return Py_None;
}

What gives? Shouldn't the exception be raised within PyString_AsString()
before it even gets a chance to return a value to strcpy()?

First rule of using the Python C API: _always_ check the return value of
API functions.

In the case you exhibit, if the return value of PyString_AsString is NULL,
a TypeError exception will have been set and you should immediately
return NULL.

When the interpreter gets a NULL return from a called function, it checks
for an exception. If you don't return NULL, it can't detect the exception
until some other unrelated event turns it up.

 

[Stephen Horne]

It isn't true that PyLong_AsLong does no type checking. It does. It
reports problems through the standard Python exception system.

This illustrates a case where a crash occurs instead of an
exception being thrown. Feed spam.getstring() something besides a
string:

static PyObject *spam_getstring(PyObject *self, PyObject *args) {
PyObject *tup, *a, *b;

char strstr[100];

if (!PyArg_ParseTuple(args, "O", &strobj))
return NULL;

// PyString_AsString(strobj); // raises exception
strcpy(strstr,PyString_AsString(strobj)); /* crashes!! */

Py_INCREF(Py_None);
return Py_None;
}

What gives? Shouldn't the exception be raised within PyString_AsString()
before it even gets a chance to return a value to strcpy()?

I assume you're talking about when the PyString_AsString *isn't*
commented out ;-)


Python exceptions aren't converted to C exceptions, and the Python C
API doesn't use C exceptions at all. There is a good reason for this.
C doesn't have exceptions - C++ does, but this is a C API.

Given that C lacks exceptions, there is no practical (and portable)
way to forcibly unwind the stack. Even if something could be done with
longjump, for instance, there would be the issue of how your function
gets to do its cleanup.

Therefore, the simplest way for a Python C API function to propogate a
Python exception is by returning an error return value to the caller,
and keeping associated data in global variables. The caller then takes
on the responsibility for propogating the exception up to its caller
and so on. Which is precisely what Python appears to do. The caller
needs to check for the error value, do any neccessary cleanup, and
then propogate the error out of the function.

Python C API functions often return zero for error IIRC, so...

if (PyString_AsString(strobj) == 0) { return 0; }

Should ensure that the Python exception raised by PyString_AsString is
propogated correctly.

 

[Jon Perez]

Python exceptions aren't converted to C exceptions, and the Python C
API doesn't use C exceptions at all. There is a good reason for this.
C doesn't have exceptions - C++ does, but this is a C API.

Ah... now I get this seemingly mysterious behaviour.

If PyString_AsString() attempts to convert a non-string Python object to a
C string, the exception is not thrown from within it, it only sets a flag
indicating that such an exception should eventually be thrown.

While the C extension function itself does need to return a NULL for said
exception occur, it does need to to return to Python runtime first to
trigger the exception.

Python C API functions often return zero for error IIRC, so...

if (PyString_AsString(strobj) == 0) { return 0; }

Should ensure that the Python exception raised by PyString_AsString is
propogated correctly.

This also answers my earlier question. So there is no need to do a
PyString_Check() before feeding a value to PyString_AsString(), just
check the return value of PyString_AsString()...

Thanks everyone.

 

[Jon Perez]

First rule of using the Python C API: _always_ check the return value of
API functions.

This only applies to API functions which return PyObject*, right?

 

[Jon Perez]

This only applies to API functions which return PyObject*, right?

....because it is not hard to find instances in the source code for
the modules that come with Python where, for example, PyTuple_SetItem()'s
return value is not checked for.

 

[Stephen Horne]

This only applies to API functions which return PyObject*, right?

It isn't that simple. There are API functions returning an int, IIRC,
where that int can indicate an error condition.

However, if you can ensure that an error cannot exist OR if you can
arrange things such that the error is naturally handled, THEN you
don't need an explicit check.

For instance, sometimes you can just return the result of an API
function. If the API function returns a valid result, so does your
function. If the API function returns NULL, your function does so too
so the exception is propogated. If you don't need any further
processing or cleanup, why make a fuss?

"_always_ check the return value" is a very good principle, however,
even if not quite literally correct.