Compiler bug. The return statement does not require ( ), incidentally.
It's not a compiler bug. On FreeBSD, and thus I assume OpenBSD, NULL is
an unadorned zero:
#define NULL 0
On Linux, it's
#define NULL ((void*)0)
Both are legal definitions of NULL. However, in some circumstances the
differences can bite you. This is one of those.
Since the function is defined as returning a pointer, you should return
a pointer (surprise). The issue here is what, exactly, a null pointer
is.
A plain old zero is allowed to be a null pointer, given the proper
context. One of the constraints is that it must be a constant
expression, and the comma operator is not allowed in a constant
expression.
So, assuming NULL is just zero, we have something like:
return (errno = EINVAL, 0);
The value that's being returned has type integer, and is zero. But
it's not a constant expression, and thus converting it to a pointer type
isn't going to create a null pointer (or rather, it might or might not
depending on your implementation, but it's safer to assume it won't.)
However, if NULL is ((void*)0), then it's:
return (errno = EINVAL, ((void*)0));
The value being returned here has type void*, and is a null pointer. It
is not a constant expression, but that's OK, because a constant
expression is only required to convert an integer zero to a null
pointer, not to convert a null pointer to another pointer type.
This is a very confusing part of C, but it does show why you should
always pretend NULL is just a plain old zero. In fact, some people
refuse to use NULL at all because of issues like this; instead, they
just use a zero as a null pointer. I like NULL, personally, because I
know when to cast it and using NULL just makes the code a bit more self
documenting, IMO.
To be complete, the proper fix for the above code is:
return (errno = EINVAL, (void*)NULL);
or
return (errno = EINVAL, (void*)0);
or
... something that doesn't use the comma operator!
Chris