Building extensibility into an API

[Francis Johnson]

I've been having a discussion with some folks here about the wisdom or
otherwise of "building extensibility into an API".

Consider your favorite function in one of your libraries, perhaps
void foo(int bar, char *baz);

Now maybe you think of some cool improvement to foo that you could make,
but this relies on being able to pass an additional parameter to foo.
You now have some unattractive choices:
1) change the signature of foo and break API compatability between
versions of the library
2) have a new function
foo_v2(int bar, char *baz, long double newparam);
and make foo a wrapper for foo_v2.
3) don't make the improvement.

Now, this dilemma would never have arisen if you'd "built extensibility
into the API" in the following way.

Each function you write takes an unused void * parameter. So foo starts
out as
void foo(int bar, char *baz, void *p);
and the last parameter should be NULL.

Now, consider extending the parameters of foo. You now use a
foo_v2_extra_params structure. Its first field encodes the additional
parameters, and the remaining fields are these parameters. Then version
2 of foo can do this:

void foo(int bar, char *baz, void *p)
{
/* define additional params as auto variables */
if(p)
/* process *p, set up extra args */
else
/* set additional params to defaults */
/* do stuff */
}

If foo improves again, the struct just gets enlarged, and the function
casts p appropriately before proceeding.

Have people here implemented something like this in the past? What do
you think of it as a solution to the "API extension" problem?

 

[Eric Sosman]

I've been having a discussion with some folks here about the wisdom or
otherwise of "building extensibility into an API".

Consider your favorite function in one of your libraries, perhaps
void foo(int bar, char *baz);

Now maybe you think of some cool improvement to foo that you could make,
but this relies on being able to pass an additional parameter to foo.
You now have some unattractive choices:
1) change the signature of foo and break API compatability between
versions of the library
2) have a new function
foo_v2(int bar, char *baz, long double newparam);
and make foo a wrapper for foo_v2.
3) don't make the improvement.

Option 2 seems most sensible for C. Nor does it seem
especially "unattractive:" the new, improved, all-singing,
all-dancing foo() must be doing something the old foo()
did not. The "function" of the function is different, so
why shouldn't the name be different, too?

Now, this dilemma would never have arisen if you'd "built extensibility
into the API" in the following way.

Each function you write takes an unused void * parameter. So foo starts
out as
void foo(int bar, char *baz, void *p);
and the last parameter should be NULL.

Now, consider extending the parameters of foo. You now use a
foo_v2_extra_params structure. Its first field encodes the additional
parameters, and the remaining fields are these parameters. Then version
2 of foo can do this:

void foo(int bar, char *baz, void *p)
{
/* define additional params as auto variables */
if(p)
/* process *p, set up extra args */
else
/* set additional params to defaults */
/* do stuff */
}

If foo improves again, the struct just gets enlarged, and the function
casts p appropriately before proceeding.

The struct "just gets enlarged" isn't quite enough: all
the callers must now fill the additional fields with something.
Even if your documentation carefully warned everybody to set
up a fooParams like

struct fooParams foop = { 0 };
foop.x = 42;
foop.y = -99;
/* the newly-added foop.z remains zero */

and not like

struct fooParams foop;
foop.x = 42;
foop.y = -99;
/* foop.z has indeterminate content */

or like

struct fooParams *pfoop = malloc(sizeof *pfoop);
/* assume success */
pfoop->x = 42;
pfoop->y = -99;
/* pfoop->z has indeterminate content */

you can bet your bottom dollar that somebody, somewhere will
have done it anyhow. Then you get to the unwinnable argument
about whether the user is at fault for disregarding the
documentation, or you are at fault for designing an error-
inviting interface. Better to avoid the argument altogether
than to try to win it.

Have people here implemented something like this in the past? What do
you think of it as a solution to the "API extension" problem?

Personally, I don't like it. You begin by requiring all
the users of foo() to supply an extra, unused, "must be NULL"
argument, just on the off-chance that someday you'll think of
a use for it -- that sucks. Second, by using `void*' you defeat
any attempt at type-checking when and if you eventually invent
a struct fooParams -- that sucks, too. And thirdly, as noted
above, it creates its own set of problems.

Perhaps a more compelling case could be made if foo() and
its ilk were left behind and a concrete, real-life example
were presented. Tell us about your most recent encounter with
"the API extension problem:" what was the original function,
what was the nature of the improvement, what did the additional
parameter(s) convey?

 

[cr88192]

I've been having a discussion with some folks here about the wisdom or
otherwise of "building extensibility into an API".

Consider your favorite function in one of your libraries, perhaps
void foo(int bar, char *baz);

Now maybe you think of some cool improvement to foo that you could make,
but this relies on being able to pass an additional parameter to foo.
You now have some unattractive choices:
1) change the signature of foo and break API compatability between
versions of the library
2) have a new function
foo_v2(int bar, char *baz, long double newparam);
and make foo a wrapper for foo_v2.
3) don't make the improvement.

2.
VirtualAllocEx anyone?...

you know, maybe in the future, this too will be extended:
VirtualAllocXXX
and:
VirtualAllocHardCore
VirtualAllocExtreme
VirtualAllocTooTheMax

or maybe:
VirtualAllocReloaded
VirtualAllocRessurection
....

now, they have to merge some of these together into a more powerful
function:
VirtualAllocHardCoreExtremeTooTheMax

or something...

Now, this dilemma would never have arisen if you'd "built extensibility
into the API" in the following way.

Each function you write takes an unused void * parameter. So foo starts
out as
void foo(int bar, char *baz, void *p);
and the last parameter should be NULL.

Now, consider extending the parameters of foo. You now use a
foo_v2_extra_params structure. Its first field encodes the additional
parameters, and the remaining fields are these parameters. Then version
2 of foo can do this:

If foo improves again, the struct just gets enlarged, and the function
casts p appropriately before proceeding.

Have people here implemented something like this in the past? What do
you think of it as a solution to the "API extension" problem?

syscall, ioctl, ...

4. or, how about, we encode all our arguments and data as a big string?...
then, on call, the API parses the string and figures out what is going on.

5. or, even more spiffy, how about we make each argument its own function
call.
so, we call a function telling the API we are sending something, and a call
for each arg, and then a call to indicate when we are done.

....

hmm, was trying to come up with ideas which were bizarre, but oddly enough,
these are actually useful (and I have made good use of them in the past).

4. works good when the other end of the API is some elaborate piece of
machinery (such as a compiler or interpreter).
5. works well when you are building something, such as in a physics library
or GUI framework.

 

[Tor Rustad]

I've been having a discussion with some folks here about the wisdom or
otherwise of "building extensibility into an API".

Consider your favorite function in one of your libraries, perhaps
void foo(int bar, char *baz);

Now maybe you think of some cool improvement to foo that you could make,
but this relies on being able to pass an additional parameter to foo.
You now have some unattractive choices:
1) change the signature of foo and break API compatability between
versions of the library

Many times, backward compability is important, but in some cases the API
is simply broken, and there is no way to avoid fixing existing code,
unless you change the API. gets() is a prime example here.

2) have a new function
foo_v2(int bar, char *baz, long double newparam);
and make foo a wrapper for foo_v2.

If the existing code isn't broken, adding a new API, is the bets way
IMO, but instead of using a wrapper, the usual case involve duplicating
some code and providing a new independent function to ensure you don't
break existing code.

3) don't make the improvement.

On a complex system, it might not be an improvement...

 

[ymuntyan]

I've been having a discussion with some folks here about the wisdom or
otherwise of "building extensibility into an API".

Consider your favorite function in one of your libraries, perhaps
void foo(int bar, char *baz);

Now maybe you think of some cool improvement to foo that you could make,
but this relies on being able to pass an additional parameter to foo.
You now have some unattractive choices:
1) change the signature of foo and break API compatability between
versions of the library
2) have a new function
foo_v2(int bar, char *baz, long double newparam);
and make foo a wrapper for foo_v2.
3) don't make the improvement.

Now, this dilemma would never have arisen if you'd "built extensibility
into the API" in the following way.

Each function you write takes an unused void * parameter. So foo starts
out as
void foo(int bar, char *baz, void *p);
and the last parameter should be NULL.

Now, consider extending the parameters of foo. You now use a
foo_v2_extra_params structure. Its first field encodes the additional
parameters, and the remaining fields are these parameters. Then version
2 of foo can do this:

void foo(int bar, char *baz, void *p)
{
/* define additional params as auto variables */
if(p)
/* process *p, set up extra args */
else
/* set additional params to defaults */
/* do stuff */

}

If foo improves again, the struct just gets enlarged, and the function
casts p appropriately before proceeding.

Have people here implemented something like this in the past? What do
you think of it as a solution to the "API extension" problem?

Microsoft likes to put the size of a structure argument into itself
as a version indicator. This has an advantage (for poor programmer)
of not having to have/use magic constants.

SOMESTUFF stuff;
ZeroMemory (&stuff, sizeof stuff);
stuff.cbSize = sizeof stuff;
/* ... */
CallFunctionExSoWhatIfItsExAlready (&stuff);

Because Microsoft cares.

Best regards,
Yevgen

 

[Eric Sosman]

Microsoft likes to put the size of a structure argument into itself
as a version indicator. This has an advantage (for poor programmer)
of not having to have/use magic constants.

DEC used to do that in OpenVMS (so I imagine H-P still does).
It didn't mesh well with C, though, and I once had to swat a bug
that the practice caused.

As I recall, the structure in question was an RMS ("file
system") control block of some kind. C didn't and doesn't have
a good construct for initializing such things in a way that's
"opaque" to the programmer, so VMS' library provided `const'
versions of the various control blocks, initialized with assorted
defaults. Your program set up its own control block by copying
the library's version and then filling in extra fields:

struct RMSthing cb = preInitializedRMSthing;
cb.this = 42;
cb.that = "foobar.txt";

The control block contained its own self-describing size field,
which got copied from the library to the program's version -- and
since both of them were `struct RMSthing' instances, all was well,
right?

Wrong. DEC introduced a new feature (I think it was in
support of what was then called the "High Sierra" CD-ROM format)
that involved expanding the RMSthing by a few bytes -- let's say
it went from 40 bytes to 48, although I don't recall the actual
numbers.

Alas, preInitializedRMSthing lived in a shared library (like
a "DLL" or ".so" in other systems). Our code, already compiled
and running in the field, had allocated 40 bytes for `cb' above,
and copied only 40 bytes into it from preInitializedRMSthing.
So far, so good -- but among those 40 bytes was the size field,
which advertised the control block as being 48 bytes long, not 40.
So when we used the block to tell VMS to do something, VMS looked
at the size field, said "Oh, goodie: it's a new-style RMSthing!"
and started messing around with the eight bytes past the end of
our control block ...

 

[Malcolm McLean]

Microsoft likes to put the size of a structure argument into itself
as a version indicator. This has an advantage (for poor programmer)
of not having to have/use magic constants.

SOMESTUFF stuff;
ZeroMemory (&stuff, sizeof stuff);
stuff.cbSize = sizeof stuff;
/* ... */
CallFunctionExSoWhatIfItsExAlready (&stuff);

Because Microsoft cares.

That's the way to do it. If you need make a function extensible, wrap the
parameters into a structure and pass a size in so you can extend it. The
fact that MS do it means the paradigm will be familiar to a large base of
programmers.

However it's not a good way of writing software. Filling out the struct will
be a grief to your caller. It's the best that can be done with C, and in
fact with most languages.

 

[Charlie Gordon]

That's the way to do it. If you need make a function extensible, wrap the
parameters into a structure and pass a size in so you can extend it. The
fact that MS do it means the paradigm will be familiar to a large base of
programmers.

However it's not a good way of writing software. Filling out the struct
will be a grief to your caller. It's the best that can be done with C, and
in fact with most languages.

Unix has even worse solutions for similar but related issues:

int accept(int sockfd, struct sockaddr *addr, socklen_t *addrlen);

 

[Roland Pibinger]

That's the way to do it. If you need make a function extensible, wrap the
parameters into a structure and pass a size in so you can extend it. The
fact that MS do it means the paradigm will be familiar to a large base of
programmers.

More often MS uses different names, e.g. CreateWindowEx vs.
CreateWindow. BTW, from a C programmer's point of view the Windows API
is a relatively good API.

 

[Charlie Gordon]

More often MS uses different names, e.g. CreateWindowEx vs.
CreateWindow. BTW, from a C programmer's point of view the Windows API
is a relatively good API.

That's a case of a glas half full or half empty. I as a C programmer
consider the Windows API to a relatively bad API.

 

[Richard Heathfield]

Charlie Gordon said:

I as a C programmer
consider the Windows API to a relatively bad API.

Which just goes to show that it takes all sorts, since I think the Win32 C
API is not only very good indeed, but in fact the only part of Windows I
actually enjoy using.