preprocessor implementation GURU question

[Dan W.]

I'm trying to resolve a disagreement between friends --Digital Mars
and BOOST-- about what the precompiler should do in a given situation:

The problem arose when I was trying to compile a boost example program
with the DM compiler, and the name of a file which was put together by
a set of macros, ended up as,

....\...\list10 .cpp
vs.
....\...\list10.cpp

(notice the space before the period)

The macro where the problem occurs is in a file called
"...boost\mpl\list.hpp", which goes:

# define MPL_AUX_LIST_HEADER
BOOST_PP_CAT(list,BLAH_LIST_SIZE).hpp \
/**/

which I was able to work-around by adding the ##, such as:
BOOST_PP_CAT(list,BLAH_LIST_SIZE)##.hpp \
/**/

I reported the problem to boost mailing list, and they said that the
compiler shouldn't be adding a space.

I reported this to the Digital Mars people and they say that the only
legal way to concatenate is with ##.

I reported again to boost, and they say that the above does not
involve concatenation, as it does not cause tokens to merge.

I reported this to DM and they insist that that is the old kludge of
'concatenation by juxtaposition'.

I reported this to boost and they say the DM compiler is relying on
the old kludge of pure textual processing, whereas tokenization should
happen first.

I think I agree with boost. Given,

#defina a(x) x
a(the_file).ext

should result in

the_file.ext

But then I'm not sure what the compiler should do if I write,

a(the_file)ext //no dot

Could someone please shed light on this issue before they shoot the
messenger? >:^0 Thanks in advance.

 

[Gianni Mariani]

Dan W. wrote:
....

I think I agree with boost. Given,

#defina a(x) x
a(the_file).ext

should result in

the_file.ext

But then I'm not sure what the compiler should do if I write,

a(the_file)ext //no dot

Could someone please shed light on this issue before they shoot the
messenger? >:^0 Thanks in advance.

I don't know what the right answer is, however, you need to boost.

a) Shoot boost because the usage of this kind of behaviour is so
uncommon that you're bound to raise issues like this. If boost relies
on bug prone features, it's use is going to be limited.

b) I'm not sure, you might need to shoot DM. They should be considering
the possibility they might be wrong. "##" has been the standard way of
concatenating for a while, I suspect that this is probably more
universally supported than what boost is relying on.

 

[Rob Williscroft]

Dan W. wrote in
[snip]

I think I agree with boost. Given,

#defina a(x) x
a(the_file).ext

should result in

the_file.ext

But then I'm not sure what the compiler should do if I write,

a(the_file)ext //no dot

It should produce the output "the_fileext" (not the quotes), however
this is 2 preproesor tokens (though only one C++ identifier). Add:

#define the_fileext Expansion
#define B() a(the_file)

B()

should produce: "the_fileext" Not "Expansion".

I.e. juxtaposition is fine for producing C++ tokens but not for
preprocessor tokens (i.e. tokens that can themselves be expanded).

The pre ## way was something like:

#define cat(a, b ) a/**/b
#define XY Expanded

cat(X, Y)

produced "Expanded".

This is what (IMO) ## was designed to fix.

Rob.

 

[Dan W.]

I don't know what the right answer is, however, you need to boost.

a) Shoot boost because the usage of this kind of behaviour is so
uncommon that you're bound to raise issues like this. If boost relies
on bug prone features, it's use is going to be limited.

b) I'm not sure, you might need to shoot DM. They should be considering
the possibility they might be wrong. "##" has been the standard way of
concatenating for a while, I suspect that this is probably more
universally supported than what boost is relying on.

Yeah, I'd shoot them both; I just don't want Armageddon to start over
a blank space ;-)

But the thing is, the boost guys may be right. What they say is that
tokenization should come before macro substitution, and that the use
of ## is prescribed only for situations where tokens, otherwise
separated by white space, should be forcibly violated into a merger.
Juxtaposing ".ext" does not violate the tokens, therefore does not
justify the use of ##. So, making the macro like

#define filename(name,num,ext) name##num##.##ext

would not exactly be right either; maybe it shouldn't even compile.
And yet I see DM's point: Allowing,

#defina a(x) x
a(b)c
would produce
bc
and therefore allow concatenation by juxtaposition.

But then again, this may not be the precomp's responsibility. The
boost guy said, paraphrasing, "Show me one place in the entire
Standard where it says that the compiler should add a space."

And the thing is, I wouldn't know what the Standard looks like if I
stepped on it. (I should get a copy, one of these days.)

Cheers!

 

[Dan W.]

The pre ## way was something like:

#define cat(a, b ) a/**/b
#define XY Expanded

cat(X, Y)

produced "Expanded".

This is what (IMO) ## was designed to fix.

Rob.

Wow! My head is spinning. Ok, you're saying that in the old days a
macro could modify the instance of another macro before the latter was
applied. But I thought that was the way it is currently, isn't it?
Would your last example NOT do that now? And how does ## prevent...
what exactly?
Maybe if you could give me a 4-way example: What worked and didn't
work / then and now. Thx.

 

[Rob Williscroft]

Dan W. wrote in

Wow! My head is spinning. Ok, you're saying that in the old days a
macro could modify the instance of another macro before the latter was
applied. But I thought that was the way it is currently, isn't it?
Would your last example NOT do that now? And how does ## prevent...
what exactly?
Maybe if you could give me a 4-way example: What worked and didn't
work / then and now. Thx.

The problem with this method is when the C standard came out it
required comments to be replaced with a single space character.

Presumably this was so you couldn't write print/* whatever */f( "\n" );
and have the compiler see printf( "\n" );

So code that relied on this behaviour needed another way of achiving
the same thing hence the invention of ##.

Rob.

 

[Dan W.]

The problem with this method is when the C standard came out it

required comments to be replaced with a single space character.

Presumably this was so you couldn't write print/* whatever */f( "\n" );
and have the compiler see printf( "\n" );

So code that relied on this behaviour needed another way of achiving
the same thing hence the invention of ##.

Ahhh! I see now. Thanks!

 

[Paul Mensonides]

It should produce the output "the_fileext" (not the quotes), however
this is 2 preproesor tokens (though only one C++ identifier). Add:

No. It is two preprocessing tokens *and* two identifier tokens. Each
preprocessing token is converted directly to a token--there is no
retokenization.

Regards,
Paul Mensonides

 

[Rob Williscroft]

Paul Mensonides wrote in

No. It is two preprocessing tokens *and* two identifier tokens. Each
preprocessing token is converted directly to a token--there is no
retokenization.

Right so the only way to write a preprocessor as a separate
programme is to introduce a space as the DM one does.

But with:

#define A() <c
#define B(X) X
#define C() >
#define INC(X) A()B(X)C()

#include INC(stdio)

since the expansion can always be internal it should include <cstdio>
not <c stdio>.

Rob.

 

[Paul Mensonides]

Paul Mensonides wrote in

Right so the only way to write a preprocessor as a separate
programme is to introduce a space as the DM one does.

But with:

#define A() <c
#define B(X) X
#define C() >
#define INC(X) A()B(X)C()

#include INC(stdio)

since the expansion can always be internal it should include <cstdio>
not <c stdio>.

Yes, but the whitespace can only be inserted *after* the preprocessor does what
it normally does. E.g. when the preprocessing tokens are output to a text
stream.

Regards,
Paul Mensonides