Quick question on StreamTokenizer

[overbored]

How do I prevent StreamTokenizer from returning any TT_NUMBER items?
Basically all I want is to have a steady stream of plain TT_WORD items,
even if they're all digits. I tried wordChars('0', '9') but to no avail.
Thanks in advance.

 

[Boudewijn Dijkstra]

How do I prevent StreamTokenizer from returning any TT_NUMBER items?
Basically all I want is to have a steady stream of plain TT_WORD items,
even if they're all digits. I tried wordChars('0', '9') but to no avail.

This will just assign a group of chars to two types. Try calling
resetSyntax() first.

 

[overbored]

But won't that clear out *everything*? Then I'd have to rebuild the entire
table, and I would have to do research into locales and character sets and
whatnot.... Is there no alternative?

 

[Yakov]

How do I prevent StreamTokenizer from returning any TT_NUMBER items?
Basically all I want is to have a steady stream of plain TT_WORD items,
even if they're all digits. I tried wordChars('0', '9') but to no avail.
Thanks in advance.

When you get TT_NUMBER, convert it to a String and concatenate all
strings and numbers into one string

Regards,
Yakov

 

[Boudewijn Dijkstra]

But won't that clear out *everything*? Then I'd have to rebuild the entire
table, and I would have to do research into locales and character sets and
whatnot....

That is what you might expect in Java, but quite the opposite is true:
StreamTokenizer isn't very advanced. It says in the docs: "Each byte read
from the input stream is regarded as a character in the range '\u0000' through
'\u00FF'." So there are only 256 characters to be reckoned for.

Is there no alternative?

I just read in the docs that the ordinaryChars() method does the same as
resetSyntax(), but only with the specified characters.

Remember this: javadoc is your friend.