use streamtokenizer to implement lexical analyzer

[Eric]

Dear all:
I want to build a lex analyzer,
and streamtokenizer is seems a powerful API
but I have 2 question about it
1).how to use method slashSlashComments(boolean flag)
I try to set its flag as true,but it doesn't work when the comment
char be encountered. I also get a token that contains comments.
slashStarComments() has the same condition
2).I want to parse the java source code, and get the token
for example:
str1 = str2;
str1.compareTo("abc");
====================>use whitespace(' ','\t') and semicolon(';') to delimte
str1
=
str2
str1.compareTo("abc")
====================>but I want semicolon is also a token, like this
str1
=
str2
;
str1.compareTo("abc")
;

can only use streamtokenizer to implement this function?
thanks
Eric

 

[Perica Milosevic]

Eric wrote

Dear all:
I want to build a lex analyzer,
and streamtokenizer is seems a powerful API
but I have 2 question about it
1).how to use method slashSlashComments(boolean flag)
I try to set its flag as true,but it doesn't work when the comment
char be encountered. I also get a token that contains comments.
slashStarComments() has the same condition
2).I want to parse the java source code, and get the token
for example:
str1 = str2;
str1.compareTo("abc");
====================>use whitespace(' ','\t') and semicolon(';') to delimte
str1
=
str2
str1.compareTo("abc")
====================>but I want semicolon is also a token, like this
str1
=
str2
;
str1.compareTo("abc")
;

can only use streamtokenizer to implement this function?
thanks
Eric

Hello Eric,
Have you considered using StringTokenizer instead of StreamTokenizer.
It's constructor StringTokenizer(String str, String delim, boolean
returnDelims) allows you to choose whether to return the delimiters as
tokens or not. Just set the returnDelims flag to true.

Regards,
Perica

 

[Eric]

Hello Eric,

Have you considered using StringTokenizer instead of StreamTokenizer.
It's constructor StringTokenizer(String str, String delim, boolean
returnDelims) allows you to choose whether to return the delimiters as
tokens or not. Just set the returnDelims flag to true.

Regards,
Perica

thanx for your suggestion
but, I want to parse the java program source code
the java code is so complex that I need powerful function to do this
job
1.I need more than one delimiter to get token once a time (e.g. '
','\t','\n')
2.comments in java code must ignore
3.white space char mention in 1. must ignore
4.some delimiter(';','{','}') must be reserve, and others must
discard(e.g. ' ','\t','\n')

StringTokenizer has not enough functionality for my requirement

StreamTokenizer is "almost" perfect, but can't return the delimiters
as tokens

any one can give more help?

Regards, Eric

 

[nos]

thanx for your suggestion
but, I want to parse the java program source code
the java code is so complex that I need powerful function to do this
job
1.I need more than one delimiter to get token once a time (e.g. '
','\t','\n')
2.comments in java code must ignore
3.white space char mention in 1. must ignore
4.some delimiter(';','{','}') must be reserve, and others must
discard(e.g. ' ','\t','\n')

StringTokenizer has not enough functionality for my requirement

StreamTokenizer is "almost" perfect, but can't return the delimiters
as tokens

any one can give more help?

Regards, Eric

it's not all that complicated to do yourself
go for it