use streamtokenizer to implement lexical analyzer

Discussion in 'Java' started by Eric, Dec 25, 2003.

  1. Eric

    Eric Guest

    Dear all:
    I want to build a lex analyzer,
    and streamtokenizer is seems a powerful API
    but I have 2 question about it
    1).how to use method slashSlashComments(boolean flag)
    I try to set its flag as true,but it doesn't work when the comment
    char be encountered. I also get a token that contains comments.
    slashStarComments() has the same condition
    2).I want to parse the java source code, and get the token
    for example:
    str1 = str2;
    str1.compareTo("abc");
    ====================>use whitespace(' ','\t') and semicolon(';') to delimte
    str1
    =
    str2
    str1.compareTo("abc")
    ====================>but I want semicolon is also a token, like this
    str1
    =
    str2
    ;
    str1.compareTo("abc")
    ;

    can only use streamtokenizer to implement this function?
    thanks
    Eric
    Eric, Dec 25, 2003
    #1
    1. Advertising

  2. Eric wrote
    > Dear all:
    > I want to build a lex analyzer,
    > and streamtokenizer is seems a powerful API
    > but I have 2 question about it
    > 1).how to use method slashSlashComments(boolean flag)
    > I try to set its flag as true,but it doesn't work when the comment
    > char be encountered. I also get a token that contains comments.
    > slashStarComments() has the same condition
    > 2).I want to parse the java source code, and get the token
    > for example:
    > str1 = str2;
    > str1.compareTo("abc");
    > ====================>use whitespace(' ','\t') and semicolon(';') to

    delimte
    > str1
    > =
    > str2
    > str1.compareTo("abc")
    > ====================>but I want semicolon is also a token, like this
    > str1
    > =
    > str2
    > ;
    > str1.compareTo("abc")
    > ;
    >
    > can only use streamtokenizer to implement this function?
    > thanks
    > Eric


    Hello Eric,
    Have you considered using StringTokenizer instead of StreamTokenizer.
    It's constructor StringTokenizer(String str, String delim, boolean
    returnDelims) allows you to choose whether to return the delimiters as
    tokens or not. Just set the returnDelims flag to true.

    Regards,
    Perica
    Perica Milosevic, Dec 26, 2003
    #2
    1. Advertising

  3. Eric

    Eric Guest

    > Hello Eric,
    > Have you considered using StringTokenizer instead of StreamTokenizer.
    > It's constructor StringTokenizer(String str, String delim, boolean
    > returnDelims) allows you to choose whether to return the delimiters as
    > tokens or not. Just set the returnDelims flag to true.
    >
    > Regards,
    > Perica



    thanx for your suggestion
    but, I want to parse the java program source code
    the java code is so complex that I need powerful function to do this
    job
    1.I need more than one delimiter to get token once a time (e.g. '
    ','\t','\n')
    2.comments in java code must ignore
    3.white space char mention in 1. must ignore
    4.some delimiter(';','{','}') must be reserve, and others must
    discard(e.g. ' ','\t','\n')

    StringTokenizer has not enough functionality for my requirement

    StreamTokenizer is "almost" perfect, but can't return the delimiters
    as tokens

    any one can give more help?

    Regards, Eric
    Eric, Dec 26, 2003
    #3
  4. Eric

    nos Guest

    "Eric" <> wrote in message
    news:...
    > > Hello Eric,
    > > Have you considered using StringTokenizer instead of StreamTokenizer.
    > > It's constructor StringTokenizer(String str, String delim, boolean
    > > returnDelims) allows you to choose whether to return the delimiters as
    > > tokens or not. Just set the returnDelims flag to true.
    > >
    > > Regards,
    > > Perica

    >
    >
    > thanx for your suggestion
    > but, I want to parse the java program source code
    > the java code is so complex that I need powerful function to do this
    > job
    > 1.I need more than one delimiter to get token once a time (e.g. '
    > ','\t','\n')
    > 2.comments in java code must ignore
    > 3.white space char mention in 1. must ignore
    > 4.some delimiter(';','{','}') must be reserve, and others must
    > discard(e.g. ' ','\t','\n')
    >
    > StringTokenizer has not enough functionality for my requirement
    >
    > StreamTokenizer is "almost" perfect, but can't return the delimiters
    > as tokens
    >
    > any one can give more help?
    >
    > Regards, Eric


    it's not all that complicated to do yourself
    go for it
    nos, Dec 26, 2003
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. D. Lane
    Replies:
    1
    Views:
    2,495
    Dave Rathnow
    Jun 30, 2003
  2. John C. Bollinger

    Re: How does StreamTokenizer work

    John C. Bollinger, Jun 30, 2003, in forum: Java
    Replies:
    0
    Views:
    868
    John C. Bollinger
    Jun 30, 2003
  3. ATC Productions
    Replies:
    3
    Views:
    2,584
    ATC Productions
    May 28, 2005
  4. Chris Seberino
    Replies:
    1
    Views:
    518
    Hao Lian
    Sep 12, 2009
  5. Jay G. Scott

    lexical analyzer generators

    Jay G. Scott, May 26, 2005, in forum: Perl Misc
    Replies:
    2
    Views:
    104
    Jay G. Scott
    Jun 1, 2005
Loading...

Share This Page