Lexing the ' char

Discussion in 'VHDL' started by Ole Nielsby, Nov 2, 2007.

  1. Ole Nielsby

    Ole Nielsby Guest

    I'm writing a lexer for VHDL and I don't know how to treat the ' char.
    It can be used both for character literals and as an operator similar to ::
    or .
    in C++ if I understand correctly. How would a lexer decide?
     
    Ole Nielsby, Nov 2, 2007
    #1
    1. Advertising

  2. Ole Nielsby

    Guest

    Re: Lexing the ' char

    On Nov 2, 7:58 am, "Ole Nielsby" <ole.niel...@tekare-you-
    spamminglogisk.dk> wrote:
    > I'm writing a lexer for VHDL and I don't know how to treat the ' char.
    > It can be used both for character literals and as an operator similar to ::
    > or .
    > in C++ if I understand correctly. How would a lexer decide?


    This is where you need a few characters of lookahead in your lex
    buffer. If you match ( TICK, char, TICK) you have a character literal.
    Otherwise it's the TICK token (attribute or type qualifier).

    - Kenn
     
    , Nov 2, 2007
    #2
    1. Advertising

  3. Ole Nielsby

    Ole Nielsby Guest

    Re: Lexing the ' char

    <> wrote:
    > Ole Nielsby wrote:
    >> I'm writing a lexer for VHDL and I don't know how to treat the ' char.
    >> It can be used both for character literals and as an operator similar to
    >> ::
    >> or .
    >> in C++ if I understand correctly. How would a lexer decide?

    >
    > This is where you need a few characters of lookahead in your lex
    > buffer. If you match ( TICK, char, TICK) you have a character literal.
    > Otherwise it's the TICK token (attribute or type qualifier).


    Thanks. That's what I already implemented but I wasn't sure...
     
    Ole Nielsby, Nov 2, 2007
    #3
  4. Ole Nielsby

    diogratia Guest

    Re: Lexing the ' char

    On Nov 3, 12:58 am, "Ole Nielsby" <ole.niel...@tekare-you-
    spamminglogisk.dk> wrote:
    > I'm writing a lexer for VHDL and I don't know how to treat the ' char.
    > It can be used both for character literals and as an operator similar to ::
    > or .
    > in C++ if I understand correctly. How would a lexer decide?


    On Nov 3, 12:58 am, "Ole Nielsby" <ole.niel...@tekare-you-
    spamminglogisk.dk> wrote:
    > I'm writing a lexer for VHDL and I don't know how to treat the ' char.
    > It can be used both for character literals and as an operator similar to ::
    > or .
    > in C++ if I understand correctly. How would a lexer decide?


    case '\'': /* IR1045 check */

    if ( last_token == DELIM_RIGHT_PAREN ||
    last_token == DELIM_RIGHT_BRACKET ||
    last_token == KEYWD_ALL ||
    last_token == IDENTIFIER_TOKEN ||
    last_token == STR_LIT_TOKEN ||
    last_token == CHAR_LIT_TOKEN || !
    (buff_ptr<BUFSIZ-2) )
    token_flag = DELIM_APOSTROPHE;
    else if (is_graphic_char(NEXT_CHAR) &&
    line_buff[buff_ptr+2] == '\'') {
    CHARACTER_LITERAL:
    buff_ptr+= 3; /* lead,trailing \'
    and char */
    last_token = CHAR_LIT_TOKEN;
    token_strlen = 3;
    return (last_token);
    }
    else token_flag = DELIM_APOSTROPHE;
    break;

    See Issue Report IR1045:
    http://www.eda-stds.org/isac/IRs-VHDL-93/IR1045.txt

    As you can see from the above code fragment, the last token can be
    captured and used to di"sambiguate something like:

    foo <= std_logic_vector'('a','b','c');

    without a large look ahead or backtracking.

    Mind you you could try to argue that LRM 13.2:

    ...

    "In some cases an explicit separator is required to separate adjacent
    lexical elements (namely when, without separation, interpretation as a
    single lexical element is possible). A separator is either a space
    character (SPACE or NBSP),a format effector, or the end of a line. A
    space character (SPACE or NBSP) is a separator except within a
    comment, a string literal, or a space character literal."

    could simply require the inclusion of disambiguating whitespace. The
    accepted practice would be against you, however.
     
    diogratia, Nov 4, 2007
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. wwj
    Replies:
    7
    Views:
    558
  2. lovecreatesbeauty
    Replies:
    1
    Views:
    1,065
    Ian Collins
    May 9, 2006
  3. John Carter

    Lexing and Parsing in Ruby.

    John Carter, Nov 19, 2003, in forum: Ruby
    Replies:
    2
    Views:
    168
    Robert Klemme
    Nov 19, 2003
  4. Martin DeMello

    simple lexing/parsing task

    Martin DeMello, Feb 9, 2004, in forum: Ruby
    Replies:
    4
    Views:
    142
    Martin DeMello
    Feb 10, 2004
  5. Aryeh M. Friedman
    Replies:
    34
    Views:
    577
    Aryeh M. Friedman
    Jan 7, 2013
Loading...

Share This Page