C tokens

Discussion in 'C Programming' started by Bill Cunningham, Apr 19, 2004.

  1. Lexical generators such as Bison, Flex, Lex, &c produce C tokens for the
    parser or compiler. What do these C tokens look like? According to ANSI C,
    what does the standard have to say about C tokens? Does anyone know?

    Bill
     
    Bill Cunningham, Apr 19, 2004
    #1
    1. Advertising

  2. Bill Cunningham

    Eric Sosman Guest

    Bill Cunningham wrote:
    >
    > Lexical generators such as Bison, Flex, Lex, &c produce C tokens for the
    > parser or compiler. What do these C tokens look like? According to ANSI C,
    > what does the standard have to say about C tokens? Does anyone know?


    The Standard defines "preprocessing token" and "token,"
    but nothing called a "C token."

    A "preprocessing token" is a header name, an identifier,
    a "pp-number," a character constant, a string literal, or a
    punctuator.

    A "token" is a keyword, an identifier, a constant, a string
    literal, or a punctuator.

    You'll notice a certain amount of overlap. This arises from
    the way the Standard describes the translation of C source code
    into executable programs: in the early stages of translation
    (roughly speaking, up through the point where the preprocessor
    has finished its work), the translation is described in terms of
    converting incoming characters into preprocessing tokens and
    performing various manipulations on them. Later stages convert
    the preprocessing tokens into tokens, and attach various meanings
    to them. For example, the two terms make it easy to explain why
    `sizeof' cannot be evaluated by the preprocessor.

    What do these preprocessing tokens and tokens "look like?"
    Whatever the implementor finds convenient and pleasing. The
    compiler will typically build data structures describing the
    preprocessing tokens and tokens constructed from the source, and
    will record various bits of useful information to assist the
    further actions of the translation. An "identifier," for example,
    will probably carry an indication of its scope, of its linkage
    (internal, external, or none), and a description of the thing
    it names. It might also carry additional handy information like
    "The `&' operator is never applied to this identifier, so it's
    eligible to be put into a register" -- but all such decorations
    are at the implementor's whim.

    Personally, I favor a sort of deep teal -- long on the blue,
    and not too much green.

    --
     
    Eric Sosman, Apr 19, 2004
    #2
    1. Advertising

  3. Bill Cunningham

    Servé La Guest

    "Bill Cunningham" <> wrote in message
    news:...
    > Lexical generators such as Bison, Flex, Lex, &c produce C tokens for

    the
    > parser or compiler. What do these C tokens look like? According to ANSI C,
    > what does the standard have to say about C tokens? Does anyone know?


    Bill, I see your name popping up in lots of different newsgroups with *very*
    confused posts.
    You are learning C, win32 API, ActiveX/OLE and DirectX all at once? You
    would be better off if you spend some time coding only standard C and only
    when you are comfortable with it you should learn a new API. And please, one
    at a time.
     
    Servé La, Apr 20, 2004
    #3

  4. > Bill, I see your name popping up in lots of different newsgroups with

    *very*
    > confused posts.
    > You are learning C, win32 API, ActiveX/OLE and DirectX all at once? You
    > would be better off if you spend some time coding only standard C and only
    > when you are comfortable with it you should learn a new API. And please,

    one
    > at a time.
    >

    I've given up. Atleast for now on directx. I am interested in win32 API
    and COM as well as C++. <sigh> yes Ibelieve I have a little too much on my
    plate. I've found a couple of good C tutorials though and I'm going to focus
    on them. Then after learning C thoughly as I should've long ago, I may move
    on to C++ and win32/COM programming.

    Bill
     
    Bill Cunningham, Apr 20, 2004
    #4
  5. Bill Cunningham

    kal Guest

    "Bill Cunningham" <> wrote in message news:<>...
    >
    > I've given up. Atleast for now on directx. I am interested in win32 API
    > and COM as well as C++. <sigh> yes Ibelieve I have a little too much on my
    > plate. I've found a couple of good C tutorials though and I'm going to focus
    > on them. Then after learning C thoughly as I should've long ago, I may move
    > on to C++ and win32/COM programming.


    FYI: "COM" is now obsolete. It has been replaced by .NET.
     
    kal, Apr 21, 2004
    #5
  6. Bill Cunningham

    Dan Pop Guest

    In <> (kal) writes:

    >FYI: "COM" is now obsolete. It has been replaced by .NET.


    Who cares?

    If you feel compelled to make such a comment, use private email for this
    purpose!

    Dan
    --
    Dan Pop
    DESY Zeuthen, RZ group
    Email:
     
    Dan Pop, Apr 21, 2004
    #6
  7. Bill Cunningham

    Fao, Sean Guest

    Dan Pop wrote:
    > In <> (kal) writes:
    >
    >
    >>FYI: "COM" is now obsolete. It has been replaced by .NET.

    >
    >
    > Who cares?
    >
    > If you feel compelled to make such a comment, use private email for this
    > purpose!


    Especially when it's not even a true statement. I'll leave it at that
    since COM and .NET totally off-topic for this newsgroup.
     
    Fao, Sean, Apr 23, 2004
    #7
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Ben Holness
    Replies:
    0
    Views:
    5,452
    Ben Holness
    Jan 6, 2006
  2. =?Utf-8?B?RWx0b24gVw==?=

    RE: string into tokens

    =?Utf-8?B?RWx0b24gVw==?=, Oct 13, 2005, in forum: ASP .Net
    Replies:
    0
    Views:
    438
    =?Utf-8?B?RWx0b24gVw==?=
    Oct 13, 2005
  3. =?Utf-8?B?TFc=?=

    string into tokens

    =?Utf-8?B?TFc=?=, Oct 13, 2005, in forum: ASP .Net
    Replies:
    1
    Views:
    397
    =?Utf-8?B?TFc=?=
    Oct 13, 2005
  4. Dale

    Struts Tokens - Newbie

    Dale, Feb 8, 2004, in forum: Java
    Replies:
    1
    Views:
    3,581
    Matt Parker
    Feb 10, 2004
  5. Per Magnus L?vold
    Replies:
    4
    Views:
    12,911
    Per Magnus L?vold
    Aug 12, 2004
Loading...

Share This Page