Where regexs listed for Python language's tokenizer/lexer?

[Chris Seberino]

Where regexs listed for Python language's tokenizer/lexer?

If I'm not mistaken, the grammar is not sufficient to specify the
language....
you also need to specify the regexs that define the tokens
right?..where is that?

chris

 

[Miles Kaufmann]

Where regexs listed for Python language's tokenizer/lexer?

If I'm not mistaken, the grammar is not sufficient to specify the
language....
you also need to specify the regexs that define the tokens
right?..where is that?

The Python tokenization process is described here:

http://docs.python.org/reference/lexical_analysis.html

The tokenizer can't be expressed in terms of regular expressions,
because it's non-regular (thanks to things like matching nested braces
and keeping track of the indentation level).

-Miles

 

[Dennis Lee Bieber]

Where regexs listed for Python language's tokenizer/lexer?

If I'm not mistaken, the grammar is not sufficient to specify the
language....
you also need to specify the regexs that define the tokens
right?..where is that?

Pardon... I've been out of the "market", but I don't recall EVER
seeing a "regex" used in a textbook for compiler/interpreter design.

BNF (or Pascal's bubble diagram equivalent) has always been used to
define the syntactical components in those books in my possession, and
parsers (tokenizers) were written using those implied algorithms (if the
first character is numeric or "." it starts a number, otherwise treat it
as an identifier, etc.),

 

[Paul McGuire]

Where regexs listed for Python language's tokenizer/lexer?

If I'm not mistaken, the grammar is not sufficient to specify the
language....
you also need to specify the regexs that define the tokens
right?..where is that?

I think the OP is asking for the regexs that define the terminals
referenced in the Python grammar, similar to those found in yacc token
definitions. He's not implying that there are regexs that implement
the whole grammar.

-- Paul

 

[Robert Kern]

Pardon... I've been out of the "market", but I don't recall EVER
seeing a "regex" used in a textbook for compiler/interpreter design.

BNF (or Pascal's bubble diagram equivalent) has always been used to
define the syntactical components in those books in my possession, and
parsers (tokenizers) were written using those implied algorithms (if the
first character is numeric or "." it starts a number, otherwise treat it
as an identifier, etc.),

In actual implementations of lexers and the lexical analysis components of
parsers, regexes are fairly common. For example, from ply:

http://www.dabeaz.com/ply/ply.html#ply_nn6

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco