Where find regexs needed to build lexical analyzer for Python sourcecode?

[Chris Seberino]

I'd like to build a lexer aka lexical analyzer aka tokenizer for
Python source code as a learning exercise.

Where can I find the regexs that define the tokens of Python source?
(I am aware of tokenizer.py but I was hoping there was a web page w/ a
list somewhere.)

cs

 

[Hao Lian]

Chris Seberino writes with great flourish:

I'd like to build a lexer aka lexical analyzer aka tokenizer for
Python source code as a learning exercise.

Where can I find the regexs that define the tokens of Python source?
(I am aware of tokenizer.py but I was hoping there was a web page w/ a
list somewhere.)

cs

The tokenize library! bum bum bum

http://svn.python.org/view/python/trunk/Lib/tokenize.py