tokenize module

[Jim]

I'm trying to understand the output of the tokenize.generate_tokens()
generator. The token types returned seem to be more general than I'd
expect. For example, when fed the following line of code:

def func_a():

the (abbreviated) returned token tuples are as follows:

(NAME, def, ..., def func_a()
(NAME , func_a, ..., def func_a()
(OP, (, ..., def func_a()
(OP, ), ..., def func_a()
(OP, :, ..., def func_a()
(NEWLINE, NEWLINE, ..., def func_a()

It seems to me that the token '(' should be identified as 'LPAR' and
')' as 'RPAR', as found in the dictionary token.tok_name. What am I
missing here?

 

[bootkey]

Did you read the module?  Right at the top, it says:

<quote>
It is designed to match the working of the Python tokenizer exactly, except
that it produces COMMENT tokens for comments and gives type OP for all
operators
</quote>

Thanks for the reply. I wonder why the tokenizer classifies all
operators simply as OP, instead of the various operators listed in the
tok_name dictionary.

Jim Cook