SyntaxError not honoured in list comprehension?

[jmfauth]

Python all versions.

It's not a bug, but I'm suprised the following does
not raise a SyntaxError (missing space between
'9' and 'for').

[9for c in 'abc'] [9, 9, 9]

Side effect: If this behaviour is considered as correct,
it makes a correct Python code styling (IDLE, editors, ...)
practically impossible to realise.

 

[Mark Dickinson]

Python all versions.

It's not a bug, but I'm suprised the following does
not raise a SyntaxError (missing space between
'9' and 'for').

[9for c in 'abc']

[9, 9, 9]

Side effect: If this behaviour is considered as correct,
it makes a correct Python code styling (IDLE, editors, ...)
practically impossible to realise.

Why? If Python itself has no problem parsing this code, why should it
be so difficult for editors? Python's grammar is fairly simple: it's
LL(1) (unlike C's, for example), so can be parsed with only 1 token of
lookahead.

 

[Mark Dickinson]

Why?  If Python itself has no problem parsing this code, why should it
be so difficult for editors?  Python's grammar is fairly simple:  it's
LL(1) (unlike C's, for example), so can be parsed with only 1 token of
lookahead.

Bah. Ignore the bit about C. I was thinking that the dangling else
issue made this a problem, but now I'm not sure that's true.

 

[Carl Banks]

Python all versions.

It's not a bug, but I'm suprised the following does
not raise a SyntaxError (missing space between
'9' and 'for').

[9for c in 'abc']

[9, 9, 9]

It does seem strange that Python's lexer wouldn't consider 9for as a
single token. Even tough it's not a valid token in Python, your eye
kind of sees it as one, so wouldn't it be better to raise a syntax
error?

Some other places were keyword can follow a number:

9if 0 else 1 (but not "9if 0else 1")
9and 0
9or 0
9in (1,2,3)
9is None

Side effect: If this behaviour is considered as correct,
it makes a correct Python code styling (IDLE, editors, ...)
practically impossible to realise.

I'm not sure why an odd corner of the grammar would mess the whole
thing up. Most code stylers only approximate the actual grammar
anyway.


Carl Banks

 

[jmfauth]

Thanks for having explained in good English my feelings.

Some other places were keyword can follow a number:

Note, that this does not envolve numbers only.

['z' for c in 'abc'] ['z', 'z', 'z']
'z'if True else 'a' z

Side effect: If this behaviour is considered as correct,
it makes a correct Python code styling (IDLE, editors, ...)
practically impossible to realise.

I'm not sure why an odd corner of the grammar would mess the whole
thing up.  Most code stylers only approximate the actual grammar
anyway.

I guess, most editors (so do I) are mainly using
a "re" engine for their styling.

---

Not a keyword, but space related, what should I thing
about this?
Traceback (most recent call last):
File "<psi last command>", line 1
print9.0
^
SyntaxError: invalid syntax
Regards,
jmf

 

[Thomas Jollans]

Thanks for having explained in good English my feelings.

Some other places were keyword can follow a number:

Note, that this does not envolve numbers only.

['z' for c in 'abc'] ['z', 'z', 'z']
'z'if True else 'a' z

Side effect: If this behaviour is considered as correct,
it makes a correct Python code styling (IDLE, editors, ...)
practically impossible to realise.

I'm not sure why an odd corner of the grammar would mess the whole
thing up. Most code stylers only approximate the actual grammar
anyway.

I guess, most editors (so do I) are mainly using
a "re" engine for their styling.

---

Not a keyword, but space related, what should I thing
about this?

looks like an identifier

Traceback (most recent call last):

can't be a single identifier. Maybe it's a print statement followed by
stuff? (stop being a statement, print!)

9

can't be an identifier or string literal. Maybe it's a print statement
followed by stuff?

abc

looks like getattr(print9, '0') - but '0' is not a valid name.
Impossible. Error!

File "<psi last command>", line 1
print9.0
^
SyntaxError: invalid syntax

somewhat strange, yes.

 

[John Machin]

somewhat strange, yes.

There are two tokens, "print9" (a name) and ".0" (a float constant) --
looks like SyntaxError to me.

 

[Mark Dickinson]

There are two tokens, "print9" (a name) and ".0" (a float constant) --
looks like SyntaxError to me.

Yep. Looks that way to me, too.

Python 2.7.0+ (release27-maint:82569, Jul 5 2010, 08:35:08)
[GCC 4.0.1 (Apple Inc. build 5490)] on darwin
Type "help", "copyright", "credits" or "license" for more information..... print token.tok_name[tok[0]], tok[1]
....
NAME print9
NUMBER .0
ENDMARKER

 

[jmfauth]

Thank you all for the discussion and the explanations.

Mark Dickinson

I toyed a littled bit this afternoon and I wrote a colouriser
(British spelling?) with the tokenize module. It is quite
simple and easy.

BTW, if I understand correctly the module tokenize import
the module token. So your example becomes:
print tokenize.tok_name[tok[0]], tok[1]

NAME print9
NUMBER .0
ENDMARKER

 

[Mark Dickinson]

BTW, if I understand correctly the module tokenize import
the module token. So your example becomes:

        print tokenize.tok_name[tok[0]], tok[1]

Ah yes; you're right. Thanks!

Mark