Python3 Grammar: Unexpected indent not recognized

[antlr]

Python3 Grammar: Unexpected indent not recognized

[whitten]

I understand that you expect that the error message should be presented.
Is there something in the language standard or spec that supports your view
?

On Mon, Jan 12, 2015 at 8:27 AM, maiermic [email protected] wrote:

input:

def func():
return 42

parser reports:
no error
expected:
An error message that indent before function definition is
unexpected/extraneous.

—
Reply to this email directly or view it on GitHub
#133.

[maiermic]

I ran my example on the command line (Python 3.4.0):

$ python my-example.py
  File "my-example.py", line 1
    def func():
    ^
IndentationError: unexpected indent

[whitten]

Is Python on the command line considered to be the reference server for
Python?
( I'm really asking because I don't know. I have no dog in this fight.)

I DO know making a grammar that is whitespace aware is harder than making
one
that ignores whitespace. This is probably going to require some tweaking of
the grammar,
possibly even intentionally putting a rule in to generate the error.

David

On Mon, Jan 12, 2015 at 9:48 AM, maiermic [email protected] wrote:

I ran my example on the command line (Python 3.4.0):

$ python my-example.py
File "my-example.py", line 1
def func():
^IndentationError: unexpected indent

—
Reply to this email directly or view it on GitHub
#133 (comment).

[bkiers]

When looking at the rule that is responsible for the creation of INDENT tokens, the NEWLINE rule, you can see that INDENT tokens are only created when the lexer "sees" a line break. Fixing it in the grammar would mean to use a predicate to check if we're at the start of the input stream followed by one or more spaces.

At the moment it looks like:

NEWLINE
 : ( '\r'? '\n' | '\r' ) SPACES?
   {
     ...
   }
 ;

and should be:

NEWLINE
 : ( {atStartOfInput()}? SPACES
   | ( '\r'? '\n' | '\r' ) SPACES?
   )
   {
     ...
   }
 ;

where atStartOfInput() would look likt his:

boolean atStartOfInput() {
  return super.getCharPositionInLine() == 0 && super.getLine() == 1;
}

Will make a PR later this day.

[bkiers]

I suppose this can be closed @parrt