Trying to replicate `tinylang.yy` and `tinylang.l` syntax with `nimbleparse`

[jymchng]

Trying to replicate `tinylang.yy` and `tinylang.l` syntax with `nimbleparse`

[ltratt]

I think the error is because MULT/HEXDIGIT et al. don't appear in the grammar itself. However, the error is definitely confusing because the %epp is suppressing the better error message. If I get rid of the %epp lines I get this:

Warning: these tokens are defined in the lexer but not referenced in the
grammar:
  DIGIT
  GT
  GTEQ
  HEX
  HEXDIGIT
  HEXDIGITS
  LT
  LTEQ
  MULT
  SPACE
Error: these tokens are referenced in the grammar but not defined in the lexer:
  GREATER
  GREATEREQUAL
  HASH
  LESS
  LESSEQUAL
  SLASH
  STAR

[ratmice]

this is the error(s) I got, each run can yield different errors lol

Hmm, I knew this was the case with error recovery that it's errors were non-deterministic, but I wasn't aware of it in a case such as this. One other thing we could look at here is whether we should try and emit both the %epp errors, and the 'unreferenced in the lexer/grammar' warnings here. When I was adding output of multiple errors, I was somewhat conservative in that I was trying to avoid adding new cascading failures.

[ltratt]

One other thing we could look at here is whether we should try and emit both the %epp errors, and the 'unreferenced in the lexer/grammar' warnings here

This would definitely be good, but it's definitely some work.

[jymchng]

I think the error is because MULT/HEXDIGIT et al. don't appear in the grammar itself. However, the error is definitely confusing because the %epp is suppressing the better error message. If I get rid of the %epp lines I get this:

Warning: these tokens are defined in the lexer but not referenced in the
grammar:
  DIGIT
  GT
  GTEQ
  HEX
  HEXDIGIT
  HEXDIGITS
  LT
  LTEQ
  MULT
  SPACE
Error: these tokens are referenced in the grammar but not defined in the lexer:
  GREATER
  GREATEREQUAL
  HASH
  LESS
  LESSEQUAL
  SLASH
  STAR

@ltratt Thank you sir for your response.

For the first error,

> Warning: these tokens are defined in the lexer but not referenced in the
> grammar:
>   DIGIT

How can show me how can I define DIGIT in tinylang.l since it is a regex? The Java7 example wasn't helpful to me. Thank you.

[ratmice]

How can show me how can I define DIGIT in tinylang.l since it is a regex? The Java7 example wasn't helpful to me. Thank you.

These errors are a bit difficult to give general repair advice about. Here tinylang.l produces a token DIGIT, but the parser has no rules for how to consume one, comparing the linked lexer definitions:

[0-9] "DIGIT"
[0-9]+ "DIGITS"

Here, we have inlined the usage of DIGIT on the left hand side, in the original this was

digit    [0-9]
{digit}+      return tinylang::Parser::token::T_integer_literal;

While doing so we have added to the lexer the ability to produce the DIGIT token, it may never produce such a token depending on match precedence. But this token never existed in the original tinylang.l, and has nothing in tinylang.yy which prepares the parser to consume it, subsequently the lexer could produce a token which the parser does not expect.

So in this case I believe it makes sense to just remove the [0-9] "DIGIT" rule from the lexer. But in general the solution for either of these types of errors could involve modifying either or both of the lex and yacc files. E.g. when I typo the token in both.

hope that helps.