Improve indentation tracking

[danielxmoore]

Improve indentation tracking

[STRd6]

I agree that indent tracking is a little complicated but I'm not sure that adding an extra parsing pass would make it less complicated. I think it is one of the core complexities of an indentation based parser. I'd be happy to take a look at other languages that have solved the problem well but I personally haven't found many good references yet.

[edemaine]

FWIW, Python and CoffeeScript have a tokenization pass that generates indent/dedent tokens separate from newline tokens, which makes for easier grammar rules. In Python this is relatively easy to do, as lines are single lines unless they end with \ or are nested in parens. But it's extremely complicated for CoffeeScript where e.g. + at end of line is a continuation, and some indentation is ignored — I believe the rewriter gets involved too — so I'm not sure its approach is better.

To me the main downside of the current approach is that we can't use caching. I think it would be possible to cache using not just the range of characters but also the current indent state. But it's a lot more complicated; the cache would also need to encode the changes to the indent state.