Implement lexer and parser using virtual block tokens

[imaqtkatt]

Implement lexer and parser using virtual block tokens

[developedby]

I think the easiest and most flexible way would be to lex to this token structure

struct Token {
  Char(char),
  LineBreak,
  Begin,
  End,
}

Which keeps the text identical but consumes all whitespaces and converts it into just the indentation information. We could swap between the two behaviours by just flipping a flag in the lexer that tells it whether or not to generate the non-text tokens.

I think with this the conversion of our current parser is pretty trivial since all the text searching stays the same, we just need to change starts_with and consume to use char tokens instead of chars.

I'm not sure what VirtualSemi is in your example.

[imaqtkatt]

Here VirtualSemi will be produced between lexemes starting in the same column, the example input can be read as:

test {
  wow {
    lol yay;
    bend
  }
  ok
}