For starters, it can be Tokenizer
struct right inside the crate.
The struct should recognize lexical grammar described in https://262.ecma-international.org/14.0/#sec-ecmascript-language-lexical-grammar.
The struct should provide methods to switch between goal symbols on the fly. The spec section linked above gives the following reasoning for such a feature:
There are several situations where the identification of lexical input elements is sensitive to the syntactic grammar context that is consuming the input elements. This requires multiple goal symbols for the lexical grammar. The InputElementRegExpOrTemplateTail goal is used in syntactic grammar contexts where a RegularExpressionLiteral, a TemplateMiddle, or a TemplateTail is permitted.
Syntactic grammar contexts is defined by a parser state so it's the parser that needs to switch a current lexical grammar goal symbol to adjust tokenization.
The user entry point is Tokenizer
object with its get_next_symbol
method:
/// From <https://262.ecma-international.org/14.0/#sec-ecmascript-language-lexical-grammar>:
///
/// > There are several situations where the identification of lexical input
/// > elements is sensitive to the syntactic grammar context that is consuming
/// > the input elements. This requires multiple goal symbols for the lexical
/// > grammar.
pub enum GoalSymbols {
/// From <https://262.ecma-international.org/14.0/#sec-ecmascript-language-lexical-grammar>:
///
/// > Used at the start of a Script or Module
InputElementHashbangOrRegExp,
/// From <https://262.ecma-international.org/14.0/#sec-ecmascript-language-lexical-grammar>:
///
/// > Used in syntactic grammar contexts where a `RegularExpressionLiteral`,
/// > a `TemplateMiddle`, or a `TemplateTail` is permitted.
InputElementRegExpOrTemplateTail,
/// From <https://262.ecma-international.org/14.0/#sec-ecmascript-language-lexical-grammar>:
///
/// > Used in all syntactic grammar contexts where a
/// > `RegularExpressionLiteral` is permitted but neither a
/// > `TemplateMiddle`, nor a TemplateTail is permitted.
InputElementRegExp,
/// From <https://262.ecma-international.org/14.0/#sec-ecmascript-language-lexical-grammar>:
///
/// > Used in all syntactic grammar contexts where a `TemplateMiddle` or a
/// > `TemplateTail` is permitted but a `RegularExpressionLiteral` is
/// > not permitted
InputElementTemplateTail,
/// From <https://262.ecma-international.org/14.0/#sec-ecmascript-language-lexical-grammar>:
///
/// > In all other contexts, [...] used as the lexical goal symbol.
InputElementDiv
}
#[derive(Debug)]
pub struct Tokenizer {
/// A current set of lexical grammar
current_goal: GoalSymbols,
}
Grammar parameter implementation
For notation like Production[Param1, Param2]
add the following comment into the code:
Notes on how specification features are mapped into YACC features:
- grammatical parameters (like
Nonterminal[Param1, Param2]
) are implemented as a special FecerBrowser_...KeywordUsage
static semantics returning a boolean, one for each pair of a left side parameter and its right side usage
- lookahead restriction
- [No Line Terminator Here]
- automatic semicolon insertion
Cover grammar
Also add a note on what cover grammar is.
Todo list: