[API Proposal]: SourceTextTokenParser

[333fred]

[API Proposal]: SourceTextTokenParser

[DustinCampbell]

What happens if LexSyntax() is called at the end of the text?

[CyrusNajmabadi]

Ideally you get an EOF token.

[333fred]

What happens if LexSyntax() is called at the end of the text?

I expect to get an EOF token repeatedly.

[chsienki]

Feedback from API review:

• Does the underlying lexer handle double disposal? This is a struct, so could get copied and disposed twice which would dispose the underlying lexer twice.

[sharwell]

My main question regards the Result.Token. I would expect the lexer primarily produces green nodes, but only the red nodes are public API. What does the Parent property return here?

Separately, I would likely be in favor of moving to allow API access to green nodes, since they perform better in a number of feature scenarios.

[333fred]

Does the underlying lexer handle double disposal?

No. I can move to a reference type for the public type if desired. A struct wrapper seemed like the nice simple approach to me, but I don't think the extra allocation will matter in grand scheme of things, and it would allow us to be safe around double disposal.

What does the Parent property return here?

null, just like the existing ParseToken(s) methods. I think exposing green nodes is very out of scope for this proposal so I'm not going to proceed with that.

[333fred]

API Review (email edition)

• We investigated whether we could pull this token parser type into the common layer. This proved too complex due to needing to be able to save state for the language-specific lexer.
• Final API shape:

namespace Microsoft.CodeAnalysis.CSharp;

public static class SyntaxFactory
{
    // Existing methods...

    /// <summary>
    /// Creates a token parser that can be used to parse tokens from a given source text.
    /// </summary>
    /// <param name="sourceText">The source to parse tokens from.</param>
    /// <param name="options">Parse options for the source.</param>
    public static SyntaxTokenParser CreateTokenParser(SourceText sourceText, CSharpParseOptions? options = null)
}

/// <summary>
/// A token parser that can be used parse tokens continuously from a source. This parser parses continuously; every call to
/// <see cref="ParseNextToken"/> will return the next token in the source text, starting from position 0. <see cref="SkipForwardTo(int)"/>
/// can be used to skip forward in the file to a specific position, and <see cref="ResetTo(Result)"/> can be used to reset the parser
/// to a previously-lexed position.
/// </summary>
/// <remarks>
/// This type is safe to double dispose, but it is not safe to use after it has been disposed. Behavior in such scenarios
/// is undefined.
/// <para />
/// This type is not thread safe.
/// </remarks>
public sealed class SyntaxTokenParser : IDisposable
{
   public void Dispose();


   /// <summary>
   /// Parse the next token from the input at the current position. This will advance the internal position of the token parser to the
   /// end of the returned token, including any trailing trivia.
   /// </summary>
   /// <remarks>
   /// The returned token will have a parent of <see langword="null"/>.
   /// </remarks>
   public Result ParseNextToken();


   /// <summary>
   /// Skip forward in the input to the specified position. Current directive state is preserved during the skip.
   /// </summary>
   /// <param name="position">The absolute location in the original text to move to.</param>
   /// <exception cref="ArgumentOutOfRangeException">If the given position is less than the current position of the lexer.</exception>
   public void SkipForwardTo(int position);


   /// <summary>
   /// Resets the token parser to an earlier position in the input. The parser is reset the start of the token that was previously
   /// parsed, before any leading trivia, with the directive state that existed at the start of the token.
   /// </summary>
   public void ResetTo(Result result);


   /// <summary>
   /// The result of a call to <see cref="ParseNextToken"/>. This is also a context object that can be used to reset the parser to
   /// before the token it represents was parsed.
   /// </summary>
   /// <remarks>
   /// This type is not default safe. Attempts to use <code>default(Result)</code> will result in undefined behavior.
   /// </remarks>
   public readonly struct Result
   {
       /// <summary>
       /// The token that was parsed.
       /// </summary>
       public readonly SyntaxToken Token { get; }


       /// <summary>
       /// If the parsed token is potentially a contextual keyword, this will return the contextual kind of the token. Otherwise, it
       /// will return <see cref="SyntaxKind.None"/>.
       /// </summary>
       public readonly SyntaxKind ContextualKind { get; }
   }
}