Best practices for skipping tokens?

[Beakerboy]

Best practices for skipping tokens?

[KvanTTT]

I discourage using skip command and always recommend using channel(HIDDEN) (or with channel other than HIDDEN) command to preserve all tokens.

[Beakerboy]

The VBA grammar currently does not hide anything, except this line continuation skip. Would you recommend this just be un-skipped for grammar consistency, or should it be hidden for some sort of backwards compatibility?

[kaby76]

You mark the content as "skip", but you can still access the whitespace through the char buffer. For example, with the "example7linecontinuation.bas" example, the tokens are:

ken@DESKTOP-DL44R7B:~/orig/vba/Generated-CSharp$ ./bin/Debug/net7.0/Test ../examples/example7linecontinuation.bas -tokens
[@0,0:2='Sub',<165>,1:0]
[@1,3:3=' ',<220>,1:3]
[@2,4:31='TestLineContinuedMemberCalls',<221>,1:4]
[@3,32:32='(',<193>,1:32]
[@4,33:33=')',<202>,1:33]
[@5,34:34='\n',<214>,1:34]
[@6,35:73=''   Valid line continuation syntax with',<216>,2:0]
[@7,74:74='\n',<214>,2:39]
[@8,75:105=''   method / property chaining.',<216>,3:0]
[@9,106:106='\n',<214>,3:31]
[@10,107:110='    ',<220>,4:0]
[@11,111:114='With',<182>,4:4]
[@12,115:115=' ',<220>,4:8]
[@13,116:116='A',<221>,4:9]
[@14,117:117='.',<4>,4:10]
[@15,118:118='B',<221>,4:11]
[@16,130:130='.',<4>,5:8]
[@17,131:131='B',<221>,5:9]
[@18,132:132='\n',<214>,5:10]
[@19,133:136='    ',<220>,6:0]
[@20,137:144='End With',<61>,6:4]
[@21,145:146='\n\n',<214>,6:12]
[@22,147:150='    ',<220>,8:0]
[@23,151:154='With',<182>,8:4]
[@24,155:155=' ',<220>,8:8]
[@25,156:156='A',<221>,8:9]
[@26,168:168='.',<4>,9:8]
[@27,169:169='B',<221>,9:9]
[@28,170:170='\n',<214>,9:10]
[@29,171:174='    ',<220>,10:0]
[@30,175:182='End With',<61>,10:4]
[@31,183:184='\n\n',<214>,10:12]
[@32,185:188='    ',<220>,12:0]
[@33,189:192='With',<182>,12:4]
[@34,193:193=' ',<220>,12:8]
[@35,194:194='A',<221>,12:9]
[@36,195:195='.',<4>,12:10]
[@37,207:207='B',<221>,13:8]
[@38,208:208='\n',<214>,13:9]
[@39,209:212='    ',<220>,14:0]
[@40,213:220='End With',<61>,14:4]
[@41,221:221='\n',<214>,14:12]
[@42,222:228='End Sub',<59>,15:0]
[@43,229:229='\n',<214>,15:7]
[@44,230:229='<EOF>',<-1>,16:0]

CSharp 0 ../examples/example7linecontinuation.bas success 0.3964643
Total Time: 0.5403111
ken@DESKTOP-DL44R7B:~/orig/vba/Generated-CSharp$ cat ../examples/example7linecontinuation.bas
Sub TestLineContinuedMemberCalls()
'   Valid line continuation syntax with
'   method / property chaining.
    With A.B _
        .B
    End With

    With A _
        .B
    End With

    With A. _
        B
    End With
End Sub
ken@DESKTOP-DL44R7B:~/orig/vba/Generated-CSharp$

Notice the large gap for the char buffer indices between token 15 and token 16. This indicates skipped content, which you can directly access using the buffer, start and stop indices into the buffer.

That is why people recommend that you marked lexer content as "HIDDEN" (channel 1). Then, you can get the content in the token stream itself, instead of using the char buffer.

The main problem with both of these approaches is that HIDDEN content is not attached to the parse tree. Antlr parse trees do not have "attributes", aka "HIDDEN" nodes. Therefore, you cannot use a query language to reason about attributes, move or delete subtrees containing attributes.

[KvanTTT]

You mark the content as "skip", but you can still access the whitespace through the char buffer.

Yes, but this solutions looks "hacky", I adhere approach with full-fidelity tokens (and parse tree).

The main problem with both of these approaches is that HIDDEN content is not attached to the parse tree.

Correct and I'm thinking about implementing such functionality in ANTLR5 (trailing and leading HIDDEN tokens that attached to main tokens).

[Beakerboy]

While reading through the VBA standard from Microsoft, there are occasions where line continuation and whitespace are disallowed, so it seems that these tokens should be made available to the parser.