The antlr4parsetreevisualizer from zspitz

Reduce boilerplate in VisualizerConfig

Every additional property needs to be updated in 5 different places in the class.

ConditionalMessageContainer control

The token list and treeview both have the same behavior: the control is only shown if the ItemsSource is not null. Otherwise a text message is displayed.

This behavior should be written into a custom control. Not sure how to implement this.

Update notification

Global setting - "Check for new release on visualizer load"

New release check: check latest release version against current version. If there's a new release, display a messagebox and ask about opening the releases page. If yes, open both the releases page and the destination folder (based on the current path of the assembly).

If the setting is true, then each time the visualizer loads, checks if there's a new release, as above.

If the setting is false, then show an explicit "Check now" button next to the setting UI. On click, check as above.

Token list controls

Checkbox -- show/hide whitespace tokens
Checkbox -- show/hide error tokens
List of token types to filter by

Use rule name instead of type name in tree nodes

Instead of ModuleContext, SubStmtContext etc., use Module, SubStmt. in the tree nodes.

Settings

We have to figure out how to persist settings. If we can we read/write from the debugger side to %ProgramData% that would be best. Otherwise, perhaps a local JSON file?

Some settings are global:

Maximum window dimensions, as a percentage of entire monitor; or absolute pixels

Some settings relate to a given parser class; the key should be the class name + the executing assembly location. But if we haven't found the parser class, they should be set globally; or perhaps using a default key

Last used parser class and assembly, based on the executing assembly location?
Pane locations, once we use some sort of docking control (AvalonDock?)

Allow expanding collection properties

Properties that return collections should expand to a full datagrid of those items, if they have multiple properties; or a single column if a collection of scalar values.

Source box selection doesn't keep up

Algorithm for identifying the parser class

This is important for getting rule and token names (#7) and for storing settings per class/assembly.

On the debugger side, iterate through all the assemblies, find all the classes that inherit from Antlr4.Runtime.Parser and are marked with the GeneratedCode attribute by ANTLR. If there is only one, that is the parser class.

On the debuggee side, present the user with all the available parser classes, and pass it along in the settings object.

Could not load this custom viewer

Hello.
After install this viewer got error on runtime:

Package artifacts as a single ZIP per VS version

It's better to have a single download (and unblock) from the Releases page, than to have multiple downloads; each file has to be unblocked, and through the UI it can only be done one file at a time.

We could include relative paths in the ZIP, which might be an alternative to #17 -- the main DLL always needs to be in the root Visualizers folder, but the dependency DLLs could be in a subfolder.

Column headers in property grid

Currently labelled as Key and Value. Find some way to give them better captions.

Selection in source chooses an entire token

Text in source corresponding to error tokens should be in red

Crash with large grammars and parse trees

Attached is a zip file of a solution/project containing a Java parser example. Displaying the tree with ANTLR4ParseTreeVisualizer crashes, presumably because it is a large parse tree, on release 0.1.0.63. It's good to be able to visualize non-trivial large trees, especially for nodes deep in the tree in a tree visitor. GetText() of a node doesn't work well, and I need to visualize where in the tree I am, e.g., with a variableDeclarator in what block.
AntlrConsole3.zip

Move filtering to viewmodel

Currently filtering is done at the backend. It's more appropriate to do in the UI directly.

Properties with null value

Currently returns the NRE error message, because we're trying to call ToString on null.

A number of choices:

Exclude such properties entirely? Don't like this.
Different background color.
Special string? -- ?

We could do both of the latter 2.

Visualize strings / standalone visualizer

This is the shared workflow for:

a Visual Studio debugging visualizer attached to the string type
a standalone application that shows the parse tree for user-input source code, and a selected lexer and parser

User	Code
Choose lexer and parser assembly and class
Optionally, choose rule
Either enter source code, or load from a file (within the debugging visualizer, this would be set by calling code)
Click Parse
	Instantiate lexer, common token source, parser
	Create instance of `VisualizerData` class from returned parser rule context
	Show visualized

@bclothier @retailcoder Does the above work for visualizing strings?

Enforce proper MVVM architecture

Relates to #22

Also important if we want to create a separate visualizer on token streams. We'll have to tease apart the token list control and model into a reusable form.

Token list should show more than just IDs

The list of tokens should include the token names:

In a grey brush font, maybe (like the "(Type)" under the "Filter rule contexts" box).

Visualizer should target additional types -- string, and token source

When visualizing a token source, return a VisualizerData instance, but with a null Root. If there's a selected parser class, we can parse the tree using the token source, and return a VisualizerData with the Root filled.

Similar logic applies to the visualization of a string. Once there's a lexer class selected, we can populate the token list. Once there's a parser class, we can populate the root node. The view should be able to handle the token list and root node being empty.

This involves the following parts:

Define the DebuggerVisualizer attribute on both string and CommonTokenStream types
Handle in VisualizerData.cs
The config has to be able to specify whether the selected lexer can be changed (true for string), and whether the selected parser can be changed (true for string and CommonTokenStream).
The UI has to respect the above
When choosing a parser class, the user also needs to choose a parse method (we can try startRule by default, if it exists).
Additionaly, if no lexer has been selected, replace the datagrid in the UI with a message "Select a lexer class from settings to see tokens"
Same for parser class and treeview

For lists of token types and rule contexts in settings, show count of usages instead of ID

Put the ID in a tooltip. It feels more natural for the gray number to represent the count.

Allow viewing entire tree from node in middle of tree

When visualizing a node in the middle of a tree, the current behavior is to only display the subtree of that particular node.

Alternative behavior might be to show the entire tree, but the visualized node should be selected.

Perhaps the second behavior should be the default, and the first could be gotten to using the "Scope to this" context menu option?

@kaby76, would that cover all the bases?

Replace TreeNodeType with ElementType

public enum TreeNodeType {
    RuleContext,
    Token,
    Error
}

with:

public enum ElementType {
    RuleContext,
    TextToken,
    ErrorToken,
    WhitespaceToken
}

MVVM testing

Relevant: https://www.pluralsight.com/courses/wpf-mvvm-test-driven-development-viewmodels

Questions:

Should we test autofill of lexer and parser when there is no selected? Currently we can test this, because there is only one lexer and parser in all the loaded assemblies. But we may add more lexers and parsers in the future.

Allow setting the filler character from the UI for GetPositionedText

Antlr4.Runtime.Standard support

There are two versions of the ANTLR runtime for .NET:

Sam Harwell's original C# target, which complements his .NET ANTLR code generator (used by Rubberduck). This version is not up to date with the latest version of the Java ANTLR generator.
The official ANTLR runtime, called Antlr.Runtime.Standard, which supports the Java ANTLR generator. C# code is generated from the grammar using the ANTLR Java tool.

Currently, we support Sam Harwell's runtime, and we really should support both.

ATM this requires two additional end projects:

Standard.2019 -- for VS 2019
Standard.2017 -- for VS 2017

which reference the Antlr.Runtime.Standard package instead of the Antlr.Runtime package.

AFAICT there are some API differences between the two runtimes; this will have to be accounted for in the shared code somehow.

How would this play with AppVeyor?

Eventually, it would be nice to incorporate everything into a single .csproj file, and use compiler constants to control the output.

When source is generated from tree, channels which don't exist in the tree are not included

For example, given the following grammar:

// define a grammar called Hello
grammar Hello;
r   : 'hello' ID;
ID  : [a-z]+ ;
WS  : [ \t\r\n]+ -> skip ;

the UI might end up as follows:

because the selected node's span is (6, 10) which relates to the source including the skipped whitespace.

A workaround would be to construct the source by iterating over the nodes and inserting a filler character until each node's text would be in the right position.

Related: antlr/antlr4#1302 (comment)

Include VS 2017 debuggers DLL as third-party reference

Currently, the AppVeyor build runs against both the VS 2019 image and the VS 2017 image, in order to get hold of the VS 2017 debugger DLL.

It would be far simpler to add a reference to the DLL without having a complete install of VS 2017.

This would also allow using later C# features that are unsupported in VS 2017.

Context menu options for treeview

Databinding in listboxes

Find some way to do databinding from listboxes and other multiselect controls, and their SelectedItems and SelectedValues properties..

Source text not highlighted

The behavior of #1 is fully realized, but the focus on the textbox somehow doesn't stick.

GridSplitter / DockSplitter

Currently the user can't change the layout. It would be expected that the user could drag the boundaries of each pane.

The standard way to do this is with a Grid and a GridSplitter, which I don't like, because different parts of the layout come out wrong.

I'd prefer to have a DockPanel, with some kind of splitter to adjust.

Alternatively, there are docking controls for WPF; perhaps I could use one of them. I'm wary of doing so, because it means users have to download and copy another DLL into the visualizers subfolder.

Visualization of token stream

Determine parser class from rule context instance

Each rule context is autogenerated as a nested type within the parser class 🤦‍♂

(Thanks @MDoerner.)

There's thus no need to prompt the user for the parser class in order to get hold of the rule names and token types.

Settings persistence

Store JSON in %ProgramData%.

File should be read when visualizer is first started.

File should be written at two points:

after TransferObject returns (as the debuggee might have changed settings)
after PropertyChanged is triggered on settings object

How can we persist parser specific settings? If we have a single settings file, we'll have to find some way of merging settings from multiple parser/lexer classes into a single object.

Alternatively, we could store each lexer / parser settings in its own file. We'd need some sort of naming scheme, as we couldn't use \ from the path as the file name.

We could simply use numbers and store the corresponding assembly / classname in the global settings file.

Selection management control

An independent control, for managing the selection of another control.

Can be used with DataGrid, ListBox among others.

Text filter
Select all
Clear selection
Invert selection

Properties:

Control (wire up in code, for now)'
Filter member path - path on which to apply filter

Navigate to parent node via properties datagrid

Navigation history + back/foward buttons

Show build/version number somewhere

It would be useful to show the version number somewhere on the settings panel:

A smaller font, grey brush perhaps.

Allow scrolling source view

This is important because the text will often not fit in the textbox. Ordinarily it might be less of an issue, because you could scroll the selection to bring various parts into focus. But that doesn't work here, because the selection in the source changes the selection in other parts of the UI.

Consider using the annotater to add nullable-reference attributes to reference assemblies

https://github.com/tunnelvisionlabs/ReferenceAssemblyAnnotator

Load parser rules from file once parser class is loaded

Perhaps display in a tooltip, or some related area on the window?

This depends on #10 .

Assuming of course the parser class contains this information somewhere.

Selection sync

The visualizer currently has three areas whose selections should be synced:

the source
the treeview
the token list

Starting control	Results
Source	Determine the start and end tokens covered by the selection (a selection could include partial tokens). Select those tokens in the token list. Find the deepest tree node whose tokens encompass the selected tokens, and select that node in the treeview
Token list	Select the text in the source based on the first and last selected tokens. (It is possible to select non-contiguous tokens, but in order to reflect that in the source selection, we'd need a multi-selection textbox, which the built-in WPF textbox is not. Perhaps later.) As above, find the deepest tree node whose tokens encompass the selected tokens, and select that node in the treeview.
Treeview	Select the tokens in the token list covered by the tree node. Select the source covered by the tokens.

Filter tree nodes

There should be two filtering options.

based on node type - rulecontext, token or trivia
provide a list of reflection types to be filtered on

Combine multiple DLLs into single final DLL

This would be useful for #5, and also for #9 if we need to include our own copy of JSON.NET.

Use XAML Behaviors for WPF

Currently fails with an exception:

System.Windows.Markup.XamlParseException: 'Could not load file or assembly 'Microsoft.Xaml.Behaviors, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.'

Inner Exception:
FileNotFoundException: Could not load file or assembly 'Microsoft.Xaml.Behaviors, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.

The DLL is in the bin folder.

https://github.com/Microsoft/XamlBehaviorsWpf/wiki