rrevenantt/antlr4rust

ANTLR4 parser generator runtime for Rust programming laguage

antlr4rust

ANTLR4 runtime for Rust programming language.

For examples you can see grammars, tests/gen for corresponding generated code and tests/my_tests.rs for actual usage examples

ANTLR4 Tool(parser generator)

Generator part is currently located in rust-target branch of my antlr4 fork rrevenantt/antlr4/tree/rust-target Latest version is automatically built to releases on this repository. So if you just want to generate parser or if you want to contribute to only runtime part you don't have to do build it yourself.

But if you want to build or change generator yourself:

• git clone -b rust-target https://github.com/rrevenantt/antlr4 - clone my antlr4 fork
• git submodule update --init --recursive --remote - update Rust target submodule
• mvn -DskipTests install - build generator

Implementation status

For now development is going on in this repository but eventually it will be merged to main ANTLR4 repo

Since version 0.3 works on stable rust. Previous versions are not maintained any more so in case of nightly breakage you should migrate to the latest version.

Usage

You should use the ANTLR4 "tool" to generate a parser, that will use the ANTLR runtime located here. You can run it with the following command:

java -jar <path to ANTLR4 tool> -Dlanguage=Rust MyGrammar.g4

For a full list of antlr4 tool options, please visit the tool documentation page.

You can also see build.rs as an example of build.rs configuration to rebuild parser automatically if grammar file was changed.

Then add following to Cargo.toml of the crate from which generated parser is going to be used:

[dependencies]
antlr-rust = "0.3"

Parse Tree structure

It is possible to generate idiomatic Rust syntax trees. For this you would need to use labels feature of ANTLR tool. You can see Labels grammar for example. Consider following rule :

e   : a=e op='*' b=e   # mult
    | left=e '+' b=e   # add
		 

For such rule ANTLR will generate enum EContextAll containing mult and add alternatives, so you will be able to match on them in your code. Also corresponding struct for each alternative will contain fields you labeled. I.e. for MultContext struct will contain a and b fields containing child subtrees and op field with TerminalNode type which corresponds to individual Token. It also is possible to disable generic parse tree creation to keep only selected children via parser.build_parse_trees = false, but unfortunately currently it will prevent visitors from working.

Differences with Java

Although Rust runtime API has been made as close as possible to Java, there are quite some differences because Rust is not an OOP language and is much more explicit.

• If you are using labeled alternatives, struct generated for the rule is an enum with variant for each alternative
• Parser needs to have ownership for listeners, but it is possible to get listener back via ListenerId otherwise ParseTreeWalker should be used.
• In embedded actions to access parser you should use recog variable instead of self/this. This is because predicates have to be inserted into two syntactically different places in generated parser and in one of them it is impossible to have parser as self.
• str based InputStream have different index behavior when there are unicode characters. If you need exactly the same behavior, use [u32] based InputStream, or implement custom CharStream.
• In actions you have to escape ' in rust lifetimes with \ because ANTLR considers them as strings, e.g. Struct<\'lifetime>
• To make custom tokens you should use @tokenfactory custom action, instead of usual TokenLabelType parser option. ANTLR parser options can accept only single identifiers while Rust target needs know about lifetime as well. Also in Rust target TokenFactory is the way to specify token type. As example you can see CSV test grammar.
• All rule context variables (rule argument or rule return) should implement Default + Clone.

Benchmarks

Here is comparison of antlr generated XML lexer and parser (from default XML grammar but with custom minimal Token/TokenFactory/InputStream/RuleContext) to hand-written implementations in rust ecosystem. Keep in mind that xmlparser and quick_xml are much closer to being lexer than parser, so they should be compared with antlr lexer. Also while structs used by generated lexer and parser were customized to track as minimum data as required (which is possible by any user of antlr-rust), internals of the lexer cannot be customized enough yet and still track quite a lot of data that might not be used in particular case. So there is still room for improvement.

lexers:
large/large_xmlparser        time:   [1.8598 ms 1.8607 ms 1.8619 ms]                                   
large/large_quick_xml        time:   [1.4623 ms 1.4645 ms 1.4675 ms]                                   
large/large_antlr_xml_lexer  time:   [5.7866 ms 5.7877 ms 5.7891 ms]
parsers:
large/large_xmlrs            time:   [16.734 ms 16.748 ms 16.766 ms]
large/large_minidom          time:   [7.0639 ms 7.0792 ms 7.0975 ms]                                
large/large_roxmltree        time:   [4.9341 ms 4.9360 ms 4.9380 ms]                                   
large/large_antlr_xml_full   time:   [10.243 ms 10.248 ms 10.252 ms]                                  

Unsafe

Currently, unsafe is used only for downcasting (through separate crate) and to update data inside Rc via get_mut_unchecked(returned mutable reference is used immediately and not stored anywhere)

Versioning

In addition to usual Rust semantic versioning, patch version changes of the crate should not require updating of generator part

Licence

BSD 3-clause. Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in this project by you shall be licensed as above, without any additional terms or conditions.

antlr4rust's Issues

Generated lexer depends on Java function: self.IsNewlineAtPos(-1)

Hello. I've tried to build the official antlr4 gramamr for sql/plsql which gives a successful build but fails on comple

I've used the release for 0.3-beta generator tool:

The Rust compiler errors are

... 14 more similar errors upwards
error[E0424]: expected value, found module `self`
    --> src/plsqllexer.rs:4024:7
     |
4019 |         fn START_CMD_sempred(_localctx: Option<&LexerContext<'input>>, pred_index:isize,
     |            ----------------- this function doesn't have a `self` parameter
...
4024 |                         self.IsNewlineAtPos(-2)
     |                         ^^^^ `self` value is a keyword only available in methods with a `self` parameter
     |
help: add a `self` receiver parameter to make the associated `fn` a method
     |
4019 |         fn START_CMD_sempred(&self, _localctx: Option<&LexerContext<'input>>, pred_index:isize,
     |                              ++++++

Some errors have detailed explanations: E0405, E0424.
For more information about an error, try `rustc --explain E0405`.
error: could not compile `plsql-antlr4` due to 14 previous errors

Issue when parsing grammar with case-insensitive keywords (maybe fragment or precedence related)

I'm using the following pattern to parse keywords in a case-insensitive way:

SELECT  : S E L E C T;
FROM    : F R O M;
WHERE   : W H E R E;
IDENTIFIER
        : [a-zA-Z_] [a-zA-Z_0-9]*
        ;

fragment A  : ('a'|'A');
fragment B  : ('b'|'B');
fragment C  : ('c'|'C');
fragment D  : ('d'|'D');
fragment E  : ('e'|'E');
fragment F  : ('f'|'F');
...

The full grammar and test case can be found at https://github.com/jhorstmann/rust-antlr-case-insensitive-keywords. The lexer and parser are generated by build.rs and the project can be run with cargo run to output the parsed as of a sample query.

select foo as f, bar as b from table where baz limit 10

I see the following output which shows an error message from the parser:

line 1:37 mismatched input 'where' expecting {<EOF>, WHERE, LIMIT}
[src/main.rs:55] query = Query {
    columns: [
        Column {
            name: "foo",
            alias: Some(
                "f",
            ),
        },
        Column {
            name: "bar",
            alias: Some(
                "b",
            ),
        },
    ],
    from: "table",
    filter: None,
    limit: None,
}

It's interesting that other keywords like AS or FROM seem to be parsed fine. A similar grammar in a java project also handles all keywords like this in a case insensitive way.

My guess would be that it's somehow related to the precedence of the lexer rules between, where the keywords should take precedence because they are listed before the IDENTIFIER rule in the grammar.

NullPointerException

Hi I am building antlr binary and use it to generate my parser and I saw a NPE in the runtime
Here is what I did

git clone -b rust-target https://github.com/rrevenantt/antlr4
git submodule update --init --recursive --remote
mvn -DskipTests install

Then I get

tool/target/antlr4-4.8-2-SNAPSHOT-complete.jar

Run it to against Expr.g4

grammar Expr;
prog:   (expr NEWLINE)* ;
expr:   expr ('*'|'/') expr
    |   expr ('+'|'-') expr
    |   INT
    |   '(' expr ')'
    ;
NEWLINE : [\r\n]+ ;
INT     : [0-9]+ ;

java -jar antlr4-4.8-2-SNAPSHOT-complete.jar -Dlanguage=Rust Expr.g4 -o generated/expr -visitor
error(31):  ANTLR cannot generate Rust code as of version 4.8
Exception in thread "main" java.lang.NullPointerException
        at org.antlr.v4.analysis.LeftRecursiveRuleAnalyzer.getArtificialOpPrecRule(LeftRecursiveRuleAnalyzer.java:211)
        at org.antlr.v4.analysis.LeftRecursiveRuleTransformer.translateLeftRecursiveRule(LeftRecursiveRuleTransformer.java:116)
        at org.antlr.v4.analysis.LeftRecursiveRuleTransformer.translateLeftRecursiveRules(LeftRecursiveRuleTransformer.java:71)
        at org.antlr.v4.semantics.SemanticPipeline.process(SemanticPipeline.java:71)
        at org.antlr.v4.Tool.processNonCombinedGrammar(Tool.java:382)
        at org.antlr.v4.Tool.process(Tool.java:369)
        at org.antlr.v4.Tool.processGrammarsOnCommandLine(Tool.java:328)
        at org.antlr.v4.Tool.main(Tool.java:172)

Master not compatible with rust-target branch of antlr4 fork

Is the antlr4 fork, https://github.com/rrevenantt/antlr4/tree/rust-target, up to date?

It seems the the Rust.stg doesn't generate code that is compatible with the master branch of antlr4rust.

Auto-substitution of TokenSource methods ?

For example, I have the following rule

LINE_COMMENT
    : {getCharPositionInLine() == 0}? (' ' | '\t')*? '--' ~('\n'|'\r')* '\r'? '\n' ->skip
    ;

It would generate the following lexer in Rust

impl ASN_3gppLexerActions{

  fn LINE_COMMENT_sempred(_localctx: &LexerContext, pred_index:isize,
            recog:&mut <Self as Actions>::Recog
    ) -> bool {
    match pred_index {
        0=>{
          getCharPositionInLine() == 0
        }
      _ => true
    }
  }
}

Which results in a compilation error since Rust lexer does not have getCharPositionInLine(). Instaed, it has get_char_position_in_line().

I think it would be better if Rust runtime automatically substitutes Java-specific TokenSource methods including getCharPositioninInLine() to recog.get_char_position_in_line()

ANTLR Generates invalid Rust for Java 9 grammar

As stated in the title, I tried to generate a parser for the official Java9 grammar in the antlr/grammars-v5 repository and it generated the following code that causes syntax and import errors:

	impl<'input, Input:CharStream<From<'input> >> Java9Lexer<'input,Input>{
		fn JavaLetter_sempred(_localctx: Option<&LexerContext<'input>>, pred_index:isize,
							recog:&mut <Self as Deref>::Target
			) -> bool {
			match pred_index {
					0=>{
						Character.isJavaIdentifierStart(_input.LA(-1))
					}
					1=>{
						Character.isJavaIdentifierStart(Character.toCodePoint((char)_input.LA(-2), (char)_input.LA(-1)))
					}
				_ => true
			}
		}
		fn JavaLetterOrDigit_sempred(_localctx: Option<&LexerContext<'input>>, pred_index:isize,
							recog:&mut <Self as Deref>::Target
			) -> bool {
			match pred_index {
					2=>{
						Character.isJavaIdentifierPart(_input.LA(-1))
					}
					3=>{
						Character.isJavaIdentifierPart(Character.toCodePoint((char)_input.LA(-2), (char)_input.LA(-1)))
					}
				_ => true
			}
		}


}

I did not change any indentation. Apart from that, it requires some Character structure which is not imported and I do not know where it is defined. I looked through the generated files and it is not defined there. (char)_input.LA(-2) seems to be a Java leftover. I think that this code originally casted a Java int to a char, which does not exist in Rust.

The raw syntax errors:

cannot find value `Character` in this scope
not found in this scope rustc(E0425)

cannot find value `_input` in this scope
not found in this scope rustc(E0425)

missing `,` rustc
java9lexer.rs (340, 67): original diagnostic
Syntax Error: expected COMMA rust-analyzer
Syntax Error: expected COMMA rust-analyzer
Syntax Error: expected SEMICOLON rust-analyzer
expected one of `)`, `,`, `.`, `?`, or an operator, found `_input`
expected one of `)`, `,`, `.`, `?`, or an operator rustc
java9lexer.rs (340, 67): missing `,`

Slow parsing performance

I'm using antlr4rust to build a Kotlin parser. I was able to successfully generate the parser using the v0.3 branch of this library and I can now parse Kotlin programs. However, I'm noticing that parsing is very slow. Parsing a small Kotlin script takes over two seconds. One reason I wanted to write this parser in Rust was for performance, but this feels on par with my JVM-based parsers I've had in the past and those weren't fast enough for my needs.

Is there anything I might be doing wrong? Here's my code so far: https://github.com/ionic-team/trapeze/tree/main/packages/parser

In the meantime I'm going to try the C++ runtime and compare parsing the same program. Will follow up once I have results.

Splitting lexer and parser grammar into separate files causes error

To repro:

• use grammars here: https://github.com/sepp2k/antlr4-string-interpolation-examples/tree/master/with-duplication

• run commands

  • java -jar ./antlr4-4.8-2-SNAPSHOT-complete.jar -Dlanguage=Rust StringLexer.g4
  • java -jar ./antlr4-4.8-2-SNAPSHOT-complete.jar -Dlanguage=Rust StringParser.g4

• Observe errors, e.g:
  error[E0412]: cannot find type `StringParserParserContextType` in this scope --> src/parsing/jasm/gen/stringparser.rs:532:49 | 156 | pub struct StringParserContextType; | ----------------------------------- similarly named struct `StringParserContextType` defined here ... 532 | fn RPAR(&self) -> Option<Rc<TerminalNode<'input,StringParserParserContextType>>> where Self:Sized{ | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |

As I understand, there is no way to use lexer modes (e.g. in that example) without separate files. Any way to get this to work?

Thank you!
Stuart

Anything we can help to improve on this project?

The motivation in this project is great in the rust community but it seems to be inactive for quite a long time. Would the author mind sharing the current status, and giving advice on how contributors can help to improve?

Not building anymore on new versions

Repo needs an update

Compiling antlr-rust v0.2.2 (/home/raja/work/rust/antlr4rust)
error[E0557]: feature has been removed
--> src/lib.rs:11:12
|
11 | #![feature(crate_visibility_modifier)]
| ^^^^^^^^^^^^^^^^^^^^^^^^^ feature has been removed
|
= note: removed in favor of pub(crate)

error: expected one of ! or ::, found keyword fn
--> src/atn.rs:58:11
|
57 | impl ATN {
| - while parsing this item list starting here
58 | crate fn new_atn(grammar_type: ATNType, max_token_type: isize) -> ATN {
| ^^ expected one of ! or ::
...
175 | }
| - the item list ends here

error: expected one of ! or ::, found keyword struct
--> src/lexer.rs:113:7
|
113 | crate struct LexerPosition {
| ^^^^^^ expected one of ! or ::

error[E0432]: unresolved imports lexer::BaseLexer, lexer::Lexer
--> src/lib.rs:104:17
|
104 | pub use lexer::{BaseLexer, Lexer};
| ^^^^^^^^^ ^^^^^ no Lexer in lexer
| |
| no BaseLexer in lexer

error[E0432]: unresolved import crate::lexer::Lexer
--> src/lexer_action.rs:3:5
|
3 | use crate::lexer::Lexer;
| ^^^^^^^^^^^^^^^^^^^ no Lexer in lexer

error[E0432]: unresolved imports crate::lexer::LEXER_MAX_CHAR_VALUE, crate::lexer::LEXER_MIN_CHAR_VALUE
--> src/transition.rs:7:20
|
7 | use crate::lexer::{LEXER_MAX_CHAR_VALUE, LEXER_MIN_CHAR_VALUE};
| ^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^ no LEXER_MIN_CHAR_VALUE in lexer
| |
| no LEXER_MAX_CHAR_VALUE in lexer

error[E0432]: unresolved import crate::lexer::Lexer
--> src/lexer_action_executor.rs:6:5
|
6 | use crate::lexer::Lexer;
| ^^^^^^^^^^^^^^^^^^^ no Lexer in lexer

error[E0432]: unresolved imports crate::lexer::Lexer, crate::lexer::LexerPosition, crate::lexer::LEXER_MAX_CHAR_VALUE, crate::lexer::LEXER_MIN_CHAR_VALUE
--> src/lexer_atn_simulator.rs:21:20
|
21 | use crate::lexer::{Lexer, LexerPosition, LEXER_MAX_CHAR_VALUE, LEXER_MIN_CHAR_VALUE};
| ^^^^^ ^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^ no LEXER_MIN_CHAR_VALUE in lexer
| | | |
| | | no LEXER_MAX_CHAR_VALUE in lexer
| | no LexerPosition in lexer
| no Lexer in lexer

warning: unused attribute allow
--> src/token_factory.rs:17:1
|
17 | #[allow(non_upper_case_globals)]
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= note: #[warn(unused_attributes)] on by default
note: the built-in attribute allow will be ignored, since it's applied to the macro invocation lazy_static
--> src/token_factory.rs:18:1
|
18 | lazy_static! {
| ^^^^^^^^^^^

warning: unused import: crate::dfa::ScopeExt
--> src/atn.rs:6:5
|
6 | use crate::dfa::ScopeExt;
| ^^^^^^^^^^^^^^^^^^^^
|
= note: #[warn(unused_imports)] on by default

warning: unused import: crate::interval_set::IntervalSet
--> src/atn.rs:7:5
|
7 | use crate::interval_set::IntervalSet;
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

warning: unused import: crate::ll1_analyzer::LL1Analyzer
--> src/atn.rs:9:5
|
9 | use crate::ll1_analyzer::LL1Analyzer;
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

warning: unused import: crate::parser::ParserNodeType
--> src/atn.rs:10:5
|
10 | use crate::parser::ParserNodeType;
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

warning: unused import: crate::rule_context::EmptyContextType
--> src/atn.rs:11:5
|
11 | use crate::rule_context::EmptyContextType;
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

warning: unused imports: TOKEN_EOF, TOKEN_EPSILON
--> src/atn.rs:12:20
|
12 | use crate::token::{TOKEN_EOF, TOKEN_EPSILON};
| ^^^^^^^^^ ^^^^^^^^^^^^^

warning: unused import: crate::token_factory::CommonTokenFactory
--> src/atn.rs:13:5
|
13 | use crate::token_factory::CommonTokenFactory;
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

warning: unused import: crate::transition::RuleTransition
--> src/atn.rs:14:5
|
14 | use crate::transition::RuleTransition;
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Some errors have detailed explanations: E0432, E0557.
For more information about an error, try rustc --explain E0432.
warning: antlr-rust (lib) generated 9 warnings
error: could not compile antlr-rust due to 8 previous errors; 9 warnings emitted

`enter_XXX` for alternative labels are not invoked

enter_XXX for alternative labels are not invoked, while enter_every_rule and enter_XXX for rules are called.

Given the grammar:

grammar Test;

prog
    : ident EOF
    ;

ident
    : 'tohka'   # Tohka
    ;

and the input tohka, the following program

#![feature(try_blocks)]
#![feature(exclusive_range_pattern)]

#[macro_use]
extern crate lazy_static;

#[allow(warnings)]
pub mod testlexer;
#[allow(warnings)]
pub mod testlistener;
#[allow(warnings)]
pub mod testparser;

use testlexer::*;
use testlistener::*;
use testparser::*;

use antlr_rust::common_token_stream::CommonTokenStream;
use antlr_rust::input_stream::InputStream;
use antlr_rust::tree::ParseTreeListener;

struct Listener;

impl ParseTreeListener for Listener {}

impl TestListener for Listener {
    fn enter_prog(&mut self, _ctx: &ProgContext) {
        println!("enter prog");
    }

    fn exit_prog(&mut self, _ctx: &ProgContext) {
        println!("exit prog");
    }

    fn enter_Tohka(&mut self, _ctx: &TohkaContext) {
        println!("enter Tohka");
    }

    fn exit_Tohka(&mut self, _ctx: &TohkaContext) {
        println!("exit Tohka");
    }
}

fn main() {
    let lexer = TestLexer::new(Box::new(InputStream::new("tohka".into())));
    let token_source = CommonTokenStream::new(lexer);
    let mut parser = TestParser::new(Box::new(token_source));
    let listener = Listener;

    let id = parser.add_parse_listener(Box::new(listener));
    parser.prog().expect("failed to parse");

    let _ = parser.remove_parse_listener(id);
}

prints

enter prog
exit Tohka
exit prog

which should be like the following:

enter prog
enter Tohka
exit Tohka
exit prog

Generating Listenable::exit

There doesn't seem to be any code generated for the exit method of listeners. E.g. it generates

impl<'input,'a> Listenable<dyn GroundedListener<'input> + 'a> for RulesContext<'input>{
	fn enter(&self,listener: &mut (dyn GroundedListener<'input> + 'a)) {
		listener.enter_every_rule(self);
		listener.enter_rules(self);
	}
}  

when I expect

impl<'input,'a> Listenable<dyn GroundedListener<'input> + 'a> for RulesContext<'input>{
	fn enter(&self,listener: &mut (dyn GroundedListener<'input> + 'a)) {
		listener.enter_every_rule(self);
		listener.enter_rules(self);
	}
	fn exit(&self,listener: &mut (dyn GroundedListener<'input> + 'a)) {
		listener.exit_rules(self);
		listener.exit_every_rule(self);
	}
}

This results in the walker not being able to perform post-order traversal. Am I missing something to generate these methods?

Build instructions are not correct for V0.3 prerelease

You seem to have forgot that your antlr fork still contain and old pointer for the git submodule Rust@76d0a7d
You should update it or add instruction to update it : git submodule update --remote --merge
Actually a user following your instructions will have a hard way to understand what is wrong with the generated code but nothing will work.
(Beside that, i have done some extensive tests with V0.3 prerelease and everything is fine, good job thanks)

Listenable trait is not fully implemented for labeled contexts

exit() method from Listenable trait is not implemented for labeled contexts. As a result, fn exit_or(&mut self, _ctx: &OrContext<'a>) not being called when walking the generated tree :(

boolExpr:
  ...
  | boolExpr '||'  boolExpr                                  #or
  ...

See also impl<'input, 'a> Listenable<dyn LabelsListener<'input> + 'a> for AddContext<'input>

Add sponsoring

I am using this and would like to support the project by sponsoring it.

It does not matter if no further development is being done right now, but I would like to show some appreciation.

I would surely be happy if further development is done, but that is no must!

Integer overflow w/ generated lexer

I was attempting to compile a lexer generated w/ the Rust backend, and I got the following errors on my tokens:

error: this arithmetic operation will overflow
     |
8379 | ...ze << (FLOAT - 1)) | (1usize << (VALID_IDENTIFIER - 1)) | (1usize << (FIELD - 1)) | (1usize << (INDEXED_FIELD - 1)))) != 0) {
     |                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ attempt to shift left by `61_isize`, which would overflow

I think this is because the template portion that is doing a bitset emulation of the Antlr code is presuming that 1usize is 64-bit width trying to match the original implementation?.

If my understanding here is correct, should it be that the template is instead masking with 1u64 instead ? Happy to provide a PR if this is right.

BailErrorStrategy calls unimplemented set_exception method

BaseLexer deref / superClass

I am trying to use antlr4rust with a stateful lexer that is already implemented in C++ and TypeScript.

It seems the best way to do this is to add to the lexer fields / members, but there's no way to reference them from from the BaseLexer. BaseParser seems to deref to its inner ext, so I thought that BaseLexer should probably be doing something similar by derefing to its inner recog.

Here's the change I am thinking of:
dfrankland@6f3ac16

Does that seem correct?

[Feature Request] add 'input lifetime parameter to ParserExt defination

To allow ParserExt hold reference on input string

Annotating parse trees?

Hello,
Is there a way to annotate parse trees (equivalent of https://www.antlr.org/api/Java/org/antlr/v4/runtime/tree/ParseTreeProperty.html in java)?

I'm using 0.3.0-beta

Visitor's return type

Comparing the Rust's ParseTreeVisitor to the TypeScript version I can see that the TS version returns a generic object.

Should the Rust version be doing the same?

Currently attempting to convert a JS library to Rust so it can be used with WASM

how did you learn how to do this port?

I'd like to do an ANTLR port (to OCaml) but haven't actually looked into how to get it done. I saw you'd done this port, and wondered if you had pointers to where to start learning how to port ANTLR to a new language runtime ?

How to use listeners and visitors in 0.2?

The code in my_test.rs indicates that a parser should have an add_parse_listener function that a listener can be paused to to run during parsing?

However, my parser in the code below has no such function:

let tf = CommonTokenFactory::default();
let input_stream = InputStream::new("test");
let lexer = OCRLexer::new_with_token_factory(input_stream, &tf);
let token_stream = CommonTokenStream::new(lexer);
let mut parser = OCRParser::new(token_stream);

// parser has no add_parse_listener

How can I add a listener to the parser, similar to these lines from my_test.rs?

let mut parser = CSVParser::new(token_source);
parser.add_parse_listener(Box::new(Listener {}));
println!("\nstart parsing parser_test_csv");
let result = parser.csvFile();
assert!(result.is_ok());
assert_eq!(
    result.unwrap().to_string_tree(&*parser),
    "(csvFile (hdr (row (field V123) , (field V2) \\n)) (row (field d1) , (field d2) \\n))"
);

Return Types for Visitors

I was wondering if there is any plan to support parse tree visitors that return something as in the Java implementation of antlr. I find this is a convenient way to write a visitor that constructs AST node objects. Concretely, the current implementation of visit_terminal has signature:
fn visit_terminal(self, TerminalNode<...>) {},

but if it were instead

fn visit_terminal<T>(self, TerminalNode<...>) -> T {}
(and similar for the other functions in the ParseTreeVisitor trait)

then it would be possible to write a visitor that builds an AST by returning the AST nodes with each of these calls.

File tests/my_tests.rs does not exist

The file tests/my_tests.rs is linked from the README, but does not exist. master branch 360bc02

ANTLR cannot generate Rust code as of version 4.13.1

I get this error:

error(31):  ANTLR cannot generate Rust code as of version 4.13.1

I've followed the steps described:

git clone -b rust-target https://github.com/rrevenantt/antlr4
git submodule update --init --recursive --remote 
mvn -DskipTests install

Running the following fails:

antlr4 -Dlanguage=Rust Idl.g4

I was trying to build Avro's g4:
https://github.com/apache/avro/blob/main/share/idl_grammar/org/apache/avro/idl/Idl.g4

Generator is incompatible with runtime

After I download the generator release 0.3.0-beta, generate parser using it and try to run it using master branch of this repo (or crate 0.3.0-beta from crates.io), then I encounter a lot of incompatibility errors:

error[E0433]: failed to resolve: could not find `impl_tid` in `antlr_rust`
   --> src/frontend/parser/latteparser.rs:206:15
    |
206 | #[antlr_rust::impl_tid]
    |               ^^^^^^^^ could not find `impl_tid` in `antlr_rust`

error[E0433]: failed to resolve: could not find `type_id` in `antlr_rust`
   --> src/frontend/parser/latteparser.rs:210:13
    |
210 | antlr_rust::type_id!{LatteParserContextType}
    |             ^^^^^^^ could not find `type_id` in `antlr_rust`
...    
error: cannot find derive macro `Tid` in this scope
   --> src/frontend/parser/lattelexer.rs:118:10
    |
118 | #[derive(Tid)]
    |          ^^^
    |
...
error[E0200]: the trait `TidAble<'input>` requires an `unsafe impl` declaration
   --> src/frontend/parser/latteparser.rs:204:1
    |
204 | impl<'input> antlr_rust::TidAble<'input> for dyn LatteParserContext<'input> + 'input{}
    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = note: the trait `TidAble<'input>` enforces invariants that the compiler can't check. Review the trait documentation and make sure this implementation upholds those invariants before adding the `unsafe` keyword
help: add `unsafe` to this trait implementation
    |
204 | unsafe impl<'input> antlr_rust::TidAble<'input> for dyn LatteParserContext<'input> + 'input{}
    | ++++++

all of which, I believe, originate from better_any crate. So far, I haven't managed to solve it myself.

As I need to use your great crate ASAP, I'd be very grateful for quick help. @rrevenantt

Build fails with latest Rust nightly

Reproducible steps:

1. git clone https://github.com/rrevenantt/antlr4rust.git
2. cargo +nightly build

Expected: build completing successfully
Got:

error[E0635]: unknown feature `raw`
 --> src/lib.rs:4:12
  |
4 | #![feature(raw)]
  |            ^^^

Version:

$ rustc +nightly --version
rustc 1.56.0-nightly (371f3cd3f 2021-09-02)

Is It possible that generate AST for g4 file dynamically?

As I know, there is a plugin in Intellij IDEA that can preview the AST on the fly, is it possible in pure rust?

could not find `impl_tid` in `antlr_rust`

Situation:

• found version 0.3-beta on crates.io (no final 0.3 to be found anywhere)
• could not find the matching parser generator (antlr4.jar) either. Therefore, compiled it with maven from https://github.com/rrevenantt/antlr4/tree/rust-target (rust-target branch from today)
• Using grammar: https://github.com/antlr/grammars-v4/blob/master/c/C.g4
• Used Rust: from Debian Unstable; rustc 1.63.0

Problem: the gencode does not compile!

error[E0433]: failed to resolve: could not find `impl_tid` in `antlr_rust`
   --> src/cparser.rs:419:15
    |
419 | #[antlr_rust::impl_tid]
    |               ^^^^^^^^ could not find `impl_tid` in `antlr_rust`

Official Java identifier rule breaks produces a lexer that doesn't compile.

The following grammar file>

`grammar Colloid;

identifier: Identifier;

// lexer

TERMINATOR : '\n';
TAB : '\t';

WS : [ \r\u000C]+ -> skip;
COMMENT : '/' .? '/' -> skip;
LINE_COMMENT : '//' ~[\r\n] -> skip;

//Identifier :
// 'a' .. 'z';

Identifier
: JavaLetter JavaLetterOrDigit*
;

fragment
JavaLetter
: [a-zA-Z$_] // these are the "java letters" below 0x7F
| // covers all characters above 0x7F which are not a surrogate
~[\u0000-\u007F\uD800-\uDBFF]
{Character.isJavaIdentifierStart(_input.LA(-1))}?
| // covers UTF-16 surrogate pairs encodings for U+10000 to U+10FFFF
[\uD800-\uDBFF] [\uDC00-\uDFFF]
{Character.isJavaIdentifierStart(Character.toCodePoint((char)_input.LA(-2), (char)_input.LA(-1)))}?
;

fragment
JavaLetterOrDigit
: [a-zA-Z0-9$_] // these are the "java letters or digits" below 0x7F
| // covers all characters above 0x7F which are not a surrogate
~[\u0000-\u007F\uD800-\uDBFF]
{Character.isJavaIdentifierPart(_input.LA(-1))}?
| // covers UTF-16 surrogate pairs encodings for U+10000 to U+10FFFF
[\uD800-\uDBFF] [\uDC00-\uDFFF]
{Character.isJavaIdentifierPart(Character.toCodePoint((char)_input.LA(-2), (char)_input.LA(-1)))}?
;`

Produces this Rust lexer:

`
// Generated from Colloid.g4 by ANTLR 4.8
#![allow(dead_code)]
#![allow(non_snake_case)]
#![allow(non_upper_case_globals)]
#![allow(unused_imports)]
use antlr_rust::atn::ATN;
use antlr_rust::char_stream::CharStream;
use antlr_rust::lexer::{BaseLexer, Lexer, LexerRecog};
use antlr_rust::atn_deserializer::ATNDeserializer;
use antlr_rust::dfa::DFA;
use antlr_rust::lexer_atn_simulator::{LexerATNSimulator, ILexerATNSimulator};
use antlr_rust::PredictionContextCache;
use antlr_rust::recognizer::{Recognizer,Actions};
use antlr_rust::error_listener::ErrorListener;
use antlr_rust::token_source::TokenSource;
use antlr_rust::common_token_factory::TokenFactory;
use antlr_rust::token::*;
use antlr_rust::rule_context::BaseRuleContext;
use antlr_rust::parser_rule_context::{ParserRuleContext,LexerContext,cast};
use antlr_rust::vocabulary::{Vocabulary,VocabularyImpl};

use std::sync::Arc;
use std::cell::RefCell;
use std::rc::Rc;
use std::ops::{Deref, DerefMut};

	pub const TERMINATOR:isize=1; 
	pub const TAB:isize=2; 
	pub const WS:isize=3; 
	pub const COMMENT:isize=4; 
	pub const LINE_COMMENT:isize=5; 
	pub const Identifier:isize=6;
pub const channelNames: [&'static str;0+2] = [
	"DEFAULT_TOKEN_CHANNEL", "HIDDEN"
];

pub const modeNames: [&'static str;1] = [
	"DEFAULT_MODE"
];

pub const ruleNames: [&'static str;8] = [
	"TERMINATOR", "TAB", "WS", "COMMENT", "LINE_COMMENT", "Identifier", "JavaLetter", 
	"JavaLetterOrDigit"
];


pub const _LITERAL_NAMES: [Option<&'static str>;3] = [
	None, Some("'\n'"), Some("'\t'")
];
pub const _SYMBOLIC_NAMES: [Option<&'static str>;7]  = [
	None, Some("TERMINATOR"), Some("TAB"), Some("WS"), Some("COMMENT"), Some("LINE_COMMENT"), 
	Some("Identifier")
];
lazy_static!{
    static ref _shared_context_cache: Arc<PredictionContextCache> = Arc::new(PredictionContextCache::new());
	static ref VOCABULARY: Box<dyn Vocabulary> = Box::new(VocabularyImpl::new(_LITERAL_NAMES.iter(), _SYMBOLIC_NAMES.iter(), None));
}

pub struct ColloidLexer {
base: BaseLexer,
// static { RuntimeMetaData.checkVersion("4.8", RuntimeMetaData.VERSION); }
}

impl Deref for ColloidLexer{
type Target = BaseLexer;

fn deref(&self) -> &Self::Target {
	&self.base
}

}

impl DerefMut for ColloidLexer{
fn deref_mut(&mut self) -> &mut Self::Target {
&mut self.base
}
}

impl ColloidLexer {
fn get_rule_names(&self) -> &'static [&'static str] {
&ruleNames
}
fn get_literal_names(&self) -> &[Option<&str>] {
&_LITERAL_NAMES
}

fn get_symbolic_names(&self) -> &[Option<&str>] {
    &_SYMBOLIC_NAMES
}

fn add_error_listener(&mut self, _listener: Box<dyn ErrorListener>) {
    self.base.add_error_listener(_listener);
}

fn remove_error_listeners(&mut self) {
    self.base.remove_error_listeners()
}

fn get_grammar_file_name(&self) -> &'static str {
    "ColloidLexer.g4"
}

pub fn new (input: Box<dyn CharStream>) -> Self {
	antlr_rust::recognizer::check_version("0","1");
	Self {
		base: BaseLexer::new_base_lexer(
			input,
			LexerATNSimulator::new_lexer_atnsimulator(
				_ATN.clone(),
				_decision_to_DFA.clone(),
				_shared_context_cache.clone(),
			),
			Box::new(ColloidLexerActions{})
		)
    }
}

}

pub struct ColloidLexerActions {
}

impl ColloidLexerActions{
}

impl LexerRecog for ColloidLexerActions{
}

impl Recognizer for ColloidLexerActions {}

impl Actions for ColloidLexerActions{
type Recog = BaseLexer;
fn sempred(localctx: &dyn ParserRuleContext, rule_index: isize, pred_index: isize,
recog:&mut ::Recog
) -> bool {
match rule_index {
6 =>
Self::JavaLetter_sempred(cast::<,LexerContext >(localctx), pred_index, recog),
7 =>
Self::JavaLetterOrDigit_sempred(cast::<,LexerContext >(_localctx), pred_index, recog),
_ => true
}
}

}

impl ColloidLexerActions{

	fn JavaLetter_sempred(_localctx: &LexerContext, pred_index:isize,
						recog:&mut <Self as Actions>::Recog
		) -> bool {
		match pred_index {
				0=>{
					Character.isJavaIdentifierStart(_input.LA(-1))
				}
				1=>{
					Character.isJavaIdentifierStart(Character.toCodePoint((char)_input.LA(-2), (char)_input.LA(-1)))
				}
			_ => true
		}
	}

	fn JavaLetterOrDigit_sempred(_localctx: &LexerContext, pred_index:isize,
						recog:&mut <Self as Actions>::Recog
		) -> bool {
		match pred_index {
				2=>{
					Character.isJavaIdentifierPart(_input.LA(-1))
				}
				3=>{
					Character.isJavaIdentifierPart(Character.toCodePoint((char)_input.LA(-2), (char)_input.LA(-1)))
				}
			_ => true
		}
	}

}

impl TokenSource for ColloidLexer {
fn next_token(&mut self) -> Box {
self.base.next_token()
}

fn get_line(&self) -> isize {
    self.base.get_line()
}

fn get_char_position_in_line(&self) -> isize {
    self.base.get_char_position_in_line()
}

fn get_input_stream(&mut self) -> &mut dyn CharStream {
    self.base.get_input_stream()
}

fn get_source_name(&self) -> String {
	self.base.get_source_name()
}

fn get_token_factory(&self) -> &dyn TokenFactory {
    self.base.get_token_factory()
}

}

lazy_static! {
    static ref _ATN: Arc<ATN> =
        Arc::new(ATNDeserializer::new(None).deserialize(_serializedATN.chars()));
    static ref _decision_to_DFA: Arc<Vec<DFA>> = {
        let mut dfa = Vec::new();
        let size = _ATN.decision_to_state.len();
        for i in 0..size {
            dfa.push(DFA::new(
                _ATN.clone(),
                _ATN.get_decision_state(i),
                i as isize,
            ))
        }
        Arc::new(dfa)
    };
}



const _serializedATN:&'static str =
	"\x03\u{608b}\u{a72a}\u{8133}\u{b9ed}\u{417c}\u{3be7}\u{7786}\u{5964}\x02\
	\x08\x4e\x08\x01\x04\x02\x09\x02\x04\x03\x09\x03\x04\x04\x09\x04\x04\x05\
	\x09\x05\x04\x06\x09\x06\x04\x07\x09\x07\x04\x08\x09\x08\x04\x09\x09\x09\
	\x03\x02\x03\x02\x03\x03\x03\x03\x03\x04\x06\x04\x19\x0a\x04\x0d\x04\x0e\
	\x04\x1a\x03\x04\x03\x04\x03\x05\x03\x05\x03\x05\x03\x05\x07\x05\x23\x0a\
	\x05\x0c\x05\x0e\x05\x26\x0b\x05\x03\x05\x03\x05\x03\x05\x03\x05\x03\x05\
	\x03\x06\x03\x06\x03\x06\x03\x06\x07\x06\x31\x0a\x06\x0c\x06\x0e\x06\x34\
	\x0b\x06\x03\x06\x03\x06\x03\x07\x03\x07\x07\x07\x3a\x0a\x07\x0c\x07\x0e\
	\x07\x3d\x0b\x07\x03\x08\x03\x08\x03\x08\x03\x08\x03\x08\x03\x08\x05\x08\
	\x45\x0a\x08\x03\x09\x03\x09\x03\x09\x03\x09\x03\x09\x03\x09\x05\x09\x4d\
	\x0a\x09\x03\x24\x02\x0a\x03\x03\x05\x04\x07\x05\x09\x06\x0b\x07\x0d\x08\
	\x0f\x02\x11\x02\x03\x02\x09\x04\x02\x0e\x0f\x22\x22\x04\x02\x0c\x0c\x0f\
	\x0f\x06\x02\x26\x26\x43\x5c\x61\x61\x63\x7c\x04\x02\x02\u{81}\u{10802}\
	\u{10c01}\x03\x02\u{10802}\u{10c01}\x03\x02\u{10c02}\u{e001}\x07\x02\x26\
	\x26\x32\x3b\x43\x5c\x61\x61\x63\x7c\x02\x53\x02\x03\x03\x02\x02\x02\x02\
	\x05\x03\x02\x02\x02\x02\x07\x03\x02\x02\x02\x02\x09\x03\x02\x02\x02\x02\
	\x0b\x03\x02\x02\x02\x02\x0d\x03\x02\x02\x02\x03\x13\x03\x02\x02\x02\x05\
	\x15\x03\x02\x02\x02\x07\x18\x03\x02\x02\x02\x09\x1e\x03\x02\x02\x02\x0b\
	\x2c\x03\x02\x02\x02\x0d\x37\x03\x02\x02\x02\x0f\x44\x03\x02\x02\x02\x11\
	\x4c\x03\x02\x02\x02\x13\x14\x07\x0c\x02\x02\x14\x04\x03\x02\x02\x02\x15\
	\x16\x07\x0b\x02\x02\x16\x06\x03\x02\x02\x02\x17\x19\x09\x02\x02\x02\x18\
	\x17\x03\x02\x02\x02\x19\x1a\x03\x02\x02\x02\x1a\x18\x03\x02\x02\x02\x1a\
	\x1b\x03\x02\x02\x02\x1b\x1c\x03\x02\x02\x02\x1c\x1d\x08\x04\x02\x02\x1d\
	\x08\x03\x02\x02\x02\x1e\x1f\x07\x31\x02\x02\x1f\x20\x07\x2c\x02\x02\x20\
	\x24\x03\x02\x02\x02\x21\x23\x0b\x02\x02\x02\x22\x21\x03\x02\x02\x02\x23\
	\x26\x03\x02\x02\x02\x24\x25\x03\x02\x02\x02\x24\x22\x03\x02\x02\x02\x25\
	\x27\x03\x02\x02\x02\x26\x24\x03\x02\x02\x02\x27\x28\x07\x2c\x02\x02\x28\
	\x29\x07\x31\x02\x02\x29\x2a\x03\x02\x02\x02\x2a\x2b\x08\x05\x02\x02\x2b\
	\x0a\x03\x02\x02\x02\x2c\x2d\x07\x31\x02\x02\x2d\x2e\x07\x31\x02\x02\x2e\
	\x32\x03\x02\x02\x02\x2f\x31\x0a\x03\x02\x02\x30\x2f\x03\x02\x02\x02\x31\
	\x34\x03\x02\x02\x02\x32\x30\x03\x02\x02\x02\x32\x33\x03\x02\x02\x02\x33\
	\x35\x03\x02\x02\x02\x34\x32\x03\x02\x02\x02\x35\x36\x08\x06\x02\x02\x36\
	\x0c\x03\x02\x02\x02\x37\x3b\x05\x0f\x08\x02\x38\x3a\x05\x11\x09\x02\x39\
	\x38\x03\x02\x02\x02\x3a\x3d\x03\x02\x02\x02\x3b\x39\x03\x02\x02\x02\x3b\
	\x3c\x03\x02\x02\x02\x3c\x0e\x03\x02\x02\x02\x3d\x3b\x03\x02\x02\x02\x3e\
	\x45\x09\x04\x02\x02\x3f\x40\x0a\x05\x02\x02\x40\x45\x06\x08\x02\x02\x41\
	\x42\x09\x06\x02\x02\x42\x43\x09\x07\x02\x02\x43\x45\x06\x08\x03\x02\x44\
	\x3e\x03\x02\x02\x02\x44\x3f\x03\x02\x02\x02\x44\x41\x03\x02\x02\x02\x45\
	\x10\x03\x02\x02\x02\x46\x4d\x09\x08\x02\x02\x47\x48\x0a\x05\x02\x02\x48\
	\x4d\x06\x09\x04\x02\x49\x4a\x09\x06\x02\x02\x4a\x4b\x09\x07\x02\x02\x4b\
	\x4d\x06\x09\x05\x02\x4c\x46\x03\x02\x02\x02\x4c\x47\x03\x02\x02\x02\x4c\
	\x49\x03\x02\x02\x02\x4d\x12\x03\x02\x02\x02\x09\x02\x1a\x24\x32\x3b\x44\
	\x4c\x03\x08\x02\x02";

`

The compilation of which produces the following errors:

error: expected one of ), ,, ., ?, or an operator, found _input
--> colloid-parser/src/parser/colloidlexer.rs:158:67
|
158 | Character.isJavaIdentifierStart(Character.toCodePoint((char)_input.LA(-2), (char)_input.LA(-1)))
| -^^^^^
| |
| expected one of ), ,, ., ?, or an operator
| help: missing ,

error: expected one of ), ,, ., ?, or an operator, found _input
--> colloid-parser/src/parser/colloidlexer.rs:158:88
|
158 | Character.isJavaIdentifierStart(Character.toCodePoint((char)_input.LA(-2), (char)_input.LA(-1)))
| -^^^^^
| |
| expected one of ), ,, ., ?, or an operator
| help: missing ,

error: expected one of ), ,, ., ?, or an operator, found _input
--> colloid-parser/src/parser/colloidlexer.rs:172:66
|
172 | Character.isJavaIdentifierPart(Character.toCodePoint((char)_input.LA(-2), (char)_input.LA(-1)))
| -^^^^^
| |
| expected one of ), ,, ., ?, or an operator
| help: missing ,

error: expected one of ), ,, ., ?, or an operator, found _input
--> colloid-parser/src/parser/colloidlexer.rs:172:87
|
172 | Character.isJavaIdentifierPart(Character.toCodePoint((char)_input.LA(-2), (char)_input.LA(-1)))
| -^^^^^
| |
| expected one of ), ,, ., ?, or an operator
| help: missing ,

error: cannot find macro lazy_static in this scope
--> colloid-parser/src/parser/colloidlexer.rs:55:2
|
55 | lazy_static!{
| ^^^^^^^^^^^

error: cannot find macro lazy_static in this scope
--> colloid-parser/src/parser/colloidlexer.rs:209:2
|
209 | lazy_static! {
| ^^^^^^^^^^^

error: cannot find macro lazy_static in this scope
--> colloid-parser/src/parser/colloidparser.rs:54:2
|
54 | lazy_static!{
| ^^^^^^^^^^^

error: cannot find macro lazy_static in this scope
--> colloid-parser/src/parser/colloidparser.rs:212:1
|
212 | lazy_static! {
| ^^^^^^^^^^^

error[E0425]: cannot find value _ATN in this scope
--> colloid-parser/src/parser/colloidlexer.rs:111:6
|
111 | _ATN.clone(),
| ^^^^ not found in this scope

error[E0425]: cannot find value _decision_to_DFA in this scope
--> colloid-parser/src/parser/colloidlexer.rs:112:6
|
112 | _decision_to_DFA.clone(),
| ^^^^^^^^^^^^^^^^ not found in this scope

error[E0425]: cannot find value _shared_context_cache in this scope
--> colloid-parser/src/parser/colloidlexer.rs:113:6
|
113 | _shared_context_cache.clone(),
| ^^^^^^^^^^^^^^^^^^^^^ not found in this scope

error[E0425]: cannot find value Character in this scope
--> colloid-parser/src/parser/colloidlexer.rs:155:7
|
155 | Character.isJavaIdentifierStart(_input.LA(-1))
| ^^^^^^^^^ not found in this scope

error[E0425]: cannot find value _input in this scope
--> colloid-parser/src/parser/colloidlexer.rs:155:39
|
155 | Character.isJavaIdentifierStart(_input.LA(-1))
| ^^^^^^ not found in this scope

error[E0425]: cannot find value Character in this scope
--> colloid-parser/src/parser/colloidlexer.rs:158:7
|
158 | Character.isJavaIdentifierStart(Character.toCodePoint((char)_input.LA(-2), (char)_input.LA(-1)))
| ^^^^^^^^^ not found in this scope

error[E0425]: cannot find value Character in this scope
--> colloid-parser/src/parser/colloidlexer.rs:158:39
|
158 | Character.isJavaIdentifierStart(Character.toCodePoint((char)_input.LA(-2), (char)_input.LA(-1)))
| ^^^^^^^^^ not found in this scope

error[E0423]: expected value, found builtin type char
--> colloid-parser/src/parser/colloidlexer.rs:158:62
|
158 | Character.isJavaIdentifierStart(Character.toCodePoint((char)_input.LA(-2), (char)_input.LA(-1)))
| ^^^^ not a value

error[E0425]: cannot find value _input in this scope
--> colloid-parser/src/parser/colloidlexer.rs:158:67
|
158 | Character.isJavaIdentifierStart(Character.toCodePoint((char)_input.LA(-2), (char)_input.LA(-1)))
| ^^^^^^ not found in this scope

error[E0423]: expected value, found builtin type char
--> colloid-parser/src/parser/colloidlexer.rs:158:83
|
158 | Character.isJavaIdentifierStart(Character.toCodePoint((char)_input.LA(-2), (char)_input.LA(-1)))
| ^^^^ not a value

error[E0425]: cannot find value _input in this scope
--> colloid-parser/src/parser/colloidlexer.rs:158:88
|
158 | Character.isJavaIdentifierStart(Character.toCodePoint((char)_input.LA(-2), (char)_input.LA(-1)))
| ^^^^^^ not found in this scope

error[E0425]: cannot find value Character in this scope
--> colloid-parser/src/parser/colloidlexer.rs:169:7
|
169 | Character.isJavaIdentifierPart(_input.LA(-1))
| ^^^^^^^^^ not found in this scope

error[E0425]: cannot find value _input in this scope
--> colloid-parser/src/parser/colloidlexer.rs:169:38
|
169 | Character.isJavaIdentifierPart(_input.LA(-1))
| ^^^^^^ not found in this scope

error[E0425]: cannot find value Character in this scope
--> colloid-parser/src/parser/colloidlexer.rs:172:7
|
172 | Character.isJavaIdentifierPart(Character.toCodePoint((char)_input.LA(-2), (char)_input.LA(-1)))
| ^^^^^^^^^ not found in this scope

error[E0425]: cannot find value Character in this scope
--> colloid-parser/src/parser/colloidlexer.rs:172:38
|
172 | Character.isJavaIdentifierPart(Character.toCodePoint((char)_input.LA(-2), (char)_input.LA(-1)))
| ^^^^^^^^^ not found in this scope

error[E0423]: expected value, found builtin type char
--> colloid-parser/src/parser/colloidlexer.rs:172:61
|
172 | Character.isJavaIdentifierPart(Character.toCodePoint((char)_input.LA(-2), (char)_input.LA(-1)))
| ^^^^ not a value

error[E0425]: cannot find value _input in this scope
--> colloid-parser/src/parser/colloidlexer.rs:172:66
|
172 | Character.isJavaIdentifierPart(Character.toCodePoint((char)_input.LA(-2), (char)_input.LA(-1)))
| ^^^^^^ not found in this scope

error[E0423]: expected value, found builtin type char
--> colloid-parser/src/parser/colloidlexer.rs:172:82
|
172 | Character.isJavaIdentifierPart(Character.toCodePoint((char)_input.LA(-2), (char)_input.LA(-1)))
| ^^^^ not a value

error[E0425]: cannot find value _input in this scope
--> colloid-parser/src/parser/colloidlexer.rs:172:87
|
172 | Character.isJavaIdentifierPart(Character.toCodePoint((char)_input.LA(-2), (char)_input.LA(-1)))
| ^^^^^^ not found in this scope

error[E0425]: cannot find value _ATN in this scope
--> colloid-parser/src/parser/colloidparser.rs:79:4
|
79 | _ATN.clone(),
| ^^^^ not found in this scope

error[E0425]: cannot find value _decision_to_DFA in this scope
--> colloid-parser/src/parser/colloidparser.rs:80:4
|
80 | _decision_to_DFA.clone(),
| ^^^^^^^^^^^^^^^^ not found in this scope

error[E0425]: cannot find value _shared_context_cache in this scope
--> colloid-parser/src/parser/colloidparser.rs:81:4
|
81 | _shared_context_cache.clone(),
| ^^^^^^^^^^^^^^^^^^^^^ a field by this name exists in Self

error[E0425]: cannot find value VOCABULARY in this scope
--> colloid-parser/src/parser/colloidparser.rs:125:51
|
125 | fn get_vocabulary(&self) -> &dyn Vocabulary { &**VOCABULARY }
| ^^^^^^^^^^ not found in this scope

warning: unnecessary parentheses around method argument
--> colloid-parser/src/parser/colloidlexer.rs:158:61
|
158 | Character.isJavaIdentifierStart(Character.toCodePoint((char)_input.LA(-2), (char)_input.LA(-1)))
| ^^^^^^ help: remove these parentheses
|
= note: #[warn(unused_parens)] on by default

warning: unnecessary parentheses around method argument
--> colloid-parser/src/parser/colloidlexer.rs:158:82
|
158 | Character.isJavaIdentifierStart(Character.toCodePoint((char)_input.LA(-2), (char)_input.LA(-1)))
| ^^^^^^ help: remove these parentheses

warning: unnecessary parentheses around method argument
--> colloid-parser/src/parser/colloidlexer.rs:172:60
|
172 | Character.isJavaIdentifierPart(Character.toCodePoint((char)_input.LA(-2), (char)_input.LA(-1)))
| ^^^^^^ help: remove these parentheses

warning: unnecessary parentheses around method argument
--> colloid-parser/src/parser/colloidlexer.rs:172:81
|
172 | Character.isJavaIdentifierPart(Character.toCodePoint((char)_input.LA(-2), (char)_input.LA(-1)))
| ^^^^^^ help: remove these parentheses

warning: unused import: std::any::Any
--> colloid-parser/src/parser/colloidlistener.rs:6:5
|
6 | use std::any::Any;
| ^^^^^^^^^^^^^
|
= note: #[warn(unused_imports)] on by default

error: aborting due to 31 previous errors

Some errors have detailed explanations: E0423, E0425.
For more information about an error, try rustc --explain E0423.
error: could not compile colloid-parser.

To learn more, run the command again with --verbose.
matijak@Matijas-MBP colloid-llvm % clear
matijak@Matijas-MBP colloid-llvm % cargo build
Compiling colloid-parser v0.1.0 (/Users/matijak/ClionProjects/colloid-llvm/colloid-parser)
error: expected one of ), ,, ., ?, or an operator, found _input
--> colloid-parser/src/parser/colloidlexer.rs:158:67
|
158 | Character.isJavaIdentifierStart(Character.toCodePoint((char)_input.LA(-2), (char)_input.LA(-1)))
| -^^^^^
| |
| expected one of ), ,, ., ?, or an operator
| help: missing ,

error: expected one of ), ,, ., ?, or an operator, found _input
--> colloid-parser/src/parser/colloidlexer.rs:158:88
|
158 | Character.isJavaIdentifierStart(Character.toCodePoint((char)_input.LA(-2), (char)_input.LA(-1)))
| -^^^^^
| |
| expected one of ), ,, ., ?, or an operator
| help: missing ,

error: expected one of ), ,, ., ?, or an operator, found _input
--> colloid-parser/src/parser/colloidlexer.rs:172:66
|
172 | Character.isJavaIdentifierPart(Character.toCodePoint((char)_input.LA(-2), (char)_input.LA(-1)))
| -^^^^^
| |
| expected one of ), ,, ., ?, or an operator
| help: missing ,

error: expected one of ), ,, ., ?, or an operator, found _input
--> colloid-parser/src/parser/colloidlexer.rs:172:87
|
172 | Character.isJavaIdentifierPart(Character.toCodePoint((char)_input.LA(-2), (char)_input.LA(-1)))
| -^^^^^
| |
| expected one of ), ,, ., ?, or an operator
| help: missing ,

error: cannot find macro lazy_static in this scope
--> colloid-parser/src/parser/colloidlexer.rs:55:2
|
55 | lazy_static!{
| ^^^^^^^^^^^

error: cannot find macro lazy_static in this scope
--> colloid-parser/src/parser/colloidlexer.rs:209:2
|
209 | lazy_static! {
| ^^^^^^^^^^^

error: cannot find macro lazy_static in this scope
--> colloid-parser/src/parser/colloidparser.rs:54:2
|
54 | lazy_static!{
| ^^^^^^^^^^^

error: cannot find macro lazy_static in this scope
--> colloid-parser/src/parser/colloidparser.rs:212:1
|
212 | lazy_static! {
| ^^^^^^^^^^^

error[E0425]: cannot find value _ATN in this scope
--> colloid-parser/src/parser/colloidlexer.rs:111:6
|
111 | _ATN.clone(),
| ^^^^ not found in this scope

error[E0425]: cannot find value _decision_to_DFA in this scope
--> colloid-parser/src/parser/colloidlexer.rs:112:6
|
112 | _decision_to_DFA.clone(),
| ^^^^^^^^^^^^^^^^ not found in this scope

error[E0425]: cannot find value _shared_context_cache in this scope
--> colloid-parser/src/parser/colloidlexer.rs:113:6
|
113 | _shared_context_cache.clone(),
| ^^^^^^^^^^^^^^^^^^^^^ not found in this scope

error[E0425]: cannot find value Character in this scope
--> colloid-parser/src/parser/colloidlexer.rs:155:7
|
155 | Character.isJavaIdentifierStart(_input.LA(-1))
| ^^^^^^^^^ not found in this scope

error[E0425]: cannot find value _input in this scope
--> colloid-parser/src/parser/colloidlexer.rs:155:39
|
155 | Character.isJavaIdentifierStart(_input.LA(-1))
| ^^^^^^ not found in this scope

error[E0425]: cannot find value Character in this scope
--> colloid-parser/src/parser/colloidlexer.rs:158:7
|
158 | Character.isJavaIdentifierStart(Character.toCodePoint((char)_input.LA(-2), (char)_input.LA(-1)))
| ^^^^^^^^^ not found in this scope

error[E0425]: cannot find value Character in this scope
--> colloid-parser/src/parser/colloidlexer.rs:158:39
|
158 | Character.isJavaIdentifierStart(Character.toCodePoint((char)_input.LA(-2), (char)_input.LA(-1)))
| ^^^^^^^^^ not found in this scope

error[E0423]: expected value, found builtin type char
--> colloid-parser/src/parser/colloidlexer.rs:158:62
|
158 | Character.isJavaIdentifierStart(Character.toCodePoint((char)_input.LA(-2), (char)_input.LA(-1)))
| ^^^^ not a value

error[E0425]: cannot find value _input in this scope
--> colloid-parser/src/parser/colloidlexer.rs:158:67
|
158 | Character.isJavaIdentifierStart(Character.toCodePoint((char)_input.LA(-2), (char)_input.LA(-1)))
| ^^^^^^ not found in this scope

error[E0423]: expected value, found builtin type char
--> colloid-parser/src/parser/colloidlexer.rs:158:83
|
158 | Character.isJavaIdentifierStart(Character.toCodePoint((char)_input.LA(-2), (char)_input.LA(-1)))
| ^^^^ not a value

error[E0425]: cannot find value _input in this scope
--> colloid-parser/src/parser/colloidlexer.rs:158:88
|
158 | Character.isJavaIdentifierStart(Character.toCodePoint((char)_input.LA(-2), (char)_input.LA(-1)))
| ^^^^^^ not found in this scope

error[E0425]: cannot find value Character in this scope
--> colloid-parser/src/parser/colloidlexer.rs:169:7
|
169 | Character.isJavaIdentifierPart(_input.LA(-1))
| ^^^^^^^^^ not found in this scope

error[E0425]: cannot find value _input in this scope
--> colloid-parser/src/parser/colloidlexer.rs:169:38
|
169 | Character.isJavaIdentifierPart(_input.LA(-1))
| ^^^^^^ not found in this scope

error[E0425]: cannot find value Character in this scope
--> colloid-parser/src/parser/colloidlexer.rs:172:7
|
172 | Character.isJavaIdentifierPart(Character.toCodePoint((char)_input.LA(-2), (char)_input.LA(-1)))
| ^^^^^^^^^ not found in this scope

error[E0425]: cannot find value Character in this scope
--> colloid-parser/src/parser/colloidlexer.rs:172:38
|
172 | Character.isJavaIdentifierPart(Character.toCodePoint((char)_input.LA(-2), (char)_input.LA(-1)))
| ^^^^^^^^^ not found in this scope

error[E0423]: expected value, found builtin type char
--> colloid-parser/src/parser/colloidlexer.rs:172:61
|
172 | Character.isJavaIdentifierPart(Character.toCodePoint((char)_input.LA(-2), (char)_input.LA(-1)))
|

...

Sorry for the poor formatting, doesn't seem I can get it to work properly.

Dependency gets recompiled every time due to rerun-if-changed with non-existing path

The build.rs references a local antlr.jar that usually does not exist when using antlr4rust as a dependency. This seems to trigger a full rebuild every time, according to a cargo issue that is the expected behaviour.

I can imagine two solutions, either check in the antlr.jar so it can be referenced using a local file (but not sure about the license implications), or the build.rs logic could be triggered only when an environment variable is set, since this seems to be needed for tests only. Detecting the test profile from within build.rs does not seem to be possible yet.

Rule contexts returned from `Attrs` traits need to be optional.

First of all, thanks for this implementation, it is already working quite well. I'm running into some issues however with optional rules and I saw that a similar change happened recently for optional tokens.

Consider the following grammar:

query   : SELECT expression (',' expression)* whereClause? EOF
        ;

whereClause
        : WHERE expression
        ;

Currently there is no way to check whether the whereClause rule matched since the return type of QueryContextAttrs::whereClause is Rc<WhereClauseContextAll> and would panic if called without matches. I think this also needs to be wrapped into an Option. That would also match the java implementation, which would return null when calling that method.

As a workaround I could probably inline the rule and check for presence of the WHERE token first, but that might not work in all cases.

`foo_all()` getters output children that are not of the kind `foo`

I encountered this issue while implementing grammarVisitorCompat using antlr-rust v0.3.0-beta.

Consider the following grammar:

grammar all_test;

list: GET VAR_ ( ',' VAR_ )* ;
GET: 'get';
VAR_: '$' [a-zA-Z0-9][a-zA-Z0-9_-]* ;

WS : [ \t\r\n]+ -> channel(HIDDEN) ;

When parsing the input, we would like to collect all the variables into a list. I.e,, when parsing the string "get $abc, $def", we expect to parse that into ["$abc", "$def"].

Using ctx.VAR__all() (where ctx is the &ListContext argument of visit_list()) results in the following output: "get", "$abc", ",", "$def"

Note that, at the same time, ctx.VAR_(i) produces the expected result:

ctx.VAR_(0) == Some($abc);
ctx.VAR_(1) == Some($def);
ctx.VAR_(2) == None;

The entire MRE can be found at https://github.com/dmitrii-ubskii/antlr-rust-all-issue

-no-listener option generates code that refers to the (non existant) listener

running test VisitorCalc through antlr

java.exe -jar ..\..\antlr4-4.8-2-SNAPSHOT-complete.jar -no-listener -visitor -Dlanguage=Rust .\VisitorCalc.g4

produces code that refernces the listener

here for example in the generated parser

antlr_rust::tid! { impl<'input> TidAble<'input> for dyn VisitorCalcListener<'input> + 'input }

error generate cpp lexer

I tried to generate cpp lexer from https://github.com/antlr/grammars-v4/blob/master/cpp/pom.xml, but have a error like:
error: expected {, found throw
I think the problem is throw because rust not using throw new like java or C# for error handling.

 --> src\cpp\cpp14parser.rs:23478:64
  |

23478 | .val { it.get_text() } else { "null" } .compareTo("0")!=0) throw new InputMismatchException(this);
| ^^^^^ expected {
|
note: the if expression is missing a block after this condition
--> src\cpp\cpp14parser.rs:23477:6
|
23477 | if(if let Some(it) = &(cast::<_,PureSpecifierContext >(&*_localctx))
| ^
23478 | | .val { it.get_text() } else { "null" } .compareTo("0")!=0) throw new InputMismatchException(this);
| |________________________________________________________^
help: try placing this code inside a block
|
23478 | .val { it.get_text() } else { "null" } .compareTo("0")!=0) { throw } new InputMismatchException(this);
| + +

New release with updated dependencies or less strict version ranges

I tried updating dependencies in a cargo project and ran into an issue with a transitive dependency on once_cell = "1.2.*". Assuming once_cell correclty follows semantic versioning, the newer version 1.3.x or 1.4 should be compatible. Or was there a specific breaking change that requires staying with 1.2?

error: failed to select a version for `once_cell`.
    ... required by package `futures-task v0.3.5`
    ... which is depended on by `futures-util v0.3.5`
    ... which is depended on by `xxxxx`
versions that meet the requirements `^1.3.1` are: 1.4.0, 1.3.1

all possible versions conflict with previously selected packages.

  previously selected package `once_cell v1.2.0`
    ... which is depended on by `antlr-rust v0.1.1`
    ... which is depended on by `yyyyy`
    ... which is depended on by `xxxxx`

failed to select a version for `once_cell` which could resolve this conflict

Anybody have a visitor pattern example?

Thank you very much!

Hidden channel tokens at the start of the parsed string are not ignored

Consider the following grammar:

grammar hidden_test;

list: VAR_ ( ',' VAR_ )* ;
VAR_: '$' [a-zA-Z0-9][a-zA-Z0-9_-]* ;

WS : [ \t\r\n]+ -> channel(HIDDEN) ;

Parsing " $abc, $def" in Java or on http://lab.antlr.org/ produces the expected parse tree. But trying to parse this string in Rust via parser.list() produces the following error instead:

line 1:0 extraneous input ' ' expecting VAR_

Unable to parse fhirpath grammar

Grammar taken from here: https://github.com/HL7/fhirpath.js/blob/master/src/parser/FHIRPath.g4

Parsed like:

#![feature(try_blocks)]

#[macro_use]
extern crate lazy_static;

mod parser;

use antlr_rust::common_token_stream::CommonTokenStream;
use antlr_rust::input_stream::InputStream;
use antlr_rust::parser_rule_context::ParserRuleContext;
use antlr_rust::tree::{ErrorNode, ParseTreeListener, TerminalNode};

use parser::fhirpathlexer::FHIRPathLexer;
use parser::fhirpathlistener::FHIRPathListener;
use parser::fhirpathparser::*;

struct Listener;

impl ParseTreeListener for Listener {
    fn visit_terminal(&mut self, node: &TerminalNode) {}
    fn visit_error_node(&mut self, node: &ErrorNode) {}
    fn enter_every_rule(&mut self, ctx: &dyn ParserRuleContext) {}
    fn exit_every_rule(&mut self, ctx: &dyn ParserRuleContext) {}
}

impl FHIRPathListener for Listener {
...
}

fn main() {
    let input = String::from("Observation.value");
    let lexer = FHIRPathLexer::new(Box::new(InputStream::new(input)));
    let token_source = CommonTokenStream::new(lexer);
    let mut parser = FHIRPathParser::new(Box::new(token_source));
    parser.add_parse_listener(Box::new(Listener {}));
    let result = parser.expression();
    assert!(result.is_ok());
}

Getting error:

thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', /home/alex/.cargo/registry/src/github.com-1ecc6299db9ec823/antlr-rust-0.1.1/src/parser.rs:442:79
stack backtrace:
   0: backtrace::backtrace::libunwind::trace
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.46/src/backtrace/libunwind.rs:86
   1: backtrace::backtrace::trace_unsynchronized
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.46/src/backtrace/mod.rs:66
   2: std::sys_common::backtrace::_print_fmt
             at src/libstd/sys_common/backtrace.rs:78
   3: <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt
             at src/libstd/sys_common/backtrace.rs:59
   4: core::fmt::write
             at src/libcore/fmt/mod.rs:1076
   5: std::io::Write::write_fmt
             at src/libstd/io/mod.rs:1537
   6: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:62
   7: std::sys_common::backtrace::print
             at src/libstd/sys_common/backtrace.rs:49
   8: std::panicking::default_hook::{{closure}}
             at src/libstd/panicking.rs:198
   9: std::panicking::default_hook
             at src/libstd/panicking.rs:217
  10: std::panicking::rust_panic_with_hook
             at src/libstd/panicking.rs:526
  11: rust_begin_unwind
             at src/libstd/panicking.rs:437
  12: core::panicking::panic_fmt
             at src/libcore/panicking.rs:85
  13: core::panicking::panic
             at src/libcore/panicking.rs:50
  14: core::option::Option<T>::unwrap
             at /home/alex/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/macros/mod.rs:10
  15: antlr_rust::parser::BaseParser<Ext,T>::unroll_recursion_context
             at /home/alex/.cargo/registry/src/github.com-1ecc6299db9ec823/antlr-rust-0.1.1/src/parser.rs:442
  16: fhir_server_fhirpath::parser::fhirpathparser::FHIRPathParser::expression_rec
             at src/parser/fhirpathparser.rs:1493
  17: fhir_server_fhirpath::parser::fhirpathparser::FHIRPathParser::expression
             at src/parser/fhirpathparser.rs:1097
  18: fhir_server_fhirpath::main
             at src/main.rs:534
  19: std::rt::lang_start::{{closure}}
             at /home/alex/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/rt.rs:67
  20: std::rt::lang_start_internal::{{closure}}
             at src/libstd/rt.rs:52
  21: std::panicking::try::do_call
             at src/libstd/panicking.rs:348
  22: std::panicking::try
             at src/libstd/panicking.rs:325
  23: std::panic::catch_unwind
             at src/libstd/panic.rs:394
  24: std::rt::lang_start_internal
             at src/libstd/rt.rs:51
  25: std::rt::lang_start
             at /home/alex/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/rt.rs:67
  26: main
  27: __libc_start_main
  28: _start

rust version: 1.46.0-nightly

Update ANTLR

Hi,

Thanks for this interesting initiative. Would it be possible to update Antlr to a later version? v4.10 seems to be a major update and is not fully compatible with older versions.
https://github.com/antlr/antlr4/releases/tag/4.10

Hence it would be better to build new projects using that.

Usage questions and comments

First of all, thanks a lot for implementing this tool. I have generated a parser and visitor for the OpenCypher syntax using your generator. Everything has worked well out of the box.
I have two small usability questions about working with the DAG.

Is there a way to get the type of a node using a helper function? Right now I am using a hacky method:
ruleNames[parserNode.get_rule_index()], but this is a bit unwieldy. However, implementing a helper function seems complicated with the way the traits are set up.

Second, how can you visit a sub node explicitly using the visitor pattern? Currently, I am using child.accept(self);, but I am not sure if that is right way to visit or if it has unintended side effects.

I also have two minor comments on issues I encountered while using the generated code

There is a style issue regarding the generated code. The generated visitor and listener function calls use CamelCase style. Rust-analyzer complains about the lack of snake case, so I have to manually disable these checks. Are there plans to switch to snake case for the generated function calls?

Also another thing I have noticed is that the parser seems to continue even though it encounters an illegal expression. For example, for this piece of code:

let mut parser = CypherParser::new(token_source);
let result = parser.oC_Cypher();
match result {
    Err(e) => {
        log::error!("Error parsing query: {:?}", e);
    }
    Ok(v) => {
        to_ir::visit_result(v);
    }
}

I am unable to catch an error that is thrown in the parser. The program continues despite a parsing error. What is the right way to handle an error here?

generated code with a java style package header

Hi, thanks for the huge effort first!

java -jar ~/antlr4-4.8-2-SNAPSHOT-complete.jar -Dlanguage=Rust -visitor -o ../gen SqlBase.g4

The generated visitor.rs or lexer.rs has both a java style package header. Are there something wrong with the options?

unknown feature `raw`

Unable to compile antlr4rs because of the following:

error[E0635]: unknown feature `raw`
 --> ~/.cargo/registry/src/github.com-1ecc6299db9ec823/antlr-rust-0.2.0/src/lib.rs:4:12
  |
4 | #![feature(raw)]
  |  

If I remove that line compilation continues without anymore errors.

Thanks!

I try to use visitor pattern, but failed.

g4 file:

grammar CSV;

csvFile: hdr row+ ;
hdr : row ;

row : field (',' field)* '\r'? '\n' ;

field
    : TEXT
    | STRING
    |
    ;

TEXT   : ~[,\n\r"]+ ;
STRING : '"' ('""'|~'"')* '"' ; // quote-quote is an escaped quote

generate codes:

java -cp /opt/antlr/antlr4-4.8-2-SNAPSHOT-complete.jar org.antlr.v4.Tool -Dlanguage=Rust CSV.g4  -no-listener -visitor

When I compile the codes, got errors:

    Compiling lixcep v0.0.1 (/home/lix/works/repos/project/lix/lixcep)
error: expected one of `:`, `@`, or `|`, found `)`
  --> src/csv_visitor/csvvisitor.rs:17:47
   |
17 |     fn visitCsvFile(ctx: CSVParser.CsvFileContext) -> T;
   |                                                  ^ expected one of `:`, `@`, or `|`
   |
   = note: anonymous parameters are removed in the 2018 edition (see RFC 1685)
help: if this is a parameter name, give it a type
   |
17 |     fn visitCsvFile(ctx: CSVParser.CsvFileContext: TypeName) -> T;
   |                                    ^^^^^^^^^^^^^^^^^^^^^^^^
help: if this is a type, explicitly ignore the parameter name
   |
17 |     fn visitCsvFile(ctx: CSVParser._: CsvFileContext) -> T;
   |                                    ^^^^^^^^^^^^^^^^^

error: expected one of `!`, `(`, `)`, `+`, `,`, `::`, or `<`, found `.`
  --> src/csv_visitor/csvvisitor.rs:17:32
   |
17 |     fn visitCsvFile(ctx: CSVParser.CsvFileContext) -> T;
   |                                   ^
   |                                   |
   |                                   expected one of 7 possible tokens
   |                                   help: missing `,`

error: expected one of `:`, `@`, or `|`, found `)`
  --> src/csv_visitor/csvvisitor.rs:23:39
   |
23 |     fn visitHdr(ctx: CSVParser.HdrContext) -> T;
   |                                          ^ expected one of `:`, `@`, or `|`
   |
   = note: anonymous parameters are removed in the 2018 edition (see RFC 1685)
help: if this is a parameter name, give it a type
   |
23 |     fn visitHdr(ctx: CSVParser.HdrContext: TypeName) -> T;
   |                                ^^^^^^^^^^^^^^^^^^^^
help: if this is a type, explicitly ignore the parameter name
   |
23 |     fn visitHdr(ctx: CSVParser._: HdrContext) -> T;
   |                                ^^^^^^^^^^^^^

error: expected one of `!`, `(`, `)`, `+`, `,`, `::`, or `<`, found `.`
  --> src/csv_visitor/csvvisitor.rs:23:28
   |
23 |     fn visitHdr(ctx: CSVParser.HdrContext) -> T;
   |                               ^
   |                               |
   |                               expected one of 7 possible tokens
   |                               help: missing `,`

error: expected one of `:`, `@`, or `|`, found `)`
  --> src/csv_visitor/csvvisitor.rs:29:39
   |
29 |     fn visitRow(ctx: CSVParser.RowContext) -> T;
   |                                          ^ expected one of `:`, `@`, or `|`
   |
   = note: anonymous parameters are removed in the 2018 edition (see RFC 1685)
help: if this is a parameter name, give it a type
   |
29 |     fn visitRow(ctx: CSVParser.RowContext: TypeName) -> T;
   |                                ^^^^^^^^^^^^^^^^^^^^
help: if this is a type, explicitly ignore the parameter name
   |
29 |     fn visitRow(ctx: CSVParser._: RowContext) -> T;
   |                                ^^^^^^^^^^^^^

error: expected one of `!`, `(`, `)`, `+`, `,`, `::`, or `<`, found `.`
  --> src/csv_visitor/csvvisitor.rs:29:28
   |
29 |     fn visitRow(ctx: CSVParser.RowContext) -> T;
   |                               ^
   |                               |
   |                               expected one of 7 possible tokens
   |                               help: missing `,`

error: expected one of `:`, `@`, or `|`, found `)`
  --> src/csv_visitor/csvvisitor.rs:35:43
   |
35 |     fn visitField(ctx: CSVParser.FieldContext) -> T;
   |                                              ^ expected one of `:`, `@`, or `|`
   |
   = note: anonymous parameters are removed in the 2018 edition (see RFC 1685)
help: if this is a parameter name, give it a type
   |
35 |     fn visitField(ctx: CSVParser.FieldContext: TypeName) -> T;
   |                                  ^^^^^^^^^^^^^^^^^^^^^^
help: if this is a type, explicitly ignore the parameter name
   |
35 |     fn visitField(ctx: CSVParser._: FieldContext) -> T;
   |                                  ^^^^^^^^^^^^^^^

error: expected one of `!`, `(`, `)`, `+`, `,`, `::`, or `<`, found `.`
  --> src/csv_visitor/csvvisitor.rs:35:30
   |
35 |     fn visitField(ctx: CSVParser.FieldContext) -> T;
   |                                 ^
   |                                 |
   |                                 expected one of 7 possible tokens
   |                                 help: missing `,`

error[E0433]: failed to resolve: use of undeclared type or module `antlr_runtime`
 --> src/csv_visitor/csvvisitor.rs:2:5
  |
2 | use antlr_runtime::tree::ParseTreeVisitor;
  |     ^^^^^^^^^^^^^ use of undeclared type or module `antlr_runtime`

error[E0405]: cannot find trait `ParseTreeVisitor` in this scope
  --> src/csv_visitor/csvvisitor.rs:11:27
   |
11 | pub trait  CSVVisitor<T>: ParseTreeVisitor<T> {
   |                           ^^^^^^^^^^^^^^^^ not found in this scope

error[E0412]: cannot find type `CSVParser` in this scope
  --> src/csv_visitor/csvvisitor.rs:17:23
   |
17 |     fn visitCsvFile(ctx: CSVParser.CsvFileContext) -> T;
   |                          ^^^^^^^^^ not found in this scope
   |
help: consider importing this struct
   |
2  | use crate::csv_visitor::CSVParser;
   |

error[E0412]: cannot find type `CSVParser` in this scope
  --> src/csv_visitor/csvvisitor.rs:23:19
   |
23 |     fn visitHdr(ctx: CSVParser.HdrContext) -> T;
   |                      ^^^^^^^^^ not found in this scope
   |
help: consider importing this struct
   |
2  | use crate::csv_visitor::CSVParser;
   |

error[E0412]: cannot find type `CSVParser` in this scope
  --> src/csv_visitor/csvvisitor.rs:29:19
   |
29 |     fn visitRow(ctx: CSVParser.RowContext) -> T;
   |                      ^^^^^^^^^ not found in this scope
   |
help: consider importing this struct
   |
2  | use crate::csv_visitor::CSVParser;
   |

error[E0412]: cannot find type `CSVParser` in this scope
  --> src/csv_visitor/csvvisitor.rs:35:21
   |
35 |     fn visitField(ctx: CSVParser.FieldContext) -> T;
   |                        ^^^^^^^^^ not found in this scope
   |
help: consider importing this struct
   |
2  | use crate::csv_visitor::CSVParser;
   |

warning: unused import: `antlr_rust::common_token_stream::CommonTokenStream`
 --> src/csv_visitor/mod.rs:1:5
  |
1 | use antlr_rust::common_token_stream::CommonTokenStream;
  |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |
  = note: `#[warn(unused_imports)]` on by default

warning: unused import: `antlr_rust::input_stream::InputStream`
 --> src/csv_visitor/mod.rs:2:5
  |
2 | use antlr_rust::input_stream::InputStream;
  |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

warning: unused imports: `ParseTreeListener`, `ParseTree`
 --> src/csv_visitor/mod.rs:3:24
  |
3 | use antlr_rust::tree::{ParseTree, ParseTreeListener};
  |                        ^^^^^^^^^  ^^^^^^^^^^^^^^^^^

warning: unused import: `csvlexer::CSVLexer`
 --> src/csv_visitor/mod.rs:8:5
  |
8 | use csvlexer::CSVLexer;
  |     ^^^^^^^^^^^^^^^^^^

warning: unused import: `csvparser::*`
 --> src/csv_visitor/mod.rs:9:5
  |
9 | use csvparser::*;
  |     ^^^^^^^^^^^^

warning: unused import: `csvvisitor::*`
  --> src/csv_visitor/mod.rs:10:5
   |
10 | use csvvisitor::*;
   |     ^^^^^^^^^^^^^

error: aborting due to 14 previous errors; 6 warnings emitted

Some errors have detailed explanations: E0405, E0412, E0433.
For more information about an error, try `rustc --explain E0405`.
error: could not compile `lixcep`.

To learn more, run the command again with --verbose.

Any suggestions?

antlr does not compile due to depricated raw feature

If one uses nightly <=1.53, this crate compiles and works as expected. However, once the switch is made to 1.55, it no longer works and gives the error

 error[E0635]: unknown feature `raw`
 --> /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/antlr-rust-0.2.0/src/lib.rs:4:12
  |
4 | #![feature(raw)]
  | 

I am importing this crate through

antlr-rust = "=0.2"

as mentioned in the readme.

I believe the error is because as mentioned here in the Rust documentation, the raw feature has been deprecated. Can this be updated so antlr doesn't use the raw feature?

Any example for the Python-like grammar?

Some languages use INDENT\DEDENT tokens such as python.
Does this crate have the function to handle them?
https://github.com/yshavit/antlr-denter/blob/master/Java/core/src/main/java/com/yuvalshavit/antlr4/DenterHelper.java
and is there any example for that?

failed to resolve: could not find `impl_tid` in `antlr_rust`

I'm not sure if my build environment is set up properly when I get the errors below.

Cargo.toml

[package]
name = "parser-test"
version = "0.1.0"
edition = "2021"

[dependencies]
antlr-rust = { git = "https://github.com/rrevenantt/antlr4rust", branch = "v0.3" }

A script that generates rust code from a grammar file:

#!/bin/bash
pushd grammars
java -jar ~/bin/antlr4-4.8-2-SNAPSHOT-complete.jar -Dlanguage=Rust -o ../src/parsers ArrayInit.g4
popd

antlr4-4.8-2-SNAPSHOT-complete.jar is built from rust-target branch.

Tool chain is nightly:

stable-x86_64-unknown-linux-gnu (default)
beta-x86_64-unknown-linux-gnu
nightly-x86_64-unknown-linux-gnu (override)

I had to add #![feature(try_blocks)] to main.rs to disable two try expression is experimental errors:

#![feature(try_blocks)]

mod parsers;

parsers/mod.rs:

mod arrayinitparser;
mod arrayinitlexer;
mod arrayinitlistener;

Any help on how to properly setup a working build environment would be greatly appreciated.

XXX_all returns all TerminalNode regardless of type

If there is more than one TerminalNode type in the child node, XXX_all cannot obtain the correct leaf node type, and children_of_type will return all TerminalNode.

For example, the results returned by STRING_TEXT_all here include STRING_START and STRING_END