aphyr/clj-antlr

Clojure bindings for the ANTLR 4 parser

clj-antlr's Issues

Python Grammar Fails: Cannot read field "grammarType" because "this.atn" is null

Hello,
I am attempting to run the following code snippet:

(def python-file (slurp "convert_logseq_md_to_org.py"))

(let [g (antlr/parser
         "./external/parsers/grammars-v4/python/python3/Python3Lexer.g4"
         "./external/parsers/grammars-v4/python/python3/Python3Parser.g4"
         {:root "file_input"})]
  (antlr/parse g python-file))

Upon running this code snippet, I get the following error message:
Cannot read field "grammarType" because "this.atn" is null

There is no Augmented Transition Network set anywhere in the Parser Grammar; only in the Lexer Grammar.

I have debugged the code, and the lexer and parser seem to be set properly within the Clojure code, however once the grammar is passed into ANTLR4 Grammar code, specifically the following section:

Excerpt from: https://github.com/antlr/antlr4/blob/4.11.1/tool/src/org/antlr/v4/tool/Grammar.java#L1345

/** @since 4.5.1 */
public GrammarParserInterpreter createGrammarParserInterpreter(TokenStream tokenStream) {
	if (this.isLexer()) {
		throw new IllegalStateException("A parser interpreter can only be created for a parser or combined grammar.");
	}
	// must run ATN through serializer to set some state flags
	IntegerList serialized = ATNSerializer.getSerialized(atn);
	ATN deserializedATN = new ATNDeserializer().deserialize(serialized.toArray());

	return new GrammarParserInterpreter(this, deserializedATN, tokenStream);
}

It seems to be expecting an ATN to be bundled with the Parser Grammar.

The referenced ANTLR code above is called from clj-antlr in the singlethreaded-parser function in the clj-antlr/interpreted namespace.

(defn singlethreaded-parser
  "Creates a new single-threaded parser for a grammar."
  [^LexerGrammar lexer-grammar ^Grammar grammar]
  (let [^Lexer lexer (.createLexerInterpreter
                       lexer-grammar (common/char-stream ""))
        parser       (.createGrammarParserInterpreter          ;; Exception is thrown to us somewhere around here
                       grammar (common/tokens lexer))]         
    (SinglethreadedParser. grammar lexer parser)))

I have attached several screenshots that have more detailed stack traces and debug views from Cursive IDE.

I will continue investigating potential solutions, but I am curious if I am simply doing something wrong, or if maybe ANTLR has made a breaking change on their end.

Appreciate any insights, and thank you for your wonderful project!

Regards,
Aeonik

Mention that you provide line number (row/column) as metadata

I got really disappointed in instaparsers run walk speed. People pointed me to ANTLR but before actually trying it I was worrier that digging up line number will involve java interop hassle since clj-antlr docs states that none of Instaparse's delightful features present while line numbers only mentioned in the context of error handling.

Mentioning the metadata can be helpful for convincing lazy people 😄

Python3.g4 grammar not working

The Python3.g4 grammar works fine with the standard ANTLR 4.5 "grun" macro. However, it doesn't work with clj-antlr in lein repl:

(require ['clj-antlr.core :as 'antlr])
(def python3 (antlr/parser "grammars-v4/python3/Python3.g4" {:root "file_input"}))
(python3 "2 + 2\n")

==> ParseError no viable alternative at input '+ '  clj-antlr.common/parse-error (common.clj:146)

Sorry if this is a noob issue. I'm very new to Clojure.

Add support for deps.edn / publish the Java classes as a separate jar

Hello,

I would like to see support for deps.edn for this project.
The feature I find most appealing is the ability to use git versions https://clojure.org/reference/deps_and_cli#_dependencies .

This will allow users (me :D) to use unreleased versions (from git branches to test things out).

I've done the work to build a deps.edn (not that hard).
The only blocker are the 3 java classes since Clojure needs those compiled.

I asked on Slack and the current solutions are:

• compile and commit the classes in the project
• publish a jar file with the java dependencies
• remove them (replace them with clojure variants)

I think option 2 is the simplest / best long term IMO but I don't care that much.

NOTE: I know the classes use packages from org.antlr namespace and this should probably change to be aligned with java modules.

@aphyr : What is your take on this?
Would you be ok with publishing a small Jar for the 3 classes (optionally with a changed package for the 2 org.antlr )

Getting position of tokens in original input stream as metadata ?

Hi,

I'm looking a way to find determine the original position of an element of the parser tree (in order to be able to point to the source in case of a semantic error found after analysis of the parse tree). But I don't think it is currently possible with clj-antlr. How difficult would it be to add this information as meta-data in the parse tree ?

Support load the grammar with separate Lexer & Parser files.

Is this library support load grammar files with separate Lexer & Parser, e.g. this xml grammar https://github.com/antlr/grammars-v4/tree/master/xml. If not, there is any workaround for it?

Thanks,

incompatibility with spark

spark 1.6 depends on ST4 4.0.4 https://github.com/apache/spark/blob/branch-1.6/dev/deps/spark-deps-hadoop-2.2#L3

this, however depends on 4.0.8 which includes a value that was added since 4.0.4. Running scarecrow (which you're probably familiar with) on spark, fails in this way:

Caused by: java.lang.NoSuchFieldError: GROUP_FILE_EXTENSION
        at org.antlr.v4.tool.ErrorManager.setFormat(ErrorManager.java:255)
        at org.antlr.v4.Tool.<init>(Tool.java:216)
        at org.antlr.v4.Tool.<init>(Tool.java:211)
        at clj_antlr.interpreted$tool.invoke(interpreted.clj:18)
        at clj_antlr.interpreted$load_string_grammar.invoke(interpreted.clj:26)
        at clj_antlr.interpreted$grammar.invoke(interpreted.clj:38)
        at clj_antlr.interpreted$parser.invoke(interpreted.clj:127)
        at clj_antlr.core$parser.invoke(core.clj:67)
        at scarecrow.parser$antlr_parser.invoke(parser.clj:148)

here's the scarecrow side of this:

(defn antlr-parser [grammar-string]
  (antlr/parser grammar-string {:case-sensitive? false
                                :format transform-antlr-node}))

I'm looking into other ways around this problem, but if it's simple for you, ST4 compatibility would be very helpful.

I am getting unable to resolve visitor in this context error when I try to run the demo project.

When I run the demo using lein repl I am getting the following error under Ubuntu.

#error {
 :cause Unable to resolve symbol: visitor in this context
 :via
 [{:type clojure.lang.Compiler$CompilerException
   :message java.lang.RuntimeException: Unable to resolve symbol: visitor in this context, compiling:(clj_antlr_demo/edn.clj:59:3)
   :at [clojure.lang.Compiler analyze Compiler.java 6688]}
  {:type java.lang.RuntimeException
   :message Unable to resolve symbol: visitor in this context
   :at [clojure.lang.Util runtimeException Util.java 221]}]
 :trace
 [[clojure.lang.Util runtimeException Util.java 221]
  [clojure.lang.Compiler resolveIn Compiler.java 7164]
  [clojure.lang.Compiler resolve Compiler.java 7108]
...

ANLTR Alternative Labels support

Hi!

It seems to me that at the moment there is no support for ANTLRv4 Alternative Labels in clj-antlr. Am I wrong? Is there any hope it will be introduced? If not, what would be the best way to introduce it in given the current structure of the library?

Best,
Adam

A warning about deprecated `multi-hinted-let` is issued even if the user code doesn't use it

That's because multi-hinted-let is a macro that's used by clj-antlr itself.

Enhancement: Support for clojure.java.io.resource?

For example:

(def grammar (antlr/parser "./Grammar.g4"))

This works fine, because a string is passed to the parser function.

However, this won't work:

(def grammar (antlr/parser (io/resource "Grammar.g4")))

But both 2 cases works for slurp:

(slurp "hello.txt")
(slurp (io/resource "hello.txt" ))

The use case of io/resource matters because sometimes it's hard to know the path of the files.

No such var: clj-antlr.static/∂ while using

I'm trying to use clj-antlr.static/visitor but I get an error: No such var: clj-antlr.static/∂.
I wonder if this namespace is maintained at all. My intention is to use a statically compiled Java parser because my project will be compiled into a JAR file. In this case, I cannot use *.g4 files anymore.