treebank make-tree uses clojure reader, chokes on some tokens from natural language

[dakrone]

treebank make-tree uses clojure reader, chokes on some tokens from natural language

[dakrone]

I agree, it'd be better to have an actual parser for the S-expressions treebank string. I'd like to be able to write one that handled it well. In the meantime though, the string is still available for people (which is why I didn't run the treebank stuff through the make-tree method by default).

I'll leave this open, in the future I may write an actual parser for it (or have one contributed).

[croeder]

It may turn out to be a fun Clojure exercise. I've parsed treebank strings in Python before so I see the challenge partly as involving Clojure in general, and partly how to identify the different types of scalar. If you're itching to take a crack at it, I probably won't for a little while. I want to integrate found names with the parse.

[alexott]

@croeder I can try to write such parser - can you provide test data as example?

[croeder]

@alexott It'll be a few weeks before I can make the time. My wife and I just moved.

To create some, run the treebank-parser here on a nearly any sentence, then modify the tokens so they aren't proper lisp s-expressions. I mentioned 2:30 above. I think it's the leading digit that makes lisp expect either an integer or a float, but the colon throws it off. Quotes would make it the string of characters. Some creative thinking while reading the LISP spec.s may reveal more.

OK, maybe I can do one off the top of my head:
"2:30 is bad." might create a treebank like this: (S (SUBJ (2:30 NN)) (VP (is VB) (bad ADJ)) )
It's horrid linguistics, the structure and tags are wrong, but it captures the idea. LISP/Clojure is happy to read that string, but it, IIRC, wants numbers, strings and identifiers, and the 2:30 breaks it.
(S (SUBJ ("2:30" NN)) (VP (is VB) (good ADJ)) ) would work. As would
(S (SUBJ (230 NN)) (VP (is VB) (good ADJ)) )

[alexott]

Ok, thank you - I'll try to investigate this issue

[dakrone]

Merged Alex's PR and released 0.3.0 with his fix; thanks @alexott!