clir/clearnlp-guidelines

Guidelines.

ClearNLP Guidelines

The ClearNLP project provides software and resources for natural language processing. The project is currently developed by the Center for Language and Information Research at Emory University.

• The latest version: 3.2.0 (release notes).
• Contact: google group, [email protected].

Quick Start

• How to install.
• How to add models.
• How to decode.
• How to train.
• How to use APIs.

Components

• Constituent to dependency conversion.
• Tokenization and segmentation.
• Part-of-speech tagging.
• Morphological analysis.
• Dependency parsing.
• Semantic role labeling.
• Named entity recognization.
• Coreference resolution (coming in Nov, 2015).

Specifications

• ClearNLP dependency labels.
• ClearNLP semantic role labels.
• ClearNLP javadoc.
• Feature template.
• Headrule format.
• Data format.

Under Construction

• How to develop an NLP component using our API.
• Use APIs with Maven and Eclipse.
• How to extract built-in features.
• PropBank format.

clearnlp-guidelines's Issues

Exception while trying to run SRL

I am having the following trace when attempting to use decoding for SRL on a given .txt file
Installation was successful (I have everything on my classpath) and the log4j.properties has been added to the classpath. Any information on how to deal with this?

java -Xmx5g -XX:+UseConcMarkSweepGC edu.emory.clir.clearnlp.bin.NLPDecode -mode srl -c config_decode_srl.xml -i data/myfile.txt

java.io.IOException: Stream closed
    at java.io.BufferedInputStream.getInIfOpen(BufferedInputStream.java:159)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
    at java.io.DataInputStream.readFully(DataInputStream.java:195)
    at java.io.DataInputStream.readFully(DataInputStream.java:169)
    at org.tukaani.xz.SingleXZInputStream.initialize(Unknown Source)
    at org.tukaani.xz.SingleXZInputStream.<init>(Unknown Source)
    at org.tukaani.xz.XZInputStream.<init>(Unknown Source)
    at org.tukaani.xz.XZInputStream.<init>(Unknown Source)
    at edu.emory.clir.clearnlp.component.utils.NLPUtils.getObjectInputStream(NLPUtils.java:173)
    at edu.emory.clir.clearnlp.component.utils.NLPUtils.getNERecognizer(NLPUtils.java:126)
    at edu.emory.clir.clearnlp.bin.NLPDecode.getComponents(NLPDecode.java:180)
    at edu.emory.clir.clearnlp.bin.NLPDecode.decode(NLPDecode.java:94)
    at edu.emory.clir.clearnlp.bin.NLPDecode.<init>(NLPDecode.java:74)
    at edu.emory.clir.clearnlp.bin.NLPDecode.main(NLPDecode.java:236)
java.lang.NullPointerException
    at edu.emory.clir.clearnlp.component.AbstractStatisticalComponent.load(AbstractStatisticalComponent.java:157)
    at edu.emory.clir.clearnlp.component.AbstractStatisticalComponent.initDecode(AbstractStatisticalComponent.java:126)
    at edu.emory.clir.clearnlp.component.AbstractStatisticalComponent.<init>(AbstractStatisticalComponent.java:99)
    at edu.emory.clir.clearnlp.component.mode.ner.AbstractNERecognizer.<init>(AbstractNERecognizer.java:61)
    at edu.emory.clir.clearnlp.component.mode.ner.EnglishNERecognizer.<init>(EnglishNERecognizer.java:52)
    at edu.emory.clir.clearnlp.component.utils.NLPUtils.getNERecognizer(NLPUtils.java:119)
    at edu.emory.clir.clearnlp.component.utils.NLPUtils.getNERecognizer(NLPUtils.java:126)
    at edu.emory.clir.clearnlp.bin.NLPDecode.getComponents(NLPDecode.java:180)
    at edu.emory.clir.clearnlp.bin.NLPDecode.decode(NLPDecode.java:94)
    at edu.emory.clir.clearnlp.bin.NLPDecode.<init>(NLPDecode.java:74)
    at edu.emory.clir.clearnlp.bin.NLPDecode.main(NLPDecode.java:236)
    Exception in thread "main" java.lang.NullPointerException
    at edu.emory.clir.clearnlp.component.mode.ner.NERState.init(NERState.java:59)
    at edu.emory.clir.clearnlp.component.mode.ner.NERState.<init>(NERState.java:52)
    at edu.emory.clir.clearnlp.component.mode.ner.AbstractNERecognizer.process(AbstractNERecognizer.java:98)
    at edu.emory.clir.clearnlp.bin.NLPDecode.process(NLPDecode.java:163)
    at edu.emory.clir.clearnlp.bin.NLPDecode.process(NLPDecode.java:132)
    at edu.emory.clir.clearnlp.bin.NLPDecode.decode(NLPDecode.java:108)
    at edu.emory.clir.clearnlp.bin.NLPDecode.<init>(NLPDecode.java:74)
    at edu.emory.clir.clearnlp.bin.NLPDecode.main(NLPDecode.java:236)

Installation without maven still refers to version 3.1.2

Cannot run the constituency-to-dependency program

1. I directly install on maven.
2. Then I directly run
   C2DConvert.main(args);
3. where the args are exactly same as your example.

Error:

java.lang.IllegalArgumentException: InputStream cannot be null
    at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:117)
    at edu.emory.clir.clearnlp.util.XmlUtils.getDocumentElement(XmlUtils.java:173)
    at edu.emory.clir.clearnlp.component.mode.morph.EnglishMPAnalyzer.<init>(EnglishMPAnalyzer.java:73)
    at edu.emory.clir.clearnlp.component.utils.NLPUtils.getMPAnalyzer(NLPUtils.java:79)
    at edu.emory.clir.clearnlp.bin.C2DConvert.<init>(C2DConvert.java:82)
    at edu.emory.clir.clearnlp.bin.C2DConvert.main(C2DConvert.java:264)
    at data.preprocess.Test.main(Test.java:13)
java.lang.IllegalArgumentException: InputStream cannot be null
    at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:117)
    at edu.emory.clir.clearnlp.util.XmlUtils.getDocumentElement(XmlUtils.java:173)
    at edu.emory.clir.clearnlp.component.mode.morph.EnglishMPAnalyzer.<init>(EnglishMPAnalyzer.java:74)
    at edu.emory.clir.clearnlp.component.utils.NLPUtils.getMPAnalyzer(NLPUtils.java:79)
    at edu.emory.clir.clearnlp.bin.C2DConvert.<init>(C2DConvert.java:82)
    at edu.emory.clir.clearnlp.bin.C2DConvert.main(C2DConvert.java:264)
    at data.preprocess.Test.main(Test.java:13)
java.lang.NullPointerException
    at edu.emory.clir.clearnlp.util.XmlUtils.getFirstElementByTagName(XmlUtils.java:75)
    at edu.emory.clir.clearnlp.component.mode.morph.EnglishMPAnalyzer.getInflectionRules(EnglishMPAnalyzer.java:141)
    at edu.emory.clir.clearnlp.component.mode.morph.EnglishMPAnalyzer.<init>(EnglishMPAnalyzer.java:78)
    at edu.emory.clir.clearnlp.component.utils.NLPUtils.getMPAnalyzer(NLPUtils.java:79)
    at edu.emory.clir.clearnlp.bin.C2DConvert.<init>(C2DConvert.java:82)
    at edu.emory.clir.clearnlp.bin.C2DConvert.main(C2DConvert.java:264)
    at data.preprocess.Test.main(Test.java:13)

How to cite this repo?

Hi all, I'm writing a research paper and I'd like to cite this repo's list of dependencies. Is there a bibliography for this repo?

train

I want to train with follow command.
java -Xmx10g -XX:+UseConcMarkSweepGC edu.emory.clir.clearnlp.bin.NLPTrain -mode dep -c config_train_dep.xml -f feature_en_dep.xml -t wsj_0001.parse.dep -d clearnlp.txt.cnlp -m dummy-dep.xz

but follow erorr represented:
Picked up JAVA_TOOL_OPTIONS: -javaagent:/usr/share/java/jayatanaag.jar
java.io.IOException: Stream closed
at java.io.BufferedInputStream.getInIfOpen(BufferedInputStream.java:159)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at java.io.DataInputStream.readFully(DataInputStream.java:195)
at java.io.DataInputStream.readFully(DataInputStream.java:169)
at org.tukaani.xz.SingleXZInputStream.initialize(Unknown Source)
at org.tukaani.xz.SingleXZInputStream.(Unknown Source)
at org.tukaani.xz.XZInputStream.(Unknown Source)
at org.tukaani.xz.XZInputStream.(Unknown Source)
at edu.emory.clir.clearnlp.component.utils.NLPUtils.getObjectInputStream(NLPUtils.java:173)
at edu.emory.clir.clearnlp.component.utils.NLPUtils.getDistributionalSemantics(NLPUtils.java:166)
at edu.emory.clir.clearnlp.component.utils.GlobalLexica.lambda$initDistributionalSemanticsWords$23(GlobalLexica.java:58)
at edu.emory.clir.clearnlp.component.utils.GlobalLexica$$Lambda$1/553264065.apply(Unknown Source)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:512)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:502)
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
at edu.emory.clir.clearnlp.component.utils.GlobalLexica.initDistributionalSemanticsWords(GlobalLexica.java:58)
at edu.emory.clir.clearnlp.component.utils.GlobalLexica.init(GlobalLexica.java:46)
at edu.emory.clir.clearnlp.bin.helper.AbstractNLPTrain.train(AbstractNLPTrain.java:87)
at edu.emory.clir.clearnlp.bin.helper.AbstractNLPTrain.(AbstractNLPTrain.java:78)
at edu.emory.clir.clearnlp.bin.NLPTrain.(NLPTrain.java:37)
at edu.emory.clir.clearnlp.bin.NLPTrain.main(NLPTrain.java:44)
log4j:WARN No appenders could be found for logger (edu.emory.clir.clearnlp.util.BinUtils).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
java.lang.NullPointerException
at edu.emory.clir.clearnlp.component.utils.NLPUtils.getDistributionalSemantics(NLPUtils.java:157)
at edu.emory.clir.clearnlp.component.utils.NLPUtils.getDistributionalSemantics(NLPUtils.java:166)
at edu.emory.clir.clearnlp.component.utils.GlobalLexica.lambda$initDistributionalSemanticsWords$23(GlobalLexica.java:58)
at edu.emory.clir.clearnlp.component.utils.GlobalLexica$$Lambda$1/553264065.apply(Unknown Source)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:512)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:502)
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
at edu.emory.clir.clearnlp.component.utils.GlobalLexica.initDistributionalSemanticsWords(GlobalLexica.java:58)
at edu.emory.clir.clearnlp.component.utils.GlobalLexica.init(GlobalLexica.java:46)
at edu.emory.clir.clearnlp.bin.helper.AbstractNLPTrain.train(AbstractNLPTrain.java:87)
at edu.emory.clir.clearnlp.bin.helper.AbstractNLPTrain.(AbstractNLPTrain.java:78)
at edu.emory.clir.clearnlp.bin.NLPTrain.(NLPTrain.java:37)
at edu.emory.clir.clearnlp.bin.NLPTrain.main(NLPTrain.java:44)
java.lang.NullPointerException
at edu.emory.clir.clearnlp.component.utils.GlobalLexica.getDistributionalSemanticFeatures(GlobalLexica.java:69)
at edu.emory.clir.clearnlp.feature.common.CommonFeatureExtractor.getFeatures(CommonFeatureExtractor.java:82)
at edu.emory.clir.clearnlp.feature.common.CommonFeatureExtractor.getFeatures(CommonFeatureExtractor.java:35)
at edu.emory.clir.clearnlp.feature.AbstractFeatureExtractor.getFeatures(AbstractFeatureExtractor.java:197)
at edu.emory.clir.clearnlp.feature.AbstractFeatureExtractor.addSetFeatures(AbstractFeatureExtractor.java:151)
at edu.emory.clir.clearnlp.feature.AbstractFeatureExtractor.addFeatures(AbstractFeatureExtractor.java:116)
at edu.emory.clir.clearnlp.feature.AbstractFeatureExtractor.createStringFeatureVector(AbstractFeatureExtractor.java:98)
at edu.emory.clir.clearnlp.component.mode.dep.AbstractDEPParser.createStringFeatureVector(AbstractDEPParser.java:117)
at edu.emory.clir.clearnlp.component.mode.dep.AbstractDEPParser.createStringFeatureVector(AbstractDEPParser.java:36)
at edu.emory.clir.clearnlp.component.AbstractStatisticalComponent.train(AbstractStatisticalComponent.java:278)
at edu.emory.clir.clearnlp.component.AbstractStatisticalComponent.process(AbstractStatisticalComponent.java:265)
at edu.emory.clir.clearnlp.component.mode.dep.AbstractDEPParser.process(AbstractDEPParser.java:96)
at edu.emory.clir.clearnlp.component.trainer.AbstractNLPTrainer.process(AbstractNLPTrainer.java:268)
at edu.emory.clir.clearnlp.component.trainer.AbstractNLPTrainer.process(AbstractNLPTrainer.java:251)
at edu.emory.clir.clearnlp.component.trainer.AbstractNLPTrainer.train(AbstractNLPTrainer.java:104)
at edu.emory.clir.clearnlp.component.trainer.AbstractNLPTrainer.train(AbstractNLPTrainer.java:53)
at edu.emory.clir.clearnlp.bin.helper.AbstractNLPTrain.train(AbstractNLPTrain.java:89)
at edu.emory.clir.clearnlp.bin.helper.AbstractNLPTrain.(AbstractNLPTrain.java:78)
at edu.emory.clir.clearnlp.bin.NLPTrain.(NLPTrain.java:37)
at edu.emory.clir.clearnlp.bin.NLPTrain.main(NLPTrain.java:44)

In addition,I wrtie follow command in terminal :
CLEARNLP=/home/clearnlp
export CLASSPATH=$CLEARNLP/clearnlp-3.1.2.jar:$CLEARNLP/args4j-2.0.29.jar:$CLEARNLP/log4j-1.2.17.jar:$CLEARNLP/hppc-0.6.1.jar:$CLEARNLP/xz-1.5.jar:$CLEARNLP/log4j.properties:.

Deprecated Labels

It says here https://github.com/clir/clearnlp-guidelines/blob/master/md/specifications/dependency_labels.md that NPADVMOD was replaced by NPMOD and that NMOD was replaced by NOUNMOD, but I still get these two tags in the trees. I'm using ClearNLP 3.1.

NER specification

Hi,

Could anyone tell me where I can find the tagset for the default named entity recognizer?

How to run SRL with my own tokenization?

My sentences are already tokenized. Is there an option in ClearNLP that I can turn on (or off) to let it perform semantic role labeling using the existing tokenization?

Statement in README is outdated: "Coreference resolution (coming in Nov, 2015)"

decoding with new trained model

Hello Jinho,

I managed to get it to train, but I'm having trouble decoding with my model.

I went as far as renaming my model to "general-en-pos.xz", replacing the original model in the "clearnlp-general-en-pos-3.2.jar" with mine and creating a new jar keeping the name "clearnlp-general-en-pos-3.2.jar" the same. It still gives me the error below.

java.io.IOException: Stream closed
at java.io.BufferedInputStream.getInIfOpen(BufferedInputStream.java:159)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at java.io.DataInputStream.readFully(DataInputStream.java:195)
at java.io.DataInputStream.readFully(DataInputStream.java:169)
at org.tukaani.xz.SingleXZInputStream.initialize(Unknown Source)
at org.tukaani.xz.SingleXZInputStream.(Unknown Source)
at org.tukaani.xz.XZInputStream.(Unknown Source)
at org.tukaani.xz.XZInputStream.(Unknown Source)
at edu.emory.clir.clearnlp.component.utils.NLPUtils.getObjectInputStream(NLPUtils.java:173)
at edu.emory.clir.clearnlp.component.utils.NLPUtils.getPOSTagger(NLPUtils.java:94)
at edu.emory.clir.clearnlp.bin.NLPDecode.getComponents(NLPDecode.java:207)
at edu.emory.clir.clearnlp.bin.NLPDecode.decode(NLPDecode.java:89)
at edu.emory.clir.clearnlp.bin.NLPDecode.(NLPDecode.java:74)
at edu.emory.clir.clearnlp.bin.NLPDecode.main(NLPDecode.java:236)
log4j:WARN No appenders could be found for logger (edu.emory.clir.clearnlp.util.BinUtils).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
java.lang.NullPointerException
at edu.emory.clir.clearnlp.component.AbstractStatisticalComponent.load(AbstractStatisticalComponent.java:157)
at edu.emory.clir.clearnlp.component.AbstractStatisticalComponent.initDecode(AbstractStatisticalComponent.java:126)
at edu.emory.clir.clearnlp.component.AbstractStatisticalComponent.(AbstractStatisticalComponent.java:99)
at edu.emory.clir.clearnlp.component.mode.pos.AbstractPOSTagger.(AbstractPOSTagger.java:60)
at edu.emory.clir.clearnlp.component.mode.pos.EnglishPOSTagger.(EnglishPOSTagger.java:49)
at edu.emory.clir.clearnlp.component.utils.NLPUtils.getPOSTagger(NLPUtils.java:87)
at edu.emory.clir.clearnlp.component.utils.NLPUtils.getPOSTagger(NLPUtils.java:94)
at edu.emory.clir.clearnlp.bin.NLPDecode.getComponents(NLPDecode.java:207)
at edu.emory.clir.clearnlp.bin.NLPDecode.decode(NLPDecode.java:89)
at edu.emory.clir.clearnlp.bin.NLPDecode.(NLPDecode.java:74)
at edu.emory.clir.clearnlp.bin.NLPDecode.main(NLPDecode.java:236)
Exception in thread "main" java.lang.NullPointerException
at edu.emory.clir.clearnlp.component.mode.pos.AbstractPOSTagger.createStringFeatureVector(AbstractPOSTagger.java:120)
at edu.emory.clir.clearnlp.component.mode.pos.AbstractPOSTagger.createStringFeatureVector(AbstractPOSTagger.java:33)
at edu.emory.clir.clearnlp.component.AbstractStatisticalComponent.decode(AbstractStatisticalComponent.java:294)
at edu.emory.clir.clearnlp.component.AbstractStatisticalComponent.process(AbstractStatisticalComponent.java:267)
at edu.emory.clir.clearnlp.component.mode.pos.AbstractPOSTagger.process(AbstractPOSTagger.java:105)
at edu.emory.clir.clearnlp.bin.NLPDecode.process(NLPDecode.java:163)
at edu.emory.clir.clearnlp.bin.NLPDecode.process(NLPDecode.java:153)
at edu.emory.clir.clearnlp.bin.NLPDecode.decode(NLPDecode.java:107)
at edu.emory.clir.clearnlp.bin.NLPDecode.(NLPDecode.java:74)
at edu.emory.clir.clearnlp.bin.NLPDecode.main(NLPDecode.java:236)

Would appreciate help with moving forward.

Thank you!