NullPointerException in StandardTokenizerImpl in Solr 4.1.0

[nolanlawson]

NullPointerException in StandardTokenizerImpl in Solr 4.1.0

[sarowe]

Looks like zzReader wasn't initialized - here's the StandardTokenizerImpl statement where the NPE occurs:

923: int numRead = zzReader.read(zzBuffer, zzEndRead,
924: zzBuffer.length-zzEndRead);

If you look at the package javadocs for o.a.l.analysis: http://lucene.apache.org/core/4_1_0/core/org/apache/lucene/analysis/package-summary.html and the tokenstream workflow http://lucene.apache.org/core/4_0_0/core/org/apache/lucene/analysis/TokenStream.html (step 2), you can see the the reset() method on a tokenstream must be called before it's used.

You didn't do that.

Why is this suddenly a problem in Lucene 4.1? Calling reset() prior to consuming a tokenstream has been part of the contract since Lucene 2.9, but has not been strictly enforced. This commit, between the 4.0 and 4.1 releases, triggered the change: http://svn.apache.org/viewvc?view=revision&revision=1401449

[nolanlawson]

Thanks for the insight! That certainly fixed it; Solr 4.1.0 works fine now. Bad on me for not reading the documentation for TokenStream.