Stemmer, tokenizer, thesaurus, etc

[mkosieradzki]

Stemmer, tokenizer, thesaurus, etc

[jessebenson]

This was more of a PoC that I pitched to the Service Fabric team. They are implementing indexing but not full-text indexing. Stemming/tokenizers/etc. would absolutely necessary to make the full-text indexing generically useful.

Additionally, the current implementation performs significantly worse for low cardinality indexes with large datasets. This is because the index maintains a (key, value) of (index key, [array of keys]) in a Reliable Dictionary. Service Fabric performs poorly when the value (i.e. [array of keys]) is large, because there is no differential replication. That is, the entire value gets serialized, logged, and replicated.

[mkosieradzki]

@jessebenson Thanks for your answer.

Yup, lack of incremental replication/merge operator is a pain in reliable collections.

There are some stemmers and stuff implemented in Lucene.NET https://github.com/apache/lucenenet/tree/master/src , unfortunately it is many versions behind the Java version.

It's a bit disappointing they are not implementing full text search, however it is also understandable due to the complexity of the analytical part...