Investigate generalization

[healthonnet]

Investigate generalization

[nolanlawson]

I agree. It would be nice if this code could work in either (pure) Lucene, Solr, or ElasticSearch. In truth, the idea itself is not fairly complicated: take the user query, expand it into all possible synonym combinations, parse each of those queries as you normally would, and then combine them back together. There's nothing in there that explicitly depends on EDisMax, or even Solr.

I think the best way I could accomplish this for now would be to make QParserPlugin its own class rather than a subclass of ExtendedDisMaxQParserPlugin. It could take ExtendedDisMaxQParserPlugin as a constructor argument and delegate to it, rather than subclassing it. Then the code would be fairly self-contained.

[janhoy]

Yep. If done right, the core code could be a new Lucene module, usable from any query parser.

Doing a Lucene module would be a nice first step, then we can tackle Solr and eDisMax next. It's also the preferred/proper way of separating code between Lucene and Solr.

[mickdelaney]

hi,
and firstly great work, i'm a newbie to lucene/solr but am getting my head around what you've done pretty quickly.

I was thinking could it be generalized further? i.e. not just handling synonyms but any sort of related term expansion.
e.g. narrower terms, related terms etc.

At the moment I'm looking at using the code here to allow our taxonomy which contains terms to be expanded into the query, where the taxonomy has narrower terms they'd be added to the query with a lower priority.

does that make sense?

[nolanlawson]

@mickdelaney : I think I understand what you're proposing. It's a little bit beyond the scope of what I tried to do with this plugin, though. I'm only trying to use a flat list of synonyms, whereas it sounds like you want to weight certain synonyms over others.

I think you could probably modify the code pretty easily to do what you want. Although, since the default SynonymFilterFactory (which I use) doesn't support weighted synonyms, you'd have to rewrite a lot of that from scratch, as well.

[nolanlawson]

@janhoy: The more I work on this plugin, the more it seems to become inextricably dependent upon Solr. I don't think it's very realistic anymore to imagine generalizing it to Lucene/ElasticSearch.

And anyway, the idea itself is pretty trivial. If folks want it for Lucene or ElasticSearch, they should be able to code it up themselves using a basic string manipulation layer on top of the query engine.

However, I'm at a point now where it would be easy to allow people to use plain ol' dismax and lucene instead of edismax as the query parser. I dunno if there's much interest in that, though - aren't all of us just using edismax anyway? 😉