Combine Tokenizers for better results

[dogeweb]

Combine Tokenizers for better results

[manishobhatia]

Hi,
Thanks for taking interest in this project. I like the idea of concatTokenizer , and I think it can be a useful addition.

I do want to make sure I understand the problem a little better. Here is my take on the issue you laid out

• "Caputo" and "Chabot" matched equal with wordSoundexEncodeTokenizer : This relies on Apache Soundex library which tries to find words with similar phonemes. I can see in this case it might have generate the same code for both these words
• "Nikolau" and "Nikolaou" did not match with wordTokenizer : This again is expected, as it is doing a string equals. But "wordSoundexEncodeTokenizer" would been a better fit here
• "Leao" and "Rafael Leao" did not match with triGramTokenizer: I think it would have matched, but did not go above the default threshold. This function breaks the words in tokens of 3 , and since the "Leao" will generate less tokens [lea, eao] (2) compared with "Rafael Leao" [Raf, afa, fae, ael, el , l L, Le, Lea, eao] (9). This match score will only be around 2/9 . I think in this case either changing the threshold or using the above two tokenizer will be better

But what I understand in your case, you probably have all these variation of data in the same element, and changing Tokenizers is not an option, but if you add all types of tokens, you have a better chance of matching. Is that correct ?

I do see concatTokenizer a useful add for such scenarios. Just a caution on performance, as you data size grows large, the additional token will slow down the match.

But in any case, I am open to add this into the library. If you would like to do a Pull Request with some unit tests, I can have this out in our next release

[dogeweb]

Thanks for the reply.

But what I understand in your case, you probably have all these variation of data in the same element, and changing Tokenizers is not an option, but if you add all types of tokens, you have a better chance of matching. Is that correct ?

Yes, I have a single data set where I have to match a mix of exact matches, typos and other variations like missing spaces, so the use of a single tokenizer was skipping some. Using a combination of two or three tokenizers got all the cases covered in a single run.

Just a caution on performance, as you data size grows large, the additional token will slow down the match.

I understand, I have a small set of ~200 docs matched against ~500 so I didn't consider performance. I will keep in mind that.

If you would like to do a Pull Request with some unit tests, I can have this out in our next release

Sure, I hope to do it in relative short time.