Thai chars are in the wrong charsets

[maximium]

Thai chars are in the wrong charsets

[sanikolaev]

@maximium Thanks for bringing this to our attention and for the PR.

@Nick-S-2018 pls review the PR.

[Nick-S-2018]

@maximium The reason why Thai chars are currently in 'cjk' is that our 'cjk' charset, in fact, comprises the languages which don't use spaces between words. We've done that to help users to deal with setting ngram_chars for such languages.

@sanikolaev It appears that the optimal solution is to move the Thai script to a separate charset that can be used along with cjk and non_cjk.

[maximium]

@sanikolaev It appears that the optimal solution is to move the Thai script to a separate charset that can be used along with cjk and non_cjk.

Indeed. The problem with having thai in cjk is tokenizing every letter as a word, which is not correct and gives a lot of irrelevant results. The only way I see to handle thai text is to split text to separate words in app before indexing with some dictionary based tokenizer. And separate Thai charset fits perfectly for this.