Issue with detection of English text

[patrickschur]

Issue with detection of English text

[iquito]

I have used a maxLength of 4, it helped my detection results quite a bit (together with setting maxNgrams to 9000).

The library also "only" includes the Universal Declaration of Human Rights as the base for the language detection, so many common phrases and words will be missing (like country names, idioms, greetings, specific nouns). Depending on the usage it makes sense to add additional words/sentences to the language samples, so they can factor in. I did it for my use case (analyzing emails) and it worked very well.

[patrickschur]

You have to know that english is very similiar to other languages and the sentences you choosed are not very long, this two things make it very hard to detect the right language. On the other side this library is more or less optimized on speed. If you want to improve the detection phase please have a look at here. Where I described the procedure how to improve the detection.

Maybe I should increase the detection by default a little bit?

If you need any further help let me know. :)

[nachitox]

What about setting maxLength? Does a bigger number improve detection?

[patrickschur]

The length of ngrams does not improve the detection much. Maybe setting the maxLength to 4 or 5 will make a difference but I wouldn't go further otherwise it's getting even worse.

Update: If you want to improve the detection be sure to set the maxNgrams to a value greater than 300. A good value would be 3000. For more information look here.