failed to detect utf8 "FAHR•WERK"

[chardet]

failed to detect utf8 "FAHR•WERK"

[asdfsx]

maybe this can also solve the problem, I mentioned before, #60
@dan-blanchard @sigmavirus24

[asdfsx]

any response???about this issue???@dan-blanchard

[dan-blanchard]

Since chardet uses statistics to determine encodings, it has poor performance for short strings.

Also, we must only rely on multi-byte characters for determining if a string is UTF-8 because we want to be able to detect single-byte encodings like ASCII and Windows-1252.

Could you please explain what exactly you're trying to use chardet for and how you're calling it? Did you run chardet.detect(b'FAHR%E2%80%A2WERK')?

[asdfsx]

No, I run somethink like this

import urlparse
import chardet
a = 'FAHR%E2%80%A2WERK'
b = urlparse.unquote(a)
chardet.detect(b)

I use it mainly for detect short characters, such as website's title.
In china, japan or somewhere else, website use not only utf-8 but also gbk, shift-jis, etc. So I need to detect what kind of character the title is, then translate them into utf-8. And people sometime likes to add some special symbols into the title like '★'，'•'. These thing may make the detecting failed.

I got another example, may be it's not so good

chardet.detect("'我' means I in chinese")

[sils]

Have a similar problem, running chardet on a simple python file containing umlauts which reads totally fine when just using utf-8, but chardet detects iso-8859-2. Here's our code: https://gitlab.com/coala/coala-utils/commit/d5b707689a150d1c561d3fa0912e5d1cb79f0441#59e25ab47f5113468cf960272779acd36dcfb680_0_38 which fails the test

[dan-blanchard]

chardet is inaccurate for short strings, as has been stated many times. Being able to restrict the encoding set as suggested in #70 would go a long way to fixing some of these common issues, so I will prioritize adding that in the next version.