Misdetection of utf-8 as ISO-8859-2 for german umlauts

[chardet]

Misdetection of utf-8 as ISO-8859-2 for german umlauts

[dan-blanchard]

Unfortunately, it's an artifact of having to rely on statistical models to detect encodings. Sometimes we're wrong because there's not enough evidence. One character with an umlaut doesn't give us much to go on. If you know you're going to be processing things like Python source files that might have a header that actually tells you the encoding, that's the better thing to rely on.

Thanks for the reply. OK, i understand. I just have been wondering why file gets it correctly.

[dan-blanchard]

file probably just has models that are biased toward UTF-8. It's also possible it checks for headers. I'm not sure the exact algorithm uses, but it can detect far fewer encodings than we can.

[Tronic]

Valid UTF-8 sequences (w or w/o BOM) as well as any Unicode sequence with BOM should always take precedence over 8-bit encodings. Unfortunately I'm getting more failures than correct detections from chardet, and often an encoding that cannot even be decoded is reported. I was forced to write my own function that, despite being simple, performs much better. https://paste.zi.fi/p/decode.py/view