Can not distinguish UTF-8 and UTF-8-SIG

[chardet]

Can not distinguish UTF-8 and UTF-8-SIG

[sigmavirus24]

@atbest notice your result. The confidence value is 0.99, not 1.0 That means that this is not matching the way you expect it to, so this conditional is not evaluating to true. That means that chardet then starts using the probers to determine the character encoding. You'll likely have to determine how best to make a UTF8SIGProber in the likeness of UTF8Prober and a similar State Machine Model. Even then, given how similar the two encodings are (utf-8, utf-8-sig) you should understand that chardet works with probabilities and returns the encoding that statistically seems most likely so getting a consistently correct answer could be extraordinarily difficult.

I do appreciate that you're trying to fix this though.

[atbest]

Thanks for the information. And I found it's very easy to fix. Just change the codecs.BOM to codecs.BOM_UTF8.

            if aBuf[:3] == codecs.BOM_UTF8:
                # EF BB BF  UTF-8 with BOM
                self.result = {'encoding': "UTF-8-SIG", 'confidence': 1.0}

[dan-blanchard]

Could you please make a PR for this?

[sigmavirus24]

@atbest please be sure to add this instead of modifying the existing lines.

[atbest]

@dan-blanchard Sorry, what does PR stand for? I am new here...
@sigmavirus24 According to this page, codecs.BOM is FF FE, while aBuf[:3] is three bytes, and aBuf[:3] == codecs.BOM is always False, so it is not necessary to keep existing lines.

[sigmavirus24]

@atbest you're right. Looking at the official documentation codecs.BOM is an alias for codecs.BOM_UTF16.

A PR is a Pull Request. You fork this repository, push your changes to your copy (preferably on a separate branch) and then click the button to create a pull request to this repository.

[dan-blanchard]

@atbest you can also just click the "edit" button on the GitHub page for the specific file you want to edit. That'll create the fork for you and make a PR when you hit save. It can be handy when you only have to make a one file change.

[sigmavirus24]

@dan-blanchard it can be handy but I've also seen it destroy formatting because the ACE editor is terrible.

[atbest]

Allright, done. Thanks.

[sigmavirus24]

Thanks @atbest

[dan-blanchard]

Fixed via #32.

Thanks @atbest!