filter_with_english_letters

[chardet]

filter_with_english_letters

[sigmavirus24]

You're referring to filter_with_english_letters right? I'm surprised that function is doing anything when used from here. Could you expand on this?

[rsnair2]

Oh sorry, I should have been more specific. I was referring to the original implementation by Mozilla which can be found at cChardet as well: cChardet .

[dan-blanchard]

The implementation used by cChardet is not actually the current reference implementation anymore. I'm trying to target revision 207643 on the Mozilla Hg repository because that was the last version before they pulled out support for several older encodings. I've got a bit of a start on that in #42 .

[dan-blanchard]

I've now got this added to #42. It's currently a fairly literal C translation, but could probably be improved in places.

As I'm going over more of the upstream code, I'm becoming more and more convinced that we likely do not want to emulate the Mozilla implementation anymore when it comes to Western encodings. Like @rsnair2 said, filter_with_english_letters does some really bizarre HTML tag removal, but it's only used for Latin1Prober. I'm not sure it makes sense to apply to all documents, since there are plenty of non-XML/HTML documents that use < and > that will probably be thrown off by that filtering.

Another weird bit I've discovered in the current Mozilla code is that the Hungarian detectors are just commented out and unused. We don't have them commented out, and we frequently detect them instead of Windows-1252.

[dan-blanchard]

Fixed via #42.