Turkish (iso-8859-9) detecting.

[chardet]

Turkish (iso-8859-9) detecting.

[queeup]

I add Turkish ISO-8859-9 detection support d0749bc

But I have a problem. If I leave this line 0.970290 it always detect ISO-8859-2. It's wrong detection.

debug:

$ chardetect divxplanet.srt 
SHIFT_JIS prober hit error at byte 8
EUC-JP prober hit error at byte 9
EUC-KR prober hit error at byte 9
CP949 prober hit error at byte 9
EUC-TW prober hit error at byte 9
GB2312 prober hit error at byte 20
Big5 prober hit error at byte 20
windows-1251 confidence = 0.0754944175953
KOI8-R confidence = 0.0
ISO-8859-5 confidence = 0.412478438097
MacCyrillic confidence = 0.0789081955208
IBM866 confidence = 0.01
IBM855 confidence = 0.0
ISO-8859-7 confidence = 0.141775571572
windows-1253 confidence = 0.141775571572
ISO-8859-5 confidence = 0.253281202428
windows-1251 confidence = 0.0
ISO-8859-2 confidence = 0.877661831311
windows-1250 confidence = 0.877661831311
TIS-620 confidence = 0.01
ISO-8859-9 confidence = 0.636039383618
ISO-8859-8 confidence = 0.0
ISO-8859-8 confidence = 0.349085294633
ISO-8859-8 confidence = 0.349085294633
windows-1251 confidence = 0.0754944175953
KOI8-R confidence = 0.0
ISO-8859-5 confidence = 0.412478438097
MacCyrillic confidence = 0.0789081955208
IBM866 confidence = 0.01
IBM855 confidence = 0.0
ISO-8859-7 confidence = 0.141775571572
windows-1253 confidence = 0.141775571572
ISO-8859-5 confidence = 0.253281202428
windows-1251 confidence = 0.0
ISO-8859-2 confidence = 0.877661831311
windows-1250 confidence = 0.877661831311
TIS-620 confidence = 0.01
ISO-8859-9 confidence = 0.636039383618
ISO-8859-8 confidence = 0.0
ISO-8859-8 confidence = 0.349085294633
ISO-8859-8 confidence = 0.349085294633
divxplanet.srt: ISO-8859-2 with confidence 0.877661831311

But if I change that number on langturkishmodel.py to 0.670290 it detects correct.
debug:

$ chardetect divxplanet.srt 
SHIFT_JIS prober hit error at byte 8
EUC-JP prober hit error at byte 9
EUC-KR prober hit error at byte 9
CP949 prober hit error at byte 9
EUC-TW prober hit error at byte 9
GB2312 prober hit error at byte 20
Big5 prober hit error at byte 20
windows-1251 confidence = 0.0754944175953
KOI8-R confidence = 0.0
ISO-8859-5 confidence = 0.412478438097
MacCyrillic confidence = 0.0789081955208
IBM866 confidence = 0.01
IBM855 confidence = 0.0
ISO-8859-7 confidence = 0.141775571572
windows-1253 confidence = 0.141775571572
ISO-8859-5 confidence = 0.253281202428
windows-1251 confidence = 0.0
ISO-8859-2 confidence = 0.877661831311
windows-1250 confidence = 0.877661831311
TIS-620 confidence = 0.01
ISO-8859-9 confidence = 0.92070992187
ISO-8859-8 confidence = 0.0
ISO-8859-8 confidence = 0.349085294633
ISO-8859-8 confidence = 0.349085294633
windows-1251 confidence = 0.0754944175953
KOI8-R confidence = 0.0
ISO-8859-5 confidence = 0.412478438097
MacCyrillic confidence = 0.0789081955208
IBM866 confidence = 0.01
IBM855 confidence = 0.0
ISO-8859-7 confidence = 0.141775571572
windows-1253 confidence = 0.141775571572
ISO-8859-5 confidence = 0.253281202428
windows-1251 confidence = 0.0
ISO-8859-2 confidence = 0.877661831311
windows-1250 confidence = 0.877661831311
TIS-620 confidence = 0.01
ISO-8859-9 confidence = 0.92070992187
ISO-8859-8 confidence = 0.0
ISO-8859-8 confidence = 0.349085294633
ISO-8859-8 confidence = 0.349085294633
divxplanet.srt: ISO-8859-9 with confidence 0.92070992187

I need some guidance about how can I do it correct Turkish detection.
I took Turkish code from cChardet: https://github.com/PyYoshi/cChardet/blob/master/src/ext/libcharsetdetect/mozilla/extensions/universalchardet/src/base/LangTurkishModel.cpp

[dan-blanchard]

I'm fairly new to the chardet team, so I'm not sure how everything works yet. That said, I think whatever you have to set that number to to get it to work is fine.

The more I look at cChardet, I think their system probably works better than chardet does. We'll have to see if there's more we can do to close the gap.

[queeup]

Thanks for feedback. What do you think? Do you want me to pull request or just wait a bit and see what going to be your direction?

[dan-blanchard]

Please submit a pull request. I think we can have this in the next release.

On Wed, Jan 22, 2014 at 3:12 PM, queeup [email protected] wrote:

Thanks for feedback. What do you think? Do you want me to pull request or
just wait a bit and see what going to be your direction?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/20#issuecomment-33062534
.

[queeup]

Pull request submitted. #21

[dan-blanchard]

Fixed by #41.