Comments (8)
Thanks for the bug report. I actually came across this difference as well when I was working on trying to update a lot of our code to incorporate upstream changes. I wasn't sure if it was necessary for our Python port or not, so I didn't actually incorporate the change.
Also, @PyYoshi didn't actually make that change, he just wrapped uchardet-enhanced, which (as you said) is the only place that has that change.
Unfortunately all of the other predictions from uchardet-enhanced seem to be much less accurate on our test set, so I wasn't sure if I should trust that change or not. You can see some discussion at the bottom of issue #48.
from chardet.
Thanks for the link to uchar-enhanced. Their database for creating language models is funny small. I have spent one month collecting approx. 10MB raw text per language :) (I have created 7 Central European language models).
It would be very useful to have more infos month ago when I was to learn how to build a language model. I had only one relevant info from mozilla doc. Therefore I had to read "between the lines" :)
I have one question. How to estimate/calculate good "typical positive ratio"?
Which expression is good?
typical_ratio = float(total_num_sequences - total_num_negative_sequences) / total_num_sequences * total_num_frequent_chars / total_num_chars
vs
typical_ratio = float(total_num_positive_sequences) / total_ num_sequences * total_num_frequent_chars / total_num_chars
from chardet.
Let me explain what is the difference with or without "-1":
The order (mModel->charToOrderMap[(unsigned char)aBuf[i]])
is index to twochars (sequences) table (with values 0=negative, 1, 2, 3=positive occurence) created from 64 (=SAMPLE_SIZE) the most frequented chars sorted by their occurence probability in descending order. This table is list with first index=0 (standard python behavior).
If you omit "-1" you not only cut off the most frequent twochar occurences (exactly SAMPLE_SIZE+1 length) but you make twochars table inconsistent to charmap table.
For example: Let 'e' and 'a' are the most frequent letters. 'e' has order=1 in the charmap table and 'a' has order=2. The twochars table therefore has 'ee' "occurence value" (0,1,2 or 3) in the first place in the list (index "0") and 'ea' value has the second place (index "1"). If the line of the source code above gives order=1 for 'e' and order=2 for 'a' then expression
i = (last_orderSAMPLE_SIZE)+order
gives i=64
1+2
for 'ea', and it points somewhere else but not to second item in our twochars table. If you add "-1" to order evaluation then you get
i=64*0+1=1
what is correct index value
from chardet.
Thanks for trying to explain the reasons why you think - 1
should be included.
In spite of the seeming correctness of adding that - 1
, if we add that our detection accuracy plummets. We fail 41 unit tests (instead of the 1 known failure we have right now) when adding that in, so I'm not very inclined to make the change.
It seems to me that what is likely happening is that when the tables were originally created by Mozilla, they were created in such a fashion that this error was happening on the generation side too.
When we address #48 and retrain all of the models in a more sensible fashion, we will want to address this issue, but I think we shouldn't until then.
from chardet.
"It seems to me that what is likely happening is that when the tables were originally created by Mozilla, they were created in such a fashion that this error was happening on the generation side too."
Ok I understand, it is pity, but it is very simple to shift current data to the left and add zeros to the end of the list, because data in the list are sorted by probability. Btw first 64+1 values are unnecesary because they will never be indexed and also it is very improbable that some twochars sequences will be indexed to the end of table.
from chardet.
Since this is a longstanding bug, we can certainly shift the data over and add the adjustment like you suggested. My wife just had a baby on the 8th, so I don't have much time to dedicate to this at the moment, so if you could make a pull request that includes this fix (and shifts the data), that would be enormously helpful.
from chardet.
Congratulations!!!!!
Of course. I've tested models (nosetests) with shifted data and the results was exactly same, therefore we can do it what you suggest.
from chardet.
The changes discussed here are part of #99.
from chardet.
Related Issues (20)
- detect encode wrong!
- Detect pep-0263
- test_detect_all_and_detect_one_should_agree fails on Python 3.11b3 HOT 4
- Dependency warning (v5.0.0) HOT 1
- chardet 5.0 KeyError with Python 3.10 on Windows HOT 5
- Is the license LGPL v2.1 or later or just LGPLv2.1 only? HOT 3
- Documentation licensed only to non-commercial and personal use found
- Documentation licensed only to non-commercial and personal use found HOT 1
- Allow running of the package via `python3 -m chardet ...` HOT 4
- Encoding error
- Next release for Python 3.11 HOT 1
- type annotation and implementation mismatch HOT 2
- How to use Chardet for this Python code, as to read files that have ANSI encoder?
- chardetect cli: UnicodeEncodeError when filename is not utf8
- wrong result. actual johab - expected latin1 HOT 4
- Failed to detect CP932 encoded file
- pip intall chardet
- `chardet.detect` a lot slower than using `UniversalDetector.feed` with chunks
- chardet detect UTF-8 XML File as EUC_KR - Possibility to exclude encodings?
- Wrong detection UTF-8 with ΓΆ symbol
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from chardet.