Comments (8)
And utf-7 ? :P
from charset_normalizer.
Yes you can test it. To compare with Python 3.7, a feature in unicode 12 needs to be found, because Python 3.7 iirc still has unicode 11. The most well known differences are the emoji list https://emojipedia.org/unicode-12.0/ , but there are also 4 new scripts. As unicodedata only provides the characters, and doesnt provide more "encodings", using a more recent unicodedata
is mostly useful for avoiding decode errors when newer characters are encountered in a stream being processed.
The case I am most interested in at the moment is streams that have 'magic' or 'BOM' which distinctly indicate their encoding. Surprisingly most of the libraries tested so far do not handle the known boms listed on Wikipedia. timrburnham/bom_open#13 (comment) I havent run this library against my bom tests yet to see how it compares with others.
I did look through the existing test suite here, and the code, and it looks quite good, and probably better for Asian languages than chardet
, but cchardet is also quite good at Asian languages.
from charset_normalizer.
Hi,
Is this modification safe ? Can we test it first ? Do we gain anything ? What do they update compare to Python embed ?
from charset_normalizer.
What do you think of this lib ? Are there any reason to use it compared to others (chardet and cchardet)?
from charset_normalizer.
self.assertEqual(
CnM.from_bytes(
'我没有埋怨,磋砣的只是一些时间。'.encode('gb18030')
).best().first().encoding,
'gb18030'
)
self.assertEqual(
CnM.from_bytes(
(u'\uFEFF' + '我没有埋怨,磋砣的只是一些时间。').encode('gb18030')
).best().first().encoding,
'gb18030'
)
self.assertEqual(
CnM.from_bytes(
'我没有埋怨,磋砣的只是一些时间。'.encode('utf_8')
).best().first().encoding,
'utf_8'
)
self.assertEqual(
CnM.from_bytes(
b'\xef\xbb\xbf' + '我没有埋怨,磋砣的只是一些时间。'.encode('utf_8')
).best().first().encoding,
'utf_8'
)
self.assertEqual(
CnM.from_bytes(
'Bсеки човек има право на образование. Oбразованието трябва да бъде безплатно, '
'поне що се отнася до началното и основното образование.'.encode('utf_8')
).best().first().encoding,
'utf_8'
)
self.assertEqual(
CnM.from_bytes(
'Bсеки човек има право на образование.'.encode(
'utf_8')
).best().first().encoding,
'utf_8'
)
charset_normalizer 0.2.2 comply with those tests 🎉
from charset_normalizer.
I'm going to fix utf_7 support in next minor. (was always considered as ascii)
from charset_normalizer.
@jayvdb did you have time to try the newest version ? 🎉
from charset_normalizer.
Set as optional feature. Op as of release 1.3.0
pip install charset-normalizer[UnicodeDataBackport]
from charset_normalizer.
Related Issues (20)
- [BUG] Unable to force-reinstall on Windows while in use - >=3.0.0 HOT 2
- IS this code reachable? HOT 1
- [DETECTION] no encoding found. HOT 1
- [DETECTION] EUC-KR files are not detected correctly. HOT 1
- [DETECTION] CP932 containing half-width kana characters cannot be detected correctly. HOT 1
- [BUG] Basic encoding failing and detecting as chinese, croatian and others when it is standard spanish HOT 2
- [DETECTION] Wrong encodings for GBK text HOT 1
- [BUG] Incorrect encoding detected in 3.3.1 HOT 2
- [BUG] Memory usage increase for big files HOT 1
- html file is not reported as UTF8 after conversion HOT 1
- [DETECTION] The results are not stable depending on the package version and do not work correctly HOT 4
- [Proposal] allow to disable language detection HOT 1
- Detection and Tiny sequences HOT 1
- [BUG] encoding is not detected when the input contains multiple encodings HOT 1
- TypeError: __eq__ cannot be invoked on <class 'NoneType'> and <class 'charset_normalizer.models.CharsetMatch'> HOT 1
- [BUG] Windows-1252 encoding is not detected in turkish text HOT 2
- [DETECTION] On the question of speed HOT 2
- [Proposal] Vendoring charset-normalizer in pip HOT 4
- [DETECTION] latin1 text misdetected as mac_latin2 HOT 7
- [DETECTION] fails on short input HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from charset_normalizer.