Comments (4)
Could be related to file size again:
(venv) atg@vapvdatg01:/tmp> size=10057858 && tail -c $size original.in > v2 && python test.py v2
MacCyrillic
(venv) atg@vapvdatg01:/tmp> size=10057859 && tail -c $size original.in > v2 && python test.py v2
Traceback (most recent call last):
File "test.py", line 6, in <module>
encoding = detect(open(fname, "rb").read())["encoding"]
File "/tmp/venv/lib64/python3.8/site-packages/charset_normalizer/legacy.py", line 28, in detect
r = from_bytes(byte_str).best()
File "/tmp/venv/lib64/python3.8/site-packages/charset_normalizer/api.py", line 452, in from_bytes
and fallback_u8.fingerprint != fallback_ascii.fingerprint
File "/tmp/venv/lib64/python3.8/site-packages/charset_normalizer/models.py", line 274, in fingerprint
return sha256(self.output()).hexdigest()
File "/tmp/venv/lib64/python3.8/site-packages/charset_normalizer/models.py", line 265, in output
self._output_payload = str(self).encode(encoding, "replace")
File "/tmp/venv/lib64/python3.8/site-packages/charset_normalizer/models.py", line 114, in __str__
self._string = str(self._payload, self._encoding, "strict")
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe9 in position 2711776: ordinal not in range(128)
Where:
(venv) atg@vapvdatg01:/tmp> diff <(tail -c 10057858 original.in) <(tail -c 10057859 original.in)
1c1
< __true_type __type;
---
> __true_type __type;
(so we gained a single ASCII space)
from charset_normalizer.
Feel free to give feedback on that matter.
from charset_normalizer.
Feel free to give feedback on that matter.
Tested! Works great! Thanks, @Ousret!
from charset_normalizer.
I had the same issue, this patch also fixes it for me.
from charset_normalizer.
Related Issues (20)
- [DETECTION] no encoding found. HOT 1
- [DETECTION] EUC-KR files are not detected correctly. HOT 1
- [DETECTION] CP932 containing half-width kana characters cannot be detected correctly. HOT 1
- [BUG] Basic encoding failing and detecting as chinese, croatian and others when it is standard spanish HOT 2
- [DETECTION] Wrong encodings for GBK text HOT 1
- [BUG] Incorrect encoding detected in 3.3.1 HOT 2
- [BUG] Memory usage increase for big files HOT 1
- html file is not reported as UTF8 after conversion HOT 1
- [DETECTION] The results are not stable depending on the package version and do not work correctly HOT 4
- [Proposal] allow to disable language detection HOT 1
- Detection and Tiny sequences HOT 1
- [BUG] encoding is not detected when the input contains multiple encodings HOT 1
- TypeError: __eq__ cannot be invoked on <class 'NoneType'> and <class 'charset_normalizer.models.CharsetMatch'> HOT 1
- [BUG] Windows-1252 encoding is not detected in turkish text HOT 2
- [DETECTION] On the question of speed HOT 2
- [Proposal] Vendoring charset-normalizer in pip HOT 4
- [DETECTION] latin1 text misdetected as mac_latin2 HOT 7
- [DETECTION] fails on short input HOT 1
- [BUG]
- [BUG] identifies UTF16LE for a pair of ascii punctuation characters
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from charset_normalizer.