[BUG] `UnicodeDecodeError: 'ascii' codec can't decode` when using `detect`

[aytey]

[BUG] `UnicodeDecodeError: 'ascii' codec can't decode` when using `detect`

[aytey]

Could be related to file size again:

(venv) atg@vapvdatg01:/tmp> size=10057858 && tail -c $size original.in > v2 && python test.py v2
MacCyrillic
(venv) atg@vapvdatg01:/tmp> size=10057859 && tail -c $size original.in > v2 && python test.py v2
Traceback (most recent call last):
  File "test.py", line 6, in <module>
    encoding = detect(open(fname, "rb").read())["encoding"]
  File "/tmp/venv/lib64/python3.8/site-packages/charset_normalizer/legacy.py", line 28, in detect
    r = from_bytes(byte_str).best()
  File "/tmp/venv/lib64/python3.8/site-packages/charset_normalizer/api.py", line 452, in from_bytes
    and fallback_u8.fingerprint != fallback_ascii.fingerprint
  File "/tmp/venv/lib64/python3.8/site-packages/charset_normalizer/models.py", line 274, in fingerprint
    return sha256(self.output()).hexdigest()
  File "/tmp/venv/lib64/python3.8/site-packages/charset_normalizer/models.py", line 265, in output
    self._output_payload = str(self).encode(encoding, "replace")
  File "/tmp/venv/lib64/python3.8/site-packages/charset_normalizer/models.py", line 114, in __str__
    self._string = str(self._payload, self._encoding, "strict")
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe9 in position 2711776: ordinal not in range(128)

Where:

(venv) atg@vapvdatg01:/tmp> diff <(tail -c 10057858 original.in)  <(tail -c 10057859 original.in)
1c1
< __true_type __type;
---
>  __true_type __type;

(so we gained a single ASCII space)

[Ousret]

Feel free to give feedback on that matter.

[aytey]

Feel free to give feedback on that matter.

Tested! Works great! Thanks, @Ousret!

[JensTimmerman]

I had the same issue, this patch also fixes it for me.