thombashi/mbstrdecoder

Python library for multi-byte character string decoder.

mbstrdecoder

Summary

mbstrdecoder is a Python library for multi-byte character string decoder.

Installation

Install from PyPI

pip install mbstrdecoder

Install from PPA (for Ubuntu)

sudo add-apt-repository ppa:thombashi/ppa
sudo apt update
sudo apt install python3-mbstrdecoder

Usage

Sample Code

from mbstrdecoder import MultiByteStrDecoder

encoded_multibyte_text = "マルチバイト文字".encode("utf-8")
decoder = MultiByteStrDecoder(encoded_multibyte_text)

print("encoded bytes: {}".format(encoded_multibyte_text))
print("unicode: {}".format(decoder.unicode_str))
print("codec: {}".format(decoder.codec))

Output

encoded bytes: b'\xe3\x83\x9e\xe3\x83\xab\xe3\x83\x81\xe3\x83\x90\xe3\x82\xa4\xe3\x83\x88\xe6\x96\x87\xe5\xad\x97'
unicode: マルチバイト文字
codec: utf_8

Dependencies

• Python 3.7+
• Python package dependencies (automatically installed)

mbstrdecoder's Issues

Typing module is not included in requirements

_func.py imports from typing module, but it's not included in requirements and thus not installed with mbstrdecoder.

GB18030

GB18030 encoded text is being detected as utf_16, big5 and cp037, and only big5 can decode it.

Detection as utf_16 is very wrong as that codec must contain the utf16 BOM, so the library should be very cautious about that result, but chardet has a patch to do exactly that chardet/chardet#109

The GB18030 BOM tends to result in detection as cp037 The BOM is regularly causing problems in chardet-like-libraries. c.f. chardet/chardet#178

Curly quotes

The example at PyYoshi/cChardet#26 also has a problem here.

The detected codec of Bob’s Burgers is big5hkscs, instead of windows-1252.

What is this library for?

Doesn't python support multibyte codepages out of the box?