bb33bb/text-encoding-detect

C# and C++ UTF8/UFT16 encoding detection library.

text-encoding-detect

This is a C++ and C# library for detecting UTF8 and UTF16 text encoding. I recently had to upgrade the text file handling feature of AutoIt to better handle text files where no byte order mark (BOM) was present. The older version of code I was using worked fine for UTF-8 files (with or without BOM) but it wasn't able to detect UTF-16 files without a BOM. I tried to use the IsTextUnicode Win32 API function but this seemed extremely unreliable and wouldn't detect UTF-16 Big-Endian text in my tests.

Please see the project page for details on how the library works and how to use it: https://www.autoitconsulting.com/site/development/utf-8-utf-16-text-encoding-detection-library/