marekru/unidecode

C++ port of the Text::Unidecode Perl module, which provides pure ASCII transliterations of Unicode strings.

unidecode's Issues

Unidecode for `std::string`

Create version of the Unidecode function which takes std::string as an argument.

Usage:

std::string decoded = Unidecode( u8"mačka" )

Example implementation:

auto Unidecode(... utf8_string)
{
    std::string result;
    unidecode::Utf8StringIterator begin{utf8_string.c_str()};
    unidecode::Utf8StringIterator end{utf8_string.c_str() + utf8_string.length()};
    unidecode::Unidecode(begin, end, std::back_inserter(result));
    return result;        
}

Iterator typetraits

Specify that the template type has to be iterator using typetraits.

[?] values in the table

Invesigate why aren't some characters defined in the table.

Wrong values in the table

section: 1, position: 51, '' != 'ij' 
section: 1, position: 242, 'D' != 'Dz' 
section: 2, position: 32, '[?]' != 'N' 
section: 2, position: 33, '[?]' != 'd' 
section: 2, position: 52, '[?]' != 'l' 
section: 2, position: 53, '[?]' != 'n' 
section: 2, position: 54, '[?]' != 't' 
section: 2, position: 55, '[?]' != 'j' 
section: 2, position: 56, '[?]' != 'db' 
section: 2, position: 57, '[?]' != 'qp' 
section: 2, position: 58, '[?]' != 'A' 
section: 2, position: 59, '[?]' != 'C' 
section: 2, position: 60, '[?]' != 'c' 
section: 2, position: 61, '[?]' != 'L' 
section: 2, position: 62, '[?]' != 'T' 
section: 2, position: 63, '[?]' != 's' 
section: 2, position: 64, '[?]' != 'z' 
section: 2, position: 67, '[?]' != 'B' 
section: 2, position: 68, '[?]' != 'U' 
section: 2, position: 69, '[?]' != '^' 
section: 2, position: 70, '[?]' != 'E' 
section: 2, position: 71, '[?]' != 'e' 
section: 2, position: 72, '[?]' != 'J' 
section: 2, position: 73, '[?]' != 'j' 
section: 2, position: 74, '[?]' != 'q' 
section: 2, position: 75, '[?]' != 'q' 
section: 2, position: 76, '[?]' != 'R' 
section: 2, position: 77, '[?]' != 'r' 
section: 2, position: 78, '[?]' != 'Y' 
section: 2, position: 79, '[?]' != 'y' 
section: 2, position: 121, 'R' != 'r' 
section: 2, position: 122, 'R' != 'r' 
section: 2, position: 123, 'R' != 'r' 
section: 2, position: 124, 'R' != 'r' 
section: 2, position: 127, 'R' != 'r' 
section: 2, position: 137, 'U' != 'u' 
section: 2, position: 141, 'W' != 'w' 
section: 2, position: 142, 'Y' != 'y' 
section: 2, position: 174, '[?]' != 'h' 
section: 2, position: 175, '[?]' != 'h' 
section: 2, position: 176, 'k' != 'h' 

Benchmark GetCodePoint method

unidecode/include/unidecode/common.hpp

Line 37 in 38f28e9

inline unicode_char GetCodePoint(const char* ptr) {

Look at this code and try to benchmark this using google benchmark!
There might be an improvement using fixed size of masks 4.

example code:

    static const uint8_t kMask0[] = {0xff, 0x00, 0x00, 0x00};
    static const uint8_t kMask1[] = {0x3f, 0x1f, 0x00, 0x00};
    static const uint8_t kMask2[] = {0x3f, 0x3f, 0x0f, 0x00};
    static const uint8_t kMask3[] = {0x3f, 0x3f, 0x3f, 0x07};

//...

   inline unicode_char GetCodePoint(const char* ptr) {
        uint32_t code_point_size = GetCodePointSize(*ptr);
        const uint8_t* mask = kCodePointMasks[code_point_size - 1];
        unicode_char result = 0;
        for (uint32_t i = 4; i-->0; ptr++) {
            result |= (*ptr & mask[i]) << 6 * i;
        }
        return result;
    }