BUG: index duplicates keys with non ascii chars

[aquirin]

BUG: index duplicates keys with non ascii chars

[dontgoto]

Interesting. This seems to happen due to the malformed unicode characters, somewhere in the MultIndex calls a search for duplicates (in e.g., ['x-\udce9', 'y-\udce9']) in the index happens via a cython hash map, this duplicate check does not happen for a single level index.

In the end this filters out a false positive duplicate, leading to the ['x-\udce9', 'x-\udce9'] index values in your example.

I don't think changing the hashmap implementation is a good idea, but one could at least throw an exception if the user wants to put malformed unicode into a MultiIndex.

[kvnwng11]

take