janlelis/unicode-confusable

Unicode::Confusable.confusable? "ℜսᖯʏ", "Ruby"

Unicode::Confusable

Compares two strings if they are visually confusable as described in Unicode® Technical Standard #39: Both strings get transformed into a skeleton format before comparing them. The skeleton is generated by normalizing the string (NFD), replacing confusable characters, and normalizing the string again.

Unicode version: 15.1.0 (September 2023)

* The Unicode normalization depends on your Ruby version

Supported Rubies: 3.2, 3.1, 3.0

Old Rubies which might still work: 2.7, 2.6, 2.5, 2.4, 2.3, 2.2

Usage

Confusable?

require "unicode/confusable"

Unicode::Confusable.confusable? "a", "b" # => false
Unicode::Confusable.confusable? "C", "С" # => true
Unicode::Confusable.confusable? "ℜ𝘂ᖯʏ", "Ruby" # => true
Unicode::Confusable.confusable? "Michael", "Michae1" # => true
Unicode::Confusable.confusable? "⁇", "?" # => false
Unicode::Confusable.confusable? "⁇", "??" # => true

Skeleton

Unicode::Confusable.skeleton "ℜ𝘂ᖯʏ" # => "Ruby"

Please note: The skeleton is an intermediate representation, not meant for any other use than testing confusability, according to the standard.

List

List all confusables of a specific character:

Unicode::Confusable.list("o", false)
# => ["ం", "ಂ", "ം", "ං", "०", "੦", "૦", "௦", "౦", "೦", "൦", "๐", "໐", "၀", "٥", "۵", "ｏ", "ℴ", "𝐨", "𝑜", "𝒐", "𝓸", "𝔬", "𝕠", "𝖔", "𝗈", "𝗼", "𝘰", "𝙤", "𝚘", "ᴏ", "ᴑ", "ꬽ", "ο", "𝛐", "𝜊", "𝝄", "𝝾", "𝞸", "σ", "𝛔", "𝜎", "𝝈", "𝞂", "𝞼", "ⲟ", "о", "ჿ", "օ", "ס", "ه", "𞸤", "𞹤", "𞺄", "ﻫ", "ﻬ", "ﻪ", "ﻩ", "ھ", "ﮬ", "ﮭ", "ﮫ", "ﮪ", "ہ", "ﮨ", "ﮩ", "ﮧ", "ﮦ", "ە", "ഠ", "ဝ", "𐓪", "𑣈", "𑣗", "𐐬"]

If you omit the second parameter, it will also show confusables, where the given character is just a part of:

Unicode::Confusable.list("o")
# => ["⒪", "ꜵ", "℅", "ᴔ", "ꭁ", "ꭂ", "ﷲ", "№", "ం", "ಂ", "ം", "ං", "०", "੦", "૦", "௦", "౦", "೦", "൦", "๐", "໐", "၀", "٥", "۵", "ｏ", "ℴ", "𝐨", "𝑜", "𝒐", "𝓸", "𝔬", "𝕠", "𝖔", "𝗈", "𝗼", "𝘰", "𝙤", "𝚘", "ᴏ", "ᴑ", "ꬽ", "ο", "𝛐", "𝜊", "𝝄", "𝝾", "𝞸", "σ", "𝛔", "𝜎", "𝝈", "𝞂", "𝞼", "ⲟ", "о", "ჿ", "օ", "ס", "ه", "𞸤", "𞹤", "𞺄", "ﻫ", "ﻬ", "ﻪ", "ﻩ", "ھ", "ﮬ", "ﮭ", "ﮫ", "ﮪ", "ہ", "ﮨ", "ﮩ", "ﮧ", "ﮦ", "ە", "ഠ", "ဝ", "𐓪", "𑣈", "𑣗", "𐐬", "ۿ", "ø", "ꬾ", "ɵ", "ꝋ", "ө", "ѳ", "ꮎ", "ꮻ", "ꭴ", "ﳙ", "ơ", "œ", "ɶ", "∞", "ꝏ", "ꚙ", "ﳗ", "ﱑ", "ﳘ", "ﱒ", "ﶓ", "ﶔ", "ﱓ", "ﱔ", "ൟ", "တ", "ꭣ", "ﲠ", "ﳢ", "ﲥ", "ﳤ", "ﷻ", "ﴱ", "ﳨ", "ﴲ", "ﳪ", "ﷺ", "ﷷ", "ﳍ", "ﳖ", "ﳯ", "ﳞ", "ﳱ", "ﳦ", "ﲛ", "ﳠ", "ﯭ", "ﯬ"]

No Advanced Detection

TR 39 also describes mechanisms for a more exact recognition of confusables, also within the same string:

• Single-script confusable
• Mixed-script confusable
• Whole-script confusable

This is currently not supported by this gem.

See unicode-x for more Unicode related micro libraries.

MIT License

• Copyright (C) 2016-2023 Jan Lelis https://janlelis.com. Released under the MIT license.
• Unicode data: https://www.unicode.org/copyright.html#Exhibit1

unicode-confusable's Issues

Canonical versions?

It would be cool to be able to use this to prevent similar looking usernames in a DB. What would it take to make that happen? I imagine one thing that would be helpful is to convert a string into some identifier or canonical function.

Unicode::Confusable.clarify("ℜ𝘂ᖯʏ") =>  "Ruby" 

# or

Unicode::Confusable.id_for("ℜ𝘂ᖯʏ") =>  "123abc" 
Unicode::Confusable.id_for("Ruby") =>  "123abc"