Unicode character formatting on keyboard layout pages

[DJm00n]

Unicode character formatting on keyboard layout pages

[jowilco]

@DJm00n - Good catch. Fixed now.

Justy FYI: Internally, ligatures are stored in the same way as the non-BMP characters. The non-BMP characters have to be converted from surrogate pairs, while the ligatures are just combinations of characters.

[jowilco]

The KLC doesn't always correctly reflect what the keyboard is returning. The keyboard layout generator used to depend on KLC, but we found some issues with the .klc files that are generated, so we now use user32.dll directly (https://learn.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-tounicodeex, https://learn.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-getkeyboardlayoutlist, etc.)

Let me spend some more time with this when I have more time available.

Oh, thats sad.
It solved an issue at the time...

how to detect physical keyboard layout type
I don't believe that the keyboard driver knows anything about the keyboard itself.

[DJm00n]

@jowilco

[jowilco]

@DJm00n I've fixed the title case -> upper case issue. Reformatting the ligature descriptions with "WITH" will take a little longer.

[jowilco]

@DJm00n After thinking about it a little more, I'm not sure that <U+XXXX, U+YYYY> XXXXXXXXXXX [WITH/AND] YYYYYYYYY is the correct approach here. While some ligatures are "letter + combining character" (not accurate, but I think that you'll understand what I am getting at), other ligatures are simply combinations or groups of characters. I went with:
<U+XXXX, U+YYYY> XXXXXXXXXXX, YYYYYYYYY

[DJm00n]

@jowilco I understand you objection concerning [WITH/AND] but I think current U+09B0, U+09CD BENGALI LETTER RA, BENGALI SIGN VIRAMA format that you're submitted is not readable.

Possible alternative would be <U+09B0 BENGALI LETTER RA, U+09CD BENGALI SIGN VIRAMA>

<> is mandatory here since it means sequences of code points (UCS Sequence Identifier or USI).

[DJm00n]

Also, it would be great if you can add VK_* codes on the button popups. It is mandatory for a user to see which code is emitted on a key press too (think about US QWERTY vs French AZERTY vs new French BÉPO) - and this value is not clear for non-US keyboards at all.

Look at VK_* keys here:

https://kbdlayout.info/kbdus/virtualkeys

https://kbdlayout.info/kbdfrna/virtualkeys

https://kbdlayout.info/kbdfrnb/virtualkeys

[DJm00n]

Also, it would be great if you can add character and VK codes for this scan codes in all shift states:

• 0x0e (usually mapped to VK_BACK with U+0008 BACKSPACE)
• 0x0f (usually mapped to VK_TAB with U+0009 CHARACTER TABULATION)
• 0x39 (usually mapped to VK_SPACE with U+0020 SPACE)
• 0x1c (usually mapped to VK_RETURN with U+000D CARRIAGE RETURN)

Since these can be remapped in keyboard layout too.

[jowilco]

@DJm00n - Please try now. I hope that is closer.

[DJm00n]

@jowilco now it looks much better! thank you!

one thing I noted that <> is rendered for non-BMP Unicode chars. See https://learn.microsoft.com/en-us/globalization/keyboards/kbdgthc for example.

[DJm00n]

Thank you again, @jowilco!

[DJm00n]

Hi @jowilco!
Minor nit. There SHOULD BE <> in this case with surrogate + ligature:

[jowilco]

Thanks @DJm00n - good catch!
Fortunately, this was the only instance.
#267

[DJm00n]

@jowilco thank you for a quick fix!

[DJm00n]

Hi @jowilco!

Dead Key characters are missing Unicode character names:

Minor: also, there should be space after : char (not before) in DeadKey :^ â Â ê Ê î Î ô Ô û Û string.

[jowilco]

Hi @DJm00n - Good to hear from you!

Good suggestions. Implemented in #281

[DJm00n]

Thank you, @jowilco!

Here is name of character is still missing, it should be ^ U+005E CIRCUMFLEX ACCENT:

[DJm00n]

Also, it seems some keyboards can output chars/deadchars from the Private Use Area (range: E000—F8FF).
But current implementation just omits them.

For example ADLaM layout have such keys in Ctrl+Alt (AltGr) and Shift+Ctrl+Alt (Shift+AltGr) modes and on GRAVE ACCENT deadkey:

https://learn.microsoft.com/en-us/globalization/keyboards/kbdadlm
https://kbdlayout.info/kbdadlm

Maybe you can output them too? Something like U+EF1E Private Use Area?

https://kbdlayout.info/features/unicode

[jowilco]

@DJm00n - Thanks again for finding the issues. I really do appreciate that you are willing to help us make the information that we are publishing better!

I added the character names to the Deadkey base characters.
The issue with Osage and ADLaM is that they were set as keyboards that didn't support AltGr, which of course they do. Once they were set correctly, the PUA characters appeared correctly.

[DJm00n]

The issue with Osage and ADLaM is that they were set as keyboards that didn't support AltGr, which of course they do

How do you set that? It can be detected by KLLF_ALTGR flag in low word of KBDTABLES.fLocaleFlags if I got it right.

Also, dead key list is still empty here. There should be a bunch of Private Use Area chars:

[jowilco]

That would be a better way of doing this. Unfortunately, all of the changes that I have been making are to a tool that I inherited that is solely for generating the keyboards for this site. It relies on a static database table with records for each keyboard. I really should remove that dependency and instead generate the data when I generate the keyboards, but that will take time/resources.

For the ADLaM keyboard, it isn't a dead key. If you use the AltGr key, you'll see the PUA chars on VK_OEM_3 and several others.

[DJm00n]

VK_OEM3 without shift states is a deadkey according to https://kbdlayout.info/kbdadlm/download/klc

DEADKEY	0060	// GRAVE ACCENT

0042	ef1e	// B -> 
004a	ef1f	// J -> 
004b	ef1d	// K -> 
0050	ef20	// P -> 
0053	ef21	// S -> 
0056	ef1c	// V -> 
0062	ef40	// b -> 
006a	ef41	// j -> 
006b	ef3f	// k -> 
0070	ef42	// p -> 
0073	ef43	// s -> 
0076	ef3e	// v -> 
0060	0060	// ` -> `

[DJm00n]

It relies on a static database table with records for each keyboard

Oh, thats sad. I had a related question - how to detect physical keyboard layout type from KBDTABLES?....and it seems now answer is "there is no way". Right?

[jowilco]

Hi @DJm00n - I had some time to look at the ADLaM issue again. I installed the keyboard on Windows 11 and tried using the ` deadkey. It appears that it can be used with any character, not just the set mentioned at the bottom of the page on https://kbdlayout.info/kbdadlm/shiftstates. Unfortunately, there is very little information on the Internet about using the ADLaM keyboard, at least in English, so I'm not sure if this is an issue with the keyboard driver or if this is by design.

I don't think that it is worth listing every single character in the ` tooltip, but I am open to suggestions.

If nothing else, I plan on updating the PUA keys (AltGr) to display the correct glyphs. I was also thinking about adding the base character to each of the deadkey combinations in the tooltip.

Happy holidays!