Fails to encode non-ASCII characters spam

[Emik03]

Fails to encode non-ASCII characters spam

[Emik03]

Update: This is definitely an issue with PrettyPrompt as I was able to use the example code and reproduce this exact behavior. The higher the key repeat speed, the easier it breaks.

[waf]

Narrowed this down to an issue in the core .NET libs. dotnet/runtime#88343

In the meantime, I think we can do a workaround to remove a call to Console.CursorTop which is ultimately causing the issue.

[Emik03]

Upon closer inspection, it seems that the problem arises because the program appears to push byte-by-byte. I was able to confirm this by making it print the ? character just from tapping ä and ' at the same time with some luck. A character that spans multiple bytes can have its first byte written, then a different character, and then the aforementioned character's remaining bytes. As a result, the resulting string is not valid UTF-8.

As an example, if I were to type a and ä just right, it probably appends the first ä byte 195, then the a byte 97, then the ä byte 164. The resulting buffer becomes [195, 97, 164] when it should have been [195, 164, 97].

[waf]

Hi @Emik03, thank you for the bug report and research. I'm not currently able to reproduce this. Would you mind sharing which terminal and OS you're using?

Also, does this still happen if start CSharpRepl with the --useUnicode true flag? I'm guessing it does still happen as that doesn't change how bytes are sent to the terminal, but just looking to narrow down differences between my setup and yours. Thanks!

[Emik03]

It indeed does happen regardless of the value set in useUnicode.
I can still replicate this after removing config.rsp and prompt-history, then launching it with no arguments.

OS: Pop!_OS 22.04 LTS x86_64
Shell: bash 5.1.16
DE: GNOME 42.5
Terminal: gnome-terminal
Layout: Swedish (no dead keys)

[Emik03]

I suspect it might be to do with how Windows uses UTF16 everywhere, which includes command-line interfaces, meanwhile Linux uses UTF8.

[waf]

Nice, thanks for that. I'm able to reproduce this on Windows Terminal with Ubuntu on WSL. Cannot reproduce at all on Windows Terminal on Windows.

[waf]

Note for myself:

• This appears to have broken between 0.5.1 and 0.6.0 of CSharpRepl (at least for my repro steps on WSL).
• Tracking this down to PrettyPrompt, it broke between v4.0.1 and v4.0.2 of PrettyPrompt.
• Verified that if I make CSharpRepl compile against PrettyPrompt v4.0.1 (requires a bit of hacking and slashing due to API differences) the issue goes away.

So the offending change is somewhere in here: v4.0.1...v4.0.2

[waf]

I've implemented this in the underlying PrettyPrompt library, when we upgrade that library CSharpRepl will get this fixed. Thanks for the report!