`pyzet add` - UnicodeDecodeError: 'utf-8' codec can't decode byte 0xed; invalid continuation byte

[tpwo]

`pyzet add` - UnicodeDecodeError: 'utf-8' codec can't decode byte 0xed; invalid continuation byte

[tpwo]

The root cause of the problem is the fact that vim from git bash on Windows seems to not support emojis. When I try to paste 😉, <de09> appears instead, and Python raises the exception with invalid continuation byte when trying to save the file.

<de09> seems to be something that is called surrogate:

And surrogate is something more complicated:

UCS uses surrogates to address characters outside the initial Basic Multilingual Plane without resorting to more-than-16-bit byte representations.

But it seems that it's wrong, because if saved externally in vim (i.e. outside pyzet) it cannot print correctly in the terminal:

> cat .\surrogate-test
This is surrogate test:
���

What is more interesting, is the fact that git bash's vim actually can open a file with emoji -- the problem lies only in pasting from external source (because it can yank and paste line with emoji without issues):

Vim from WSL2 works fine when pasting emojis, and it shares exactly the same configuration file. A quick search told me that the issue might be in locale setting. IDK how it's done in git bash, but it's the hint for the further searching.

One of the workarounds is to actually use vim from WSL2 in pyzet. WSL2 adds command bash that is visible from PowerShell and I found a way to run Vim with the help of it:

# run Linux Substystem "vim" program like it's a normal Windows program
# something like an alias but really a shortcut to Windows Linux Subsystem vim application
function vim ($File){
    # XXX: need to allow variable length argument list so "vim" can be passed options
    $File = $File -replace '\\', '/'
    bash -c "vim -- '$File'"
}

But WSL2 is not as standard as git bash, so I don't like this solution that much.

I'm leaving this issue open for now. I'm not using emojis that much, and they will appear practically only when pasting text authored by someone else. For now, it'd need some input sanitation before trying to save the file.