[BUG] Detects ISO-8859-1 incorrectly as ANSI; then, empties document when converting to UTF-8

[darthstark1138]

[BUG] Detects ISO-8859-1 incorrectly as ANSI; then, empties document when converting to UTF-8

[srutzky]

Hello @darthstark1138 . I'm the author of SQL# and am trying to reproduce this issue. So far, I cannot reproduce the behavior you are describing; it works as expected. I'm using Notepad++ v8.6.8 (64-bit) (same as you) on Windows 11 Home (64-bit), OS Build : 22631.3737 (very similar to you environment). I'm loading SQLsharp_SETUP.sql, and the script is detected as being ANSI (same as you are seeing). However, when I go to Encoding | Convert to UTF-8, the entire script remains.

One difference between our environments is that your "Current ANSI codepage" is 65001 (i.e. UTF-8) while mine is 1252.

What SQL tool are you using that was complaining about "non-Unicode characters"? Can you please provide the exact error message? I find the error (as currently stated) a bit odd since Unicode supports most characters across most (if not all) code pages. However, it is possible that your SQL tool is complaining about an invalid UTF-8 byte sequence as there's a single character in the file that is not standard ASCII (i.e. decimal values 0 - 127): A9 copyright sign ( "©" in ISO-8859-1, Windows-1252, and others). The UTF-8 encoding of "©" is: C2 A9. Since the encoding of the file is ISO-8859-1 or Windows-1252 (effectively the same thing, with only minor variation), it does not have the C2 byte preceding the A9 byte.

I'm guessing that the issue you are having is related to the combination of 1) your current ANSI code page being set to 65001 / UTF-8, and 2) the invalid UTF-8 byte-sequence of A9 without the preceding C2 byte.

Regardless of if / when this gets fixed in Notepad++ (might be a low-priority edge-case as I'm not sure how many Windows systems have their ANSI code page set to 65001 / UTF-8 as opposed to 1252, 437, or some other 12xx series Windows code page), I will update the SQL# install script to convert the "©" symbol to instead be "(c)", which will make the script be entirely standard ASCII (which should avoid such problems in the first place).

[srutzky]

@darthstark1138 Also, this might not matter, but what type of install do you have: Installer, Portable, or Mini-Portable? I'm using the Installer.

[darthstark1138]

Hi @srutzky, thank you for going to the trouble of testing this!
Indeed, I checked "Use Unicode UTF-8 for worldwide language support" in Windows.

I'll uncheck it and test it again as soon as I can restart it to see if the problem still happens.

[srutzky]

@darthstark1138 Thanks for the info. Please don't forget to let me / us know the following:

• What SQL tool were you using that complained about "non-Unicode characters"
• What was the exact and full error message regarding "non-Unicode characters"
• What type of Notepad++ install do you have: Installer, Portable, or Mini-Portable

Thanks...

P.S. If you did not see the notice just above the previous comment of mine, others have experienced similar behavior in #15324 .

[SupportRainbow]

Hey Everyone,

You tagged my post #15324 and I just wanted to say, currently I haven't found a workaround or fix for version 8.6.8 when trying to convert through the "Encoding" tab, if it's a timely manner and you need to convert correctly like I did, I uninstalled 8.6.8 and installed the previous version 8.6.7 and it works there, currently that's the only "workaround" I know to get it working if you need it timely!

I've been trying to do some digging myself to see if I can find some sort of fix or work around for version 8.6.8 so if I do I'll update this post as well with the fix but, it seems pretty "buggy" and I'm hoping that we get enough traction to have the Notepad ++ devs see this and hopefully create a hotfix soon on their next update :)!

[srutzky]

I don't know what change, but I'm now able to reproduce this issue. I don't think I changed anything. AND, I've been able to reproduce the issue on any ANSI file, including the SQLsharp_SETUP.sql file and even the C:\Program Files\Notepad++\change.log file. This is definitely not specific to the SQL# setup script, and as a result of that, the issue is also not due to the invalid UTF-8 byte sequence of the A9 byte by itself, nor due to "current ANSI codepage" (as this seems to be happening across several of those).

I'm now thinking this has something to do with some internal state of Notepad++ given that I was for a while able to reproduce this behavior with the C:\Program Files\Notepad++\change.log file (in fact, I was not able to not get this behavior), but I did something and now I can no longer reproduce the issue, on either the SQLsharp_SETUP.sql file or the C:\Program Files\Notepad++\change.log file. I have not closed / re-opened Notepad++, but I do have many tabs open and made an edit in one of them (which I have tried before in this testing and it hasn't seemed to have had any effect), and maybe changed to various tabs. Hmmm, I think I will need to check what encodings I am using in those other tabs.

[srutzky]

@darthstark1138 looks like the problem was figured out in the other issue. Seems that if the clipboard history is empty the conversion problem will happen, but with something in the clipboard the conversion works.

So I guess there's no need to answer the question about what type of install, but I'm still curious as to what SQL tool you have been using, if you don't mind answering. I just want to be able to test it for myself, if possible. Thanks...

[darthstark1138]

Hi,

The tool was SSMS 20. Sorry, I forgot to answer before.

[srutzky]

@darthstark1138 Thanks again for the info. And, I had actually downloaded SSMS 20 a while ago but never installed it. Now I have the push that I needed to actually do the install 😉 . And, I've done the install and do not get that error message. I suspect the issue is (or was) due to you enabling the "Use UTF-8 for worldwide language support" option, which set the ANSI code page to 65001 (i.e. UTF-8), and the script was assumed to be ANSI by SSMS, which in your case was UTF-8, yet it had an invalid UTF-8 byte sequence (i.e. the A9 byte by itself) since it was not truly a UTF-8 file.

You mentioned above that you were planning on unchecking that option, restarting, and testing Notepad++ again. While it's not necessary at this point to test Notepad++ (since the cause has been found and it's not the encoding), if you have already disabled this option, or if it would be easy enough to do, do you mind trying again to open the SQLsharp_SETUP.sql file (unaltered; directly from the zip file) in SSMS 20 to see if you get that error? And I suppose I can try to enable that option on my system to see if that triggers the error.

Thanks and take care!!

[srutzky]

@darthstark1138 Yep, that was it. I enabled the UTF-8 support on my system, rebooted, opened SSMS 20, open the SQL# setup file, and got the Unicode error. In which case, my plan to replace the "©" symbol with "(c)" will solve that issue (moral of the story: that's what I get for trying to be clever 😸 ).