XoopsFormDhtmlTextArea preview with UTF-8

[xoops]

XoopsFormDhtmlTextArea preview with UTF-8

[GregMage]

I think it's OK now

[geekwright]

I took a closer look, and cleared off some cob webs in a dark corner in my brain. 😄

I think the new utf8_encode() on the PHP side is just masking the real issue. The problem seems to start from the escape() call in formdhtmltextarea.js

escape("é") results in "%E9" which is just the raw code point % encoded.
escape("跠") results in "%u8DE0" which is an encoding of the raw unicode code point

There is no single function to reverse this in PHP. Changing that "%uXXXX" into something PHP can use is the what that preg_match_all code is about.

A further consideration is that escape() is no longer part of any javascript standard. Implementations still include it, but it is considered obsolete.

In contrast, let's look at what happens with encodeURIComponent()

encodeURIComponent("é") results in "%C3%A9" which is the correct UTF-8 encoding for the code point, percent encoded for transmission.

encodeURIComponent("跠") results in "%E8%B7%A0" which is again correct UTF-8 ready for transmission.

Since encodeURIComponent (and its reverse decodeURIComponent) are based on the rfc3986 standard, we can reverse both of these in compatible PHP functions, rawurldecode and rawurlencode.

If we use encodeURIComponent() in formdhtmltextarea.js, and then decode it with rawurldecode() in formdhtmltextarea_preview.php, I think everything should pass through end to end, no matter what the page encoding is set to. With what we have now, the high ascii range (0x80-0xFF) gets handled out of bounds and differently on both ends.

Also, looking closer, we probably should decode everything before we process the xoopscode with the text sanitizer?

[geekwright]

I just want to add a short note on what brought all of this to my attention in the first place.

The protector filter I was testing is checking language characteristics. (The idea is that if you are a new user posting in korean on a french site, you are probably a spammer.) The funny thing was, the ajax preview sailed right through, even though anything that submitted the actual form got rejected. The custom hybrid encoding didn't trigger any warnings in the preview, even though it showed up as korean text.

[GregMage]

You're right, my code was not good. I did like as you said and it works.

Also, looking closer, we probably should decode everything before we process the xoopscode with the text sanitizer?

Apparently, this is not necessary. I think now it's ok! #214

This is a very good idea to test language

[geekwright]

There are a lot of subtle interactions in this issue - thanks for being patient!

[geekwright]

Works well!