strings generated by the browser’s JSON.stringify are not parseable by Boost JSON

[vaishnavkatiyar]

strings generated by the browser’s JSON.stringify are not parseable by Boost JSON

[grisumbras]

So, first of all "\uD83D\uD83D" is not a valid UTF-8 string. It does not represent any sequence of unicode characters. Whatever gives you strings like that does not give you UTF-8 encoded strings. My intuition is that you need to escape the slashes prior to JSON.stringify.

Second, parse_options::allow_invalid_utf8 is intended to disable checking if UTF-8 code units in the text are forming a valid code point. That is, it affects how unescaped UTF-8 is treated.

Now, there's an argument that we need to provide users with a way to allow invalid surrogate sequences in the input. Probably with the same option. The question then becomes, how to deal with such sequences, as they do not represent UTF-8 characters? If we leave them as-is, we ignore the JSON spec, because they are supposedly escaped characters. But we cannot treat them as surrogate pairs, because they do not represent any unicode characters.

[pdimov]

If invalid UTF-16 is allowed, unpaired surrogates would probably need to be UTF-8 encoded, and the result will be WTF-8.

[grisumbras]

Oh, so there is a somewhat popular method of handling it? Would you recommend using a different option for this, or the same allow_invalid_utf8 ?

[pdimov]

I think that this deserves a different option, allow_invalid_utf16. The input isn't invalid UTF-8, so it's not the same thing.

[vaishnavkatiyar]

Is introducing the new option allow_invalid_utf16 under consideration for upcoming release?

[grisumbras]

Currently we're not committed to having this option at all.