JSON Error Message: Malformed UTF-8 characters, possibly incorrectly encoded

[bquimby1]

JSON Error Message: Malformed UTF-8 characters, possibly incorrectly encoded

[youkidearitai]

° is convert to \u00b0 of JSON.
https://3v4l.org/iA3HU

Please check character encoding. Maybe ISO-8859-1 and other character encodings.

[bukka]

Please provide exact string that is failing. Note that it needs to be valid UTF-8 so if exif data is binary, then it might be obviously failing.

[bquimby1]

I can say with 100% certainty that the string is UTF-8.  I have thousands of photos updated in the exact same way with the exact same Perl code.  The EXIFTool app and various editors have no problem with this particular string as well. I provided the exact strings (one working and one that doesn’t) in my initial post.  I’ve since done a work around so any provided strings would not be a valid representation. Bruce QuimbySent from my iPadOn Apr 10, 2024, at 4:51 PM, Jakub Zelenka ***@***.***> wrote: Please provide exact string that is failing. Note that it needs to be valid UTF-8 so if exif data is binary, then it might be obviously failing. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: ***@***.***>

[bukka]

So provide a string that is failing otherwise we cannot check it out. Basically we need to be able to reproduce it somehow.

[bquimby1]

The exact string that is failing is in my original post - please get it from there. There are two strings there, one that is working and the one that is failing, both containing the ° character inserted with Option+Shift+8 on my Mac’s keyboard. I’ve run thousands of photos through json_encode() with no issues except this single one (so far). I couldn’t fix it with this string in this one photo so I worked around it. The original failing string no longer exists. Bruce Quimby ***@***.***

…

On Apr 11, 2024, at 8:12 AM, Jakub Zelenka ***@***.***> wrote: So provide a string that is failing otherwise we cannot check it out. Basically we need to be able to reproduce it somehow. — Reply to this email directly, view it on GitHub <#13935 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AYQMVTZIE4KMA4RBUNCZYM3Y4Z43XAVCNFSM6AAAAABGAS2FCOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANBZGU2TSNZTGA>. You are receiving this because you authored the thread.

[bukka]

The string that you provided is not failing:

var_dump(json_encode('+First Winter hike of the season. 20°F during the hike and dropping. This little bird posed nicely. There wasn\'t much wildlife because of the human traffic.'));
string(163) ""+First Winter hike of the season. 20\u00b0F during the hike and dropping. This little bird posed nicely. There wasn't much wildlife because of the human traffic.""

So please provide string that is actually failing.

[bukka]

As @youkidearitai pointed out, your encoding is most likely incorrect. I'm going to close it. If you provide example that is reliable failing, I will be happy to re-open this.

[bukka]

you can also verify it using bin2hex to compare both strings and see what the difference is. Then you should see how ° is encoded. UTF-8 should give you

echo bin2hex('°');
c2b0

[bquimby1]

Then I suppose we can go no further with this issue. I can tell you it was failing consistently (returning false) on my Mac. Close the ticket. I’ll open another one if I find another instance. Here’s the code that I put in place to capture the issue: public function makeObjectJS2024():bool { $photosjsonarray = []; $cnt=0; //$this->writeFetchLog("MyEXIF timeindex - " . count($this->timeindex)); foreach ( $this->timeindex as $timestamp => $filename ) { $o = $this->olist[$filename]; if ( $json = json_encode($o->exifobj) ) { $photosjsonarray[] = $json; } else { $this->writeFetchLog(sprintf("%s did not produce JSON at count %d - %s",$filename,$cnt,json_last_error_msg())); //continue; } $cnt++; } $this->photosjson = sprintf("[%s]",implode(",",$photosjsonarray)); //$this->writeFetchLog($this->photosjson); return true; } Bruce Quimby ***@***.***

…

On Apr 11, 2024, at 8:26 AM, Jakub Zelenka ***@***.***> wrote: The string that you provided is not failing: var_dump(json_encode('+First Winter hike of the season. 20°F during the hike and dropping. This little bird posed nicely. There wasn\'t much wildlife because of the human traffic.')); string(163) ""+First Winter hike of the season. 20\u00b0F during the hike and dropping. This little bird posed nicely. There wasn't much wildlife because of the human traffic."" So please provide string that is actually failing. — Reply to this email directly, view it on GitHub <#13935 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AYQMVT2MGLDP45QPPLLR66TY4Z6P7AVCNFSM6AAAAABGAS2FCOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANBZGU4DGMJZGI>. You are receiving this because you authored the thread.

[bukka]

If you don't mind potentially longer log entries, try to add there bin2hex($o->exifobj) - this should give info how that string is encoded

[bquimby1]

It could’ve been encode differently but I don’t see how as I only use Option+Shift+8 in vim. Thanks for the lead. If I find another instance I’ll definitely run it through bin2hex before I complain :-) Bruce Quimby ***@***.***

…

On Apr 11, 2024, at 8:35 AM, Jakub Zelenka ***@***.***> wrote: you can also verify it using bin2hex to compare both string and see what the difference is. The you should see how ° is encoded. UTF-8 should give you echo bin2hex('°'); c2b0 — Reply to this email directly, view it on GitHub <#13935 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AYQMVT7O3FM2EE3SH5MRU63Y4Z7RRAVCNFSM6AAAAABGAS2FCOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANBZGU4TQMRRHE>. You are receiving this because you authored the thread.

[bukka]

No probs. Feel free to just update this issues and if there's anything not right, I will re-open it.