is_utf8 helper

[krlmlr]

is_utf8 helper

[krlmlr]

This will need C code to distinguish between "unknown" encoding and truly "all ASCII", basically an enhanced do_encoding() that also checks IS_ASCII(). Eventually this could be added upstream.

[hadley]

What's the difference between unknown and ascii? I thought unknown implied ascii.

[krlmlr]

Need to double-check.

[krlmlr]

There' a dedicated ASCII bit that is set if the code of all characters is 127 or less.

[hadley]

But that never gets used in Encoding(): https://github.com/wch/r-source/blob/2f2e4711ad7089f97f22c6b1ae25ba582d2e99a6/src/main/util.c#L1110

[hadley]

Also, none of the internal string representation stuff is in the exported API, so I think that means doing checks in R with Encoding() will be the easiest way forward.

[krlmlr]

ASCII implies unknown, but not the other way round. It will be difficult to detect pure ASCII strings using Encoding() only.

[hadley]

Yes, but it'll be accurate >95% of the time, I'd imagine

[github-actions]

This old thread has been automatically locked. If you think you have found something related to this, please open a new issue and link to this old issue if necessary.