Python3 text encoding handled incorrectly for shortcuts.run

[release-engineering]

Python3 text encoding handled incorrectly for shortcuts.run

[rohanpm]

I'm not sure this is a bug in kobo.shortcuts.run. I see you pass universal_newlines=True. It's mentioned in https://docs.python.org/3/library/subprocess.html#popen-objects that passing universal_newlines=True will cause Popen stdout to be treated as a text stream (i.e. it causes decoding to happen), rather than a byte stream. I think that text stream would be using your default locale from the environment.

Is it possible that you're running a process which produces bytes not decodable according to your configured locale? e.g. your locale uses ASCII and your command is producing UTF-8? Can you share output of "locale" command where this is reproduced?

It seems that the correct ways to use kobo.shortcuts.run are:

• If you want text and you know the encoding used by the process you're running, then specify it in "encoding" argument.
• If you want text and you want to use the default encoding, you can omit "encoding" argument and provide universal_newlines=True.
• If you want bytes then provide neither argument.

In either of the text cases, if the process produces bytes which can't be decoded with the given encoding, then it'll crash, and I suspect that's what happened here.

[puiterwijk]

The issue we hit has been fixed on the Pungi side of things.