Python 3 UnicodeEncodeError when passing unicode options

[pallets]

Python 3 UnicodeEncodeError when passing unicode options

[zart]

$ python3.4 -c "import sys;print(sys.argv)" "Sabão em pó Omo 1kg"
['-c', 'Sabão em pó Omo 1kg']
$ LC_ALL=C python3.4 -c "import sys;print(sys.argv)" "Sabão em pó Omo 1kg"
['-c', 'Sab\udcc3\udca3o em p\udcc3\udcb3 Omo 1kg']

This happens in cron/upstart/init because they run tasks with restricted environment and LANG* and LC_* variables are usually missing. See http://upstart.ubuntu.com/cookbook/#job-environment for example or crontab(5) manpage.

So either put LANG=en_US.UTF-8 in your crontab or run your script there with /usr/bin/env LANG=en_US.UTF-8 script.py arguments.... Ditto for upstart jobs where it's spelt like env LANG=en_US.UTF-8.

Of course you can adjust that to your preferred locale.

sudo and friends also have special rules about environment handling.

This whole mess is due to inability of POSIX to deal with encodings at filesystem level.
Working this around at python level is too late by the time the script is running.

[mitsuhiko]

That sounds like a bug in Python 3 or a misconfiguration somewhere. Can you check if the same happens on Python 2?

[fiatjaf]

Just checked. On Python 2 it runs perfectly.

[mitsuhiko]

Can you paste the code you are using?

[mengzhuo]

I can confirm it's not a Py3 problem while example worked on my CentOS 6.3 & Python 3.3.5.

$ python example.py --name="你好，世界" --count=3

Hello 你好，世界!
Hello 你好，世界!
Hello 你好，世界!

My ENV

COLORTERM=gnome-terminal
LANG=zh_CN.UTF-8
LC_CTYPE=zh_CN.UTF-8
OLDPWD=/home/meng/Python-3.3.5
ORBIT_SOCKETDIR=/tmp/orbit-meng
PAGER=less
PATH=/home/meng/.virtualenvs/py3/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/meng/bin:/home/meng/bin
PWD=/home/meng
QTDIR=/usr/lib64/qt-3.3
QTINC=/usr/lib64/qt-3.3/include
QTLIB=/usr/lib64/qt-3.3/lib
QT_IM_MODULE=ibus
SESSION_MANAGER=local/unix:@/tmp/.ICE-unix/1914,unix/unix:/tmp/.ICE-unix/1914
SHELL=/bin/zsh
SHLVL=3
SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass
SSH_AUTH_SOCK=/tmp/keyring-75qgcm/socket.ssh
TERM=xterm-256color
TMUX=/tmp/tmux-500/default,2561,0
TMUX_PANE=%0
USER=meng
USERNAME=meng
VIRTUALENVWRAPPER_HOOK_DIR=/home/meng/.virtualenvs
VIRTUALENVWRAPPER_LOG_DIR=/home/meng/.virtualenvs
VIRTUALENVWRAPPER_PROJECT_FILENAME=.project
VIRTUAL_ENV_DISABLE_PROMPT=1
WINDOWID=33554436
WORKON_HOME=/home/meng/.virtualenvs
XMODIFIERS=@im=ibus
ZSH=/home/meng/.oh-my-zsh
_=/usr/bin/env
VIRTUAL_ENV=/home/meng/.virtualenvs/py3

---

helper_raw_input.py code:

#!/usr/bin/env python
# encoding: utf-8

from click.helpers import raw_input

b = raw_input("Test: ")
print(b.lower().strip())

Which works fine with UTF8 terminal settings.

[mitsuhiko]

Python 3 detects the encoding from the environment variables but there are situations where it will assume the system is ASCII even though it's not. This for instance happens for scripts launched from upstart/systemd or cron. Not sure yet how to deal with this.

Currently the plan is to manually force the encoding to utf-8.

[mitsuhiko]

I will not fix this. This is in my eyes a problem with Python 3 and needs to be fixed there. In fact I cannot find a way to reliably recover from this in all situations which is a clear regression over Python 2.

Click will now immediately die with an error on Python 3 if the environment is misconfigured.

See 9ab74c3.

[fiatjaf]

That's pretty good, I think. Thanks for the help.

I couldn't provided more information because I lost myself in a jungle of Python versions and only reached the civilization again now, with a new Python 3.2 running the same code of before, now misteriously working fine.