UnicodeDecodeError: when running with python 3

[bebef1987]

UnicodeDecodeError: when running with python 3

[tarekziade]

We're far away from vendoring tooltool (that's the plan)
In the meantime for this bug what I propose we do is:

• fix it here
• copy over the new tooltool.py version in all tooltool.py instances in mozilla-central
• bump the version number in that file, it was never changed I think

[bhearsum]

At this point, 3.6 is Old (3.8.1 is already out). I don't think it's worth fixing this unless it's reproducible on 3.7 or 3.8.

[tarekziade]

@mozbhearsum you should reopen this bug, it will happen on 3.7 and 3.8 as well.

The bug is that for some reason, the PY3 version is not saving the data using a binary mode. it converts for some reason the data received into text and then writes it into a text file.

But it makes the assumption that the data received can always be decoded in utf8, which is not the case all the time (and the response headers don't always give us a hint and we don't check it anyways)

We should keep the 'wb' mode for py3 too and simply write through the data received as bytes.
I got it working locally, happy to provide a PR is needed.

[tarekziade]

This is my quick fix to make it work locally :)

tarek@Tareks-MacBook-Air-3 Downloads % diff tooltool.py.original tooltool.py.mine
63c63
<     open_attrs = dict(mode='w', encoding='utf-8')
---
>     open_attrs = dict(mode='wb')  #, encoding='utf-8')
740,741c740,741
<                     if PY3:
<                         indata = to_text(indata)
---
>                     #if PY3:
>                     #    indata = to_text(indata)

[bhearsum]

Okay, let's reopen then :)

[bebef1987]

This is blocking a mozproxy enhancement: https://phabricator.services.mozilla.com/D62443

Here is a try example of the error:
https://treeherder.mozilla.org/#/jobs?repo=try&selectedJob=289676975&revision=73a706c3397b0ed213c6727f2a347d3e375d6608

[bhearsum]

I have a plausible fix for this in #27, although it can't be merged yet due to test failures.

If anyone is up for it, I'd appreciate any additional testing you can do with the new tooltool.py.

[bhearsum]

https://lando.services.mozilla.com/D63870/ to import the fix in tree.

[bebef1987]

@mozbhearsum looks like the import was backed out can you take a new look pls

[bhearsum]

Yeah, sorry about that. I've got that fixed here, and will be updating mozilla-central again after #28 lands.

In the meantime, feel free to import the current tooltool.py into try pushes if it helps you.

[bhearsum]

#28 is taking a bit longer than I thought. I opened https://phabricator.services.mozilla.com/D65139 to get the python 3 support landed in the meantime.

[bhearsum]

This landed on central and stuck this time. Python 3 should work now. Please re-open or open a new issue if it's still problematic.