yapf does not explicitly handle different encodings

[google]

yapf does not explicitly handle different encodings

[gotgenes]

yapf needs to open source files using codecs.open, and accept an optional explicit file encoding parameter on the command line interface, as well as for any functions exposed in the API that create a file handle. See #19 for an example usage.

[DRMacIver]

I learned about tokenize.open recently. It handles encoding comments in python source files to infer the correct file encoding.

I think it's python 3 only but the source code might be open source in a way you could use.

[gotgenes]

Guessing encoding based on the header comment in the source file would be convenient. Keep in mind, though, that it's not always accurate. See Fact of Life 5: Sometimes you are told wrong. This is why it's important to allow users to pass it in explicitly.

[DRMacIver]

Right, but the correct way to pass it in explicitly is to fix your source code so it's not lying. :-) yapf already does not correctly reformat syntactically invalid source after all.

[gotgenes]

I hadn't thought of it that way, but you make a good point. For convenience, if the user provides a file with encoding declared inside the file via a magic comment matching one of the PEP 263 specifications, then yapf should read this comment line and select this encoding for decoding the file.

I still advocate for yapf and its API to accept encoding as an explicit parameter when reading source files, as there are still cases for which it would be useful; for example, when the source file does not contain an explicit encoding magic comment, or when tokenize.open (and by dependency, tokenize.detect_encoding) guess incorrectly. Otherwise, yapf will force users to declare a magic comment to any source file they run it through the tool, which is very opinionated (even though I agree with it) and may overstep the scope of this tool. Such opinion seems better relegated to style checkers like pep8 and Pylint.

[asottile]

Does it make sense to maintain the original encoding or to fix it to something reasonable (utf-8)? In py2 it kind of makes sense to maintain the original encoding as bytestrings will be encoded using that encoding. For py3 there's very little reason to specify a source encoding afaict.

Also if we're going by how the interpreter handles non-ascii (non-utf-8 py3) I think it's reasonable to reject files without an explicit encoding if they are not UTF-8).

Also probably relevant to this discussion: https://www.python.org/dev/peps/pep-3120/

Here's how we detect encoding in another project I've worked on (also dealing with 2to3 + automatic refactorings!) https://github.com/bukzor/RefactorLib/blob/master/refactorlib/python/parse.py#L5-L22

I'm interested in working on this issue.

[bwendling]

I think that this is taken care of by: 1d42a26