File encoding should be detected correctly

[pycqa]

File encoding should be detected correctly

[jquast]

Here is a PR for another tool implementing precisely that, landscapeio/prospector#71

[The-Compiler]

There's tokenize.detect_encoding and in later Python versions, tokenize.open which is probably a lot simpler than that PR 😉

[jquast]

nice hint @The-Compiler, unfortunately tokenize.detect_encoding isn't available in python2.7, presume pep257 aims support for python2.7

[peterjc]

I just traced a problem in my plugin based on pydocstyle to this bit of code:

https://github.com/PyCQA/pydocstyle/blob/2.0.0/src/pydocstyle/checker.py#L23

# If possible (python >= 3.2) use tokenize.open to open files, so PEP 263
# encoding markers are interpreted.
try:
    tokenize_open = tk.open
except AttributeError:
    tokenize_open = open

Would a backport of tokenize.open for Python 2 be acceptable here?

Note it seems that because later the contents of the file get executed, it is helpful to also remove/replace the encoding line to avoid triggering SyntaxError: encoding declaration in Unicode string when passing it to exec(...):

https://github.com/PyCQA/pydocstyle/blob/2.0.0/src/pydocstyle/parser.py#L286

        try:
            compile(src, filename, 'exec')
        except SyntaxError as error:
            six.raise_from(ParseError(), error)

I'm not proud of this hacked together back port, but it might serve if Python 2.7 support is desired:

peterjc/flake8-rst-docstrings@80697a6

[peterjc]

See also pull request #152 which added a test case using # -*- coding: Windows-1255 -*

[sigmavirus24]

pycodestyle has code to work around the absence of tokenize.open. Flake8 has taken and improved it as well.

[peterjc]

@sigmavirus24 Can you point me at this code please? And can this issue be closed?

[aphedges]

pydocstyle has been using tokenizer.open when reading source files since 4948f22. This is therefore no longer a problem.