robertcraigie/pytest-pyright

Pytest plugin for type checking code with Pyright

pytest-pyright

⚠️ This project was created for internal use within another project of mine, support will be minimal.

Pytest plugin for type checking code with Pyright.

---

This pytest plugin was generated with Cookiecutter along with @hackebrot's cookiecutter-pytest-plugin template.

Why Should You Use it?

pytest-pyright was created to ensure that complex types are correctly constrained, i.e will raise an error if used incorrectly.

if your project makes use of any complex types you should find some value out pytest-pyright.

Installation

You can install "pytest-pyright" via pip from PyPI:

$ pip install pytest-pyright

Usage

See the documentation

Contributing

Contributions are very welcome. Tests can be run with tox, please ensure the coverage at least stays the same before you submit a pull request.

License

Distributed under the terms of the MIT license, "pytest-pyright" is free and open source software

Issues

If you encounter any problems, please file an issue along with a detailed description.

pytest-pyright's Issues

Tests are failing

I cloned this repository and ran tox, and it seems like the tests failed:

tests/test_formatting.py ..  
tests/test_plugin.py ..FF.FFFF..

[snip]

py37: SKIP (0.03 seconds)
py38: SKIP (0.01 seconds)
py39: FAIL code 1 (15.66=setup[7.52]+cmd[8.14] seconds)
py310: SKIP (0.01 seconds)
docs: OK (7.72=setup[3.74]+cmd[3.98] seconds)
pypy3: SKIP (0.01 seconds)
lint: FAIL code 1 (6.73=setup[3.77]+cmd[0.85,2.12] seconds)
evaluation failed :( (30.36 seconds)

I'm on macOS 13.3.1. I'm not very familiar with tox, but it seems like it sets up its own environment so it shouldn't be anything related to my Python environment, right?

Here's the output of the first failing test:

================================================================================================================= FAILURES =================================================================================================================
____________________________________________________________________________________________________ test_reveal_type_incorrect_comment ____________________________________________________________________________________________________

pytester = <Pytester PosixPath('/private/var/folders/t6/lff4nkz567qcb589_dpx3s000000gn/T/pytest-of-gsgx/pytest-1/test_reveal_type_incorrect_comment0')>

    def test_reveal_type_incorrect_comment(pytester: Pytester) -> None:
        content = '''
        def foo(a: str) -> None:
            reveal_type(a)  # T: int
        '''
        pytester.makepyfile(**{'typesafety/bar.py': content})
        result = pytester.runpytest()
        result.assert_outcomes(failed=1)
>       result.stdout.fnmatch_lines(
            [
                '1 | def foo(a: str) -> None:',
                '2 |     reveal_type(a)  # T: int',
                'E | Expected revealed type to be "int" but got "str" instead',
                *SUMMARY_LINES,
            ],
            consecutive=True,
        )
E       Failed: nomatch: '1 | def foo(a: str) -> None:'
E           and: '=========================================================================================================== test session starts ============================================================================================================'
E           and: 'platform darwin -- Python 3.9.6, pytest-7.2.0, pluggy-1.0.0'
E           and: 'rootdir: /private/var/folders/t6/lff4nkz567qcb589_dpx3s000000gn/T/pytest-of-gsgx/pytest-1/test_reveal_type_incorrect_comment0'
E           and: 'plugins: pyright-0.0.3, syrupy-3.0.5'
E           and: 'collected 1 item'
E           and: ''
E           and: 'typesafety/bar.py F                                                                                                                                                                                                                  [100%]'
E           and: ''
E           and: '================================================================================================================= FAILURES ================================================================================================================='
E           and: '_____________________________________________________________________________________________________________ pyright: bar.py ______________________________________________________________________________________________________________'
E       exact match: '1 | def foo(a: str) -> None:'
E       exact match: '2 |     reveal_type(a)  # T: int'
E       exact match: 'E | Expected revealed type to be "int" but got "str" instead'
E       no consecutive match: '--------------------------- snapshot report summary ----------------------------'
E          with: '--------------------------------------------------------------------------------------------------------- snapshot report summary ----------------------------------------------------------------------------------------------------------'

/Users/gsgx/code/pytest-pyright/tests/test_plugin.py:67: Failed
----------------------------------------------------------------------------------------------------------- Captured stdout call -----------------------------------------------------------------------------------------------------------
=========================================================================================================== test session starts ============================================================================================================
platform darwin -- Python 3.9.6, pytest-7.2.0, pluggy-1.0.0
rootdir: /private/var/folders/t6/lff4nkz567qcb589_dpx3s000000gn/T/pytest-of-gsgx/pytest-1/test_reveal_type_incorrect_comment0
plugins: pyright-0.0.3, syrupy-3.0.5
collected 1 item

typesafety/bar.py F                                                                                                                                                                                                                  [100%]

================================================================================================================= FAILURES =================================================================================================================
_____________________________________________________________________________________________________________ pyright: bar.py ______________________________________________________________________________________________________________
1 | def foo(a: str) -> None:
2 |     reveal_type(a)  # T: int
E | Expected revealed type to be "int" but got "str" instead
--------------------------------------------------------------------------------------------------------- snapshot report summary ----------------------------------------------------------------------------------------------------------

========================================================================================================= short test summary info ==========================================================================================================
FAILED typesafety/bar.py::bar.py
============================================================================================================ 1 failed in 0.88s =============================================================================================================

Improve error message checking

Error checking should pass if the expected error matches any of the pyright error lines

Trim error output

We currently output the entire file contents, we should change this so that only the lines surrounding an error are displayed

Fix case sensitive file paths on windows

Windows tests can fail with an unrecognised file error in some cases, e.g.

E |  Received diagnostic for unknown file: d:\projects\prisma-client-py\typesafety\pyright\validator.py; Expected D:\projects\prisma-client-py\typesafety\pyright\validator.py

Type check all items at once

This would significantly improve performance, should still add flag to disable this behaviour

Add a `--pyright-update` flag to automatically write missing checks to the source file

Refactor file collection to collect test_*.pyright files

This would add support for mixing and matching standard pytest files and pyright files side by side which would improve dx as type tests are located next to the unit tests

Difficulty with this is that even if we pass non .py files directly to Pyright, they are ignored so we would need to copy the .pyright files to a temporary directory, I don't know how this would effect module resolving within pyright.

Massive benefit of this would be that pytest currently imports all the typesafety test files looking for test functions, structuring tests like this would get around that.