max-sixty/pytest-accept

A pytest plugin for automatically updating doctest outputs

pytest-accept's Issues

Issue with docstrings containing backslashes at the end of line

Hi,

I am currently using pytest-accept for this issue of xarray: pydata/xarray#8690

I noticed a bug when a docstring contains a backslash at the end of line, then the lines are not properly replaced, an offset is introduced and it breaks the docstring

Example of a docstring that will surely fail on the xarray repo

Zoom on the section containing a backslash:

   combine_attrs : {"drop", "identical", "no_conflicts", "drop_conflicts", \
                     "override"} or callable, default: "override"

This docstring will make the auto-replacement fail. I attach here a screenshot to visually see the issue:

When removing the backslash, the pytest-accept tool successfully replace. Currently as a way to mitigate, I plan to remove all the backslashes at ends of line, running pytest-accept and then git revert the backslash removal.

Thanks!

Edit: the issue is already known and mentioned in the README

it reports line numbers incorrectly in some cases — two docstring lines separated with continuation character \ is counted as one, meaning this library will not have access to the correct line number for doctest inputs and outputs.

Long lines support

Thanks for this project! I was trying to put some colour on this approach of doing testing. I have read the same blogpost as you from Jane Street folks and was looking if someone did something similar in the python land.
I really like the approach you took to leverage the already existing doctest-modules facility.

I want to propose the implementation of a --long-lines option to make pytest-accept avoid wrapping up "big" lines and I want all the output to be shown. We have a bunch of scrapers and we want to integrate tests altogether with code.

This is an example of how one can wrap long output in doctests, the --long-lines option ought to produce something like:

def large_return_value():
    """
def large_return_value():
    """
    >>> large_return_value()
    [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1\
, 1, 1, 1, 1, 1, 1, 1,1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, \
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\
 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1\
, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, \
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\
 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1\
, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, \
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\
 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1\
, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, \
1, 1, 1, 1, 1, 1, 1]
    """

    return [ 1 for _ in range(300)]

    """

    return [ 1 for _ in range(300)]

This is a hacky PR I created just to have a baseline for this conversation: #42

Would you accept a contribution for the above?

Also can you give me guidance on how to setup the development environment? I come from a virtualenv-inspired workflow but so far I had no success in installing pytest_accept in there. I am totally unfamiliar with poetry.
If I do pip install -e . I find it does not execute the pytest11 hook to install the plugin and I have to supply manually the -p option for pytest to find the plugin.

Issue with doctest result containing escaped backslashes

When a doctest result contains escaped backslashes, after a run of pytest-accept, they are not escaped anymore

Potential solution according to the README: use a raw string as a docstring

correcting linebreaks: `\n` --> `\\n`

Assume a function with doctest where the return create a linebreak \n:

import xarray as xr

def func(a, b):
    """
    Example:
    >>> import xarray as xr
    >>> func(xr.DataArray([5], dims='a'), xr.DataArray([1], dims='a'))
    """
    c = a + b
    c = c.assign_attrs(a=a, b=b)
    return c

pytest --doctest-modules file.py --accept adds the expected:

import xarray as xr

def func(a, b):
    """
    Example:
    >>> import xarray as xr
    >>> func(xr.DataArray([5], dims='a'), xr.DataArray([1], dims='a'))
    <xarray.DataArray (a: 1)>
    array([6])
    Dimensions without coordinates: a
    Attributes:
        a:        <xarray.DataArray (a: 1)>\narray([5])\nDimensions without coord...
        b:        <xarray.DataArray (a: 1)>\narray([1])\nDimensions without coord...
    """
    c = a + b
    c = c.assign_attrs(a=a, b=b)
    return c

Usually I expect that I can run pytest --doctest-modules file.py --accept and all doctests pass. Now they dont because a single \ is present but docstrings somehow expect \\n. I get an error message:

――――――――――――――――――――――――――――――――――――――――――――――――― ERROR collecting file.py ――――――――――――――――――――――――――――――――――――――――――――――――――
../../anaconda3/envs/climpred-dev/lib/python3.9/doctest.py:939: in find
    self._find(tests, obj, name, module, source_lines, globs, {})
../../anaconda3/envs/climpred-dev/lib/python3.9/site-packages/_pytest/doctest.py:522: in _find
    doctest.DocTestFinder._find(  # type: ignore
../../anaconda3/envs/climpred-dev/lib/python3.9/doctest.py:1001: in _find
    self._find(tests, val, valname, module, source_lines,
../../anaconda3/envs/climpred-dev/lib/python3.9/site-packages/_pytest/doctest.py:522: in _find
    doctest.DocTestFinder._find(  # type: ignore
../../anaconda3/envs/climpred-dev/lib/python3.9/doctest.py:989: in _find
    test = self._get_test(obj, name, module, globs, source_lines)
../../anaconda3/envs/climpred-dev/lib/python3.9/doctest.py:1073: in _get_test
    return self._parser.get_doctest(docstring, globs, name,
../../anaconda3/envs/climpred-dev/lib/python3.9/doctest.py:675: in get_doctest
    return DocTest(self.get_examples(string, name), globs,
../../anaconda3/envs/climpred-dev/lib/python3.9/doctest.py:689: in get_examples
    return [x for x in self.parse(string, name)
../../anaconda3/envs/climpred-dev/lib/python3.9/doctest.py:651: in parse
    self._parse_example(m, name, lineno)
../../anaconda3/envs/climpred-dev/lib/python3.9/doctest.py:720: in _parse_example
    self._check_prefix(want_lines, ' '*indent, name,
../../anaconda3/envs/climpred-dev/lib/python3.9/doctest.py:805: in _check_prefix
    raise ValueError('line %r of the docstring for %s has '
E   ValueError: line 10 of the docstring for tes.func has inconsistent leading whitespace: 'array([5])'

Manually correcting \n to \\n lets tests pass:

import xarray as xr

def func(a, b):
    """
    Example:
    >>> import xarray as xr
    >>> func(xr.DataArray([5], dims='a'), xr.DataArray([1], dims='a'))
    <xarray.DataArray (a: 1)>
    array([6])
    Dimensions without coordinates: a
    Attributes:
        a:        <xarray.DataArray (a: 1)>\\narray([5])\\nDimensions without coord...
        b:        <xarray.DataArray (a: 1)>\\narray([1])\\nDimensions without coord...
    """
    c = a + b
    c = c.assign_attrs(a=a, b=b)
    return c

So would it be possible to add a small correction .replace("\n", "\\n") to pytest-accept?
Probably \n is not the only issue here.

Detect and redact memory addresses?

De we want to automatically remove strings that are unlikely to be consistent across runs, e.g. memory addresses (<__main__.A at 0x13ab3cd90>)?

Thoughts:

• Most of the time this is useful, but occasionally someone might want to produce and test a string like this, so do we provide opt-out?
• We don't want to put pytest-accept options into the test file, because it violates the principle of decoupling the library from the tests.
• It could be a commandline flag? But then do these proliferate? We could have a single one for all the output adjusting (e.g. trimming > 1000 rows or columns, adding blankline comments, etc)
• What do we redact? Matching memory addresses only, e.g. 0x1[0-f]{8}? Or something to cover more examples?

Issue in case of excess Ellipsis `...` at the end of a multi-line doctest statement

If there is a trailing ellipsis at the end of a doctest statement

My IDE syntax coloring, and various other examples in the xarray codebase suggests me that there should not be any trailing Ellipsis at the end of a multi-line doctest statement, but I thought it was still worth mentioning this edge case!

Screenshot: Diff before and after running pytest-accept

Truncate long lines

Currently if a result is one line of 100K characters, pytest-accept will add it to the file, which can then crash an editor.

Probably we should truncate the middle and leave X characters at the beginning and end of each line, joined with ...

Check if a file has changed before overwriting

As discussed in the readme, currently pytest-accept will overwrite a file unless --accept-copy is passed.

Generally that's fine. But recently I've been using it as a realtime tool recently (i.e. it runs with pytest -f --accept, running on every save), which is really cool. And it requires pausing on each save for it to generate and write new results, and getting that wrong causes an overwrite and resolving the change in an editor (or losing it), which can get confusing, even if it's only a couple of seconds of changes.

I think by default it should check that the file hasn't changed before overwriting. If it's changed, it still fails the test as it would otherwise, and prints a message saying it didn't overwrite and suggesting to run again