max-sixty / pytest-accept Goto Github PK
View Code? Open in Web Editor NEWA pytest plugin for automatically updating doctest outputs
License: Apache License 2.0
A pytest plugin for automatically updating doctest outputs
License: Apache License 2.0
Hi,
I am currently using pytest-accept for this issue of xarray: pydata/xarray#8690
I noticed a bug when a docstring contains a backslash at the end of line, then the lines are not properly replaced, an offset is introduced and it breaks the docstring
Example of a docstring that will surely fail on the xarray repo
Zoom on the section containing a backslash:
combine_attrs : {"drop", "identical", "no_conflicts", "drop_conflicts", \
"override"} or callable, default: "override"
This docstring will make the auto-replacement fail. I attach here a screenshot to visually see the issue:
When removing the backslash, the pytest-accept tool successfully replace. Currently as a way to mitigate, I plan to remove all the backslashes at ends of line, running pytest-accept and then git revert
the backslash removal.
Thanks!
Edit: the issue is already known and mentioned in the README
it reports line numbers incorrectly in some cases — two docstring lines separated with continuation character \ is counted as one, meaning this library will not have access to the correct line number for doctest inputs and outputs.
Assume a function with doctest where the return create a linebreak \n
:
import xarray as xr
def func(a, b):
"""
Example:
>>> import xarray as xr
>>> func(xr.DataArray([5], dims='a'), xr.DataArray([1], dims='a'))
"""
c = a + b
c = c.assign_attrs(a=a, b=b)
return c
pytest --doctest-modules file.py --accept
adds the expected:
import xarray as xr
def func(a, b):
"""
Example:
>>> import xarray as xr
>>> func(xr.DataArray([5], dims='a'), xr.DataArray([1], dims='a'))
<xarray.DataArray (a: 1)>
array([6])
Dimensions without coordinates: a
Attributes:
a: <xarray.DataArray (a: 1)>\narray([5])\nDimensions without coord...
b: <xarray.DataArray (a: 1)>\narray([1])\nDimensions without coord...
"""
c = a + b
c = c.assign_attrs(a=a, b=b)
return c
Usually I expect that I can run pytest --doctest-modules file.py --accept
and all doctests pass. Now they dont because a single \
is present but docstrings somehow expect \\n
. I get an error message:
――――――――――――――――――――――――――――――――――――――――――――――――― ERROR collecting file.py ――――――――――――――――――――――――――――――――――――――――――――――――――
../../anaconda3/envs/climpred-dev/lib/python3.9/doctest.py:939: in find
self._find(tests, obj, name, module, source_lines, globs, {})
../../anaconda3/envs/climpred-dev/lib/python3.9/site-packages/_pytest/doctest.py:522: in _find
doctest.DocTestFinder._find( # type: ignore
../../anaconda3/envs/climpred-dev/lib/python3.9/doctest.py:1001: in _find
self._find(tests, val, valname, module, source_lines,
../../anaconda3/envs/climpred-dev/lib/python3.9/site-packages/_pytest/doctest.py:522: in _find
doctest.DocTestFinder._find( # type: ignore
../../anaconda3/envs/climpred-dev/lib/python3.9/doctest.py:989: in _find
test = self._get_test(obj, name, module, globs, source_lines)
../../anaconda3/envs/climpred-dev/lib/python3.9/doctest.py:1073: in _get_test
return self._parser.get_doctest(docstring, globs, name,
../../anaconda3/envs/climpred-dev/lib/python3.9/doctest.py:675: in get_doctest
return DocTest(self.get_examples(string, name), globs,
../../anaconda3/envs/climpred-dev/lib/python3.9/doctest.py:689: in get_examples
return [x for x in self.parse(string, name)
../../anaconda3/envs/climpred-dev/lib/python3.9/doctest.py:651: in parse
self._parse_example(m, name, lineno)
../../anaconda3/envs/climpred-dev/lib/python3.9/doctest.py:720: in _parse_example
self._check_prefix(want_lines, ' '*indent, name,
../../anaconda3/envs/climpred-dev/lib/python3.9/doctest.py:805: in _check_prefix
raise ValueError('line %r of the docstring for %s has '
E ValueError: line 10 of the docstring for tes.func has inconsistent leading whitespace: 'array([5])'
Manually correcting \n
to \\n
lets tests pass:
import xarray as xr
def func(a, b):
"""
Example:
>>> import xarray as xr
>>> func(xr.DataArray([5], dims='a'), xr.DataArray([1], dims='a'))
<xarray.DataArray (a: 1)>
array([6])
Dimensions without coordinates: a
Attributes:
a: <xarray.DataArray (a: 1)>\\narray([5])\\nDimensions without coord...
b: <xarray.DataArray (a: 1)>\\narray([1])\\nDimensions without coord...
"""
c = a + b
c = c.assign_attrs(a=a, b=b)
return c
So would it be possible to add a small correction .replace("\n", "\\n")
to pytest-accept
?
Probably \n
is not the only issue here.
Currently if a result is one line of 100K characters, pytest-accept will add it to the file, which can then crash an editor.
Probably we should truncate the middle and leave X characters at the beginning and end of each line, joined with ...
As discussed in the readme, currently pytest-accept will overwrite a file unless --accept-copy
is passed.
Generally that's fine. But recently I've been using it as a realtime tool recently (i.e. it runs with pytest -f --accept
, running on every save), which is really cool. And it requires pausing on each save for it to generate and write new results, and getting that wrong causes an overwrite and resolving the change in an editor (or losing it), which can get confusing, even if it's only a couple of seconds of changes.
I think by default it should check that the file hasn't changed before overwriting. If it's changed, it still fails the test as it would otherwise, and prints a message saying it didn't overwrite and suggesting to run again
If there is a trailing ellipsis at the end of a doctest statement
My IDE syntax coloring, and various other examples in the xarray codebase suggests me that there should not be any trailing Ellipsis at the end of a multi-line doctest statement, but I thought it was still worth mentioning this edge case!
Screenshot: Diff before and after running pytest-accept
De we want to automatically remove strings that are unlikely to be consistent across runs, e.g. memory addresses (<__main__.A at 0x13ab3cd90>
)?
Thoughts:
0x1[0-f]{8}
? Or something to cover more examples?Thanks for this project! I was trying to put some colour on this approach of doing testing. I have read the same blogpost as you from Jane Street folks and was looking if someone did something similar in the python land.
I really like the approach you took to leverage the already existing doctest-modules facility.
I want to propose the implementation of a --long-lines
option to make pytest-accept avoid wrapping up "big" lines and I want all the output to be shown. We have a bunch of scrapers and we want to integrate tests altogether with code.
This is an example of how one can wrap long output in doctests, the --long-lines
option ought to produce something like:
def large_return_value():
"""
def large_return_value():
"""
>>> large_return_value()
[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1\
, 1, 1, 1, 1, 1, 1, 1,1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, \
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1\
, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, \
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1\
, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, \
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1\
, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, \
1, 1, 1, 1, 1, 1, 1]
"""
return [ 1 for _ in range(300)]
"""
return [ 1 for _ in range(300)]
This is a hacky PR I created just to have a baseline for this conversation: #42
Would you accept a contribution for the above?
Also can you give me guidance on how to setup the development environment? I come from a virtualenv
-inspired workflow but so far I had no success in installing pytest_accept in there. I am totally unfamiliar with poetry
.
If I do pip install -e .
I find it does not execute the pytest11 hook to install the plugin and I have to supply manually the -p
option for pytest to find the plugin.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.