withsecureopensource / pytest-rts Goto Github PK
View Code? Open in Web Editor NEWCoverage-based regression test selection (RTS) plugin for pytest
License: Apache License 2.0
Coverage-based regression test selection (RTS) plugin for pytest
License: Apache License 2.0
At the moment to get the list of files and list of changes two separate calls to git diff
are used. It is good to perform one call to git to get a structure (a new class GitChanges
?) containing information about changed files and changes in files.
Maybe it would be good to add this functionality to pydriller
and create a PR there
$ git grep -B5 file_diff_data_branch
tests_selector/evaluation/start.py-from tests_selector.utils.git import (
tests_selector/evaluation/start.py- get_git_repo,
tests_selector/evaluation/start.py- file_diff_data_current,
tests_selector/evaluation/start.py- get_test_lines_and_update_lines,
tests_selector/evaluation/start.py- changed_files_branch,
tests_selector/evaluation/start.py: file_diff_data_branch,
--
tests_selector/evaluation/start.py-
tests_selector/evaluation/start.py- changed_files = changed_files_branch(PROJECT_FOLDER)
tests_selector/evaluation/start.py- tests_line_level = set()
tests_selector/evaluation/start.py- tests_file_level = set()
tests_selector/evaluation/start.py- for f in changed_files:
tests_selector/evaluation/start.py: diff = file_diff_data_branch(f, PROJECT_FOLDER)
It's not defined anywhere, as far as I can tell
@guotin not sure that it is still relevant. If you have plans please describe it here if not and it is in good shape - just close the issue
We have the following situations:
It is pretty clear how to handle the first situation. How do we handle the latter two?
@guotin or @alexey-vyskubov could you please describe the problem?
Decided to post an issue here to have a cheat-sheet on what to exactly have in the test.
helper_project
to a temp folder, run tests_selector_init
and checkout a new branch
tests_selector
, test that same tests found and database updated for that runtests_selector
, test that same new test ran and database updated@matveypashkovskiy
Comment if I am missing some crucial step. I can't quite remember if you did a step with a normal addition (like shift existing lines by 1 or something) and whether that might be something important to test here.
Instead of having multiply entry points, we should support one executable with multiple commands (tests_selector init
instead of tests_selector_init
) via something like click
We might also want to invent shorter name for the executable (I'd hate if git
would be instead version_control
:D), but that's "naming things" -- one of the two most difficult problem in the computer science (funny that the tool itself is pretty much about the second, which is "cache invalidation") ๐
Here is an example from a project:
return (
df
.withColumnRenamed("a", "b")
)
if the code is modified to:
return self.reset(
df
.withColumnRenamed("a", "b")
)
no tests will be found for the change as DB contains no mapping info about return (
line.
Any ideas on how to fix it? Any magic pytest-cov flag maybe?
https://github.com/ishepard/pydriller/releases/tag/2.0
It was time to rename and shorten the main classes:
- RepositoryMining -> Repository
- GitRepository -> Git
Quite common scenario is the tests reading input and/or expected results from files. How can we make sure that changes in those "data files" will trigger re-run of the corresponding tests? Some approaches, with varying degree of success and complexity could be:
open()
somehowATM if pytest --rts
executed not in a git repo (no .git
folder) execution fails. Instead, let's print a warning. Something like: "Not a git repository! pytest-rts is disabled. Run git init
before using pytest-rts"
Here is an example how tests are executed in one of F-Secure projects:
pytest --ignore=node_modules \
--cov=b2b \
--cov-report=html \
--cov-report=term \
--cov-fail-under=90 \
We have several cases where coverage is inaccurate:
Related discussion: nedbat/coveragepy#974
The rule is pretty simple: no DB run init, DB exists run tests_selector
shall we employ some ORM framework to simplify work with DB?
I was trying to produce an example for the problem with decorators we discussed, and put my code into pytest-rts-try-decorators
directory. Because of the (incorrect) hardcoded condition in the save_mapping_data()
the tool does not work at all: "pytest-rts" in filename
is wrong.
INTERNALERROR> File "/Users/pashma/workspace/b2b-data-collection-pipeline/.venv/lib/python3.6/site-packages/pytest_rts/pytest/update_phase_plugin.py", line 57, in pytest_collection_modifyitems
INTERNALERROR> items[:] = sorted(selected, key=lambda item: updated_runtimes[item.nodeid])
INTERNALERROR> TypeError: '<' not supported between instances of 'NoneType' and 'float'
Not sure how to reproduce the situation where the execution time of a test is None, but it happens.
we have to add somethign like that to one of our documents (copied from the paper):
Our technique uses Git version control system and Coverage.py for tracking changes in the code. For the test runner, PyTest is used. The procedure of the technique is as follows: First, an initial run of all tests is performed. While performing the firsts run of the tests, a locally stored SQLite database is constructed with the coverage data provided by Coverage.py. The database contains six tables:
After the initial full test suite run, the tool is ready to be used. When changes are made to the target project's files, the tool checks for changes in the Git working directory. The tool first constructs a list of changed files according to Git and checks which of those files are either source code files or test code files in our local database. After the tool has determined which files are taken into consideration, it checks the Git diff -output for each of those files. From this 'diff', the tool can determine which lines have changed and which lines have shifted from their original position. Then the tool can query all the test functions from our database according to the list of line numbers for the changed lines and run them with PyTest. No database state updating is performed during this. If a user wishes to make these changes final, a Git commit operation is required. When the changes are committed, the tool checks whether the current Git HEAD hash differs from the one that is marked as a last update hash. If so, the tool queries the changes and tests for those changes almost as before. As a small addition, it does two additional things. The tool calculates how unchanged lines have shifted in the files and performs a database update based on this information. It also checks for newly added test functions by checking what test functions PyTest can find and comparing it to the current state in the database. When the tests are run after this, new coverage data is collected and inserted into the database.
Create a project where there is some logic inside __init__.py
file. Add a test for that logic. Run pytes_rts_init
. Change the logic breaking the tests. Run pytest_rts
-- notice that no tests are run. Looking into mapping.db
one can notice that __init__.py
is not in the src_file
table.
Looks like I noticed one problem. If you have statement like: return 1 + 2
and you change it to return 1
tests related to that line won't be found. @guotin could you please check it?
I'm getting non-zero return code after running pytest-rts.
> pytest --rts --rts-coverage-db=mapping.db
============ test session starts =============
platform darwin -- Python 3.6.8, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
rootdir: /Users/pashma/workspace/b2b-data-collection-pipeline
plugins: cov-2.11.1, rts-2.1.0
collected 52 items / 52 deselected
============= warnings summary ===============
....
===== 52 deselected, 1 warning in 1.62s ======
> echo $?
5
Any ideas about why that happens?
Hey @alexey-vyskubov & @matveypashkovskiy
What should we do with the existing evaluation code? I spent some time this week to fix it again (could make a PR soon) but should it be somewhere else entirely? I was just thinking that if this tool is to be transformed to a pytest-plugin then is the evaluation code ever going to be used. Does it make sense that a pytest plugin contains something like that ?
Just FYI, I noticed a small bug related to updating the database:
save_data
method/functionI will try to fix this next.
During a conversation with @Twan00 we found out that it would be beneficial to integrate the module with Robot Framework and build mapping between Robot Framework test cases and lines of code. Let's see how we can do that.
Created a small project to reproduce coverage percentage problem but run into problems during the mapping database initialization.
If I run make test-rts
it fails with:
=========================================================================== test session starts ============================================================================
platform darwin -- Python 3.6.8, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
rootdir: /Users/pashma/workspace/pytest-rts-cov
plugins: cov-2.11.1, rts-1.1.10
collected 1 item
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR> File "/Users/pashma/workspace/pytest-rts-cov/.venv/lib/python3.6/site-packages/_pytest/main.py", line 269, in wrap_session
INTERNALERROR> session.exitstatus = doit(config, session) or 0
INTERNALERROR> File "/Users/pashma/workspace/pytest-rts-cov/.venv/lib/python3.6/site-packages/_pytest/main.py", line 322, in _main
INTERNALERROR> config.hook.pytest_collection(session=session)
INTERNALERROR> File "/Users/pashma/workspace/pytest-rts-cov/.venv/lib/python3.6/site-packages/pluggy/hooks.py", line 286, in __call__
INTERNALERROR> return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR> File "/Users/pashma/workspace/pytest-rts-cov/.venv/lib/python3.6/site-packages/pluggy/manager.py", line 93, in _hookexec
INTERNALERROR> return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR> File "/Users/pashma/workspace/pytest-rts-cov/.venv/lib/python3.6/site-packages/pluggy/manager.py", line 87, in <lambda>
INTERNALERROR> firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
INTERNALERROR> File "/Users/pashma/workspace/pytest-rts-cov/.venv/lib/python3.6/site-packages/pluggy/callers.py", line 208, in _multicall
INTERNALERROR> return outcome.get_result()
INTERNALERROR> File "/Users/pashma/workspace/pytest-rts-cov/.venv/lib/python3.6/site-packages/pluggy/callers.py", line 80, in get_result
INTERNALERROR> raise ex[1].with_traceback(ex[2])
INTERNALERROR> File "/Users/pashma/workspace/pytest-rts-cov/.venv/lib/python3.6/site-packages/pluggy/callers.py", line 187, in _multicall
INTERNALERROR> res = hook_impl.function(*args)
INTERNALERROR> File "/Users/pashma/workspace/pytest-rts-cov/.venv/lib/python3.6/site-packages/_pytest/main.py", line 333, in pytest_collection
INTERNALERROR> session.perform_collect()
INTERNALERROR> File "/Users/pashma/workspace/pytest-rts-cov/.venv/lib/python3.6/site-packages/_pytest/main.py", line 638, in perform_collect
INTERNALERROR> session=self, config=self.config, items=items
INTERNALERROR> File "/Users/pashma/workspace/pytest-rts-cov/.venv/lib/python3.6/site-packages/pluggy/hooks.py", line 286, in __call__
INTERNALERROR> return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR> File "/Users/pashma/workspace/pytest-rts-cov/.venv/lib/python3.6/site-packages/pluggy/manager.py", line 93, in _hookexec
INTERNALERROR> return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR> File "/Users/pashma/workspace/pytest-rts-cov/.venv/lib/python3.6/site-packages/pluggy/manager.py", line 87, in <lambda>
INTERNALERROR> firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
INTERNALERROR> File "/Users/pashma/workspace/pytest-rts-cov/.venv/lib/python3.6/site-packages/pluggy/callers.py", line 208, in _multicall
INTERNALERROR> return outcome.get_result()
INTERNALERROR> File "/Users/pashma/workspace/pytest-rts-cov/.venv/lib/python3.6/site-packages/pluggy/callers.py", line 80, in get_result
INTERNALERROR> raise ex[1].with_traceback(ex[2])
INTERNALERROR> File "/Users/pashma/workspace/pytest-rts-cov/.venv/lib/python3.6/site-packages/pluggy/callers.py", line 187, in _multicall
INTERNALERROR> res = hook_impl.function(*args)
INTERNALERROR> File "/Users/pashma/workspace/pytest-rts-cov/.venv/lib/python3.6/site-packages/pytest_rts/pytest/init_phase_plugin.py", line 36, in pytest_collection_modifyitems
INTERNALERROR> for testfile_path in self.testfiles
INTERNALERROR> File "/Users/pashma/workspace/pytest-rts-cov/.venv/lib/python3.6/site-packages/pytest_rts/pytest/init_phase_plugin.py", line 36, in <dictcomp>
INTERNALERROR> for testfile_path in self.testfiles
INTERNALERROR> File "/Users/pashma/workspace/pytest-rts-cov/.venv/lib/python3.6/site-packages/coverage/python.py", line 61, in get_python_source
INTERNALERROR> raise NoSource(exc_msg)
INTERNALERROR> coverage.misc.NoSource: No source for code: 'tests/foo_test.py'.
INTERNALERROR> Aborting report output, consider using -i.
========================================================================== no tests ran in 0.06s ===========================================================================
make: *** [test-rts] Error 3
It looks like a config problem of coverage.py.
Steps to reproduce:
git add
)pytest --rts
Observer behaviour:
Expected behaviour:
It happens because git diff --name-only
doesn't list untracked files. Probably we need to add something like: + subprocess.check_output("git ls-files --others --exclude-standard".split()).split()
to the end of the line
For example coverage plugin for pytest could be invoked like:
pytest -x
--cov=tests_selector
--cov-report=html
--cov-report=term
--cov-fail-under=90
can we do the same?
In the same small project execution of make test-rts
gives:
pytest_rts_cov/tests/foo_test.py . [100%]
---------- coverage: platform darwin, python 3.6.8-final-0 -----------
Name Stmts Miss Cover
------------------------------------------------
pytest_rts_cov/__init__.py 0 0 100%
pytest_rts_cov/bar.py 3 3 0%
pytest_rts_cov/foo.py 3 1 67%
------------------------------------------------
TOTAL 6 4 33%
while make test
gives:
pytest_rts_cov/tests/foo_test.py . [100%]
---------- coverage: platform darwin, python 3.6.8-final-0 -----------
Name Stmts Miss Cover
------------------------------------------------
pytest_rts_cov/__init__.py 0 0 100%
pytest_rts_cov/bar.py 3 3 0%
pytest_rts_cov/foo.py 3 0 100%
------------------------------------------------
TOTAL 6 3 50%
so we have something like that:
import logging
logger = logging.getLogger()
logger.debug("debug message!")
...and documentation clearly says:
DESCRIPTION
Instead of importing this module directly, import os and refer to this
module as os.path.
Hey @alexey-vyskubov & @matveypashkovskiy
What should be done with the README of this repo? Is anyone ever going to read some of those articles in the table or should they just be removed? Also the other old stuff in there might be out of place.
For added lines there could be different approaches:
As agreed let's integrate with pytest-cov and move with small steps with a limited scope of PRs to do reviews faster. @guotin three "features" were suggested before:
And as you said, from pytest-cov pp. 2 and 3 are the same. Let's select the simplest functionality that we can build fast. For example, it could be part of p 1.: running all uncommitted tests. How does it sound? Do you see any simpler functionality?
I suppose we need a snapshot(copy) of coverage database to find tests that are not listed there thus, root folder will contain .coverage.[COMMIT_HASH]
(any name could be used) file and a flag --rts-coverage-db=.coverage.[COMMIT_HASH]
will be passed.
From the technical point of view let's:
__version__ = "2.0.0"
in pytest-rts/__init__.py
one of the options is to do that in two steps:
master
branch: git cherry -v master
git show [COMMIT_HASH] -U0
Here's a few questions I have regarding making the integration test pass
What would be the way to approach this? Git diff data is as follows (this is what the integration test now has):
diff --git a/src/car.py b/src/car.py
index 34b567f..f5a4555 100644
--- a/src/car.py
+++ b/src/car.py
@@ -23 +23,5 @@ class Car:
- self.passengers = self.passengers - 1
\ No newline at end of file
+ self.passengers = self.passengers - 1
+
+ def new_method():
+ i = self.speed + self.seats
+ return i + 8
Should it somehow parse the diff data to find new def
declarations with +
-sign? This would probably also cover renames too but that's one way I thought of. Or should it somehow use the line numbers inside the @@
symbols to determine that new lines are added to the bottom of the file?
tests_selector
Running tests_selector
with changes in the working directory should get the tests from working directory changes and new tests addition in the working directory. The new test discovery is handled with pytest by comparing pytest --collect-only
to tests that exist in the database.
Now, after merging tests_selector
and tests_selector_branch
scripts, the new tests are only checked when a database update is made, meaning not in the working directory changes but only when checking for committed changes. The tests_selector
script checks if there are any working directory changes and runs them and exits if there was any. If there were no changes in the working directory, it continues to check the committed changes...
Integration test expects the newly added tests to be found in the working directory changes. If this is added, when would be the time to update the database (add the test)? If a test is discovered in the working directory changes but never added to the database before the branch test selection, the test is always found in the working directory changes and the script never progresses to look at the committed changes. So should the new test discovery be only left to the committed changes phase, as it is now or should some other trick be performed to make the integration test pass with the current test code?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.