withsecureopensource / pytest-rts Goto Github PK

View Code? Open in Web Editor NEW

7.0 7.0 3.0 1.92 MB

Coverage-based regression test selection (RTS) plugin for pytest

License: Apache License 2.0

Python 98.51% Makefile 1.49%

pytest rts

pytest-rts's People

Contributors

Stargazers

Watchers

Forkers

tacky-gewgaw yigiterinc mikamove

pytest-rts's Issues

Test selector imports `ntpath` module

...and documentation clearly says:

DESCRIPTION
    Instead of importing this module directly, import os and refer to this
    module as os.path.

Statements "return (" have no coverage data

Here is an example from a project:

return (
  df
  .withColumnRenamed("a", "b")
)

if the code is modified to:

return self.reset(
  df
  .withColumnRenamed("a", "b")
)

no tests will be found for the change as DB contains no mapping info about return ( line.

Any ideas on how to fix it? Any magic pytest-cov flag maybe?

Add LICENSE.txt

avoid failing if executed not in a git repo

ATM if pytest --rts executed not in a git repo (no .git folder) execution fails. Instead, let's print a warning. Something like: "Not a git repository! pytest-rts is disabled. Run git init before using pytest-rts"

Improve coverage collection `defs/decorators/pyspark UDFs`

We have several cases where coverage is inaccurate:

declaration of methods
decorators
pyspark user defined functions

Related discussion: nedbat/coveragepy#974

Non-zero return code from --rts

I'm getting non-zero return code after running pytest-rts.

> pytest --rts --rts-coverage-db=mapping.db
============ test session starts =============
platform darwin -- Python 3.6.8, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
rootdir: /Users/pashma/workspace/b2b-data-collection-pipeline
plugins: cov-2.11.1, rts-2.1.0
collected 52 items / 52 deselected                                                                                                                                                                                                     

============= warnings summary ===============
....
===== 52 deselected, 1 warning in 1.62s ======
> echo $?
5

Any ideas about why that happens?

Tests using data files should be re-run at data files change

Quite common scenario is the tests reading input and/or expected results from files. How can we make sure that changes in those "data files" will trigger re-run of the corresponding tests? Some approaches, with varying degree of success and complexity could be:

search tests for literals which are filenames
intercept open() somehow
monitor the filesystem during test run

Coverage percentage is lower with --rts flag

In the same small project execution of make test-rts gives:

pytest_rts_cov/tests/foo_test.py .                                                      [100%]

---------- coverage: platform darwin, python 3.6.8-final-0 -----------
Name                         Stmts   Miss  Cover
------------------------------------------------
pytest_rts_cov/__init__.py       0      0   100%
pytest_rts_cov/bar.py            3      3     0%
pytest_rts_cov/foo.py            3      1    67%
------------------------------------------------
TOTAL                            6      4    33%

while make test gives:

pytest_rts_cov/tests/foo_test.py .                                                      [100%]

---------- coverage: platform darwin, python 3.6.8-final-0 -----------
Name                         Stmts   Miss  Cover
------------------------------------------------
pytest_rts_cov/__init__.py       0      0   100%
pytest_rts_cov/bar.py            3      3     0%
pytest_rts_cov/foo.py            3      0   100%
------------------------------------------------
TOTAL                            6      3    50%

return statements are not recorded to db

Looks like I noticed one problem. If you have statement like: return 1 + 2 and you change it to return 1 tests related to that line won't be found. @guotin could you please check it?

PRs merge strategy

I'm thinking to apply following config for PRs:

@alexey-vyskubov, @guotin what do you think?

README updating

Hey @alexey-vyskubov & @matveypashkovskiy

What should be done with the README of this repo? Is anyone ever going to read some of those articles in the table or should they just be removed? Also the other old stuff in there might be out of place.

Code in init.py is not tracked

Create a project where there is some logic inside __init__.py file. Add a test for that logic. Run pytes_rts_init. Change the logic breaking the tests. Run pytest_rts -- notice that no tests are run. Looking into mapping.db one can notice that __init__.py is not in the src_file table.

If directory name contains 'pytest-rts', the tool does not work

I was trying to produce an example for the problem with decorators we discussed, and put my code into pytest-rts-try-decorators directory. Because of the (incorrect) hardcoded condition in the save_mapping_data() the tool does not work at all: "pytest-rts" in filename is wrong.

Problems with PyDriller 2.x

https://github.com/ishepard/pydriller/releases/tag/2.0

It was time to rename and shorten the main classes:

RepositoryMining -> Repository

GitRepository -> Git

Integration test to verify operation

Decided to post an issue here to have a cheat-sheet on what to exactly have in the test.

Test flow

Init my helper_project to a temp folder, run tests_selector_init and checkout a new branch
- Perhaps check that correct src-files and test-files are mapped
Add a new method to an existing src-file, test that it should run tests for all the lines in that src-file without updating database yet
Commit changes, run tests_selector, test that same tests found and database updated for that run
Add a new test to an existing test file, test that it should run it without updating database
Commit changes, run tests_selector, test that same new test ran and database updated

@matveypashkovskiy
Comment if I am missing some crucial step. I can't quite remember if you did a step with a normal addition (like shift existing lines by 1 or something) and whether that might be something important to test here.

Plan prioritization strategy implementation

What to do with evaluation code

Hey @alexey-vyskubov & @matveypashkovskiy

What should we do with the existing evaluation code? I spent some time this week to fix it again (could make a PR soon) but should it be somewhere else entirely? I was just thinking that if this tool is to be transformed to a pytest-plugin then is the evaluation code ever going to be used. Does it make sense that a pytest plugin contains something like that ?

Finish evaluation code

@guotin not sure that it is still relevant. If you have plans please describe it here if not and it is in good shape - just close the issue

Merge tests_selector_init with tests_selector

The rule is pretty simple: no DB run init, DB exists run tests_selector

problems running pytest-rts first time

Created a small project to reproduce coverage percentage problem but run into problems during the mapping database initialization.

If I run make test-rts it fails with:

=========================================================================== test session starts ============================================================================
platform darwin -- Python 3.6.8, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
rootdir: /Users/pashma/workspace/pytest-rts-cov
plugins: cov-2.11.1, rts-1.1.10
collected 1 item                                                                                                                                                           
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/Users/pashma/workspace/pytest-rts-cov/.venv/lib/python3.6/site-packages/_pytest/main.py", line 269, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/Users/pashma/workspace/pytest-rts-cov/.venv/lib/python3.6/site-packages/_pytest/main.py", line 322, in _main
INTERNALERROR>     config.hook.pytest_collection(session=session)
INTERNALERROR>   File "/Users/pashma/workspace/pytest-rts-cov/.venv/lib/python3.6/site-packages/pluggy/hooks.py", line 286, in __call__
INTERNALERROR>     return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>   File "/Users/pashma/workspace/pytest-rts-cov/.venv/lib/python3.6/site-packages/pluggy/manager.py", line 93, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/Users/pashma/workspace/pytest-rts-cov/.venv/lib/python3.6/site-packages/pluggy/manager.py", line 87, in <lambda>
INTERNALERROR>     firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
INTERNALERROR>   File "/Users/pashma/workspace/pytest-rts-cov/.venv/lib/python3.6/site-packages/pluggy/callers.py", line 208, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/Users/pashma/workspace/pytest-rts-cov/.venv/lib/python3.6/site-packages/pluggy/callers.py", line 80, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/Users/pashma/workspace/pytest-rts-cov/.venv/lib/python3.6/site-packages/pluggy/callers.py", line 187, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/Users/pashma/workspace/pytest-rts-cov/.venv/lib/python3.6/site-packages/_pytest/main.py", line 333, in pytest_collection
INTERNALERROR>     session.perform_collect()
INTERNALERROR>   File "/Users/pashma/workspace/pytest-rts-cov/.venv/lib/python3.6/site-packages/_pytest/main.py", line 638, in perform_collect
INTERNALERROR>     session=self, config=self.config, items=items
INTERNALERROR>   File "/Users/pashma/workspace/pytest-rts-cov/.venv/lib/python3.6/site-packages/pluggy/hooks.py", line 286, in __call__
INTERNALERROR>     return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>   File "/Users/pashma/workspace/pytest-rts-cov/.venv/lib/python3.6/site-packages/pluggy/manager.py", line 93, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/Users/pashma/workspace/pytest-rts-cov/.venv/lib/python3.6/site-packages/pluggy/manager.py", line 87, in <lambda>
INTERNALERROR>     firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
INTERNALERROR>   File "/Users/pashma/workspace/pytest-rts-cov/.venv/lib/python3.6/site-packages/pluggy/callers.py", line 208, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/Users/pashma/workspace/pytest-rts-cov/.venv/lib/python3.6/site-packages/pluggy/callers.py", line 80, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/Users/pashma/workspace/pytest-rts-cov/.venv/lib/python3.6/site-packages/pluggy/callers.py", line 187, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/Users/pashma/workspace/pytest-rts-cov/.venv/lib/python3.6/site-packages/pytest_rts/pytest/init_phase_plugin.py", line 36, in pytest_collection_modifyitems
INTERNALERROR>     for testfile_path in self.testfiles
INTERNALERROR>   File "/Users/pashma/workspace/pytest-rts-cov/.venv/lib/python3.6/site-packages/pytest_rts/pytest/init_phase_plugin.py", line 36, in <dictcomp>
INTERNALERROR>     for testfile_path in self.testfiles
INTERNALERROR>   File "/Users/pashma/workspace/pytest-rts-cov/.venv/lib/python3.6/site-packages/coverage/python.py", line 61, in get_python_source
INTERNALERROR>     raise NoSource(exc_msg)
INTERNALERROR> coverage.misc.NoSource: No source for code: 'tests/foo_test.py'.
INTERNALERROR> Aborting report output, consider using -i.

========================================================================== no tests ran in 0.06s ===========================================================================
make: *** [test-rts] Error 3

It looks like a config problem of coverage.py.

Run tests related to changed lines

We have the following situations:

modified line
added line
removed line

It is pretty clear how to handle the first situation. How do we handle the latter two?

Describe how the tool works

we have to add somethign like that to one of our documents (copied from the paper):

Our technique uses Git version control system and Coverage.py for tracking changes in the code. For the test runner, PyTest is used. The procedure of the technique is as follows: First, an initial run of all tests is performed. While performing the firsts run of the tests, a locally stored SQLite database is constructed with the coverage data provided by Coverage.py. The database contains six tables:

'source file' which contains the full path to the source file, along with an id
'test file' which contains the full path to the test file, along with an id
'test function' which contains the test function name extracted from PyTest and information about where it's located, such as test file id and start and end line numbers in that test file
'test map' which contains information about which test function ran specific line in a specific source file
'new tests' which contains information about newly added tests after the previous run
'last update hash' which contains information when the database was last updated

After the initial full test suite run, the tool is ready to be used. When changes are made to the target project's files, the tool checks for changes in the Git working directory. The tool first constructs a list of changed files according to Git and checks which of those files are either source code files or test code files in our local database. After the tool has determined which files are taken into consideration, it checks the Git diff -output for each of those files. From this 'diff', the tool can determine which lines have changed and which lines have shifted from their original position. Then the tool can query all the test functions from our database according to the list of line numbers for the changed lines and run them with PyTest. No database state updating is performed during this. If a user wishes to make these changes final, a Git commit operation is required. When the changes are committed, the tool checks whether the current Git HEAD hash differs from the one that is marked as a last update hash. If so, the tool queries the changes and tests for those changes almost as before. As a small addition, it does two additional things. The tool calculates how unchanged lines have shifted in the files and performs a database update based on this information. It also checks for newly added test functions by checking what test functions PyTest can find and comparing it to the current state in the database. When the tests are run after this, new coverage data is collected and inserted into the database.

Introduce tox-based test setup to test compatibility with different python versions

It is important that test selector works with different versions of Python, starting from 3.6 (see issue #25 ) to 3.9. The easiest way to support testing with different Python versions is to use tox

Integrate with Robot Framework

During a conversation with @Twan00 we found out that it would be beneficial to integrate the module with Robot Framework and build mapping between Robot Framework test cases and lines of code. Let's see how we can do that.

Run tests that are not yet added to git

Steps to reproduce:

Create new source file
Create a test file for it (do not run git add)
run pytest --rts

Observer behaviour:

The test won't be executed

Expected behaviour:

The test is executed

It happens because git diff --name-only doesn't list untracked files. Probably we need to add something like: + subprocess.check_output("git ls-files --others --exclude-standard".split()).split() to the end of the line

Improve command-line usage

Instead of having multiply entry points, we should support one executable with multiple commands (tests_selector init instead of tests_selector_init) via something like click

We might also want to invent shorter name for the executable (I'd hate if git would be instead version_control :D), but that's "naming things" -- one of the two most difficult problem in the computer science (funny that the tool itself is pretty much about the second, which is "cache invalidation") 😄

pytest-cov integration: execution of new tests

As agreed let's integrate with pytest-cov and move with small steps with a limited scope of PRs to do reviews faster. @guotin three "features" were suggested before:

all the new tests (both committed and uncommitted)
all the changed tests (both committed and uncommitted)
all tests that are related to the changed source lines

And as you said, from pytest-cov pp. 2 and 3 are the same. Let's select the simplest functionality that we can build fast. For example, it could be part of p 1.: running all uncommitted tests. How does it sound? Do you see any simpler functionality?

I suppose we need a snapshot(copy) of coverage database to find tests that are not listed there thus, root folder will contain .coverage.[COMMIT_HASH] (any name could be used) file and a flag --rts-coverage-db=.coverage.[COMMIT_HASH] will be passed.

From the technical point of view let's:

In very first commit add __version__ = "2.0.0" in pytest-rts/__init__.py
Remove all irrelevant code and tests - we can always bring it back from history. I think we don't need git module for discovering new tests as pytest, in theory, will pass us test items and we can check if we see them in the DB snapshot. Do I understand the process right? DB utils integration not needed here as well as we are using pytest-cov API, isn't it?

utilization of ORM

shall we employ some ORM framework to simplify work with DB?

Introduce the option to stop execution of tests after first failed

Develop (data) fixture changes detection

Migrate to python 3.6 for better compatibility

Indetermenistic database update

@guotin or @alexey-vyskubov could you please describe the problem?

Make integration test pass

@matveypashkovskiy

Here's a few questions I have regarding making the integration test pass

Feature: Running all tests if a new method is added to class-file.

What would be the way to approach this? Git diff data is as follows (this is what the integration test now has):

diff --git a/src/car.py b/src/car.py
index 34b567f..f5a4555 100644
--- a/src/car.py
+++ b/src/car.py
@@ -23 +23,5 @@ class Car:
-            self.passengers = self.passengers - 1
\ No newline at end of file
+            self.passengers = self.passengers - 1
+
+    def new_method():
+        i = self.speed + self.seats
+        return i + 8

Should it somehow parse the diff data to find new def declarations with +-sign? This would probably also cover renames too but that's one way I thought of. Or should it somehow use the line numbers inside the @@ symbols to determine that new lines are added to the bottom of the file?

Feature: New test discovery

Current state of `tests_selector`

Running tests_selector with changes in the working directory should get the tests from working directory changes and new tests addition in the working directory. The new test discovery is handled with pytest by comparing pytest --collect-only to tests that exist in the database.

Now, after merging tests_selector and tests_selector_branch scripts, the new tests are only checked when a database update is made, meaning not in the working directory changes but only when checking for committed changes. The tests_selector script checks if there are any working directory changes and runs them and exits if there was any. If there were no changes in the working directory, it continues to check the committed changes...

Problem with current state

Integration test expects the newly added tests to be found in the working directory changes. If this is added, when would be the time to update the database (add the test)? If a test is discovered in the working directory changes but never added to the database before the branch test selection, the test is always found in the working directory changes and the script never progresses to look at the committed changes. So should the new test discovery be only left to the committed changes phase, as it is now or should some other trick be performed to make the integration test pass with the current test code?

Add possibility to pass pytest flags to `tests_selector`

Here is an example how tests are executed in one of F-Secure projects:

pytest --ignore=node_modules \
       --cov=b2b \
       --cov-report=html \
       --cov-report=term \
       --cov-fail-under=90 \

Make it possible to run the tool through pytest

For example coverage plugin for pytest could be invoked like:

pytest -x
	    --cov=tests_selector
	    --cov-report=html
	    --cov-report=term
	    --cov-fail-under=90

can we do the same?

Fix test files mapping as source code files in some situations

@matveypashkovskiy

Just FYI, I noticed a small bug related to updating the database:

Run tests for committed changes -> some subset of tests is selected and database is updated while running them
Condition should check whether the coverage is collected for a test file or a source code file BUT in the updating phase, a full list of test files is missing because I forgot to check the database test files after refactoring the save_data method/function

I will try to fix this next.

RTS strategy for newly added lines

For added lines there could be different approaches:

run all tests related to the file where the line was added
run the tests mapped to the previous or next covered line
run the tests mapped to the previous covered statement in AST

Add posibility to run tests basing on changes in current branch

one of the options is to do that in two steps:

list commits in the current branch comparing to master branch: git cherry -v master
iterate through commits with git show [COMMIT_HASH] -U0

Write a tutorial on how to use the tool with open source project as an example

Migrate from print() to logger.info/warn/error

so we have something like that:

import logging

logger = logging.getLogger()
logger.debug("debug message!")

Test selection fails when execution time is None

INTERNALERROR>   File "/Users/pashma/workspace/b2b-data-collection-pipeline/.venv/lib/python3.6/site-packages/pytest_rts/pytest/update_phase_plugin.py", line 57, in pytest_collection_modifyitems
INTERNALERROR>     items[:] = sorted(selected, key=lambda item: updated_runtimes[item.nodeid])
INTERNALERROR> TypeError: '<' not supported between instances of 'NoneType' and 'float'

Not sure how to reproduce the situation where the execution time of a test is None, but it happens.

The code uses undefined function

$ git grep -B5 file_diff_data_branch
tests_selector/evaluation/start.py-from tests_selector.utils.git import (
tests_selector/evaluation/start.py-    get_git_repo,
tests_selector/evaluation/start.py-    file_diff_data_current,
tests_selector/evaluation/start.py-    get_test_lines_and_update_lines,
tests_selector/evaluation/start.py-    changed_files_branch,
tests_selector/evaluation/start.py:    file_diff_data_branch,
--
tests_selector/evaluation/start.py-
tests_selector/evaluation/start.py-        changed_files = changed_files_branch(PROJECT_FOLDER)
tests_selector/evaluation/start.py-        tests_line_level = set()
tests_selector/evaluation/start.py-        tests_file_level = set()
tests_selector/evaluation/start.py-        for f in changed_files:
tests_selector/evaluation/start.py:            diff = file_diff_data_branch(f, PROJECT_FOLDER)

It's not defined anywhere, as far as I can tell

Refactor git diff data extraction

At the moment to get the list of files and list of changes two separate calls to git diff are used. It is good to perform one call to git to get a structure (a new class GitChanges?) containing information about changed files and changes in files.

Maybe it would be good to add this functionality to pydriller and create a PR there