withsecureopensource/flaky-tests-detection

flaky-tests-detection's Issues

For some reason git is required

In here https://github.com/MarketSquare/robotframework-browser/runs/3731288969?check_suite_focus=true there is error about missing git:

    ---> Running in b15bcfa2d006
  Obtaining file:///
      ERROR: Command errored out with exit status 1:
       command: /usr/local/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/setup.py'"'"'; __file__='"'"'/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-x7dw41k7
           cwd: /
      Complete output (13 lines):
      Traceback (most recent call last):
        File "<string>", line 1, in <module>
        File "/setup.py", line 11, in <module>
          subprocess.check_output("git describe --always".split())
        File "/usr/local/lib/python3.9/subprocess.py", line 424, in check_output
          return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
        File "/usr/local/lib/python3.9/subprocess.py", line 505, in run
          with Popen(*popenargs, **kwargs) as process:
        File "/usr/local/lib/python3.9/subprocess.py", line 951, in __init__
          self._execute_child(args, executable, preexec_fn, close_fds,
        File "/usr/local/lib/python3.9/subprocess.py", line 1821, in _execute_child
          raise child_exception_type(errno_num, err_msg, err_filename)
      FileNotFoundError: [Errno 2] No such file or directory: 'git'
      ----------------------------------------
  WARNING: Discarding file:///. Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
  ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
  The command '/bin/sh -c pip install -e .' returned a non-zero code: 1

export command result to action output

Overview

The flaky command outputs the result of all found flaky tests.
Export it to the action output

Details

As a user of this action, I have a workflow that uses this action after running my test suite.
From there I can read the output of the action Falky.

I would like to use that output from the console and put it somewhere, upload it somewhere, send notifications somewhere when I find flaky tests etc.

It would be useful to print the flaky command output to the console and export it to the github action output system so the calling workflow can take the output and do something with it.

If interested I can pull up a pull request with it.

Fewer digits after the decimal separator in score

Exclude tests with zero score from the output lists

Add GitHub Actions history configuration section to README.md

Only around ~5 builds/artefacts are available. Is it configurable?

Flip rate calculation should not count in skipped tests

In here: https://github.com/F-Secure/fliprate_actions/blob/master/check_flakes.py#L189

                # junitparser has "failure", "skipped" or "error" in result list if any
                if len(testcase.result) == 0:
                    test_status = "pass"
                else:
                    test_status = testcase.result[0]._tag

But I think it should not be considered a flip when test is skipped. Example one can run same test suite for multiple different configurations, like Python versions and in some configurations some test are not possible. Therefore test might be skipped, but not skip is counted towards flip rate calculation.

Proposal to fix it in: #7

Need a better explanation of what the arguments actually do

Better docs and examples of the CSV file would be good. It is not possible to use it without reading the code.

In windows, pip install -e .[dev] does not work

In Windows:

(.venv) C:\workspace\3party\fliprate_actions>pip install -e .[dev]
Obtaining file:///C:/workspace/3party/fliprate_actions
  Preparing metadata (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: 'c:\workspace\3party\fliprate_actions\.venv\scripts\python.exe' -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\workspace\\3party\\fliprate_actions\\setup.py'"'"'; __file__='"'"'C:\\workspace\\3party\\fliprate_actions\\setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base 'C:\Users\aaltta\AppData\Local\Temp\pip-pip-egg-info-gu7jm4h9'
       cwd: C:\workspace\3party\fliprate_actions\
  Complete output (9 lines):
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "C:\workspace\3party\fliprate_actions\setup.py", line 30, in <module>
      long_description=_read_long_description(),
    File "C:\workspace\3party\fliprate_actions\setup.py", line 8, in _read_long_description
      return readme.read()
    File "C:\Python\Python39\lib\encodings\cp1252.py", line 23, in decode
      return codecs.charmap_decode(input,self.errors,decoding_table)[0]
  UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 2780: character maps to <undefined>
  ----------------------------------------
WARNING: Discarding file:///C:/workspace/3party/fliprate_actions. Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

With Python 3.9.7

If there is not flaky test, log to the console that that no flaky tests found

Instead of doing this:

Just log no flaky test found.

Documentation lacks to mention the 'flaky' command (Python)

I tried to use this tool in the python package version. But, reading the (not so comprehensive) documentation, I did not discover what command should be used from CLI. It turned out to be 'flaky'.
The so-called 'full examples' could be extended with this information.

Package in PyPi might be good idea too

Not all CI are in GitHub and creating package to Pypi would be also nice.

Fix also doccer file

This is not correct: https://github.com/F-Secure/flaky-tests-detection/blob/master/Dockerfile#L3

Print only one list of flaky tests

Reference to original Apple paper

Link to the original paper is needed in README.md.

Probably we can link the repo to the paper on https://paperswithcode.com/