boostorg/regression

Regression testing tools for the Boost C++ Libraries project.

regression's Issues

Tests fail to run the second time through on Windows.

Regression tests fail the second time through because Git for Windows removes the files pointed to by links when "git clean -fdx" is run. See git-for-windows/git#607.

To reproduce:
git clone --recursive https://github.com/boostorg/boost.git
cd boost
bootstrap.bat
b2 headers
At this point there is a subdirectory boost with links to library headers.

git clean -fdx
The links are deleted along with the header files they point to.

git submodule foreach git reset --hard
Puts the files back.

Add CI to regression testing results.

We need to add CI tests to the regular regression testing so that we can better keep up with making sure regular user level builds don't break and will work when it's time for a release.

Optionally add user-config.jam and site-config.jam to the html

Add options to regression.py called --send-user-config and --send-site-config that will copy the files and put them at the bottom of the comment html file. This way, users that do not have private info in those files can choose to upload them as part of their run configuration.

Test matrix truncates error output

the current test matrix illustrates a problem. The compile error/warning text is truncated to the extent I can't see the complete error message. This makes it impossible for me to know what the problem is so I can fix it and thus defeats the whole point of the test system. Here is a link to an example of this (valid as I write this issue).

http://www.boost.org/development/tests/develop/developer/output/teeks99-08f-win2012R2-64on64-boost-bin-v2-libs-serialization-test-test_array_binary_archive-test-msvc-14-0-dbg-adrs-mdl-64-thrd-mlt.htm

Can't build process_jam_log on develop

It is depending on the filesystem/convenience.hpp, which was just removed in this commit.

Document master results approval process.

Check toolset configuration before running tests

From time to time misconfigured runners appear on regression matrix. It would be better to perform some kind of a toolset check before actually running the tests, it will prevent regression matrix pollution and will allow runner mainteiners faster recognize that something went wrong.

Fail running test on macOS

System information:

• macOS v10.13.3
• Python 2.7.10
• git version 2.16.2
• Apple LLVM version 9.0.0 (clang-900.0.39.2)
  Target: x86_64-apple-darwin17.4.0
  Thread model: posix

Rin as:

python run.py --runner=terik23 --tag=develop --toolsets=darwin

Fail with:

# Running Boost.Build tests
Traceback (most recent call last):
  File "run.py", line 71, in <module>
    runner(root)
  File "/Users/terik23/Desktop/boost_test/boost_regression_src/regression.py", line 286, in init
    self.main()
  File "/Users/terik23/Desktop/boost_test/boost_regression_src/regression.py", line 623, in main
    getattr(self,action_m)()
  File "/Users/terik23/Desktop/boost_test/boost_regression_src/regression.py", line 588, in command_regression
    self.command_test()
  File "/Users/terik23/Desktop/boost_test/boost_regression_src/regression.py", line 367, in command_test
    self.command_test_boost_build()
  File "/Users/terik23/Desktop/boost_test/boost_regression_src/regression.py", line 409, in command_test_boost_build
    "boost/bin.v2/libs/any/test/any_test.test"));
OSError: [Errno 2] No such file or directory: '/Users/terik23/Desktop/boost_test/results/boost/bin.v2/libs/any/test/any_test.test'

I checked, and there really is no such directory

 ls /Users/terik23/Desktop/boost_test/results/boost/bin.v2/libs
compute  integer  math  stacktrace winapi
config  locale  multiprecision system
context  log  regex  thread

Collecting results fails with Cygwin

The test suite seems to run fine, however, when collecting the results the used path is wrong. It is prepended with the physical location of the folder ("E:/Boost\Develop\boost_root\status"), though it probably should only use the virtual path ("\cygdrive\e\Boost\Develop") and maybe only forward slashes.

[...]
*****Warning - can't open output file: E:/Boost\Develop\boost_root\status\cygdrive\e\Boost\Develop\results\xt~boost~fusion~deque~auto~drop_front.test/gcc-5.3.0/debug/ext/boost/fusion/deque/auto\test_log.xml
*****Warning - can't open output file: E:/Boost\Develop\boost_root\status\cygdrive\e\Boost\Develop\results\xt~boost~fusion~deque~auto~drop_while.test/gcc-5.3.0/debug/ext/boost/fusion/deque/auto\test_log.xml
*****Warning - can't open output file: E:/Boost\Develop\boost_root\status\cygdrive\e\Boost\Develop\results\xt~boost~fusion~deque~auto~group.test/gcc-5.3.0/debug/ext/boost/fusion/deque/auto\test_log.xml
# Collecting test logs into "/cygdrive/e/Boost/Develop/results/timber-cygwin.xml"...
# Reading comments file "/cygdrive/e/Boost/Develop/full-comment.html"...
# Collecting test logs ...
# Walking directory "/cygdrive/e/Boost/Develop/results" ...
#     Processing test log "/cygdrive/e/Boost/Develop/results/boost-build-gcc-5.3.0/test_log.xml"
# Done writing "/cygdrive/e/Boost/Develop/results/timber-cygwin.xml".
# Compressing "/cygdrive/e/Boost/Develop/results/timber-cygwin.xml"...
# Done writing "/cygdrive/e/Boost/Develop/results/timber-cygwin.zip".
# Uploading log archive "/cygdrive/e/Boost/Develop/results/timber-cygwin.zip" to develop

too broad warning detection pattern

regression/reports/src/add_expected_results.cpp

Line 252 in 4d4c503

if ( find_regex(val, "warning") ) {

Currently the test is marked as having warnings if there is warning word in b2 compiling pass output (not only compiler output). I found at least 5 tests across boost libraries that have warning in their name and marked because of this.

Tests are marked fail with only a link to a successful compile.

There are times when a test indicates failure but the only explanation is a link to a successful compile step. For example http://www.boost.org/development/tests/develop/developer/output/teeks99-08f-win2012R2-64on64-boost-bin-v2-libs-serialization-test-test_array_binary_archive-test-msvc-14-0-dbg-adrs-mdl-64-thrd-mlt.html

Most likely this is log parsing bug.

why is there no master branch for this repo?

I'm sort of mystified that there is no master branch for this repo. I switched from develop to master and everything disappeared. Took me quite a while to figure out what happened. Isn't it required that anything in any of the git repos have a master and develop branch in order for the release process to work correctly?

Robert Ramey

Support python 3

Python 2's end of life is coming in just over a year. Can we make the run.py script (and its dependencies, regression.py, etc) compatible with both?

I'm guessing most of the updates will be adding parens around print statements. We should also pay attention to any output that we check from making subprocess calls (string vs. bytes).

library_status: Infinity loop in argument parsing

I tried to use library_status but it hangs with 100% CPU. Only after jumping with a debugger I have figured out that it does not support passing arguments like this --boost-root=../../../boost. It loops and does not consume any input.

Make library_status tool operational again.

Make sure the library_status tool works as expected and update its docs for new location and procedure.

The patterns for matching action names are too broad.

In particular, "compile-" and "-compile" can show up in source file names.
This recently appeared in graph and is breaking the regression tests.
Some of the patterns also appear to be out-dated (perhaps left over from Boost.Build 1).

Failure running `run.py`

Following the online docs I downloaded and executed the run.py script using

python run.py --runner=Stefan --toolsets=gcc-6.2.1 --tag=develop

which eventually resulted in

# Executing GIT command: /home/stefan/projects/boost_root> git submodule foreach "git submodule deinit ."
Entering 'libs/accumulators'
error: pathspec '.' did not match any file(s) known to git.
Stopping at 'libs/accumulators'; script returned non-zero status.
# git_source_checkout failed with message "GIT command "git submodule foreach "git submodule deinit ."" failed with code 256"
# Retrying (5 more attempts).
...
# Retrying (1 more attempts).
# Executing GIT command: /home/stefan/projects/boost_root> git submodule foreach "git submodule deinit ."
Entering 'libs/accumulators'
error: pathspec '.' did not match any file(s) known to git.
Stopping at 'libs/accumulators'; script returned non-zero status.
# git_source_checkout failed with message "GIT command "git submodule foreach "git submodule deinit ."" failed with code 256"
# Giving up.
Traceback (most recent call last):
  File "run.py", line 71, in <module>
    runner(root)
  File "/home/stefan/projects/boost_regression_src/regression.py", line 287, in __init__
    self.main()
  File "/home/stefan/projects/boost_regression_src/regression.py", line 624, in main
    getattr(self,action_m)()
  File "/home/stefan/projects/boost_regression_src/regression.py", line 579, in command_regression
    self.command_get_source()
  File "/home/stefan/projects/boost_regression_src/regression.py", line 331, in command_get_source
    self.retry(self.git_source_checkout)
  File "/home/stefan/projects/boost_regression_src/regression.py", line 663, in retry
    return f()
  File "/home/stefan/projects/boost_regression_src/regression.py", line 876, in git_source_checkout
    self.git_checkout(git_info['boost'], self.git_branch(), clean)
  File "/home/stefan/projects/boost_regression_src/regression.py", line 847, in git_checkout
    self.git_command( 'submodule foreach', 'git submodule deinit .')
  File "/home/stefan/projects/boost_regression_src/regression.py", line 835, in git_command
    raise Exception( 'GIT command "%s" failed with code %d' % (git_cli, rc) )
Exception: GIT command "git submodule foreach "git submodule deinit ."" failed with code 256