boostorg / regression Goto Github PK
View Code? Open in Web Editor NEWRegression testing tools for the Boost C++ Libraries project.
Home Page: https://rawgit.com/boostorg/regression/develop/index.html
Regression testing tools for the Boost C++ Libraries project.
Home Page: https://rawgit.com/boostorg/regression/develop/index.html
Regression Test Reporting Tools ------------------------------- To run Boost's automated regression testing and reporting, and to report the results back to the Boost community, see www.boost.org/development/running_regression_tests.html To build and run Boost's local regression testing and reporting tools, clone the GitHub boostorg/regression repository to your local machine: git clone [email protected]:boostorg/regression.git -- or -- git clone https://github.com/boostorg/regression.git Then view index.html with a web browser for more information, including build instructions -------------------------- Copyright Beman Dawes 2015 Distributed under the Boost Software License, Version 1.0. http://www.boost.org/LICENSE_1_0.txt
Regression tests fail the second time through because Git for Windows removes the files pointed to by links when "git clean -fdx" is run. See git-for-windows/git#607.
To reproduce:
git clone --recursive https://github.com/boostorg/boost.git
cd boost
bootstrap.bat
b2 headers
At this point there is a subdirectory boost with links to library headers.
git clean -fdx
The links are deleted along with the header files they point to.
git submodule foreach git reset --hard
Puts the files back.
We need to add CI tests to the regular regression testing so that we can better keep up with making sure regular user level builds don't break and will work when it's time for a release.
Add options to regression.py called --send-user-config and --send-site-config that will copy the files and put them at the bottom of the comment html file. This way, users that do not have private info in those files can choose to upload them as part of their run configuration.
the current test matrix illustrates a problem. The compile error/warning text is truncated to the extent I can't see the complete error message. This makes it impossible for me to know what the problem is so I can fix it and thus defeats the whole point of the test system. Here is a link to an example of this (valid as I write this issue).
It is depending on the filesystem/convenience.hpp
, which was just removed in this commit.
From time to time misconfigured runners appear on regression matrix. It would be better to perform some kind of a toolset check before actually running the tests, it will prevent regression matrix pollution and will allow runner mainteiners faster recognize that something went wrong.
Rin as:
python run.py --runner=terik23 --tag=develop --toolsets=darwin
Fail with:
# Running Boost.Build tests
Traceback (most recent call last):
File "run.py", line 71, in <module>
runner(root)
File "/Users/terik23/Desktop/boost_test/boost_regression_src/regression.py", line 286, in init
self.main()
File "/Users/terik23/Desktop/boost_test/boost_regression_src/regression.py", line 623, in main
getattr(self,action_m)()
File "/Users/terik23/Desktop/boost_test/boost_regression_src/regression.py", line 588, in command_regression
self.command_test()
File "/Users/terik23/Desktop/boost_test/boost_regression_src/regression.py", line 367, in command_test
self.command_test_boost_build()
File "/Users/terik23/Desktop/boost_test/boost_regression_src/regression.py", line 409, in command_test_boost_build
"boost/bin.v2/libs/any/test/any_test.test"));
OSError: [Errno 2] No such file or directory: '/Users/terik23/Desktop/boost_test/results/boost/bin.v2/libs/any/test/any_test.test'
I checked, and there really is no such directory
ls /Users/terik23/Desktop/boost_test/results/boost/bin.v2/libs
compute integer math stacktrace winapi
config locale multiprecision system
context log regex thread
The test suite seems to run fine, however, when collecting the results the used path is wrong. It is prepended with the physical location of the folder ("E:/Boost\Develop\boost_root\status"), though it probably should only use the virtual path ("\cygdrive\e\Boost\Develop") and maybe only forward slashes.
[...]
*****Warning - can't open output file: E:/Boost\Develop\boost_root\status\cygdrive\e\Boost\Develop\results\xt~boost~fusion~deque~auto~drop_front.test/gcc-5.3.0/debug/ext/boost/fusion/deque/auto\test_log.xml
*****Warning - can't open output file: E:/Boost\Develop\boost_root\status\cygdrive\e\Boost\Develop\results\xt~boost~fusion~deque~auto~drop_while.test/gcc-5.3.0/debug/ext/boost/fusion/deque/auto\test_log.xml
*****Warning - can't open output file: E:/Boost\Develop\boost_root\status\cygdrive\e\Boost\Develop\results\xt~boost~fusion~deque~auto~group.test/gcc-5.3.0/debug/ext/boost/fusion/deque/auto\test_log.xml
# Collecting test logs into "/cygdrive/e/Boost/Develop/results/timber-cygwin.xml"...
# Reading comments file "/cygdrive/e/Boost/Develop/full-comment.html"...
# Collecting test logs ...
# Walking directory "/cygdrive/e/Boost/Develop/results" ...
# Processing test log "/cygdrive/e/Boost/Develop/results/boost-build-gcc-5.3.0/test_log.xml"
# Done writing "/cygdrive/e/Boost/Develop/results/timber-cygwin.xml".
# Compressing "/cygdrive/e/Boost/Develop/results/timber-cygwin.xml"...
# Done writing "/cygdrive/e/Boost/Develop/results/timber-cygwin.zip".
# Uploading log archive "/cygdrive/e/Boost/Develop/results/timber-cygwin.zip" to develop
Currently the test is marked as having warnings if there is warning
word in b2 compiling pass output (not only compiler output). I found at least 5 tests across boost libraries that have warning
in their name and marked because of this.
There are times when a test indicates failure but the only explanation is a link to a successful compile step. For example http://www.boost.org/development/tests/develop/developer/output/teeks99-08f-win2012R2-64on64-boost-bin-v2-libs-serialization-test-test_array_binary_archive-test-msvc-14-0-dbg-adrs-mdl-64-thrd-mlt.html
Most likely this is log parsing bug.
I'm sort of mystified that there is no master branch for this repo. I switched from develop to master and everything disappeared. Took me quite a while to figure out what happened. Isn't it required that anything in any of the git repos have a master and develop branch in order for the release process to work correctly?
Robert Ramey
Python 2's end of life is coming in just over a year. Can we make the run.py script (and its dependencies, regression.py, etc) compatible with both?
I'm guessing most of the updates will be adding parens around print statements. We should also pay attention to any output that we check from making subprocess calls (string vs. bytes).
I tried to use library_status
but it hangs with 100% CPU. Only after jumping with a debugger I have figured out that it does not support passing arguments like this --boost-root=../../../boost
. It loops and does not consume any input.
Make sure the library_status tool works as expected and update its docs for new location and procedure.
In particular, "compile-" and "-compile" can show up in source file names.
This recently appeared in graph and is breaking the regression tests.
Some of the patterns also appear to be out-dated (perhaps left over from Boost.Build 1).
Following the online docs I downloaded and executed the run.py
script using
python run.py --runner=Stefan --toolsets=gcc-6.2.1 --tag=develop
which eventually resulted in
# Executing GIT command: /home/stefan/projects/boost_root> git submodule foreach "git submodule deinit ."
Entering 'libs/accumulators'
error: pathspec '.' did not match any file(s) known to git.
Stopping at 'libs/accumulators'; script returned non-zero status.
# git_source_checkout failed with message "GIT command "git submodule foreach "git submodule deinit ."" failed with code 256"
# Retrying (5 more attempts).
...
# Retrying (1 more attempts).
# Executing GIT command: /home/stefan/projects/boost_root> git submodule foreach "git submodule deinit ."
Entering 'libs/accumulators'
error: pathspec '.' did not match any file(s) known to git.
Stopping at 'libs/accumulators'; script returned non-zero status.
# git_source_checkout failed with message "GIT command "git submodule foreach "git submodule deinit ."" failed with code 256"
# Giving up.
Traceback (most recent call last):
File "run.py", line 71, in <module>
runner(root)
File "/home/stefan/projects/boost_regression_src/regression.py", line 287, in __init__
self.main()
File "/home/stefan/projects/boost_regression_src/regression.py", line 624, in main
getattr(self,action_m)()
File "/home/stefan/projects/boost_regression_src/regression.py", line 579, in command_regression
self.command_get_source()
File "/home/stefan/projects/boost_regression_src/regression.py", line 331, in command_get_source
self.retry(self.git_source_checkout)
File "/home/stefan/projects/boost_regression_src/regression.py", line 663, in retry
return f()
File "/home/stefan/projects/boost_regression_src/regression.py", line 876, in git_source_checkout
self.git_checkout(git_info['boost'], self.git_branch(), clean)
File "/home/stefan/projects/boost_regression_src/regression.py", line 847, in git_checkout
self.git_command( 'submodule foreach', 'git submodule deinit .')
File "/home/stefan/projects/boost_regression_src/regression.py", line 835, in git_command
raise Exception( 'GIT command "%s" failed with code %d' % (git_cli, rc) )
Exception: GIT command "git submodule foreach "git submodule deinit ."" failed with code 256
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.