Documents test-results inside Sphinx
Home Page: https://sphinx-test-reports.readthedocs.io/en/latest/
License: MIT License
If need-parts are used, tr_link does not check, if a given id (containing a string like 'need_id.part_id`) fits the target option.
Used like:
.. test-file:: My Test Data
:file: teen_car/test_run_1.xml
:id: TESTFILE_1
:auto_suites:
:auto_cases:
:links: [[tr_link('case_name', 'id_complete')]]Where [[tr_link('case_name', 'id_complete')]] is the part which should make it work.
Using id instead of id_complete works.
Hi. First of all thank you for the sphinx-needs and sphinx-test-reports extensions. They are great and helps as to keep everything as code.
I discovered a strange behavior with pytest and sphinx-test-reports when a fixture raises an error the test cases using that fixture are still passing.
Here is a small example, which I hope is reproducible:
# test_example.py
import pytest
@pytest.fixture
def fixture_foo():
raise Exception('ERROR on fixture')
def test_foo(fixture_foo):
print("I'm OK")The above test file is called using the following command:
pytest test_example.py --junitxml=junit.xmlThen in an RST file I simply call the test-report directive:
FOO
=====
.. test-report:: My Report
:id: FOO
:file: /workspaces/tmp/junit.xmlIt's important to mention that pytest doesn't contain any additional configuration nor sphinx. Also the test suite detects that there is 1 error.
HTML Results
Test case is shown as passed
In the imported data, there is 1 error and one test case which is passing.
pytest 6.2.5
Sphinx 4.4.0
sphinx-data-viewer 0.1.2
sphinx-test-reports 0.3.6
sphinxcontrib-jsmath 1.0.1
sphinxcontrib-needs 0.7.5
sphinxcontrib-plantuml 0.22
After I increase Sphinx, Test-Reports and Sphinx-Needs versions I got an error message after I built my project:
Sphinx parallel build error:
NotImplementedError: <class 'sphinx.writers.html5.HTML5Translator'> departing unknown node type: NeedIncoming
or
Sphinx parallel build error:
NotImplementedError: <class 'sphinx.writers.html5.HTML5Translator'> departing unknown node type: NeedOutgoing
Versions:
I played with the versions and with my implementation and I realized that problem is coming up, when increasing Sphinx-Needs version above 1.0.2. When I used <=1.0.2 everything works fine.
I have connection between Test Results (Test Cases, Test Suites from Test-Reports extension), and other own created need types using extra options. Between Test Cases (or Test Suites) and Test Specifications I am using the :links: extra option from Test-Results extension.
When I want to link from a Test Case or Test Suite to a Test Specification I got this error message, without linking the project works.
Example implementations of Test Suite and Test Case in reStructuredText:
.. test-suite:: No title
:id: TESTSUITE_1
:file: ./source/tests/test_res_addproduct.xml
:suite: test_res_addproduct-123
:collapse: true
:links: TESTSPEC_123.. test-case:: No title
:id: TESTCASE_1
:file: ./source/tests/test_res_addproduct.xml
:suite: test_res_addproduct-123
:case: test_button
:collapse: true
:links: TESTSPEC_123I got a feedback from Useblocks side to remove Sphinx parallel build while this bug has been fixed, after I removed I got the error message below:
Exception occurred:
File "/home/dockeruser/.local/lib/python3.10/site-packages/docutils/nodes.py", line 2056, in unknown_departure
raise NotImplementedError(
NotImplementedError: <class 'sphinx.writers.html5.HTML5Translator'> departing unknown node type: NeedOutgoing
Important finding:
If I put the "target" need in the file where the Test Case or Test Suite is, the build process is successful.
Hi there,
I found an issue about the rendered documentation through the make docs-html target on the current master commit b28c01e.
This issue is located in the topic Test framework related examples for the CasperJs one : only one testsuite is captured within test report, whereas all test suites are effectively described on the website (see attached screenshot, left is website, right is what my local build produced).
After digging a bit the codebase, and watch how casperjs.xml is built, I pointed out the following line which seems to be the one leading to this behaviour, within junitparser.py:26 :
if self.junit_xml_object.tag == "testsuites":
self.junit_xml_object = self.junit_xml_object.testsuite
So two questions arise :
testsuites XML tag is parsed differently ?The following quick workaround fixes this issue, but I'm not 100% sure of any side effects :
+++ aggron/junitparser.py 2022-07-11 14:11:20.080362490 +0200
@@ -131,8 +131,9 @@
# main flow starts here
junit_dict = []
- complete_testsuite = parse_testsuite(self.junit_xml_object)
- junit_dict.append(complete_testsuite)
+ for o in self.junit_xml_object:
+ complete_testsuite = parse_testsuite(o)
+ junit_dict.append(complete_testsuite)
return junit_dict
Thanks for your help 🙂
It does not appear that adding a custom test_report template is supported in the TestReportDirective class:
It would be nice if this could be pulled out as a configurable setting.
Hi!
I try to use check_linked_values dynamic function to resolve the results from importet junit file to my previously defined test cases.
Attached files contains the junit xml (one tests with status fail) and the index.rst with the report definition.
Expected: in the needstable one row gets red, other rows have no color due to check_linked_values returning only failed for ONE case.
But the result is that EVERY row is marked failed. Does check_linked_values perform a deep search of all linked values (and therefore for the complete file test file itself?). Or what is wrong in my usage of dynamic function?
Content of report index.rst
Checked linked values
=====================
.. test:: usage_test
:id: TEST_0001
.. test:: success_test
:id: TEST_0002
.. test:: fail_test
:id: TEST_0003
.. test:: fail_test_output
:id: TEST_0004
.. needtable::
:style: table
:filter: id.startswith('TEST_') and any('CTESTFILE_1' in s for s in links_back)
:columns: id, title, links_back as "Tests"
:style_row: [[check_linked_values('tr_failure', 'result', 'failure', filter_string='title==case')]]
.. test-file:: My CTest Data
:file: ../utils/ctest_result_filter.xml
:id: CTESTFILE_1
:auto_suites:
:auto_cases:
:links: [[tr_link('classname', 'title')]]
The content of the xml file is:
<?xml version="1.0" encoding="UTF-8"?>
<testsuite name="Linux-c++"
tests="4"
failures="1"
disabled="0"
skipped="0"
hostname=""
time="0"
timestamp="2022-12-08T15:15:07"
>
<testcase name="usage_test" classname="usage_test" time="0.000609293" status="run">
<system-out>usage: test <argument>
If <argument> is 0, print SUCCESS. Otherwise print FAIL.
</system-out>
</testcase>
<testcase name="success_test" classname="success_test" time="0.000605907" status="run">
<system-out>SUCCESS
</system-out>
</testcase>
<testcase name="fail_test" classname="fail_test" time="0.000599379" status="run">
<system-out>FAIL
</system-out>
</testcase>
<testcase name="fail_test_output" classname="fail_test_output" time="0.000576548" status="fail">
<failure message=""/>
<system-out>FAIL
</system-out>
</testcase>
</testsuite>
Attached files:
doc_test_ctest_file_result_filter.zip
Add support for sphinx parallel read and write
tr_rootdir being a Path obj causes esbonio v0.92.1 (pre-release) to crash
app.add_config_value("tr_rootdir", app.confdir, "html")since app.confdir is a Path, it causes esbonio to raise an exception as it cant convert the configuration to a json
def as_json(data: Any) -> str:
if dataclasses.is_dataclass(data):
data = dataclasses.asdict(data)
return json.dumps(data, default=_serialize_message)one way to fix this issue would be:
app.add_config_value("tr_rootdir", str(app.confdir), "html")I have sphinx-needs configured with a very specific set of allowed tags. But sphinx-test-reports always includes the report ID as a tag, and so each individual report I make fails with invalid tags:
Sphinx error:
Tag REPORT2 of need id REPORT2 is not allowed by config value 'needs_tags'.
Input causing this looks like:
.. test-report:: Unit Test Results
:id: REPORT
:file: ../proj/unit-test-results/proj-unit-test-results.xml
.. test-report:: Cypress Tests
:id: REPORT2
:tags: REPORT
:file: ../frontend/proj/results/proj-e2e-test-output-5b4d6459a038b4cd8595b7b99702.xml
Seems that this is coming from here:
where it hard-codes adding the test ID as a tag. Should this be removed? What purpose is this serving? Can I make a PR to remove that feature, or should it be a new setting to toggle it off?
Writing the test_report_template for the report works on linux but seems broken on windows for Latin1 encodings.
My test_report_template is written in German with German Umlaute and the file encoding is utf-8.
In the file sphinxcontrib/test_reports/directives/test_report.py this is opened without encoding information and i assume it opens it with system encoding (utf8 on linux, latin1 or window cpxxxx). This breaks the handling on both platforms.
Best would be to open the file ALWAYS with utf8 encoding or make in a configuration value.
There shall be one new directive: test-env
.. test-env:: env_1/env-report.json # file name
or
.. test-env:: py27 # env name
One new configuration option must be added: test-env-files
test-env-files = ["../.tox/tox-report.json"]
If a file name is given for "test-env", the complete file shall be used as input data.
If an env-name is given, the name shall be searched inside given files by "test-env-files" and only found data shall be used as input data.
If file name or env name do not exist, an error note shall be printed at the current place of the directive.
Another option "test-env-data" must be added, which stores the parametes, which shall be used for output:
test-env-data = ["name", "host", "installed_packages"]
Supported parameters are:
This configuration can be overwritten by each "test-env" directive by using the option "data":
.. test-env: my_env
:data: name, host, test
"data" is a comma separated list, which support "," and ";" as delimiter.
If a given parameter is not available, a warning shall be printed on command line.
If :raw: is given as option for test-env, the shall be only a single code-block, which contains the complete env data.
The data shall be printed as follows:
Each :test-env: output is framed by a container with class=test-env
Each parameter shall be printed inside a "parameter" container which class=test-env-parameter and class=param_PARAM_NAME (e.g. param_host)
Each string/number parameter shall be printed by using sphinx field-names
:param_name: param_value
dics/lists shall be printed inside a code-block without any special style.
The param_name shall be printed above the code-block (in bold).
Hello,
I am looking for a way to generate a sphinx test-plan from docstrings included in my pytest test-suite.
I came across this stackoverflow question, where danwos answer suggests an approach based on the sphinx-test-reports plugin.
I tried to use the test-case directive but I got following error:
File "/local/home/sabria/.local/lib/python3.8/site-packages/sphinxcontrib/test_reports/directives/test_common.py", line 98, in prepare_basic_options
self.test_file_given = self.test_file[:]
TypeError: 'NoneType' object is not subscriptable
It looks like junit xml test reports are mandatory to generate documentation when using this directive (i didn't provide the file option).
I didn't encounter this problem when I tried the test directive (presented in danwos's answer).
def my_test():
"""
.. test:: My test case
:id: TEST_001
:status: in progress
:author: me
This test case checks for **awesome** stuff.
"""
a = 2
b = 5
# ToDo: chek if a+b = 7
The issue is that I was not able to find in the project's documentation information about this one. I would like to have something that not depends on the junit xml file.
Is the test directive still supported by your plugin? Where can I find examples?
Thank you
If a list of values is given for :data: option of ```.. test-env::``, then the ordering of this list gets lost.
Instead all requested values get printed in alphabetical order.
Example:
.. test-env:: xy.json
:data: version, name, python
Output order is: name, python, version.
It would be good, if the ordering of :data: can be kept, so that the author can chose how the final result shall look like.
Previously, xUnit reports generated by robot framework were handled correctly by sphinx-test-reports.
With the latest version Robot Framework 5.0, however, there was a change in the layout of the XML report file.
See discussion here: robotframework/robotframework#2982
The reports now contain nested elements like this:
<testsuite>
<testsuite>
<testcase>
and so on
When importing such a report with test-file/test-report directive, I get the error message:
AttributeError: no such child: testcase
It would be great if sphinx-test-reports would support nested testsuites to continue compatibility with robot framework.
There shall be more configuration options:
We are currently using this extension (locked in version 0.3.6 until issue 40 gets fixed) to link GTEST results.
When using GTEST and typed tests, it is very common to have xml files with many suites (e.g., 60 suites or more, depending on the number of test cases and tested types in the xml). For this use case, we have experienced sphinx-build compilation errors about repeated IDs.
The titles of the suites are different, but somehow they get the same hash, and hence the problem.
Here is an example xml file which is generating this problem:
<?xml version="1.0" encoding="UTF-8"?>
<testsuites tests="63" failures="0" disabled="0" errors="0" timestamp="2022-10-21T14:47:39" time="0.002" name="AllTests">
<testsuite name="vfc_hash" tests="7" failures="0" disabled="0" errors="0" time="0">
<testcase name="TEST_UNIT_VFC_hash_testHashBoolAndEnum" status="run" time="0" classname="vfc_hash" />
<testcase name="TEST_UNIT_VFC_hash_testSerialHash" status="run" time="0" classname="vfc_hash" />
<testcase name="TEST_UNIT_VFC_hash_testSerialHashComplex" status="run" time="0" classname="vfc_hash" />
<testcase name="TEST_UNIT_VFC_hash_testSerialHashNested" status="run" time="0" classname="vfc_hash" />
<testcase name="TEST_UNIT_VFC_hash_testVfcComplexTypesTCArray" status="run" time="0" classname="vfc_hash" />
<testcase name="TEST_UNIT_VFC_hash_testVfcComplexTypesTSIUnit" status="run" time="0" classname="vfc_hash" />
<testcase name="TEST_UNIT_VFC_hash_testVfcComplexTypesRect" status="run" time="0" classname="vfc_hash" />
</testsuite>
<testsuite name="vfc_THash/0" tests="1" failures="0" disabled="0" errors="0" time="0">
<testcase name="TEST_UNIT_VFC_hash_testHash" type_param="vfc_cppunit::THash<int, 100l, 2l, 1000l, 3l, 0l, 0l, -100l, 2l, -1l, 31l>" status="run" time="0" classname="vfc_THash/0" />
</testsuite>
<testsuite name="vfc_THash/1" tests="1" failures="0" disabled="0" errors="0" time="0">
<testcase name="TEST_UNIT_VFC_hash_testHash" type_param="vfc_cppunit::THash<short, 100l, 2l, 1000l, 3l, 0l, 0l, -100l, 2l, -1l, 31l>" status="run" time="0" classname="vfc_THash/1" />
</testsuite>
<testsuite name="vfc_THash/2" tests="1" failures="0" disabled="0" errors="0" time="0">
<testcase name="TEST_UNIT_VFC_hash_testHash" type_param="vfc_cppunit::THash<unsigned short, 100l, 2l, 1000l, 3l, 0l, 0l, 100l, 2l, 1l, 31l>" status="run" time="0" classname="vfc_THash/2" />
</testsuite>
<testsuite name="vfc_THash/3" tests="1" failures="0" disabled="0" errors="0" time="0">
<testcase name="TEST_UNIT_VFC_hash_testHash" type_param="vfc_cppunit::THash<unsigned int, 100l, 2l, 1000l, 3l, 0l, 0l, 100l, 2l, 1l, 31l>" status="run" time="0" classname="vfc_THash/3" />
</testsuite>
<testsuite name="vfc_THash/4" tests="1" failures="0" disabled="0" errors="0" time="0">
<testcase name="TEST_UNIT_VFC_hash_testHash" type_param="vfc_cppunit::THash<long, 100l, 2l, 1000l, 3l, 0l, 0l, -100l, 2l, -1l, 31l>" status="run" time="0" classname="vfc_THash/4" />
</testsuite>
<testsuite name="vfc_hash_Primitives/0" tests="1" failures="0" disabled="0" errors="0" time="0">
<testcase name="TEST_UNIT_VFC_hash_testHashPrimitives" type_param="vfc_cppunit::hash_Primitives<int, 85l, 2l, 1000l, 3l, 0l, 0l, 100l, 2l, 1l, 31l>" status="run" time="0" classname="vfc_hash_Primitives/0" />
</testsuite>
<testsuite name="vfc_hash_Primitives/1" tests="1" failures="0" disabled="0" errors="0" time="0">
<testcase name="TEST_UNIT_VFC_hash_testHashPrimitives" type_param="vfc_cppunit::hash_Primitives<short, 85l, 2l, 1000l, 3l, 0l, 0l, 100l, 2l, 1l, 31l>" status="run" time="0" classname="vfc_hash_Primitives/1" />
</testsuite>
<testsuite name="vfc_hash_Primitives/2" tests="1" failures="0" disabled="0" errors="0" time="0">
<testcase name="TEST_UNIT_VFC_hash_testHashPrimitives" type_param="vfc_cppunit::hash_Primitives<unsigned short, 85l, 2l, 1000l, 3l, 0l, 0l, 100l, 2l, 1l, 31l>" status="run" time="0" classname="vfc_hash_Primitives/2" />
</testsuite>
<testsuite name="vfc_hash_Primitives/3" tests="1" failures="0" disabled="0" errors="0" time="0">
<testcase name="TEST_UNIT_VFC_hash_testHashPrimitives" type_param="vfc_cppunit::hash_Primitives<unsigned int, 85l, 2l, 1000l, 3l, 0l, 0l, 100l, 2l, 1l, 31l>" status="run" time="0" classname="vfc_hash_Primitives/3" />
</testsuite>
<testsuite name="vfc_hash_Primitives/4" tests="1" failures="0" disabled="0" errors="0" time="0">
<testcase name="TEST_UNIT_VFC_hash_testHashPrimitives" type_param="vfc_cppunit::hash_Primitives<long, 85l, 2l, 1000l, 3l, 0l, 0l, 100l, 2l, 1l, 31l>" status="run" time="0" classname="vfc_hash_Primitives/4" />
</testsuite>
<testsuite name="vfc_hash_Primitives/5" tests="1" failures="0" disabled="0" errors="0" time="0">
<testcase name="TEST_UNIT_VFC_hash_testHashPrimitives" type_param="vfc_cppunit::hash_Primitives<float, 85l, 2l, 1000l, 3l, 0l, 0l, 100l, 2l, 1l, 31l>" status="run" time="0" classname="vfc_hash_Primitives/5" />
</testsuite>
<testsuite name="vfc_hash_Tuple/0" tests="1" failures="0" disabled="0" errors="0" time="0.001">
<testcase name="TEST_UNIT_VFC_hash_testVfcComplexTypesTuple" type_param="vfc_cppunit::hash_Tuple<int, 5921370, 10855845, 5921370, 10855845>" status="run" time="0.001" classname="vfc_hash_Tuple/0" />
</testsuite>
<testsuite name="vfc_hash_Tuple/1" tests="1" failures="0" disabled="0" errors="0" time="0">
<testcase name="TEST_UNIT_VFC_hash_testVfcComplexTypesTuple" type_param="vfc_cppunit::hash_Tuple<unsigned int, 5921370u, 10855845u, 5921370u, 10855845u>" status="run" time="0" classname="vfc_hash_Tuple/1" />
</testsuite>
<testsuite name="vfc_hash_Tuple/2" tests="1" failures="0" disabled="0" errors="0" time="0">
<testcase name="TEST_UNIT_VFC_hash_testVfcComplexTypesTuple" type_param="vfc_cppunit::hash_Tuple<long, 5921370l, 10855845l, 5921370l, 10855845l>" status="run" time="0" classname="vfc_hash_Tuple/2" />
</testsuite>
<testsuite name="vfc_hash_TFixed/0" tests="2" failures="0" disabled="0" errors="0" time="0">
<testcase name="TEST_UNIT_VFC_hash_testVfcComplexTypesTFixedVector" type_param="vfc_cppunit::hash_TFixed<int, 5ul, -10l>" status="run" time="0" classname="vfc_hash_TFixed/0" />
<testcase name="TEST_UNIT_VFC_hash_testVfcComplexTypesTFixedList" type_param="vfc_cppunit::hash_TFixed<int, 5ul, -10l>" status="run" time="0" classname="vfc_hash_TFixed/0" />
</testsuite>
<testsuite name="vfc_hash_TFixed/1" tests="2" failures="0" disabled="0" errors="0" time="0">
<testcase name="TEST_UNIT_VFC_hash_testVfcComplexTypesTFixedVector" type_param="vfc_cppunit::hash_TFixed<unsigned int, 15ul, 200l>" status="run" time="0" classname="vfc_hash_TFixed/1" />
<testcase name="TEST_UNIT_VFC_hash_testVfcComplexTypesTFixedList" type_param="vfc_cppunit::hash_TFixed<unsigned int, 15ul, 200l>" status="run" time="0" classname="vfc_hash_TFixed/1" />
</testsuite>
<testsuite name="vfc_hash_TFixed/2" tests="2" failures="0" disabled="0" errors="0" time="0">
<testcase name="TEST_UNIT_VFC_hash_testVfcComplexTypesTFixedVector" type_param="vfc_cppunit::hash_TFixed<long, 5ul, -300l>" status="run" time="0" classname="vfc_hash_TFixed/2" />
<testcase name="TEST_UNIT_VFC_hash_testVfcComplexTypesTFixedList" type_param="vfc_cppunit::hash_TFixed<long, 5ul, -300l>" status="run" time="0" classname="vfc_hash_TFixed/2" />
</testsuite>
<testsuite name="vfc_hash_TFixed/3" tests="2" failures="0" disabled="0" errors="0" time="0">
<testcase name="TEST_UNIT_VFC_hash_testVfcComplexTypesTFixedVector" type_param="vfc_cppunit::hash_TFixed<float, 25ul, 50l>" status="run" time="0" classname="vfc_hash_TFixed/3" />
<testcase name="TEST_UNIT_VFC_hash_testVfcComplexTypesTFixedList" type_param="vfc_cppunit::hash_TFixed<float, 25ul, 50l>" status="run" time="0" classname="vfc_hash_TFixed/3" />
</testsuite>
<testsuite name="vfc_hash_TFixedCircularBuffer/0" tests="1" failures="0" disabled="0" errors="0" time="0">
<testcase name="TEST_UNIT_VFC_hash_testVfcComplexTypesTFixedCircularBuffer" type_param="vfc_cppunit::hash_TFixedCircularBuffer<int, 100l, 2l, 1000l, 3l, 0l, 0l, -100l, 2l, -1l, 31l>" status="run" time="0" classname="vfc_hash_TFixedCircularBuffer/0" />
</testsuite>
<testsuite name="vfc_hash_TFixedCircularBuffer/1" tests="1" failures="0" disabled="0" errors="0" time="0">
<testcase name="TEST_UNIT_VFC_hash_testVfcComplexTypesTFixedCircularBuffer" type_param="vfc_cppunit::hash_TFixedCircularBuffer<short, 100l, 2l, 1000l, 3l, 0l, 0l, -100l, 2l, -1l, 31l>" status="run" time="0" classname="vfc_hash_TFixedCircularBuffer/1" />
</testsuite>
<testsuite name="vfc_hash_TFixedCircularBuffer/2" tests="1" failures="0" disabled="0" errors="0" time="0">
<testcase name="TEST_UNIT_VFC_hash_testVfcComplexTypesTFixedCircularBuffer" type_param="vfc_cppunit::hash_TFixedCircularBuffer<unsigned short, 100l, 2l, 1000l, 3l, 0l, 0l, 100l, 2l, 1l, 31l>" status="run" time="0" classname="vfc_hash_TFixedCircularBuffer/2" />
</testsuite>
<testsuite name="vfc_hash_TFixedCircularBuffer/3" tests="1" failures="0" disabled="0" errors="0" time="0">
<testcase name="TEST_UNIT_VFC_hash_testVfcComplexTypesTFixedCircularBuffer" type_param="vfc_cppunit::hash_TFixedCircularBuffer<unsigned int, 100l, 2l, 1000l, 3l, 0l, 0l, 100l, 2l, 1l, 31l>" status="run" time="0" classname="vfc_hash_TFixedCircularBuffer/3" />
</testsuite>
<testsuite name="vfc_hash_TFixedCircularBuffer/4" tests="1" failures="0" disabled="0" errors="0" time="0">
<testcase name="TEST_UNIT_VFC_hash_testVfcComplexTypesTFixedCircularBuffer" type_param="vfc_cppunit::hash_TFixedCircularBuffer<long, 100l, 2l, 1000l, 3l, 0l, 0l, -100l, 2l, -1l, 31l>" status="run" time="0" classname="vfc_hash_TFixedCircularBuffer/4" />
</testsuite>
<testsuite name="vfc_hash_TFixedCircularBuffer/5" tests="1" failures="0" disabled="0" errors="0" time="0">
<testcase name="TEST_UNIT_VFC_hash_testVfcComplexTypesTFixedCircularBuffer" type_param="vfc_cppunit::hash_TFixedCircularBuffer<float, 100l, 2l, 1000l, 3l, 0l, 0l, -100l, 2l, -1l, 31l>" status="run" time="0" classname="vfc_hash_TFixedCircularBuffer/5" />
</testsuite>
<testsuite name="vfc_hash_TFixedMap/0" tests="1" failures="0" disabled="0" errors="0" time="0">
<testcase name="testVfcComplexTypesTFixedMap" type_param="vfc_cppunit::hash_TFixedMap<int, int, 101l, 21l, 1000l, 3l, 0l, 50l, -100l, 2l, -1l, 31l>" status="run" time="0" classname="vfc_hash_TFixedMap/0" />
</testsuite>
<testsuite name="vfc_hash_TFixedMap/1" tests="1" failures="0" disabled="0" errors="0" time="0">
<testcase name="testVfcComplexTypesTFixedMap" type_param="vfc_cppunit::hash_TFixedMap<short, int, 101l, 21l, 1000l, 3l, 0l, 50l, -100l, 2l, -1l, 31l>" status="run" time="0" classname="vfc_hash_TFixedMap/1" />
</testsuite>
<testsuite name="vfc_hash_TFixedMap/2" tests="1" failures="0" disabled="0" errors="0" time="0">
<testcase name="testVfcComplexTypesTFixedMap" type_param="vfc_cppunit::hash_TFixedMap<unsigned int, int, 101l, 21l, 1000l, 3l, 0l, 50l, 100l, 2l, 1l, 31l>" status="run" time="0" classname="vfc_hash_TFixedMap/2" />
</testsuite>
<testsuite name="vfc_hash_TFixedMap/3" tests="1" failures="0" disabled="0" errors="0" time="0">
<testcase name="testVfcComplexTypesTFixedMap" type_param="vfc_cppunit::hash_TFixedMap<unsigned int, short, 101l, 21l, 1000l, 3l, 0l, 50l, 100l, 2l, 1l, 31l>" status="run" time="0" classname="vfc_hash_TFixedMap/3" />
</testsuite>
<testsuite name="vfc_hash_TFixedMap/4" tests="1" failures="0" disabled="0" errors="0" time="0">
<testcase name="testVfcComplexTypesTFixedMap" type_param="vfc_cppunit::hash_TFixedMap<long, int, 101l, 21l, 1000l, 3l, 0l, 50l, -100l, 2l, -1l, 31l>" status="run" time="0" classname="vfc_hash_TFixedMap/4" />
</testsuite>
<testsuite name="vfc_hash_TFixedMap/5" tests="1" failures="0" disabled="0" errors="0" time="0">
<testcase name="testVfcComplexTypesTFixedMap" type_param="vfc_cppunit::hash_TFixedMap<int, short, 101l, 21l, 1000l, 3l, 0l, 50l, -100l, 2l, -1l, 31l>" status="run" time="0" classname="vfc_hash_TFixedMap/5" />
</testsuite>
<testsuite name="vfc_hash_TFixedMap/6" tests="1" failures="0" disabled="0" errors="0" time="0">
<testcase name="testVfcComplexTypesTFixedMap" type_param="vfc_cppunit::hash_TFixedMap<short, int, 101l, 21l, 1000l, 3l, 0l, 50l, -100l, 2l, -1l, 31l>" status="run" time="0" classname="vfc_hash_TFixedMap/6" />
</testsuite>
<testsuite name="vfc_hash_TFixedMap/7" tests="1" failures="0" disabled="0" errors="0" time="0">
<testcase name="testVfcComplexTypesTFixedMap" type_param="vfc_cppunit::hash_TFixedMap<unsigned int, unsigned short, 101l, 21l, 1000l, 3l, 0l, 50l, 100l, 2l, 1l, 31l>" status="run" time="0" classname="vfc_hash_TFixedMap/7" />
</testsuite>
<testsuite name="vfc_hash_TFixedMap/8" tests="1" failures="0" disabled="0" errors="0" time="0">
<testcase name="testVfcComplexTypesTFixedMap" type_param="vfc_cppunit::hash_TFixedMap<long, long, 101l, 21l, 1000l, 3l, 0l, 50l, -100l, 2l, -1l, 31l>" status="run" time="0" classname="vfc_hash_TFixedMap/8" />
</testsuite>
<testsuite name="vfc_hash_TFixedMap/9" tests="1" failures="0" disabled="0" errors="0" time="0">
<testcase name="testVfcComplexTypesTFixedMap" type_param="vfc_cppunit::hash_TFixedMap<int, float, 101l, 21l, 1000l, 3l, 0l, 50l, -100l, 2l, -1l, 31l>" status="run" time="0" classname="vfc_hash_TFixedMap/9" />
</testsuite>
<testsuite name="vfc_hash_TVectorN/0" tests="1" failures="0" disabled="0" errors="0" time="0">
<testcase name="TEST_UNIT_VFC_hash_testVfcComplexTypesTVectorN" type_param="vfc_cppunit::hash_TVectorN<int, 1l, 7l, 100l, 2l, 1000l, 3l, 0l, 0l, -100l, 2l, -1l, 31l>" status="run" time="0" classname="vfc_hash_TVectorN/0" />
</testsuite>
<testsuite name="vfc_hash_TVectorN/1" tests="1" failures="0" disabled="0" errors="0" time="0">
<testcase name="TEST_UNIT_VFC_hash_testVfcComplexTypesTVectorN" type_param="vfc_cppunit::hash_TVectorN<unsigned int, 1l, 6l, 100l, 2l, 1000l, 3l, 0l, 0l, 100l, 2l, 1l, 31l>" status="run" time="0" classname="vfc_hash_TVectorN/1" />
</testsuite>
<testsuite name="vfc_hash_TVectorN/2" tests="1" failures="0" disabled="0" errors="0" time="0">
<testcase name="TEST_UNIT_VFC_hash_testVfcComplexTypesTVectorN" type_param="vfc_cppunit::hash_TVectorN<long, 1l, 7l, 100l, 2l, 1000l, 3l, 0l, 0l, -100l, 2l, -1l, 31l>" status="run" time="0" classname="vfc_hash_TVectorN/2" />
</testsuite>
<testsuite name="vfc_hash_TVectorN/3" tests="1" failures="0" disabled="0" errors="0" time="0">
<testcase name="TEST_UNIT_VFC_hash_testVfcComplexTypesTVectorN" type_param="vfc_cppunit::hash_TVectorN<float, 10l, 3l, 100l, 2l, 1000l, 3l, 0l, 0l, -100l, 2l, -1l, 31l>" status="run" time="0" classname="vfc_hash_TVectorN/3" />
</testsuite>
<testsuite name="vfc_hash_TVector234/0" tests="1" failures="0" disabled="0" errors="0" time="0">
<testcase name="TEST_UNIT_VFC_hash_testVfcComplexTypesTVector234" type_param="vfc_cppunit::hash_TVector234<int, 1l, 100l, 2l, 1000l, 3l, 0l, 0l, -100l, 2l, -1l, 31l>" status="run" time="0" classname="vfc_hash_TVector234/0" />
</testsuite>
<testsuite name="vfc_hash_TVector234/1" tests="1" failures="0" disabled="0" errors="0" time="0">
<testcase name="TEST_UNIT_VFC_hash_testVfcComplexTypesTVector234" type_param="vfc_cppunit::hash_TVector234<unsigned int, 1l, 100l, 2l, 1000l, 3l, 0l, 0l, 100l, 2l, 1l, 31l>" status="run" time="0" classname="vfc_hash_TVector234/1" />
</testsuite>
<testsuite name="vfc_hash_TVector234/2" tests="1" failures="0" disabled="0" errors="0" time="0">
<testcase name="TEST_UNIT_VFC_hash_testVfcComplexTypesTVector234" type_param="vfc_cppunit::hash_TVector234<long, 1l, 100l, 2l, 1000l, 3l, 0l, 0l, -100l, 2l, -1l, 31l>" status="run" time="0" classname="vfc_hash_TVector234/2" />
</testsuite>
<testsuite name="vfc_hash_TVector234/3" tests="1" failures="0" disabled="0" errors="0" time="0">
<testcase name="TEST_UNIT_VFC_hash_testVfcComplexTypesTVector234" type_param="vfc_cppunit::hash_TVector234<float, 10l, 100l, 2l, 1000l, 3l, 0l, 0l, -100l, 2l, -1l, 31l>" status="run" time="0" classname="vfc_hash_TVector234/3" />
</testsuite>
<testsuite name="vfc_hash_TMatrixMN/0" tests="1" failures="0" disabled="0" errors="0" time="0">
<testcase name="TEST_UNIT_VFC_hash_testVfcComplexTypesTMatrixMN" type_param="vfc_cppunit::hash_TMatrixMN<int, 1l, 7l, 5l, 100l, 2l, 1000l, 3l, 0l, 0l, -100l, 2l, -1l, 31l>" status="run" time="0" classname="vfc_hash_TMatrixMN/0" />
</testsuite>
<testsuite name="vfc_hash_TMatrixMN/1" tests="1" failures="0" disabled="0" errors="0" time="0">
<testcase name="TEST_UNIT_VFC_hash_testVfcComplexTypesTMatrixMN" type_param="vfc_cppunit::hash_TMatrixMN<short, 1l, 7l, 5l, 100l, 2l, 1000l, 3l, 0l, 0l, -100l, 2l, -1l, 31l>" status="run" time="0" classname="vfc_hash_TMatrixMN/1" />
</testsuite>
<testsuite name="vfc_hash_TMatrixMN/2" tests="1" failures="0" disabled="0" errors="0" time="0.001">
<testcase name="TEST_UNIT_VFC_hash_testVfcComplexTypesTMatrixMN" type_param="vfc_cppunit::hash_TMatrixMN<unsigned short, 1l, 7l, 5l, 100l, 2l, 1000l, 3l, 0l, 0l, 100l, 2l, 1l, 31l>" status="run" time="0.001" classname="vfc_hash_TMatrixMN/2" />
</testsuite>
<testsuite name="vfc_hash_TMatrixMN/3" tests="1" failures="0" disabled="0" errors="0" time="0">
<testcase name="TEST_UNIT_VFC_hash_testVfcComplexTypesTMatrixMN" type_param="vfc_cppunit::hash_TMatrixMN<unsigned int, 1l, 7l, 5l, 100l, 2l, 1000l, 3l, 0l, 0l, 100l, 2l, 1l, 31l>" status="run" time="0" classname="vfc_hash_TMatrixMN/3" />
</testsuite>
<testsuite name="vfc_hash_TMatrixMN/4" tests="1" failures="0" disabled="0" errors="0" time="0">
<testcase name="TEST_UNIT_VFC_hash_testVfcComplexTypesTMatrixMN" type_param="vfc_cppunit::hash_TMatrixMN<long, 1l, 7l, 5l, 100l, 2l, 1000l, 3l, 0l, 0l, -100l, 2l, -1l, 31l>" status="run" time="0" classname="vfc_hash_TMatrixMN/4" />
</testsuite>
<testsuite name="vfc_hash_TMatrixMN/5" tests="1" failures="0" disabled="0" errors="0" time="0">
<testcase name="TEST_UNIT_VFC_hash_testVfcComplexTypesTMatrixMN" type_param="vfc_cppunit::hash_TMatrixMN<float, 10l, 7l, 5l, 100l, 2l, 1000l, 3l, 0l, 0l, -100l, 2l, -1l, 31l>" status="run" time="0" classname="vfc_hash_TMatrixMN/5" />
</testsuite>
<testsuite name="vfc_hash_TMatrix234/0" tests="1" failures="0" disabled="0" errors="0" time="0">
<testcase name="TEST_UNIT_VFC_hash_testVfcComplexTypesTMatrix234" type_param="vfc_cppunit::hash_TMatrix234<int, 1l, 100l, 2l, 1000l, 3l, 0l, 0l, -100l, 2l, -1l, 31l>" status="run" time="0" classname="vfc_hash_TMatrix234/0" />
</testsuite>
<testsuite name="vfc_hash_TMatrix234/1" tests="1" failures="0" disabled="0" errors="0" time="0">
<testcase name="TEST_UNIT_VFC_hash_testVfcComplexTypesTMatrix234" type_param="vfc_cppunit::hash_TMatrix234<unsigned int, 1l, 100l, 2l, 1000l, 3l, 0l, 0l, 100l, 2l, 1l, 31l>" status="run" time="0" classname="vfc_hash_TMatrix234/1" />
</testsuite>
<testsuite name="vfc_hash_TMatrix234/2" tests="1" failures="0" disabled="0" errors="0" time="0">
<testcase name="TEST_UNIT_VFC_hash_testVfcComplexTypesTMatrix234" type_param="vfc_cppunit::hash_TMatrix234<long, 1l, 100l, 2l, 1000l, 3l, 0l, 0l, -100l, 2l, -1l, 31l>" status="run" time="0" classname="vfc_hash_TMatrix234/2" />
</testsuite>
<testsuite name="vfc_hash_TMatrix234/3" tests="1" failures="0" disabled="0" errors="0" time="0">
<testcase name="TEST_UNIT_VFC_hash_testVfcComplexTypesTMatrix234" type_param="vfc_cppunit::hash_TMatrix234<float, 10l, 100l, 2l, 1000l, 3l, 0l, 0l, -100l, 2l, -1l, 31l>" status="run" time="0" classname="vfc_hash_TMatrix234/3" />
</testsuite>
</testsuites>
The xml is then used in Sphinx-Needs via the following directive:
.. test-file:: TEST_UNIT_RESULT_VFC_HASH
:id: TEST_UNIT_RESULT_VFC_HASH
:tags: unit-test
:file: unit_tests/vfc_unit_tests_hashtest_11_GCC.xml
:auto_suites:
:auto_cases:
I have done some debugging by printing information in the test-reports extension and I can see the problem as such:
File: ../../../build_with_GCC/Testing/xml/vfc/unit_tests/vfc_unit_tests_hashtest_11_GCC.xml, Suite: vfc_hash, ID: TEST_UNIT_RESULT_VFC_HASH_FFD
File: ../../../build_with_GCC/Testing/xml/vfc/unit_tests/vfc_unit_tests_hashtest_11_GCC.xml, Suite: vfc_THash/0, ID: TEST_UNIT_RESULT_VFC_HASH_D63
File: ../../../build_with_GCC/Testing/xml/vfc/unit_tests/vfc_unit_tests_hashtest_11_GCC.xml, Suite: vfc_THash/1, ID: TEST_UNIT_RESULT_VFC_HASH_885
File: ../../../build_with_GCC/Testing/xml/vfc/unit_tests/vfc_unit_tests_hashtest_11_GCC.xml, Suite: vfc_THash/2, ID: TEST_UNIT_RESULT_VFC_HASH_AE3
File: ../../../build_with_GCC/Testing/xml/vfc/unit_tests/vfc_unit_tests_hashtest_11_GCC.xml, Suite: vfc_THash/3, ID: TEST_UNIT_RESULT_VFC_HASH_3D5
File: ../../../build_with_GCC/Testing/xml/vfc/unit_tests/vfc_unit_tests_hashtest_11_GCC.xml, Suite: vfc_THash/4, ID: TEST_UNIT_RESULT_VFC_HASH_C4D
File: ../../../build_with_GCC/Testing/xml/vfc/unit_tests/vfc_unit_tests_hashtest_11_GCC.xml, Suite: vfc_hash_Primitives/0, ID: TEST_UNIT_RESULT_VFC_HASH_0E5
File: ../../../build_with_GCC/Testing/xml/vfc/unit_tests/vfc_unit_tests_hashtest_11_GCC.xml, Suite: vfc_hash_Primitives/1, ID: TEST_UNIT_RESULT_VFC_HASH_0E8
File: ../../../build_with_GCC/Testing/xml/vfc/unit_tests/vfc_unit_tests_hashtest_11_GCC.xml, Suite: vfc_hash_Primitives/2, ID: TEST_UNIT_RESULT_VFC_HASH_528
File: ../../../build_with_GCC/Testing/xml/vfc/unit_tests/vfc_unit_tests_hashtest_11_GCC.xml, Suite: vfc_hash_Primitives/3, ID: TEST_UNIT_RESULT_VFC_HASH_89D
File: ../../../build_with_GCC/Testing/xml/vfc/unit_tests/vfc_unit_tests_hashtest_11_GCC.xml, Suite: vfc_hash_Primitives/4, ID: TEST_UNIT_RESULT_VFC_HASH_FE7
File: ../../../build_with_GCC/Testing/xml/vfc/unit_tests/vfc_unit_tests_hashtest_11_GCC.xml, Suite: vfc_hash_Primitives/5, ID: TEST_UNIT_RESULT_VFC_HASH_3CE
File: ../../../build_with_GCC/Testing/xml/vfc/unit_tests/vfc_unit_tests_hashtest_11_GCC.xml, Suite: vfc_hash_Tuple/0, ID: TEST_UNIT_RESULT_VFC_HASH_3A1
File: ../../../build_with_GCC/Testing/xml/vfc/unit_tests/vfc_unit_tests_hashtest_11_GCC.xml, Suite: vfc_hash_Tuple/1, ID: TEST_UNIT_RESULT_VFC_HASH_417
File: ../../../build_with_GCC/Testing/xml/vfc/unit_tests/vfc_unit_tests_hashtest_11_GCC.xml, Suite: vfc_hash_Tuple/2, ID: TEST_UNIT_RESULT_VFC_HASH_D7B
File: ../../../build_with_GCC/Testing/xml/vfc/unit_tests/vfc_unit_tests_hashtest_11_GCC.xml, Suite: vfc_hash_TFixed/0, ID: TEST_UNIT_RESULT_VFC_HASH_4AD
File: ../../../build_with_GCC/Testing/xml/vfc/unit_tests/vfc_unit_tests_hashtest_11_GCC.xml, Suite: vfc_hash_TFixed/1, ID: TEST_UNIT_RESULT_VFC_HASH_9A7
File: ../../../build_with_GCC/Testing/xml/vfc/unit_tests/vfc_unit_tests_hashtest_11_GCC.xml, Suite: vfc_hash_TFixed/2, ID: TEST_UNIT_RESULT_VFC_HASH_F01
File: ../../../build_with_GCC/Testing/xml/vfc/unit_tests/vfc_unit_tests_hashtest_11_GCC.xml, Suite: vfc_hash_TFixed/3, ID: TEST_UNIT_RESULT_VFC_HASH_796
File: ../../../build_with_GCC/Testing/xml/vfc/unit_tests/vfc_unit_tests_hashtest_11_GCC.xml, Suite: vfc_hash_TFixedCircularBuffer/0, ID: TEST_UNIT_RESULT_VFC_HASH_909
File: ../../../build_with_GCC/Testing/xml/vfc/unit_tests/vfc_unit_tests_hashtest_11_GCC.xml, Suite: vfc_hash_TFixedCircularBuffer/1, ID: TEST_UNIT_RESULT_VFC_HASH_F63
File: ../../../build_with_GCC/Testing/xml/vfc/unit_tests/vfc_unit_tests_hashtest_11_GCC.xml, Suite: vfc_hash_TFixedCircularBuffer/2, ID: TEST_UNIT_RESULT_VFC_HASH_D7A
File: ../../../build_with_GCC/Testing/xml/vfc/unit_tests/vfc_unit_tests_hashtest_11_GCC.xml, Suite: vfc_hash_TFixedCircularBuffer/3, ID: TEST_UNIT_RESULT_VFC_HASH_155
File: ../../../build_with_GCC/Testing/xml/vfc/unit_tests/vfc_unit_tests_hashtest_11_GCC.xml, Suite: vfc_hash_TFixedCircularBuffer/4, ID: TEST_UNIT_RESULT_VFC_HASH_871
File: ../../../build_with_GCC/Testing/xml/vfc/unit_tests/vfc_unit_tests_hashtest_11_GCC.xml, Suite: vfc_hash_TFixedCircularBuffer/5, ID: TEST_UNIT_RESULT_VFC_HASH_9DD
File: ../../../build_with_GCC/Testing/xml/vfc/unit_tests/vfc_unit_tests_hashtest_11_GCC.xml, Suite: vfc_hash_TFixedMap/0, ID: TEST_UNIT_RESULT_VFC_HASH_E34
File: ../../../build_with_GCC/Testing/xml/vfc/unit_tests/vfc_unit_tests_hashtest_11_GCC.xml, Suite: vfc_hash_TFixedMap/1, ID: TEST_UNIT_RESULT_VFC_HASH_8D8
File: ../../../build_with_GCC/Testing/xml/vfc/unit_tests/vfc_unit_tests_hashtest_11_GCC.xml, Suite: vfc_hash_TFixedMap/2, ID: TEST_UNIT_RESULT_VFC_HASH_45F
File: ../../../build_with_GCC/Testing/xml/vfc/unit_tests/vfc_unit_tests_hashtest_11_GCC.xml, Suite: vfc_hash_TFixedMap/3, ID: TEST_UNIT_RESULT_VFC_HASH_904
File: ../../../build_with_GCC/Testing/xml/vfc/unit_tests/vfc_unit_tests_hashtest_11_GCC.xml, Suite: vfc_hash_TFixedMap/4, ID: TEST_UNIT_RESULT_VFC_HASH_921
File: ../../../build_with_GCC/Testing/xml/vfc/unit_tests/vfc_unit_tests_hashtest_11_GCC.xml, Suite: vfc_hash_TFixedMap/5, ID: TEST_UNIT_RESULT_VFC_HASH_CCE
File: ../../../build_with_GCC/Testing/xml/vfc/unit_tests/vfc_unit_tests_hashtest_11_GCC.xml, Suite: vfc_hash_TFixedMap/6, ID: TEST_UNIT_RESULT_VFC_HASH_37A
File: ../../../build_with_GCC/Testing/xml/vfc/unit_tests/vfc_unit_tests_hashtest_11_GCC.xml, Suite: vfc_hash_TFixedMap/7, ID: TEST_UNIT_RESULT_VFC_HASH_DD7
File: ../../../build_with_GCC/Testing/xml/vfc/unit_tests/vfc_unit_tests_hashtest_11_GCC.xml, Suite: vfc_hash_TFixedMap/8, ID: TEST_UNIT_RESULT_VFC_HASH_B95
File: ../../../build_with_GCC/Testing/xml/vfc/unit_tests/vfc_unit_tests_hashtest_11_GCC.xml, Suite: vfc_hash_TFixedMap/9, ID: TEST_UNIT_RESULT_VFC_HASH_210
File: ../../../build_with_GCC/Testing/xml/vfc/unit_tests/vfc_unit_tests_hashtest_11_GCC.xml, Suite: vfc_hash_TVectorN/0, ID: TEST_UNIT_RESULT_VFC_HASH_8DD
File: ../../../build_with_GCC/Testing/xml/vfc/unit_tests/vfc_unit_tests_hashtest_11_GCC.xml, Suite: vfc_hash_TVectorN/1, ID: TEST_UNIT_RESULT_VFC_HASH_10A
File: ../../../build_with_GCC/Testing/xml/vfc/unit_tests/vfc_unit_tests_hashtest_11_GCC.xml, Suite: vfc_hash_TVectorN/2, ID: TEST_UNIT_RESULT_VFC_HASH_2D9
File: ../../../build_with_GCC/Testing/xml/vfc/unit_tests/vfc_unit_tests_hashtest_11_GCC.xml, Suite: vfc_hash_TVectorN/3, ID: TEST_UNIT_RESULT_VFC_HASH_154
File: ../../../build_with_GCC/Testing/xml/vfc/unit_tests/vfc_unit_tests_hashtest_11_GCC.xml, Suite: vfc_hash_TVector234/0, ID: TEST_UNIT_RESULT_VFC_HASH_8D8
where we see ID TEST_UNIT_RESULT_VFC_HASH_8D8 benig repeated.
This questions actually whether a hash is not an overkill solution to this problem, which is not even sufficient in this case.
Namely, if the user already needs to specify an unique ID for the test file name, why not simply use an incremental counter for each suite, and another incremental counter for the test cases? We can then guarantee that the ID will never get repeated.
I assume that the hash was chosen because as long as the title of the suite does not change, it does not matter if you re-order it in the xml, you always get the same ID, whereas using an incremental counter means that adding a new suite or test results in a different ID.
If the hash is a preferred solution then, we either need a longer hash for the suites, or some mechanism to make sure that the ID does not get repeated. For example, while parsing a file and selecting the hashed IDs of both suites and test cases, the extension could keep a list of the IDs selected so far for this file, and if it produces a hashed ID which has by chance already been selected, then simply increase the value by one and check again until an unused ID is found, or another similar solution. In this case one should verify that we hace not made a complete loop for the limit of encodable values for the chosen lenght of the hash, such that we do not enter an infinite loop. We can then simply issue an error that the number of supported suites for a given file has been reached, or otherwise simply increase the lenght of the hash?
I have tested testsphinx-test-reports with Google test XML Report and I get some errors.
As far as i can tell, google uses more XML tags as your test sample xml_data_2.xml (one and multiple with ). I have deleted the extra tags and have produced a Sphinx output.
Is the Google test XML Report something that will be supported in future or am i doing something wrong?
Example:
<?xml version="1.0" encoding="UTF-8"?>
<testsuites tests="40" failures="0" disabled="0" errors="0" timestamp="2019-01-07T14:21:56" time="0.047" name="AllTests">
<testsuite name="testsuite1" tests="1" failures="0" disabled="0" errors="0" time="0">
<testcase name="testsuite1case1" status="run" time="0" classname="testsuite1" />
</testsuite>
<testsuite name="testsuite2" tests="8" failures="0" disabled="0" errors="0" time="0">
<testcase name="testsuite2case1" status="run" time="0" classname="testsuite2" />
<testcase name="testsuite2case2" status="run" time="0" classname="testsuite2" />
<testcase name="testsuite2case3" status="run" time="0" classname="testsuite2" />
</testsuite>
</testsuites>
Hi,
I would like to know if test_reports currently supports our use case.
Namely, we have multiple software requirements, where each requirement usually matches a method of a class.
For example, let's say, we have ClassA with methods get() and set(), and we then have ReqA_get, and ReqA_set.
For testing we use GTest, and we normally have a test file with a test class that tests an entire class.
For example, we would have GTest class TestClassA for testing ClassA.
The test involves multiple test functions, where multiple test functions are involved in testing a method of ClassA.
For example, we would have TestClassA_get_1, and TestClassA_get_2 for test method get() of ClassA, and we would have TestClassA_set_1, TestClassA_set_2, and TestClassA_set_3 for testing method set() of ClassA.
We then define the Test Specification in such a way that we have one specification per test.
The xml output from GTest for the test cases would be (summarizing only the class name and case name):
classname="TestClassA" and name="get_1"
classname="TestClassA" and name="get_2"
classname="TestClassA" and name="set_1"
classname="TestClassA" and name="set_2"
classname="TestClassA" and name="set_3"
So far, by having a configuration as for example
needs_global_options = {
'tests': "[[tr_link('classname', 'title')]]",
}we need to have the same name as the test class in all test specifications, e.g., .. test:: TestClassA, and we would get ALL test cases in the test file TestClassA linked to every test specification. However, we would like to have each test case only linked with its corresponding test specification.
One way to do this would be to adapt the tr_link function such that it could take a tupple of multiple things that need to be matched. We could then add classname and casename to the test specification and then have both things matched to the xml.
Thank you for the great work.
If I have the following code:
chapter 1
=========
chapter 1.1
------------
chapter 1.1.1
~~~~~~~~~~~~~
.. test-env:: xy.json
:env: env_1, env_2
Then the sections structure looks likes this.
That's not so nice, as for normal cases the author wants the env-information be part of the last defined section. So it should look like this:
I am trying out this library on Windows and have gotten into a little trouble with paths. In doing a test-report:: directive and passing a unix-style relative path for :file:
.. test-report:: Unit Test Results
:id: REPORT
:file: ../project/unit-test-results/project-unit-test-results.xml
I'm seeing the following rendered out:
Test file: C:Usersusernamecodesprojectdoc../project/unit-test-results/unit-test-results.xml
Template used: C:Usersusernamecodesvenvsatom39libsite-packagessphinxcontribtest_reportsdirectives/test_report_template.txt
Statistics
Test suites: 0
...
A similar problem happens when I give a windows-style relative path for :file:, including:
:file: ..\\..\\project\\unit-test-results\\project-unit-test-results.xml:file: ..\project\unit-test-results\project-unit-test-results.xmlIf this used pathlib rather than os.path I believe it might work better? Are you interested in a PR along theses lines?
Hey, can you please release a 1.0.3 release to the pip registry with your recent updates? There's commits in the past year but no increase in version
affected page: https://sphinx-test-reports.readthedocs.io/en/latest/install.html
correct command is pip install sphinx-test-reports (instead of pip install sphinxcontrib-test-reports)
Currently test-file and co. do not allow to set values for the extra_options and extra_links.
So Sphinx would claim about a not registered option.
test-file and co. should allow all options/links and just copy&paste them to the next level without any interpretation.
This would allow to set values for user specific options/links.
In pytest-5.1.0 bugfix pytest-dev/pytest#5477 was included which adds a testsuites root element.
see also #7
This breaks test reports in sphinx.
If a given json file for .. test-env:: is somehow corrupted, the whole build crashes.
A meaningful warning would be better.
Hi,
I have a test report file (one made myself) and I am trying to get it to work with sphinx. I can create a test-results directive but it goes wrong when trying to make a test-report directive. It gives certain errors when implementing it as followed:
.. test-report:: My report
:id: REPORT
:file: ../testreports/testresults-20230502-154149.xml
Without any sphinx-test-report configurations in my conf.py it gives the following error trying to generate sphinx html:
Exception occurred:
File "C:\Python311\Lib\site-packages\sphinxcontrib\test_reports\directives\test_file.py", line 113, in run
raise Exception(
Exception: Suite ID REPORT_FF0 already exists by 6.4.X.xml (6.6.X.xml)
The error makes it look like it's trying to regenerate the same ID for 2 suites, so I tried changing the tr_suite_id_length to 4 when I had this it started giving the following error:
Warning, treated as error:
C:\Users\201026868\source\repos\Sphinx\source\1_introduction.rst:50:Include file 'C:\\Users\\201026868\\source\\repos\\sphinxcontrib\\test_reports\\directives\\test_report_template.txt' not found or reading it failed
I found this weird because there is no line 50 in my .rst file and I thought maybe by adding the tr_report_template option to my conf.py it would be resolved but that did nothing.
The xml file being used is as followed:
<testsuites><testsuite name="5.6.X.xml" errors="0" failures="2" skipped="0" tests="2" time="0.0028" timestamp="2023-05-02 16:17:50.206889"><testcase classname="test_functions" name="test_case01" time="0.0006"><failure type="AssertionError" message="First element: PARAMETERS/INTERFACE/Value does not exist">AssertionError: First element: PARAMETERS/INTERFACE/Value does not exist
</failure></testcase><testcase classname="test_functions" name="test_case02" time="0.0021"><failure type="AssertionError" message="Second element: ./PARAMETERS/INTERFACE/Value does not exist">AssertionError: Second element: ./PARAMETERS/INTERFACE/Value does not exist
</failure></testcase></testsuite><testsuite name="5.7.X.xml" errors="0" failures="2" skipped="0" tests="2" time="0.0024" timestamp="2023-05-02 16:17:50.209336"><testcase classname="test_functions" name="test_case01" time="0.0014"><failure type="AssertionError" message="First element: PARAMETERS/INTERFACE/Value does not exist">AssertionError: First element: PARAMETERS/INTERFACE/Value does not exist
</failure></testcase><testcase classname="test_functions" name="test_case02" time="0.0010"><failure type="AssertionError" message="Second element: ./PARAMETERS/INTERFACE/Value does not exist">AssertionError: Second element: ./PARAMETERS/INTERFACE/Value does not exist
</failure></testcase></testsuite><testsuite name="5.8.X.xml" errors="0" failures="2" skipped="0" tests="2" time="0.0017" timestamp="2023-05-02 16:17:50.211552"><testcase classname="test_functions" name="test_case01" time="0.0010"><failure type="AssertionError" message="First element: PARAMETERS/INTERFACE/Value does not exist">AssertionError: First element: PARAMETERS/INTERFACE/Value does not exist
</failure></testcase><testcase classname="test_functions" name="test_case02" time="0.0007"><failure type="AssertionError" message="Second element: ./PARAMETERS/INTERFACE/Value does not exist">AssertionError: Second element: ./PARAMETERS/INTERFACE/Value does not exist
</failure></testcase></testsuite><testsuite name="5.9.X.xml" errors="0" failures="2" skipped="0" tests="2" time="0.0019" timestamp="2023-05-02 16:17:50.213430"><testcase classname="test_functions" name="test_case01" time="0.0014"><failure type="AssertionError" message="First element: PARAMETERS/INTERFACE/Value does not exist">AssertionError: First element: PARAMETERS/INTERFACE/Value does not exist
</failure></testcase><testcase classname="test_functions" name="test_case02" time="0.0005"><failure type="AssertionError" message="Second element: ./PARAMETERS/INTERFACE/Value does not exist">AssertionError: Second element: ./PARAMETERS/INTERFACE/Value does not exist
</failure></testcase></testsuite><testsuite name="6.0.X.xml" errors="0" failures="2" skipped="0" tests="2" time="0.0026" timestamp="2023-05-02 16:17:50.216038"><testcase classname="test_functions" name="test_case01" time="0.0019"><failure type="AssertionError" message="First element: PARAMETERS/INTERFACE/Value does not exist">AssertionError: First element: PARAMETERS/INTERFACE/Value does not exist
</failure></testcase><testcase classname="test_functions" name="test_case02" time="0.0007"><failure type="AssertionError" message="Second element: ./PARAMETERS/INTERFACE/Value does not exist">AssertionError: Second element: ./PARAMETERS/INTERFACE/Value does not exist
</failure></testcase></testsuite><testsuite name="6.1.X.xml" errors="0" failures="2" skipped="0" tests="2" time="0.0022" timestamp="2023-05-02 16:17:50.218288"><testcase classname="test_functions" name="test_case01" time="0.0009"><failure type="AssertionError" message="First element: PARAMETERS/INTERFACE/Value does not exist">AssertionError: First element: PARAMETERS/INTERFACE/Value does not exist
</failure></testcase><testcase classname="test_functions" name="test_case02" time="0.0013"><failure type="AssertionError" message="Second element: ./PARAMETERS/INTERFACE/Value does not exist">AssertionError: Second element: ./PARAMETERS/INTERFACE/Value does not exist
</failure></testcase></testsuite><testsuite name="6.2.X.xml" errors="0" failures="2" skipped="0" tests="2" time="0.0018" timestamp="2023-05-02 16:17:50.220541"><testcase classname="test_functions" name="test_case01" time="0.0013"><failure type="AssertionError" message="First element: PARAMETERS/INTERFACE/Value does not exist">AssertionError: First element: PARAMETERS/INTERFACE/Value does not exist
</failure></testcase><testcase classname="test_functions" name="test_case02" time="0.0005"><failure type="AssertionError" message="Second element: ./PARAMETERS/INTERFACE/Value does not exist">AssertionError: Second element: ./PARAMETERS/INTERFACE/Value does not exist
</failure></testcase></testsuite><testsuite name="6.3.X.xml" errors="0" failures="0" skipped="0" tests="2" time="0.0011" timestamp="2023-05-02 16:17:50.221650"><testcase classname="test_functions" name="test_case01" time="0.0006">Passed</testcase><testcase classname="test_functions" name="test_case02" time="0.0005">Passed</testcase></testsuite><testsuite name="6.4.X.xml" errors="0" failures="0" skipped="0" tests="2" time="0.0011" timestamp="2023-05-02 16:17:50.223261"><testcase classname="test_functions" name="test_case01" time="0.0011">Passed</testcase><testcase classname="test_functions" name="test_case02" time="0.0000">Passed</testcase></testsuite><testsuite name="6.5.X.xml" errors="0" failures="0" skipped="0" tests="2" time="0.0010" timestamp="2023-05-02 16:17:50.224292"><testcase classname="test_functions" name="test_case01" time="0.0007">Passed</testcase><testcase classname="test_functions" name="test_case02" time="0.0004">Passed</testcase></testsuite><testsuite name="6.6.X.xml" errors="0" failures="0" skipped="0" tests="2" time="0.0000" timestamp="2023-05-02 16:17:50.224292"><testcase classname="test_functions" name="test_case01" time="0.0000">Passed</testcase><testcase classname="test_functions" name="test_case02" time="0.0000">Passed</testcase></testsuite><testsuite name="6.7.X.xml" errors="0" failures="0" skipped="0" tests="2" time="0.0000" timestamp="2023-05-02 16:17:50.224292"><testcase classname="test_functions" name="test_case01" time="0.0000">Passed</testcase><testcase classname="test_functions" name="test_case02" time="0.0000">Passed</testcase></testsuite><testsuite name="6.8.X.xml" errors="0" failures="0" skipped="0" tests="2" time="0.0035" timestamp="2023-05-02 16:17:50.227767"><testcase classname="test_functions" name="test_case01" time="0.0030">Passed</testcase><testcase classname="test_functions" name="test_case02" time="0.0005">Passed</testcase></testsuite><testsuite name="6.9.X.xml" errors="0" failures="0" skipped="0" tests="2" time="0.0011" timestamp="2023-05-02 16:17:50.229372"><testcase classname="test_functions" name="test_case01" time="0.0011">Passed</testcase><testcase classname="test_functions" name="test_case02" time="0.0000">Passed</testcase></testsuite><testsuite name="7.0.X.xml" errors="0" failures="0" skipped="0" tests="2" time="0.0025" timestamp="2023-05-02 16:17:50.231896"><testcase classname="test_functions" name="test_case01" time="0.0000">Passed</testcase><testcase classname="test_functions" name="test_case02" time="0.0025">Passed</testcase></testsuite><testsuite name="7.1.X.xml" errors="0" failures="2" skipped="0" tests="2" time="0.0024" timestamp="2023-05-02 16:17:50.234284"><testcase classname="test_functions" name="test_case01" time="0.0000"><failure type="AssertionError" message="First element value is: Ethernet and not the expected: USB">AssertionError: First element value is: Ethernet and not the expected: USB
</failure></testcase><testcase classname="test_functions" name="test_case02" time="0.0024"><failure type="AssertionError" message="First element value is: Ethernet and not the expected: USB">AssertionError: First element value is: Ethernet and not the expected: USB
</failure></testcase></testsuite><testsuite name="7.2.X.xml" errors="0" failures="2" skipped="0" tests="2" time="0.0020" timestamp="2023-05-02 16:17:50.236289"><testcase classname="test_functions" name="test_case01" time="0.0000"><failure type="AssertionError" message="First element value is: Ethernet and not the expected: USB">AssertionError: First element value is: Ethernet and not the expected: USB
</failure></testcase><testcase classname="test_functions" name="test_case02" time="0.0020"><failure type="AssertionError" message="First element value is: Ethernet and not the expected: USB">AssertionError: First element value is: Ethernet and not the expected: USB
</failure></testcase></testsuite><testsuite name="7.3.X.xml" errors="0" failures="2" skipped="0" tests="2" time="0.0023" timestamp="2023-05-02 16:17:50.238627"><testcase classname="test_functions" name="test_case01" time="0.0000"><failure type="AssertionError" message="First element value is: Ethernet and not the expected: USB">AssertionError: First element value is: Ethernet and not the expected: USB
</failure></testcase><testcase classname="test_functions" name="test_case02" time="0.0023"><failure type="AssertionError" message="First element value is: Ethernet and not the expected: USB">AssertionError: First element value is: Ethernet and not the expected: USB
</failure></testcase></testsuite><testsuite name="7.4.X.xml" errors="0" failures="2" skipped="0" tests="2" time="0.0019" timestamp="2023-05-02 16:17:50.240576"><testcase classname="test_functions" name="test_case01" time="0.0011"><failure type="AssertionError" message="First element value is: Ethernet and not the expected: USB">AssertionError: First element value is: Ethernet and not the expected: USB
</failure></testcase><testcase classname="test_functions" name="test_case02" time="0.0008"><failure type="AssertionError" message="First element value is: Ethernet and not the expected: USB">AssertionError: First element value is: Ethernet and not the expected: USB
</failure></testcase></testsuite><testsuite name="7.5.X.xml" errors="0" failures="2" skipped="0" tests="2" time="0.0033" timestamp="2023-05-02 16:17:50.244385"><testcase classname="test_functions" name="test_case01" time="0.0019"><failure type="AssertionError" message="First element value is: Ethernet and not the expected: USB">AssertionError: First element value is: Ethernet and not the expected: USB
</failure></testcase><testcase classname="test_functions" name="test_case02" time="0.0014"><failure type="AssertionError" message="First element value is: Ethernet and not the expected: USB">AssertionError: First element value is: Ethernet and not the expected: USB
</failure></testcase></testsuite><testsuite name="NOTRBBASELINE.xml" errors="0" failures="0" skipped="0" tests="2" time="0.0010" timestamp="2023-05-02 16:17:50.245382"><testcase classname="test_functions" name="test_case01" time="0.0010">Passed</testcase><testcase classname="test_functions" name="test_case02" time="0.0000">Passed</testcase></testsuite></testsuites>
I know it is not clear to see but basically there are multiple testsuite elements with each 2 testcase elements in them, and some have failure elements. I would say the xml is correct because it is possible to generate a sphinx test-results directive with it.
Hopefully it is just an issue on my side otherwise it would be great to see a solution.
Kind regards,
Jesse
Now that Sphinx 4 has been released, sphinx-test-report is no longer compatible with it because it uses the add_stylesheet method of Sphinx application object instead of the current add_css_file method.
This has apparently been fixed in sphinx-test-reports 0.3.4 but that version is not yet available in PyPi. This break default installation with and prevents us from upgrading to Sphinx 4.0.
Hello,
We have the need to include javascript cypress coverage reports in our sphinx documentation. Unfortunately, cypress outputs multiple junit xml report files, one per test-file.
This is really awkward to get working with sphinx-test-reports.
Would it make sense to modify the directive's :file: argument to accept 1+ files & bundle them together?
Thanks,
In CMake/CTest Tool used for test runs could generate junit xml files. The files are not properly parsed due to some missing tags seemed to be required by sphinx-test-reports.
Upgrade sphinx-test-reports on master to use Sphinx-Needs > 1.0.0 so that it can be tested in a setup where Sphinx-Needs > 1.0.0 is used.
I've just started playing around with the needs system and this library. I was wondering if it is possible to create a trace matrix (table) to link other needs such as requirements/specs/implementations with test cases.
I am using pytest and have tried to load the test results via junit XML. It works, but I cannot seem to link the tests. Is there some way to declare each test manually, for example within the docstring of the test function itself? If I did that I could manually link them to other needs.
Alternatively, I assume you can add links dynamically when loading test data from XML. For example, I could link the test classname to some classname declared on the other need(s). I tried this but got an error:
.. test-report:: Test Report
:id: TEST_REPORT
:file: pytest_report.xml
:tags: tests
:links: [[tr_link('classname', 'classname')]]
The error was Unknown dynamic sphinx-needs function: tr_link. Found in need: TEST_REPORT
Ideally for my use case it would be best to declare each test as a need, but somehow also link that to the XML junit test suite results... is that possible?
We can use https://www.sphinx-doc.org/en/master/extdev/appapi.html#sphinx.application.Sphinx.setup_extension to load all needed external extensions instead of forcing the user to configure it for us in conf.py
Should be fixed with #9.
As I'm not using Google test, I'm unable to test it.
Originally posted by @danwos in #7 (comment)
current now in the sphinx-test-reports v0.3.7, the Multi test suit tags import still not supported, only the test cases in the 1st test suit tag is imported by the tool, the later test suit tag is ignored.
below is 1 google test output xml for your reference and test, pls help check this issue is fixed or not, tks.
<?xml version="1.0" encoding="UTF-8"?>
<testsuites tests="6" failures="1" disabled="0" errors="0" time="0.001" timestamp="2022-08-15T09:08:41.743" name="AllTests">
<testsuite name="TimerTest" tests="4" failures="0" disabled="0" skipped="0" errors="0" time="0" timestamp="2022-08-15T09:08:41.743">
<testcase name="test_Timer_increment" status="run" result="completed" time="0" timestamp="2022-08-15T09:08:41.743" classname="TimerTest" />
<testcase name="test_Timer_checkTime" status="run" result="completed" time="0" timestamp="2022-08-15T09:08:41.743" classname="TimerTest" />
<testcase name="test_Timer_getdT" status="run" result="completed" time="0" timestamp="2022-08-15T09:08:41.743" classname="TimerTest" />
<testcase name="test_Timer_getCurrentTime" status="run" result="completed" time="0" timestamp="2022-08-15T09:08:41.743" classname="TimerTest" />
</testsuite>
<testsuite name="TimerCounterImplTest" tests="2" failures="1" disabled="0" skipped="0" errors="0" time="0.001" timestamp="2022-08-15T09:08:41.743">
<testcase name="CheckTimerFunction_Init" status="run" result="completed" time="0" timestamp="2022-08-15T09:08:41.743" classname="TimerCounterImplTest">
<failure message="C:\CPPComponent\CPPTimeCounter\tst\tests\UT\test_c_TimeCounter_timecounter_5msImpl.cpp:85
Expected equality of these values:
 TimerValue
 Which is: 500
 testOuputs.m_TimerCurrentValue
 Which is: 0" type=""><![CDATA[C:\CPPComponent\CPPTimeCounter\tst\tests\UT\test_c_TimeCounter_timecounter_5msImpl.cpp:85
Expected equality of these values:
TimerValue
Which is: 500
testOuputs.m_TimerCurrentValue
Which is: 0]]></failure>
<failure message="C:\CPPComponent\CPPTimeCounter\tst\tests\UT\test_c_TimeCounter_timecounter_5msImpl.cpp:86
Expected equality of these values:
 TimeReached
 Which is: true
 testOuputs.m_TimeReached
 Which is: false" type=""><![CDATA[C:\CPPComponent\CPPTimeCounter\tst\tests\UT\test_c_TimeCounter_timecounter_5msImpl.cpp:86
Expected equality of these values:
TimeReached
Which is: true
testOuputs.m_TimeReached
Which is: false]]></failure>
</testcase>
<testcase name="CheckTimerFunction_StartTimer" status="run" result="completed" time="0.001" timestamp="2022-08-15T09:08:41.743" classname="TimerCounterImplTest" />
</testsuite>
</testsuites>Minor warning issue
\sphinxcontrib\test_reports\environment.py:10: RemovedInSphinx80Warning: The alias 'sphinx.util.status_iterator' is deprecated, use 'sphinx.util.display.status_iterator' instead. Check CHANGES for Sphinx API modifications. from sphinx.util import status_iterator # NOQA Sphinx 1.5
Other directives like image or include use relative path starting from rSt file. Looks like this is the standard.
.. test-file:: uses relative path starting from folder where conf.py is. This behavior is not consistent with the other directives.
Hint: Change to this behavior will break existing builds. Possibly it shall be changed asap or never.
See similar issue: useblocks/sphinx-needs#351
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.