donutreport/donut

Sweet reports for the icing over your automated tests

donut's Issues

Read non-BDD json files and publish a single merged report

We need to read one set of non-BDD json files and publish a single merged report with both BDD and non-BDD reports.

Group scenarios from different json files by feature name

With the latest version of cucumber-parallel plugin (the one written by Tim), we have the ability to split the execution of cucumber feature files by scenarios as well as feature.

When we split execution by scenarios, we get 1 json report file per scenario, and in this case donut displays that many features in the donut html report.

We need to modify donut to group scenarios under the same feature name, but in different json files, by feature name.

Java exception when running particular json outputs

When running Donut directly from terminal using java -jar donut-0.0.3-one-jar.jar -s . I sometimes receive an error that prevents the report from being generated. This has been reproduced with many different feature files of varying formats and complexities and I can't spot any obvious pattern.

I have now managed to pin it down to two scenarios, when I run scenario 1, it passes and I can successfully generate a Donut report with the JSON output (test1.json). When I run scenario 2, it also passes and I can successfully generate a Donut report with that JSON output (test2.json). However, if I run scenario 1 and 2 as part of the same test run, attempting to use Donut with that JSON output (test1andtest2.json) throws the following error:

Exception in thread "main" java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at com.simontuffs.onejar.Boot.run(Boot.java:340)
    at com.simontuffs.onejar.Boot.main(Boot.java:166)
Caused by: java.lang.IndexOutOfBoundsException: 0
    at scala.collection.LinearSeqOptimized$class.apply(LinearSeqOptimized.scala:65)
    at scala.collection.immutable.List.apply(List.scala:84)
    at io.magentys.donut.transformers.cucumber.CucumberTransformer$$anonfun$mapToDonutScenarios$1.apply(CucumberTransformer.scala:55)
    at io.magentys.donut.transformers.cucumber.CucumberTransformer$$anonfun$mapToDonutScenarios$1.apply(CucumberTransformer.scala:53)
    at scala.collection.immutable.List.flatMap(List.scala:327)
    at io.magentys.donut.transformers.cucumber.CucumberTransformer$.mapToDonutScenarios(CucumberTransformer.scala:53)
    at io.magentys.donut.transformers.cucumber.CucumberTransformer$.mapToDonutFeature(CucumberTransformer.scala:30)
    at io.magentys.donut.transformers.cucumber.CucumberTransformer$$anonfun$mapToDonutFeatures$1.apply(CucumberTransformer.scala:24)
    at io.magentys.donut.transformers.cucumber.CucumberTransformer$$anonfun$mapToDonutFeatures$1.apply(CucumberTransformer.scala:21)
    at scala.collection.immutable.List.map(List.scala:273)
    at io.magentys.donut.transformers.cucumber.CucumberTransformer$.mapToDonutFeatures(CucumberTransformer.scala:21)
    at io.magentys.donut.transformers.cucumber.CucumberTransformer$.transform(CucumberTransformer.scala:12)
    at io.magentys.donut.gherkin.Generator$$anonfun$3.apply(Generator.scala:61)
    at io.magentys.donut.gherkin.Generator$$anonfun$3.apply(Generator.scala:59)
    at io.magentys.donut.performance.PerformanceSupport$class.timed(PerformanceSupport.scala:13)
    at io.magentys.donut.gherkin.Generator$.timed(Generator.scala:15)
    at io.magentys.donut.gherkin.Generator$.createReport(Generator.scala:59)
    at io.magentys.donut.gherkin.Generator$.apply(Generator.scala:32)
    at io.magentys.donut.Boot$.delayedEndpoint$io$magentys$donut$Boot$1(Boot.scala:82)
    at io.magentys.donut.Boot$delayedInit$body.apply(Boot.scala:11)
    at scala.Function0$class.apply$mcV$sp(Function0.scala:34)
    at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
    at scala.App$$anonfun$main$1.apply(App.scala:76)
    at scala.App$$anonfun$main$1.apply(App.scala:76)
    at scala.collection.immutable.List.foreach(List.scala:381)
    at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
    at scala.App$class.main(App.scala:76)
    at io.magentys.donut.Boot$.main(Boot.scala:11)
    at io.magentys.donut.Boot.main(Boot.scala)
    ... 6 more

I have attached the respective JSON files for reference. Possibly I'm missing something obvious or doing something wrong when running but this seems to be quite common across our projects.

JSONOutputs.zip

Failure logged in diagram but no information displayed on it below

failure is shown here:

but no further information displayed on it:

failed test:

(...)

Failure message reported by donut, in spite of creating the reports, in case of a test failures.

Below message is reported in the maven build, in case of test failures, even though donut report is generated successfully:

[ERROR] Failed to execute goal io.magentys:donut-maven-plugin:0.0.3:generate (execution) on project eas-module: Error Found: BUILD FAILED - Check Report For Details -> [Help 1]

toggle events don't work in firefox

Common Data Model

We need to implement a standard model that isn't based on gherkin
Ideally Test Suite, Test Case, Test Step
Test Suite -> Feature
Test Case -> Scenario
Test Step -> Step

This will allow us to more easily support non-gherkin based test tools

Print scenario type ex: Scenarios above all the scenarios

Print scenario type ex: Scenarios above all the scenarios, instead of printing 'Scenario' for every scenario.

Refactor donut-maven-plugin to use the latest version of donut

404 Error when accessing donut.html remotely on VPN

The pattern in the ResultLoader does not consider Windows paths

The pattern in the ResultLoader does not consider Windows paths contain a drive followed by a colon. If a format is excluded from the path then donut fails with an exception relating to an unsupported format. The pattern regex needs to be tightened.

TeamCity plug in

It would be nice

Enhance Feature Summary search to include searching for scenarios

On the Donut Feature summary page, the search function above the feature summary table works only for features currently.
Recently, while presenting Donut as living documentation to a client, they asked if they can search for scenarios. Currently, we can only search for a feature name, but not the scenario names.

Allow custom attributes to be specified for the report

Currently, the build metadata provided with the report is limited to project version and project name. Users should be able to provide their own additional information.

Support passing in the file encoding type of the JSON as a parameter

Had a case where setting the output from cucumber to UTF-8 was causing donut to fail (we were testing against special characters). For now we managed to solve this error by setting Java's default file encoding in the JAVATOOL_OPTIONS variable to use UTF-8, but it would be useful if it was also a parameter that could be passed in.

Build fails when there are special characters in cucumber.json

[ERROR] Failed to execute goalreport.donut:donut-maven-plugin:0.0.6:generate (execution) on project taf-chips: Error Found:: Input length = 1 -> [Help 1]
The error is not very meaningful but we discovered the build fails when there are special characters in the cucumber feature file. For example:

"%!@£$%^&*( “ ” ' ' « » … ° © ® ™ • ½ ¼ ¾ ⅓ ⅔ № † ‡ µ ¢ £ € ♤ ♧ ♥️ ♢ ✓)"

Removing those characters from cucumber.json fixes the problem, however it's painful having to manually go through json and remove all special characters especially if the feature file is massive.

Interesting fact is that it only happens on windows machines. Works fine one Mac no matter if the special characters are there or not.

Add Test Automation HTML ID's

Please can you add Test Automation ID's to the Donut HTML. This will aid in the creation of the UI Acceptance tests.

The initial requirement would be for any interaction elements e.g links, buttons and dropdowns)

As a first trial it would be good to add an ID to the Donut Chart Failures link if possible

Pyramid Generator for Current Run

It would be nice to see whether my set of tests executed for a single project & run conformed (or not) to the test pyramid:
https://www.google.com.au/url?sa=i&rct=j&q=&esrc=s&source=images&cd=&cad=rja&uact=8&ved=0ahUKEwi8oJWbnsbTAhXCEpQKHXpbAXEQjRwIBw&url=https%3A%2F%2Fmartinfowler.com%2Fbliki%2FTestPyramid.html&psig=AFQjCNE3berHhPEtexTKPYnwEhKmBVE77w&ust=1493437457556413

Unable to Generate Donut Report getting below issue

java -jar donut-1.2.2-one-jar.jar -s cucumber:C:\Users\VDI023\Desktop\ECS\seleniumDemo\target\cucumber-reports\Cucumber.json -n seleniumDemo
JarClassLoader: Warning: org/slf4j/impl/StaticLoggerBinder.class in lib/logback-classic-1.1.2.jar is hidden by lib/slf4j-simple-1.6.4.jar (with different bytecode)
JarClassLoader: Warning: org/slf4j/impl/StaticMarkerBinder.class in lib/logback-classic-1.1.2.jar is hidden by lib/slf4j-simple-1.6.4.jar (with different bytecode)
JarClassLoader: Warning: org/slf4j/impl/StaticMDCBinder.class in lib/logback-classic-1.1.2.jar is hidden by lib/slf4j-simple-1.6.4.jar (with different bytecode)
Exception in thread "main" java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.simontuffs.onejar.Boot.run(Boot.java:340)
at com.simontuffs.onejar.Boot.main(Boot.java:166)
Caused by: java.lang.NullPointerException
at scala.collection.mutable.ArrayOps$ofRef$.length$extension(ArrayOps.scala:192)
at scala.collection.mutable.ArrayOps$ofRef.length(ArrayOps.scala:192)
at scala.collection.SeqLike$class.size(SeqLike.scala:106)
at scala.collection.mutable.ArrayOps$ofRef.size(ArrayOps.scala:186)
at report.donut.gherkin.processors.JSONProcessor$.getValidFiles(JSONProcessor.scala:49)
at report.donut.gherkin.processors.JSONProcessor$.loadFrom(JSONProcessor.scala:22)
at report.donut.ResultLoader$CucumberResultLoader.load(ResultLoader.scala:24)
at report.donut.Generator$$anonfun$loadResultSources$1.apply(Generator.scala:83)
at report.donut.Generator$$anonfun$loadResultSources$1.apply(Generator.scala:82)
at scala.collection.immutable.List.foreach(List.scala:381)
at report.donut.Generator$.loadResultSources(Generator.scala:82)
at report.donut.Generator$$anonfun$createReport$2$$anonfun$apply$1.apply(Generator.scala:61)
at report.donut.Generator$$anonfun$createReport$2$$anonfun$apply$1.apply(Generator.scala:61)
at report.donut.performance.PerformanceSupport$class.timed(PerformanceSupport.scala:13)
at report.donut.Generator$.timed(Generator.scala:15)
at report.donut.Generator$$anonfun$createReport$2.apply(Generator.scala:60)
at report.donut.Generator$$anonfun$createReport$2.apply(Generator.scala:59)
at scala.util.Either$RightProjection.flatMap(Either.scala:522)
at report.donut.Generator$.createReport(Generator.scala:59)
at report.donut.Generator$.apply(Generator.scala:33)
at report.donut.Boot$.delayedEndpoint$report$donut$Boot$1(Boot.scala:85)
at report.donut.Boot$delayedInit$body.apply(Boot.scala:8)
at scala.Function0$class.apply$mcV$sp(Function0.scala:34)
at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
at scala.App$$anonfun$main$1.apply(App.scala:76)
at scala.App$$anonfun$main$1.apply(App.scala:76)
at scala.collection.immutable.List.foreach(List.scala:381)
at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
at scala.App$class.main(App.scala:76)
at report.donut.Boot$.main(Boot.scala:8)
at report.donut.Boot.main(Boot.scala)
... 6 more

C:\Users\VDI023\Downloads>

Refactor donut-jenkins-plugin to use the latest version of donut

Split Generator responsibilities

Problem:
Generator is currently producing the html file as well as returning an object ReportConsole. This breaks the single responsibility, is not easily testable as the file generator also contains the side effect, and also forces the file generation which could be optional in case only the console output is required.

Solution:
Break this in 2 different steps so each client may use only what is necessary or both if that's the case.

jUnit Adaptor

To make Donut ubiquitous, we need to add a jUNit adaptor to read jUnit XML

Scenario output not being rendered to HTML template

The latest version of Cinnamon (0.2.0) uses a newer version of Cucumber (4.2.6). The cucumber.json file that gets generated puts any output (when using Scenario.write within an after hook) into the json structure elements.[index].after.[index].output. Donut does not output this custom output as it is looking in the older location of elements.[index].steps.[index].output which was in the older version of Cucumber (with Cinnamon 0.1.2).

See https://github.com/DonutReport/donut/blob/master/src/main/scala/report/donut/gherkin/processors/HTMLProcessor.scala#L157

and

https://github.com/DonutReport/donut/blob/master/src/main/scala/report/donut/gherkin/processors/HTMLProcessor.scala#L107

We rely on the after hooks to write custom output when a scenario fails and can no longer see this output with the latest Cinnamon/Cucumber. This issue probably extends to writing to scenarios in before hook too (but haven't checked).

Examples tables for Scenario outlines (<parameters> not rendering)

Examples tables are not being rendered in the report for non-expanded Scenario Outlines. I've created an examples-tables branch on my fork to add them but the <> delimited parameters are lost in the report.

The branch contains an example 9a.json JSON file used to test this feature. The following is a snippet of a step containing an outline parameter in that file..

            "keyword": "Given ",
            "name": "string 1 is \"<string 1>\""

I'd expect this to render as Given string 1 is "<string1>"
but instead get Given string 1 is ""

I've tried escaping the value in the HTMLProcessor class but I still get the same problem.

Here is a screenshot showing the examples tables rendering but the parameters not..

Question: How exactly to use the Generator feature?

Hi,
In the project Wiki I found that we can create HTML report with Generator. Can you guys pls give an example of where this piece of code will @go?

ReportConsole report =
    Generator.apply(sourceDirectory, outputDirectory, filePrefix, timestamp,
    template, countSkippedAsFailure, countPendingAsFailure,
    countUndefinedAsFailure, countMissingAsFailure, projectName, projectVersion);

Enormous Log Generation

Why does the donut report generate enormous logs? is there a way to disable the logs to just "Report Generated Successfully"?

Support for jbehave

First of all, donut is the cool test report I have seen.. Great effort!! Any thought about adding support to "Jbehave" ?

Visual indication of skipped steps

Can we do something about the steps colouring when there are skipped steps - it's not immediately obvious when there are skipped steps.

e.g.

Add support for displaying Unit Tests relating to a feature

Details to be added soon...

Unit test and step statistics are incorrect

If Unit tests are included in the result sources the statistics currently being reported are incorrect.

• Orphaned unit test failures are not considered in the "Unit tests" chart
• All unit test step failures are not considered in the Steps statistics
• Unit tests are not considered in the Tag Summary if the adapted JSON contains tags