do we have any way to combine multiple test runs report into one report (when running test in parallel)

[adamgruber]

do we have any way to combine multiple test runs report into one report (when running test in parallel)

[mjhea0]

I have mochawesome working here -> https://github.com/testdrivenio/cypress-mochawesome-s3

Essentially-

1. cypress runs
2. each spec generates a new report
3. a custom script combines each mochawesome json file and calculates the stats
4. html report is then generated

[fsmaia]

Just inserting a suggestion here too: we could implement a [file] wildcard in reporterFilename to be replaced with current spec, then we would have predictable filenames at least.

I think the implementation is easy: just insert a regex replace from [file] to current report data filename in .getFilename(). We would need to send reportData parameter from .prepare(), to .getOptions() and then .getFilename().

If someone thinks this can be helpful, I can give it a try and open a PR if it works.

[izydorekjakub]

I have mochawesome working here -> https://github.com/testdrivenio/cypress-mochawesome-s3

Essentially-

1. cypress runs
2. each spec generates a new report
3. a custom script combines each mochawesome json file and calculates the stats
4. html report is then generated

It solves all my problems @mjhea0

[stephenchan9]

I have mochawesome working here -> https://github.com/testdrivenio/cypress-mochawesome-s3

Essentially-

1. cypress runs
2. each spec generates a new report
3. a custom script combines each mochawesome json file and calculates the stats
4. html report is then generated

I am getting this issue when running tests that are in a "pending" state. Any possible fix for this? Great job on the code so far.

Invalid value "pending" supplied to /suites/suites/0/tests/0/state: TestState

[SeriousM]

merge reports is the magic word. it's not supported as far as I know

[adamgruber]

@SeriousM is correct. This is something that's been on my radar for a while and held up mostly by lack of time to work on it.

[SeriousM]

It would be awesome to have it because cypress.io is now generating a single report for every spec (which are a lot) and how I end up with mochawesome_001.html to mochawesome_130.html

This is really bad and cypress.io team is not gonna fix it.
What data in the json file should be joined to generate a single report?

[adamgruber]

I'm not really familiar with cypress.io and how it runs the tests. Seems strange that it generates one report for each spec. Is it running each in a separate process? If so I wonder if there's even a way at the reporter level to aggregate everything at the end. You may still need to post-process all the reports through the mochawesome-report-generator CLI.

Each report JSON should have a top level suite. These would need to be combined into one. But the overall stats would be off.

[SeriousM]

Each report JSON should have a top level suite. These would need to be combined into one. But the overall stats would be off.

Can I just omit the overall stats then and just do something like this:

{
  "suites": {
    "suites": [
      ...<all tests here>...
    ]
  }
}

[SeriousM]

ok, I can combine the json files as described. of course my stats are off (maybe I can calculate them later) but the report is ok.

mochawesome-report.zip

I noticed that missing json properties create validation errors. That's ok, but the latest version of mochawesome creates an incompatible json: Invalid value undefined supplied to /suites/suites/0/tests/0/timedOut: Boolean. Adding it let marge work again.

Btw: is there a documentation what all the json properties mean? "beforeHooks", "afterHooks", "pending", etc... when is what filled with and do I have to care?

[adamgruber]

Most of the JSON properties come directly from mocha.

the latest version of mochawesome creates an incompatible json

Is this with cypress.io? Can you provide a repo to reproduce this? I put the validation in place to avoid bad data when generating the report but it has caused some headaches. I may need to revisit how the JSON is validated and consider loosening the requirements.

[SeriousM]

Sure, here is the setup I'm using. I guess the content of the package.json is the important aspect: https://gitlab.com/seriousm/CypressDockerTester https://gitlab.com/seriousm/CypressDockerTester/blob/master/cypress-package.json

[izydorekjakub]

I noticed that missing json properties create validation errors. That's ok, but the latest version of mochawesome creates an incompatible json: Invalid value undefined supplied to /suites/suites/0/tests/0/timedOut: Boolean. Adding it let marge work again.

@SeriousM
Have you managed to solve this problem?

[izydorekjakub]

I noticed that missing json properties create validation errors. That's ok, but the latest version of mochawesome creates an incompatible json: Invalid value undefined supplied to /suites/suites/0/tests/0/timedOut: Boolean. Adding it let marge work again.

@SeriousM
Have you managed to solve this problem?

OK i got it. @mjhea0 do the job.

[mjhea0]

what job? @izydorekjakub

[milankrstin]

@SeriousM

I noticed that missing json properties create validation errors. That's ok, but the latest version of mochawesome creates an incompatible json: Invalid value undefined supplied to /suites/suites/0/tests/0/timedOut: Boolean. Adding it let marge work again.

When I fixed this and ran marge command I always got 2 separate files.
For example: I have report1.json and report2.json When I run: marge report1.json report2.json
Result will be: report1.html and report2.html. Merge is not executed.
Did you have the same issue?

[SeriousM]

Currently we just generate one report per fixture. Sorry that I can't help you any further but we decided to stop investigate this issue any longer as it costs more than it gives us.

[mjhea0]

@stephenchan9 - Yes, this has been fixed.

[sandeepbantode]

@govindn Do you found solution for original issue ? merge reports when run in parallel

[0xc0d3r]

@adamgruber Any update on the original issue?