Allowing separate report file at parallel execution

[jls]

Allowing separate report file at parallel execution

[tyleralves]

Is there anyway to have parallel reports appended, maybe just with the xml filename in the report title? Thanks!

[jls]

Hello! There was some discussion about this in issue #21.

I think the solution there was to use an environment variable to make sure the filename was unique.

Would it work for you if there was an option that allowed you to guarantee unique filenames for each report? Something like { uniqueFilename: true } that would cause a timestamp to be appended to the filename so that would make sure each test run would have a unique report filename?

[nemethgabor83]

I'm using nightwatch-html-reporter from globals.js. I think adding a timestamp to report filename would make the report file unique, but it is not convenient to find out from the report filename which testsuit is in the report file, that's why I would suggest adding the name of the testfile.
If the timestamp are also contained in the report file name, it would be better from tracking point of you.

[nemethgabor83]

Hi James, what do you think about adding testcase name to report file name?

[jls]

Hello! I wouldn't be opposed to seeing each suite in it's own report file if you wanted to try out a pull request. It looks like your example above it is just using the last suite name as the filename. If they run multiple suites should it generate multiple report files each with the suite name prepended (one per suite)?

[nemethgabor83]

Hi James, I've tested this solution, and it seems to be working properly. When I execute multiple suites, each suite has its own report file.

reportFilename: 'report.html',
    separateReportPerSuite: true,
    uniqueFilename: false,
    themeName: 'default',
    logLevel: 1,
    debug: {
      saveNightwatch: false
    }
  });

  logger.setLevel(opts.logLevel);

  this.fn = function(results, done) {
    var testName = '';
    var oneObject = Object.keys(results.modules);      
    oneObject.forEach(function (nextLevel) {
            testName = nextLevel;
    });
    if (opts.separateReportPerSuite) {
       opts.reportFilename = opts.reportFilename.replace(/\.html/,'') + '-' + testName.replace(/\\/g,'-') + '.html';
    }
    var generate = function generate(next) {

That's all what I've changed. Would you push it to master, or should I create a pull request?
Thank you!

[nemethgabor83]

Hi James, I've created a pull request with this solution.

[jls]

Thank you! There are a couple of other issues right now around report filename generation. I'm going to refactor that part of the lib so I'll incorporate your change shortly after that! Thank you again!

[jls]

@tyleralves I'm not sure if I follow exactly what you mean. If you have access XML filenames to put in the report title then you are running the reporter on a directory of XML files from a previous test run. Do you mean be able to specify multiple report XML directories?

[tyleralves]

@jls Sorry for the LONG delay.

I mean if I'm running in parallel on Chrome and Firefox, it would be great to have the results output to the same report file. Maybe separate tabs, or just a separate heading for each (chrome and firefox). If it is impossible to see the actual browser used, then just some other identifier.

My comments regarding the xml filenames were uneducated and can be ignored.

Edit: Looks like what I'm asking about is #21

• They closed it but the original request wasn't actually addressed. That is, having results from all parallel runs, compiled into one report.

[jls]

Hi @tyleralves, no problem!

Ah ok I think I'm understanding. #21 was about report files being overwritten during parallel runs so the timestamp helped with that. I don't know if combining multiple parallel runs into a single report is possible just because the reporter is called from nightwatch for each test run, it doesn't really know (as far as I can tell) that it's a part of parallel execution.

If you were using the XML files to generate the html report we could probably look at passing multiple XML file directories through the -d option on the CLI and build a combined report that maybe used the XML file directory name as the heading.

But using the built in nightwatch reporter I'm not sure how I would detect the multiple runs that should be included in a single report.

[jls]

Feedback I've gotten for parallel tests is the timestamp solution solves the majority of problems (at least the vocal ones!). Going to close this out but I'll look back over the earlier pull request to see if that can be pulled in.