Serenity Console reporter improvements

[1MateuszKruk]

Serenity Console reporter improvements

[1MateuszKruk]

Ok, I figured it out (I guess). Turns out that Playwright's wrapper for reporters expects onError method from custom reporter, and that's something SerenityReporterForPlaywrightTest doesn't have.

Now to Serenity's part @jan-molak , do you have a reliable way to test serenity's console reporter?

(gonna also change to my main account, I want to have first real contribution well documented on my GH account 😄 )

[1MateuszKruk]

And also I think running npm ls --all is kinda too long for an issue 😄

[1MateuszKruk]

Another thing I discovered, Serenity reporter leaves ambiguous report when test times out from playwright setup test.

This test times out on custom Interaction waiting for invoice, but it's not visible.

Also, @serenity-js/code:StreamReporter doesn't indicate the test was timed out

{"type":"SceneFinishes","event":{"sceneId":"j9eesfa8xwgfoaa2c3ec8hpf","timestamp":"2024-02-20T14:08:43.087Z"}}
{"type":"AsyncOperationAttempted","event":{"correlationId":"fzfrpma1qgycaicdk59qjzpv","description":"Dismissing Steve...","name":"Stage","timestamp":"2024-02-20T14:08:43.104Z"}}

It even seems that reporter labels tests in the same case (interaction is so long that test times out) as execution ignored or failed

[jan-molak]

Nice catch; thanks for reporting! I'll need to investigate that! (Unless you already have an idea for a fix, in which case PRs are also very welcome)

[1MateuszKruk]

I can work on that one in free time, but I don't have any implementation in mind and I'm not doing any cookie-licking, so feel free to investigate that yourself.

[jan-molak]

Nice!

OK, so Serenity/JS uses SerenityReporterForPlaywrightTest to extract data from a Playwright Test run -

serenity-js/packages/playwright-test/src/reporter/SerenityReporterForPlaywrightTest.ts

Line 71 in 30bda10

export class SerenityReporterForPlaywrightTest implements Reporter {

That's where we'd need the onError handler.

However, Serenity/JS also decouples the process of extracting the data from reporting by emitting domain events.

For example, you have a TestRunFinishes and TestRunFinished events, the latter of which could be emitted with an outcome indicating an error.

Playwright docs state that onError is:

Called on some global error, for example unhandled exception in the worker process.

I assume this means that the handler should be invoked at most once.
However, I don't know if Playwright executes both onError and onEnd handlers should a global error occur.

If it does, then onError could remember the error, and then the existing onEnd could emit that, if present.

Regarding testing, you can simulate error conditions and how a global error is picked up by the SerenityReporterForPlaywrightTest using the integration/playwright-test suite.

To test the ConsoleReporter, you can use the specs inside packages/console-reporter

Any questions just let me know! And thanks for offering to help out 👍

[In1th]

Alright, alright. I decided to create a new Domain Event for that - GlobalExceptionEncountered. I guess it would be clearer in StreamReporter.

I'll try as well your approach with emitting this during onEnd, but...

I wanted to run this to just see how it works, but I can't get it working with my new event type and building throws me an error complaining it doesn't know this type. I couldn't find it on the Contribute docs, any tips? 😄

[jan-molak]

I couldn't find it on the Contribute docs, any tips?

Haha, sure. You're digging deep into the heart of Serenity/JS :-)

What's the error you're getting? Any stack trace you could share? If you want, feel free to open a draft PR and we can discuss there.

[In1th]

added a draft PR. I'm more of a fan of trying to do and try things several times so I can see what sticks. The problem is that my new Domain Event I created in core package is not recognised by console-reporter package. What should I run to mak those packaged sync between each other?

[jan-molak]

Fixed by #2252, released in Serenity/JS 3.19.0 🚀