Failed Synthetic tests doesn't fail the Github action step

[datadog]

Failed Synthetic tests doesn't fail the Github action step

[Drarig29]

Hey @29decibel!

It looks like your test jqh-uv5-fqx has an execution rule set to non-blocking. So it is expected that your GHA doesn't exit with 1.

You can change your test to be blocking, or override this test's execution rule with a test file.

More information here: https://docs.datadoghq.com/continuous_testing/troubleshooting/

[29decibel]

@Drarig29 🤦🏼‍♂️ Can't believe I missed this config. It works all good now after switching to blocking.

Thank you so much for the quick response!

[Drarig29]

No problem, have a good day! 😁

[CsabaSzabo]

@Drarig29 I wonder if this feature is part of the v0.2.1 older version of this github action, as setting this flag to no blocking still blocked our CI job.

I just updated our action to v0.18.0 latest version and will check this behaviour again.

One of your synthetic test CI integration blogpost still contains v0.2.1, so I would suggest updating the version number in that blogpost: https://www.datadoghq.com/blog/datadog-github-action-synthetics-ci-visibility/#set-up-the-github-action-to-run-synthetic-tests