Determine why github CI is failing

[eclipse-birt]

Determine why github CI is failing

[hvbtup]

Good question! I am facing the same issue.
Personally, I think that the CI test is fundamentally broken. If and which errors occur seems to be more or less random. I guess that's the reason why the -DskipTests was the default before the BIRT reboot.

[xavierdpt]

Maybe it could be worth doing the build with -fn instead of -DskipTests. This will report all failing tests instead of skipping them, and make it easier to identify which tests are failing randomly and which tests are failing more consistently.

[SteveSchafer-Innovent]

Using PR 650 as a test, I reverted it to the code that existed when PR 643 (issue 642) passed successfully, and it fails with

Failures:
org.eclipse.birt.report.model.metadata.ElementDefnTest.testGetGroupNames(org.eclipse.birt.report.model.metadata.ElementDefnTest)
Run 1: ElementDefnTest.testGetGroupNames:84 expected:<6> but was:<0>
Run 2: PASS
Run 3: PASS

If I run the same mvn commands that are in ci.yml locally it fails with different errors, so that's not going to be a good way to troubleshoot this. What I can do is make changes to PR 650 to see what effect they have. I don't fully understand what's happening when CI is executed, but it's pretty clear that something is different from April 23 when PR 643 CI succeeded.

When CI executes, does it use the ci.yml that's in the PR or the one that's in master? I'll find out experimentally.

[ruspl-afed]

When CI executes, does it use the ci.yml that's in the PR or the one that's in master?

It should use the ci.yml and all the rest from the PR's branch

[SteveSchafer-Innovent]

@ruspl-afed Yes I just verified that it does. So I can make it work by just getting rid of the tests, haha.

[ruspl-afed]

@ruspl-afed Yes I just verified that it does. So I can make it work by just getting rid of the tests, haha.

Firm but fair, sir :)

[SteveSchafer-Innovent]

The particular error I'm getting in PR 650 is caused by MetaDataDictionary not containing the correct objects. MetaDataDictionary is a static object that normally gets loaded from a file named rom.def, however it also gets loaded with other files as part of some unit tests. The failure in CI may be caused by the order in which unit tests are executed. I'm continuing to research this...

[ruspl-afed]

Thank you @SteveSchafer-Innovent for investing so much of your time to this issue

[SteveSchafer-Innovent]

I was able to fix it with a line of code. I submitted a PR (658) which passes the CI tests. Hopefully once this PR is merged, the other PR's will work.

[SteveSchafer-Innovent]

PR merged and closed so closing this issue.