Determine if external repos should provide their own komodotest.sh

[pinkwah]

Determine if external repos should provide their own komodotest.sh

[lars-petter-hauge]

Argument against xtgeo having this is that if they change their testing procedure..

If they do, and it is all kept in komodo-releases, it would be us that have to update other repo's test procedures.

Should we allow them to use komodo-integration PRs?

That would occupy resources on our nodes, and all of a sudden we are the maintainers of one of their testing framework. I am not entirely against this, but there are definitive drawbacks as well. We could start out with only testing when we create a new release and, if deemed necessary, include the possibility to use on a per PR basis (it isn't certain that they would want this part either).

The alternative is that they have a komodotest.sh that doesn't do anything until we complain because bleeding stopped passing tests.

That is true. But they do want to be included in testing when we run new releases of komodo, it is in their interest to avoid faulty versions. It will be a silent pass.

I would vote for stating that if they add a komodotest.sh that behaves in a certain way, we will run those functions whenever we create a new release, and that should be fine.

[pinkwah]

If they do, and it is all kept in komodo-releases, it would be us that have to update other repo's test procedures.

The issue here is that, suppose that xtgeo 1.0 uses pytest and xtgeo 2.0 uses tox. In komodo stable we have xtgeo 1.0 while bleeding has 2.0. Then, we'd need to determine which version of xtgeo we are running so that we know whether to install pytest or tox.

This is a non-issue if the testkomodo.sh is present in the repo, and the commit that changed from pytest to tox also includes the change in testkomodo.sh. However, this won't be the case if this script is external to the repo, nor will it be that good if the script isn't run for each PR.

[lars-petter-hauge]

I have to say, the arguments you have in this issue kind of contradicts the arguments you post in #132, where it seems you argue for having komodo on its own figure out what to install and whether it is to run tox or pytest.

This is a non-issue if the testkomodo.sh is present in the repo, and the commit that changed from pytest to tox also includes the change in testkomodo.sh.

This I agree, and therefore i also want the testkomodo.sh to initiate the testing. If komodo is supposed to do some clever thinking on it's own, as suggested in #132, I think we'll quickly end up in a place where that fails and it really doesn't help us much. Sorry for dragging in the other issue, but they're starting to correlate.

However, this won't be the case if this script is external to the repo,

I don't think this script should be external to the repo, I would want them to treat it as any other test-setup config.

nor will it be that good if the script isn't run for each PR.

It won't be as good, no, as they don't get any immediate feedback. But we can probably extend that later. Let's at least first start with having each repo contain their own configuration, which is run on komodo-release builds.

[lars-petter-hauge]

But we can probably extend that later. Let's at least first start with having each repo contain their own configuration, which is run on komodo-release builds.

After thinking about this a bit more I think that the testkomodo.sh should be run on every PR for external repo as well. The reason is, we could end up with a faulty testkomodo.sh for any version in an external repo, and then the komodo-unit tests will always fail for the release with that specific version in it - and we couldn't have fixed it in any other way than to create a new version in the repo and put that in the release. That would be just terrible.

[sondreso]

After a discussion the conclusion was that each repository should include their own komodotest.sh file.