Test Suite aggregation should occur in runtime

[dylanhitt]

Test Suite aggregation should occur in runtime

[dylanhitt]

@SimonBaeumer I've looking at this some more as I'm up for another refactor. I was looking through the code base and how to implement this. The current goal of this issue would be to move the TestSuite aggregation into runtime. We have defined in the past that the computation of results and other measurements is the responsibility of runtime, so it would make sense to move the convergence of runtime.Result's into runtime.

In order to do this we will need to move convergeResults into runtime. It's not possible to just move app.Execute into runtime as we will be creating a circular dependency between runtime and suite.

Currently my thoughts/prototypes to accomplish aggregation in runtime.

1. Performing something like this in app.Execute

// iterate and add all runtimes
myRuntimes.AddRuntime(out.GetEventHandler(), tests, s.Nodes...)
// after iterating 
result := myRuntimes.Start()

Meaning that we would simply need to support a higher level runtime struct composed of a []runtime.Runtime. I'm not too sure about going down this route as I don't really see the benefit.

2. Create a higher level struct in runtime. Something like:

type RuntimeMeta struct {
    Nodes []runtime.Node
    Tests []runtime.TestCase
}

This could then be passed to runtime which would then be responsible for configuring every single runtime.Runtime.

3. Remove the runtime dependency in suite and change the suite.Suite struct too:

type Suite struct {
	TestCases []suite.TestCase
	Config    suite.GlobalTestConfig
	Nodes     []suite.Node
}

This would allow for a slice of []suite.Suite to be passed to runtime, this would then allow runtime to compose it's runtimes and aggregate them. We could then move more of the functionality in app into runtime. However, I think would be nice to allow app to continue performing I/O and generating the Suites. At most we could be passing []bytes too runtime. It would then be possible to begin executing each suite in a channel without having to pass said channel across packages (This isn't really in the scope of this, just a thought).

What are your thoughts on this? I personally like number 3, but this would require moving some of the runtime structs into suite which on the surface I don't see a problem with.

[dylanhitt]

One thing to note moving the runtime dependency in suite would then require the majority of the structs in runtime to be moved to suite. I don't really see a problem with this on the surface (could be some other issue somewhere else). IMO a lot the structs fit in suite already. The docs state:

Runtime controls the execution of the test suite. It will be initialized by the main package and will be a given a Suite which contains all tests to be executed. The runtime also contains all executors which are different types of ways to execute a test, i.e. on a local machine or a node viá ssh.
I think moving these structs to suite embodies this.

Sorry for the spam btw. I'm sure you understand how prototyping ideas go 😄