Summary
Cucumber does not offer a parallelization option out of the box.
I'd like to get a Runtime option like "--threads 4".
The option will put all scenarios of all .features files in a queue.
As soon as a thread is idle, the thread will pick a scenario from the queue and execute it.
I'd also like to have a tagging mechanism to forbid some scenarios to run in parallel if they are prone to interfere with each other.
Current Behavior
Maven plugins or other external components tend to offer hacky ways to implement some sort of parallelism.
They almost all span multiple Cucumber processes, each one running a different .feature file, and then they run a process that merges all the JSON reports into one big report.json.
I'm not happy with this situation because I (anybody can) have .feature files that vary in number, length, and duration of scenarios (depending on varying things like network load required by each scenario).
This results in having one thread taking very long to execute a bigger .feature file while other threads have finished executing all other small .feature files.
Expected Behavior
Given the current behavior, I'd like the parallelism to be smarter, to be optimal about execution time, and a better thread occupation efficiency.
Possible Solution
For our project, I've made a quick fork of Cucumber with this feature, and we use it with success:
https://github.com/slaout/cucumber-jvm/tree/parallel-scenarios-execution-1.2.4
It has the following additions:
- --threads parameter to tell Cucumber how much executor threads must be spanned
- At the begin of the execution, all .feature files are scanned and all scenarios are put in a queue
- Each thread picks a scenario to execute from the queue, and as soon as it finished, it picks another scenario from the queue
- Some scenario needs to be synchronized:
- For instance, say two scenarios begin with setting the online stock of a product before testing it: they must be executed sequentially for no interference
- Say two other scenarios are modifying the in-store stock of a product before testing it: the two of them must also be executed sequentially BUT they can perfectly be executed in parallel with the first two scenarios that modify the online stock
- To get that behavior, all four scenarios are tagged with a tag starting with "@synchronized-". In this case, we will tag the two first scenarios with @synchronized-online-stock and the two others are tagged with @synchronized-store-stock
- Cucumber will group all scenarios with the same @synchronized- tag name in one "execution group" and will place them first in the queue, because they contain several not-parallelizable scenarios, so executing them first is more efficient if one group is quite large (imagine this large group being at the bottom of the queue: we would end up with the poor parallelization efficiency discussed in "Current Behavior" with other threads having finished their job and being idle)
- Not implemented, but we could imagine the order of scenarios in the queue could be randomized, so if there are potential concurrency problems, they get spotted quicker, and developers can either fix the scenarios (and ensure scenarios are independent from each other, not just working by luck of current schedule) or add "@synchronized-" tags to concurrent scenarios.
- Not implemented too, but some heuristics could put scenarios with a lot of steps at the start of the queue, as they could be longer to execute, which could lead to poorer threading efficiency (but as the gain could be marginal, it could as well not be implemented, it's just an idea)
The fork is launching multiple theads: each scenario has its own Formatter and Reporter: executing a scenario will "print" to those formater and reporter: hey will just record method calls in memory.
At the end, I list all scenarios in the order they are declarer and I tell the formater and reporter of these scenarios to replay what they memorized into the "real" formatter and reporter to construct the report.json exactly the same as it would have been created without parallelism.
To not change the code too much, I heavily added ThreadLocal variables and a few synchronized blocks where needed.
A proper refactor of all the code base will be needed for it to be cleaner, but I read somewhere you were already planning to do so for the next major version of Cucumber.
It's developed mainly for the Java backend, but I think it would need basic adjustments to expose the new threads option to other languages.
It's done for the CLI runner.
It should work for the JUnit runner too, but because all reports are stored in memory and merged at the very end, the JUnit view of the IDE will not move in real time, and it will change all at once at the end.
Note: this fork also contains the @ContinueNextStepsFor({Throwable.class}) evolution seen on issue #79.
Because our project needs both parallel execution and continuing execution on Then step failures.
You can see the changes introduced by the fork with this comparison:
slaout/cucumber-jvm@continue-next-steps-for-exceptions-1.2.4...slaout:parallel-scenarios-execution-1.2.4
Mainly:
- ThreadLocals
- synchronized blocks
- added PlaybackFormatter (implements Reporter, Formatter): records all events of the current thread and is able to play them back to another instance that will merge the reports of all scenario executions in the expected order
- added ProxyFormatter (implements Reporter, Formatter): just for debug purpose: will be removed
- ScenarioExecutionRunnable.java
- ThreadedRuntime.java