badoo/parallel_cucumber

parallel_cucumber's Issues

No help of how to install and use

Information, tutorial and examples are missing on how to install and use parallel_cucumber

It seems that this conflicts as an executable name with parallel_cucumber being used from this repo: https://github.com/grosser/parallel_tests

One important explanation is how to support both or what needs to be done to avoid any conflicts and be able to swap between the two.

Many thanks

Recover lost batches

If a batch-run times out or otherwise fails to return results for some/all the tests in the batch, we should consider re-queueing the tests. (Implies maintaining the currently running tests in the database.)

A test should not be re-queued endlessly - it's probably genuinely timing out or killing its agent. (Implies multiple queues.)

Re-queued tests are re-run in isolation, to separate bad tests from innocent batch-mates. (Implies workers knows about secondary queuing.)

We should recover even (especially) if the agent is killed with extreme prejudice. (Implies worker-tracking.)

Workers should not terminate until the queue is empty and all workers are idle. (Implies coordination)

Proposal:

1. Workers should use transactions (http://redis.io/topics/transactions) to pull 'n' tests off the primary queue (or only 1 from the requeue) and into their own set, and then run them. Once the run is finished, any tests from the primary queue that weren't executed for any reason are added to the requeue.
2. Worker-controller maintains a set listing each worker, removing a worker from the set when it terminates. If a worker terminates with tests in its set, worker-controller adds those tests to the requeue.
3. The worker-controller polls for an empty queue and requeue, and for all worker sets to be empty, whereupon the worker-controller puts a 'tests complete' marker in a controller set and workers terminate in response.
4. The various queues and sets have names based on that of the primary queue - e.g. queue, queue_requeue, queue_worker0, queue_control. These will be ensured empty at start-up by the processes using them (in case of previous catastrophic failure) but should be naturally empty by the end of a normally completed run.

parallel_cucumber doesn't do the scenarios per process calculation properly if cucumber profiles are used instead of tags

It does the calculation correctly if only tags are used but if a profile defined in cucumber.yml is used which has all the tags defined, it fails to calculate exactly how many scenarios will be executed so it does the calculation based on all the scenarios in the project across all feature files due to which the allocation of tests is per process is done incorrectly causing the execution to take more time than expected as tests within a process are executed sequentially.

For example, if my project has total 400 scenarios but I have a profile with which only 40 scenarios should run when I do parallel_cucumber -n 10 '-p test_profile_name' it will say 10 processes for 400 scenarios, ~ 40 scenarios per process

For the same test if I use the tags like: parallel_cucumber -n 10 '-t @abc -t ~@xyz' it will do the calculation correctly and say: 10 processes for 40 scenarios, ~ 4 scenarios per process

Add documentation

argument -e -batch-size

Hi, I have just read the documentation and I'm trying to run more than 1 test with 1 worker

I'm trying to run like this with no sucess:

parallel_cucumber -e --batch-size 3 features/

Log above:

Traceback (most recent call last):
        4: from C:/Ruby25-x64/bin/parallel_cucumber:23:in `<main>'
        3: from C:/Ruby25-x64/bin/parallel_cucumber:23:in `load'
        2: from C:/Ruby25-x64/lib/ruby/gems/2.5.0/gems/parallel_tests-2.31.0/bin/parallel_cucumber:9:in `<top (required)>'
        1: from C:/Ruby25-x64/lib/ruby/gems/2.5.0/gems/parallel_tests-2.31.0/lib/parallel_tests/cli.rb:12:in `run'
C:/Ruby25-x64/lib/ruby/gems/2.5.0/gems/parallel_tests-2.31.0/lib/parallel_tests/cli.rb:229:in `parse_options!': invalid option: --batch-size (OptionParser::InvalidOption)

What am I doing wrong?

It's not possible to override ENV var from config/cucumber.yml

If in config/cucumber.yml for any project env var is specified it is impossible to override it from CLI.

For example

# in config/cucumber.yml:
some_profile: SOME_VAR=1

$ parallel_cucumber --debug --cucumber-options ' SOME_VAR=2 -p some_profile' features/functional/some.feature
...
== Running command `cucumber ... SOME_VAR=2 ... SOME_VAR=1 features/functional/some.feature ...`
...

It seems there is a problem in batch_mapped_files function — it should handle duplicated ENV vars (and replace old ones cucumber config with new ones from CLI/--cucumber-options).

parallel_cucumber needs tag formatting update to match new tagging style of cucumber 3.0

New conditional operator tagging style in cucumber 3.0 allows user to set tags like --tags @abc and @def not @123 not @xyz

parallel_cucumber doesn't like this formatting and miscalculates the scenarios per process for total scenarios in the project rather the actual count scenarios that satisfies the tag conditions.

Process stdout and stderr separately

https://github.com/bayandin/parallel_cucumber/blob/master/lib/parallel_cucumber/runner.rb#L33

Is that possible to share data between different processes?

If we are using parallel cucumber to run features in parallel, is it possible to share data between different processes? Could we use Ruby global variable to achieve this? If not, how we could do it?

Use queue for better tests distribution

Support separated worker daemons

Requirements...

The 'test publisher' (i.e. parallel_cucumber) makes an ?http service available, through which workers can retrieve the application and test files - probably as zip archives. It publishes its address into the database's 'awaiting workers' table, receives results (again, probably as zips), and unpublishes itself once complete.

Worker-owners monitor the 'awaiting workers' table and somehow coordinate to share resources between 'awaiting workers' tasks, so that if there is only one task, all available suitable workers run it, but resources are evenly and dynamically allocated if additional tasks dis/appear - down to a minimum service level - e.g. min=4 phones. Unclear what happens if several tasks are already at minimum service, and a phone disappears: a task becomes underpowered or is suspended pending workers? Where is the minimum service level defined? What prevents or permits hogs demanding lots more?

Implies worker-owners also register with the database.

'Somehow' suggests a task-resource allocator.

By default, parallel_cucumber will launch its own worker-owner, task-resource allocator, assuming solitary operation and test-execution facilities.

Parallel Cucumber only outputs JUnit .xml for 1 of the scenarios in each feature file

I am running parallel from a Rakefile using parallel_cucumber. What am I doing wrong if I only get 1 JUnit file for 1 of the scenarios in my .feature file?

I think both of my threads are getting the same file-name (named by feature file path: TEST-features-the_internet-basics.xml ) and one over-writes the other?

I get a process_0.log and a process_1.log file in my root folder but the JUnit .xml output files are NOT named this way, ending with the thread id.

Here is my project: https://github.com/djangofan/watir-cuke-starter
Here is my test output (with 2 .feature files containing 3 scenarios: resulting in only 2 junit .xml output files):

$ bundle exec rake test_parallel
0> Custom env: TEST=1 TEST_PROCESS_NUMBER=0; command: cucumber --format junit --out reports/win10_chrome/junit/ --format pretty features/the_internet/basics1.feature:6
1> Custom env: TEST=1 TEST_PROCESS_NUMBER=1; command: cucumber --format junit --out reports/win10_chrome/junit/ --format pretty features/the_internet/basics2.feature:6
2> Custom env: TEST=1 TEST_PROCESS_NUMBER=2; command: cucumber --format junit --out reports/win10_chrome/junit/ --format pretty features/the_internet/basics2.feature:13
0> Pid: 14744
1> Pid: 7308
2> Pid: 23348
1> # features/the_internet/the_internet.feature
1> Feature: Test The Internet
0> # features/the_internet/the_internet.feature
0> Feature: Test The Internet
2> # features/the_internet/the_internet.feature
2> Feature: Test The Internet
    Given I am on the internet                                       # features/step_definitions/the_internet_steps.rb:3
2>     When I see a heading on the page                                 # features/step_definitions/the_internet_steps.rb:7
2>     Then I see the text 'Welcome' contained in the heading           # features/step_definitions/the_internet_steps.rb:12
2>     And show me the test setup                                       # features/step_definitions/common_steps.rb:1
    Given I am on the internet                             # features/step_definitions/the_internet_steps.rb:3
0>     When I see a heading on the page                       # features/step_definitions/the_internet_steps.rb:7
0>     Then I see the text 'Welcome' contained in the heading # features/step_definitions/the_internet_steps.rb:12
0>     And show me the test setup                             # features/step_definitions/common_steps.rb:1
2>       Test Setup: Selenium::WebDriver::Remote::Capabilities, WIN10, chrome, latest
2> 1 scenario (1 passed)
2> 4 steps (4 passed)
2> 0m9.216s
2> Report output at: reports/win10_chrome
2> Exited with status 0
Thread 2 has finished. Remaining(2): 0, 1
    Given I am on the internet                             # features/step_definitions/the_internet_steps.rb:3
1>     When I see a heading on the page                       # features/step_definitions/the_internet_steps.rb:7
1>     Then I see the text 'Welcome' contained in the heading # features/step_definitions/the_internet_steps.rb:12
1>     And show me the test setup                             # features/step_definitions/common_steps.rb:1
1 scenario (1 passed)Selenium::WebDriver::Remote::Capabilities, WIN10, chrome, latest
0> 4 steps (4 passed)
0> 0m9.464s
0> Report output at: reports/win10_chrome
0> Exited with status 0
Thread 0 has finished. Remaining(1): 1
1>       Test Setup: Selenium::WebDriver::Remote::Capabilities, WIN10, chrome, latest
1> 1 scenario (1 passed)
1> 4 steps (4 passed)
1> 0m9.817s
1> Report output at: reports/win10_chrome
1> Exited with status 0
Thread 1 has finished. Remaining(0):
All threads are complete
************ FINAL SUMMARY ************
2 scenarios (2 passed)
12 steps (12 passed)
Took 0 Minutes, 16.74 Seconds

Add tests

Is parallel_cucumber compatible with Cucumber 3.0.0 yet?

I get this error when I try to install (using Ruby 2.3 on Windows x64) the latest cucumber pre-release:

bundle update cucumber
Fetching gem metadata from https://rubygems.org/...........
Fetching version metadata from https://rubygems.org/..
Fetching dependency metadata from https://rubygems.org/.
Resolving dependencies...
Bundler could not find compatible versions for gem "cucumber":
  In Gemfile:
    cucumber (~> 3.0.0.pre.1) x64-mingw32

    cucumber (~> 3.0.0.pre.1)

    parallel_cucumber was resolved to 0.1.22, which depends on
      cucumber (~> 2) x64-mingw32

    parallel_cucumber was resolved to 0.1.22, which depends on
      cucumber (~> 2)

Any details on the change of ownership?

Hey,

I was keeping a close eye on how this developed and noticed the handoff from @bayandin. Will this lib remain public and active under badoo?

Cucumber 7 support

Hi, Is Parallel_cucumber supports cucumber v7? especially the ~tags(not tags) are not working.