mkorpela/pabot

Parallel executor for Robot Framework test cases.

pabot's Issues

Tests for pabot

Add tests for the tool

Parsing massive test case file takes lot of time

We are auto generating robot framework testcase through some automation scripts the generated robot framework testcase size approximately 50 MB, while i used to

pabot --name "Conformance Tests" --doc --timestampoutputs --splitlog --outputdir olt_reports generated

It took around 30 minutes to analyse the testcase after that it started actual execution and completed in 4 minutes

The number of lines in testcase approximately 4,00,000

Is there any methods to find why it took so much time analyse the testcase?

If not are there any methods to solve this issue?

The same issue we raised at robot-framework robotframework/robotframework#2231

Pabot -processes not working

Hi.

I'm trying to execute scripts in parallel on a folder. The problem that I have is when I run PABOT --processes 5 C:\Scripts\1\test.tsv displays No test to execute. I can execute the test using RIDE but not with Pabot. I'm missing something?

Thanks.

Execute last failed suites first

Tune alive message interval

Currently set to 15 seconds. Which is too often in cases where the execution takes multiple hours.

resource management

I've seen on the mailing list some messages that suggests that I'm not the only one to require a resource management.

I plan to implement and propose a pull request with the following feature:

• add a new option --resources filename.txt
• filename.txt would be made of several lines, one line per resource.
• each line would consist in a set of arguments that have to be provided to pybot.

For instance, if filename.txt is build like this:
--variable servername:server1
--variable servername:server2
--variable servername:server3

pabot --resources filename.txt TEST_DIR reduces the number of subprocesses to 3 and adds to the pybot arguments: '--variable servername:serverX' to the subprocess. It also ensures that only one subprocess at a time is using the same servername.

What do you think of this idea ?

result merger does not preserve suite setups and tear downs correctly

Dont work --name and --reporttitle options

By pybot I use command (for Jenkins):

pybot --suite check_availability --name Check_availability --loglevel INFO --outputdir %WORKSPACE%\output\release_stand --output availability_output.xml --log availability_log.html --report availability_report.html --exclude develop --reportbackground white:white:white --monitorwidth 150 --reporttitle "Availability tests" --noncritical noncritical --randomize suites --removekeywords FOR --removekeywords WUKS %WORKSPACE%\test\release_stand\test & exit 0

I have log.html with title and header "Check availability" and parent suite start with "Check availability".

If I run tests by pabot:

pabot --suite check_availability --name Check_availability --loglevel INFO --outputdir %WORKSPACE%\output\release_stand --output availability_output.xml --log availability_log.html --report availability_report.html --exclude develop --reportbackground white:white:white --monitorwidth 150 --reporttitle "Availability tests" --noncritical noncritical --randomize suites --removekeywords FOR --removekeywords WUKS %WORKSPACE%\test\release_stand\test & exit 0

I have log.html with full path to suites

warning messages with robotframework 2.9

Hi @mkorpela,

I have updated robotframework to 2.9.

After that I am getting following warning messages when using pabot.

[ WARN ] Option '--monitorcolors' is deprecated. Use '--consolecolors' instead.
[ WARN ] Option '--monitormarkers' is deprecated. Use '--consolemarkers' instead.

Looks like parameter names are changed in robotframework 2.9

Thanks & Regards,
Sachin Nikam

Unable to re-run failed testcases

HI All,

Need help with this issue. I have made changes suggested by you but still facing the issue.

I have a test suite with name test. My pybot command is :

pybot --name "Test" --loglevel INFO --outputdir G:\workspace\output --output output.xml --log log.html --report report.html --reporttitle "Robot test" g:\workspace\test

g:\workspace\test is the path of my main suite.

This works fine.

pybot --name "Test" --loglevel INFO --outputdir G:\workspace\output --output output.xml --log log.html --report report.html --reporttitle "Robot test" g:\workspace\test

which gives me output as below

Under Statistics by Suite section, it adds Test.Test due to which if there is any failure in the suite I am unable to re-run those cases.

I got following error:

G:>pybot -R g:\workspace\output\output.xml g:\workspace\test
[ ERROR ] Suite 'Test' contains no tests named 'Test.Test.Main suite3.Child suite3.test'.

@dmizverev @mkorpela Can you please let me know what is wrong with my execution.

Thanks & Regards,
Sachin Nikam

Pabot is broken with RF 3.0

Launching test suite with pabot fails immediately.

python: 2.7.11
pabot: 0.20
robotframework: 3.0 beta1

Traceback (most recent call last):
  File "/Users/USER/robot_venv/venv/bin/pabot", line 3, in <module>
    from pabot.pabot import main
  File "/Users/USER/robot_venv/venv/lib/python2.7/site-packages/pabot/pabot.py", line 40, in <module>
    from result_merger import merge
  File "/Users/USER/robot_venv/venv/lib/python2.7/site-packages/pabot/result_merger.py", line 22, in <module>
    from robot.result.testsuite import TestSuite
ImportError: No module named testsuite

add a --deterministic option

I'm going to implement a --deterministic option. Before, I would appreciate your opinion on this idea.

I would like that, when launched several times, pabot distributes the test suites in a deterministic manner.

This will require to add a new distribution function that split the list of suite names into N parts and then launch N processes, one for each part.

What do you think ?

Pabot issues with running failed testcases.

HI @dmizverev,

Apologies for filling this new issue. I was unable to reopen the issue #10 so filling this one.

I solved the problem while re-running the failed testcases by using --name and --suite variables. However when I tried to merge the re-run result with first run result I am getting the same issue which you have mentioned in the issue #10 .

I observed output.xml from both runs and found that in first run has following line for suite name.

However in re-run adds one additional line:

I guess this is causing the problem. It would be great if you could have a look at it.

Thanks & Regards,
Sachin Nikam

Parallel Distributing TestSuite Execution Remotely

Add a parameter (hostsfile) for in parallel distributing testsuite execution on remote machines

Run long suites first

If small proportion of suites is big part of the execution time, this should have a very good performance gain.

Invalid syntax error when using pabot with Python 2.6.9

We've been successfully using pabot with 2.7.5, but recently we have a server which we can't upgrade and is using 2.6.9.

The following is the output:

Traceback (most recent call last):
File "/Users/ramses/od-tests/odvenv2.6/bin/pabot", line 3, in
from pabot.pabot import main
File "/Users/ramses/od-tests/odvenv2.6/lib/python2.6/site-packages/pabot/pabot.py", line 58
with open(os.path.join(outs_dir, 'stdout.txt'), 'w') as stdout,
^
SyntaxError: invalid syntax

Again, we've done some debugging and the same code works fine on 2.7.5.

Enhancement: Post-process ALL .png files and references, not just selenium-screenshot-*.png

Some Robot Framework libraries derived from Selenium2Library capture and log their own screenshots in their own ways. These are not currently handled by pabot and result_merger which only handle screenshot files matching the pattern: selenium-screenshot-*.png.

See the attached image for an example of the multiple screenshots my RF library puts into the logs:

Improve command line output

• Status for each test suite execution should be shown in a more user-friendly way ( ON GOING / PASSED / FAILED etc )
• Improve ability to kill certain test execution (process PID should be present at this time)

Add run setup only once style keyword

Not ending execution

I'm sure you know this already but test execution never ends (at least not for me), which of course can be problematic when running through TeamCity

Great tool, by the way, been looking for robot parallel execution for months

Can't call --name from inside a config file

Hi guys,

Thank you for the great work on pabot. Thank you, really.

Introduction

On pybot, I have a config file looking like this:

--variable env:test
--variable platform:chrome
--name A Test
tests/PublicSite/

Then I call:

$ pybot --argumentfile globals/scenarios/test

This is a simple example, but it is a very nice way to define which tests to run against which scenarios, all inside a single (or multiple) config files. That way I have well defined and clean command line calls.

The Issue

That doesn't work on pabot (at least for me). Here is the error:

$ pybot --argumentfile globals/scenarios/test
get_suite_names: output_file='./pabot_results/suite_names.xml' does not exist
No tests to execute
Elapsed time: 0 minutes 0.12 seconds

Please let me know if you are in need of any other information. I would be more than happy to help.

Thank you all!

pabot.PabotLib.Get Value conflicts with Selenium2Library

Hi Mikko,

In my testcases, I am using Get Value keyword from selenium2library. I found that in new version of pabot there is a keyword with same name. Due to which my testcases starts failing.

Found: 'Selenium2Library.Get Value' and 'pabot.PabotLib.Get Value'

To fix this i had to create alias for the resp. library.

Not sure if we can consider this as a bug.

Run same suite with different resource file args

Get test names from listener in dryrun

Currently must wait for the dryrun to end - could improve execution time when parsing a lot of tests.

Preserve criticality when merging output.xml

https://groups.google.com/d/msg/robotframework-users/mY4qIyFu0aw/SBd2W_KfyugJ

Now all pobot processes are killed after test execution

In my windows 7 environment[1], when I run test with Selenium2Library, there is one pabot python process left hanging after the test has been executed.

The problem can be reproduced with this test suite: https://www.dropbox.com/s/71jsy33mt563ib1/suites.zip?dl=0
When that suite is run, I see this in my process explorer:

[1]: c:\workspace\pabot_example>pybot --version
Robot Framework 2.8.7 (Python 2.7.6 on win32)

Web dashboard

Show realtime test execution on a webpage

Warning for missing resource file

If the resource file is not found in the code we just silently continue.
In pabotlib.py we could add some simple logging that the resource file cannot be found. This would make more sense when the 'resourcefile' is not None.

In my opinion if the resource file is not set to empty and is not found, we should throw an exception, but if this is not wanted a mere warning is enough.

options not recognized

When I run for example:

pabot --verbose .

I get this error:

Traceback (most recent call last):
  File "/usr/local/bin/pabot", line 6, in <module>
    main(sys.argv[1:])
  File "/Library/Python/2.7/site-packages/pabot/pabot.py", line 356, in main
    options, datasources, pabot_args = _parse_args(args)
  File "/Library/Python/2.7/site-packages/pabot/pabot.py", line 190, in _parse_args
    options, datasources = ArgumentParser(USAGE, auto_pythonpath=False, auto_argumentfile=False).parse_args(args)
  File "/Library/Python/2.7/site-packages/robot/utils/argumentparser.py", line 149, in parse_args
    opts, args = self._parse_args(args)
  File "/Library/Python/2.7/site-packages/robot/utils/argumentparser.py", line 179, in _parse_args
    raise DataError(err.msg)
robot.errors.DataError: option --verbose not recognized

The same happens with all command-line options listed in readme, for example with --processes.

pabot -h results in stacktrace before dumping pybot help output

When I did a "pabot -h" I got the following. Should pabot offer its own help first and then maybe dump out pybot's help screen?

pabot --help
Elapsed time: 0 minutes 0.15 seconds
Traceback (most recent call last):
File "/usr/local/bin/pabot", line 6, in
main(sys.argv[1:])
File "/Library/Python/2.7/site-packages/pabot/pabot.py", line 236, in main
options, datasources, pabot_args = _parse_args(args)
File "/Library/Python/2.7/site-packages/pabot/pabot.py", line 135, in _parse_args
options, datasources = ArgumentParser(USAGE, auto_pythonpath=False, auto_argumentfile=False).parse_args(args)
File "/Library/Python/2.7/site-packages/robot/utils/argumentparser.py", line 132, in parse_args
opts, args = self._handle_special_options(opts, args)
File "/Library/Python/2.7/site-packages/robot/utils/argumentparser.py", line 142, in _handle_special_options
self._raise_help()
File "/Library/Python/2.7/site-packages/robot/utils/argumentparser.py", line 329, in _raise_help
raise Information(msg)
robot.errors.Information: Robot Framework -- A generic test automation framework

Version: 2.8.4 (Python 2.7.5 on darwin)

Usage: pybot|jybot|ipybot [options] data_sources
or: python|jython|ipy -m robot.run [options] data_sources
or: python|jython|ipy path/to/robot/run.py [options] data_sources
or: java -jar robotframework.jar run [options] data_sources

Screenshot path is wrong when using Selenium2Library's 'Set Screenshot Directory' keyword

I am using Selenium2Library's 'Set Screenshot Directory' keyword to set different path for screenshots in all tests.

Set Screenshot Directory  robot_screenshots  persist=True

Then I run couple of tests with Pabot and after I opened log file the path for screenshots is wrong.

test_suite_folder example_user$ pabot test1.robot test2.robot

# After that opened log.html file and tried to view screenshots
What it should be: file:///Users/example_user/test_suite_folder/robot_screenshots/selenium-screenshot-3.png
How it actually is: file:///Users/robot_screenshots/selenium-screenshot-3.png

OS X: 10.10.5
robotframework-pabot: 0.20
robotframework: 1.9.2
robotframework-selenium2library: 1.7.4

PABOTLIBURI variable gets repeatedly added to the underlying pybot command line options

Every time _options_for_executor method is executed another copy of the PABOTLIBURI variable gets added to the options. This results in very long pybot command lines with many redundant settings of the PABOTLIBURI variable.

Pictures doesn't show up in report and log files

In OSX when final report and log files are generated the log file has wrong file paths for pictures that RF has taken (or the pictures are not moved to right place with correct name). At the moment pictures are stored in pabot_results folder in own folders (in my case each test case has it's own subfolder inside pabot_results folder) and there pictures are named selenium-screenshot-1...n.png. In log.html file paths to pictures are referred to same directory where the html file is.

Online docs for PabotLib

error when used with -x (xunit output)

Hello, I'm trying to incorporate pabot into our CI process which demands xunit output, but I get this error:

$ pabot --processes 4 -x robot-xunit.xml mytests
[PID:7112] EXECUTING mytests.Test mytest
PASSED mytests.Test mytest
Elapsed time: 0 minutes 11.893 seconds
Traceback (most recent call last):
  File "~/env/bin/pabot", line 6, in <module>
    main(sys.argv[1:])
  File "~/env/lib/python2.7/site-packages/pabot/pabot.py", line 260, in main
    sys.exit(_report_results(outs_dir, options, start_time_string))
  File "~/env/lib/python2.7/site-packages/pabot/pabot.py", line 246, in _report_results
    merge(*sorted(glob(os.path.join(outs_dir, '**/*.xml')))).save(output_path)
  File "~/env/lib/python2.7/site-packages/pabot/result_merger.py", line 82, in merge
    merger.merge(ExecutionResult(result))
  File "~/env/lib/python2.7/site-packages/robot/result/resultbuilder.py", line 44, in ExecutionResult
    return _single_result(sources[0], options)
  File "~/env/lib/python2.7/site-packages/robot/result/resultbuilder.py", line 68, in _single_result
    raise DataError("Reading XML source '%s' failed: %s" % (unicode(ets), error))
robot.errors.DataError: Reading XML source './pabot_results/mytests.Test mytest/robot-xunit.xml' failed: Incompatible XML element 'testsuite'.

Not sure what the problem is here, am I missing something?

Doesn't pick outputdir from an argument file

Would be nice if there were a commandline param that would allow pabot_results to be created in a different location than ./pabot_results/

code contains a reference to GPL

LICENSE.txt contains the Apache license, but toward the end of pabot.py is the string "Copyright 2015 Mikko Korpela - GPLv3".

Pabot takes too long to start test in case of multiple files

Issue:
My project directories has six sub directories (which has sub directories in it), in which there are around 215 test suites files (there are equal number of resource files). When I run the test for it, it takes more then 107 minutes to start the execution. When I did the debugging I found that it was taking time on this line "run(_datasources, *_opts)" in function solve_suite_names in file pabot.py.

Some other help needed:
I have different script written in Java which does the same thing, and it starts immediately. Only reason I am coming to Pabot is that it combines all the reports file to one output.xml. Even If combning alone can work then can continue till this is fixed.

-Vinay

pabot in Jenkins does not give realtime console output

When we run pabot in Jenkins, it does not print any output to the console until the whole run is over. After that all the output is shown normally.

When I added --verbose to the command, I noticed that the extra outputs related to the verbose option get printed immediately:

[PID:12250] EXECUTING PARALLEL SUITE Tests.Accept Terms In Login with command:
pybot --variable PABOTLIBURI:127.0.0.1:8270 --outputdir "./pabot_results/Tests.Accept Terms In Login" --log NONE --consolecolors off --report NONE --consolemarkers off --xunit NONE --suite "Tests.Accept Terms In Login" .

When the last "EXECUTING PARALLEL..." output is printed, the non-verbose outputs won't appear until the run is finished. Maybe the verbose outputs are flushed differently than the normal output?

Screenshoots and delays between execution

Hi, I would like to know:

1. How to stop the creation of screenshoots.
2. How to add a delay between executions.

Thanks.

RF 3.0: New 'robot' start-up script and compatibility with older RF versions in the future

https://github.com/robotframework/robotframework/blob/master/doc/releasenotes/rf-3.0b1.rst#new-robot-start-up-script

New generic robot start-up script has been introduced to allow running tests like robot tests.robot regardless the Python interprete. Old interpreter specific pybot, jybot and ipybot scripts still work as earlier, but the plan is to deprecate and remove them in the future major releases.

This might be possible issue in the future if there is need to offer support for RF version 2.9 and older.

pabot calculates elapsed time wrong in Total Statistics

See the attached image. I'm able to repeat this every time.

Pabot fails when using asterisk wildcard in file names

Asterisk wildcard works fine when there are only few test scripts that matches but in following case there were 12 files matching to given argument for files to include and result was Assertion Error.

(venv) pabot regression-set/Example_Test_*
Elapsed time: 0 minutes 3.287 seconds
Traceback (most recent call last):
  File "/Users/USER/robot_venv/venv/bin/pabot", line 6, in <module>
    main(sys.argv[1:])
  File "/Users/USER/robot_venv/venv/lib/python2.7/site-packages/pabot/pabot.py", line 372, in main
    sys.exit(_report_results(outs_dir, options, start_time_string, _get_suite_root_name(suite_names)))
  File "/Users/USER/robot_venv/venv/lib/python2.7/site-packages/pabot/pabot.py", line 301, in _report_results
    merge(sorted(glob(os.path.join(outs_dir, '**/*.xml'))), options, tests_root_name).save(output_path)
  File "/Users/USER/robot_venv/venv/lib/python2.7/site-packages/pabot/result_merger.py", line 125, in merge
    assert len(result_files) > 0
AssertionError

pabot: 0.22
robotframework: 3.0

Document all keywords properly

Assert error when running pabot using variables.py with ;-separator

I noticed that when using --variablefile variables.py with ;-separator that quotes don't work the same way they work with pybot. When using either "- or '-quotes (i.e. "variables.py;arg1;arg2" or 'variables.py;arg1;arg2') pabot crashes with AssertError.

I have a simple test.robot:

(robot)# cat test.robot
*** TEST CASES ***
test0001
  Log To Console      ${TEST1}
  Log To Console      ${TEST2}

variables.py

(robot)# cat variables.py
def get_variables(arg1, arg2):
    variables = {   "TEST1": arg1,
                    "TEST2": arg2
                }
    return variables

my environment:

(robot)# python --version
Python 2.7.10
(robot)# pip list|grep robotframework
robotframework (2.9)
robotframework-pabot (0.16)

When I run the test with pybot:

(robot)# pybot --variablefile "variables.py;http://www.google.com;http://www.yahoo.com" test.robot
==============================================================================
Test
==============================================================================
test0001                                                              http://www.google.com
.http://www.yahoo.com
test0001                                                              | PASS |
------------------------------------------------------------------------------
Test                                                                  | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
Output:  /Users/jiuweigui/Examples/robot/output.xml
Log:     /Users/jiuweigui/Examples/robot/log.html
Report:  /Users/jiuweigui/Examples/robot/report.html

When I try to run the same thing with pabot:

(robot)# pabot --variablefile "variables.py;http://google.com;http://www.yahoo.com" test.robot
[PID:2310] EXECUTING Test
FAILED Test
Elapsed time: 0 minutes 0.454 seconds
Traceback (most recent call last):
  File "/Users/jiuweigui/.virtualenvs/robot/bin/pabot", line 6, in <module>
    main(sys.argv[1:])
  File "/Users/jiuweigui/.virtualenvs/robot/lib/python2.7/site-packages/pabot/pabot.py", line 354, in main
    sys.exit(_report_results(outs_dir, options, start_time_string, _get_suite_root_name(suite_names)))
  File "/Users/jiuweigui/.virtualenvs/robot/lib/python2.7/site-packages/pabot/pabot.py", line 286, in _report_results
    merge(sorted(glob(os.path.join(outs_dir, '**/*.xml'))), options, tests_root_name).save(output_path)
  File "/Users/jiuweigui/.virtualenvs/robot/lib/python2.7/site-packages/pabot/result_merger.py", line 116, in merge
    assert len(result_files) > 0
AssertionError
(robot)#

and 

(robot)# pabot --variablefile 'variables.py;http://google.com;http://www.yahoo.com' test.robot
[PID:2319] EXECUTING Test
FAILED Test
Elapsed time: 0 minutes 0.450 seconds
Traceback (most recent call last):
  File "/Users/jiuweigui/.virtualenvs/robot/bin/pabot", line 6, in <module>
    main(sys.argv[1:])
  File "/Users/jiuweigui/.virtualenvs/robot/lib/python2.7/site-packages/pabot/pabot.py", line 354, in main
    sys.exit(_report_results(outs_dir, options, start_time_string, _get_suite_root_name(suite_names)))
  File "/Users/jiuweigui/.virtualenvs/robot/lib/python2.7/site-packages/pabot/pabot.py", line 286, in _report_results
    merge(sorted(glob(os.path.join(outs_dir, '**/*.xml'))), options, tests_root_name).save(output_path)
  File "/Users/jiuweigui/.virtualenvs/robot/lib/python2.7/site-packages/pabot/result_merger.py", line 116, in merge
    assert len(result_files) > 0
AssertionError
(robot)#

However if I use both quotes everything passes and there's no assert:

(robot)# pabot --variablefile "'variables.py;http://google.com;http://www.yahoo.com'" test.robot
[PID:2328] EXECUTING Test
PASSED Test
Output:  /Users/jiuweigui/Examples/robot/output.xml
Log:     /Users/jiuweigui/Examples/robot/log.html
Report:  /Users/jiuweigui/Examples/robot/report.html
Elapsed time: 0 minutes 0.487 seconds
(robot)# cat pabot_results/Test/stdout.txt
==============================================================================
Test
==============================================================================
test0001                                                              http://google.com
http://www.yahoo.com
| PASS |
------------------------------------------------------------------------------
Test                                                                  | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
Output:  /Users/jiuweigui/Examples/robot/pabot_results/Test/output.xml
(robot)#

Based on the errors seen it looks like pabot interprets variables with ;-separator as shell commands and doesn't seem to honour quotes as pybot does. Using double and single quotes combined seems to solve the problem but is most likely not the intended way.

When `-d or --outputdir` not specified, pabot throws an error

When executing a robot test (e.g. pabot RobotTest.robot), the following error is generated:

Traceback (most recent call last):
  File "/usr/local/bin/pabot", line 6, in <module>
    main(sys.argv[1:])
  File "/usr/local/lib/python2.7/dist-packages/pabot/pabot.py", line 337, in main
    sys.exit(_report_results(outs_dir, options, start_time_string))
  File "/usr/local/lib/python2.7/dist-packages/pabot/pabot.py", line 278, in _report_results
    _copy_screenshots(options)
  File "/usr/local/lib/python2.7/dist-packages/pabot/pabot.py", line 265, in _copy_screenshots
    outputdir = options['outputdir']
KeyError: 'outputdir'

Pabot process keeps running for long time on jenkins

Hi,
I am using pabot to run automation test scripts for UI written in python + selenium. While running these scripts on jenkins, I found that most of the times a random process keeps running till timeout and after timeout ends in failure with BadStatusLine error. Unable to sort it out

Selenium2Library's screenshots are not copied properly

We had to proxy Selenium2Library so screenshot image files are put in the right place.

It seems pabot does try to copy these files, but they are not.

Python 2 and 3 compatible code

RF 3.0 is coming soon and the biggest change there is support for Python 3. Pabot should also add support for Python 3 (3.4 & 3.5).

--timestampoutputs breaks pabot

If you use the pybot option --timestampoutputs in the command line to pabot, it breaks pabot because the suite_names.xml file does not exist (suite_names_.xml does exist but pabot doesn't find it.