damies13/rfswarm

Robot Framework Swarm

rfswarm (Robot Framework Swarm)

Version	Manager	Agent	Reporter

Build Status

About

rfswarm is a testing tool that allows you to use Robot Framework test cases for performance or load testing.

Swarm being the collective noun for Robots, just as Flock is for Birds and Herd for Sheep, so it made sense to use swarm for a performance testing tool using Robot Framework, hence rfswarm

While Robot Framework is normally used for functional or regression testing, it has long been considered the holy grail in testing for there to be synergies between the functional and performance testing scripts so that effort expended in creating test cases for one does not need to be duplicated for the other which is currently the normal case.

rfswarm aims to solve this problem by allowing you to take an existing functional or regression test case written in Robot Framework, make some minor adjustments to make the test case suitable for performance testing and then run the Robot Framework test case with as many virtual users (robots) as needed to generate load on the application under test.

rfswarm is written completely in python, so if you are already using Robot Framework, then you will already have most of what you need to use rfswarm and will be familiar with pip to get any extra components you need.

To learn more about rfswarm please refer to the Documentation

Getting Help

Community Support

• rfswarm Documentation
• Discord
• Reporting Issues / Known Issues

An example of how your rfswarm setup might look.

Commercial Support

• The easiest way to get commercial support is to sponsor this project on GitHub

Contribute

If you'd like to help make rfswarm better the are a number of ways you can help (only the last two require programming skills)

• Update the Documentation, all the documentation is written in Markdown (GitHub flavour) so is very easy to update but time consuming so help with this will be very appreciated
• Testing
  • Automated testing, With v1.1.0 GitHub actions were setup to run robotframework test cases from the Tests/Regression folder of this repository, the tests can be either command-line tests or GUI tests using ImageHorizonLibrary, and the tests will run on Windows-latest, macos-latest & ubuntu-latest for all supported versions of python. The initial set of tests are very basic but gives a start point, contributing test cases will help make future versions of rfswarm more reliable and will be greatly appreciated.
  • Manual testing, any testing that finds bugs is appreciated, but manual testing is not reliably repeatable so automation is preferred
• Raise an Issue, yes finding a but is helpful!
• Create a Feature Request, if you have an idea you think will make rfswarm better, please let us know.
• Add a reaction to or comment on an Issue (especially Bugs and Feature Requests), adding a 👍 tells us you want this implemented too, and a 👎 lets us know you think it's a bad idea (please add a comment to for why). this will help for prioritising which issues get implemented next.
• Fix a bug, please assign the issue to you self or add a comment on the issue first, to prevent duplication of effort.
• Implement a Feature Request, please assign the issue to you self or add a comment on the issue first, to prevent duplication of effort.

Donations

If you would like to thank me for this project please consider using one of the sponsorship methods:

• GitHub
• PayPal.me (the $5 is a suggestion, feel free to change to any amount you would like)

See our sponsors

rfswarm's Issues

Don't prompt to save if scenario hasn't changed

Configure GUI/Server Port &| IP in ini file

Configure GUI/Server's http server Port and or the bind ip address in the ini file

updatenetpct errors if no nic's have a speed > 0

causes a crash dump on the agent, doesn't prevent the agent working otherwise.

Display 90 percentile on the run screen

Display 90 percentile on the run screen:

Actually I plan to add a percentile column to the Run results table the percentile will default to the 90th percentile but will be user configurable.

While some of the code require for this feature has already been added, unfortunately this functionality is not natively available in sqlite (which is used to store the results when they are sent from the agents). While there is a math extension to sqlite available this is not available in the python implementation of the sqlite driver and doesn't look like it is likely to be anytime (https://bugs.python.org/issue30952), so it looks like I will need to use a sqlite create_function() to do this.

Since v0.5.0 launching the GUI fails if there is no existing ini file

Describe the bug
Since v0.5.0 launching the GUI fails if there is no existing ini file

To Reproduce
Steps to reproduce the behavior:

1. delete the ini file
2. launch GUI

Expected behavior
Application launches and creates new ini file

Additional context

Robot Framework Swarm: GUI/Server
Version v0.5.0-beta
Traceback (most recent call last):
File "rfswarm.py", line 3885, in
core = RFSwarmCore()
File "rfswarm.py", line 1090, in init
base.gui = RFSwarmGUI()
File "rfswarm.py", line 1856, in init
self.BuildUI()
File "rfswarm.py", line 1993, in BuildUI
self.BuildRun(r)
File "rfswarm.py", line 3273, in BuildRun
self.display_run['display_index'].set(base.str2bool(base.config['Run']['display_index']))
File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/configparser.py", line 958, in getitem
raise KeyError(key)
KeyError: 'Run'

Command line switches for CI/CD

Add Command line switches to both the GUI/Server and Agent to enable integration with CI/CD build processes (and for any other automation uses of rfswarm.

Command line switches probably needed for the GUI/Server:

• Scenario File
• Auto play (so the user can decide to automatically or manually start the test)
• Wait for n Agents (for use with auto play)
• Hide GUI (for use with auto play)

Command line switches probably wanted for the Agent (these would simply override the default values):

• rfswarm host name
• agent name

If you think of any other please add your suggestions below

rare bug where robot count doesnt decrement

Describe the bug
In rare occurrences if robot crashed the robot count would not get decremented, so rfswarm would report more running robots than actually running and would not get back to zero

To Reproduce
Actually quite hard to reproduce, I can't remember what I did to make robot crash regularly but it was a syntax issue of some type in my robot file

Expected behavior
when robot stops running for any reason the count of running robots should be decremented

Handle *** Test Case as well as *** Test Cases

On Tuesday, 5 November 2019 22:29:29 UTC+10, Pekka Klärck wrote:
ti 5. marrask. 2019 klo 14.04 Dave Amies [email protected] kirjoitti:

1. Is *** Test Case *** valid in robot? if it is then I will need to fix that.

Yes it is. The format is perhaps too flexible:

• Both "Test Cases" and "Test Case" are allowed.
• In RPA mode also "Tasks" and "Task" are OK.
• It's case-insensitive.
• Spaces between the text and asterisks are optional.
• Only one leading asterisk is mandatory. Others are optional and there can even be more that three.
• There can be other content on the same line after this header. Just need to separate it with two or more spaces.

Scenario not loading properly on rfswarm launch

Scenario not loading properly on rfswarm launch

From the screen shot, scenario 5u_test is loaded, but the script and test are not selected.

Scenario Files use local path for scripts should be relative path

Describe the bug

The scenario files use the local path (non relative / full path) to the scripts (.robot) files, when opening these scenarios on another computer (esp. if the OS is different) the scenario will not load the script files correctly.

To Reproduce
Steps to reproduce the behavior:

1. In rfswarm create a scenario with at least 1 .robot script
2. Save the scenario
3. edit the scenario file (.rfs) and check the script = lines

Expected behavior
Path to the script should be relative to the .rfs file

Additional context
When opening these scenarios on another computer (esp. if the OS is different) the scenario will not load the script files correctly.

Run robot scripts in separate "Sessions"

Describe the solution you'd like
Run robot scripts in separate "Sessions" so you can run multiple GUI robots on the same agent machine (Sikuli, White, AutoIt, etc)

Additional context
On Windows this will probably look like RunAs (Win 7 +?) on Linux (unix?) systems this might be su -u or even creating a separate XWindows session, No idea what this will look like on MacOS yet.

More investigation is going to be needed to determine if this is possible and will probably need do use different strategies for different os's

Problem with test case name

To get things to work on Windows 10 with Python 3.7.5 I had to change line 441 in rfswarm_agent.py from cmd.append("'"+test+"'") to cmd.append('"'+test+'"'). Single quotes around the test case name in the Robot command line weren't working - they need to be double quotes for Windows.

libraries whose results are excluded

Ok another thing I need to document, the agent doesn't return results for the following libraries:
["BuiltIn", "String", "OperatingSystem", "perftest"]

Also allow the library list to be configured in the agent ini file.

try getting agent to use listener-interface to collect results and return to gui/server

Hopefully this will allow the GUI/Server to get results as the the cases are progressing instead of waiting until the end of an iteration.

https://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#listener-interface
https://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#listener-examples

currently the agent simply parses the output.xml file.

Display stdev on the run screen

Display standard deviation on the run screen

report in html format

For this feature to be implemented I need some idea of the number of people who want this functionality and what information is needed in this type of report.

Some ideas could be:

• summary table
• running users graph
• response times graphs
  • broken down by test case?

This would be in addition to what is already provided by Issue #17

Redirect robot stdout and stderr to log file to make diagnostic easier

Redirect robot stdout and stderr to log file to make diagnostic easier

Document how the rampup process works and why a rampup time should be more that 10 seconds

Ok something else I need to document, rampups of less than 10 seconds are pretty meaningless as the agent only polls the server once every 10 seconds until a test is assigned, then it polls every 0.1 seconds, so the agent probably kicked them both off at the same time.

Unable to open rfswarm.py (GUI / Server)

Receive the error below after running python rfswarm.py:

C:\Users\raphaelr\Desktop\rfswarm-master>python rfswarm.py
Traceback (most recent call last):
File "rfswarm.py", line 2690, in
rfs = RFSwarmGUI()
File "rfswarm.py", line 355, in init
self.BuildUI()
File "rfswarm.py", line 430, in BuildUI
self.BuildPlan(p)
File "rfswarm.py", line 577, in BuildPlan
self.pln_graph.pack(fill="both", expand=True)
File "C:\Python37\lib\tkinter_init_.py", line 2143, in pack_configure
+ self._options(cnf, kw))
_tkinter.TclError: cannot use geometry manager pack inside .!rfswarmgui.!notebook.!frame which already has slaves managed by grid

Running Python 3.7 on Windows 10, have configparser==4.0.2, requests==2.22.0 and psutil==5.6.5 installed.

Enhancements to the listner

Enhancements already done, just catching up

• added debugging option, passes through the debugging level from the rfswarm_agent
• now picks up the documentation string for the transaction name as a fall back if there is no info string to use. This is important so you can get timings for your keywords, as previously there no way to make them get timings.

Cannot save/load a scenario

As Title: Cannot save/load a scenario

This is a planned feature but hasn't been implemented yet.

Group/ungroup results buy test index, test iteration or step sequence

Add the ability to group/ungroup results buy test index, test iteration or step sequence.

This is a planned feature but hasn't been implemented yet.

GUI/Server results directory and db file name still Scenario_<TS>

GUI/Server results directory and db file name still Scenario_ (e.g. Scenario_1574839662), 'Scenario' is supposed to be replaced with the scenario file name if run from a previously saved scenario.

I also noticed that a new file is not being generated when play is clicked again after the first test stops, each test should have it's own result file.

Document GUI Features and units

All the time units in the GUI are in seconds, I need to document this too.

Agent Groups

Is your feature request related to a problem? Please describe.

• when testing an application with some external and some internal agents
• when testing with a mix of robot libraries and some agent machines only support one library type.
• having a mix of agent os's, i.e. want the SeleniumLibrary scripts to run on any but the WhiteLibrary scripts should only be assigned to windows agents

Describe the solution you'd like

Ability to assign Agent Groups and select which scripts run on which groups

After a period of time Agent runs out of connections to GUI/Server

After a period of time, the requests library in the Agent appears to run out of connections to GUI/Server, at least that is what I currently believe is happening.

Will try to fix this by creating a session object for the connection to the GUI/Server, then all calls can use the session.

Might need to set keep_alive to false
(https://stackoverflow.com/questions/10115126/python-requests-close-http-connection/15511852)

RFSwarmAgent: mainloop: Running 2019-11-12 21:51:38 ( 1573559498 )isrunning: False isconnected: True
RFSwarmAgent: mainloop: Running 2019-11-12 21:51:48 ( 1573559508 )isrunning: False isconnected: True
RFSwarmAgent: mainloop: Running 2019-11-12 21:51:58 ( 1573559518 )isrunning: False isconnected: True
getscripts: Exception: HTTPConnectionPool(host='davesmbpsg.fritz.box', port=8138): Max retries exceeded with url: /Scripts (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f9d88a10390>: Failed to establish a new connection: [Errno -2] Name or service not known',))
getjobs: Exception: HTTPConnectionPool(host='davesmbpsg.fritz.box', port=8138): Max retries exceeded with url: /Jobs (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f9d8819b3c8>: Failed to establish a new connection: [Errno -2] Name or service not known',))
Exception in thread Thread-117:
Traceback (most recent call last):
  File "/home/dave/.local/lib/python3.6/site-packages/urllib3/connection.py", line 157, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/home/dave/.local/lib/python3.6/site-packages/urllib3/util/connection.py", line 61, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/lib/python3.6/socket.py", line 745, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/dave/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 672, in urlopen
    chunked=chunked,
  File "/home/dave/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 387, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.6/http/client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1285, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1234, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1026, in _send_output
    self.send(msg)
  File "/usr/lib/python3.6/http/client.py", line 964, in send
    self.connect()
  File "/home/dave/.local/lib/python3.6/site-packages/urllib3/connection.py", line 184, in connect
    conn = self._new_conn()
  File "/home/dave/.local/lib/python3.6/site-packages/urllib3/connection.py", line 169, in _new_conn
    self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f9d8819b3c8>: Failed to establish a new connection: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/dave/.local/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/home/dave/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 720, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/home/dave/.local/lib/python3.6/site-packages/urllib3/util/retry.py", line 436, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='davesmbpsg.fritz.box', port=8138): Max retries exceeded with url: /Jobs (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f9d8819b3c8>: Failed to establish a new connection: [Errno -2] Name or service not known',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "rfswarm_agent.py", line 318, in getjobs
    r = requests.post(uri, json=payload)
  File "/home/dave/.local/lib/python3.6/site-packages/requests/api.py", line 116, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/home/dave/.local/lib/python3.6/site-packages/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/dave/.local/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/dave/.local/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/home/dave/.local/lib/python3.6/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='davesmbpsg.fritz.box', port=8138): Max retries exceeded with url: /Jobs (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f9d8819b3c8>: Failed to establish a new connection: [Errno -2] Name or service not known',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "rfswarm_agent.py", line 361, in getjobs
    print("getjobs: resp: ", r.status_code, r.text)
UnboundLocalError: local variable 'r' referenced before assignment

RFSwarmAgent: mainloop: Running 2019-11-12 21:52:08 ( 1573559528 )isrunning: False isconnected: False
RFSwarmAgent: connectserver: Try connecting to http://DavesMBPSG.fritz.box:8138/
getjobs: Exception: HTTPConnectionPool(host='davesmbpsg.fritz.box', port=8138): Max retries exceeded with url: /Jobs (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f9d8c087a20>: Failed to establish a new connection: [Errno -2] Name or service not known',))
getscripts: Exception: HTTPConnectionPool(host='davesmbpsg.fritz.box', port=8138): Max retries exceeded with url: /Scripts (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f9d88a36dd8>: Failed to establish a new connection: [Errno -2] Name or service not known',))
Exception in thread Thread-122:
Traceback (most recent call last):
  File "/home/dave/.local/lib/python3.6/site-packages/urllib3/connection.py", line 157, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/home/dave/.local/lib/python3.6/site-packages/urllib3/util/connection.py", line 61, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/lib/python3.6/socket.py", line 745, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/dave/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 672, in urlopen
    chunked=chunked,
  File "/home/dave/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 387, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.6/http/client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1285, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1234, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1026, in _send_output
    self.send(msg)
  File "/usr/lib/python3.6/http/client.py", line 964, in send
    self.connect()
  File "/home/dave/.local/lib/python3.6/site-packages/urllib3/connection.py", line 184, in connect
    conn = self._new_conn()
  File "/home/dave/.local/lib/python3.6/site-packages/urllib3/connection.py", line 169, in _new_conn
    self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f9d8c087a20>: Failed to establish a new connection: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/dave/.local/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/home/dave/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 720, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/home/dave/.local/lib/python3.6/site-packages/urllib3/util/retry.py", line 436, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='davesmbpsg.fritz.box', port=8138): Max retries exceeded with url: /Jobs (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f9d8c087a20>: Failed to establish a new connection: [Errno -2] Name or service not known',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "rfswarm_agent.py", line 318, in getjobs
    r = requests.post(uri, json=payload)
  File "/home/dave/.local/lib/python3.6/site-packages/requests/api.py", line 116, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/home/dave/.local/lib/python3.6/site-packages/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/dave/.local/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/dave/.local/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/home/dave/.local/lib/python3.6/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='davesmbpsg.fritz.box', port=8138): Max retries exceeded with url: /Jobs (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f9d8c087a20>: Failed to establish a new connection: [Errno -2] Name or service not known',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "rfswarm_agent.py", line 361, in getjobs
    print("getjobs: resp: ", r.status_code, r.text)
UnboundLocalError: local variable 'r' referenced before assignment

Document how scripts get assigned to agents

The function get_next_agent() is where the agent selection happens, it sorts the agents by number of assigned robots and takes the agent with the least assigned robots, if the select agent has more than 10 robots, then it ignores that selection, sorts the agents by load and take the agent with the least load.

I designed it like this to cope 2 scenarios:

1. your scenario, so low number of virtual users we just want to round robin
2. with higher numbers of virtual users and agents of differing hardware resources, we want to distribute the generation load more to the more powerful machines and less to the less powerful machines.

Actually Load generator assignment is something LoadRunner does quite poorly and using this algorithm I came up with method I hope rfswarm will be better in this area.

Export results from run screen

Method to export results from the run screen.

This is important as the sqlite3 db the results are stored in doesn't have a percentile function so the percentile is done as a python user-defined function.

GUI/Server window layout (resize and scrolling)

Window layout in the GUI/Server needs to be improved, some items like the graphs don't resize with the window and some items like the results table and agent list will need scroll bars as the number of data items gets large causing the window to grow larger than the screen.

• Plan
  • plan graph should auto resize, h=50% and w=100%
  • test list frame should auto resize, h=50% and w=100%, test list inside frame should scroll if list overflows window size
• Run
  • test result frame should auto resize, h=90%? and w=100%, test results inside frame should scroll if result list overflows window size, result column headings should not scroll out of view
• Agents
  • agent list frame should auto resize, h=100% and w=100%, agent list inside frame should scroll if agent list overflows window size, agent list column headings should not scroll out of view

Don't create scenario files that crash rfswarm on launch

Describe the bug
Scripts are not re-indexed properly when a script is deleted.

Perhaps there is a better way to deal with this, but the scenario file below crashes rfswarm GUI even though the GUI created it.

Perhaps the scriptcount item in the scenario section could be renamed as scriptlist and contain a list of script id's, the id's could then be GUID's.

To Reproduce
Steps to reproduce the behavior:

1. create new scenario
2. Click on '+' button to add 2 script rows
3. assign scripts to each row
4. delete rows 1 & 2
5. save scenario

Expected behavior

• Script count should be 1 not 3
• section heading for only script should not be 3

Additional context

[Scenario]
scriptcount = 3

[3]
users = 200
delay = 0
rampup = 1800
run = 7200
test = Data cycle test
script = /Users/dave/Documents/GitHub/TestDataTable/Regression_Tests/TestDataTable-API_requests_perf.robot

Pretty graphs on the Run screen

Add pretty graphs on the Run screen (like the one on the plan screen) for a visual display of the response times etc.

This is a planned feature, however it requires some investigation, I am concerned that constructing and constantly refreshing a graph with many data points on it using the same method as was used to create the plan graph will be quite CPU intensive and likely to make the application unstable. I need to find a better way of creating these graphs so I don't expect to add this feature until application is more stable and more important features have been added and I can be sure I won't be compromising application stability.

Change file hashing and referencing

Is your feature request related to a problem? Please describe.
I have been contemplating this for a while but in working on the fix for Issue #52 I Identified the importance of this.

Describe the solution you'd like
Currently the file hash is just a hash of the file content, the hash should also take into account the:

• relative file path
• the date and time the file was last modified, remove the old hashes for the same (relative) file.

Additional context

• if a file is modified the previous hashes for the same (relative) file should be removed
• if a file is referenced in a .robot file and it is in the same folder as the .robot file, it will be downloaded to the scripts folder with the .robot file, but if it is also referenced by a relative path to an included resource file that is in another folder then the resource file won't be found consider this example:

On rfswarm gui machine:

file path	reference
dir1/main.robot	import1.robot
	../dir2/import2.robot
dir1/import1.robot
dir2/import2.robot	../dir1/import1.robot

On rfswarm agent machine:

file path	exists
scripts/main.robot	yes
scripts/import1.robot	yes
dir2/import2.robot	yes
dir1/import1.robot	No

scripts/main.robot fails to load, because dir2/import2.robot fails to load, because cannot find dir1/import1.robot

dependancy scan needs to be recursive

Describe the bug
From a discussion with, Nallavan G [email protected]:

So you understand what I think is happening, when you select the main .robot file the rfswarm GUI/Server scans the .robot file for and dependent files (Resource & Variable files) and includes them for transfer to the agent, but I don't think it then scans the dependent files for more dependent files.

To Reproduce
Steps to reproduce the behavior:

1. (File A) create .robot file with a dependent .robot resource file
2. (File B) create the .robot file that File A is dependent on, with another dependent .robot resource file
3. (File C) create the .robot file that File B is dependent on.
4. Select File A in rfswarm
5. Select test case from File A

Expected behavior
Files A, B & C should be downloaded to the agent machine

Iteration counter not resetting to zero after test finished

when you run several tests on the agent without restarting the agent, the iteration counts start from the last test +1

First 2 users go to the same agent

User Report:

I planned a run with two users and set up two agents, one on the localhost and one on a remote machine. The system consistently assigned two users to one of the machines and zero to the other. The machine that got the users was the first machine in the table of agents as displayed on the agents tab on the server UI.

I have been able to reproduce this issue and I now understand what is going on.

In the meantime, if you slow down your rampup so that there is >15-20 seconds per user, it should work as intended.

what is going on is the robot gets assigned it takes up to 10 seconds from when it was assigned till the agent launches the first robot, then it takes up to 10 seconds from launching the robot for the agent to report back it's count of running robots, so in the up to 20 seconds the agents robot count on the server is still zero.

To fix this add an assigned count to the agents and sorting on that instead.

Code Cleanup

There are some unused functions that need to be removed and some print statements that should be commented out (init)

UpdateAgents KeyErrors

No actual impact on the application working, just an ugly error message we should try to clean up

Exception in thread Thread-17:
Traceback (most recent call last):
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "rfswarm.py", line 2403, in UpdateAgents
    self.display_agents[rnum]["CPU%"].set("  {}  ".format(self.Agents[agnt]["CPU%"]))
KeyError: 'CPU%'



Exception in thread Thread-25:
Traceback (most recent call last):
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "rfswarm.py", line 2405, in UpdateAgents
    self.display_agents[rnum]["NET%"].set("  {}  ".format(self.Agents[agnt]["NET%"]))
KeyError: 'NET%'

UpdateAgents - RuntimeError: dictionary changed size during iteration

No actual impact on the application working, just an ugly error message we should try to clean up

Exception in thread Thread-14:
Traceback (most recent call last):
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "rfswarm.py", line 2372, in UpdateAgents
    for agnt in self.Agents.keys():
RuntimeError: dictionary changed size during iteration

find_dependancies downloads commented out dependancies

find_dependancies downloads commented out dependancies, should only download dependancies files on lines that are not commented out

Agent throws exception when GUI/Server is shutdown

handle the exception cleanly

RFSwarmAgent: mainloop: Running 2019-11-25 15:28:17 ( 1574659697 )isrunning: False isconnected: True
RFSwarmAgent: mainloop: Running 2019-11-25 15:28:27 ( 1574659707 )isrunning: False isconnected: True
RFSwarmAgent: mainloop: Running 2019-11-25 15:28:37 ( 1574659717 )isrunning: False isconnected: True
RFSwarmAgent: mainloop: Running 2019-11-25 15:28:47 ( 1574659727 )isrunning: False isconnected: True
RFSwarmAgent: mainloop: Running 2019-11-25 15:28:57 ( 1574659737 )isrunning: False isconnected: True
getjobs: Exception: HTTPConnectionPool(host='localhost', port=8138): Max retries exceeded with url: /Jobs (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x1059167d0>: Failed to establish a new connection: [Errno 61] Connection refused'))
getscripts: Exception: HTTPConnectionPool(host='localhost', port=8138): Max retries exceeded with url: /Scripts (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x1058628d0>: Failed to establish a new connection: [Errno 61] Connection refused'))
Exception in thread Thread-30:
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 160, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw)
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 80, in create_connection
    raise err
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 70, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 603, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 355, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1244, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1290, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1239, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1026, in _send_output
    self.send(msg)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 966, in send
    self.connect()
  File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 183, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 169, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x1059167d0>: Failed to establish a new connection: [Errno 61] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 641, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 399, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=8138): Max retries exceeded with url: /Jobs (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x1059167d0>: Failed to establish a new connection: [Errno 61] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "rfswarm_agent.py", line 318, in getjobs
    r = requests.post(uri, json=payload)
  File "/usr/local/lib/python3.7/site-packages/requests/api.py", line 116, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=8138): Max retries exceeded with url: /Jobs (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x1059167d0>: Failed to establish a new connection: [Errno 61] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "rfswarm_agent.py", line 361, in getjobs
    print("getjobs: resp: ", r.status_code, r.text)
UnboundLocalError: local variable 'r' referenced before assignment

report in word format

For this feature to be implemented I need some idea of the number of people who want this functionality and what information is needed in this type of report.

Some ideas could be:

• summary table
• running users graph
• response times graphs
  • broken down by test case?

This would be in addition to what is already provided by Issue #17

Version Numbers

Need to update version numbers in rfswarm.py and rfswarm_agent.py so that people can know what version they are using.

dependancy check needs to download custom libraries

Describe the bug

From a discussion with, Nallavan G [email protected]:

Library | ../CustomLibrary/ExcelUtilities.py
This one is obvious, rfswarm never even tried to copy this one across because it's a Library line, I never even considered custom libraries.

As a workaround to make this file get transferred you can add the line:
Metadata | File | ../CustomLibrary/ExcelUtilities.py
to your .robot file, anywhere in the *** Settings section
you don't need to worry about putting in the "Metadata | File" sequence Robot Framework will just store the file name as a metadata variable but because your test doesn't use that metadata value noting else will happen with it in the test, but putting it there will tell the server to tell the agent to download the file to the correct path

To Reproduce
Steps to reproduce the behavior:

1. Create a .robot file with a Library entry in the *** Settings section that is a custom library
2. Select File A in rfswarm
3. Select test case from File A

Expected behavior
custom library should be downloaded to the rfswarm agent

Additional context
I have given some thought about how we identify if a library is "custom" or not, so what I have come up with is:

• if library string has an extention (e.g. .py) then consider it a custom library in the current path.
• if library string contains a path, I guess we call this a / or ?
• maybe we can skip both of those and simply check if the library string referenced exists as a file relative to the current robot file?

Improve error handling when agent can't read the _output.xml file

RFSwarmAgent: mainloop: Running 2019-11-25 10:33:27 ( 1574658207 )isrunning: True isconnected: True
[ ERROR ] Parsing 'World'' failed: File or directory to execute does not exist.

Try --help for usage information.
Exception in thread Thread-85:
Traceback (most recent call last):
File "C:\Users\ROHITKUM\AppData\Local\Programs\Python\Python37-32\lib\threading.py", line 917, in _bootstrap_inner
self.run()
File "C:\Users\ROHITKUM\AppData\Local\Programs\Python\Python37-32\lib\threading.py", line 865, in run
self._target(*self._args, **self._kwargs)
File "C:/Users/ROHITKUM/PycharmProjects/Python_rf1/rfswarm_agent.py", line 479, in run_process_output
tree = ET.parse(outputFile)
File "C:\Users\ROHITKUM\AppData\Local\Programs\Python\Python37-32\lib\xml\etree\ElementTree.py", line 1197, in parse
tree.parse(source, parser)
File "C:\Users\ROHITKUM\AppData\Local\Programs\Python\Python37-32\lib\xml\etree\ElementTree.py", line 587, in parse
source = open(source, "rb")
FileNotFoundError: [Errno 2] No such file or directory: 'C:\Users\ROHITKUM\AppData\Local\Temp\rfswarmagent\logs\Scenario_1574658179\TC001_OpenCloseBrowser_1_5_1574658208\Hello_World_output.xml'

RFSwarmAgent: mainloop: Running 2019-11-25 10:33:29 ( 1574658209 )isrunning: True isconnected: True

Remove Case sensitivity to Test Cases detector

Describe the bug
When rfswarm detects the test case names in a .robot file it only finds them if the section is called:

• *** Test Case
• *** Test Cases
• *** Task
• *** Tasks

To Reproduce
Steps to reproduce the behavior:

1. create robot file with test case section:
   *** Test cases
2. load .robot file in rfswram
3. try to select test case name

Expected behavior
test case names should show in list

Report end time at the end of the test

Describe the solution you'd like

And also test end time in the console?

report in excel format

For this feature to be implemented I need some idea of the number of people who want this functionality and what information is needed in this type of report.

Some ideas could be:

• summary table
• running users graph
• response times graphs
  • broken down by test case?

This would be in addition to what is already provided by Issue #17

filename.rfs.rfs

When saving a scenario, should only add the .rfs extension only if the file name doesn't already have the .rfs extenion to prevent filename.rfs.rfs happening

Prompt with dialogue when No Agents available to run Robots

Another quirk I have noticed, if you click play when there are no agents available it won't start the test, but it will switch to the run tab and look like it did start the test. You can tell if this happened because the command window you ran the GUI/Server from will have the message:

No Agents available to run Robots!

For now I recommend waiting until the agent shows up in the agents tab before clicking play, I guess I need to pop an alert dialogue for this.

Plan graph bug with total users graph display

see screenshots:

Actual:

Expected: