qmetry/qaf

Quality Automation Framework for web, mobileweb, mobile native and rest web-service using Selenium, webdrier, TestNG and Java Jersey

QMetry Automation Framework

for web, mobileweb, mobile native and rest web-service functional test automation

Online documentation | Step-By-Step-Tutorial

Support libraries:

• qaf-support - steps for web and mobile web
• qaf-support-mobile - steps for and mobile native
• qaf-support-ws - request call repository support and steps for web services
• qaf-support-openapi - auto generate request call repository and bdd from open api specification
• qaf-support-elasticsearch - elasticsearch integration for cross execution dashboard
• qaf-support-angular - custom locator stretegies specific to angularjs
• qaf-support-aem - aem support
• qaf-support-galen - galen framework support
• qaf-support-flutter - flutter native app support
• qaf-cucumber - cucumber step with QAF BDD2 and/or cucumber runner support

Powerful Automation Platform for Test Authoring, Test Execution and Execution Analysis

QMetry Automation Framework provides a powerful and versatile platform to author Test Cases in Behavior Driven, Keyword Driven or Code Driven approach. It helps to significantly reduce costs involved in setting up Test Automation at any organization. It is a right tool for Web Platform, Mobile Platform (Native, Mobile Web, HTML5,etc) and Web Service test automation solution using Selenium and other related technologies. QMetry Automation Framework benefits any QA Team in developing highly maintainable and repeatable tests that utilize reusable test assets, proper modularity and semantic structure. Descriptive Reporting satisfies high-level as well as low-level (debugging) aspects.

QMetry Automation Framework is designed to solve common industry problems related to testing complex web systems. The framework is best suited for writing Automated Web, Mobile Web and Mobile Native Application UI Tests simulate real user activities on the page.

	The framework is built upon java and integrates TestNG,
	Selenium/Webdriver, Appium and Perfecto. Due to the framework's
	thorough design, test developer does not need to worry about common
	tasks such as thread safe browser session for running test in
	parallel, reporting or to incorporate result with test management
	tools. The tests run can be configured through standard testing
	configuration files, the test run filters and behavior can be changed
	within the config File. <br /> The architecture of the framework
	ensures a low cost of maintenance while supporting extensibility by:

• Abstracting the technical implementation away from the operational components
• Following accepted design patterns in creating the core functionalities within the framework.
• Providing an approach to develop highly maintainable and repeatable tests that utilize reusable test assets, proper modularity and semantic structure. Framework has readymade services for assertions, browser, reporting and data. Also, framework concept is based on page services so your page and related actions will be reusable from any test case. There is a good logging functionality and screen shot facility for assertions.

Features

QMetry Automation Framework supports integrations with Test Management Platform (QMetry, Rally, ApTest, HP ALM, etc), Continuous Integration Systems (Jenkins, Bamboo, etc), Mobile Device Cloud Solutions (SauceLabs, Perfecto Mobile, etc).

QMetry Automation Framework Key Features

Test Authoring:

• Behavior Driven Development (QAF BDD or Gherkin)
• Keyword Driven (Test case in CSV or EXCEL or XML)
• Code Driven Development (TestNG test)

Test Data Management:

Data driven testing with external Test Data Support (CSV, JSON, XML, excel, database), locator repository, support for multiple locales and multiple environments

Execution Reporting:

Execution Dashboard, Detailed Analysis and Screenshot Capturing, Integrations with Test Management Tools, and more

Some benefits of using the framework are:

• Less maintenance
• More reusability of code
• Reduced execution time
• Data-driven capability
• Easy configurable parallel execution
• Descriptive report
• Utility classes
• Test Results integration with test management tools like QMetry, Rally.
• Integration with Perfecto Mobile cloud, Sauce labs, Supports parallel execution
• Enabling testing across multiple platforms with or without selenium grid

qaf's Issues

Gherkin parser does does not include last step in scenario

Looks like the AbstractScenarioFileParser is not adding the last step of a scenario in a feature file. I believe line 264 in that class is the culprit.

In the following feature file the last step (*Then I ask it to say hi") is not included in the list of steps that the parseScenario() function creates. It looks like the do while loop always terminates 1 step too early.

Feature: Switch Frame Test
Scenario: Switch Frame Test
Given I open browser to webpage "http://nunzioweb.com/iframes-example.htm"
Then "iframe1" should exist
Then I switch to "iframe1" frame by element
Then I click on "iframe1.play"
Then I ask it to say hi

POM file in distribution for maven has dependency misspelled

Hi,

We are trying to add the QAF as maven dependency and the running the project is failing as the following dependency is misspelled:

javax.media
jai-core
1.1.3
compile

Actually the jai-core is "jai_core" on maven central:
http://repo1.maven.org/maven2/javax/media/jai_core/1.1.3/

We are getting following error:
The following artifacts could not be resolved: javax.media:jai-core:jar:1.1.3, com.sun.media:jai-codec:jar:1.1.3: Could not find artifact javax.media:jai-core:jar:1.1.3 in qaf (https://qmetry.github.io/qaf/dist) -> [Help 1]

Is there any workaround - our team is blocked.

Allow class variable sharing among steps in same class

Unable to load step from the class extending another class

When executing following bdd it throws step not found exception. It was working fine with 2.1.8.

Steps to Reproduce

BDD Scenario:

SCENARIO: SampleTest
META-DATA: {"description":"Sample Test Scenario","groups":["SMOKE"]}
 
 open application

END

Step implementation:

public class Steps extends BaseClass {
 @QAFTestStep(description = "open application")
 public void openApp() {
  System.out.println("it worked.....");
 }
}

support for bdd, kwd dry-run

New step creation code snippet when step not found

NotYetImplementedException should add logger that logs code snippet for new step creation.

Not working with groovy code,gettining "Classnotinitilization Error"

Thanks for such wonderful open source product. While exploring your open source framework we faced some issues. Below are details and objective of experiment.

Objective : Using BDD approach , we need to execute our groovy files including step definition. Cucumber will be used for steps definition , gherkin can use as feature file while we are trying groovy for scripting .

Steps with QAF framework :

1. Define application property file with basic require properties and step.provider.pkg definition .
2. In "step.provider.pkg" define groovy step definition file but after execution of runner XML, it throws 'Class not Initialize " error . However, if we tried same with step definition in java file it worked perfectly. So do we support groovy ? or are we doing something wrong ?

Please help us to move so we can leverage this seamless framework. Thanks again.

dependsOnMethods not working for BDD scenarios

For following bdd scenarios, I have given dependsOnMethods meta data. So scenario "ONE" should execute first and then scenario "TWO". But currently its not working and throws an exception

SCENARIO: TWO
META-DATA: {"description":"Test Scenario TWO","groups":["SMOKE"],"dependsOnMethods":["ONE"]}

    COMMENT: "THIS IS Scenario <TWO> depends on <ONE>"

END

SCENARIO: ONE
META-DATA: {"description":"Test Scenario ONE","groups":["SMOKE"]}

     COMMENT: "THIS IS Scenario <TWO> depends on <ONE>"

END

Here is stack trace,

org.testng.TestNGException: 
Method "scenarios.denpendsOnMethod.bdd.TWO()[pri:1000, instance:com.qmetry.qaf.automation.step.client.Scenario@3e8]" depends on nonexistent method "ONE"
	at org.testng.DependencyMap.getMethodDependingOn(DependencyMap.java:100)
	at org.testng.TestRunner.createDynamicGraph(TestRunner.java:1082)
	at org.testng.TestRunner.privateRun(TestRunner.java:772)
	at org.testng.TestRunner.run(TestRunner.java:655)
	at org.testng.SuiteRunner.runTest(SuiteRunner.java:366)
	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:361)
	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:319)
	at org.testng.SuiteRunner.run(SuiteRunner.java:268)
	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1244)
	at org.testng.TestNG.runSuitesLocally(TestNG.java:1169)
	at org.testng.TestNG.run(TestNG.java:1064)
	at org.testng.remote.RemoteTestNG.run(RemoteTestNG.java:113)
	at org.testng.remote.RemoteTestNG.initAndRun(RemoteTestNG.java:206)
	at org.testng.remote.RemoteTestNG.main(RemoteTestNG.java:177)

Provide maintenance mode

During execution user should be prompt to provide valid locator in case of element not found exception

Reporter.log(message,Messagetypes) don't have option to set to log to standard output

Problem:

• Can't see Reporter.log (html logs) in eclipse console .
  I see their is existing support for TestNG , but qmetery Utility Class" Reporter.log "don't have an option to set to log standard output as it is supported in testNG, as this helps debugging easier when working with TestNG Eclipse plugin.
  Existing support in testng has something that can be set to boolean if required "http://testng.org/javadocs/org/testng/Reporter.html"
  log(java.lang.String s, boolean logToStandardOut)

Support for xml for test-authoring

XML test step client #9
In addition to CSV, and Excel support XML for test-authoring.

Duplicate result file for data driven testcase

Using data driven if thousand of testcases are executed then for some testcases it was creating same resultFile, so in report it was creating ambiguity.

Support to have on driver Initialization Failure in driver listener

TestStep executed from the wrong package

Case: same step is implemented for two different platform android and ios in two different package com.company.automation.android and com.company.automation.ios respectively.
When step provider package set as com.company.automation.android steps get executed from com.company.automation.ios and vise-versa.

QAF Report should display description for locator instead of locator key

For locator arguments in BDD scenario currently it display locator key in report, it should display locator description for more readability. Currently its like

click on 'btn.search.loc'

If user has provided description for this key then it should be like

click on 'Search Button'

Control flow support in non-java test step

Support for basic loop and conditional statement in non-java test step so for such requirements user don't need to create java step and can do code less automation.

Unable to Launch firefox and IE in QAF 2.1.9

Details for Firefox error:
firefox Version 50.1.0
Below error we are getting when running from config file.
Unable to create driver instance in 1st attempt with retry timeout of 0 seconds. You can check/set value of 'driver.init.retry.timeout' appropriately to set retry timeout on driver initialization failure.Unable to Create Driver Instance for firefox: Failed to connect to binary FirefoxBinary(C:\Program Files\Mozilla Firefox\firefox.exe) on port 7055; process output follows: null

Details for IE Browser error-
IE driver Server Product version-2.35.1.0
Below error we are getting when running from config file:
Unable to create driver instance in 1st attempt with retry timeout of 0 seconds. You can check/set value of 'driver.init.retry.timeout' appropriately to set retry timeout on driver initialization failure.Unable to Create Driver Instance for iexplorer: java.lang.NoSuchMethodException: org.openqa.selenium.ie.InternetExplorerDriver.(java.net.URL, org.openqa.selenium.Capabilities)

Step reporting broken in when test authored in java

In case where test case written in java (standard TestNG test) step from common step library are not coming in report.

Unable to capture screenshot for IE driver

Verification steps unable to capture screenshot for IE driver

Step to reproduce:

SCENARIO: Google Search
META-DATA: {"description":"google search for test in ie"}
	get '/'
	verify 'name=r' is present
END

So while capturing screenshot it throws below exception:

java.lang.NullPointerException
	at com.qmetry.qaf.automation.ui.webdriver.QAFExtendedWebDriver.getScreenshotAs(QAFExtendedWebDriver.java:268)
	at com.qmetry.qaf.automation.ui.webdriver.QAFExtendedWebDriver.takeScreenShot(QAFExtendedWebDriver.java:460)
	at com.qmetry.qaf.automation.core.QAFTestBase.takeScreenShot(QAFTestBase.java:278)
	at com.qmetry.qaf.automation.core.QAFTestBase.addAssertionLog(QAFTestBase.java:320)
	at com.qmetry.qaf.automation.ui.WebDriverCommandLogger.addMessage(WebDriverCommandLogger.java:192)
	at com.qmetry.qaf.automation.ui.webdriver.QAFExtendedWebElement.report(QAFExtendedWebElement.java:1140)
	at com.qmetry.qaf.automation.ui.webdriver.QAFExtendedWebElement.ensurePresent(QAFExtendedWebElement.java:622)
	at com.qmetry.qaf.automation.ui.webdriver.QAFExtendedWebElement.verifyPresent(QAFExtendedWebElement.java:630)

BDD Scenarios are getting skipped in QAS during execution

“Unable to execute script in QAS".

1. The Perfecto Lab and Dashboard is getting disconnected often in QAS. Attached screenshot on the Perfecto Issue disconnection. When trying to open in website this issue is not happening.
   PerfectoIssue.docx

2. We are getting below issue when executing the automated scripts after @12 PM IST, and we are blocked. Is there is any Issues/downtime.?
   Skipping Issue.docx

Issue1: Unable to Create Driver Instance for appium: java.lang.reflect.InvocationTargetException.
Issue2: Connect to heb.perfectomobile.com:443 [heb.perfectomobile.com/192.80.63.78] failed: Connection timed out: connect
Issue3: ERROR com.perfectomobile.selenium.util.PerfectoLabConnector -
Failed to connect to Perfecto Lab for Eclipse plugin on host: localhost and port: 3287.
Possible reasons are: The code doesn't run in Eclipse or the Perfecto Lab view is not open. In these cases use the MobileDriver constructor that receives the host, user and password.
“Unable to execute script due to slowness in perfecto performance”
Issue4: Error communicating with the remote browser. It may have died.

Attached screenshot for the above mentioned skip issues.

support retry on driver initialization failure

Scenario object access in teststep listener

Cucumber stub for not implemented steps

if using gherkin syntax to write features, it would be great that in case of not implemented step the auto generate code will be in gherkin.

Java class generation for test authored in non-java format

There can be a way to have java classes generated for test-case authored in non-java format for example bdd. The code-generator can be implemented to dynamically generate byte-code or generate source files.

Test fails on NullPointerException when using verify functions

TestNG 6.9.10 support

Support to set Class Finder for TestStepFinder

There should be a way to set class that can be provide list of the classes where to find test steps. User can provide implementation using different reflection implementation or library and set the class to be used to get all classes to find test steps.

Parallel configuration share parameters values.

When user run the cucumber tests in parallel, sometimes driver tries to open the device that already used by another test thread.
If deviceName parameter was define for one test (in testng.xml) it is using the same parameter in another test configuration, even if it wasn't defined.
So the test is ignored with next output:
Unable to create driver instance in 1st attempt with retry timeout of 60 seconds. You can check/set value of 'driver.init.retry.timeout' appropriately to set retry timeout on driver initialization failure.Unable to Create Driver Instance Failed to execute command handset open: Cannot open device 9A90117A5FF7AEB655944A1D7FA8C70F3C2C81BD for user [email protected], device is in use.

Steps to reproduce:

Create feature file with several test scenarios
Create xml suite which will run those scenarios in 2 tests, for one test set the "deviceName" parameter
Run the xml suite

Support XML node key in BDD step call where to JavaStep requires map

Allow passing entire Map object from BDD to JavaStep parameter.

Having one property for all QAF listener

As of now there are following 3 different properties to register 3 different types of QAF listeners

• teststep.listeners
• wd.command.listeners
• we.command.listeners

If there is single property provided that can used to register any of the QAF listeners it will be more user friendly.

FormDataBean Custom Component support

With form data bean UI mapping support to provide custom component class.

Some scripts taking more hours than expected time, is there any cutoff time there to limit execution of a test script

Few test scripts executed more than 7 hours, even though there is some error like error communicating with the server and element is not visible, which usually will be executed within 5 minutes.

Is there any cutoff time available in QAF so that we can limit the test execution which will avoid such kind of issues.

How to send the QAF execution summary as email ?

How to send the QAF execution summary report as email for the project team ? is there any provision through framework ? Do we have any documentation for this ?

Test result should display description instead of stepname in KWD scenarios

In kwd scenarios currently it displays stepname in resport instead of formatted description.
For example,

SCENARIO|GoogleSearch|{"description":"Search Google","groups":["P1","SMOKE"]}
get|["http://www.google.com"]|
sendKeys|["qmetry","txt.search.loc"]|
click|["btn.search.loc"]|
verifyLinkWithPartialTextPresent|["qmetry"]|
END||

Currently steps displays in report is

get
sendKeys
click
verifyLinkWithPartialTextPresent

which is not readable as bdd scenarios, and user cannot see any arguments here.

It should be like this.

get 'http://www.google.com'
sendKeys 'qmetry' into 'Search Input'
click on 'Search Button'
verify link with partial text 'qmetry' is present

After updating 2.1.9 Jar files, lot of testcases are getting “skipped “.

Please let us know on what condition/scenario the testcase will be “skipped “ ? because TC which gets skipped is again retried and either it gets pass / fail but it updated the TC count also in the summary report which becomes as duplicate entry point.

Support JSON report for multiple XML suite in single run

Support to define component with locator repository

Self descriptive locator has capability to provide meta-data with the locator. There can be a way where user can define component class in meta-data that can be used to create element from the locator by the framework.

This is a test issue - should be ignored

Please ignore this.

Unable to launch firefox and IE browsers using QAF2.1.9.

Details for Firefox error:
firefox Version 50.1.0
Below error we are getting when running from config file.
Unable to create driver instance in 1st attempt with retry timeout of 0 seconds. You can check/set value of 'driver.init.retry.timeout' appropriately to set retry timeout on driver initialization failure.Unable to Create Driver Instance for firefox: Failed to connect to binary FirefoxBinary(C:\Program Files\Mozilla Firefox\firefox.exe) on port 7055; process output follows: null

Details for IE Browser error-
IE driver Server Product version-2.35.1.0
Below error we are getting when running from config file:
Unable to create driver instance in 1st attempt with retry timeout of 0 seconds. You can check/set value of 'driver.init.retry.timeout' appropriately to set retry timeout on driver initialization failure.Unable to Create Driver Instance for iexplorer: java.lang.NoSuchMethodException: org.openqa.selenium.ie.InternetExplorerDriver.(java.net.URL, org.openqa.selenium.Capabilities)

Gherkin support for test case authoring

QAF supports cucumber as underlying framework for Gherkin but not all the features as QAF-BDD available because of some limitation of cucumber. If QAF supports for Gherkin it will be more useful for those who want to opt Gherkin instead of QAF-bdd for testcase authoring.

excel xlsx file support

Data driven scenario count issue in case of verification failure

please run the attached code for DataDriven to reproduce the issue. Results are also uploaded along with source which displays the count issue.

DataDrivenDemo.zip

How to create a mutiple chrome webdriver instances

As a part of my project I need to create multiple sessions so I am trying to create multiple webdriver instance to create multiple session. When I try to create a new instance using new WebDriverTestCase().getDriver(), it is opening in the same chrome browser instead of opening in new window.

Is there any method available in QAF to create multiple chrome driver instances in multiple windows instead of opening in the same window

TestBase for MobileWebDriver

At present, in case of mobile webdriver, user have to cast driver to underlying mobile driver. Required to create TestBase for MobileDriver in order to get direct mobile driver object from getDriver().

JSON report missing test results sometimes when running parrallel

This is very random behavior and very few time we found test get executed but not in json report (QAF dashboard).

Test case specific data driven parallelism property does not working

Test written in java
`package com.p1;

import java.util.Map;

import org.testng.annotations.Test;

import com.qmetry.qaf.automation.testng.dataprovider.QAFDataProvider;
import com.qmetry.qaf.automation.ui.WebDriverTestCase;

public class TestDataDrivenParallel extends WebDriverTestCase {

@QAFDataProvider(dataFile = "resources/logindata1.txt")
@Test(groups = { "SMOKE" }, description = "Google Search")
public void test1(Map<String, String> data) {
    System.out.println(Thread.currentThread().getId());
}

@QAFDataProvider(dataFile = "resources/logindata2.txt")
@Test(groups = { "P1", "P2" }, description = "Yahooo Search")
public void test2(Map<String, String> data) {
    System.out.println(Thread.currentThread().getId());
}

}
`

property setting
global.datadriven.parallel = 1
test1.parallel=0

data driven and retry combination problem

For data driven scenario when retry.count is used, then it retries for the passed testcases also.

Step To Reproduce

SCENARIO: SampleTest
META-DATA: {"description":"Sample Test Scenario","groups":["SMOKE"],"dataFile":"resources/testdata.xls","sheetName":"datasheet"}
	
	COMMENT: '${recId}'
	Given login step
	
END

Testdata

Step Implementation

public class StepsBackLog {
	public static int count = 0;
	@QAFTestStep(description = "login step")
	public void loginStep() {
		count = count + 1;
		if (count <= 1) {
			throw new NullPointerException("Failure to test retry");
		}
	}
}

And retry.count=2

Actual Result

Expected Result:
It should not retry for pass testcases, here it retries for TD2 and TD3 even it was passed.

Duplicate report entries with QAF + Cucumber + TestNG

When using Cucumber with TestNG in QAF report it created two entries for the execution.

"Your connection is not secure" issue while launching script in Firefox.

How to resolve this issue. One of the approach we have tried is below in java but its not working. unable to proceed ahead after "Confirm Security Exception" page. I have uploaded the screen shot of issue page.
First I have created a profile in firefox named "automationtest" and then
Following desired capabilities we have set for Firefox:

   ProfilesIni allProfiles = new ProfilesIni();
FirefoxProfile myProfile = allProfiles.getProfile("automationtest");
myProfile.setAcceptUntrustedCertificates(true);
myProfile.setAssumeUntrustedCertificateIssuer(false);
WebDriver myProfilename = new FirefoxDriver(myProfile); // FirefoxProfile
myProfile = new FirefoxProfile();

Required help on this issue.
Firefox version-50.0.1
Selenium server-3.0.1
geockdriver-0.11.1

Getting Error "Error communicating with the remote browser. It may have died."

Getting Error "Error communicating with the remote browser. It may have died."

Experiencing this issue in parallel execution with chrome driver when the thread count is configured as more than 1 in config xml files.

Platform: Windows

Tried below solutions, but nothing had worked out.

1. Updated windows reg edit tcp maximum ports
2. Made the test step methods as thread safe by adding synchronized modifier

Frequency of this error is more when the thread count increases. Please help us on this.