wttech/aet

AET - a system that detects visual changes on web sites and performs basic page health checks

aet's Issues

Strange error message for XUnit failures

Sometimes we get the error after test suite execution saying:

There are 2 parameter(s) but only 1 format argument(s) were provided.

This is related to wrong usage of jcabi-log in our code:

TestSuiteRunner.java:147

Add fallback for screen comparator with selector

When I use

<open/>
<click xpath="selector-to-a-button-that-opens-an-overlay" timeout="5000"/>
<screen name="Desktop" css'"selector-to-an-overlay"/>

and the overlay fails to open then I receive a "Processing error" that indicates that AET were unable to take the screenshot.

It would be useful to have a fallback: if screen with selector fails, then simply take a screenshot of whole page (it would be simpler to debug the issue by a not experienced QA)

ESXI or Hyper-V

Hey are you gonna support ESXI or Hyper-V in near future? I would need to migrate VM from VirtualBox to some other virtualization server. And I'm thinking also about calling eat:run remotely. Is it possible at all?

Add possibility to retest only selected page from the suite

Sometimes there is a need to retest only particular page. For example, there was some environment/browser glitch and we want to retest that page without running the whole suite.

Ideally there should be some button in the report which will trigger that.

No way to define endpoint URL for AET reports

Instance setup:

1. Windows PC with 2 VM's installed using cookbook for windows wttech/aet-cookbook#9 created by @karoldrazek
2. network for VM's using NAT - all relevant ports are forwarded
3. we have no proxy setup so all requests have to have port defined

Steps to reproduce

1. Run AET in given setup
2. open up report.html and wait for redirect

Expected
I should be able to see report

Actual
Report app is displaying connection error alerts

Reason
In JS Console we can see that reports app is trying to contact "http://192.168.37.12/api/metadata?company=cognifide&project=pioneer&correlationId=cognifide-pioneer-pioneer-ci-1493362630699", but due to fact that we have no proxy, nothing is dispatching request to Apache Karaf. If we execute the same request with port "http://192.168.37.12:8181/api/metadata?company=cognifide&project=pioneer&correlationId=cognifide-pioneer-pioneer-ci-1493362630699" we can see that report is there.

In order to fix this issue we need to have an ability to define endpoint for reports app as well. At the moment in "endpointConfiguration" module we can se that it's simply using current domain as endpoint.

xUnit generation fails when stepResult is missing

Following exception is being thrown when trying to access xunit report for Test Suite Run with missing stepResults e.g. after some ProcessingError/timeout that would cause not persisting results:

| WARN  | qtp1838949591-92 | ServletHandler                   | pse.jetty.servlet.ServletHandler  505 | 70 - org.eclipse.jetty.util - 7.6.9.v20130131 | /xunit
java.lang.NullPointerException
	at com.cognifide.aet.xunit.MetadataToXUnitConverter.traverseSteps(MetadataToXUnitConverter.java:97)[86:com.cognifide.aet.rest-endpoint:2.1.1]
	at com.cognifide.aet.xunit.MetadataToXUnitConverter.convertStepsToXUnitCase(MetadataToXUnitConverter.java:91)[86:com.cognifide.aet.rest-endpoint:2.1.1]
	at com.cognifide.aet.xunit.MetadataToXUnitConverter.convertTestToXUnitSuite(MetadataToXUnitConverter.java:82)[86:com.cognifide.aet.rest-endpoint:2.1.1]
	at com.cognifide.aet.xunit.MetadataToXUnitConverter.convert(MetadataToXUnitConverter.java:68)[86:com.cognifide.aet.rest-endpoint:2.1.1]
	at com.cognifide.aet.rest.XUnitServlet.generateXUnitAndRespondWithIt(XUnitServlet.java:112)[86:com.cognifide.aet.rest-endpoint:2.1.1]
	at com.cognifide.aet.rest.XUnitServlet.process(XUnitServlet.java:85)[86:com.cognifide.aet.rest-endpoint:2.1.1]
	at com.cognifide.aet.rest.BasicDataServlet.doGet(BasicDataServlet.java:79)[86:com.cognifide.aet.rest-endpoint:2.1.1]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:618)[243:org.apache.felix.http.servlet-api:1.1.2]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:725)[243:org.apache.felix.http.servlet-api:1.1.2]
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:652)[81:org.eclipse.jetty.servlet:7.6.9.v20130131]
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:445)[81:org.eclipse.jetty.servlet:7.6.9.v20130131]
	at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:70)[99:org.ops4j.pax.web.pax-web-jetty:1.1.17]
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)[78:org.eclipse.jetty.server:7.6.9.v20130131]
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:556)[80:org.eclipse.jetty.security:7.6.9.v20130131]
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227)[78:org.eclipse.jetty.server:7.6.9.v20130131]
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1044)[78:org.eclipse.jetty.server:7.6.9.v20130131]
	at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:124)[99:org.ops4j.pax.web.pax-web-jetty:1.1.17]
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:372)[81:org.eclipse.jetty.servlet:7.6.9.v20130131]
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:189)[78:org.eclipse.jetty.server:7.6.9.v20130131]
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:978)[78:org.eclipse.jetty.server:7.6.9.v20130131]
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)[78:org.eclipse.jetty.server:7.6.9.v20130131]
	at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:81)[99:org.ops4j.pax.web.pax-web-jetty:1.1.17]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)[78:org.eclipse.jetty.server:7.6.9.v20130131]
	at org.eclipse.jetty.server.Server.handle(Server.java:367)[78:org.eclipse.jetty.server:7.6.9.v20130131]
	at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:486)[78:org.eclipse.jetty.server:7.6.9.v20130131]
	at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:926)[78:org.eclipse.jetty.server:7.6.9.v20130131]
	at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:988)[78:org.eclipse.jetty.server:7.6.9.v20130131]
	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640)[73:org.eclipse.jetty.http:7.6.9.v20130131]
	at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)[73:org.eclipse.jetty.http:7.6.9.v20130131]
	at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)[78:org.eclipse.jetty.server:7.6.9.v20130131]
	at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:628)[72:org.eclipse.jetty.io:7.6.9.v20130131]
	at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)[72:org.eclipse.jetty.io:7.6.9.v20130131]
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)[70:org.eclipse.jetty.util:7.6.9.v20130131]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)[70:org.eclipse.jetty.util:7.6.9.v20130131]
	at java.lang.Thread.run(Thread.java:745)[:1.7.0_79]

Links get cut when the user changes the screen size

urls get cut when the user expands the list of browsers and changes the size of the screen.

To reproduce:

1. Open your chrome browser in a full-screen mode
2. Open any report with a number of long urls
3. Expand the list
4. Note that long urls are shortened as intended
5. Collapse the list
6. Change the size of the screen
7. Set the full screen again
8. Expand a list of urls
9. Note that some urls got cut

Actual result: urls get cut when the user expands the list of browsers and changes the size of the screen.

Expected result: urls do not get cut when the user expands the list of browsers and changes the size of the screen.

Report - unable to save notes(comments) on case level

Steps to reproduce:

• open any AET report
• click on any test
• click on any url
• click on 'notes' icon on case level
• fill form and clic "updata"
• click "save all changes"
• reload page

Expected :

• entered comment is saved, and visible
  Actual:
• comment is not saved

Also there are no functional test coverage for comments feature
We should add CRUD scenarios for each nesting level (suite, tests, case) in sanity-functional module.

SharedPatterns - take latest suite

Currently the Shared Patterns functionality requires us to add the full correlationId. This is useful to e.g. test against a particular release.

However during heavy development on multiple feature branches we would like to take the latest patterns from develop or master branch without adding the full correlationId.

I would love to be able to add something like:
-Dpattern=company-project-master-latest

instead of
-Dpattern=company-project-master-149519161234

AET as clientless application

Scheduling AET job should be as simle as performing http request. Remove client application (AET Maven Plugin) and move its logic to AET Server. Create a web service that consumes e.g. HTTP request with suite XML file. Thanks to this scheduling AET job will be available without need to install Java and Maven.
Technical challenges:

• communication between client and AET server (e.g. progress and other feedback) is currently performed by JMS temporary queue. It should be still possible to track the job progress (e.g. by simple service that shows current job status and displayed in a web view).
• documentation should be updated.

After this enhancement user should be able to schedule AET job with this simple command:
curl POST http://server/api/scheduleJob -d @suite.xml

Login Modifier - refactor and stabilize

Login modifier currently works not stable and causes many problems.
Current implementation enables to save cookie with authentication token.
Probably this solution may cause problems - please check this.

Login modifier just provides username and password and clicks submit button.
If we need to wait for login then <sleep> is required.

Within this task Integration Tests should be prepared (sanity site).

Login modifier should work on AEM author page!

Update Hide Modifier with find elements by css selector

Update Hide Modifier and add possibility to find element to hide by cssSelector.
Finding elements by xpath should be still possible, example:

<collect>
  ...
  <hide css="#learnMore"/>
  <hide xpath="//*[@id='about-anim']" />
  ...
</collect>

As part of this task, extend integration-tests with case where hiding by css selector will be used.
Update proper page in the documentation.

HTTP Not Found error message in client is confusing

Steps to reproduce:

1. Navigate to <aet-url>/system/console/bundles on the AET instance.
2. Find the AET :: Test Executor bundle.
3. Stop this bundle.
4. Run test suite.

As a result of the suite run, the following message is displayed in client logs: Failed to process test suite: Not Found. This message is confusing and should be more meaningful.

Prepare Integration Tests For Login Modifier

Within this task Integration Tests should be prepared (sample site) for Login Modifier.

End of the URL is cut off

When URL is to long is just cut off. There should be ... or some other indicator when URL is to long.

(at the end should be: /grid960-12.html)

.

Support for comparing patterns between suites

It would be very useful to be able to compare gathered snapshots with patterns from other suites. Use case:

1. I have two environments: staging and prod
2. I want to deploy a code to staging, run AET and eventually accept all new changes as new patterns
3. I want to deploy the same code to prod, run AET and automatically compare snapshots with these new patterns from staging. Thanks to this, I don't need to review and accept changes on prod manually.

The mechanism would need to take into consideration obvious differences between environments, like different domains, etc

Prepare AET Developer Guide

Developer Guide should describe in details how to create new AET module (e.g. collector and comparator).
Maybe a good idea would be to create maven archetype?

Selenium Grid

Idea of this epic is to get rid of current strong connection between AET core and Firefox browser. Instead of using selenium driver. AET should use Selenium Grid to use remote browser.
This epic will provide new possibilities:

• separate AET infrastructure and browser (no selenium/webdriver dependency),
• support wider range of browsers.

Report UX improvements

Within this epic, all AET Report improvements will be performed.

Support for different environments in integration tests test suite

For login modifier the login page URL needs to be provided with domain.
This URL differs for our tests environments (we are using more complicated environments setup).
We need to have ability to configure the domain for the login page URL in order to run the test successfully on our different environments.

PROCESSING_ERROR should be properly showed on main view

On the left hand side and test level it seems fail (see the screenshots).
But on the url level it seems success.

While checking metadata it looks like the status is PROCESSING_ERROR however the message is not displayed.

There are other collector statuses that should be supported. See:

• com.cognifide.aet.communication.api.metadata.CollectorStepResult.Status

The colors should be consistent on all levels; the message should be displayed.

Note:
DUPLICATES_PATTERN means that collector doesn't store new version in database (as it is same as pattern),

Task	Status
if there is processing error it should be displayed in test case report
message (description) is provided for error (copied from metadata)

clean configuration legacy for `useProxy` configuration

Currently there is a possibility to configure web proxy with "true" value.
This makes the code unclear: see also the discussion for pull request at:

• #53 (comment)

For resolution I like the second approach more:

there will be only named type of a proxy (remove true as a possible value) after deprecating it first for e.g. 2 minor versions

(Alternative solution is to have the configuration value "default" be interpreted as "rest" proxy.)

Currently all suites using web proxy have it configured to "rest" as otherwise it doesn't work.

Under certain circumstances AET can collect wrong page content

Hi,

during one of AET run completely wrong page was collected and presented in the report. Here's a screenshot that presents the problem:

I had to blur a lot above, but it's quite obvious that these 2 pages are completely different. Surprisingly, subsequent AET run produced expected result - the appropriate page was collected and compared with the pattern.

It definitely wasn't an application issue, as collected content has never been present under that path (it was one of other paths we defined in our test suite).

I had a quick chat with @Skejven about this problem and he mentioned that it can be side effect of multiple <open/> elements in our XML file. Here's a snippet of its current shape:

<collect>
    <resolution width="1920" height="1024"/>
    <open/>
    <sleep duration="4000"/>
    <screen name="extralarge"/>
    <resolution width="1250" height="900"/>
    <open/>
    <sleep duration="4000"/>
    <screen name="large"/>
    <resolution width="1150" height="900"/>
    <open/>
    <sleep duration="4000"/>
    <screen name="medium"/>
    <resolution width="769" height="900"/>
    <open/>
    <sleep duration="4000"/>
    <screen name="medium2"/>
    <resolution width="400" height="900"/>
    <open/>
    <sleep duration="4000"/>
    <screen name="small"/>
    <source/>
    <status-codes/>
    <js-errors/>
</collect>

What could be a possible root cause? How can we overcome that problem?

It happened just once so far (1 out of hundreds collections), so unfortunately I don't have exact reproduction steps. I can consider that as an incident, but since it took place I'd like to get some insights and see what can be done to prevent similar situations in the future.

Cheers,
Kuba

Update Click Modifier with find elements by css selector

Update Click Modifier and add possibility to find element to hide by cssSelector.
Finding elements by xpath should be still possible, example:

<collect>
  ...
  <click css="#learnMore" timeout="3000"/>
  <click xpath="//*[@id='about-anim']" timeout="3000"/>
  ...
</collect>

As part of this task, extend integration-tests with case where click by css selector will be used.
Update proper page in the documentation.

Switch to new Karaf version

Following steps should help upgrading to newest Karaf framework

Phase A:

1. Remove start level from aet-features.xml file (so that default is used).
2. Group dependencies into sub-features
3. Make aet-features.xml fle a filtered resource so that we could change it easily in maven lifecycle.
4. For some dependencies versions (the easy ones) that are in aet-features.xml make them to be read from project properties so that same versions are used as Maven dependencies
5. Validate and update not-so-obvious dependencies, remove them if it is possible

Phase B:

Upgrade those dependencies that don't work on new Karaf and swith to new Karaf instance. (It might be easier to switch to ActiveMQ v 5.15.x instead of resolving Spring dependencies for ActiveMQ 5.13.1.)

Report link generated by AET Maven client doesn't work

Hey,

something odd just happened for one of our test suites. AET run finally went green (build 115), but report link that was generated at the very end of processing suddenly stopped working. Links generated for previous attempts (builds 106-114) worked without issues. Here's what exactly happened:

• Jenkins job that runs AET finally turns green
  
  
• when I click on the link the following gets displayed in the browser
  
• when I click "OK" button, report says it can't load metadata
  

This is a bit related to #71. Build is green, theoretically all was successfully committed to MongoDB, but for some reason report doesn't work anymore.

Do you have any ideas what went wrong and how we can get that resolved?

Thanks,
Kuba

Cannot compare source code after switching to TLS 1.2 only

It seems that after disabling anything different than TLS 1.2 we're getting following errors on source code comparison tab:

Processing error occurred!
Details:

["Step: source (with parameters: {}) thrown an exception when collecting url: /en-us/aet/analytics/product-detail-page in: cp-zg-extension-aet-test-sample test. Cause: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake"]

Research a tool that will validate code quality at GitHub

Research a tool that will validate code quality on GitHub and apply it as a PoC on existing AET GitHub repo, then integrate it with current codebase and check it (no fixes are in scope of this task).

Remove timeouts limitation

According to documentation https://github.com/Cognifide/aet/wiki/LoginModifier login timeout is limited to 10 seconds. This value can be to little in some cases. Also there are other places where maximal timeout value is fixed:

• https://github.com/Cognifide/aet/wiki/ClickModifier
• https://github.com/Cognifide/aet/wiki/ReplaceTextModifier

this time to 15 seconds. Maybe it would be better to remove such limitations at all?

Missing fonts

Hi

would you please consider to install more fonts to your OS? Especially those like Chinese or Arabic. Currently, it's been missing those and even the page is looking good at my local web browser
at EAT it looks as on the screen:

Improve tooltip message layout for notes

Whenever a user move over a "add note" button, a tooltip is dispalyed.
Currently there is a message "Click to update this ... note" with some empty space left above which does not look well:

The purpose of that one-line-space is to separate a tip from the actual note if it is added:

Let's try to improve the UX. A suggested solution is to remove that empty space and simply make a tip message visually different than a note, e.g. use two different font colors, bold vs. italic, sizes, etc.

Add ability to wait for some element to appear on the page

Current implementation of wait for page loaded modifier is not sufficent for pages containing dynamic web applications, especially when there is some delay of page load (our client servers are in US, thus the page load time takes a bit)
The scenario is as follows:

1. Page loads with most of the stuff on the page
2. Dynamic application starts running, contacts few backend services, etc.
3. Once the dynamic data comes back page gets modified

What we've seen was that wait for page load was stopping at random moments during dynamic application load. To overcome that we had to add a whole bunch of sleeps here and there, which made our tests run for a very long time.

I would like to suggest a new modifier, where:

• You will be able to specify xpath to element that the modifier should wait for
• Timeout (optional - default value of 5? seconds)
• Treshold between checks (optional - default value o 500ms?)

how to find previous test runs?

Sorry this is probably a silly question... a couple of us have installed using the basic setup with vagrant on our locals. Is there a URL we can hit to see the latest reports, to find them without having to use the maven output? If I've closed my terminal I find myself kicking off a new run just to see the report.

Probably be less of an issue if we get a real install and get it running from Jenkins, etc.

Error messages are not verbose enough

Hey,

quite recently I encountered a bunch of AET issues. Unfortunately none of them was easy to troubleshoot and fix, mostly due to not enough feedback on client side. That's rather frustrating experience for someone who considers AET as a platform and just executes a bunch of test suites against predefined endpoint.

I have 3 stories that prove above standpoint.

First story

I have 2 environment in my project. Both of them have dedicated test suites and we used to run them on a quite regular basis. Everything was working fine on Thursday, we did app deployment and rerun AET Jenkins job the next day. It failed with the following error:

[INFO] Scanning for projects...
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building XYZ 1.0.0
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- aet-maven-plugin:2.0.1:run (default-cli) @ aet ---
[INFO] Successfully connected to tcp://10.aaa.bbb.ccc:61616
[INFO] HV000001: Hibernate Validator 4.3.2.Final
[INFO] HV000002: Ignoring XML configuration.
[INFO] CorrelationID: xyz-xyz-uat-suite-1488205130948
[INFO] ********************************************************************************
[INFO] ********************** Job Setup finished at 14:18:52.324.**********************
[INFO] *** Suite is now processed by the system, progress will be available below. ****
[INFO] ********************************************************************************
[INFO] [14:18:52.426]: COLLECTED: [success: 0, total: 49] ::: COMPARED: [success: 0, total: 0]
[WARNING] null
[WARNING] null
[WARNING] null
[WARNING] null
[WARNING] null
[INFO] [14:19:15.441]: COLLECTED: [success: 1, total: 49] ::: COMPARED: [success: 3, failed: 5, total: 8]
[WARNING] null
[WARNING] null
[WARNING] null
[WARNING] null
[WARNING] null
[INFO] [14:19:16.443]: COLLECTED: [success: 2, total: 49] ::: COMPARED: [success: 6, failed: 10, total: 16]
...
[INFO] [14:30:53.475]: COLLECTED: [success: 48, total: 49] ::: COMPARED: [success: 144, failed: 240, total: 384]
[WARNING] null
[WARNING] null
[WARNING] null
[WARNING] null
[WARNING] null
[INFO] Received report message: FinishedSuiteProcessingMessage{correlationId=xyz-xyz-uat-suite-1488205130948, status=OK, errors=[]}
[INFO] Report is available at https://<AETDOMAIN>.com/report.html?company=xyz&project=xyz&correlationId=xyz-xyz-uat-suite-1488205130948
[INFO] Suite processing finished.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 12:25 min
[INFO] Finished at: 2017-02-27T14:31:15+00:00
[INFO] Final Memory: 11M/129M
[INFO] ------------------------------------------------------------------------

I opened the report, but for all the pages there was a message that said something about Unknown error (w/o any description at all), but apparently all the pages were visited because JS and status code errors were present in the report (all confirmed by log entries on the server).

Thankfully we had an AET engineer on board and he found out something was wrong with data collected in MongoDB. We made a few attempts to take some pattern IDs directly out of the database and injected that into report app URL. Unfortunately we ended up with the following in every single case:

{
  "message": "Unable to get artifact with id : 58ad96527700471ab4ec98f0 form {company='XYZ', project='ABC'}"
}

All in all we decided to remove all the data related to that suite from MongoDB (that wasn't that important for us at that time) and rerun the test - all was fine this time. It's hard to say what exactly went wrong, but based just on client logs and w/o deep understanding of what's going on under the hood it was impossible to track that bug down. There was no information about MongoDB whatsoever in the log.

It's worth to mention that MongoDB itself was 100% healthy and had no downtime or availability issues.

Second story

Recently we made a few changes related the way HTTPS gets terminated and how we pass on HTTP traffic from load balancer to Apache web server. Unfortunately due to webserver misconfiguration we accidentally introduced basic auth dialog on certain paths.

The change was implemented and we run AET as usual, but we ended up with this:

[INFO] Scanning for projects...
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building XYZ 1.0.0
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- aet-maven-plugin:2.0.1:run (default-cli) @ aet ---
[INFO] Successfully connected to tcp://10.xxx.yyy.zzz:61616
[INFO] HV000001: Hibernate Validator 4.3.2.Final
[INFO] HV000002: Ignoring XML configuration.
[INFO] CorrelationID: XYZ-XYZ-uat-suite-1488186346394
[INFO] ********************************************************************************
[INFO] ********************** Job Setup finished at 09:05:47.727.**********************
[INFO] *** Suite is now processed by the system, progress will be available below. ****
[INFO] ********************************************************************************
[INFO] [09:05:47.809]: COLLECTED: [success: 0, total: 49] ::: COMPARED: [success: 0, total: 0]
[WARNING] Report will be generated after timeout - some results might be missing!
[INFO] [09:07:49.136]: COLLECTED: [success: 0, total: 49] ::: COMPARED: [success: 0, total: 0]
[INFO] Received report message: FinishedSuiteProcessingMessage{correlationId=XYZ-XYZ-uat-suite-1488186346394, status=OK, errors=[Failed to generate reports because it took too much time!]}
[INFO] Report is available at https://aet-XYZcom.cognifide.com/report.html?company=XYZ&project=XYZ&correlationId=XYZ-XYZ-uat-suite-1488186346394
[ERROR] Failed to obtain xUnit report from: http://10.254.167.105:8181/xunit?company=XYZ&project=XYZ&correlationId=XYZ-XYZ-uat-suite-1488186346394
[INFO] Suite processing finished.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 04:04 min
[INFO] Finished at: 2017-02-27T09:09:50+00:00
[INFO] Final Memory: 14M/135M
[INFO] ------------------------------------------------------------------------

As you can see there was information only about some timeout. Based on the output itself it's hard to say where exactly that timeout happened and what might be the root cause. As in previous story there were Unknown errors on every single page. I wish there's a message about basic auth dialog somewhere to speed up troubleshooting.

There was also a similar story with the identical effect - basic auth credentials were changed, but no one updated Authorization header in the XML file, so the entire test failed. Again - no information about 401 or any other indication that something's wrong with the request itself.

Third story

By default Apache Karaf comes with 5 collector workers. We stumbled upon a situation where we had to decrease that value.

The background story behind that was, as usual, triggered by failed Maven AET build:

[INFO] ------------------------------------------------------------------------
[INFO] Building XYZ 1.0.0
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- aet-maven-plugin:2.0.1:run (default-cli) @ aet ---
[INFO] Successfully connected to tcp://10.xxx.yyy.zzz:61616
[INFO] HV000001: Hibernate Validator 4.3.2.Final
[INFO] HV000002: Ignoring XML configuration.
[INFO] CorrelationID: XYZ-XYZ-integration-suite-1488200959249
[INFO] ********************************************************************************
[INFO] ********************** Job Setup finished at 13:09:20.591.**********************
[INFO] *** Suite is now processed by the system, progress will be available below. ****
[INFO] ********************************************************************************
[INFO] [13:09:21.022]: COLLECTED: [success: 0, total: 49] ::: COMPARED: [success: 0, total: 0]
[WARNING] Report will be generated after timeout - some results might be missing!
[INFO] [13:11:22.879]: COLLECTED: [success: 0, total: 49] ::: COMPARED: [success: 0, total: 0]
[INFO] Received report message: FinishedSuiteProcessingMessage{correlationId=XYZ-XYZ-integration-suite-1488200959249, status=OK, errors=[Failed to generate reports because it took too much time!]}
[INFO] Report is available at https://aet-XYZcom.cognifide.com/report.html?company=XYZ&project=XYZ&correlationId=XYZ-XYZ-integration-suite-1488200959249
[ERROR] Failed to obtain xUnit report from: http://10.xxx.yyy.zzz:8181/xunit?company=XYZ&project=XYZ&correlationId=XYZ-XYZ-integration-suite-1488200959249
[INFO] Suite processing finished.

URL that points to xUnit report returned 500 error code and the following body:

HTTP ERROR 500

Problem accessing /xunit. Reason:

    Server Error
Caused by:

java.lang.NullPointerException
	at com.cognifide.aet.xunit.MetadataToXUnitConverter.convert(MetadataToXUnitConverter.java:64)
	at com.cognifide.aet.rest.XUnitServlet.generateXUnitAndRespondWithIt(XUnitServlet.java:113)
	at com.cognifide.aet.rest.XUnitServlet.process(XUnitServlet.java:86)
	at com.cognifide.aet.rest.BasicDataServlet.doGet(BasicDataServlet.java:79)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:618)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:725)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:652)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:445)
	at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:70)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:556)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1044)
	at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:124)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:372)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:189)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:978)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
	at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:81)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
	at org.eclipse.jetty.server.Server.handle(Server.java:367)
	at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:486)
	at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
	at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:926)
	at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:988)
	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640)
	at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
	at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
	at org.eclipse.jetty.server.nio.BlockingChannelConnector$BlockingChannelEndPoint.run(BlockingChannelConnector.java:298)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
	at java.lang.Thread.run(Thread.java:745)

It didn't say much what's going on. After a session with AET engineer we figured out that the server we run Apache Karaf on wasn't powerful enough to handle 5 Firefox browsers simultaneously, so we decreased that value to 3 and magically it started to work.

As in previous stories there was no indication what's going on and without support from AET engineer it's almost impossible to track the problem down on your own.

---

As you can see it can be quite tricky to troubleshoot and resolve issues you may encounter during your day-to-day work with AET.

Is it possible to make AET client logs more descriptive to give users some hints what actually went wrong? I believe there's a huge room for improvement in this area.

I'm quite convinced that without help of AET engineers (btw huge thanks for that!) I wouldn't be able to fix those issues.

Thanks,
Kuba

Does screen with css selector work?

AET version 2.0.2
Suite definition below seems to not work as expected.
It does't make screen of area by css selector, It makes always screen of whole page.
Does current AET version support this functionality ?

<?xml version="1.0" encoding="UTF-8" ?>

<suite name="test-suite" company="company" project="tests">
    <test name="google-homepage">

        <collect>
            <open/>
            <sleep duration="5000"/>
            <resolution width="800" height="600" />
            <screen name="logo" css="#hplogo"/>
        </collect>
        <compare>
            <screen collectorName="logo"> </screen>
        </compare>
        <urls>
            <url href="http:/google.com"/>
        </urls>
    </test>
</suite>

Compare against another suite patterns

We would like to be able to run a suite, but compare it to another suite patterns.

The case here is to maintain the master branch suite with up-to-date patterns
and run AET against sites deployed from feature branches.

The report is just to confirm nothing has changed: no need for accepting changes as new patterns,

Navigating between pages should remain user on the same tab

Let's say you want to check W3C errors on each page in your suite. When you navigate between pages, you are getting pulled back to the first tab.

I would expect to remain on the same tab/aspect of test when navigating through pages.

Improve documentation

Hi,
i tried my best to install the aet.
The wiki pages are different (maybe older) than the Readme here:
https://github.com/Cognifide/aet/tree/master/vagrant

After a lot of problems i was able to log in via putty.
I can open http://192.168.123.100/report.html#/suite with chrome.
I see a message "Failed to load metadata!" .
In my opnion i should now create a suite file inside my virtual machine, is this right?
I ask, because there is no maven installed in the VM?

Can you please advice me through the next steps in detail.
Are the wiki pages upto date?
Thanks in advance

Issue with site width

I want to test site with width 1920px. I run aet tests (2.1.0 prod) and i have 1904px on my aet results (-16px).
I tried to add 33 px for firefox scroll, so i used 1953px width - aet results : 1937px (-16px)

Code snippet:
...

                    <resolution width="1024" height="768"/>
                    <open/>
                    <wait-for-page-loaded/>
                    <screen name="Desktop"/>

...

Support Java 8 source format?

During one of the code reviews, I noticed that the sources of AET are written in Java 7 (this is used later on in the Maven Compiler plugin config)

Is there a reason the project hasn't migrated to Java 8?

It seems there's already some requirement to build (or use, I'm not sure )the project using Java 8 that has something to do with Bobcat.

https://github.com/Cognifide/aet/blob/master/pom.xml#L135

Js-errors collectors is storing pattern but it shouldn't.

Because of that:
-We're storing first collected js-errors artifact as a pattern which stays in DB forever;
it is not deleted while running cleaner job.
-Report app is calling for this artifact even if it's not used anywhere.

Sidebar goes under AET logo while scrolling

Whenever I scroll the on the left hand side all the information displayed there go under AET logo

AET version: 2.0.1
Browser: Chrome 56.0.2924.87

Support for more levels in navigation tree

It is possible to add support for more levels in navigation tree? It would be helpful for grouping tests
example:

suite100
+suite110
++suite111
+++test1111
+++test1112
+++test1113
++suite112
+++test1121
+++test1122
+++test1123

Fail fast option

Hey,

after a while with AET I came to a conclusion that it'd be great to have a fail fast option.

Our test suite lasts for around 10 minutes (2 Karaf workers) and the goal is to keep it green all the time. Right now we have to wait until the end to find out that 3rd out of 60 pages failed.

Is it possible to stop processing upon first error? This should significantly shorten feedback loop.

Regards,
Kuba

Support for regular expressions in all reasonable parameters of Filters and Modifiers

Sometimes it is hard to target specific set of W3C errors with the same type, because they differ, they are parametrized.
At the same time, using prefix seems to be too broad.
Please provide possibility to use regular expressions.

Ambiguous messages when there is a problem with test

From time to time there are problems with running AET tests. Test do not fail but some results are unavailable. The only comment displayed is as follows:

• Processing error occurred!

in most cases

• Processing error occurred! Details: ["NullPointerException: "]

in case of source code comparison

• Processing error occurred!
  Details:
  ["Unrecognized collector error: waiting for doc.body failed\nCommand duration or timeout: 10.40 seconds\nBuild info: version: 'unknown', revision: 'unknown', time: 'unknown'\nSystem info: host: 'WIN-8GT511MADOM', ip: '10.254.189.57', os.name: 'Windows Server 2012', os.arch: 'amd64', os.version: '6.2', java.version: '1.7.0_79'\nSession ID: a06fd0d1-8c1a-4e0b-8541-bd988b1ee42c\nDriver info: com.cognifide.aet.worker.drivers.AetFirefoxDriver\nCapabilities [{platform=WINDOWS, acceptSslCerts=true, javascriptEnabled=true, cssSelectorsEnabled=true, databaseEnabled=true, browserName=firefox, handlesAlerts=true, nativeEvents=false, webStorageEnabled=true, rotatable=false, locationContextEnabled=true, applicationCacheEnabled=true, takesScreenshot=true, version=38.6.0}]"]

in case of JS Errors.

The problem is related to 2 things:

• the tool should provide more information related to the problem

Microservices Architecture

Within this epic, AET will resign from OSGi in favor of microservices.
Architecture of AET should be lighter and more scalable AET.

Change regex data filter remove => replace

We could add support for regular expressions so that instead of removing part of source by regexp, we could replace it, in order to still compare the markup structure.

Screen tabs in report app are not displayed correctly

There's 5 different screen in my project. In AET report webapp it translates to 8 tabs in total (5 screens + 3 static ones: SOURCE W3C-HTML5, STATUS-CODES and JS-ERRORS). When your screen resolution is not high enough it seems that tabs which don't fit into screen width get wrapped around. This makes the app unusable, i.e. it's quite hard to use Show Mask feature then. Here's a screenshot of that bug:

AET version: 2.0.1
Browser: Chrome 56.0.2924.87

Keyboard shortcut and navigation buttons stop working after accepting whole URL

After accepting URL, keyboard shortcut [ and ] and navigation button >, < stops working. There are no fast way to go to next URL.