redhatqe/katello.auto

Automated tests for katello

Katello.auto GUI automation

Usage

There are several options for running automation. The easiest way to run it is to install Leiningen and its runproject plugin

Running pre-packaged tests from command line

Run

$ lein runproject com.redhat.qe/katello.auto 1.0.0-SNAPSHOT -h

to get a list of command line options.

An example command line would be

$ lein runproject com.redhat.qe/katello.auto 1.0.0-SNAPSHOT -s https://my.host/katello katello.tests.suite/katello-tests  katello.tests.suite

where the last argument is the test group to run. See suite file for list of available groups.

By default this will start up several firefox browsers and run the specified tests in parallel.

Viewing results

After the tests finish running, there will be a file testng-report.xml in the current directory. You can either view this file directly in an editor or browser, or if you prefer a nicely formatted HTML report, Jenkins with the TestNG plugin can generate one for you. The internal Red Hat QE Jenkins server has a job set up to display the result - just upload the xml file.

If you are running your own Jenkins server, this job is very easy to set up. Just add a file parameter testng-report.xml, and then check the box Publish TestNG Results and fill in *.xml.

katello.auto's Issues

Values in automation-properties.clj overridden

Values in automation-properties.clj are not being honored, and are being overridden by hard-coded defaults.

upload manifest does not wait for async notification

It will pop up later and mess up whatever test is running. I think delete-organization does the right thing.

Unhelpful error message when non-existent suite is specified

get NPE when you typo a suite to run on the command line eg, ketello.tests.login/login-tests

need to catch and print better error

Add eclipse support for running interactive w binary (nREPL)

I think should just involve add nREPL as a dep, and adding the command to start nREPL to the README with instrux on how to connect from eclipse (Window -> Connect to REPL).

Pull automation-properties.clj from within project

Move the existing file to automation-properties.clj.sample, and ignore (.gitignore) automation-properties.clj. Then, look for that file in $PROJECT, instead of $HOME

Add more tasks for searching

so you need a function that takes 1) a search term and 2) a function
that takes each resulting item in the left pane and verifies it
matches the search
jweiss, yes..exactly

ok so i can add those two things - HOF for search validation, search
with :add-to-favorites true?
and then i guess a function to run a previously saved search [08:46]
jweiss, yes.. absolutely..

Long rest JSON responses are truncated

seems like http/await really does do something beyond the waiting that is currently done. add that back in.

Request for New Test: Add a test for replacing the default Admin

Test case is documented at https://url.corp.redhat.com/3e6387e

This is a post install test using the webui to create a new administrative user and then log in as that user and delete the original (default) admin user.

Create a bootstrap file and get rid of comment block in project.clj

As part of the "consumability" effort, a user of automation should be able to start slime, and just do

(load "bootstrap")
or
(require 'bootstrap)

and it should initialize, start an embedded selenium server, and log in to katello

Stop depending on Katello's notification system to determine if a test passed.

It's unreliable - use the API or existing UI construct to verify. Check for error messages, still, and report them. Whether to fail a test because of a red notification even if the verification itself works, still TBD.

Multiple unexplained failures in test run.

https://url.corp.redhat.com/d2948a3

Test results have 4 failed tests against this git repo, TAG (branch) set to SAM-1.1. Of the 4 there are 3 that failed and are unexplained as in manual testing all tests passed and evidence of missing elements is in accurate.

Create and Activation Key:
Worked Manually,
Automation: com.thoughtworks.selenium.SeleniumException: ERROR: Element /div[contains(@Class,'jbreadcrumb')]//a[normalize-space(.)='Development'] not found
Element does exist.

Rename an existing system:
Worked manually
Automation: com.thoughtworks.selenium.SeleniumException: ERROR: Element //div[@id='list']//div[starts-with(normalize-space(.),'newsystem-1338561247710')]
Element does exist

Verify system appears on systems by environment page in the proper environment
Worked Manually
Automation: com.thoughtworks.selenium.SeleniumException: ERROR: Element //a[normalize-space(.)='Development' and contains(@Class, 'path_link')] not found
System and Environment elements are present

It is possible this is all related to timing. Is it possible to add explicit or implicit waits to see if a required element appears in XX seconds. This would be in lieu of the default time out.

Test case 183426

Add a new system to cloned system group and perform package install

automation-properties is not found unless --config is specified

Should find the file using a default location of the project root dir.

Create test for synchronizing a repository where username has latin-1 characters

We a test case to cover the following scenario:

• Admin user creates user Mané
• Admin user creates organization Cuiabá with the following environments:
  • Library -> Desenvolvemento -> ControleQualidade
• User Mané creates a custom provider, with a product and a repository
• User Mané synchronizes this newly added repository.

The outcome should be that the repository gets synchronized. See https://bugzilla.redhat.com/show_bug.cgi?id=835586.

Navigate to system by env tab is failing

https://url.corp.redhat.com/1476ea0

Package as jar file

If someone wants to just run the automation, not write it, would be nice to package as a jar file and just stick it on a webserver for them to download.

Probably need a script to generate the jar (since we don't want the precompile done)

Will have to figure out how to allow both a swank session and just command line running

subscribe-system will unsubscribe from a product if it's already subscribed

it just blindly checks box and hits save, doesn't notice the product is listed in the "Current subscriptions" div.

Request for new automation: Add automation for setting YStream (Release Version)

Please review the following Test Case for inclusion in automation.

https://url.corp.redhat.com/7f17ce3

Browser Window should not open to full screen

During development this is problematic, Please set up the tests to not launch the browser window in fully screen (note this is not a maximized window, it is simply set to be size equal to window).

Test case 18347

Add a new system group and edit the max limit

Script to update autodocs

See http://tomfaulhaber.github.com/autodoc/

Should script this up and check in to the source

Test Case: 185934

[system_group] System count +1: from registration through AK

Make unique strings formatted timestamps instead of unix time in ms

unique names would look better as

redhat-0615-092611-001 (jun 15 9:26:11.001am)
rather than
redhat-1339700967530 (which a human can't easily determine what date that is)

Add (browser ...) symbol when (load "bootstrap") executes

Add the (browser ...) symbol so you can begin running browser commands immediately after bootstrapping.

[high] Browser sessions does not get closed error occures

Trying the command execution (via java -jar katello.auto-1.0.0-SNAPSHOT-standalone.jar -m katello.tests.suite -s https:///katello katello.tests.organizations/org-tests) against latest project *.jar fails on my side with leaving all the 3 firefox instances open...

Exception in thread "test.tree-thread0" com.thoughtworks.selenium.SeleniumException: ERROR: Element //input[@value='Log In'] not found
at com.thoughtworks.selenium.HttpCommandProcessor.throwAssertionFailureExceptionOrError(HttpCommandProcessor.java:97)
at com.thoughtworks.selenium.HttpCommandProcessor.doCommand(HttpCommandProcessor.java:91)
at com.thoughtworks.selenium.DefaultSelenium.click(DefaultSelenium.java:167)
at com.redhat.qe.auto.selenium.ExtendedSelenium.click(ExtendedSelenium.java:180)
at com.redhat.qe.auto.selenium.ExtendedSelenium.click(ExtendedSelenium.java:193)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28)
at com.redhat.qe.auto.selenium.selenium$call_sel.doInvoke(selenium.clj:36)
at clojure.lang.RestFn.applyTo(RestFn.java:139)
at clojure.core$apply.invoke(core.clj:601)
at fn.trace$trace_fn_call$fn__4462.invoke(trace.clj:44)
at fn.trace$trace_fn_call.invoke(trace.clj:43)
at fn.trace$rebind_map$iter__4476__4480$fn__4481$fn__4486.doInvoke(trace.clj:68)
at clojure.lang.RestFn.invoke(RestFn.java:421)
at com.redhat.qe.auto.selenium.selenium$fill_form.doInvoke(selenium.clj:84)
at clojure.lang.RestFn.invoke(RestFn.java:425)
at katello.ui_tasks$login.invoke(ui_tasks.clj:442)
at clojure.lang.AFn.applyToHelper(AFn.java:163)
at clojure.lang.AFn.applyTo(AFn.java:151)
at clojure.core$apply.invoke(core.clj:601)
at fn.trace$trace_fn_call$fn__4462.invoke(trace.clj:44)
at fn.trace$trace_fn_call.invoke(trace.clj:43)
at fn.trace$rebind_map$iter__4476__4480$fn__4481$fn__4486.doInvoke(trace.clj:68)
at clojure.lang.RestFn.invoke(RestFn.java:421)
at katello.setup$start_selenium.invoke(setup.clj:32)
at clojure.lang.AFn.applyToHelper(AFn.java:159)
at clojure.lang.AFn.applyTo(AFn.java:151)
at clojure.core$apply.invoke(core.clj:601)
at fn.trace$trace_fn_call$fn__4462.invoke(trace.clj:44)
at fn.trace$trace_fn_call.invoke(trace.clj:43)
at fn.trace$rebind_map$iter__4476__4480$fn__4481$fn__4486.doInvoke(trace.clj:68)
at clojure.lang.RestFn.invoke(RestFn.java:397)
at katello.setup$thread_runner$fn__4634.invoke(setup.clj:68)
at clojure.lang.AFn.run(AFn.java:24)

at java.lang.Thread.run(Thread.java:679)

Could you please make all the commands having java-like finally step for closing the browser session in case of ANY kind of failure(s).

providers/get-validation-data is too ugly.

could have used macro quote/unquote here but didn't. fix this

Create testcases for "Create a GPG Key", "Delete a GPG Key"

Should go in katello.tests.providers, probably at the top-level (after the entire branch for "Create a custom provider").

There is already a task in katello.ui-tasks/create-gpg-key for creating, the test will probably just be a 1-liner to call it.

Then nested inside the create test should be the delete test. (Can't test deleting if you can't create, right?) Deleting will need it's own ui-task - can call it (defn delete-gpg-key [gpg-key-name] ...)

See also katello.locators/page-tree (already has some gpg related pages in there).

Add test to create a new organization using Manage Organizations link

The new web ui now allows the creation of a new Organization via the new Orgs dropdown widget. Could we add a new test that uses it in addition to the existing tests that use the same functionality via the Administer link?

• Click the Organizations dropdown widget and select the Manage Organizations link.

Automate test: Create Katello Role apply to LDAP Group

General test here to create a role in katello and apply the role to a ldap group. For the purpose of this testing we have selected +All for permissions, essentially granting the group and its members admin privileges.

https://url.corp.redhat.com/bc15652

Testing here must detect if ldap is enabled or not. The current easiest way I know to do this is to inspect the roles/users tab to see if ldap group is one of the options for adding to a role. Another way, is by selecting the li.hello and finding that the user can not manage their password and email address details.

Verify by logging in as a user that is a member of the group used in test, NONE of this can be random because the group and its users must exist in ldap.

LDAP is currently set up on morbo, no ssl.

details in the test case.

Skip e2e tests if no clients are specified

Add package filter functions

[RFE] Add command line arguments (options) to enhance user experience

Please allow for the passing of command line options to make it easier and more intuitive to set the "BrowserType" and "SeleniumServer". I understand that these can be passed in as config file options; however, they can change so frequently is is actually easier to pass them at the command line. Having both options would be ideal; default values, configuration overrides, and / or cli options override.

Options that do not change frequently; i.e. admin user and password don't necessarily have to be available on the cli. Consider also flags to indicate test types or configurations to run against; i.e. --ldap to test against a ldap configuration v. not passing the flag to indicate testing a non-ldap configuration.

Existing search tests are giving false positives

See "Search for an organization" and "Search for a user"

they both create an entity and then search for some prefix of the entity's name. But the search term doesn't have a * at the end, so there are 0 results. And the verification is just to check that all search results contain the search term. There are zero results so it returns true.

Needs to check that the specially created entity is in the results also.

Create testcase for uploading a mangled manifest file.

Invalid manifests should be rejected.

Ping me if you need specific ways to possibly mangle said manifest.

Request for new automation: Add automation for setting SLA

Please add functionality and / or test for the following:

https://url.corp.redhat.com/494512e

Convert Eric's upload manifest tests

from headpin.auto

Process does not exit if using embedded selenium server

If you do lein run without specifying a selenium-address, it uses an embedded one. But the threads hang around after automation is finished. the main function should shut down sel server if there is one.

Required two functions: 1)clear the search and 2) remove favorite search

Properly handle multiple notifications

When there are several notifications on screen at once after some action, the current auto will only return the first one.

Sometimes that is an irrelevant notification (like "Sync successful" for some sync that was scheduled or started a long time ago). So really what needs to happen is all the notifications should be returned, and check-for-error should either fail only when all the notifications are errors, or one of them is (i suppose this could be an option for the caller). There's a lot of complication involved here, but I don't see any other way to avoid tests failing because of extra notifications.

Add ability to use an embedded selenium server

May not want to always include selenium server as a dep because it's rather large (30mb), but including it as a dev dependency might be a good idea

[org.seleniumhq.selenium/selenium-parent "2.21.0"]

Parameters in screenshot filename cause them to be too long, too many special characters

Tomcat apparently has trouble serving up a file when its name is 200 chars long and has a bunch of parens and colons and braces in it. Probably should just timestamp the test name instead of trying to append the parameters to it.

Setup for ...-tests should not report pass fail

Because "setup for ...-tests" are not actually tests it is misleading to have these report pass|fail as they're not actual tests. The tests themselves should fail if the setup fails or if the setup fails then simply fail or skip the children.

Representation of function parameters in the results are inconsistent and sometimes verbose

For data driven tests, see stuff in jenkins output like:
Parameter #1 (clojure.core/let [expected-validation-err :repository-url-invalid] (fn [result] (some #{expected-validation-err} (:validation-errors result))))

Perhaps should read:

Parameter #1 (is-validation-error? :repository-url-invalid)
Parameter #2 {:url "", :name "mytestcp-1336509424775", :description "blah"}

This can already be done, but requires a bit of extra metadata to tell the function how to print itself out (don't print its contents, just how it was called).

Add .settings to .gitignore

Rename an existing system fails

This test is consistantly failing though manually this works. I was able to navigate to systems/newsystem-xxxxx/Details/System Info and modify the system name specified. upon clicking save the green success dialogue appeared and the new name was listed in the left pane.

Update README

The README page should be updated with instructions on how to "install" the framework, any preliminary steps that need to be taken prior to using it and examples on how to use it.

Tests assume they are on the admin org (ACME_Corporation) - not always true

Check out this trace:

   [(katello.ui-tasks/upload-subscription-manifest
     "/home/hudson/workspace/katello-gui/manifests/scenario5_O1_M2.zip"
     {:repository-url "http://hudson.rhq.lab.eng.bos.redhat.com:8080/shared/cds/"})
    [(katello.ui-tasks/navigate :redhat-subscriptions-tab)
     [(com.redhat.qe.auto.selenium.selenium/call-sel "isElementPresent" :log-out) true]
     [(com.redhat.qe.auto.selenium.selenium/call-sel "isElementPresent" :confirmation-dialog) false]
     [(com.redhat.qe.auto.selenium.selenium/call-sel "mouseOver" :content) nil]
     [(com.redhat.qe.auto.selenium.selenium/call-sel "mouseOver" :subscriptions) nil]
     [(com.redhat.qe.auto.selenium.selenium/call-sel "clickAndWait" :red-hat-subscriptions) nil]
     nil]
    [(com.redhat.qe.auto.selenium.selenium/call-sel "isElementPresent" :choose-file) false]
    [(com.redhat.qe.auto.selenium.selenium/call-sel "click" :import-manifest) nil]
    [(katello.ui-tasks/in-place-edit {:redhat-provider-repository-url-text "http://hudson.rhq.lab.eng.bos.redhat.com:8080/shared/cds/"})
     [(katello.ui-tasks/activate-in-place :redhat-provider-repository-url-text)
      [(com.redhat.qe.auto.selenium.selenium/call-sel "click" "//div[@name='provider[repository_url]']") nil]
      nil]
     [(com.redhat.qe.auto.selenium.selenium/call-sel "getElementType" :redhat-provider-repository-url-text) "textbox"]
     [(com.redhat.qe.auto.selenium.selenium/call-sel
       "setText"
       :redhat-provider-repository-url-text
       "http://hudson.rhq.lab.eng.bos.redhat.com:8080/shared/cds/")
      nil]
     [(com.redhat.qe.auto.selenium.selenium/call-sel "click" :save-inplace-edit) nil]
     nil]
    [(katello.ui-tasks/check-for-success)
     [(katello.ui-tasks/notification nil)
      [(com.redhat.qe.auto.selenium.selenium/call-sel "waitForElement" :notification "15000") nil]
      [(com.redhat.qe.auto.selenium.selenium/call-sel "getText" :notification) "×Provider 'Red Hat' was updated."]
      [(com.redhat.qe.auto.selenium.selenium/call-sel "getAttributes" :notification)
       "<properties {tagname="DIV," style="background-color:" yellow;="" outline:="" 1px="" solid="" rgb(136,="" 255,="" 136);,="" class="jnotify-notification" jnotify-notification-success}="">"]
      [(katello.ui-tasks/clear-all-notifications)
       [(com.redhat.qe.auto.selenium.selenium/call-sel
         "isElementPresent"
         #=(com.redhat.qe.auto.selenium.Element. "xpath=(//div[contains(@class,'jnotify-notification-error')]//a[@class='jnotify-close'])[1]"))
        false]
       nil]
      {:type :success, :msg "×Provider 'Red Hat' was updated."}]
     [(katello.ui-tasks/success? {:type :success, :msg "×Provider 'Red Hat' was updated."}) true]
     {:type :success, :msg "×Provider 'Red Hat' was updated."}]
    [(katello.ui-tasks/fill-ajax-form {:choose-file "/home/hudson/workspace/katello-gui/manifests/scenario5_O1_M2.zip"} :upload)
     [(com.redhat.qe.auto.selenium.selenium/call-sel "getElementType" :choose-file) "input file"]
     [(com.redhat.qe.auto.selenium.selenium/call-sel "setText" :choose-file "/home/hudson/workspace/katello-gui/manifests/scenario5_O1_M2.zip") nil]
     [(com.redhat.qe.auto.selenium.selenium/call-sel "click" :upload) nil]
     ([:choose-file "/home/hudson/workspace/katello-gui/manifests/scenario5_O1_M2.zip"])]
    [(katello.ui-tasks/async-notification 600000)
     [(katello.ui-tasks/notification)
      [(com.redhat.qe.auto.selenium.selenium/call-sel "waitForElement" :notification "15000")
       #=(com.thoughtworks.selenium.SeleniumException. "Timed out after 15000ms")]
      nil]
     [(com.redhat.qe.auto.selenium.selenium/call-sel "refresh") #=(com.thoughtworks.selenium.SeleniumException. "Timed out after 60000ms")]
     #=(com.thoughtworks.selenium.SeleniumException. "Timed out after 60000ms")]
    #=(com.thoughtworks.selenium.SeleniumException. "Timed out after 60000ms")]

Just above the test fails, timing out trying to get a success notification, and then timed out refreshing the page. Then below the failed test tries to switch back to the admin org before failing. But that fails too.

   [(katello.ui-tasks/switch-org "ACME_Corporation")
    [(com.redhat.qe.auto.selenium.selenium/call-sel "click" :org-switcher)
     #=(com.thoughtworks.selenium.SeleniumException. "Timed out after 60000ms")]
    #=(com.thoughtworks.selenium.SeleniumException. "Timed out after 60000ms")]
   #=(com.thoughtworks.selenium.SeleniumException. "Timed out after 60000ms")]]

The problem is that a failure is a failure, there's no specific way to fail so that the next test knows its starting in an unexpected state. The next test needs to do that for itself.

Throughout the suite we need to remove the assumption about the current org, and switch explicitly. The switch routine should probably just check the current value before switching to be more performant.

Testcase 183431

Remove a cloned system group

Upload manifest test is not receiving cmdline opt for cdn url

https://url.corp.redhat.com/bc990a3

Force upload checkbox has been removed from web ui

As per https://bugzilla.redhat.com/show_bug.cgi?id=839005 the force checkbox has been removed from the web ui in the page for importing manifests. We should update our tests accordingly.