Giter Site home page Giter Site logo

galenframework / galen Goto Github PK

View Code? Open in Web Editor NEW
1.4K 74.0 167.0 8.87 MB

Layout and functional testing framework for websites

Home Page: http://galenframework.com

JavaScript 30.20% Shell 0.12% Java 64.60% CSS 1.13% HTML 3.86% Batchfile 0.01% Python 0.04% Ruby 0.02% FreeMarker 0.01%
layout-testing responsive-design java javascript image-comparison

galen's Introduction

master: Build Status

Galen is an open-source tool for testing layout and responsive design of web applications. It is also a powerfull functional testing framework. For more information visit http://galenframework.com

How does it work?

Galen Framework uses Selenium in order to open web browser and select the tested elements on page. It has a special language to describe the layout of web page for different browser sizes. You just need to define your own tags for devices and sizes and then using the galen spec language write checks for each element on page. The best way to test the layout is to check location and dimensions of elements relatively to each other. Here is a small example of basic syntax.

@objects
    header                  id  header
    menu                    css #menu
    content                 id  content
    side-panel              id  side-panel
    footer                  id  footer

= Main section =
    @on *
        header:
            inside screen 0px top, 0px left, 0px right

        menu:
            inside screen 0px left right
            below header 0px

        content:
            below menu 0px
            inside screen 0px left

    @on desktop
        side-panel:
            below menu 0px
            inside screen 0px right
            width 300px
            near content 0px right

    @on mobile
        content, side-panel:
            width 100% of screen/width

        side-panel:
            below content 0px

And here is a more advanced spec:

# example of using custom rules (functions)

@set userMargin 5 to 10 px

@objects
    user-*      div.users-list .user

@rule %{pattern} are below each other by %{distance} and aligned
    @forEach [pattern] as object, prev as prevObject
        ${object}:
            below ${prevObject} ${distance}
            aligned vertically all ${prevObject}

= Checking all users =
    | user-* are below each other by ${userMargin} and aligned

Conditional statements:

@objects
    banner-container    #banner-container

= Banner section =
    @if ${isVisible("banner-container")}
        banner-container:
            image file imgs/banner.png, error 5%

For more information please read Galen Framework Documentation

Contributing

If you want to contribute to this project just look for current open issues. Please let know in the comments of the issue that you are going to pick it up because somebody could already work on it. In the end just send the pull request. By the way the feature that you are going to work on should not just solve your particular problem. It should be extendable and configurable. The github issues is the best place to debate on the feature and discuss how it should be implemented.

All the work on the next version is performed in corresponding release branch (e.g. release-2.5). The master branch reflects the current live version. Most of the pull requests are accepted on release branch and not on master.

If you would like to make a change to the Galen Framework website (http://galenframework.com) you can do it here https://github.com/galenframework/galenframework.com

Add GPG key to your maven settings.xml:

      <properties>
          <gpg.keyname>C78F3CC4</gpg.keyname>
          <gpg.passphrase>Password</gpg.passphrase>
          ...

Setup the dependencies mvn clean install

The test can be run via mvn verify

To run integrations tests against chrome:

mvn clean verify -Dwebdriver.chrome.driver=/opt/dev/chromedriver -Dselenium.browser=chrome

Please ensure that you downloaded chromedriver, see here

Also you need Maven 3.3 or greater and Node with some modules:

  • sudo npm install -g grunt-cli
  • sudo npm install -g bower

Building

This project is based on Maven. You can use both Intellij or Eclipse for it. It is being developed in Linux so all the assembling scripts are in bash. In order to assemble the dist with zip archives just run ./makeDist.sh and it will create a dist folder with both binary and source folders and it will prepare zip archives there as well. If you want to quickly test the concept and install galen right after the assembling you can use script ./assembleAndInstall.sh. It uses makeDist.sh and then just invokes sudo ./install.sh

Testing

There are two levels of testing. The first one is just the regular mvn clean test. But as there is a lot of Javascript code - you need Mocha. The tests are located in folder src/test/js/. To run the just go to that folder and execute testJs.sh

License

Galen Framework is licensed under Apache License, Version 2.0

galen's People

Contributors

alecxe avatar decad avatar emalikterzi avatar fabiansawatzki1234 avatar fhoeben avatar guyarieli avatar hypery2k avatar ishubin avatar jimic avatar opx3 avatar phystem avatar sglahn avatar stefanbirkner avatar tuoris avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

galen's Issues

add spec to check that an element 'does not contain' another

In some cases, we need to check that after resizing some elements disappear. In terms of element contained in other element, it would be useful to add this expression. Something like:

elements-container:
contains: element-1, element-2, no element-3, no element-4

this way we ensure that element-3 and element-4 are not visibile

Introduce "template" special instruction for test suites

The "template" functionality should give a flexibility and improve test readability. The idea is explained below

@@ template login
    inject feature.js
    run login.js '{login: "$1", password: "$2"}'
@@ end

@@ template open
    run open.js '{pageName: "$1"}'
@@ end

Place order test on mobile device
    http://example.com/page  400x600
        @@ login [email protected]  qwerty
        @@ open order_page
        check myspec.spec --include "all,mobile"



@@ set 
    domain    example.com

@@ template page
    http://${domain}/$1  $2
@@ end

@@ template check 
    check $1 --include "$2" --exclude "$3"
@@ end

Place order test on mobile device
    @@ page /login  400x600
        @@ login [email protected]  qwerty
        @@ open order_page
        @@ check order-page.spec "all,mobile"

Or maybe we could do it even more complex with more indentations

@@ template test 
    $1
        http://example.com/$2   $3
              check $4  --include "$5"
@@ end

@@ test "Home page on mobile" "/home" "480x700" "homepage.spec" "all,mobile"

Add approximation for checking alignment

One case I came across, I wanted to check if two elements were aligned horizontally: a text input box and a label on top. However, due to the presence of 1px border probably, the driver does not return exact position.
I could not find any other spec to workaround this issue.
What if something like an offset (+1/-1px) were be allowed?

Set variables from command line or config file

Something like '-D' with maven.
One case where this would turn out useful is in parameterizing execution depending on test environment. If you need to use a different URL depending on the environment, it would be nice to have something like:

galen test testpages.test -DURL="http://example.com"

or
galen test testpages.test --config galen_config where galen_config overrides variables set in the .test file.

Names cannot be specified in a component section

Example:

@General | all
----------------------------------------
checkbox:
       below: item-heading

This is not run. Instead, when the name is removed the test is executed:

@ all
----------------------------------------
checkbox:
       below: item-heading

Need a tool for measuring a distance to other elements on page in a browser

It would be nice to have a tool that would allow to select any elements and give advices on how to construct Galen Specs.
It could be either a plugin for Chrome or Firefox or it could be a javascript which you would have to inject yourself on your webpage. Or it could be done by Galen Framework when it takes a snapshot of all elements on the page and then it would generate an html page with screenshot where you can get advices for each element.

Screenshots in IE have offset of 10 to 20 pixels

The way the screenshots are done are different depending on the browser. Perhaps we should not fix this yet and wait until this will be resolved by a selenium.
At the moment if you perform testing in IE with Galen - on the screenshots all highlighted areas are shown with offset of 10 to 20 pixels. Seems like a scrollbar is affecting this

NullPointerException if no test specifed

$ galen test --htmlreport report
Exception in thread "main" java.lang.NullPointerException
at net.mindengine.galen.GalenMain.runTests(GalenMain.java:147)
at net.mindengine.galen.GalenMain.execute(GalenMain.java:55)
at net.mindengine.galen.GalenMain.main(GalenMain.java:141)

Math operations in ranges

It should be possible to define ranges with simple math operations like here

content
    width: 100% of screen/width - 300 px + 5% of comment-box/width

Improve spec 'on' to make more useful and expressive

At the moment the spec on looks a bit awkward and it confuses with the orientation. It also does not allow to properly test that an object is located on the bottom right edge on another object.
That can be fixed by adding and edges declaration before colon like this:
We also need to introduce "up" and "down" words to make this spec less confusing but thats not a highest priority.

label
    on top left: container 10 px left, 20px down

e.g. when label is somewhere on the bottom right corner

label
    on bottom right: container 10 px up, 20px left

Dynamic ranges for template specs

Need to figure out the way to get the amount of elements inside template range.

When working with multiple (dynamic) objects on page we use the following syntax

==============
some-object-*     css .some-object
==============

[ 1 - 30 ]
some-object-@
    width: > 50px
    height: > 100px

[ 1 - 29 ] 
some-object-@
     above: some-object-@{+1} 10px

But in that case we need to know the precise amount of objects.
So if we don't know the exact amount of object - there is no way we can use template specs.

One of the ideas could be something like this

==============
some-object-*     css .some-object
==============

[ 1 - @{some-object-*} ]
some-object-@
    width: > 50px
    height: > 100px

[ 1 - @{some-object-* - 1} ] 
some-object-@
     above: some-object-@{+1} 10px

Or perhaps something like this:

==============
some-object-*     css .some-object
==============

[ 1 - @{:count(some-object-*)} ]
some-object-@
    width: > 50px
    height: > 100px

[ 1 - @{:count(some-object-*) - 1} ]
some-object-@
     above: some-object-@{+1} 10px

Another idea is to introduce functions in a bash-like processor that we have in test suites parser. so it could look in the end like:

[ 1 - ${sub(count("some-object-*"),1)}]
some-object-@
     above: some-object-@{+1} 10px

The problem is that all these template generated specs are initialized before doing actual checks. It is done when the spec is being parsed. In order to support dynamic ranges we need to change the way the spec is being loaded. Perhaps we should do it on the fly. Or we could just fetch all elements during parsing so by the time parser hits the line "[ 1 - @{:count(some-object-*)} ]" it is able to fetch the amount of objects. This means that parsing is depending on browser interaction

path relative to project path needs to be specified when using components inside a spec

How to reproduce:
All .spec files are in /spec folder.
When a spec uses a component which is in the same folder, Incorrect spec exception is shown.

E.g.
Inside page-to-test.spec

@general | all
----------------------------------------------------------
list-item-*
          component: list-item.spec

and spec folder contains list-item.spec
Then list-item.spec is found only when the path is specified:

          component:spec/list-item.spec

Html Report should allow to see passed object also on screenshot

It would be quite useful if Html report would always make a screenshot even when the tests are 100% passed.
This way it will be possible to later see what exactly is it testing and where the real area of a test object is on a screenshot.
What needs to be changed:

  1. Always make a screenshot before doing checks.
  2. Make all test objects clickable on html report page. When clicked - show popup with screenshot and highlighted object area

Another thing could also be done - highlight all objects on a single screenshot. This way we could spot things that were not covered by our tests.

OutOfMemoryError exception is thrown when making a screenshot of a very big page

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOfRange(Arrays.java:3209)
at java.lang.String.(String.java:215)
at java.lang.StringBuffer.toString(StringBuffer.java:585)
at org.json.JSONTokener.nextString(JSONTokener.java:305)
at org.json.JSONTokener.nextValue(JSONTokener.java:371)
at org.json.JSONObject.(JSONObject.java:206)
at org.json.JSONObject.(JSONObject.java:403)
at org.openqa.selenium.remote.JsonToBeanConverter.convert(JsonToBeanConverter.java:202)
at org.openqa.selenium.remote.JsonToBeanConverter.convert(JsonToBeanConverter.java:42)
at org.openqa.selenium.remote.HttpCommandExecutor.createResponse(HttpCommandExecutor.java:444)
at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:305)
at org.openqa.selenium.firefox.internal.NewProfileExtensionConnection.execute(NewProfileExtensionConnection.java:165)
at org.openqa.selenium.firefox.FirefoxDriver$LazyCommandExecutor.execute(FirefoxDriver.java:366)
at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:527)
at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:569)
at org.openqa.selenium.firefox.FirefoxDriver.getScreenshotAs(FirefoxDriver.java:327)
at net.mindengine.galen.browser.SeleniumBrowser.createScreenshot(SeleniumBrowser.java:80)
at net.mindengine.galen.reports.HtmlSuiteReportingListener.createScreenShot(HtmlSuiteReportingListener.java:232)
at net.mindengine.galen.reports.HtmlSuiteReportingListener.onSpecError(HtmlSuiteReportingListener.java:121)
at net.mindengine.galen.reports.HtmlReportingListener.onSpecError(HtmlReportingListener.java:152)
at net.mindengine.galen.runner.CombinedListener.onSpecError(CombinedListener.java:84)
at net.mindengine.galen.runner.GalenPageRunner.onSpecError(GalenPageRunner.java:113)
at net.mindengine.galen.validation.SectionValidation.tellOnSpecError(SectionValidation.java:223)
at net.mindengine.galen.validation.SectionValidation.checkObject(SectionValidation.java:210)
at net.mindengine.galen.validation.SectionValidation.checkObjects(SectionValidation.java:95)
at net.mindengine.galen.validation.SectionValidation.checkObjects(SectionValidation.java:116)
at net.mindengine.galen.validation.SectionValidation.checkSection(SectionValidation.java:61)
at net.mindengine.galen.validation.SectionValidation.check(SectionValidation.java:53)
at net.mindengine.galen.suite.actions.GalenPageActionCheck.execute(GalenPageActionCheck.java:62)
at net.mindengine.galen.runner.GalenPageRunner.run(GalenPageRunner.java:65)
at net.mindengine.galen.runner.GalenSuiteRunner.runPageTest(GalenSuiteRunner.java:100)
at net.mindengine.galen.runner.GalenSuiteRunner.runSuite(GalenSuiteRunner.java:67)

Import test suite functionality

It would be nice to have the same "import" feature in test suites like we have it for specs.
Same idea and syntax:

@@ import general.test
@@ Set 
       some_variable      ${someVariable_fromImportedTest}

Math parser should do more complicated parsing for spec templating

At the moment galen allows this:

[1 - 9]
menu-item-@
    near: menu-item-@{+1} 0 px right

But what would be nice to have is more complex parsing like this:

[1 - 9]
menu-item-@
    near: menu-item-@{*2 + 1} 0 px right

Or reusing the @ symbol together with brackets

[1 - 9]
menu-item-@
    near: menu-item-@{+(@*2 - @/(@-1))} 0 px right

Running in selenium grid should be configurable

Galen Framework should not make users choose what they should go for: local browser or selenium grid. Instead this should be configurable in a way that you can run your galen tests locally without Selenium Grid and on some CI server you could run it against Selenium Grid. This configuration could be passed via system properties with "-D" argument in command line

add screen size in the report

A small enhancement would be to add screen size in the single report page so that user do not need to jump back and forth on .test definitions. This could be particularly useful when the number of devices is large.

Screenshots are bad in Chrome

At the moment Selenium has different ways of doing screenshot for Firefox, Chrome and IE. We should think on a workaround for this as in Chrome it takes only the visible viewport area which is not useful for Galen testing

Color histograms verification and image comparison

Would be useful to check color histograms of specific page object on page.
The idea is:

  1. Take a screenshot and crop to page object area
  2. Calculate image histogram
  3. Verify it using the syntax below:
menu
    color scheme is: 80 to 90% black, 10 to 20% white

another example of using color spec

    color scheme is: ~ 40% #ff00a3

Also we could make a basic image comparison with the following syntax:

company-logo
    image like: images/company-logo.png   20% error rate

Introduce another type of object in page spec: "text"

The problem is that Galen takes boundary box of whole block instead of visible text. e.g. h1 tags.
But it would be useful to take only the area of visible text. The simplest and not the best way to go is like this:

  1. Take a screenshot
  2. Modify text (change opacity, color etc.)
  3. Take another screenshot and make a diff with previous one.
  4. Take the area of text block and analyze colors

Html reporter should also give overview of all failed specs

At the moment once you get a big report file with around 1000 checks - it is hard to scroll to failed checks. There should be a way to aggregate all failed specs with error messages in one place (perhaps in the beginning of a report we could put a "Failed" section )

Variables processing for specs

Currently in Galen Test Suite syntax we have bash-like variable processing. Would be cool to implement the same feature for page specs together with variable definitions. e.g.

@@ Set frameWidth  400

container
    width: ~ ${frameWidth} px
``

Support full page screenshots

For the moment not all browser support full page screenshots. For many browsers Selenium makes screenshot for visible area only. In order to support screenshots for a complete page Galen should scroll the page multiple times and create screenshots of all parts of a page. After that Galen should assembly those parts in a single image

"viewport" object width is not calculated properly

Currently in version 0.6.2 the "viewport" object width is returned with extra 10-20 pixels. e.g. if viewport width is 400 px than Galen thinks it is 410 px. It seems that it takes also scrollbar width into "viewport" width but it doesn't make any sense.

broken layout in html report

<html>
    <head>
        <title>Galen Report</title>
        <link rel="stylesheet" type="text/css" href="galen-report.css"></link>
        <script src="jquery-1.10.2.min.js"></script>
        <script src="galen-report.js"></script>
    </head>
    <body>
        <script>$(function() {Galen.formatMainOverview();});</script>
        <div class="suites-overview">
            <h1>Galen Report</h1>
            <div class="suites">
                <div class="status-header">
                    <div>Passed</div>
                    <div>Failed</div>
                    <div>Total</div>
                    <div>Objects</div>
                    <div>Test</div>
                </div>
                                <div class="suite">
                    <div class="status passed">11</div>
                    <div class="status failed">0</div>
                    <div class="status total">11</div>
                    <div class="status objects">6</div>
                    <div class="suite-link">
                        <a href="report-1-login-page-on-mobile-device.html">Login page on mobile device</a>
                    </div>
                </div>
                <div class="suite">
                    <div class="status passed">14</div>
                    <div class="status failed">2</div>
                    <div class="status total">16</div>
                    <div class="status objects">6</div>
                    <div class="suite-link">
                        <a href="report-2-login-page-on-tablet-device.html">Login page on Tablet device</a>
                    </div>
                </div>
                <div class="suite">
                    <div class="status passed">13</div>
                    <div class="status failed">2</div>
                    <div class="status total">15</div>
                    <div class="status objects">6</div>
                    <div class="suite-link">
                        <a href="report-3-login-page-on-desktop-device.html">Login page on Desktop device</a>
                    </div>
                </div>
            </div>
        </div>
    </body>
</html>

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.