thingsinjars/hardy

Selenium-driven, cucumber-powered CSS testing.

hardy's Issues

Pull in latest Selenium

Double-check the changelog of recent seleniums to find out if we should pull in a newer one. If so, do it.

getting started

I'm having trouble getting started with hardy. I followed all the steps here http://hardy.io/getting-started.html
but when i try and run the example test i just see a blank firefox window and this output in the console.

$ hardy . 
Hardy v1.1.0

INFO Loading browser firefox
.F
INFO Shutting down browser
firefox success

am i missing something?

Unknown system error when running first test

I'm following your First Test example from http://hardy.io/getting-started.html.
I'm running it on a 64-bit Win 7 box.
I've performed all prereq steps successfully, but when I run the executable command, I get an unknown system error:

$ hardy .
Hardy v0.0.10

events.js:72
throw er; // Unhandled 'error' event
^
Error: spawn Unknown system errno 193
at errnoException (child_process.js:975:11)
at Process.ChildProcess._handle.onexit (child_process.js:766:34)

I'm not sure if this is an issue with my environment, or something else. I noticed I don't have hardy or phantomjs in my Path (your instructions during install say a hardy executable will be placed in my path). However, when I manually add it to my path, it makes no difference. Is this something you're familiar with or maybe have a suggested workaround or fix?

Thanks!

Only first selector in the page is tested

I tryed to test group of selectors.

My rule: "h2" should have "font-size" of "32px"

Page has few h2 tags. If first h2 selector successfully passed test, then test will be successfull even in case when other h2 tag will have other font-size (modified with inline styles or classes).

Ability to test a local html file?

I don't know if I am just doing something wrong in my setup, or if it's just not possible, But I am currently working on an internal css boilerplate, and would like to be able to test the main UI styles on a local test page that has all of the various components laid out on it, rather than comparing the differences of a live page.

Thanks in advance

Travis error: throw er; // Unhandled 'error' event

I'm trying run hardy in travis, but I'm not passing throw this error, some ideas about that?
https://travis-ci.org/diegoeis/locawebstyle/builds/53061854

> [email protected] test /home/travis/build/diegoeis/locawebstyle
> node_modules/hardy/bin/hardy --browser=phantomjs test
Hardy v0.0.10
events.js:72
    throw er; // Unhandled 'error' event
    ^
Error: spawn ENOENT
at errnoException (child_process.js:1011:11)
at Process.ChildProcess._handle.onexit (child_process.js:802:34)
npm ERR! Test failed. See above for more details.
npm ERR! not ok code 0
The command "bundle exec rake jasmine:ci && npm test" exited with 1.
Done. Your build exited with 1.

My .travis.yml

before_script:
  - sh -e /etc/init.d/xvfb start
  - export DISPLAY=:99.0
  - sleep 3
  - npm install
  - node_modules/hardy/bin/hardy selenium start
  - python -m SimpleHTTPServer&

 script:
  - bundle exec rake jasmine:ci && npm test

and my package:

"scripts": {
   "test": "node_modules/hardy/bin/hardy --browser=phantomjs test"
}

Process for cross-platform screenshots

When integrating Hardy into a build-pipeline, the environment where reference image diffs are taken is usually not the same one the comparison images are created.

Hardy should be platform aware and generate new reference cases if no references exist for this platform.

For example:

1. I create an image diff test on my mac
2. I create reference files locally then run the tests
3. I will use these reference renders as my base case so I commit them
4. I push to remote VCS
5. Linux-based CI server picks up the change
6. CI builds project, runs tests
7. Runs image diff test
8. Images are different due to platform differences.
9. Test fails

If Hardy were platform-aware, this would simply result in new reference images being created the same as a typical first-run image diff test.

Command-line interface

Add a simple CLI to make basic testing easier. This should encourage more people to start using it.

Is it possible to have a grunt module ?

Hello,

Is it possible to have a grunt module that can execute hardy tests ?

Logfile output ?

Is there any way to get a file as output, instead of a stdout ?

Running Hardy from a node_modules folder goes into infinite loop

When running Hardy from a project's node_modules folder, hardy goes into an infinite loop.

To reproduce:

mkdir tmp
cd tmp
npm install hardy
./node_modules/hardy/bin/hardy init
./node_modules/hardy/bin/hardy .

Output is:
Hardy v1.0.1
Using external selenium.
CSS Utils Steps Loaded
CSS Steps Loaded
Generic Steps Loaded
Loading browser firefox
.FShutting down browser
Loading browser firefox
.FShutting down browser
Loading browser firefox
...Shutting down browser
Loading browser firefox
.F-Shutting down browser
Loading browser firefox
..Shutting down browser
Loading browser firefox
.FShutting down browser
Loading browser firefox

Granted, the instructions do say to install it globally ... ;-)

False negative on font family attribute

Scenarios involving font-family checks do not seem to work as expected.

This scenario

Scenario: billboard-top should use AntennaExtraLight
Given I visit "http://global-ux.dev/prototypes/homepage/first"
Then ".billboard-top" should have "font-family" of "AntennaExtraLight, AntennaRE, Antenna, 'Helvetica Neue', Helvetica, Arial, sans-serif"

Yields

(::) failed steps (::)

".billboard-top" should have font-family of AntennaExtraLight, AntennaRE, Antenna, 'Helvetica Neue', Helvetica, Arial, sans-serif, measured: AntennaExtraLight, AntennaRE, Antenna, 'Helvetica Neue', Helvetica, Arial, sans-serif

Two versions of selenium included

Noticing that Hardy includes both selenium 2.32.0 and 2.35.0 jars. Looks like 2.32 is what's currently being used. We should remove one.

selenium start user feedback

I added logging and a check to see if the java process started.

https://gist.github.com/kraken-paul/c7c2ab8dd6d13038b3b1

Logging improvements

With configurable cucumber reportFormats now, we need a bit of improvement to logging and output in general so that it's easier to pass --logLevel=silent and only get the report out.

It wasn't particularly easy with all the logging done with console.log so hopefully the new stuff will be a bit better. Haven't pushed to npm yet because the windows compatibility needs checked.

Using :contains() pseudo class selector

Is it possible to use the :contains() pseudo class selector to assert text() values?

Visual diffs and reporters in Hardy

We're (@frio80 and I) trying to use Hardy for visual regression testing on a JS/CSS UI library.

Hardy gets us part of the way there by providing a tool that will 'fail' on a visual difference, but we haven't figured out how to create 'reports' on what changed.

The use-case is for developers iteratively building locally, and us giving them more information on what their changes effects. For example, if they visually change Component X, lets let them know if it changes Components Y and Z. They may have expected that change, they may not have.

It seems that we'd a couple of features to make this happen:

• Add the ability to create the visual diff in Hardy to get this to happen. (I've experimented with this in my dwabyick fork).
• Create a report of all failed tests from Hardy. This seems like a general Cucumber runner question, maybe. I'm new to Cucumber.
• A third 'maybe' feature would be more explicit support in Hardy for creating a baseline. Its kind of an implicit thing that happens the first time you run Hardy. It could help make it more intuitive for this use case.

Are these use-cases interesting to you? Happy to help as I can.

Add project snippet to PhantomJS's "Related Projects" wiki page

https://github.com/ariya/phantomjs/wiki/Related-Projects

Thanks! :)

Support for other image croppers/comparisons?

Great work on Hardy! We're looking forward to using it for our visual testing.

One challenge we have is with speed of running the tests. I took a swing at swapping out the PhantomJS based crop+compare functions with one that relies on GraphicsMagick (or ImageMagick). Performance is way faster.

For my tests on OSX, I'm getting about a 7X improvement in speed, at least.

Current
real 1m9.337s
user 0m27.102s
sys 0m6.516s

My fork
real 0m11.261s
user 0m0.883s
sys 0m0.359s

I imagine you might want to keep the PhantomJS testing for some scenarios, though I've noticed that even on Travis ImageMagick is supported.

I'm wondering what approach you would consider for a Pull Request. Direct swap is obviously easiest, but it could probably be configurable.

Cheers,
-Daniel

hardy can't find page elements

I'm using this page structure for my test:

<div class="container">
  <div class="page-header">
  <div class="row ui-sortable">
    <section class="module span4 actionable draggable scrollable">
      <header class="module-header">
      <div class="module-body">
        <div>
    <ul class="message-list">
      <li class="message-item">
        <a href="#">
          <span class="label label-important pull-left">Urgent</span><h4

Previously in .10 and early in .11, my tests were running successfully with this assertion:
Then "module-body" should have "color" of "#313337"

However, that assertion now generates this error:
NoSuchElement: An element could not be located on the page using the given search parameters.: "module-body"

I've tried using "div class = module-body", "section class = module span4 actionable draggable scrollable", "

", and other various combinations that either result in a similar error or a server error. Do I need to write the tests differently in my feature files to account for a recent change/update, or is this a bug? Thanks!

Windows 7 issues

Error:

> hardy --browser=firefox,phantom,chrome ./hardy_tests

Hardy v1.1.0
events.js:141
      throw er; // Unhandled 'error' event
      ^

Error: spawn cmd ENOENT
    at exports._errnoException (util.js:870:11)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:178
:32)
    at onErrorNT (internal/child_process.js:344:16)
    at nextTickCallbackWith2Args (node.js:442:9)
    at process._tickCallback (node.js:356:17)

npm ERR! Windows_NT 6.1.7601
npm ERR! argv "C:\\Program Files\\nodejs\\node.exe" "C:\\Program Files\\n
odejs\\node_modules\\npm\\bin\\npm-cli.js" "run" "hardy_test"
npm ERR! node v4.4.0
npm ERR! npm  v2.14.20
npm ERR! code ELIFECYCLE
npm ERR! [email protected] hardy_test: `hardy --browser=firefox,phanto
m,chrome ./hardy_tests`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] hardy_test script 'hardy --br
owser=firefox,phantom,chrome ./hardy_tests'.
npm ERR! This is most likely a problem with the rhythm-testing package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     hardy --browser=firefox,phantom,chrome ./hardy_tests
npm ERR! You can get information on how to open an issue for this project
 with:
npm ERR!     npm bugs rhythm-testing
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!
npm ERR!     npm owner ls rhythm-testing
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     C:\Rhythm\rhythm-testing\hardy-tests\npm-debug.log

Move to using Selenium

Using Selenium under the hood would allow the steps to be reused against a number of browsers, not just PhantomJS.