Giter Site home page Giter Site logo

specter's Introduction

Codacy Badge Code Climate bitHound Score Stories in Ready

Specter

Specter is the automated visual regression testing framework that you've been wishing for.

For most other forms of software development there exist unit testing frameworks with which a developer can verify that new code does nothing to break existing code, but for the front-end (UI) developer, there are very few tools to test design and layout, and the ones that do exist are terribly clumsy. This has always resulted in a huge need for heavy manual testing and QA after changes are made, to ensure that a change in one place doesn't break something somewhere else.

Specter aims to change that. After you've built a page and you know it's right you can set up a Specter test to prove that page, and each of its individual components are correctly rendered. Specter will capture screenshots of the elements matching the selectors you specify, at all the screen dimensions you desire. Then, just run your suite of Specter tests the next time you make a change to ensure you haven't broken anything. If something does break, you'll get a nice screenshot with the differences highlighted in red.

Requirements

Specter is a XUL application, so it needs Firefox or XULRunner in order to work.

Installation

To use Specter, just clone this repo and run make and sudo make install to link the binary into your path and make it executable.

$ git clone https://github.com/letsgetrandy/specter.git
$ cd specter
$ make
$ sudo make install

Usage

Check out the demo for a full working example (run specter demo from the command line).

Quick Start

Create your test file(s). Eg:

open("http://www.google.com/", function() {

    test([960, 640, 320], function() {
        capture("#footer", "footer");
    });

    finish();
});

Then, invoke specter on your tests. Specter can run a single test file:

$ specter tests/test-login.js

Or, multiple files:

$ specter tests/test-login.js tests/test-homepage.js tests/test-checkout.js

Or, it can recursively run every test file in a specified directory:

$ specter tests

Baselines

The first time Specter captures a selector for a particular test, there is nothing to compare it to, so that capture becomes the baseline. After that, any time that selector is captured for that test, it will be compared to the baseline.

When you make an intentional change, you'll wish to update the baseline. You do this with the --rebase option.

# update the baseline for the homepage tests
specter --rebase tests/test-homepage.js

Configuration

Specter follows standard .rc file behavior.

  • Global defaults can be set in /etc/specterrc
  • User-level settings can be set in ~/.specterrc
  • Project-level settings can be set within the project in a .specterrc file in the current working directory, or any directory above it.
$ cat .specterrc

[paths]
testroot = static/styles/tests
baseline = static/styles/screenshots
diffdir  = /tmp/specter

[hostnames]
mycoolsite = http://localhost:5000
myothersite = http://localhost:8080

testroot is the folder that contains all the test files to run. No value is required here, but specifying the root directory of your tests will help Specter to put your baseline and diff images into more meaningful directories.

baseline is the directory in which baseline screen captures should be stored

diffdir is where diff images are stored for comparison later. This is typically a temp directory of some kind.

[hostnames] allows you to define custom hostnames where tests should be run. In this example, mycoolsite and myothersite are directories directly below the testroot (static/styles/tests), which is a path relative to the location of the .specterrc file.

Questions?

If have other questions about Specter, what it does, how it works, or why, have a look at the FAQ, and if you don't see the answers you're looking for there, contact me!

Changelog

A list of changes made in each version can be found on the releases page


Created and maintained by Randy Hunt

Portions of Specter were inspired by PhantomCSS, and by SlimerJS.

specter's People

Contributors

hdennison avatar letsgetrandy avatar sedovsek avatar thingsinjars avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

specter's Issues

Create some way to ignore certain elements.

Not even sure this is possible. I imagine it would be, because you could simply make an ignored element visibility:hidden prior to testing.

A common failure I run in to on specter tests is an element that will be different depending on, say, geolocation.

Here's a common scenario.

  1. A baseline exists with a customer service phone #.
  2. My specter test fails because locally, the customer service phone # is different.

It would be nice to be able to capture("header", "site-header") but also be able to ignore the customer service phone # in this instance.

Not sure of a good recommendations for syntax right now.. but perhaps something like this:

test(['1024', '768', '640'], ignoring['header-phone', 'some-other-el'], function() {
capture('header', 'site-header');
});

Exception... Component returned failure code: 0x80004001 -- innocuous?

This seems to just be an innocuous warning of some kind, but it happens whenever I run specter (btw: hello! :))

error cache service: [Exception... "Component returned failure code: 0x80004001 (NS_ERROR_NOT_IMPLEMENTED) [nsICacheService.evictEntries]" nsresult: "0x80004001 (NS_ERROR_NOT_IMPLEMENTED)" location: "JS frame :: jar:file:///Users/dannypeck/Sites/specter/bin/omni.ja!/chrome/components/commandline.js :: CommandLine.prototype.handle :: line 37" data: no]

Support for "specter-as-a-service" remote captures

Since each operating system renders pages with slight differences, teams working in mixed environments will needs a way to capture all baselines from the same platform in order to implement reliable team-wide regression testing.

Add support for beforeEach and afterEach functions

Specter should provide some means of reducing repetition by allowing setup and teardown functionality to be defined.

Proposed implementation is to allow an _events.js file, where handlers such as onload and onunload can be defined.

Allow _events.js definitions to work recursively

Event handlers defined in _events.js should work for all files in the current directory or in any subdirectory thereof.

Implementation detail: should definitions in subdirectories override definitions in parent directories? Or should they work in addition to those definitions above?

Add tests

There should be some way to run a set of tests to prove that specter is doing everything that we expect it to do.

Perhaps there could be a specter test command that runs unit tests and verifies that we get the output we expect. Or maybe just a directory full of shell scripts. Who knows?

I imagine this will involve firing up an on-the-spot web server, and parsing the STDOUT from Specter. Or perhaps specter could dump some more detailed results info into the environment after running.

specter throwing error when specifying only a directory

I'm in directory foo and my .specterrc is also in this directory and looks like the following:

[paths]
testroot = tests
baseline = tests/baseline
diff = tests/diff
fail = tests/fail

The tests folder is also in foo.

When I run the specter command from the foo directory, I get the following error.

specter tests

Traceback (most recent call last):
File "/usr/local/bin/specter", line 152, in
if path[0] == os.path.sep:
IndexError: string index out of range

Format errors, warnings better (newlines)

Instead of:

.......F.................NotFoundError: Unable to capture '.std_testing ' in /Users/dannypeck/Sites/analytemedia/dev-git/static/styles/tests/stx/test-site_map.js
EFFFNotFoundError: Unable to capture '.std_testing ' in /Users/dannypeck/Sites/analytemedia/dev-git/static/styles/tests/stx/test-site_map.js
EF..NotFoundError: Unable to capture '.std_testing ' in /Users/dannypeck/Sites/analytemedia/dev-git/static/styles/tests/stx/test-site_map.js
EFFFNotFoundError: Unable to capture '.std_testing ' in /Users/dannypeck/Sites/analytemedia/dev-git/static/styles/tests/stx/test-site_map.js
EF................................

Could be:

.......F.................
NotFoundError: Unable to capture '.std_testing '
in /Users/dannypeck/Sites/analytemedia/dev-git/static/styles/tests/stx/test-site_map.js

EFFF
NotFoundError: Unable to capture '.std_testing '
in /Users/dannypeck/Sites/analytemedia/dev-git/static/styles/tests/stx/test-site_map.js

EF..
NotFoundError: Unable to capture '.std_testing '
in /Users/dannypeck/Sites/analytemedia/dev-git/static/styles/tests/stx/test-site_map.js

EFFF
NotFoundError: Unable to capture '.std_testing '
in /Users/dannypeck/Sites/analytemedia/dev-git/static/styles/tests/stx/test-site_map.js

EF................................

Warn if var $ = window.jQuery; isn't defined.

I spent a good 15 minutes the other day trying to figure out why a perform() wasn't working. FINALLY, I realized I was missing:

var $ = window.jQuery;

Specter didn't show any errors or warnings when this was the case. That would've helped.

Windows support

Create a specter.bat file to enable running specter from Windows.

Add any necessary support for file paths

Add support for beforeEach and afterEach functions

Specter should provide some means of reducing repetition by allowing setup and teardown functionality to be defined.

Proposed implementation is to allow an _events.js file, where handlers such as onload and onunload can be defined.

"invalid context" message on the console

Each time a test file finishes, the console reports an "invalid context" message when the browser window closes.

I think this has something to do with the timer being used for the queue.

add an --update option to pull this repo and rebuild

Current update process for specter is to navigate to the cloned repository, issue a git pull, then make clean and make. Seems like specter could have an --update option which checks to see if it's running from inside a git repo, and if so, does this for you.

Error when specifying args in .specterrc files

Currently, the specter command parses all config options from rc files as if they're paths, and tries to compare them to cwd. This causes an error when you want to add args to your rc file.

Add a wait() function

There should be a shorthand function wait function which creates a Date object and implicitly builds the waitFor callback for the specified delay.

Can't use relative paths with file protocol.

The open() command currently requires an absolute path when using file protocol. I would like to be able to use a relative path and the absolute path automatically be resolved.

Add support for beforeEach and afterEach functions

Specter should provide some means of reducing repetition by allowing setup and teardown functionality to be defined.

Proposed implementation is to allow an _events.js file, where handlers such as onload and onunload can be defined.

Run comparisons even if tests failed

Currently, if any functional tests fail, the screenshot comparisons do not run.

There should be a CLI options to let the user choose the behavior.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.