segmentio/niffy

Perceptual diffing suite built on Nightmare

Perceptual diffing suite
built on Nightmare by Segment

Getting Started

You can look at test/index.js as an example for how to use Niffy. To run the example test just do make test after cloning this repo.

Reference

Niffy is built on Nightmare and used in combination with Mocha. You'll also need to read and use both of those library's APIs to use niffy effectively.

Niffy(basehost, testhost[, options])

To create a new Niffy differ:

let niffy = new Niffy(basehost, testhost, nightmareOptions);

• basehost is the url that is assumed "good"
• testhost is the url that you are comparing to the base
• nightmareOptions can be seen here in the Nightmare docs
  • .threshold is the maximum percentage difference for a passing test (default: 0.2%)

.test(url[, fn])

This method instructs niffy to go to a url (and optionally take additional actions like clicking, typing or checkboxing via the fn argument), and test basehost vs. testhost screenshots for pixel differences, and output the diff-highlight image. Typically you'll use .test(url, fn) in the body of a mocha test, like this:

it('/news', function* () {
  yield niffy.test('/news');
});

.goto(url[, fn])

This method instructs niffy to go to a url and optionally take additional actions like clicking, typing or checkboxing via the fn argument. Typically you'll use .goto(url, fn) in the before method of a mocha test suite, like this:

before(function* () {
  yield niffy.goto('/logout', function* (nightmare) {
    yield nightmare
      .type('input[name="email"]', '[email protected]')
      .type('input[name="password"]', 'fakepassword')
      .click('button[type="submit"]');
  });
});

.end()

This method closes the underlying Nightmare instance (e.g. freeing up memory). Typically you'll use .end() in the after method of a mocha test suite, like this:

after(function* () {
  yield niffy.end();
});

License (MIT)

WWWWWW||WWWWWW
 W W W||W W W
      ||
    ( OO )__________
     /  |           \
    /o o|    MIT     \
    \___/||_||__||_|| *
         || ||  || ||
        _||_|| _||_||
       (__|__|(__|__|

Copyright (c) 2017 Segment.io, Inc. [email protected]

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the 'Software'), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

niffy's Issues

Can't get test to fail

I'm just getting started so I'm probably doing something wrong but this test passes. I'm comparing google.com and yahoo.com, so I'd expect it to fail.

const Niffy = require('niffy');

describe('testing niffy', function () {
  let niffer = null;
  
  before(function () {
    niffer = new Niffy(
      'https://www.google.com',
      'https://www.yahoo.com',
      { show: true }
    );
  });
  
  it('/', function* () {
    yield niffer.test('/');
  });
  
  after(function* () {
    yield niffer.end();
  });
});

Why does this require mocha?

I just stumbled upon this and it looks great, but I was wondering why the readme says it must be used with mocha? From what I can tell, it should work with any generator or await/async aware test framework?

Use as an npm module?

I'm a bit new to regression testing so forgive me if this question is an obvious one, however, I spent a good amount of time attempting to pull niffy into a project as an npm module and couldn't get the test spec to run properly.

Is regression testing supposed to exist as a stand alone project, built specifically for its use case? I was expecting to add it to the test specs of our current app build so that during significant updates we can run regression testing to make sure things did not change too much (or only the right things changed).

I have npm installed the proper modules - mocha, chai, niffy (and nightmare for good measure), used the test section of Makefile, and copied code I edited (and proved worked) from /test/index.js. The test spec will run but the tests just pass automatically and the electron window will not pop up. Any ideas here?

[Feature] Expose Nightmare.wait()

mousemke#4

What needs to get done?

• the wait method from nightmare should be exposed.

Why does it need to get done?

before(function*() {
  yield niffy.goto('/', function*(nightmare) {
    yield nightmare.wait(3000);
  });
});

it seems odd that you would have to go through another call just to use wait through a callback. this exposes it to the niffy object

before(function*() {
  yield niffy.wait(3000);
});

How do I do it?

simply exposing it should do the trick

  /**
   * exposes the nightmare wait fubction
   *
   * @param {String} target selector of ms
   */
  * wait( target ){
      this.nightmare.wait( target );
    }
  }

Handling click events and screenshots with Niffy

Trying to work out how to do this given the limited documentation currently.
Im aiming to run the Niffy screenshot on page load, then do some page interaction (ex: open a menu), and take a new screenshot at that moment so in the end i will have 2 screenshots of the page to compare against my base screenshots of those same instances.

Im assuming some nightmare API interaction will be needed but just hoping I could get some wisdom on how to do it.

Thank you.

[Feature] option for 'ignore' containers

What needs to get done?
Niffy should take an list as an option for "exclude" containers. As example:

const ignore = ['video', '.animated-background-image', '#d3-animated-chart']

Current project status

Hi, what is the current status of this project? Is it still maintained?

[Feature] screenshot diff reporter

Post failed test, an html page could be generated and opened to show the diff failure.

diff screenshot

How do I compare two screenshots, check documents, and find no API

Setting threshold value for diff

I am trying to adjust the threshold value for Niffy but do not see the threshold option described in Nightmare as the Niffy info suggests.

I tried:
niffy = new Niffy(
baseUrl,
testUrl,
{ show: true,
threshold: 0.02}
)
but did not seem to change anything.
Thanks

[Feature] Dynamic Screenshots Path

It seems fairly straightforward. There could be a screenshotPath option passed into imgfilepath. Is there a specific reason why this is not implemented? (dynamic or relative paths wouldnt work for some reason?) I would be willing to implement this into a PR as long as it's a desired thing

Jest support

Hi, Wondering if there is any possibility to run niffy with Jest?
I was hoping to be able to do something like:

import Niffy from 'niffy';

describe('Header', () => {
  const basehost = 'https://www.google.com/';
  const testhost = 'https://www.google.co.uk/';

  let niffy;

  beforeAll(() => {
    niffy = new Niffy(basehost, testhost, { show: true });
  });

  afterAll(() => {
    niffy.end();
  });

  it('running visual regression tests', () => {
    niffy.goto('/?x=true');
    return niffy.test('/?x=true');
  });
});

Is there any plan for Jest support?

[Feature] Niffy should accept and test multiple resolutions

issue: mousemke#2
pr (merged): mousemke#3
pr (open): #14

What needs to get done?

• niffy should take an object as an option with different resolutions to test all. example:

const niffyTargets = {
  xl: [1205,600],
  l: [965, 600],
  m: [605, 600],
  s: [325, 600]
}

const niffy = new Niffy( url1, url2, { 
    targets: niffyTargets,
    pngPath: path.resolve(process.cwd(), `./niffy`)
 });

in tihs case, it whould output diff images to the following directories:

• ./niffy/xl
• ./niffy/l
• ./niffy/m
• ./niffy/s

Why does it need to get done?

it saves everyone making their own implementation of the loop and streamlines the process

How do I do it?

loop nightmare over the keys

How full page screen comparison is done

Am new to NightmareJs and Niffy. Impressed with easiness to use Niffy.

When i go with the default options, niffy compares the screen snap of the visible viewport. My requirement is to compare the screen snap of the whole page. Is it possible to do it with Niffy?
Please help.

Fullpage diff

What needs to get done?

I'm trying to solve issue #18 (taking fullpage screenshots). I've managed to capture fullpage screenshots. However, the diff doesn't run up to 2144px of height.

Anyone has an idea why the diff does not process the whole page ?

Why does it need to get done?

• That would solve #18
• I think that is a very important feature.

How do I do it?

Setup

• Clone the repo by running git clone https://github.com/alanpilloud/niffy
• Run npm install
• Run tests with make tests (tests will "fail" for now, as expected)
• Go check the diff for the page /services

Unexpected diff image output

What needs to get done?

While testing landing page with quite large height ( > 3000px ) running multiple tests with different scrollTo values. Diff image produced by the first test is correct, while second one is not. Tried to add extra .wait() but it didn't help.
( see produced output images below )

Why does it need to get done?

Expect to have visual diff working correctly.

How do I do it?

  describe('/Landing', function () {
    it('Homepage1', function* () {
      yield niffy.test('/?test=1', function* (nightmare) {
        yield nightmare.scrollTo(800, 0);
      });
    })
    it('Homepage2', function* () {
      yield niffy.test('/?test=2', function* (nightmare) {
        yield nightmare.scrollTo(2000, 0);
      });
    })
  });

Example test1 ( correct )

base

diff

test

---

Example test2 ( wrong )

base

diff ( looks chopped )

test