garris/ember-backstop

BackstopJS visual regression testing addon for Ember.

ember-backstop's Issues

Old reference images are not cleaned up

If I remove await backstop(assert); from a test or remove/rename a test with await backstop(assert);,
ember test && ember backstop-approve does not remove according (obsolete) image reference from bitmaps_reference/.
This is not a big problem, but since bitmaps_reference/ is committed to git, it would be great to autoremove old image references via ember backstop-approve.

For example:
1.

test('shows specific rental details', async function(assert) {
 await visit('/rentals');
 await click('.grand-old-mansion');
 await backstop(assert);
});

bitmaps_reference/ember-backstoptest_Acceptance__list_rentals__shows_specific_rental_details__assert0_0_document_0_webview.png

- test('shows specific rental details', async function(assert) {
+ test('foobar', async function(assert) {

$ ember test
$ ember backstop-approve

--->

actual:

bitmaps_reference/ember-backstoptest_Acceptance__list_rentals__shows_specific_rental_details__assert0_0_document_0_webview.png
+ bitmaps_reference/ember-backstoptest_Acceptance__list_rentals__foobar__assert0_0_document_0_webview.png

expected:

- bitmaps_reference/ember-backstoptest_Acceptance__list_rentals__shows_specific_rental_details__assert0_0_document_0_webview.png
+ bitmaps_reference/ember-backstoptest_Acceptance__list_rentals__foobar__assert0_0_document_0_webview.png

Checkboxes or Radio boxes "checked" state is not preseved

Hello,

I am seeing that checkboxes/radios in Testem are checked, but they are not checked on final screenshots.
it looks like Backstop Ember plugin copies input.value for inputs, but not input.checked attribute in prepareInputValuesForCopying

Add Support For Connecting to Docker Hosted Backstop Server

Currently, ember-backstop expects to connect to a backstop server connected on the same host as the test host. It would be really useful to have a docker based renderer server for consistent verification across dev machines and CI.

I was able to prototype this by running the backstop docker image and mounting port 3000 to local 3000. After that I needed to update the origin field in the fetch request to the following:

Current:

ember-backstop/addon-test-support/backstop.js

Line 139 in c0cd995

origin: ORIGIN,

Modified:

origin: `http://host.docker.internal:${new URL(ORIGIN).port}`

This is obviously a POC/hack, but it would be great to have first class support for connecting to a docker based server.

Enhancement: Pass options to scrollIntoView()

@garris do you think this is something that would belong in master or do you expect this type of functionality to exist in the consumer's space?

danDanV1@2ca1b82

scenario.scrollToSelector uses scrollIntoView() under the hood, and sometimes you need use the options supported by scrollIntoView()

eg.

element.scrollIntoView({
      block: "end",
      inline: "end"
    });

Implementation

    const options = {
        scenario: {
             scrollToSelector: "#myid",
             scrollIntoViewOptions: {
                  block: "end",
                  inline: "end"
             }
         }
      }
    await backstop(assert, options);

It's a pity that in the scenario option in backstopjs itself is called scrollToSelector because it's not calling window.scrollTo.

Something like this would have clearer semantics:

 {
        scenario: {
             scrollIntoView: {
                          selector: "#myid",
                          options: {
                               block: "end",
                               inline: "end"
                          }
             }
         }
   }

but that'd be a breaking change in the scenario settings...

When using a title for the screen capture, throws `assert.ok is not a function`

I made a reproduction.
danDanV1/ember-backstop-demo@2f4735d

Prior to this commit, was just using await backstop(assert); and was working just fine.

However, if I switch to await backstop("custom title"); it errors when running the test after approving the reference.

Source: | TypeError: assert.ok is not a function     at http://localhost:7357/assets/test-support.js:13825:14     at async Object.<anonymous> (http://localhost:7357/assets/tests.js:13:7)

I also had a wierd bug that I was trying to track down in my app where I first installed backstop. I had started with using a custom title, eg await backstop("custom title");, but then saw the assert.ok is not a function error, and then switched to await backstop(assert); but for some reason ember backstop-approve would not approve the reference and it got stuck in a wierd state. The console kept logging that is was approving the custom title image each time it ran. Even if I did the filter and gave it the exact filename of the autogenerated one. Anyhow, no reproduction for that yet... just wanted to mention.

Using ember-backstop in a CI pipeline

The documentation says:

You will need the backstop-remote service running for visual tests. In a seperate window run...
ember backstop-remote

While it's feasible in local development environment, how does one go about achieving the same results in a CI like travis or Github Actions?

Proposal: A command for regenerating snapshots

When working with a threshold, reference images may drift. It can be nice to update them from time to time without waiting for a failure to occur.

A command like ember backstop:regenerate might be useful to blank all reference images and create new ones.

Run backstop server on a different port

Is there a way to run backstop server on a different port other than the default 3000 ?

I'll be happy to send a PR for the same, if it's not possible already.

TypeError: Cannot read property 'content' of undefined

When running ember-backstop along with ember test, sometimes ember-backstop fails with this error and does not produce a test screenshot. On my local machine, it happens quite erratically. It fails on one run and then works properly on another. On the CI, it fails every time with this same error.

This is the Terminal log of backstop-remote:

This is the screenshot produced for the test:

testHash.testId cannot be overridden in the options, leading to multiple folders in bitmaps_test when running tests in parallel

Hello!

When running ember backstop-remote along with ember exam with the parallel option, I get multiple folders inside ember-backstop/backstop_data/bitmaps_test, each containing the snapshots taken by a particular browser instance, as visible below:

actual:

ember-backstop/backstop_data/bitmaps_test
├── 20230518-122916
│   ├── ...
│   └── report.json
├── 20230518-122917
│   ├── ...
│   └── report.json
└── 20230518-122919
    ├── ...
    └── report.json

expected:

ember-backstop/backstop_data/bitmaps_test
├── 20230518-122916
│   ├── ...
│   └── report.json

The command I'm using is: ember exam --split=12 --partition=1 --parallel=4 --load-balance. With this command I get at most 4 different folders.

The problem seems to be related to this row: https://github.com/garris/ember-backstop/blob/master/addon-test-support/backstop.js#L198

And the way the testRunTime is generated here: https://github.com/garris/ember-backstop/blob/master/addon-test-support/backstop.js#L70

Since that code will run once per each browser instance, each window will carry a different value for that variable. Thus, with 4 browser instances, I'll get 4 different values.

The goal is getting only one folder with one report.json file, as having more of them:

• breaks the html_report - that for instance, will show only some of the failing tests
• breaks the approval process as well, as visible in the code here: https://github.com/garris/BackstopJS/blob/master/core/command/approve.js#L8

The proposed solution would be passing a new testId option here: https://github.com/garris/ember-backstop/blob/master/addon-test-support/backstop.js#L234

When available, it will take precedence over the one generated on the window.

I just created a Pull Request that solves the problem here: #83

What do you think? Thanks 🙏

Dynamically configure scenario settings on per test basis in Ember.

We have some tests where we want to configure misMatchThreshold separately from the rest of the test suite in Ember.

It looks like BackstopJS/Remote/index.js is hard coded to pull the first scenario.
https://github.com/garris/BackstopJS/blob/11634c9cc7db73fa8f6a64035d31ab4c49b600ae/remote/index.js#L41
var s = config.scenarios[0];

What do you think of passing the scenario config settings dynamically with something like this?

BackstopJS/Remote/index.js L41

      var s = _merge({}, config.scenarios[0], req.body.scenarioConfig);

ember-backstop/addon-test-support/backstop.js

  const payload = JSON.stringify({
    content,
    name: name,
    widths: options.widths,
    breakpoints: options.breakpoints,
    enableJavaScript: options.enableJavaScript,
    testHash: testHash,
    origin: ORIGIN,
    scenarioConfig: options.scenario
  });

Implementation

    await backstop(
      assert,
      {
        scenario: {
            misMatchThreshold: 0.5
        }
      },

Question:Error in sample step-by-step

Followed the step-by-step tutorial
https://github.com/garris/ember-backstop-tutorial/
added "await backstop(assert);" to the last four tests and the tests failed as mentioned. Then I approved the screenshots(backstop-approve). BackstopJs Report shows all passing. But the four tests fail when run with following error;

1. Promise rejected during "should link to contact information": Unexpected end of JSON input
   Source: SyntaxError: Unexpected end of JSON input
   at parse ()

Question: do we need console.log?

Does this serve a purpose when running tests? or just for debugging when developing the add-on and can be removed?

ember-backstop/addon-test-support/backstop.js

Line 134 in 3df9829

console.log('Request succeeded with JSON response', data);

Question: Running tests for multiple view ports?

First off, this is really cool and easy to use software, great work!

The only issue I'm having is that I've assigned two viewport sizes in the backstop.js file but the tests undertaken by backstop only test both of the view sizes for one my test functions. The rest of the tests after the first one just look at the first viewport size in the array.

I call "await backstop(assert)" 4 times, but only once in each test function in the test file.

What am I doing wrong?

The sanitised output from the html_report config.js is that I have 5 tests (instead of the expected 8):

report({
  "testSuite": "BackstopJS",
  "tests": [
    {
      "pair": {
        "reference": "../bitmaps_reference/ember-backstoptest_Acceptance__login__user_can_visit_login__assert0_0_document_0_zebraDeviceView.png",
        "test": "../bitmaps_test/20200805-171235/ember-backstoptest_Acceptance__login__user_can_visit_login__assert0_0_document_0_zebraDeviceView.png",
        "fileName": "ember-backstoptest_Acceptance__login__user_can_visit_login__assert0_0_document_0_zebraDeviceView.png",
        "label": "Acceptance | login | user can visit login | assert0",
        "viewportLabel": "zebraDeviceView",
      },
      "status": "pass"
    },
    {
      "pair": {
        "reference": "../bitmaps_reference/ember-backstoptest_Acceptance__login__user_can_visit_login__assert0_0_document_1_webview.png",
        "test": "../bitmaps_test/20200805-171235/ember-backstoptest_Acceptance__login__user_can_visit_login__assert0_0_document_1_webview.png",
        "fileName": "ember-backstoptest_Acceptance__login__user_can_visit_login__assert0_0_document_1_webview.png",
        "label": "Acceptance | login | user can visit login | assert0",
        "viewportLabel": "webview",
      },
      "status": "pass"
    },
    {
      "pair": {
        "reference": "../bitmaps_reference/ember-backstoptest_Acceptance__login__error_message_displayed_when_invalid_credentials__assert0_0_document_0_zebraDeviceView.png",
        "test": "../bitmaps_test/20200805-171235/ember-backstoptest_Acceptance__login__error_message_displayed_when_invalid_credentials__assert0_0_document_0_zebraDeviceView.png",
        "fileName": "ember-backstoptest_Acceptance__login__error_message_displayed_when_invalid_credentials__assert0_0_document_0_zebraDeviceView.png",
        "label": "Acceptance | login | error message displayed when invalid credentials | assert0",
        "viewportLabel": "zebraDeviceView",
      },
      "status": "pass"
    },
    {
      "pair": {
        "reference": "../bitmaps_reference/ember-backstoptest_Acceptance__login__user_should_be_redirected_to_change-password_when_their_password_has_expired__assert0_0_document_0_zebraDeviceView.png",
        "test": "../bitmaps_test/20200805-171235/ember-backstoptest_Acceptance__login__user_should_be_redirected_to_change-password_when_their_password_has_expired__assert0_0_document_0_zebraDeviceView.png",
        "fileName": "ember-backstoptest_Acceptance__login__user_should_be_redirected_to_change-password_when_their_password_has_expired__assert0_0_document_0_zebraDeviceView.png",
        "label": "Acceptance | login | user should be redirected to change-password when their password has expired | assert0",
        "viewportLabel": "zebraDeviceView",
      },
      "status": "pass"
    },
    {
      "pair": {
        "reference": "../bitmaps_reference/ember-backstoptest_Acceptance__login__account_suspended_message_shows_when_user_account_has_been_suspended__assert0_0_document_0_zebraDeviceView.png",
        "test": "../bitmaps_test/20200805-171235/ember-backstoptest_Acceptance__login__account_suspended_message_shows_when_user_account_has_been_suspended__assert0_0_document_0_zebraDeviceView.png",
        "fileName": "ember-backstoptest_Acceptance__login__account_suspended_message_shows_when_user_account_has_been_suspended__assert0_0_document_0_zebraDeviceView.png",
        "label": "Acceptance | login | account suspended message shows when user account has been suspended | assert0",
        "viewportLabel": "zebraDeviceView",
      },
      "status": "pass"
    }
  ],
  "id": "ember-backstop test"
});

Backstop.js file with the two viewports defined:

module.exports = {
  id: `ember-backstop test`,
  viewports: [
    {
      label: 'zebraDeviceView',
      width: 360,
      height: 570,
    },
    {
      label: 'webview',
      width: 1440,
      height: 900,
    }
  ],
  onReadyScript: `puppet/onReady.js`,
  scenarios: [
    {
      label: '{testName}',
      url: '{origin}/backstop/dview/{testId}/{scenarioId}',
      delay: 500,
    },
  ],
  paths: {
    bitmaps_reference: 'backstop_data/bitmaps_reference',
    bitmaps_test: 'backstop_data/bitmaps_test',
    engine_scripts: 'backstop_data/engine_scripts',
    html_report: 'backstop_data/html_report',
    ci_report: 'backstop_data/ci_report',
  },
  report: [],
  engine: 'puppet',
  engineOptions: {
    args: ['--no-sandbox'],
  },
  asyncCaptureLimit: 10,
  asyncCompareLimit: 50,
  debug: false,
  debugWindow: false,
};

ESLint errors in ember-backstop

Using ESLint on a project with ember-backstop throws lint errors for the following lines:

In ember-backstop/addon-test-support/backstop.js

150:11 error - Unexpected Console Statement (no-console)

In ember-backstop/addon-test-support/index.js

1:1 error Parsing error: 'import' and 'export' may appear only with 'sourceType': "module"

1| import { backstop } from './backstop';
2|
3| export { backstop };

Undefined error when running without options

I'm using a simple await backstop(assert); to test, and since one of the last version it's throwing an error

Source: 	
TypeError: Cannot read properties of undefined (reading 'testId')
    at createNameHash (http://localhost:4200/assets/test-support.js:7321:33)
    at _default (http://localhost:4200/assets/test-support.js:7362:18)
    at Object.<anonymous> (http://localhost:4200/assets/tests.js:27:37)

I think it stems from this PR: #83

Because in my case I didn't pass anything options is undefined in the new line testHash.testId = options.testId ?? window._testRunTime;

Passing an empty object as option fixes the error but I'll prepare a PR to prevent this in future.

This Repo contains both yarn.lock and package-lock.json

Seems to me like one of the 2 files - yarn.lock or package-lock.json files was added by mistake. I'll be happy to send a PR as needed.

ENGINE ERROR: Cannot find module 'debug' Require stack:

I'm using the latest version of this addon + ember-cli 5.3.0 + embroider 2.x

Error stack trace:

ENGINE ERROR: Cannot find module 'debug' Require stack: - /Users/villander/Projects/my-project/ember-backstop/backstop_data/engine_scripts/puppet/onReady.js - /Users/villander/Projects/my-project/node_modules/.pnpm/github.com+garris+BackstopJS@d4536fa376a32283b3eb7678cbe281ceed3b2983_ic6gzzytlz7mjae6rwferulft4/node_modules/backstopjs/core/util/runPuppet.js - /Users/villander/Projects/my-project/node_modules/.pnpm/github.com+garris+BackstopJS@d4536fa376a32283b3eb7678cbe281ceed3b2983_ic6gzzytlz7mjae6rwferulft4/node_modules/backstopjs/core/util/createBitmaps.js - /Users/villander/Projects/my-project/node_modules/.pnpm/github.com+garris+BackstopJS@d4536fa376a32283b3eb7678cbe281ceed3b2983_ic6gzzytlz7mjae6rwferulft4/node_modules/backstopjs/core/command/reference.js - /Users/villander/Projects/my-project/node_modules/.pnpm/github.com+garris+BackstopJS@d4536fa376a32283b3eb7678cbe281ceed3b2983_ic6gzzytlz7mjae6rwferulft4/node_modules/backstopjs/core/command/index.js - /Users/villander/Projects/my-project/node_modules/.pnpm/github.com+garris+BackstopJS@d4536fa376a32283b3eb7678cbe281ceed3b2983_ic6gzzytlz7mjae6rwferulft4/node_modules/backstopjs/core/runner.js - /Users/villander/Projects/my-project/node_modules/.pnpm/github.com+garris+BackstopJS@d4536fa376a32283b3eb7678cbe281ceed3b2983_ic6gzzytlz7mjae6rwferulft4/node_modules/backstopjs/remote/index.js - /Users/villander/Projects/my-project/node_modules/.pnpm/github.com+garris+superSimpleExpressServer@b6b9684514caa7b95d0920be2d56946e5e636693_5z4ijgnifzh6id4vwrjhggxrxq/node_modules/super-simple-web-server/index.js

Optional enhancement: Steps and/or custom test name

@garris since you like my last enhancement...

I customized my backstop helper to allow steps and custom naming. Right now ember-backstop is limited to one visual assert per test without this change.

In our app, the components we are testing can change state multiple times in a single test, eg. image placeholder, preloading animation, and loaded state. Also in-viewport detection and we need to scroll, etc. So it's much easier to test multiple steps in a single test.

    await backstop(assert, {
      step: { id: 1, name: `Rendered at scrollPos Top,Left` }
    });

    scrollIntoView(image, { block: "end", inline: "end" });

    await assertImageLoaded(image, assert);
    await backstop(assert, {
      step: { id: 2, name: `Rendered at scrollPos Bottom,Right` }
    });

or even a more fun test

  for (let index = 0; index < images.length; index++) {
    backstopScrollIntoView(images[index], true);

    await assertImageLoaded(images[index], assert);
    await backstop(assert, {
      step: { id: index, name: `Image ${index} rendered` }
    });
  }

you can even just pass a custom string

await backstop(assert, "my cool test name");

In the above, I have the custom naming a second argument and the options as a third. With a little thinking it could be merged into just one options argument.

Right now my personal API for the helper is:

await backstop(assert, "my cool test name", {scenario: { selector: ".jumbo"});

Looks a bit like this:

function createNameHash(assert, nameOptions) {
  let name;
  let testHash = {};

  // Generate base names to extend
  if (
    assert.test &&
    assert.test.module &&
    assert.test.module.name &&
    assert.test.testName
  ) {
    testHash.testId = window._testRunTime;
    testHash.scenarioId = assert.test.testId;
    name = `${assert.test.module.name} | ${assert.test.testName}`;
  } else if (assert.fullTitle) {
    // Automatic name generation for Mocha tests by passing in the `this.test` object.
    name = assert.fullTitle();
    //TODO doesn't testHash need to be set here too?
  }

  // Automatic name generation for QUnit tests by passing in the `assert` object.
  if (!nameOptions) {
    return { name: name, testHash: testHash };
  }

  // Name generation based on step.id and a name for the step.
  if (
    typeof nameOptions === "object" &&
    nameOptions.step &&
    typeof nameOptions.step.id === "number" && //a step of zero is false, so check if int.
    nameOptions.step.name
  ) {
    name = `${name} | Step #${nameOptions.step.id} | ${nameOptions.step.name}`;
    return { name: name, testHash: testHash };
  }

  // Append a unique descriptor
  if (typeof nameOptions === "string") {
    name = `${name} | ${nameOptions}`;
    return { name: name, testHash: testHash };
  }
}

/**
 * I'm in your webapps -- checkin your screenz. -schooch
 */
export default async function(assert, nameOptions, options) {
  const hash = createNameHash(assert, nameOptions);
  return new Promise((res, err) => {
    backstopHelper(hash.name, hash.testHash, options, res, err);
  }).then(backstopResult => {
    assert.ok(backstopResult.ok, backstopResult.message);
  });
}

produces:

ember-backstoptest_Integration__Component__my_component__test_name__Step_1__Image_1_rendered_0_document_0_webview.png
ember-backstoptest_Integration__Component__my_component__test_name__Step_2__Image_2_rendered_0_document_0_webview.png
ember-backstoptest_Integration__Component__my_component__scrolls_in_both_x_and_y_axis__Step_1__Rendered_at_scrollPos_TopLeft_0_document_0_webview.png
ember-backstoptest_Integration__Component__my_component__scrolls_in_both_x_and_y_axis__Step_2__Rendered_at_scrollPos_BottomRight_0_document_0_webview.png

404 error for http://backstopjs.org in the README under `More Info` section

In the More Info section of README, the link with text, http://backstopjs.org is pointing to https://github.com/garris/ember-backstop/blob/master that results in a 404 error.

I would imagine that this link should probably be pointing to http://backstopjs.org or https://github.com/garris/BackstopJS/blob/master/README.md.

I will be happy to submit a PR.