bbc/a11y-tests-web

Runs automated accessibility tests against configurable lists of webpages

a11y-tests-web's Issues

bin executable for a11y-tests-web

It would be good to be able to install a11y-tests-web as a node script from npm.

npm run coverage results in one error

Summary

npm run coverage results in a failing test on branch master

Expected Behaviour

Tests should pass

Current Behaviour

output from npm run coverage is:

  1) bbcA11y build() Paths and baseUrl and visit and options outputs the basic config for the paths, with the defined baseUrl, and with the visit function:
     AssertError: expected writeFileSync to be called with arguments
a11y.js


    page(
      "http://base.url/path/1",
      {
        visit: function (){/* Do something */cov_17hr8di7tb.f[0]++;},
        "some":"option"
      }
    )

    page(
      "http://base.url/path/2",
      {
        visit: function (){/* Do something */cov_17hr8di7tb.f[0]++;},
        "some":"option"
      }
    )


   match()
      at Object.fail (node_modules/sinon/lib/sinon/assert.js:96:21)
      at failAssertion (node_modules/sinon/lib/sinon/assert.js:55:16)
      at Object.assert.(anonymous function) [as calledWith] (node_modules/sinon/lib/sinon/assert.js:80:13)
      at Context.it (test/lib/bbcA11y.js:175:24)

Your Environment

• version "5.1.0"`

Need to use latest version of lighthouse

Summary

Need to update the package.json file to use the latest version of lighthouse

Expected Behaviour

Lighthouse version should be latest

Current Behaviour

a11y-tests-web uses lighthouse v2.7.0 - see here https://github.com/bbc/a11y-tests-web/blob/master/package.json#L27

Possible Solution

We should update to use the latest version of lighthouse

Update dependencies to fix vulnerabilities

Running npm audit today gives found 62 vulnerabilities (24 low, 3 moderate, 33 high, 2 critical) in 1921 scanned packages. We should fix as many of these as possible.

Google Lighthouse job needs a pretty output option

Summary

Google Lighthouse job needs a pretty output option as at the moment, the only option is junit which isn't very human readable.

Bump bbc-a11y to v2.0.0

bbc-a11y v2.0.0 is currently in pre-release.
Once it's released, this project should be updated to work with v2.0.0
Work for this has been started on the update-to-bbc-a11y-v2 branch

No way to distinguish pages where the URL is the same but different commands are run prior to the tests running

If you are testing a URL with and without some pre-test steps, the JUnit results have no way of distinguishing these results.

`npm run start:lighthouse:junit-headless` sometimes hangs

Summary

npm run start:lighthouse:junit-headless sometimes hangs and doesn't exit

Expected Behaviour

npm run start:lighthouse:junit-headless will finish successfully or unsuccessfully and exit every time.

Current Behaviour

npm run start:lighthouse:junit-headless sometimes hangs and doesn't exit

Possible Solution

Might be fixed by #26

Environment

Seen on our Jenkins and on some Macbooks

visit option is not getting passed to bbc-a11y

Summary

The visit option in options does not work on in a11y-tests-web.
Assuming a minimal config where are not using signedInPaths:

module.exports = {
  options: {
    visit: function (frame) {
        /* code */
    },
    hide: [],
    skip: []
  },
  baseUrl: 'http://localhost:3000',
  paths: [ '/' ],
  signedInPaths: [ ]
};

Expected Behaviour

The visit option works in bbc-a11y but does not work in a11-tests-web. It would be great to have this because we would like to inject a script to wait for the page to fully render.

The output sent to bbc-a11y would ideally be like this:

page( "http://localhost:3000/", { "hide":[],"skip":[], "visit": function() { ... } })

Current Behaviour

The visit option is currently not getting passed to bbc-a11y. This is because a JSON.stringify is stripping out the function. The built-in behaviour of JSON.stringify is to remove functions:

If undefined, a Function, or a Symbol is encountered during conversion it is either omitted (when it is found in an object) or censored to null (when it is found in an array).

The output sent to bbc-a11y currently looks like:

page( "http://localhost:3000/", { "hide":[],"skip":[] })

Possible Solution

There's probably a nicer way, but this, for example, seems to fix the problem:

   return `
     page(
       "${baseUrl}${path}",
       {
         ${visitOptions}
+        visit: ${options.visit.toString()},
         ${JSON.stringify(options).slice(1, -1)}
       }
     )
   `;

We make a change to what pathToOutput returns – we explicitly take the function body of options.visit and convert it to a string, outside of the JSON.stringify.

Your Environment

• Version used: 3.1.0
• Operating System and version: macOS Sierra Version 10.12.6 (Google Chrome Version 66.0)
• Link to your project: (testing a11y-tests-web locally for now)