Giter Site home page Giter Site logo

cypress-axe's Introduction

cypress-axe

npm Node.js CI status

Test accessibility with axe-core in Cypress.

Installation

  1. Install cypress-axe from npm:
  • For Cypress v10 and above install latest cypress-axe
npm install --save-dev cypress-axe
  • For Cypress v9 install 0.x.x
npm install --save-dev [email protected]
  1. Install peer dependencies:
  • For Cypress v10 and above
npm install --save-dev cypress axe-core
  • For Cypress v9 and below install the specific cypress version you are using For example if you are using cypress v9.6.0
npm install --save-dev [email protected] axe-core
  1. Include the commands.
  • For Cypress v10 and above update cypress/support/e2e.js file to include the cypress-axe commands by adding:
  • For Cypress v9 and below update cypress/support/index.js file to include the cypress-axe commands by adding:
import 'cypress-axe'
  1. Add a task to log the messages to the terminal when Cypress executes the spec files. Example - configuring log task.

TypeScript

If you’re using TypeScript, add cypress-axe types to your Cypress’ tsconfig.json file:

{
  "compilerOptions": {
    "baseUrl": "./",
    "target": "es5",
    "lib": ["esnext", "dom"],
    "types": ["cypress", "cypress-axe"]
  },
  "include": ["."]
}

Commands

cy.injectAxe

This will inject the axe-core runtime into the page under test. You must run this after a call to cy.visit() and before you run the checkA11y command.

You run this command with cy.injectAxe() either in your test, or in a beforeEach, as long as the visit comes first.

beforeEach(() => {
  cy.visit('http://localhost:9000')
  cy.injectAxe()
})

The injectAxe function receives an optional argument injectOptions of type InjectOptions.

This injectOptions object can have a property axeCorePath of type string, which allows the user to specify the file from which axe-core will be injected.

If axeCorePath is not provided, the function will try to resolve the path to axe-core/axe.min.js using the require.resolve function, if it is available.

If require.resolve is not available, the default path node_modules/axe-core/axe.min.js will be used.

beforeEach(() => {
  cy.visit('http://localhost:9000')
  cy.injectAxe({ axeCorePath: '<path-to-axe-core>' })
})

cy.configureAxe

Purpose

To configure the format of the data used by aXe. This can be used to add new rules, which must be registered with the library to execute.

Description

User specifies the format of the JSON structure passed to the callback of axe.run

Link - aXe Docs: axe.configure

it('Has no detectable a11y violations on load (custom configuration)', () => {
  // Configure aXe and test the page at initial load
  cy.configureAxe({
    branding: {
      brand: String,
      application: String
    },
    reporter: 'option',
    checks: [Object],
    rules: [Object],
    locale: Object
  })
  cy.checkA11y()
})

cy.checkA11y

This will run axe against the document at the point in which it is called. This means you can call this after interacting with your page and uncover accessibility issues introduced as a result of rendering in response to user actions.

Parameters on cy.checkA11y (axe.run)

context (optional)

Defines the scope of the analysis - the part of the DOM that you would like to analyze. This will typically be the document or a specific selector such as class name, ID, selector, etc.

options (optional)

Set of options passed into rules or checks, temporarily modifying them. This contrasts with axe.configure, which is more permanent.

The keys consist of those accepted by axe.run's options argument, a custom includedImpacts key, and retries/interval keys for retrying the check.

The includedImpacts key is an array of strings that map to impact levels in violations. Specifying this array will only include violations where the impact matches one of the included values. Possible impact values are "minor", "moderate", "serious", or "critical".

The retries key is an integer that specifies how many times to retry the check if there are initial findings. The interval key is an integer that specifies the number of milliseconds to wait between retries, and defaults to 1000 (one second). If retries is not specified, the check will only be run once. Use this option to account for dynamic content that may not be fully loaded when the check is first run.

Filtering based on impact in combination with the skipFailures argument allows you to introduce cypress-axe into tests for a legacy application without failing in CI before you have an opportunity to address accessibility issues. Ideally, you would steadily move towards stricter testing as you address issues.

violationCallback (optional)

Allows you to define a callback that receives the violations for custom side-effects, such as adding custom output to the terminal.

NOTE: This respects the includedImpacts filter and will only execute with violations that are included.

skipFailures (optional, defaults to false)

Disables assertions based on violations and only logs violations to the console output. This enabled you to see violations while allowing your tests to pass. This should be used as a temporary measure while you address accessibility violations.

Reference : #17

Examples

Basic usage

// Basic usage
it('Has no detectable a11y violations on load', () => {
  // Test the page at initial load
  cy.checkA11y()
})

// Applying a context and run parameters
it('Has no detectable a11y violations on load (with custom parameters)', () => {
  // Test the page at initial load (with context and options)
  cy.checkA11y('.example-class', {
    runOnly: {
      type: 'tag',
      values: ['wcag2a']
    }
  })
})

it('Has no detectable a11y violations on load (filtering to only include critical impact violations)', () => {
  // Test on initial load, only report and assert for critical impact items
  cy.checkA11y(null, {
    includedImpacts: ['critical']
  })
})

// Basic usage after interacting with the page
it('Has no a11y violations after button click', () => {
  // Interact with the page, then check for a11y issues
  cy.get('button').click()
  cy.checkA11y()
})

it('Only logs a11y violations while allowing the test to pass', () => {
  // Do not fail the test when there are accessibility failures
  cy.checkA11y(null, null, null, true)
})

it('Has no a11y violations after asynchronous load', () => {
  // Retry the check if there are initial failures
  cy.checkA11y(null, {
    retries: 3,
    interval: 100
  })
})

Using the violationCallback argument

The violation callback parameter accepts a function and allows you to add custom behavior when violations are found.

This example adds custom logging to the terminal running Cypress, using cy.task and the violationCallback argument for cy.checkA11y

In Cypress plugins file

This registers a log task as seen in the Cypress docs for cy.task as well as a table task for sending tabular data to the terminal.

module.exports = (on, config) => {
  on('task', {
    log(message) {
      console.log(message)

      return null
    },
    table(message) {
      console.table(message)

      return null
    }
  })
}

In your spec file

Then we create a function that uses our tasks and pass it as the validationCallback argument to cy.checkA11y

// Define at the top of the spec file or just import it
function terminalLog(violations) {
  cy.task(
    'log',
    `${violations.length} accessibility violation${
      violations.length === 1 ? '' : 's'
    } ${violations.length === 1 ? 'was' : 'were'} detected`
  )
  // pluck specific keys to keep the table readable
  const violationData = violations.map(
    ({ id, impact, description, nodes }) => ({
      id,
      impact,
      description,
      nodes: nodes.length
    })
  )

  cy.task('table', violationData)
}

// Then in your test...
it('Logs violations to the terminal', () => {
  cy.checkA11y(null, null, terminalLog)
})

This custom logging behavior results in terminal output like this:

Custom terminal logging with cy.task and validationCallback

Standard Output

When accessibility violations are detected, your test will fail and an entry titled "A11Y ERROR!" will be added to the command log for each type of violation found (they will be above the failed assertion). Clicking on those will reveal more specifics about the error in the DevTools console.

Cypress and DevTools output for passing and failing axe-core audits

Authors

The project was created by Andy Van Slaars, and maintained by Artem Sapegin.

Contributors

Thanks goes to these wonderful people (emoji key):


Samuel Custer

πŸ’» πŸ“–

Michael Toth

πŸ’»

Nicholas Boll

πŸ’»

Mike Davis

πŸ’»

chit786

πŸ’» πŸ“–

Adrien courdavault

πŸ’»

Brett Zamir

πŸ’»

This project follows the all-contributors specification. Contributions of any kind welcome!

License

MIT License, see the included License.md file.

cypress-axe's People

Contributors

abhidp avatar acourdavaultlinode avatar allcontributors[bot] avatar anho avatar avanslaars avatar bmordue avatar brettz9 avatar chit786 avatar christine-pinto avatar cshouts-tasc avatar dependabot[bot] avatar dkryaklin avatar ijrex avatar jasonparallel avatar jkevingutierrez avatar johnhwhite avatar mehmetyararvx avatar michaeljacobdavis avatar miketoth avatar nicholasboll avatar samcus avatar sapegin avatar srikanthkyatham avatar sshaar08 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

cypress-axe's Issues

Disabaling axe rules

Hi,

Is there a way to disable certain axe rules? We're looking to run it through CI and gradually update accessibility errors as we go then enable the rules once they're updated.

Is there a way to do this?

Thanks

Identify a11y failures

Is there another way to see the a11y failures than through the console? I'm finding it somewhat time consuming to identify the violating DOM node, it'd be neat if you could use regular Cypress functionality like highlighting the DOM node through the UI, etc.

With Cypress-Axe 0.8.1 commands not showing

With 0.8.1 even after you add the import command to support/index.js the axe commands are not detected.
cy.injectAxe()
cy.injectAxe()
cy.checkA11y()

Those commands are not detected by cypress..

Specify selector on cy.checkA11y()

I'm working on a component library so my focus in a11y testing of the components rather than the test harness they're loaded in. In protractor we can do something like:

browser.runAxeTest('Breadcrumbs', 'lib-breadcrumbs');

Looking at some of the discussion here, it seems like axe-core allows specificity.

It looks like it might be a simple passing of a selector as a parameter to the cy.checkA11y(); to propagate to axe API.

Does this seem like it would be a good approach? Or would using axe directly be preferable?

Add a callback for getting the full axe-core results object

Hey!
In some cases, the full axe-core results object is needed (for logging / tracking or other uses)
It would be helpful to have another callback that returns the full results object (similar to how violationsCallback returns the violations object).

I submitted a PR for this - please consider it :)
#68

Where can I find information on adding Custom Rules for aXe

axe-core has rules provided at https://github.com/dequelabs/axe-core/tree/develop/lib/rules
but could not find an example custom json to append custom rules. What I want to do is to manually test a site using Trusted Tester Test Process, and add the test to the custom list of tests with expected / actual type assertions, so as the custom tests grow in size the amount of time needed to test against trusted tester tests, manually is reduced.

Disable a rule

Hi, is it possible to disable a rule via ID when running cy.checkA11y() on a page?

We're already aware of the color-contrast issue we have on our pages and until our site rebrand this will remain an a11y issue.

I've read the provided documentation and tried a few different variations (passing just an array through, removing [] brackets etc) of the below but the rule still appears to be checked:

        cy.injectAxe()
        cy.configureAxe({
            rules: [{ 'aria-hidden-focus': { enabled: false } }]
        })
        cy.checkA11y()

image

How run just a single rule?

Still after reading axe-core docs cannot for the life of me figure out how to run a single rule, for example how to run just "color-contrast" rule against text on the page...

Color Contrast Not Failing - Works with react-axe

Hey there. I am experience what I think is a strange issue. I am testing out the accessibility on my website built with Gatsby (React.js). I have Cypress set up as per their docs with this library and axe-core. When I run the the following I get no errors on a page that has a footer which I know is rendering links that will fail the color contrast test. I know this because I have react-axe installed as well and it is reporting the color contrast problem in the console.
However because react-axe does not lead to a nice automation testing work flow I would like to use cypress to find these color contrast issues.

What could be causing cypress-axe / axe-core not picking it up with testing but react-axe picks it up as expected?

Sample test

describe('Accessibility tests', () => {
    beforeEach(() => {
      cy.visit('http://localhost:8000')
      cy.injectAxe()
    })
    it('Has no detectable accessibility violations on load', () => {
      cy.wait(1000)
      cy.checkA11y()
    })
  })

"AssertionError: 1 accessibility violation was detected: expected 1 to equal 0"?

I'm using cypress-axe to do some accessibility testing, but the errors I'm getting are not clear enough for me to act on... Is there a way to get more detailed information here?

  6 passing (23s)
  22 pending
  2 failing

  1) Accessibility Tests Global Component Assessments Desktop Assessments Petition Petition form should be accessible.:
     AssertionError: 1 accessibility violation was detected: expected 1 to equal 0
      at Function.fn.(anonymous function) [as equal] (http://localhost:8000/__cypress/runner/cypress_runner.js:86181:34)
      at getRet (http://localhost:8000/__cypress/runner/cypress_runner.js:89187:16)
      at tryCatcher (http://localhost:8000/__cypress/runner/cypress_runner.js:139407:23)
      at Function.Promise.attempt.Promise.try (http://localhost:8000/__cypress/runner/cypress_runner.js:136682:29)
      at Context.thenFn (http://localhost:8000/__cypress/runner/cypress_runner.js:89201:23)
      at Context.then (http://localhost:8000/__cypress/runner/cypress_runner.js:89525:21)
      at Context.<anonymous> (http://localhost:8000/__cypress/runner/cypress_runner.js:100860:21)
      at http://localhost:8000/__cypress/runner/cypress_runner.js:100381:33
      at tryCatcher (http://localhost:8000/__cypress/runner/cypress_runner.js:139407:23)
      at Promise._settlePromiseFromHandler (http://localhost:8000/__cypress/runner/cypress_runner.js:137343:31)
      at Promise._settlePromise (http://localhost:8000/__cypress/runner/cypress_runner.js:137400:18)
      at Promise._settlePromiseCtx (http://localhost:8000/__cypress/runner/cypress_runner.js:137437:10)
      at Async../node_modules/bluebird/js/release/async.js.Async._drainQueue (http://localhost:8000/__cypress/runner/cypress_runner.js:134137:12)
      at Async../node_modules/bluebird/js/release/async.js.Async._drainQueues (http://localhost:8000/__cypress/runner/cypress_runner.js:134142:10)
      at Async.drainQueues (http://localhost:8000/__cypress/runner/cypress_runner.js:134016:14)

  2) Accessibility Tests Global Component Assessments Desktop Assessments Petition Petition form should be accessible.:
     AssertionError: 1 accessibility violation was detected: expected 1 to equal 0
      at Function.fn.(anonymous function) [as equal] (http://localhost:8000/__cypress/runner/cypress_runner.js:86181:34)
      at getRet (http://localhost:8000/__cypress/runner/cypress_runner.js:89187:16)
      at tryCatcher (http://localhost:8000/__cypress/runner/cypress_runner.js:139407:23)
      at Function.Promise.attempt.Promise.try (http://localhost:8000/__cypress/runner/cypress_runner.js:136682:29)
      at Context.thenFn (http://localhost:8000/__cypress/runner/cypress_runner.js:89201:23)
      at Context.then (http://localhost:8000/__cypress/runner/cypress_runner.js:89525:21)
      at Context.<anonymous> (http://localhost:8000/__cypress/runner/cypress_runner.js:100860:21)
      at http://localhost:8000/__cypress/runner/cypress_runner.js:100381:33
      at tryCatcher (http://localhost:8000/__cypress/runner/cypress_runner.js:139407:23)
      at Promise._settlePromiseFromHandler (http://localhost:8000/__cypress/runner/cypress_runner.js:137343:31)
      at Promise._settlePromise (http://localhost:8000/__cypress/runner/cypress_runner.js:137400:18)
      at Promise._settlePromiseCtx (http://localhost:8000/__cypress/runner/cypress_runner.js:137437:10)
      at Async../node_modules/bluebird/js/release/async.js.Async._drainQueue (http://localhost:8000/__cypress/runner/cypress_runner.js:134137:12)
      at Async../node_modules/bluebird/js/release/async.js.Async._drainQueues (http://localhost:8000/__cypress/runner/cypress_runner.js:134142:10)
      at Async.drainQueues (http://localhost:8000/__cypress/runner/cypress_runner.js:134016:14)

The test code is just a checka11y() call:

[...]

      context("Petition", () => {
        before(() => {
          cy.viewport('macbook-13');
          cy.visit(`/campaigns/single-page/`);
          cy.wait(500);
          cy.injectAxe().configureAxe({
            reporter: "v2"
          });
          cy.get(".sign-petition");
        });

        it("Petition content should be accessible.", () => {
          cy.checkA11y(".sign-petition .petition-content"); // passes
        });

        it("Petition form should be accessible.", () => {
          cy.checkA11y(".sign-petition form"); // generates those 2 errorrs
        });

[...]

And I updated cypress' /plugins/index.js to include the task logging:

module.exports = (on, config) => {
  on("task", {
    log(message) {
      console.log(message);
      return null;
    },
    table(message) {
      console.table(message);
      return null;
    }
  });
};

But I don't see anything in the console that lets me figure out what is actually failing and how to fix it =S

InjectAxe fails on cy.readFile error when using Cypress Webpack Preprocessor

I'm seeing the following error when running cy.injectAxe in my Cypress tests:

Timed out retrying after 4000ms: cy.readFile("0764") failed because the file does not exist at the following path:

/home/megan/voicethread/code/frontend/wizard/0764

Because this error occurred during a before each hook we are skipping the remaining tests in the current suite

node_modules/cypress-axe/dist/index.js:16:8
  14 | exports.configureAxe = exports.injectAxe = void 0;
  15 | exports.injectAxe = function () {
> 16 |     cy.readFile(require.resolve('axe-core/axe.min.js')).then(function (source) {
     |        ^
  17 |         return cy.window({ log: false }).then(function (window) {
  18 |             window.eval(source);
  19 |         }); 

require.resolve('axe-core/axe.min.js') seems to evaluate to "0764" instead of the intended JS.

This issue seems to be connected to Cypress's Webpack Preprocessor, which I'm using to handle importing files with absolute path aliases.

const webpack = require('@cypress/webpack-preprocessor')

module.exports = (on, config) => {
  on('file:preprocessor', webpack({
    webpackOptions: require('@vue/cli-service/webpack.config'),
    watchOptions: {},

    // Add the ability to use aliases like @shared
    resolve: {
      alias: require('../../../../aliases.config').webpack
    }
  }))

  return Object.assign({}, config, {
    fixturesFolder: 'tests/e2e/fixtures',
    integrationFolder: 'tests/e2e/specs',
    screenshotsFolder: 'tests/e2e/screenshots',
    videosFolder: 'tests/e2e/videos',
    supportFile: 'tests/e2e/support/index.js'
  })
}

When I remove the on('file:preprocessor') listener from my plugins.js, file, the cy.injectAxe command succeeds.

Node version: 12.13.1
Cypress version: 6.3.0
@cypress/webpack-preprocessor version: 5.5.0

checkA11y failing when called after setting cy.clock

Hi,

I noticed that checkA11y failed when called after using the cy.clock(...) function to set a date

I get the following error when trying to call checkA11y :

cy.then() timed out after waiting 4000ms.

Your callback function returned a promise that never resolved.

The callback function was:

function(a){i(e)&&(e=void 0),i(t)&&(t=void 0),i(n)&&(n=void 0);var o=t||{},s=o.includedImpacts,u=r(o,["includedImpacts"]);return a.axe.run(e||a.document,u).then((function(e){var t=e.violations;return s&&Array.isArray(s)&&Boolean(s.length)?t.filter((function(e){return s.includes(e.impact)})):t}))}

Running cypress-axe 0.8.1 and cypress 4.12.1

ReferenceError: exports is not defined when calling injectAxe() through typescript

I have this in my support/index.ts

my tsconfig.json looks like this:

{
  "compilerOptions": {
    "lib": [
      "dom",
      "es2015",
      "es5",
      "es6",
      "es2015"
    ],
    "target": "es5",
    "module": "commonjs",
    "noImplicitAny": false,
    "strictNullChecks": true,
    "noImplicitThis": true,
    "alwaysStrict": true,
  },
  "files": [
    "./support/index.d.ts"
  ]
}

I have this test:

describe("TypeScript", () => {
  it("checks shape of an object", () => {
    cy.visit('http://localhost:6001')
    cy.injectAxe();
  });
})

It fails with

ReferenceError: exports is not defined
at axeFunction (eval at cy.window.then.window (http://localhost:6001/__cypress/tests?p=cypress/support/index.ts-538:111:12), :16:14)
at eval (eval at cy.window.then.window (http://localhost:6001/__cypress/tests?p=cypress/support/index.ts-538:111:12), :15287:3)
at eval ()

Why would this be?

Not detecting missing aria-label on input field

My test file:

context('app', () => {
  beforeEach(() => {
    cy.server();
    cy.route('GET', '/api/exercises/core/exercises/*', 'fixture:exercises');
    cy.visit('/exercises/3bc94905-4b8a-4d67-815e-0652afaf22c2');
  });

  it('has no a11y violations', () => {
    cy.injectAxe();
    cy.checkA11y();
  });
});

the output (note the input field shown in element explorer)
Screen Shot 2019-08-27 at 12 11 08 PM

A11Y ERROR! landmark-one-main on 1 Node
A11Y ERROR! page-has-heading-one on 1 Node

shows nothing about missing label on input, am i doing something wrong? ive tried adding arguments to the checkA11y call to make it more specific but then i get no errors at all

Support Cypress 4.0

With Cypress 4.0 released, could this be added now to peerDeps? I don't know if other changes would be needed. Thanks for allowing us to test accessibility with cypress!

Can we bump the axe-core to 4.1.1?

Would it be possible to update the axe-core package to the latest version in order to include the larger rule set that came in with 4.1? Thank you.

Using a different assertion library?

Firstly, cypress-axe is brilliant! Previously I was using axe-core with webdriverio, and it was painfully slow! However, one frustration with cypress-axe is the use of node assert. Elsewhere in our tests I'm using the chai library bundled into Cypress, which I have also extended to provide soft assertions (ie, continue on assertion fail) based on code found here:https://stackoverflow.com/questions/55868107/does-cypress-support-soft-assertion

I have modified cypress-axe locally to use chai.softExpect, and it behaves as I'd like. this allows me to run multiple a11y checks against different pages / page sections in the same test.

Might it be possible to have some sort of configuration option that allowed the user to specify which assertion library / method to use? Or something similar?

Using custom matches for axe ruls

We are trying to use a custom matches for some of the axe rules to remove a few false positives that are currently being thrown. The documentation for axe-core states that a string with the relative path to the JS file containing the custom matches function must be used (https://github.com/dequelabs/axe-core/blob/develop/doc/developer-guide.md#matches-function).
The problem we are having is what is the path relative too as nothing we have tried has worked.

Also is there any way of importing the original matches function used by axe core ie (https://github.com/dequelabs/axe-core/blob/develop/lib/rules/color-contrast-matches.js)

How do you exclude an element from checkA11y?

I am using these versions:

    "axe-core": "4.0.1",
    "cypress": "4.11.0",
    "cypress-axe": "0.8.1",

Using this in my cypress test:

    cy.injectAxe();
    cy.checkA11y({
      exclude: [
        [['#onetrust-consent-sdk']]
      ]
    });

But getting these error:

SyntaxError
Failed to execute 'querySelectorAll' on 'Document': '[object Object]' is not a valid selector.

It seems like the first param for checkA11y should not be an object, yet looking at the documentation and examples here it should work:

Any ideas? My goal is to ignore a11y checks for the div specified by an ID.

Weird error after initial setup

cy.readFile() must be passed a non-empty string as its 1st argument. You passed: 146.Learn more

This happens after finishing setup and trying to run a sample test.

This is in my command that I use to login:

    .then((url) => {
      cy.intercept('api/users/user/profile/').as('login');
      cy.visit(url);
      cy.wait('@login');
      cy.injectAxe();
    }),

Error happens on the cy.injectAxe() line.

Thanks!

Unable to run successfully cypress-axe

Having trouble running tests and get the following error:

cypress_runner.js:157775 TypeError: Cannot read property 'run' of undefined
    at Context.<anonymous> (https://qa1.domain.com/__cypress/tests?p=cypress\support\index.js-350:18962:22)
From previous event:
    at Context.thenFn (https://qa1.domain.com/__cypress/runner/cypress_runner.js:71909:23)
    at Context.then (https://qa1.domain.com/__cypress/runner/cypress_runner.js:72229:21)
    at Context.<anonymous> (https://qa1.domain.com/__cypress/runner/cypress_runner.js:80256:21)
    at https://qa1.domain.com/__cypress/runner/cypress_runner.js:79783:33
From previous event:
    at runCommand (https://qa1.domain.com/__cypress/runner/cypress_runner.js:79765:14)
    at next (https://qa1.domain.com/__cypress/runner/cypress_runner.js:79900:14)
From previous event:
    at next (https://qa1.domain.com/__cypress/runner/cypress_runner.js:79900:34)
From previous event:
    at https://qa1.domain.com/__cypress/runner/cypress_runner.js:79937:37
From previous event:
    at run (https://qa1.domain.com/__cypress/runner/cypress_runner.js:79929:15)
    at Object.cy.<computed> [as percySnapshot] (https://qa1.domain.com/__cypress/runner/cypress_runner.js:80290:11)
    at Context.runnable.fn (https://qa1.domain.com/__cypress/runner/cypress_runner.js:80477:20)
    at callFn (https://qa1.domain.com/__cypress/runner/cypress_runner.js:30713:21)
    at Test.../driver/node_modules/mocha/lib/runnable.js.Runnable.run (https://qa1.domain.com/__cypress/runner/cypress_runner.js:30706:7)
    at https://qa1.domain.com/__cypress/runner/cypress_runner.js:84151:28
From previous event:
    at Object.onRunnableRun (https://qa1.domain.com/__cypress/runner/cypress_runner.js:84140:17)
    at $Cypress.action (https://qa1.domain.com/__cypress/runner/cypress_runner.js:78413:30)
    at Test.Runnable.run (https://qa1.domain.com/__cypress/runner/cypress_runner.js:82943:20)
    at Runner.../driver/node_modules/mocha/lib/runner.js.Runner.runTest (https://qa1.domain.com/__cypress/runner/cypress_runner.js:31180:10)
    at https://qa1.domain.com/__cypress/runner/cypress_runner.js:31286:12
    at next (https://qa1.domain.com/__cypress/runner/cypress_runner.js:31100:14)
    at https://qa1.domain.com/__cypress/runner/cypress_runner.js:31110:7
    at next (https://qa1.domain.com/__cypress/runner/cypress_runner.js:31042:14)
    at https://qa1.domain.com/__cypress/runner/cypress_runner.js:31073:7
    at next (https://qa1.domain.com/__cypress/runner/cypress_runner.js:84079:16)
    at https://qa1.domain.com/__cypress/runner/cypress_runner.js:84097:11
From previous event:
    at onNext (https://qa1.domain.com/__cypress/runner/cypress_runner.js:84094:57)
    at done (https://qa1.domain.com/__cypress/runner/cypress_runner.js:30674:5)
    at https://qa1.domain.com/__cypress/runner/cypress_runner.js:30718:11
From previous event:
    at callFn (https://qa1.domain.com/__cypress/runner/cypress_runner.js:30717:10)
    at Hook.../driver/node_modules/mocha/lib/runnable.js.Runnable.run (https://qa1.domain.com/__cypress/runner/cypress_runner.js:30706:7)
    at https://qa1.domain.com/__cypress/runner/cypress_runner.js:84151:28
From previous event:
    at Object.onRunnableRun (https://qa1.domain.com/__cypress/runner/cypress_runner.js:84140:17)
    at $Cypress.action (https://qa1.domain.com/__cypress/runner/cypress_runner.js:78413:30)
    at Hook.Runnable.run (https://qa1.domain.com/__cypress/runner/cypress_runner.js:82943:20)
    at next (https://qa1.domain.com/__cypress/runner/cypress_runner.js:31056:10)
    at https://qa1.domain.com/__cypress/runner/cypress_runner.js:31078:5
    at timeslice (https://qa1.domain.com/__cypress/runner/cypress_runner.js:26146:27)

My package:

{
	"devDependencies": {
		"@cypress/code-coverage": "^1.8.0",
		"@percy/cypress": "^2.0.0",
		"axe-core": "^3.3.1",
		"concurrently": "^4.1.1",
		"cypress": "^3.4.1",
		"cypress-axe": "^0.5.1",
		"fs-extra": "^8.1.0",
		"istanbul-lib-coverage": "^2.0.5",
		"mocha": "^5.2.0",
		"mocha-multi-reporters": "^1.1.7",
		"mochawesome": "^4.1.0",
		"mochawesome-merge": "^2.0.1",
		"mochawesome-report-generator": "^4.0.1",
		"nyc": "^14.1.1"
	},
	"dependencies": {
		"ls": "^0.2.1",
		"rimraf": "^2.6.3",
		"yargs": "^13.3.0"
	}
}

My test:

describe('APPB', function () {
	context('[Login] Login into account', function () {
		beforeEach(function(){
		  cy.visit('/login')
		  cy.injectAxe()
		})
		it('[Login] Redirects to /dashboard on success', function () {
			cy.percySnapshot('testing before')
			
			cy.get('input[name=username]').should('have.attr', 'placeholder', 'Username or Email address')
				.type('[email protected]', { delay: 50 }).should('have.value', '[email protected]')
			cy.get('input[name=password]').should('have.attr', 'placeholder', 'Password')
				.type('12345678', { delay: 50 }).should('have.value', '12345678')
			
			cy.percySnapshot('testing after')
			
			cy.get('button.button.primary').contains('Sign in').click() 
			cy.checkA11y()
		})
	})
})

Any ideas or suggestions on what could be wrong with my setup and or test?

New API RFC

Issues with the current API

  • Multiple optional positional arguments in the checkA11y() method make it awkward to use and very difficult to extend. We have several open pull requests that are adding a new argument at the end.
  • Requires too much setup to have violations summary either in the terminal or JSON that could be supplied to another too.
  • Doesn't support Cypress chaining in the checkA11y() method, and uses a custom context argument. This makes the usage less idiomatic and doesn't allow integration with tools like Cypress Testing Library.
  • No way to define defaults for the checkA11y() method, we have to pass things like violationCallback in every call or create a custom command.

Related issues: #40, #49, #62, #67, #68

New API proposal

  1. Remove the context argument of the checkA11y() method, and rely on the Cypress chaining instead:
cy.checkA11y() // Check the whole document
cy.getByTestId('my-modal').checkA11y() // Check part of the document
  1. Replace the rest of the arguments with an optional object of the following shape:
interface CypressAxeOptions {
  axeOptions?: axe.RunOptions, // This used to be `options` argument
  shouldFail?: (violations: RunResults[], results: RunResults[]) => boolean,  // Replaces `includedImpacts` and `skipFailures`
  reporters?: Reporter[]
}

type Reporter = (results: RunResults[]) => void

interface RunResults {
  filename: string, // Test file
  label?: string, // Label passed to checkA11y
  results: axe.Result[]
}

The defaults are going to be:

{
  axeOptions: undefined,
  shouldFail: (violations) => violations.length > 0,  // Fail on any number of violations
  reporters: [
   require('cypress-axe/cli-reporter')
  ]
}

Reporters are replacing the violationCallback: they are more flexible, have access to both passes and violations, and you could have global reporters that would run after all tests, or pass a reporter for a separate checkA11y() method call. The default reporter is printing an overall summary after all tests.

  1. Add a new optional label to the checkA11y() method:
cy.checkA11y(options?: CypressAxeOptions, label?: string)

Similar to labels describe() or it() methods, to identify results of a particular call.

  1. Add a new method to define defaults:
cy.configureCypressAxe(options: CypressAxeOptions)

It should accept the same object as the checkA11y() method, and set default values that could be overwritten in checkA11y().


I hope this covers all the use cases we have. I'm not sure that all the names are clear, so if you have any feedback, leave a comment ;-)

False Positives Occurring

Hi, I am currently using cypress-axe and I'm experiencing the same issues that were identified in this issue that's been closed:
#22

I left a comment there but didn't get a response and I tried implementing the other attempts on that issue board. I was just hoping to get a clear understanding of what the fix was or if there was a fix/workaround at all. Below are the versions I'm using for Cypress:

Cypress version: 6.1
cypress-axe version: 0.12.0

I am also using cypress-axe in conjunction with Storybook: 5.3.21.

Allow users to use any version of axe core

I see a couple of issues opened to upgrade axe core version, which is bundled in this library. (#36, #58)

To avoid this type of request in the future, I think we can make this awesome library more flexible by allowing users to use any axe core version.

I'm proposing to add a parameter to the injectAxe method, so users can specify their axe core source/version to overwrite the default version.

Sample scenario: Assuming the user wants to use the latest axe core

  1. Install the latest axe-core
npm install -D axe-core@latest
  1. Load axe.min.js and pass it into the injectAxe method
cy.readFile('node_modules/axe-core/axe.min.js', 'utf-8').then((content) => {
    cy.injectAxe(content);
    cy.checkA11y();
})

Add typescript/webpack configuration to readme

I am creating a new issue for #7 as the original owner closed it despite the issue not being resolved. If you are having issues getting cypress-axe to work with Typescript/Webpack, please look at that issue for solutions.

Configuring axe rules

Hello,

Love the product! This is a great idea and way to manage a11y development, creating a much tighter feedback loop.

I wanted to request support for configuring axe rule enforcement - some rules are inappropriately enforced in certain scenarios, leading to false positives. Ideally in cypress/support/index.js a config file could be passed in, leveraging axe.configure().

This may already be possible! If so, just some simple docs with an example would suffice.

Error: Arguments are invalid whening passing Options object in cy.checkA11y

Hello there,

Im using cypress with typescript and when i set the variable:

const A11yConfig: Options = {
  includedImpacts: ["critical"],
  runOnly: {
    type: "tag",
    values: ["section508"]
  }
}; 

Then, i use the cy.checkA11y(A11yConfig), when i run the tests an error happens:

axe.run arguments are invalid

Am i doing something wrong?

Thanks!

Versions:
"cypress": "^4.12.1"
"@types/cypress-axe": "^0.8.0"
"axe-core": "^3.5.5"

False positives when using Cypress-axe

I started using cypress-axe to catch a11y issues when running end to end tests automatically, but I am finding it challenging to get the tests to pass. I get two errors which I believe are false positives:

  • page-has-heading-one
  • landmark-one-main

I use react-axe, which prints a11y issues to browser console in development mode, but I don't see these two issues reported by that package. I tried auditing using Axe Chrome extension, but I do not see these.

I would be happy to provide more information, but at this point, I believe it to be a bug with cypress-axe.

Waiting arbitrary time for the checks to work

Related: #22, vercel/next.js#7945

Full repo: here (note: some of the packages there are not yet published, but the test can run without them)

I have a simple test:

import formatAxeLog from "../helpers/formatAxeLog";

describe("Home page", () => {
  beforeEach(() => {
    cy.visit("/");
  });
  it("Has no detectable a11y violations on load", () => {
    cy.injectAxe();
    // cy.wait(500);
    cy.checkA11y(null, null, formatAxeLog);
  });
});

When I run the tests multiple times they sometimes fail (due to vercel/next.js#7945, as they should), sometimes pass. The only "reliable" way is just to wait. Am I doing something wrong, or this is a bug?

exports is not defined on v0.9.1

Getting this error when using 0.9.1 with axe-core 4.0.2 as a peer dependencies

ReferenceError: exports is not defined
      at axeFunction (eval at <anonymous> (http://172.25.25.197:3000/__cypress/tests?p=cypress/support/index.js), <anonymous>:24:14)
      at eval (eval at <anonymous> (http://172.25.25.197:3000/__cypress/tests?p=cypress/support/index.js), <anonymous>:27630:3)
      at eval (<anonymous>)

Need to upgrade to axe-core 4.0.1

As mentioned here, the latest version of axe-core fixes a CSP issue where injectAxe() doesn't work with a strict content security policy that doesn't allow unsafe-eval for scripts.

I'll submit a PR for this.

Color contrast bug in modals

I have a modal that overlays the screen:
image

(note backdrop darkens)

I'm testing for a11y like so: cy.checkA11y(".modal", null, terminalLog);

But always get this contrast error:

failureSummary: "Fix any of the following:
  Element has insufficient color contrast of 2.92 (foreground color: #979797, background color: #ffffff, font size: 13.5pt (18px), font weight: normal). Expected contrast ratio of 4.5:1"

The font size is 18px, the text is #000 and the background is #FFF. I'm not sure how it's getting this #979797 value. This bug only happens when I am checking my modal element for a11y, so I think something is going wrong when I pass in the element I want to check.

Question about bundling axe-core

Thank you very much for contributing cypress-axe to the community!

I noticed that around 0.7.0 it was decided to bundle axe-core and I guess I'm wondering if would it might be better to keep axe-core as a peerDependency so that consumers can exert some level of control over their version? I notice that there is already a request (#36) to upgrade the bundled version.

Reasons why I ask:

  1. Maybe it's just a recent phenomenon, but I notice that Deque has had a few minor/patch releases to fine-tune the aggressiveness of the axe ruleset. In my application I've had to upgrade a few times (to 3.4, to 3.5) to get around "overly aggressive" (my words, not theirs) errors that were not failing in other a11y checkers.

  2. I could be doing it wrong TM (and therefore it may not be a valuable use case), but in our application we currently also use axe-core directly in some of our faster JSDOM tests that are not covered by our slightly slower full-browser cypress tests. While not maybe technically necessary, it would be nice to have the a single axe-core version/ruleset apply.

Any thoughts are appreciated. Thank you again for creating this useful library.

cannot resolve axe-core/axe.min.js

suddenly getting this error with no changes to my code. Using version 0.10.0:

have tried deleting dist and node_modules folder, running npm run update and built new dist

Module not found: Error: Can't resolve 'axe-core/axe.min.js' in /path/node_modules/cypress-axe/dist

Running cy.checkA11y three times or more causing app to freeze

I'm not sure if anyone else if having this issue. However, when running cy.checkA11y three times, with the following params:

cy.checkA11y(null, null, null, {skipFailures: true});

it's causing the Cypress test to freeze all together with no failures

Readme updates for typescript

Hi,

Great project, thanks!

Just been trying to get this running with a Cypress project in TypeScript but came across some issues, so I am proposing some readme updates for typescript to state that typings are available here

https://www.npmjs.com/package/@types/cypress-axe

https://github.com/DefinitelyTyped/DefinitelyTyped/tree/master/types/cypress-axe

And that it won't work with webpack, but will work with browserify pre-processors

Related issues

#6

dequelabs/axe-core#1406

dequelabs/axe-core#1427

Cypress Typescript users can switch from webpack, to browerify

https://github.com/cypress-io/cypress-browserify-preprocessor

Happy to make the readme changes if required πŸ‘

Suppost Cypress 5.0

With Cypress 5.0 released, could this be added now to peerDependencies? I don't know if other changes would be needed.
"peerDependencies": { "cypress": "^3.1.1 || ^4.0.2" },

cy.injectAxe() is Causing Cypress Test Runner to Freeze

Whenever I try to use cy.injectAxe() in a test, Cypress gets stuck at the step immediately prior. The Cypress test runner itself is locked and must be force-quit.

Here is code to reproduce the problem.

context('Problem with cy.injectAxe()', function () {

  beforeEach (function () {
    // These are not real credentials.
    // Please sign up (for free) at www.vidyard.com, verify your email address, then update these accordingly before running.
    const email = '[email protected]'
    const password = 'password123!'

    cy.visit('https://secure.vidyard.com/user/sign_in')
    cy.get('#username').type(email)
    cy.get('#password').type(password)
    cy.get('#sign-in').click()
    cy.wait(10000) // Just to make sure the page is fully loaded.
  })

  it('Accessibility Test', function () {
    // When the lines below are commented out, there is no issue.
    // When the lines are let to run, the Cypress Test Runner gets stuck on the cy.wait step.
    // Cypress will be completely frozen and must be force quit.
    cy.injectAxe()
    cy.checkA11y()
  })
})

I am using Cypress version 4.12.1 and cypress-axe version 0.8.1 on macOS 10.14.6 with Chrome 84.0.4147.105 and Firefox 79.0.

Having an error when injecting Axe

Hi,

I am trying to follow the instructions as documented.
When doing cypress open, I have the following error:

ReferenceError: t is not defined

This is happening when trying to eval(axe.source).
It appears that axe is not defined inside the eval.

I am not sure why.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.