pa11y/pa11y

Pa11y is your automated accessibility testing pal

pa11y's Issues

Default to TLSv1 in order to support older versions of PhantomJS

PhantomJS previously (<= 1.9.7) used SSLv3 for all secure connections and since a lot of servers are disabling it due to the 🐩 attack, the --ssl-protocol=tlsv1 flag would need to be used to change the default behaviour.

PhantomJS 1.9.8 (released 5 days ago) changed the default protocol to TLSv1, fixing the issue.

This is more of a question: should the --ssl-protocol=tlsv1 flag be passed by default in order to support older versions of PhantomJS? The alternative would be to require a newer PhantomJS binary (>= 1.9.8) when the server doesn't support SSLv3.

PhantomJS hangs on page redirect

If there's a redirect in the page load, PhantomJS will hang forever (it seems). This may be related to JavaScript redirects but I've experienced with server redirects too.

Include PhantomJS in library?

Re-investigate using https://github.com/Medium/phantomjs to install PhantomJS with pa11y. This would only be acceptable if it doesn't touch pre-existing installs and can run in CI.

PhantomJS timeouts

Occasionally (rarely) PhantomJS can hang which causes the command to just sit there doing nothing.

If we can't solve the root issue, it may be worth implementing a simple timeout to prevent this from clogging up CI servers etc. If a timeout is added, make sure it can be specified with an option like this:

-t, --timeout <ms>

Possibly related to #2.

phantomjs timeout when getting new htmlcodesniffer

trying this
pa11y -c "http://raw.githubusercontent.com/bryanrasmussen/HTML_CodeSniffer/master/HTMLCS.js" www.sundhed.dk
or this
pa11y -c "https://raw.githubusercontent.com/bryanrasmussen/HTML_CodeSniffer/master/HTMLCS.js" www.sundhed.dk

always gives me

Loading page...
Running HTML CodeSniffer...
Error: PhantomJS timeout

another thing is I want to pass a lot more info from htmlcs, for example the html that triggered the warning, an xpath to the element etc.

If I want to the above should I just make sure that it comes out in the message.msg, and then that is adequate ( as I looks like from the code) or will I have to do some adaption of pa11y itself? (if so I would like to know how to compile)

Break up and unit test `lib/pa11y.js`

This is the last un-(unit)tested part of the code base – mainly because it's quite large and difficult to test in its current state. Look at breaking this apart and making more testable.

Release 1.0

This is a placeholder issue to prevent the milestone from accidentally closing.

• Fix all outstanding issues
• Version, tag etc.
• Publish to npm

Finalise and document the JavaScript API

Make a JavaScript API for pa11y which is consistent, stable and documented. This will allow for tools to be built on top of pa11y's core (e.g. #16), as well as enable pa11y to be easily integrated into other tools such as Grunt.

This may mean moving some of the more command-line-specific stuff out of lib/pa11y. Off the top of my head, we'd need to reconsider:

• Use of process.cwd in path resolution (for config files)
• Use of console.log/console.error in reporters. Either that, or reporters should be exclusive to the command-line tool and pa11y should just deal in JavaScript objects

Release pa11y 2.0

Pa11y 2.0 is currently in development, and viewable on the 2.x branch for now. Issues will appear in the 2.0 milestone, and we'll be aiming for near feature parity with pa11y 1.x.

Pa11y 2.0 is currently in an alpha state, and is not ready to replace 1.x as part of your toolchain. This is mainly because not all of the accessibility rules present in 1.x have been built yet. Also, things may (and are likely to) change before we reach Beta. You can install the alpha version with:

npm install -g https://github.com/nature/pa11y/tarball/2.x

For contributors: feedback is hugely appreciated, and no change to the current 2.0 API/code is too big to consider.

The major changes in this version are as follows:

• Drop the PhantomJS dependency
• Drop HTML CodeSniffer as a tester
• Drop Section508 support (just for initial release – it can be built in later)
• Allow writing custom rules
• Allow configuring of existing rules
• Allow testing of HTML snippets as well as URLs
• Allow ignoring rules and changing error levels

Review Truffler

Hey @whymarrh, I've been working on splitting out the PhantomJS part of pa11y into a separate library, which is now open-sourced at https://github.com/nature/truffler. Before I start using this, can you envisage any issues? Is there anything I've missed?

There's no rush on this, but it'd be excellent if you could cast your eyes over it at some point.

Allow users to log into password-protected areas of sites

Maybe tackle this by allowing inclusion of a JS file in the page as part of the program running.

pa11y hard to use from non-internet connected machines

Hi,

Thanks for making pa11y. We're trying to use it for our project but since we have machines with no internet connectivity we have to jump through some hoops to make HTMLCS.js available to download via HTTP.

Is there a reason you're not just including this js file in your code or is there a reason it can't be fetched from a local file system?

Happy to try and make a fix but wanted to understand if there were some reasons for this not being possible.

Thanks,

Tom

running pa11y from the command line throws an error

phantom stderr: 2013-12-18 17:01:57.429 phantomjs[8694:507] *** WARNING: Method userSpaceScaleFactor in class NSView is deprecated on 10.7 and later. It should not be used in new applications. Use convertRectToBacking: instead.

Switch to using WebSockets to pass data to/from PhantomJS

See #1, #2 and this blog post for more context.

Default config file `.pa11yrc`

Allow pa11y to use a default config file <cwd>/.pa11yrc.

The .pa11yrc config file should be used if present, and if a config file hasn't been specified with the --config command-line option.

Review Pa11y 2.0 API

A first pass of pa11y 2.0 has been built, and is viewable on the 2.x branch for now. It's had a couple of big API changes, and the code itself is a ground-up rewrite.

I decided to stick with PhantomJS and HTML CodeSniffer for this release, if we ever move away from them it'll be much further down the line. The core library has been split into a few smaller modules and is a lot cleaner and easier to understand than 1.0; also a lot of attention has been given to the documentation.

I'd like to get feedback on the API changes which are significant (hence the 2.0 bump) before this gets released so that we can make changes if needed. @dotcode, @whymarrh, @stephenmathieson – it'd be much appreciated if you had time to review (the API more than the code itself).

The 2.x README is pretty comprehensive, and I've also written a Migration Guide to highlight the changes and help people make the switch.

Thank you

Confirming usage under the GPL

Greetings team pa11y! @adelevie introduce me to the awesomeness that is pa11y last week, and since then, we've set out to make an automated 508 testing tool for Jekyll and other static sites. We've lovingly called the Ruby gem Ra11y.

Before we go too far down the 🐰 hole, we wanted to check in with you and make sure that we were honoring both the project's vision and the spirit of the GPL.

Pa11y is not distributed with Ra11y, instead, the Readme instructs users on how to install Pa11y via NPM. Ra11y shells out to pa11y with default arguments, and then parses the JSON returned via stdout. The executable is even configurable if the user would like to use a different command-line testing tool.

From The GPL FAQ:

By contrast, pipes, sockets and command-line arguments are communication mechanisms normally used between two separate programs. So when they are used for communication, the modules normally are separate programs.

To me, it sounds like the letter of the law is clear, but because we'll likely want to contribute improvements upstream, wanted to check in with the project maintainers personally before we were too far along.

Edit: To answer the most obvious question ("why not just use pa11y?"), Ra11y serves as an test abstraction, allowing you run to run accessibility tests against an entire directory of static files locally, aggregating the results.

Tests fail on first run

The first time you run grunt from within pa11y, you get this:

$ grunt

…

Running "mochaTest:functional" (mochaTest) task


  pa11y --help
    ✓ should be successful
    ✓ should output usage information

  pa11y --version
    ✓ should be successful
    ✓ should output the current version


  pa11y --config ./config.json http://localhost:4117/normal
    1) "before all" hook

  4 passing (2s)
  1 failing

  1) pa11y --config ./config.json http://localhost:4117/normal "before
all" hook:
     Error: timeout of 2000ms exceeded
      at null.<anonymous>
(/Users/jrobinson/repositories/git/pa11y/node_modules/grunt-mocha-test/node
_modules/mocha/lib/runnable.js:165:14)
      at Timer.listOnTimeout [as ontimeout] (timers.js:110:15)



Warning: Task "mochaTest:functional" failed. Use --force to continue.

Aborted due to warnings.

Tests pass if run again.

Ignore SSL errors by default

For 2.0, it'd perhaps be good to ignore SSL errors by default. See #64. This should be much easier to do in the newer code-base.

Implement error/warning rules that HTML CodeSniffer implements

The following list is taken roughly from HTML CodeSniffer's implemented rules. We should aim to implement tests for all of these in pa11y 2.0 as a baseline.

I've excluded notices for now to focus on actionable items.

• 1.1.1 Non-text content
  • missing alt attribute on img (error)
  • empty alt attribute on img inside an a with no other content (error)
  • empty alt attribute on img but title attribute is set (error)
  • missing alt attribute on input type="image" (error)
  • missing alt attribute on area (error)

todo: continue adding these, also get unique codes for each

Allow HTML source from file system and not just URLs.

HTML Code Sniffer allows for files on file-system to be used as the source HTML.

This is very useful when you need to analyse many HTML files and want to break the process into 2 distinct phases:

1. Download all HTML
2. Analyse HTML files for WCAG compliance.

Pa11y fails with error URL could not be loaded

Pa11y Ignore SSL Certificate warning

Hello, I am trying to integrate Pa11y as part of Build process. However on trying to perform site validation seeing following error.

/pa11y WCAG2AA https://testsiteurl

_-::> Loading page...
_-::> URL could not be loaded

It appears my Local test environment does not have valid SSL Certificate.
for eg: Curl https://testsiteurl throws following message
curl: (60) SSL certificate problem: Invalid certificate chain

Could you please advise, if there is any way to ignore ssl certificate warning, when validating urls with Pa11y.
for eg: Curl -k testsiteurl , using -k we can ignore ssl warning with Curl

Thanks,
Raja

Don't use return codes to indicate lint error count

I feel a bit guilty about this one, since I introduced it in #25 and #32.

I still think the process should exit with an error to indicate that there were errors (as per #14), but using the actual count as the code will break if there are too many errors (more than 255). Also, using -1 wraps around, and while it is still possible to get it back, the process is a hassle (not to mention that if there are exactly 255 errors, the results count will be indistinguishable from a pa11y internal error).

Thoughts on this?

Allow for reporting levels to be set: error, warning, notice

Great tool, very good implementation of HTML Code Sniffer!

For CI integration, it would be very useful to be able to specify the reporting level, eg.

0 - error & warning & notice
1 - error & warning
2 - error

With 0 being the default.

In our case, this would allow CI testing to halt a build if an error is found in the WCAG standard but allow the build to continue if only warnings and notices are present.

Not able to get detailed accessibility report

I am using pa11y for accessibility audit on my web application and it generate a report which shows following details -

• code : WCAG2AA.Principle2.Guideline2_4.2_4_2.H25.2
• message : Check that the title element describes the document.
• type : notice

But i am not able to get details like where any particular accessibility error/warning exists in my html.
May be i am not able to understand.
so, please help me on this or let me know if there is any option to customize reports to get complete details to resolve issues in html.

Allow filtering out certain rules

Not exactly sure how we'd tackle yet. Maybe allow user to specify a JSON config file or pass rule names in as CLI flags.

Conform to GPL usage guidelines

Just noticed that pa11y doesn't follow the GPL guidelines for using the license.

Quoting from http://www.gnu.org/licenses/gpl-howto.html (emphasis is mine)...

"...the process involves adding two elements to each source file of your program: a copyright notice... and a statement of copying permission, saying that the program is distributed under the terms of the GNU General Public License..."

"The copying permission statement should come right after the copyright notices.
...
For programs that are more than one file, it is better to replace “this program” with the name of the program, and begin the statement with a line saying “This file is part of NAME”. For instance,

This file is part of Foobar.

Foobar is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

Foobar is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with Foobar.  If not, see <http://www.gnu.org/licenses/>.

This statement should go near the beginning of every source file, close to the copyright notices."

"For interactive programs, it is usually a good idea to make the program print out a brief notice about copyright and copying permission when it starts up."

"You should also include a copy of the license itself somewhere in the distribution of your program. All programs, whether they are released under the GPL or LGPL, should include the text version of the GPL. In GNU programs the license is usually in a file called COPYING."

PhantomJS Error

Hello,

I cannot launch the tool because i get an error about the path of the phantomjs. However, phantomjs path is on the system environment variables, and like you can see in the image, I can run the commands of phantomjs like for example (phantomjs --version) to check if phantomjs is installed.

Do you know what could be wrong?

Thank you,

Zombies!

First of all, thank you for such a wonderful project. We have been using it quite extensively and have thus found some problems with it. The most problematic of which is that it spawns zombie processes that consume memory and cpu and eventually crash the server.

Some details:
If you run pa11y on a page that generates an error, p11y eventually reaches its timeout but at least two processes live on (node and phantomjs).

To reproduce the error I have been able to do the following:

1. run pa11y -r json http://prairieschooner.unl.edu/?q=fusion%2Fwork%2Fbus-driver and wait for it to error
2. run ps x -o "%p %r %y %x %c " or look at the output of ps x and you should see two zombies processes (one for node and one for phantomjs). Sometimes you will see one or the other.

We are using SiteMaster to scan our sites with several metrics, including accessibility via pa11y. In order to get around this problem, I implemented our own custom timeout for the pa11y process, which recursively kills child processes. See: UNLSiteMaster/metric_pa11y@d478bc8

I'm not sure where exactly this problem lives. It might actually be a bug in phantomjs. I just thought I'd give you a heads up.

Unhandled error event

sieben@sieben-lincs:~|⇒ pa11y nature.com

Welcome to Pa11y
We'll sniff your page for you now.

Loading page...

events.js:72
throw er; // Unhandled 'error' event
^
Error: spawn ENOENT
at errnoException (child_process.js:980:11)
at Process.ChildProcess._handle.onexit (child_process.js:771:34)

spidering support

i know it's a huge feature request, but spidering support would be extremely helpful.

it'd be nice to also be able to specify a maximum depth of spidered pages:

$ pa11y http://website.org/ --depth 15

thoughts? i'd be happy to contribute, or create a separate module (pa11y-spider?) for such a task.

Code Sniffer Error

Hello!
I got a generic "Code Sniffer Error" when I launch the following command:
pa11y scuolafalconeborsellino.it

Also when use the flag -d, I can't understand what went wrong:

Welcome to Pa11y
We'll sniff your page for you now.

Debug: Starting timeout timer (30000ms)
Loading page...
Running HTML CodeSniffer...
Error: HTML CodeSniffer error

What to do to better understand the problem? I suppose this is caused by the frame included in the webpage...

Thank you very much!

remove heavy dependencies

dependencies such as _ are a bit large for what they're being used for and may become deeply coupled with this module. how would you feel about removing the underscore dependency and relying on smaller, cleaner modules for underscore-esque functionality (_.defaults, _.countBy, etc.)?

Document all of the rules you can ignore

This probably involves the joyous task of trawling through HTML CodeSniffer and grabbing each of the rules (code plus message).

At the moment you have to copy rules from the console output which is quite inefficient, and doesn't allow you to ignore rules that you haven't come across yet.

Source code and screenshot backup

Hello,
For some website, the page tested by pa11y is not the same as the one displayed in a browser (maybe cause of the user-agent ...). It would be nice to have an option to get a backup of the source page code and a screenshot (phantomjs) of the page tested by pa11y to be sure this is the good one.

Set Unique User Agent

Investigate setting a unique user agent to allow sites to filter out pa11y traffic from analytics etc.

Return non-zero status code when errors reported

This would be useful if including pa11y as part of a Git pre-commit hook or in a CI environment.

It'd only be when errors reported, not for warnings or notices.

Allow sending options to pa11y

I've been seeing errors where pa11y fails to open URLs due to failed ssl handshakes. The solution is to change the ssl protocol that phantomjs uses via this command line option:

--ssl-protocol=any

I'm sure other issues could crop up where passing user defined configuration to phantomjs would be helpful, so maybe it would be better to implement phantomjs's --config option:

--config=/path/to/config.json

Sources:

1. http://stackoverflow.com/questions/12021578/phantomjs-failing-to-open-https-site#answer-17101575
2. https://github.com/ariya/phantomjs/wiki/API-Reference

Allow customizing the viewport width

It looks like pa11y currently renders and checks pages at a mobile width. This is useful, and goes with a mobile first attitude, however it ignore other potential breakpoints.

Especially when considering responsive design, the layout, colors, content, etc can change at different breakpoints, which can affect accessibility.

So, it would be nice if there was a command line option to configure the viewport size. Ideally, you could add several breakpoints and pa11y would run the test suite against each breakpoint, and then return the breakpoints for each error that was found.

Write a `pa11y list` command

As rules/suites are now customisable and extendable, it would be useful if the command-line tool could list all of the ones present. Thinking pa11y -l.

Could testing be performed on fragments of HTML?

Not sure of the feasibility of this but it would be great to be able to run pa11y against a fragment of HTML as well testing full pages.

Investigate passing data to PhantomJS

The way we pass data to PhantomJS at the moment is awful. We're passing the 'standard' to use through the querystring.

There's almost definitely a nicer way.

I couldn't work out how to pass data into PhantomJS any other way using our chosen Node library. I may just be being blind.

pa11y installing error

I have installed NodeJS 0.10 but i could not install pa11y tool. I also have installed Windows SDK and VisualStudio.

However, when try to install pa11y tool "$ npm install -g pa11y", i got this error and the tool fails to install.

I see that the "weak 2.2" script and package fail to install, but i don't know why.

Add element XPath to result

See discussion in #61. This could be done quite easily in pa11y 2.0, you'd need to get each element's XPath in lib/inject (which run's within PhantomJS). You could use one of the functions in these SO answers:

• How to calculate the XPath position of an element using Javascript?
• Javascript get XPath of a node

Revisit (and refactor) feature tests

These were thrown together as a basic safety net before we had a decent test suite. There's now a lot of repetition and most of the JS code in test/feature is a mess.

Sync JSON config file with command line options

Idea: adding the ability to configure all the same options in the JSON config (.pa11yrc) that are available as command line flags.

I'm not 100% sure where I would draw the line on this as not all of the command line flags are worthy (e.g. help and version), but I think the others might be useful.

Thoughts? With ideas like #50/#51 and #53, maybe the JSON config is underutilised.

Ignore not working

Hello,

Hopefully I'm just doing something wrong but I can't get the ignore feature to work. Here is my code:

pa11y.sniff({
    url: url,
    standard: accessibilityStandard,
    useragent: 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.104 Safari/537.36',
    config: {
        ignore: [
            'WCAG2AA.Principle1.Guideline1_4.1_4_3.G18'
        ]
    },
}, function(err, results) {
    // Do something
});

Whenever I run this command I get nothing but WCAG2AA.Principle1.Guideline1_4.1_4_3.G18 errors (because I've eliminated everything else).

Is there something wrong in my code or is there a defect?

Release 1.1

This is a placeholder issue to prevent the milestone from accidentally closing.

• Fix all outstanding issues
• Version, tag etc.
• Publish to npm

Investigate using a local copy of HTML CodeSniffer

This will allow you to test local URLs with no internet – not currently possible.

Document writing custom reporters properly