Espresso AccessibilityChecks: Toggle for warnings rather than errors when Accessibility issues found?

[google]

Espresso AccessibilityChecks: Toggle for warnings rather than errors when Accessibility issues found?

[caseyburkhardt]

Just replied to your AOSP issue. Apologies for the double reply -- just saw you posted it here as well.

The behavior you're describing is how the Android Accessibility Test framework was designed to integrate with testing frameworks like Espresso and Robolectric. By default, we evaluate only elements (and sub-hierarchies) which are interacted with, and throw an Exception when an error is detected within the scope of that interaction. As you've discovered, you can use setRunChecksFromRootView(true) to cause ATF to crawl to the View root and evaluate all elements within the View hierarchy.

To instead fail no tests, but still report accessibility findings to logcat, you can use setThrowExceptionForErrors(false). (Note that this reports all AccessibilityCheckResultTypes, including ERROR, WARNING, INFO, and NOT_RUN) You can also write your own logic for handling results, using AccessibilityValidator#checkAndReturnResults(View) to invoke ATF from a particular starting point and return its findings, or AccessibilityValidator#setSuppressingResultMatcher(Matcher) to filter/whitelist certain results from your affecting your test results.

Can you clarify your exact request a bit more? Are you saying it would be helpful for you to be able to configure ATF to fail your Espresso tests for the WARNING result type? Generally, we demote things to WARNING when we have less certainty about the presence of an issue, but something may potentially be an accessibility issue. As such, we considered it less desirable to fail tests in these cases due to substantially increased noise in test results.

[rheesn]

My ui keeps stoping

[renatoi]

@caseyburkhardt @pweaver-google it would be nice to add a few getters/setters for the AccessibilityValidator. Looking primarily at the possibility to create my own checks or presets, and add or modify the check combinations (viewHierarchyChecks) that will run on a view.

edit: nevermind, I was using an old version and didn't see it now has setCheckPreset()

[caseyburkhardt]

Thanks for the feedback, @renatoi. I owe this project an open source drop, which I hope to devote some cycles to in the coming weeks. Are you just looking to be able to add individual AccessibilityViewChecks to the set run by AccessibilityValidator? Right now we don't support running checks not included in the test framework/our presets, but this is plausible. I'd also be interested to hear what additional functionality / checks you have in mind.

[renatoi]

@caseyburkhardt Currently my use case is to be able to modify some of the existing checks with some ad hoc use cases that might be product specific:

• Touch Target Size: Check if there are clickable parent ViewGroups within the minimum dimensions;
• Touch Target Size: Different requirements for minimum dimensions;
• ClickableSpans: Allow empty values for URLSpan because TalkBack does not display empty URLs and there's currently an issue with the back stack of URLSpans not working properly on TalkBack for deep links for app content.
• Other modifications that might be important in the future.

Sometimes we might find an issue with a check and we might need to get it fixed ASAP, so having the possibility to extend it first and later submit a PR or opening an issue here is probably the best path.

Another use case is to configure the result type of each check (WARNING, ERROR, INFO) instead of a global flag, although I believe this use case might not be very realistic since I believe that we might not want to change this.

Possible new checks:

• Check if a focusable view has a focus indicator drawable (StateListDrawable) for keyboard and D-Pad navigation.
• Check if a view is clickable but it's missing an accessibility class name of "Button" (this might be product specific). We try to keep clickable views having the "button" role by settings its accessibility class name in the Node Info.
• Check if a specific component was used without using its "accessible" version created by the team. For example, charts, cards with accessibility actions, etc.

---

If I'm not mistaken, I might already be able to add new checks with the setCheckPreset method in the AccessibilityValidator. I'd have to create a new class and extend AccessibilityPreset, overriding getViewChecksForPreset. Not sure if this is elegant or the best way though.
Edit: I guess I can't extend it because that's an enum.

However, I believe that the latest version of this framework is not yet available in espresso-contrib 2.2.2 (latest).

[caseyburkhardt]

I believe the main issue discussed here is resolved with the release of the new AccessibilityValidator#setThrowExceptionFor(AccessibilityCheckResultType) API. There are also relevant enhancements to TouchTargetSizeCheck. These have just been merged to master in our v3.1 release. You can expect binaries available on maven in the coming days.