On web accessibility evaluation environments

Modern Web sites leverage several techniques (e.g. DOM manipulation) that allow for the injection of new content into their Web pages (e.g., AJAX), as well as manipulation of the HTML DOM tree. This has the consequence that the Web pages that are presented to users (i.e., browser environment) are different from the original structure and content that is transmitted through HTTP communication (i.e., command line environment). This poses a series of challenges for Web accessibility evaluation, especially on automated evaluation software.This paper details an experimental study designed to understand the differences posed by accessibility evaluation in the Web browser. For that, we implemented a Javascript-based evaluator, QualWeb, that can perform WCAG 2.0 based accessibility evaluations in both browser and command line environments. Our study shows that, in fact, there are deep differences between the HTML DOM tree in both environments, which has the consequence of having distinct evaluation results. Furthermore, we discovered that, for the WCAG 2.0 success criteria evaluation procedures we implemented, 67% of them yield false negative answers on their applicability within the command line environment, whereas more than 13% of them are false positives. We discuss the impact of these results in the light of the potential problems that these differences can pose to designers and developers that use accessibility evaluators that function on command line environments.