W3C ACT Implementation

Overview - Accessibility Conformance Testing (ACT)

The W3C established a community of developers of accessibility evaluation methodologies and testing tools and some accessibility Subject Matter Experts (SMEs). The community establishes and documents ACT Rules for testing the conformance of WCAG and ARIA. These test rules address automated, semi-automated, and manual testing, but since the Checker is primarily an automated test tool, that is the primary interest. ACT’s goal is to make accessibility testing more transparent, and thus reduce confusion caused by different interpretations of the standards (WCAG Success Criteria). ACT members write & share with the community and implement ACT Rules into their products and services. They also test their tool against the ACT Rules and measure consistency and accuracy as reported in their ACT Implementation reports. ACT is not intended for end-users of accessibility tools, web developers, content authors, QA testers, etc. For more overview and background information, see Accessibility Conformance Testing (ACT) Overview.

ACT Rules

1. All ACT rules are listed by the standard or specification that they cover, including WCAG, ARIA, Proposed Rules, and beyond.

2. All ACT rules follow a review and publishing process.

3. Each ACT rule has a 6 character ID, description, applicability, expectations, assumptions, the WCAG criteria it maps to (reports against), and example test cases.

4. Each rule’s Test Cases (test case snippets) include Passing examples, Failing examples, and Inapplicable examples

   • See an example - Rule ID: 2779a5 - HTML page has non-empty title [w3.org/WAI/standards-guidelines/act/rules/2779a5/].

   • The ACT Rule ID is found in the URLs, such as:

     • Rule description URL: e.g. w3.org/WAI/standards-guidelines/act/rules/97a4e1/

     • Rule test case URL: e.g. w3.org/WAI/content-assets/wcag-act-rules/testcases/97a4e1/….html

IBM Checker mapping to ACT

1. Individual Checker rules may test and map (reports against) one or more ACT rules by identifying the ACT ID in the rule_name_.ts file using the act: [ ], declaration.

   1. See example img_alt_valid.ts Checker rule maps to act: "23a2a8" around line 53 in the rule’s logic code.

   2. See example aria_id_unique.ts where a single Checker rule maps to more than one ACT rule - act: ["59796f", "6a7281"],

   3. See example aria_graphic_labelled.ts where a single Checker rules maps to multiple ACT test cases depending on the Fail reason: act: [{ "7d6734": {"Pass_0": "pass", "Fail_1": "inapplicable", "Fail_2": "fail", "Fail_3": "inapplicable" } }],

   4. See example aria_img_labelled.ts where a single Checker rule maps to multiple ACT Rules and multiple test cases:

act: ["23a2a8", { "7d6734": {
                  "Pass_0": "pass",
                  "Fail_1": "inapplicable",
                  "Fail_2": "fail",
                  "Fail_3": "inapplicable"
                 }
      }],

ACT Results

See test-act-w3 in the accessibility-checker repo that use Puppeteer to run the ACT test cases and report the results.

See the ACT Test Results (act-report-v2.txt) Artifact created from Actions by each PR build that details the results of running the Checker against the ACT rules. It’s available in Actions for debugging purposes. After each deployment, it’s publicly available in the Checker’s implementation report (assuming ACT updates it). Note the date of the report. Ignore the Checker version until ACT fixes the issue.

Each ACT rule is listed, each test case is listed, and if there is a mismatch, then there is a message listed such as “Expected passed, but returned earl:failed”

If no Checker rule maps to an ACT rule, then a message: “? No checker rules“ is listed.

Sample ACT Test Results (act-report-v2.txt) saved to Box, but it’s best to view the Artifact from Actions for the most current results.

ACT Implementation report

The “official results” are publicly available from the ACT Implementation Reports page - Equal Access Accessibility Checker after each deployment. Results from other test tools are also publicly available.

The Checker results sent to ACT are formatted by ACT into the common-looking report.

[Is the ACT Community Repo still being used by ACT?] ACT Community Git repo at https://act-rules.github.io/rules/; the JSON file that indexes them all is at https://act-rules.github.io/testcases.json

Background for repo contributors

The following statements help clear up the purpose and high level understanding of these related topics and specs:

The Evaluation and Report Language (EARL) is a machine-readable format for expressing test results (not the rules).

EARL is an RDF vocabulary (not rules & not results), the terms of which are defined by Evaluation And Report Language (EARL) 1.0 Schema

Some of the terms defined by EARL are being used by the Accessibility Conformance Testing (ACT) Rules Format (v 1.0 Oct 31 2019).

The ACT Rules Format defines a format for writing accessibility test rules (not the results). This format is intended to enable a consistent interpretation of how the test was / is to be done ... and promote consistent results between complementary and competing 3rd party engines in reporting accessibility testing. Example tools include Axe-core, SortSite, Alpha (SiteImprove), etc. As of 2022 it is not clear yet if/when WebAIM's WAVE engine plans to publish an ACT implementation report.  

Accessibility Conformance Testing (ACT) Overview - introduces the ACT Rules Format, lists ACT Rules formally published by W3C, Who ACT is for, and Who develops ACT rules.

Each rule has the following sections (these links go to an example rule: ARIA required context role): 

Applicability, Expectation, Assumptions, Accessibility Support, Background, Test Cases, Glossary, Implementations, Changelog 

but as mentioned above, its easier to see / use all the ACT rules in the ACT Community Git repo at https://act-rules.github.io/rules/ 

ACT Community Git repo also includes 

the same above sections and an additional section called Acknowledgements of who created / submitted the rule (see example at https://act-rules.github.io/rules/ff89c9#acknowledgments)

and both the ACT Community Git repo and the W3C list of Rules contain an Implementation Report per rule showing which tools support that particular ACT rule. One goal of the squad would be to determine and eventually publish the support in the IBM Accessibility Checker for the particular ACT. Perhaps the squad could publish that eventual implementation support report on the IBM Equal Access Toolkit Tools (https://www.ibm.com/able/toolkit/tools) page, Equal Access Git repo (https://github.com/IBMa/equal-access), and/or in the user documentation.