[test] Add an `assertions` mode to the axe regression harness by michelengelen · Pull Request #48915 · mui/material-ui

michelengelen · 2026-08-04T18:29:59Z

Adds an assertions mode to the axe regression harness so a fixture can assert
every axe rule it exercises, not only the CSS-dependent visual ones. This is the
foundation the per-component WCAG conformance reports build on.

Also stops excluding the progress slug from the demo bundle so axe can reach
the LinearProgress demos; screenshots stay disabled slug-wide for it, since the
animated bars are flaky.

Original work by @mj12albert, extracted from #48708 so the harness change reviews
separately from the Button report.

Part of the WCAG conformance effort (see #14187).

Important

How to review this PR

Layer 1 of 14 in a stacked series (#48915 → #48926). Its branch is the base of the series. GitHub cannot chain PR bases across a fork, so every PR in the series targets master instead.

Its diff is exactly its own change — 4475eb3.

Part of the WCAG conformance effort (#14187).

code-infra-dashboard · 2026-08-04T18:32:50Z

Deploy preview

https://deploy-preview-48915--material-ui.netlify.app/

Bundle size

Bundle	Parsed size	Gzip size
@mui/material	0B^(0.00%)	0B^(0.00%)
@mui/lab	0B^(0.00%)	0B^(0.00%)
@mui/private-theming	0B^(0.00%)	0B^(0.00%)
@mui/system	0B^(0.00%)	0B^(0.00%)
@mui/utils	0B^(0.00%)	0B^(0.00%)

Details of bundle changes

Check out the code infra dashboard for more information about this PR.

michelengelen · 2026-08-06T09:38:31Z

Follow-up: #48936 adds the Playwright suite for the CSS-dependent criteria (1.4.10, 1.4.4, 1.4.12) on top of this harness change, and #48935 wires pnpm a11y:scorecard:check into test_static.

michelengelen · 2026-08-06T09:43:31Z

Correction to my earlier comment: the scorecard CI wiring is in #48925, not #48935 (that number does not exist). Follow-ups on top of this harness change are #48936 (Playwright suite for 1.4.10 / 1.4.4 / 1.4.12) and #48925 (pnpm a11y:scorecard:check in test_static).

michelengelen · 2026-08-07T11:55:14Z

Correcting myself again, properly this time: #48935 is [release] v9.3.1 — it exists, it is just unrelated to this series. The scorecard CI wiring is in #48925. Apologies for the noise.

Lets a fixture assert every axe rule it exercises rather than only the CSS-dependent visual ones, which is what the per-component WCAG reports need.

siriwatknp

👍 Thanks for splitting this out of #48708. The assertions refactor reads well — shouldAssert keeps the visual default exactly as before, and a11yRule cannot be undefined at the new call site.

A few things I found.

siriwatknp · 2026-08-10T10:23:05Z

    expect(parseRoute('/regression-Rating/FocusVisibleRating')).to.equal(null);
  });

-  it('parses a docs-components route into path/slug/demo', () => {


Is there a reason to remove these three tests?

From what I checked, parseRoute and the buttons rule are both unchanged by this PR, so all three still pass on this branch. They cover the route parser and the brace-glob matching, which is what the a11y enrolment depends on.

If the goal is to not depend on A11Y_RULES churn, maybe rewrite the two getConfig ones against a local array, the way honours last-match-wins when multiple rules apply already does?

siriwatknp · 2026-08-10T10:23:05Z

-    (v) => VISUAL_RULES.includes(v.id) && !skip.has(v.id),
-  );
+  const shouldAssert = (ruleId: string) =>
+    !skip.has(ruleId) && (assertions === 'all' || VISUAL_RULES.includes(ruleId));


This branch never runs in CI. A11Y_RULES has one entry and it does not set assertions, and there is no test file for recordA11y under test/regressions/a11y/.

recordA11y is easy to call directly, so a small unit test would cover it. Could be something like:

recordA11y({ task: { meta: {} } } as TestContext, results, { slug: 'buttons', demo: 'BasicButtons', assertions: 'all', });

Three cases would be enough — visual ignores a non-visual violation, all throws on it, and all plus skipAssertions stays silent.

siriwatknp · 2026-08-10T10:23:05Z

      });
    });
+
+    describe('Avatar', () => {


This looks out of scope for the harness layer. It is a WCAG 1.4.12 conformance check for Avatar, and avatars is not enrolled in A11Y_RULES.

Should it live in the Avatar report PR instead, so this one stays only the assertions change?

siriwatknp · 2026-08-10T10:23:06Z

+            (node) => node.textContent === 'OP',
+          );
+          const range = document.createRange();
+          range.selectNodeContents(avatar);


.find() above returns undefined if LetterAvatars ever changes its initials, and then this line throws parameter 1 is not of type 'Node'. The test still fails, which is good, but the message does not say what happened.

Suggested change

range.selectNodeContents(avatar);

if (!avatar) {

throw new Error('LetterAvatars no longer renders an "OP" avatar');

}

range.selectNodeContents(avatar);

siriwatknp · 2026-08-10T10:23:06Z

+  const shouldAssert = (ruleId: string) =>
+    !skip.has(ruleId) && (assertions === 'all' || VISUAL_RULES.includes(ruleId));
+  const assertedViolations = results.violations.filter((v) => shouldAssert(v.id));
+  const assertedIncomplete = results.incomplete.filter((v) => shouldAssert(v.id));


shouldAssert is applied to incomplete the same as violations, so assertions: 'all' also turns every axe incomplete into a hard failure.

incomplete means "needs human review", so it is the non-deterministic bucket. Under visual it is bounded to two rule ids, under all it is every WCAG-tagged rule. GLOBAL_DISABLED_RULES plus the tag-scoped runOnly limit this a lot, so maybe it is fine.

Was this intended? If yes, I think a line in the assertions JSDoc saying skipAssertions is the escape hatch for noisy incompletes would help.

siriwatknp · 2026-08-10T10:23:06Z

+ * Scope: the components with a conformance report under
+ * `packages/mui-material/src/<Component>/accessibility.md`. Others onboard
+ * incrementally.


packages/mui-material/src/<Component>/accessibility.md does not exist yet on this branch, so a reader of this layer cannot act on it. I think this describes the end state of the series.

Suggested change

* Scope: the components with a conformance report under

* `packages/mui-material/src/<Component>/accessibility.md`. Others onboard

* incrementally.

* Initial PR scope: `buttons` only. Other components onboard incrementally.

The new sentence could move to the layer that adds the first accessibility.md.

siriwatknp · 2026-08-10T10:23:06Z

 // test/regressions/demoMeta.ts
 { test: 'docs/data/material/components/alert/*', enabled: true, skipAssertions: ['color-contrast'] },
-{ test: 'docs/data/material/components/buttons/{BasicButtons,ColorButtons}', enabled: true },
+{ test: 'docs/data/material/components/buttons/{BasicButtons,ColorButtons}', enabled: true, assertions: 'all' },


This block is labelled // test/regressions/demoMeta.ts, so it reads as a copy of the file, but the real rule at demoMeta.ts:163 does not set assertions. Rules are last-match-wins with no inheritance, so what the rule literally contains matters.

Which one is intended? If buttons is meant to run in all mode, setting it on the real rule would make the docs correct and also make the new branch run in CI, which covers 3 as well.

siriwatknp · 2026-08-10T10:23:06Z


 ```ts
-{ test: 'docs/data/material/components/buttons/{BasicButtons,ColorButtons}', enabled: true }
+{ test: 'docs/data/material/components/buttons/{BasicButtons,ColorButtons}', enabled: true, assertions: 'all' }


Same as AGENTS.md — this example sets assertions: 'all' on buttons, but the real rule in demoMeta.ts omits it.

michelengelen force-pushed the a11y/stack/harness branch from a5a6319 to 3da7a80 Compare August 5, 2026 09:56

michelengelen added test accessibility a11y labels Aug 5, 2026

michelengelen force-pushed the a11y/stack/harness branch 2 times, most recently from 5dd3b74 to ca61860 Compare August 6, 2026 09:37

michelengelen force-pushed the a11y/stack/harness branch from ca61860 to 91ee8d7 Compare August 6, 2026 09:42

michelengelen force-pushed the a11y/stack/harness branch 2 times, most recently from dc0eab5 to e13e971 Compare August 6, 2026 10:43

michelengelen requested review from silviuaavram and siriwatknp August 7, 2026 10:02

michelengelen marked this pull request as ready for review August 7, 2026 10:03

michelengelen self-assigned this Aug 7, 2026

michelengelen force-pushed the a11y/stack/harness branch from e13e971 to fca727f Compare August 7, 2026 11:37

[test] Add an assertions mode to the axe regression harness

4475eb3

Lets a fixture assert every axe rule it exercises rather than only the CSS-dependent visual ones, which is what the per-component WCAG reports need.

michelengelen force-pushed the a11y/stack/harness branch from fca727f to 4475eb3 Compare August 7, 2026 13:00

siriwatknp reviewed Aug 11, 2026

View reviewed changes

michelengelen mentioned this pull request Aug 11, 2026

[test] Automate 2.4.7 Focus Visible #48945

Draft

Uh oh!

Uh oh!

Conversation

michelengelen commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

code-infra-dashboard Bot commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploy preview

Bundle size

Uh oh!

michelengelen commented Aug 6, 2026

Uh oh!

michelengelen commented Aug 6, 2026

Uh oh!

michelengelen commented Aug 7, 2026

Uh oh!

siriwatknp left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

michelengelen commented Aug 4, 2026 •

edited

Loading

code-infra-dashboard Bot commented Aug 4, 2026 •

edited

Loading