Skip to content

feat(audit): bounded real-site runs, page profiles, blocked_or_empty, false-positive fixes - #76

Merged
changeroa merged 2 commits into
mainfrom
feat/audit-realsites
Oct 19, 2026
Merged

changeroa merged 2 commits into
mainfrom
feat/audit-realsites

Conversation

@changeroa

Copy link
Copy Markdown
Owner

What

stylegallery audit check on live sites, from the 2026-10-19 run on 12 homepages (6 timed out at 420 s, the other 6 had 5-10 failures each, several of them false positives).

  1. Bounded runs: --timeout <s> (default 120). Every viewport pass, hash/short-viewport/offline/no-script scenario, and the three slop stages get a weighted share of the remaining time. An overrunning stage has its contexts closed, and the report carries timed_out, verdict: "partial", status: "timed_out" per unfinished check, and stages timings. A hard browser close at budget + 15 s is the last resort.
    • Profiled causes: a fresh page load per CTA (GitHub: 80+), 5 s hover timeouts on hidden controls, up to 6 s of settle at every scroll stop on never-idle pages, and the resting-style array re-sent on every Tab press.
    • Fixes: live pages use domcontentloaded plus a bounded load wait, a 1 s settle cap, and a 3 s font cap. CTAs share one context and are capped at 8, hovered controls at 24 (evenly sampled), and the Tab walk at 120. Resting styles stay in the page.
  2. False positives removed
    • Hidden, off-page, zero-size, clipped, aria-hidden, and inert controls are skipped and counted in evidence.skipped. A hover or click that cannot reach its target is unreachable, not a violation.
    • Skip links, cookie/consent controls (by scope or name, including Korean), and legal links are not CTAs.
    • Observable effects now include aria-expanded/selected/pressed changes, changed rendered text, and scrolling (GOV.UK's feedback buttons).
    • slop.static-interactive and slop.sluggish-state-change no longer double count: when controls do not respond at all, sluggish keeps only its speed reason.
  3. Page profiles: --profile landing|app|content (default auto, from rendered signals). The catalog's new profiles object lists, with a rationale, the landing-calibrated rules that become warnings for app and content pages. Accessibility checks and honest actions stay errors everywhere. validate-anti-patterns checks the object (6 new negative cases).
  4. blocked_or_empty: an HTTP error, a challenge page, under 80 characters with no media, or an empty main after a bounded wait means the page is reported with its reason and signals, and no check runs.
  5. Regression gate: tests/fixtures/audit-realsites/ holds 7 from-scratch reproductions (no copied markup or text) mapped to the 12 sites. test:audit:realsites runs them 4 at a time in CI (audit-kit workflow). It asserts the true violations still fail, the fixed false positives do not, the profile resolves as expected, and runs stay inside their budget. audit:realsites:live re-runs the 12 live sites one at a time, outside CI.

Before / after (live)

Before: 0.1.17, 4 sites at a time, 420 s harness limit. After: this branch, one at a time, default 120 s budget. The full 12-site table follows in a comment once the sequential run finishes.

Site Before After
www.gov.uk 108 s, 6 fails incl. dishonest-cta "Reject additional cookies", "Skip to main content", empty-screen ×2 45 s, content profile, fail: audit.contrast 3.91 (kept); empty-screen and sluggish are warnings; dishonest-cta pass (7 consent, 1 skip, and 4 legal controls excluded)
github.com 306 s, 7 fails incl. dishonest-cta "Skip to content", static-interactive 9× hover_unreachable 86 s, landing, audit.focus w320 input (kept); hover_unreachable skipped
news.ycombinator.com 401 s, 5 fails incl. static-interactive + sluggish double count 48 s, content, fail: overflow w768 and contrast 3.54 (real); static-interactive and unfinished-surfaces are warnings, sluggish deduplicated
www.airbnb.com 47 s, 8 fails on an unrendered page 9 s, blocked_or_empty: main landmark rendered no text and no media

P4 shots: no rendered surface changes in this PR (audit tooling, catalog text, fixtures), so there is no before/after screenshot pair. The real-site table is the before/after evidence.

Verification

  • bun run test:audit 19/19, test:audit:self 23/23 (clean PASS / violation FAIL), test:audit:realsites 7/7, test:audit:parity 3/3 identical shared verdicts
  • node scripts/test-validate-anti-patterns.mjs 26 cases, validate-anti-patterns ok
  • bun run validate, validate:lifecycle-dispositions (caller inventories regenerated), validate:material (registry and embeddings regenerated), test:material, test:lifecycle-dispositions: all pass

consumer_reference: not_applicable - audit tooling selects no consumer reference record.

… false-positive fixes

audit check now ends within --timeout (default 120s) with partial results and per-stage status, detects pages whose content never rendered, resolves a landing/app/content profile that relaxes landing-only rules to warnings, skips hidden and non-CTA controls, and reports an unresponsive page once. Local reproductions of the 2026-10-19 12-site run gate CI.
…se portals are content

toss.im showed only its chrome and logo svg when the preflight read it, and was reported blocked_or_empty. The preflight now waits for 200 characters or a 120x80 media box. Naver's link text share of 0.81 with ad media now resolves to content.
@changeroa

Copy link
Copy Markdown
Owner Author

Full 12-site run (sequential, this branch at b6e24d7)

Site Before (0.1.17, 4 concurrent) After (this PR, sequential, 120 s budget) Failing (errors) Warnings (profile-relaxed) Timed-out checks
stripe.com timed out (420 s) 120.1 s, partial (landing) audit.focus, audit.reduced-motion, audit.contrast, audit.offline, audit.no-script, slop.soft-shadow-excess, slop.repeated-card-triplet, slop.static-interactive - audit.load, audit.overflow, audit.short-viewport, audit.link-transition, audit.empty-screen, slop.dishonest-cta
linear.app timed out (420 s) 104.7 s, fail (landing) audit.load, audit.focus, audit.reduced-motion, audit.contrast, audit.offline, audit.no-script, audit.empty-screen, slop.empty-screen, slop.gradient-area-excess, slop.soft-shadow-excess, slop.repeated-card-triplet, slop.static-interactive - -
www.apple.com timed out (420 s) 89.5 s, fail (landing) audit.focus, audit.no-script, slop.static-interactive - -
github.com 306 s, 7 fails 102.7 s, fail (landing) audit.focus, audit.offline, audit.no-script, slop.empty-screen, slop.gradient-area-excess, slop.static-interactive - -
toss.im 142 s, 8 fails 0.4 s, blocked_or_empty - - -
www.naver.com timed out (420 s) 73.6 s, fail (landing) audit.overflow, audit.focus, audit.contrast, audit.hash-navigation, audit.no-script, audit.empty-screen, slop.unfinished-surfaces, slop.static-interactive, slop.dishonest-cta - -
www.notion.com timed out (420 s) 102.2 s, fail (landing) audit.focus, audit.reduced-motion, audit.contrast, audit.empty-screen, slop.empty-screen, slop.static-interactive, slop.oversized-display - -
vercel.com 205 s, 10 fails 120.1 s, partial (landing) audit.load, audit.focus, audit.reduced-motion, audit.contrast, audit.offline, slop.headline-only-gradient-screen, slop.empty-screen, slop.oversized-display - audit.overflow, audit.short-viewport, audit.link-transition, audit.empty-screen, slop.static-interactive, slop.dishonest-cta
www.gov.uk 108 s, 6 fails 44.4 s, fail (content) audit.contrast audit.empty-screen, slop.empty-screen, slop.sluggish-state-change -
craigslist.org timed out (420 s) 120.1 s, partial (content) audit.focus, audit.contrast, audit.offline slop.unfinished-surfaces, slop.sluggish-state-change slop.dishonest-cta
www.airbnb.com 47 s, 8 fails 9 s, blocked_or_empty - - -
news.ycombinator.com 401 s, 5 fails 48.1 s, fail (content) audit.overflow, audit.contrast slop.unfinished-surfaces, slop.static-interactive -
  • Timeouts: 0 of 12 runs exceed the budget. Vercel and craigslist end as partial at 120 s with the unfinished checks named.
  • Kept true violations: GOV.UK audit.contrast 3.91 on "Search GOV.UK"; GitHub audit.focus on the w320 input without a focus indicator.
  • Removed false positives: no skip-link, cookie, or legal control is a dishonest CTA, and no hidden or unreachable control counts as static. GOV.UK and HN text-led screens are warnings, Airbnb is blocked_or_empty, and static-interactive and sluggish no longer double count.
  • Found by this run and fixed in cd1d462: toss.im was read before rendering (logo svg only) and misreported as blocked_or_empty, and Naver (link text share 0.81) resolved to landing. Re-verified in the final 0.1.18 measurement.

@changeroa
changeroa merged commit 686fc12 into main Oct 19, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant