Skip to content

📋 [SAMPLE] Nightly Guidance Health Audit: 2026-06-04 #932

Open
0 of 2 issues completed
Open
📋 [SAMPLE] Nightly Guidance Health Audit: 2026-06-04#932
0 of 2 issues completed
Labels
nightly-investigationNightly Guidance Health Audits

Description

@micahjo7
micahjo7
opened on Jun 4, 2026

Nightly Evaluation Investigation Report

Generated: 2026-06-04
Suites Investigated:

  • claude_code: nightly-2026-06-03_17-25-12-claude_code-micahjo
  • codex_cli: nightly-2026-06-03_19-35-43-codex_cli-micahjo
  • jetski_cli: nightly-2026-06-03_17-00-02-jetski_cli-micahjo

Summary of Flagged Tasks

  • Total distinct tasks analyzed: 129
  • Total flagged tasks: 32

Flagged Tasks Table

Task / Guide Flags Details
translator HIGH_UNGUIDED_PASS_RATE Unguided pass rate was 90% or higher in all nightly runs (claude_code: 100%, codex_cli: 100%, jetski_cli: 100%).
style-parent-with-has HIGH_UNGUIDED_PASS_RATE Unguided pass rate was 90% or higher in all nightly runs (claude_code: 100%, codex_cli: 100%, jetski_cli: 100%).
stabilize-reactive-state MISSING_EXPECTED_GUIDE Expected guide was missing in 2 out of 3 agent runs.
precise-text-alignment HIGH_UNGUIDED_PASS_RATE Unguided pass rate was 90% or higher in all nightly runs (claude_code: 100%, codex_cli: 100%, jetski_cli: 100%).
position-aware-tooltips LOW_GUIDED_PASS_RATE Guided pass rate was under 75% in all nightly runs (claude_code: 70%, codex_cli: 60%, jetski_cli: 70%).
platform-controls-dismiss-dialog TOO_MANY_GUIDES_CONSUMED Consumed 3 or more guides in 2 out of 3 agent runs.
persistent-toast-notifications HIGH_UNGUIDED_PASS_RATE Unguided pass rate was 90% or higher in all nightly runs (claude_code: 100%, codex_cli: 100%, jetski_cli: 100%).
performance TOO_MANY_GUIDES_CONSUMED Consumed 3 or more guides in 3 out of 3 agent runs.
passkey-registration LOW_GUIDED_PASS_RATE Guided pass rate was under 75% in all nightly runs (claude_code: 0%, codex_cli: 60%, jetski_cli: 0%).
passkey-management MISSING_EXPECTED_GUIDE, LOW_GUIDED_PASS_RATE Expected guide was missing in 2 out of 3 agent runs.
Guided pass rate was under 75% in all nightly runs (claude_code: 50%, codex_cli: 50%, jetski_cli: 50%).
passkey-conditional-create MISSING_EXPECTED_GUIDE Expected guide was missing in 2 out of 3 agent runs.
passkey-authentication LOW_GUIDED_PASS_RATE Guided pass rate was under 75% in all nightly runs (claude_code: 67%, codex_cli: 67%, jetski_cli: 17%).
optimize-preload-priority TOO_MANY_GUIDES_CONSUMED Consumed 3 or more guides in 2 out of 3 agent runs.
move-dom-element-without-losing-state HIGH_UNGUIDED_PASS_RATE Unguided pass rate was 90% or higher in all nightly runs (claude_code: 100%, codex_cli: 100%, jetski_cli: 100%).
model-partial-time-concepts MISSING_EXPECTED_GUIDE Expected guide was missing in 3 out of 3 agent runs.
manage-recurring-intervals MISSING_EXPECTED_GUIDE, LOW_GUIDED_PASS_RATE Expected guide was missing in 3 out of 3 agent runs.
Guided pass rate was under 75% in all nightly runs (claude_code: 9%, codex_cli: 9%, jetski_cli: 27%).
language-model LOW_GUIDED_PASS_RATE Guided pass rate was under 75% in all nightly runs (claude_code: 31%, codex_cli: 31%, jetski_cli: 31%).
interactive-content-reveal HIGH_UNGUIDED_PASS_RATE Unguided pass rate was 90% or higher in all nightly runs (claude_code: 94%, codex_cli: 94%, jetski_cli: 100%).
improve-text-layout-and-legibility HIGH_UNGUIDED_PASS_RATE Unguided pass rate was 90% or higher in all nightly runs (claude_code: 100%, codex_cli: 100%, jetski_cli: 100%).
forms TOO_MANY_GUIDES_CONSUMED Consumed 3 or more guides in 3 out of 3 agent runs.
deliver-optimized-decorative-images HIGH_UNGUIDED_PASS_RATE Unguided pass rate was 90% or higher in all nightly runs (claude_code: 100%, codex_cli: 100%, jetski_cli: 100%).
declarative-button-actions HIGH_UNGUIDED_PASS_RATE Unguided pass rate was 90% or higher in all nightly runs (claude_code: 100%, codex_cli: 100%, jetski_cli: 100%).
dark-mode TOO_MANY_GUIDES_CONSUMED Consumed 3 or more guides in 3 out of 3 agent runs.
content-based-styling HIGH_UNGUIDED_PASS_RATE Unguided pass rate was 90% or higher in all nightly runs (claude_code: 100%, codex_cli: 100%, jetski_cli: 100%).
conditional-async-dependencies MISSING_EXPECTED_GUIDE Expected guide was missing in 2 out of 3 agent runs.
child-state-based-styling HIGH_UNGUIDED_PASS_RATE Unguided pass rate was 90% or higher in all nightly runs (claude_code: 100%, codex_cli: 100%, jetski_cli: 100%).
calculate-total-foreground-time LOW_GUIDED_PASS_RATE Guided pass rate was under 75% in all nightly runs (claude_code: 0%, codex_cli: 0%, jetski_cli: 0%).
autofill-sign-in-form LOW_GUIDED_PASS_RATE Guided pass rate was under 75% in all nightly runs (claude_code: 0%, codex_cli: 0%, jetski_cli: 0%).
autofill-highlight-inputs HIGH_UNGUIDED_PASS_RATE Unguided pass rate was 90% or higher in all nightly runs (claude_code: 100%, codex_cli: 100%, jetski_cli: 100%).
apply-webgl-shaders LOW_GUIDED_PASS_RATE Guided pass rate was under 75% in all nightly runs (claude_code: 57%, codex_cli: 29%, jetski_cli: 57%).
animate-element-entry-exit LOW_GUIDED_PASS_RATE Guided pass rate was under 75% in all nightly runs (claude_code: 63%, codex_cli: 63%, jetski_cli: 63%).
adapt-scrollbar-to-contrast-preferences LOW_GUIDED_PASS_RATE Guided pass rate was under 75% in all nightly runs (claude_code: 67%, codex_cli: 67%, jetski_cli: 67%).

Flagged Tasks Details

translator

Flags:

  • HIGH_UNGUIDED_PASS_RATE: Unguided pass rate was 90% or higher in all nightly runs (claude_code: 100%, codex_cli: 100%, jetski_cli: 100%).

Run Details:

Agent Guided Pass Rate Unguided Pass Rate Guides Consumed
claude_code 100% 100% translator
codex_cli 100% 100% translator
jetski_cli 17% 100% None

Qualitative Diagnostic Summary:

  • Diagnostic Details:
    • HIGH_UNGUIDED_PASS_RATE: The Translator API is a standardized web API. Modern LLMs already possess strong knowledge of this API (including window.Translator), which allows them to implement the translation buttons, availability checks, and progress monitoring without guide assistance, resulting in 100% pass rates on the unguided runs.

style-parent-with-has

Flags:

  • HIGH_UNGUIDED_PASS_RATE: Unguided pass rate was 90% or higher in all nightly runs (claude_code: 100%, codex_cli: 100%, jetski_cli: 100%).

Run Details:

Agent Guided Pass Rate Unguided Pass Rate Guides Consumed
claude_code 100% 100% style-parent-with-has, validate-input-after-interaction
codex_cli 100% 100% style-parent-with-has, validate-input-after-interaction
jetski_cli 100% 100% style-parent-with-has

Qualitative Diagnostic Summary:

  • Diagnostic Details:
    • HIGH_UNGUIDED_PASS_RATE: The :has(:user-invalid) selector is standard CSS for parent-validation styling. Modern agents natively write this CSS. In addition, the fallback test suite has a major loophole: it stubs CSS.supports to return false to simulate fallback mode, but Chromium still natively parses and applies :has(:user-invalid) selector rules. The agent passes the fallback test suite even without writing any fallback JavaScript.

stabilize-reactive-state

Flags:

  • MISSING_EXPECTED_GUIDE: Expected guide was missing in 2 out of 3 agent runs.

Run Details:

Agent Guided Pass Rate Unguided Pass Rate Guides Consumed
claude_code 100% 14% manage-recurring-intervals, coordinate-global-events
codex_cli 29% 29% stabilize-reactive-state
jetski_cli 100% 29% capture-location-agnostic-data, calculate-event-differentials

Qualitative Diagnostic Summary:

  • Diagnostic Details:
    • MISSING_EXPECTED_GUIDE: The guide's frontmatter description is abstract and lacks technical keywords such as "Temporal", "PlainDateTime", "reactivity", "immutability", or "Date". When agents search for "Temporal API" or "PlainDateTime", semantic search retrieves other more keyword-dense Temporal/Date guides (like manage-recurring-intervals), missing this guide entirely.

precise-text-alignment

Flags:

  • HIGH_UNGUIDED_PASS_RATE: Unguided pass rate was 90% or higher in all nightly runs (claude_code: 100%, codex_cli: 100%, jetski_cli: 100%).

Run Details:

Agent Guided Pass Rate Unguided Pass Rate Guides Consumed
claude_code 100% 100% precise-text-alignment
codex_cli 100% 100% precise-text-alignment
jetski_cli 100% 100% precise-text-alignment

Qualitative Diagnostic Summary:

  • Diagnostic Details:
    • HIGH_UNGUIDED_PASS_RATE: The task prompt explicitly directs the agent to "use text-box-trim to make the titles sit perfectly in the center". Because modern LLMs are fully trained on newer CSS specifications, they natively generate the correct properties (text-box-trim: both and text-box-edge: cap alphabetic), completely bypassing the need for guided documentation.

position-aware-tooltips

Flags:

  • LOW_GUIDED_PASS_RATE: Guided pass rate was under 75% in all nightly runs (claude_code: 70%, codex_cli: 60%, jetski_cli: 70%).

Run Details:

Agent Guided Pass Rate Unguided Pass Rate Guides Consumed
claude_code 70% 40% position-aware-tooltips
codex_cli 60% 70% position-aware-tooltips
jetski_cli 70% 70% position-aware-tooltips

Qualitative Diagnostic Summary:

  • Diagnostic Details:
    • Failed Assertions:
      • "conditionally polyfills the Popover API" (Test ID: 410c9cb569428846eeba-8a42b14913e76005e061)
      • "uses pseudo-elements for decorative arrows" (Test ID: 410c9cb569428846eeba-f9249efeaa4a1c5733fa)
      • "updates the visual styling and layout when the tooltip flips to a fallback position purely via CSS without JS" (Test ID: 410c9cb569428846eeba-705d1fd1fed83abbd3b6)
    • LOW_GUIDED_PASS_RATE: Guided runs are failing 3 key tests. Popover Polyfill fails because the guide is long and agents skip polyfill scripting. Decorative arrows test fails because it rejects standard empty content (content: "") used in clip-path arrows. The layout flip test fails because short pages clamp scrolling bounds, preventing the tooltip from actually flipping to fallback positions during evaluation.

platform-controls-dismiss-dialog

Flags:

  • TOO_MANY_GUIDES_CONSUMED: Consumed 3 or more guides in 2 out of 3 agent runs.

Run Details:

Agent Guided Pass Rate Unguided Pass Rate Guides Consumed
claude_code 86% 71% light-dismiss-a-dialog, platform-controls-dismiss-dialog
codex_cli 86% 86% light-dismiss-a-dialog, platform-controls-dismiss-dialog, accessibility
jetski_cli 100% 14% light-dismiss-a-dialog, platform-controls-dismiss-dialog, declarative-dialog-popover-control

Qualitative Diagnostic Summary:

  • Diagnostic Details:
    • TOO_MANY_GUIDES_CONSUMED: The prompt requires a dialog that dismisses via both clicking outside (light-dismiss) and pressing escape. Because the guides light-dismiss-a-dialog and platform-controls-dismiss-dialog both cover the same dialog-closedby API surface and overlap in target implementation guidelines, the agent retrieves both guides (often along with generic CSS/accessibility guides), leading to exceeding the guide consumption threshold.

persistent-toast-notifications

Flags:

  • HIGH_UNGUIDED_PASS_RATE: Unguided pass rate was 90% or higher in all nightly runs (claude_code: 100%, codex_cli: 100%, jetski_cli: 100%).

Run Details:

Agent Guided Pass Rate Unguided Pass Rate Guides Consumed
claude_code 100% 100% persistent-toast-notifications, animate-to-from-top-layer, dynamic-sibling-styling
codex_cli 100% 100% persistent-toast-notifications, animate-to-from-top-layer
jetski_cli 100% 100% persistent-toast-notifications, animate-to-from-top-layer

Qualitative Diagnostic Summary:

  • Diagnostic Details:
    • HIGH_UNGUIDED_PASS_RATE: The prompt is highly prescriptive (telling the agent to avoid z-index hacks, use top-layer elements, and dynamically offset the stacking position using purely CSS without wrappers). This directly gives away the CSS sibling selector technique (.toast ~ .toast), allowing the unguided runs to pass 100% easily using standard CSS layout capabilities.

performance

Flags:

  • TOO_MANY_GUIDES_CONSUMED: Consumed 3 or more guides in 3 out of 3 agent runs.

Run Details:

Agent Guided Pass Rate Unguided Pass Rate Guides Consumed
claude_code 93% 80% performance, optimize-image-priority, html, accessibility, deliver-optimized-decorative-images
codex_cli 100% 93% accessibility, performance, html
jetski_cli 73% 60% deliver-optimized-decorative-images, optimize-image-priority, performance, html

Qualitative Diagnostic Summary:

  • Diagnostic Details:
    • TOO_MANY_GUIDES_CONSUMED: The prompt is a very broad instruction ("Perform a comprehensive audit and optimization of the site"). To perform a comprehensive optimization, the agent naturally retrieves multiple specialized guides related to performance (e.g. image optimization, script priority, view transitions, and basic layout performance), exceeding the 3-guide threshold.

passkey-registration

Flags:

  • LOW_GUIDED_PASS_RATE: Guided pass rate was under 75% in all nightly runs (claude_code: 0%, codex_cli: 60%, jetski_cli: 0%).

Run Details:

Agent Guided Pass Rate Unguided Pass Rate Guides Consumed
claude_code 0% 0% passkey-registration
codex_cli 60% 0% passkey-registration
jetski_cli 0% 0% None

Qualitative Diagnostic Summary:

  • Diagnostic Details:
    • Failed Assertions:
      • "detects support using platform authenticator prior to prompting UI" (Test ID: a1190142b3a412901a26-f8411b5a3cf1e99c5087)
      • "signals signalUnknownCredential using Base64URL credentialId upon server verification failure" (Test ID: a1190142b3a412901a26-be10f4aab7a0be0c52e6)
      • LOW_GUIDED_PASS_RATE: The client-side capability check in the guide checks !capabilities.passkeyPlatformAuthenticator || !capabilities.conditionalGet to gate registration. However, conditionalGet (autofill) is only relevant for authentication, not registration. The grader's mock for getClientCapabilities only registers conditionalCreate: true and does not set conditionalGet. Consequently, agents following the guide check conditionalGet, find it undefined, and hide the registration button entirely, causing all subsequent tests to fail.

passkey-management

Flags:

  • MISSING_EXPECTED_GUIDE: Expected guide was missing in 2 out of 3 agent runs.
  • LOW_GUIDED_PASS_RATE: Guided pass rate was under 75% in all nightly runs (claude_code: 50%, codex_cli: 50%, jetski_cli: 50%).

Run Details:

Agent Guided Pass Rate Unguided Pass Rate Guides Consumed
claude_code 50% 50% passkey-management
codex_cli 50% 50% None
jetski_cli 50% 50% None

Qualitative Diagnostic Summary:

  • Diagnostic Details:
    • Failed Assertions:
      • "invokes signalAllAcceptedCredentials on DOMContentLoaded load" (Test ID: 1070b32ac3d6381c7415-8e3cdd25b9e0307dc28c)
      • "invokes signalAllAcceptedCredentials upon credentials deletion triggers" (Test ID: 1070b32ac3d6381c7415-4b93b71d5b49cd70f1f6)
      • "invokes signalCurrentUserDetails upon credential rename" (Test ID: 1070b32ac3d6381c7415-36bf778edbb8148429e7)
      • MISSING_EXPECTED_GUIDE: The guide's frontmatter description is abstract and lacks key terms like "rename", "delete", "credentials list", or "AAGUID".
      • LOW_GUIDED_PASS_RATE: The grader overwrites window.PublicKeyCredential using Object.defineProperty on page load. However, the client loads a polyfill (webauthn-polyfills) which overrides PublicKeyCredential again, wiping out the grader's mock of signalAllAcceptedCredentials and signalCurrentUserDetails, causing tests checking these calls to fail.

passkey-conditional-create

Flags:

  • MISSING_EXPECTED_GUIDE: Expected guide was missing in 2 out of 3 agent runs.

Run Details:

Agent Guided Pass Rate Unguided Pass Rate Guides Consumed
claude_code 17% 17% None
codex_cli 83% 33% passkey-conditional-create, passkey-registration
jetski_cli 17% 17% None

Qualitative Diagnostic Summary:

  • Diagnostic Details:
    • MISSING_EXPECTED_GUIDE: The guide description lacks keywords like "silent passkey enrollment" or "background registration".

passkey-authentication

Flags:

  • LOW_GUIDED_PASS_RATE: Guided pass rate was under 75% in all nightly runs (claude_code: 67%, codex_cli: 67%, jetski_cli: 17%).

Run Details:

Agent Guided Pass Rate Unguided Pass Rate Guides Consumed
claude_code 67% 17% passkey-authentication, passkeys
codex_cli 67% 0% passkey-authentication, passkeys
jetski_cli 17% 33% passkey-authentication

Qualitative Diagnostic Summary:

  • Diagnostic Details:
    • Failed Assertions:
      • "A successful credential verification updates the UI status to indicate successful authentication and session establishment" (Test ID: 6b8d7284564fb9f08e57-1c9d397f970458446cdc)
      • LOW_GUIDED_PASS_RATE: If the agent fails to properly abort the pending conditional UI request (conditional-get) before triggering the explicit biometric prompt on button click, Chromium throws a NotAllowedError: A request is already in progress. This causes the login attempt to fail and prevents the status from updating to "Authenticated Successfully!".

optimize-preload-priority

Flags:

  • TOO_MANY_GUIDES_CONSUMED: Consumed 3 or more guides in 2 out of 3 agent runs.

Run Details:

Agent Guided Pass Rate Unguided Pass Rate Guides Consumed
claude_code 100% 83% optimize-image-priority, optimize-preload-priority, performance
codex_cli 83% 83% optimize-preload-priority, optimize-image-priority
jetski_cli 100% 83% optimize-preload-priority, optimize-image-priority, performance

Qualitative Diagnostic Summary:

  • Diagnostic Details:
    • TOO_MANY_GUIDES_CONSUMED: Significant query overlap between optimize-image-priority and optimize-preload-priority.
    • HIGH_UNGUIDED_PASS_RATE: The prompt is overly prescriptive, detailing exactly which assets are critical (LCP image, brand font) and which are not.

move-dom-element-without-losing-state

Flags:

  • HIGH_UNGUIDED_PASS_RATE: Unguided pass rate was 90% or higher in all nightly runs (claude_code: 100%, codex_cli: 100%, jetski_cli: 100%).

Run Details:

Agent Guided Pass Rate Unguided Pass Rate Guides Consumed
claude_code 100% 100% move-dom-element-without-losing-state
codex_cli 75% 100% move-dom-element-without-losing-state
jetski_cli 100% 100% move-dom-element-without-losing-state

Qualitative Diagnostic Summary:

  • Diagnostic Details:
    • HIGH_UNGUIDED_PASS_RATE: The grader uses simple static regex checks for moveBefore and fallback methods, which are easily satisfied by agents writing boilerplate code.

model-partial-time-concepts

Flags:

  • MISSING_EXPECTED_GUIDE: Expected guide was missing in 3 out of 3 agent runs.

Run Details:

Agent Guided Pass Rate Unguided Pass Rate Guides Consumed
claude_code 86% 71% calculate-event-differentials
codex_cli 71% 71% calculate-event-differentials
jetski_cli 71% 71% None

Qualitative Diagnostic Summary:

  • Diagnostic Details:
    • MISSING_EXPECTED_GUIDE: Abstract description lacks terms like "birthday", "credit card expiration", or "store hours".
    • HIGH_UNGUIDED_PASS_RATE: The polyfill verification tests in grader.ts pass vacuously if no polyfill is loaded because the browser environment natively has a mocked/supported Temporal object.

manage-recurring-intervals

Flags:

  • MISSING_EXPECTED_GUIDE: Expected guide was missing in 3 out of 3 agent runs.
  • LOW_GUIDED_PASS_RATE: Guided pass rate was under 75% in all nightly runs (claude_code: 9%, codex_cli: 9%, jetski_cli: 27%).

Run Details:

Agent Guided Pass Rate Unguided Pass Rate Guides Consumed
claude_code 9% 9% None
codex_cli 9% 9% validate-input-after-interaction
jetski_cli 27% 27% None

Qualitative Diagnostic Summary:

  • Diagnostic Details:
    • Failed Assertions:
      • "should feature-detect Temporal via typeof Temporal" (Test ID: 5b2af8be7a680402a5ad-327261817190cc9345d6)
      • "should not load polyfill if native Temporal is present" (Test ID: 5b2af8be7a680402a5ad-a9834d793296d25be84c)
      • "should load polyfill when native Temporal is absent" (Test ID: 5b2af8be7a680402a5ad-d02d6947bc22f2acb453)
      • "should manually assign loaded polyfill to globalThis.Temporal" (Test ID: 5b2af8be7a680402a5ad-c94078a613ced3716c65)
      • "should use Temporal.PlainDate as primary type for calculating intervals" (Test ID: 5b2af8be7a680402a5ad-eddefdb347c43c8ebc5a)
      • "should use Temporal.PlainDate.from to parse starting date" (Test ID: 5b2af8be7a680402a5ad-5a234fd9e1ee3321b1b2)
      • "should use .add() on PlainDate instance to calculate next interval date" (Test ID: 5b2af8be7a680402a5ad-549f767c40a42bcbd442)
      • "should use new Temporal instance returned by operations like add" (Test ID: 5b2af8be7a680402a5ad-a35c2879d4f49f6bfaa4)
      • MISSING_EXPECTED_GUIDE: The guide description references abstract billing/payroll concepts instead of date math or delivery planning.
      • LOW_GUIDED_PASS_RATE: Since the guide is not retrieved, agents fail to implement feature-detection, polyfilling, and proper Temporal calculations required by the tests.

language-model

Flags:

  • LOW_GUIDED_PASS_RATE: Guided pass rate was under 75% in all nightly runs (claude_code: 31%, codex_cli: 31%, jetski_cli: 31%).

Run Details:

Agent Guided Pass Rate Unguided Pass Rate Guides Consumed
claude_code 31% 31% language-model
codex_cli 31% 31% language-model
jetski_cli 31% 31% None

Qualitative Diagnostic Summary:

  • Diagnostic Details:
    • Failed Assertions:
      • "1. LanguageModel.create() should be called using window.LanguageModel" (Test ID: 92d4b3db223c306b02e3-44b3c5af59a3f9a34ec1)
      • "3. LanguageModel.availability() should be called before attempting to create a session" (Test ID: 92d4b3db223c306b02e3-08e0a520ca1fa49dee3b)
      • "6. LanguageModel.create() should register a downloadprogress listener via monitor" (Test ID: 92d4b3db223c306b02e3-1dbbafb1454270096d20)
      • "7. session.promptStreaming() should be used with for-await and not onmessage" (Test ID: 92d4b3db223c306b02e3-0bfbec1c54189e1252b1)
      • "8. session.prompt() should be used for one-shot responses on a modern session" (Test ID: 92d4b3db223c306b02e3-4e5ca79d0521f3e613d9)
      • "10. For structured output, responseConstraint option should be passed a JSON Schema" (Test ID: 92d4b3db223c306b02e3-01b82f748b762fe0a8c6)
      • "11. Result of session.prompt() with responseConstraint should be parsed with JSON.parse()" (Test ID: 92d4b3db223c306b02e3-7f88a91ca6f64788f0d0)
      • "12. JSON.stringify() on schema not needed" (Test ID: 92d4b3db223c306b02e3-60bfde43220f72545197)
      • "13. session.destroy() should be called when a session is no longer needed" (Test ID: 92d4b3db223c306b02e3-42e6c175e3f9c91d7895)
      • "15. session.clone() usage and base destruction" (Test ID: 92d4b3db223c306b02e3-0dfbf57be80eb8038388)
      • "16. contextUsage and contextWindow should be used" (Test ID: 92d4b3db223c306b02e3-3c4a7fa32133ff50ac20)
      • LOW_GUIDED_PASS_RATE: Mismatch between the task prompt (which asks to build specific features like #ask-the-barista, feedback categorization, and image uploads) and the grader (which asserts the structure of demo.html with buttons like #sentiment-btn and #clone-btn).

interactive-content-reveal

Flags:

  • HIGH_UNGUIDED_PASS_RATE: Unguided pass rate was 90% or higher in all nightly runs (claude_code: 94%, codex_cli: 94%, jetski_cli: 100%).

Run Details:

Agent Guided Pass Rate Unguided Pass Rate Guides Consumed
claude_code 100% 94% interactive-content-reveal
codex_cli 100% 94% interactive-content-reveal
jetski_cli 100% 100% interactive-content-reveal

Qualitative Diagnostic Summary:

  • Diagnostic Details:
    • HIGH_UNGUIDED_PASS_RATE: Extremely prescriptive prompt detailing the exact CSS custom variables (--mouse-x, --mouse-y, etc.) and transitions.

improve-text-layout-and-legibility

Flags:

  • HIGH_UNGUIDED_PASS_RATE: Unguided pass rate was 90% or higher in all nightly runs (claude_code: 100%, codex_cli: 100%, jetski_cli: 100%).

Run Details:

Agent Guided Pass Rate Unguided Pass Rate Guides Consumed
claude_code 100% 100% improve-text-layout-and-legibility
codex_cli 100% 100% improve-text-layout-and-legibility
jetski_cli 100% 100% improve-text-layout-and-legibility

Qualitative Diagnostic Summary:

  • Diagnostic Details:
    • HIGH_UNGUIDED_PASS_RATE: Standard text-wrap: balance and pretty properties are easily written by modern models.

forms

Flags:

  • TOO_MANY_GUIDES_CONSUMED: Consumed 3 or more guides in 3 out of 3 agent runs.

Run Details:

Agent Guided Pass Rate Unguided Pass Rate Guides Consumed
claude_code 79% 67% forms, accessibility, accessible-error-announcement, validate-input-after-interaction
codex_cli 79% 64% forms, autofill-sign-up-form, validate-input-after-interaction
jetski_cli 79% 61% forms, accessibility, autofill-sign-up-form

Qualitative Diagnostic Summary:

  • Diagnostic Details:
    • TOO_MANY_GUIDES_CONSUMED: The prompt is a massive audit and multi-step creation task, causing the agent to fetch all forms/accessibility guides.

deliver-optimized-decorative-images

Flags:

  • HIGH_UNGUIDED_PASS_RATE: Unguided pass rate was 90% or higher in all nightly runs (claude_code: 100%, codex_cli: 100%, jetski_cli: 100%).

Run Details:

Agent Guided Pass Rate Unguided Pass Rate Guides Consumed
claude_code 100% 100% deliver-optimized-decorative-images
codex_cli 100% 100% deliver-optimized-decorative-images
jetski_cli 0% 100% None

Qualitative Diagnostic Summary:

  • Diagnostic Details:
    • HIGH_UNGUIDED_PASS_RATE: Prompt lists the exact image filenames, formats, densities, and the image-set() function.

declarative-button-actions

Flags:

  • HIGH_UNGUIDED_PASS_RATE: Unguided pass rate was 90% or higher in all nightly runs (claude_code: 100%, codex_cli: 100%, jetski_cli: 100%).

Run Details:

Agent Guided Pass Rate Unguided Pass Rate Guides Consumed
claude_code 100% 100% declarative-button-actions
codex_cli 100% 100% declarative-button-actions
jetski_cli 100% 100% declarative-button-actions

Qualitative Diagnostic Summary:

  • Diagnostic Details:
    • HIGH_UNGUIDED_PASS_RATE: Prompt defines the exact commands (--spin, etc.) and attributes.

dark-mode

Flags:

  • TOO_MANY_GUIDES_CONSUMED: Consumed 3 or more guides in 3 out of 3 agent runs.

Run Details:

Agent Guided Pass Rate Unguided Pass Rate Guides Consumed
claude_code 94% 63% dark-mode, component-specific-light-dark-theme, customize-scrollbar-color-and-thickness
codex_cli 75% 63% dark-mode, component-specific-light-dark-theme, customize-scrollbar-color-and-thickness
jetski_cli 81% 63% dark-mode, customize-scrollbar-color-and-thickness, component-specific-light-dark-theme

Qualitative Diagnostic Summary:

  • Diagnostic Details:
    • TOO_MANY_GUIDES_CONSUMED: Prompt combines general dark mode, custom scrollbars, and nested overrides, triggering multiple guides.

content-based-styling

Flags:

  • HIGH_UNGUIDED_PASS_RATE: Unguided pass rate was 90% or higher in all nightly runs (claude_code: 100%, codex_cli: 100%, jetski_cli: 100%).

Run Details:

Agent Guided Pass Rate Unguided Pass Rate Guides Consumed
claude_code 100% 100% content-based-styling
codex_cli 100% 100% content-based-styling
jetski_cli 100% 100% content-based-styling

Qualitative Diagnostic Summary:

  • Diagnostic Details:
    • HIGH_UNGUIDED_PASS_RATE: Modern models write :has() checks natively for child content.

conditional-async-dependencies

Flags:

  • MISSING_EXPECTED_GUIDE: Expected guide was missing in 2 out of 3 agent runs.

Run Details:

Agent Guided Pass Rate Unguided Pass Rate Guides Consumed
claude_code 86% 71% declarative-dialog-popover-control
codex_cli 86% 57% declarative-dialog-popover-control, conditional-async-dependencies
jetski_cli 86% 14% declarative-dialog-popover-control, html

Qualitative Diagnostic Summary:

  • Diagnostic Details:
    • MISSING_EXPECTED_GUIDE: Description is too generic.

child-state-based-styling

Flags:

  • HIGH_UNGUIDED_PASS_RATE: Unguided pass rate was 90% or higher in all nightly runs (claude_code: 100%, codex_cli: 100%, jetski_cli: 100%).

Run Details:

Agent Guided Pass Rate Unguided Pass Rate Guides Consumed
claude_code 100% 100% child-state-based-styling
codex_cli 100% 100% child-state-based-styling, brand-consistent-forms
jetski_cli 100% 100% child-state-based-styling

Qualitative Diagnostic Summary:

  • Diagnostic Details:
    • HIGH_UNGUIDED_PASS_RATE: Models write parent state-based :has() rules natively.

calculate-total-foreground-time

Flags:

  • LOW_GUIDED_PASS_RATE: Guided pass rate was under 75% in all nightly runs (claude_code: 0%, codex_cli: 0%, jetski_cli: 0%).

Run Details:

Agent Guided Pass Rate Unguided Pass Rate Guides Consumed
claude_code 0% 0% calculate-total-foreground-time
codex_cli 0% 0% calculate-total-foreground-time
jetski_cli 0% 0% None

Qualitative Diagnostic Summary:

  • Diagnostic Details:
    • Failed Assertions:
      • "should use VisibilityStateEntry API (via getEntriesByType or PerformanceObserver)" (Test ID: dd9974d8cb2863beb580-6c03bd112c1260a21cbd)
      • "should correctly calculate foreground time excluding hidden periods" (Test ID: dd9974d8cb2863beb580-799fa8072fdfd4281b9f)
      • "should check for VisibilityStateEntry support before falling back" (Test ID: dd9974d8cb2863beb580-6a24a7833f2fd4b62e6e)
      • LOW_GUIDED_PASS_RATE: The grader mocks PerformanceObserver but does not add 'visibility-state' to the mocked supportedEntryTypes list or mock window.VisibilityStateEntry. Capability checks fail and fallback to visibilitychange, failing the tests.

autofill-sign-in-form

Flags:

  • LOW_GUIDED_PASS_RATE: Guided pass rate was under 75% in all nightly runs (claude_code: 0%, codex_cli: 0%, jetski_cli: 0%).

Run Details:

Agent Guided Pass Rate Unguided Pass Rate Guides Consumed
claude_code 0% 0% autofill-sign-in-form
codex_cli 0% 0% autofill-sign-in-form
jetski_cli 0% 93% autofill-sign-in-form

Qualitative Diagnostic Summary:

  • Diagnostic Details:
    • Failed Assertions:
      • "All sign-in inputs must be within a element" (Test ID: de1be34cbc2954fc16ae-3b8c119984a858bc8db7)
      • "The form must have a submit button" (Test ID: de1be34cbc2954fc16ae-1ccc61dee53120f2a169)
      • "Every input in the form must have an associated label" (Test ID: de1be34cbc2954fc16ae-ad7b3d29636af80f2320)
      • "Labels must have a "for" attribute matching an input "id"" (Test ID: de1be34cbc2954fc16ae-b406a67f8d6ac88858e5)
      • "No element must use autocomplete="off"" (Test ID: de1be34cbc2954fc16ae-5f9fecabe53490078e41)
      • "Email input must have type="email"" (Test ID: de1be34cbc2954fc16ae-65eb4449b4434d68fdad)
      • "Email input must have autocomplete="username"" (Test ID: de1be34cbc2954fc16ae-d757e1280f73765e58e3)
      • "Password input must have type="password"" (Test ID: de1be34cbc2954fc16ae-422015f1c6b2ec5895f0)
      • "Password input must have autocomplete="current-password"" (Test ID: de1be34cbc2954fc16ae-a84c5d3054055e77147c)
      • "Password input must have id="current-password"" (Test ID: de1be34cbc2954fc16ae-ee54ab6735b0c09e3474)
      • "Email input must be required" (Test ID: de1be34cbc2954fc16ae-1961c1f8e141fcfde81d)
      • "Password input must be required" (Test ID: de1be34cbc2954fc16ae-de9edae6ce5ab74c371c)
      • "There must be exactly one email input" (Test ID: de1be34cbc2954fc16ae-c6ccb3d6466bd56bf29c)
      • "There must be exactly one password input" (Test ID: de1be34cbc2954fc16ae-2d11a0fd01dc8d0039ff)
      • LOW_GUIDED_PASS_RATE: The guide contains plain HTML snippets. When the agent copies or adapts them into SignInForm.tsx (a React TSX file), they introduce syntax errors (like missing state bindings or invalid JSX), causing Astro compile/render crashes.

autofill-highlight-inputs

Flags:

  • HIGH_UNGUIDED_PASS_RATE: Unguided pass rate was 90% or higher in all nightly runs (claude_code: 100%, codex_cli: 100%, jetski_cli: 100%).

Run Details:

Agent Guided Pass Rate Unguided Pass Rate Guides Consumed
claude_code 100% 100% autofill-highlight-inputs
codex_cli 100% 100% autofill-highlight-inputs
jetski_cli 100% 100% autofill-highlight-inputs

Qualitative Diagnostic Summary:

  • Diagnostic Details:
    • HIGH_UNGUIDED_PASS_RATE: Modern models natively write the CSS :autofill pseudo-class.

apply-webgl-shaders

Flags:

  • LOW_GUIDED_PASS_RATE: Guided pass rate was under 75% in all nightly runs (claude_code: 57%, codex_cli: 29%, jetski_cli: 57%).

Run Details:

Agent Guided Pass Rate Unguided Pass Rate Guides Consumed
claude_code 57% 14% apply-webgl-shaders, interactive-content-in-3d-scenes
codex_cli 29% 14% apply-webgl-shaders
jetski_cli 57% 14% apply-webgl-shaders, interactive-content-in-3d-scenes

Qualitative Diagnostic Summary:

  • Diagnostic Details:
    • Failed Assertions:
      • "The CSS transform property of the descendant HTML element is updated" (Test ID: 9b47ca60e16ed6df4065-e3eb502c93aad810aa91)
      • "A fallback UI strategy is implemented for unsupported browsers" (Test ID: 9b47ca60e16ed6df4065-a8255be38eed382e9886)
      • LOW_GUIDED_PASS_RATE: The grader asserts a console warning containing "not supported", which is undocumented in the guide's required fallback behaviour.

animate-element-entry-exit

Flags:

  • LOW_GUIDED_PASS_RATE: Guided pass rate was under 75% in all nightly runs (claude_code: 63%, codex_cli: 63%, jetski_cli: 63%).

Run Details:

Agent Guided Pass Rate Unguided Pass Rate Guides Consumed
claude_code 63% 13% animate-element-entry-exit
codex_cli 63% 25% animate-element-entry-exit
jetski_cli 63% 50% animate-element-entry-exit

Qualitative Diagnostic Summary:

  • Diagnostic Details:
    • Failed Assertions:
      • "includes transition-behavior: allow-discrete or allow-discrete keyword for display" (Test ID: b82ff594e67495691c1c-98167fc2c3d3aa8ff52e)
      • "includes display property in transition list" (Test ID: b82ff594e67495691c1c-eab5b92c57db2033eea8)
      • "transition durations for entry and exit are reasonable (0.3s to 1s)" (Test ID: b82ff594e67495691c1c-0f3db188b7d27662dce5)
      • LOW_GUIDED_PASS_RATE: The computed style tests for transitionBehavior fail on older browsers that lack native support.

adapt-scrollbar-to-contrast-preferences

Flags:

  • LOW_GUIDED_PASS_RATE: Guided pass rate was under 75% in all nightly runs (claude_code: 67%, codex_cli: 67%, jetski_cli: 67%).

Run Details:

Agent Guided Pass Rate Unguided Pass Rate Guides Consumed
claude_code 67% 17% adapt-scrollbar-to-contrast-preferences, customize-scrollbar-color-and-thickness
codex_cli 67% 50% adapt-scrollbar-to-contrast-preferences, customize-scrollbar-color-and-thickness
jetski_cli 67% 17% customize-scrollbar-color-and-thickness, adapt-scrollbar-to-contrast-preferences

Qualitative Diagnostic Summary:

  • Diagnostic Details:
    • Failed Assertions:
      • "CSS variables for scrollbar update to high-contrast colors when prefers-contrast: more is emulated" (Test ID: d578e6b09182e93cdaf5-a5fe29147d8401c4861c)
      • "High contrast scrollbar colors are highly distinct (e.g., black or white)" (Test ID: d578e6b09182e93cdaf5-03dbe02b579deddb4ed2)
      • LOW_GUIDED_PASS_RATE: The grader checks for literal string values like black or #000000, failing on functional notations like rgb() or hsl().

Metadata

Metadata

Assignees

No one assigned

    Labels

    nightly-investigationNightly Guidance Health Audits

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions