Skip to content

feat(server): monitor Auto mode in the provider canary and turn analytics - #930

Merged
incognitojam merged 2 commits into
mainfrom
styal/auto-mode-monitoring
Oct 20, 2026
Merged

incognitojam merged 2 commits into
mainfrom
styal/auto-mode-monitoring

Conversation

@incognitojam

@incognitojam incognitojam commented Oct 20, 2026 •

Copy link
Copy Markdown
Owner

Since #928, choosing Auto as the default starts Claude, Codex and Grok threads in Auto, but nothing noticed if a provider release changed how Auto behaves. The provider canary only ran Full access and Supervised turns, and turn analytics did not record approval prompts, so a reviewer that started asking about routine work, or a CLI that quietly stopped applying Auto, would go unseen.

Change

  • Canary auto scenario. For providers whose snapshot reports reviewsAutoMode, the verifier launches a thread in Auto and asks for a routine file write in the workspace. It fails if any approval request appears (declining it so nothing waits), if the run does not complete, or if the provider's own event log does not report applying Auto: Claude Code's system/init permissionMode and Codex's thread/settings/updated approvalsReviewer. The last check is needed because Claude Code runs a routine write without asking even in its default mode. Providers without the flag (OpenCode today) skip the scenario. Like the other scenarios, a failure blocks that release from managed updates.
  • approvalRequests on provider.turn.completed. The count of command, file-read, file-change and permission requests the turn raised, read from the thread's runtime requests when the turn ends. Questions are not counted, and no command or file content is sent. With the existing runtimeMode, this shows the share of Auto turns that needed a person, by provider and version.
  • Docs: the canary page lists the new scenario, the feature-usage guide describes how to read the count, Product usage data mentions it, and the provider-auto-default ledger entry records both checks.

Validation

  • Ran the verifier against real CLIs with this branch: Claude Code 2.1.302 on claude-haiku-5-5 passed provider, tool-turn, approval and auto (provider reported permissionMode auto); Codex 0.162.1 on gpt-6-luna passed auto (reported approvalsReviewer auto_review); OpenCode 2.0.28 skipped auto as expected.
  • Mapped Claude's Auto to default in the adapter temporarily and reran: the scenario failed with "The provider did not report Auto: permissionMode default". Before the provider-report check was added, that run passed, which is why the check exists.
  • Checked offline that Claude Code reports Auto support for every current model under API-key authentication with a clean home, which is how the canary signs in. A live Auto turn with the canary's API keys was not run; the first scheduled canary after the next nightly will be the first.
  • The Codex approval scenario failed locally on GPT-6 Luna (once the command ran without a prompt, once the model never ran it). The scenario and Codex adapter are unchanged here; the canary's retry covers this, and the last two scheduled runs passed.
  • Unit test: a completed turn with command and permission approvals, a question, and another turn's approval records approvalRequests: 2; the existing usage test now expects approvalRequests: 0. Server typecheck clean; ledger validates.

The canary runs the verifier from each build's release tag, so the auto scenario applies from the first nightly that contains it.


Written by an agent (Claude Code, claude-opus-5-5).

…tics

The provider canary gains an auto scenario for providers that report an Auto
reviewer: a routine workspace write in Auto must finish without an approval
request, and the provider must report applying Auto (Claude Code's init
permissionMode, Codex's thread approvalsReviewer). provider.turn.completed now
reports approvalRequests, the command, file and permission approvals the turn
raised, so a provider release whose reviewer starts asking shows by mode.
@github-actions

Copy link
Copy Markdown

Upstream divergence

Both sides are compared with upstream at the in-order tip 0ecb25a8d4. Excludes .repos/ and pnpm-lock.yaml.

Measure Base This PR Change
Upstream files the fork changes 1,505 1,505 0
Lines changed in them 78,248 78,367 +119
Upstream files the fork removes 42 42 0
Fork-only files 549 549 0
Fork-only lines 100,759 100,875 +116
Upstream files whose difference changes (3)
File Change Now differs by
apps/server/src/orchestration-v2/ProviderEventIngestor.test.ts +89 359 lines
apps/server/src/orchestration-v2/ProviderEventIngestor.ts +27 159 lines
docs/user/telemetry.md +3 61 lines

@github-actions

Copy link
Copy Markdown

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

ℹ️ The exact PR base did not have a successful artifact. Baseline uses the latest successful main measurement shown below.

Provider Metric Main baseline This PR Impact PR ceiling
Codex Total thread wire 5.0 KiB 5.0 KiB 0 B (0.0%) 6.8 KiB ✅
Codex Thread snapshot wire 3.8 KiB 3.8 KiB 0 B (0.0%) 4.9 KiB ✅
Codex Live turn WebSocket wire 1.2 KiB 1.2 KiB 0 B (0.0%) 2.0 KiB ✅
Codex Live turn WebSocket decoded 20.9 KiB 20.9 KiB 0 B (0.0%) 29.3 KiB ✅
Codex Live turn messages 2 2 0 (0.0%) 8 ✅
Claude Total thread wire 5.0 KiB 5.0 KiB +7 B (+0.1%) 6.8 KiB ✅
Claude Thread snapshot wire 3.8 KiB 3.8 KiB 0 B (0.0%) 4.9 KiB ✅
Claude Live turn WebSocket wire 1.2 KiB 1.2 KiB +7 B (+0.6%) 2.0 KiB ✅
Claude Live turn WebSocket decoded 21.2 KiB 21.2 KiB 0 B (0.0%) 29.3 KiB ✅
Claude Live turn messages 2 2 0 (0.0%) 8 ✅

Baseline: ff3b3e1 · PR result: 0c4ff27 · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

  • Codex decoded thread snapshot: 108.5 KiB
  • Claude decoded thread snapshot: 108.8 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

@incognitojam
incognitojam merged commit 41977ab into main Oct 20, 2026
35 checks passed
@incognitojam
incognitojam deleted the styal/auto-mode-monitoring branch October 20, 2026 11:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant