Repository navigation
feat(server): monitor Auto mode in the provider canary and turn analytics - #930
Merged
Merged
Conversation
…tics The provider canary gains an auto scenario for providers that report an Auto reviewer: a routine workspace write in Auto must finish without an approval request, and the provider must report applying Auto (Claude Code's init permissionMode, Codex's thread approvalsReviewer). provider.turn.completed now reports approvalRequests, the command, file and permission approvals the turn raised, so a provider release whose reviewer starts asking shows by mode.
Upstream divergenceBoth sides are compared with upstream at the in-order tip
Upstream files whose difference changes (3)
|
Thread transfer impact✅ Thread transfer remains within every enforced ceiling.
Baseline: Scenario and decoded snapshot size10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.
Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Since #928, choosing Auto as the default starts Claude, Codex and Grok threads in Auto, but nothing noticed if a provider release changed how Auto behaves. The provider canary only ran Full access and Supervised turns, and turn analytics did not record approval prompts, so a reviewer that started asking about routine work, or a CLI that quietly stopped applying Auto, would go unseen.
Change
autoscenario. For providers whose snapshot reportsreviewsAutoMode, the verifier launches a thread in Auto and asks for a routine file write in the workspace. It fails if any approval request appears (declining it so nothing waits), if the run does not complete, or if the provider's own event log does not report applying Auto: Claude Code'ssystem/initpermissionModeand Codex'sthread/settings/updatedapprovalsReviewer. The last check is needed because Claude Code runs a routine write without asking even in its default mode. Providers without the flag (OpenCode today) skip the scenario. Like the other scenarios, a failure blocks that release from managed updates.approvalRequestsonprovider.turn.completed. The count of command, file-read, file-change and permission requests the turn raised, read from the thread's runtime requests when the turn ends. Questions are not counted, and no command or file content is sent. With the existingruntimeMode, this shows the share of Auto turns that needed a person, by provider and version.provider-auto-defaultledger entry records both checks.Validation
claude-haiku-5-5passedprovider,tool-turn,approvalandauto(provider reportedpermissionMode auto); Codex 0.162.1 ongpt-6-lunapassedauto(reportedapprovalsReviewer auto_review); OpenCode 2.0.28 skippedautoas expected.defaultin the adapter temporarily and reran: the scenario failed with "The provider did not report Auto: permissionMode default". Before the provider-report check was added, that run passed, which is why the check exists.approvalscenario failed locally on GPT-6 Luna (once the command ran without a prompt, once the model never ran it). The scenario and Codex adapter are unchanged here; the canary's retry covers this, and the last two scheduled runs passed.approvalRequests: 2; the existing usage test now expectsapprovalRequests: 0. Server typecheck clean; ledger validates.The canary runs the verifier from each build's release tag, so the
autoscenario applies from the first nightly that contains it.Written by an agent (Claude Code, claude-opus-5-5).