feat(eval): checkpoint fidelity eval with decay round - #454
Open
bobzhang wants to merge 3 commits into
Open
Conversation
eval/compaction_fidelity scores whether a compaction summary lets the next turn continue accurately: planted-fact preservation in the summary text (error codes, crash sites, versions), scripted probes answered from the compacted projection, and a second summary-of-summary round that measures fact decay directly. Fixtures are built programmatically with complete assistant/tool batches; deterministic tests pin that the projections are API-valid and that every planted fact is present in the raw projection, so a perfect summarizer could score 100%. Probe questions carry a no-tools prefix: the projection keeps the fixture's agent system prompt, and without the prefix a probe can come back as tool-call markup instead of an answer — the summarizer-side twin of a26456b. First live run: 4/5 facts after one checkpoint, 3/5 after the decay round, 4/4 probes, and both checkpoint requests were full prompt-cache misses — the baseline the compaction-hardening plan builds against. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two subal review findings: the decay follow-up bumped the version to 0.9.2-rc1, whose text contains the original 0.9.2 release needle, so a second summary that forgot the release blocker still scored the fact as preserved — the follow-up now bumps to 1.0.0-rc1, and a deterministic test pins that follow-up content never replants any planted fact. Probe matching is now token-delimited instead of raw substring, so a "no" acceptance no longer matches inside "now" or "unknown". Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two more subal findings: a rejected token match starting with a supplementary character advanced the retry cursor into its low surrogate, and the next slice panics — the cursor now steps a whole code point (whitebox test pins the repro). The anti-replant guard now checks every assistant field the projection replays to the summarizer (reasoning and tool-call names/arguments), not just visible content. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Second slice of the compaction production-hardening plan (R0: baseline fidelity before changing summarizer behavior). Stacked on #453 (consumes
CompactionSummary's usage/duration telemetry for its cost rows).eval/compaction_fidelityscores whether a checkpoint summary lets the next turn continue accurately:Fixtures include a finished turn, a mid-session user redirection, an exact-error tool failure, and a prompt-injection line inside a tool result. Deterministic tests (run in CI) pin that fixture projections are API-valid, every planted fact exists in the raw projection, and follow-ups never replant a fact in any model-visible field.
First live baseline (deepseek-v4-pro)
Review
Three subal (codex) rounds: two substring-collision scoring bugs and a surrogate-pair panic found and fixed (each with pinning tests); final round clean.
🤖 Generated with Claude Code