feat(evals): report token and cost deltas between conditions - #95
Open
mvanhorn wants to merge 1 commit into
Open
feat(evals): report token and cost deltas between conditions#95mvanhorn wants to merge 1 commit into
mvanhorn wants to merge 1 commit into
Conversation
Adds a measure subcommand that aggregates the usage and cost_usd fields run_evals.py already records on every response row, reporting per-condition totals and the candidate-versus-baseline delta. Applies the same comparability rule the release gate uses: conditions are compared only when they cover identical (case_id, trial) sets and came from the same runner. Mixing runners or partial coverage raises rather than emitting a number that looks like a measurement. Refs ayghri#4
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
scripts/run_evals.pyalready writesusageandcost_usdon every response row it records, and nothing ever reads them back. The harness answers "did quality hold" throughscore, but has no command that answers "at what cost", which is the measurement half of issue #4. This adds ameasuresubcommand that aggregates those already-recorded fields into per-condition token, cost and response-length totals plus a candidate-versus-baseline delta. Conditions are compared only when they cover identical(case_id, trial)sets and came from the same runner; mixing runners or partial coverage raises rather than emitting a number that looks like a measurement.Verify
python3 tests/test_run_evals.pygoes from 9 tests to 14, all passing.New coverage: per-condition aggregation across both supported runner usage shapes; a different-runner pair raises with both runner names in the message; unequal
(case_id, trial)coverage raises via the rule extracted from PR #48; a missingcost_usdreportsnullplus the count of unreported rows rather than0; and a runner reporting no token counts yieldsnull, not0.scoreoutput is byte-identical before and after the_coverage_errorsextraction on the same input.What
The change is confined to one module plus its docs and tests.
measurereads the responses JSONL, groups rows by condition, sums the token and cost fields already present on each row, and prints JSON in the same shape the existingscorebranch prints. The pairing rulescorealready enforces is extracted into a shared helper so both summaries apply one definition of comparability instead of two copies that can drift.Demo
Simulated demo: an animated walkthrough, not a screen recording. The token and cost figures shown are illustrative example values, not a measured run of this repository.
Prior attempts on this issue
Issue #4 has two earlier closed-unmerged PRs, neither of which attempted the measurement work: #11 (refine ADHD-friendly response behavior) was closed once #10 landed
disable-model-invocationand settled the activation-scope question, and #24 (opt-in always-on mode via a SessionStart hook) was closed shortly after opening. Both were about skill activation behaviour. This change does not revisit either approach; it adds a read-only aggregation command over data the harness already records.Refs #4