Skip to content

feat(evals): report token and cost deltas between conditions - #95

Open
mvanhorn wants to merge 1 commit into
ayghri:mainfrom
mvanhorn:fix/evals-token-cost-delta
Open

feat(evals): report token and cost deltas between conditions#95
mvanhorn wants to merge 1 commit into
ayghri:mainfrom
mvanhorn:fix/evals-token-cost-delta

Conversation

@mvanhorn

@mvanhorn mvanhorn commented Aug 1, 2026

Copy link
Copy Markdown

scripts/run_evals.py already writes usage and cost_usd on every response row it records, and nothing ever reads them back. The harness answers "did quality hold" through score, but has no command that answers "at what cost", which is the measurement half of issue #4. This adds a measure subcommand that aggregates those already-recorded fields into per-condition token, cost and response-length totals plus a candidate-versus-baseline delta. Conditions are compared only when they cover identical (case_id, trial) sets and came from the same runner; mixing runners or partial coverage raises rather than emitting a number that looks like a measurement.

Verify

python3 tests/test_run_evals.py goes from 9 tests to 14, all passing.
New coverage: per-condition aggregation across both supported runner usage shapes; a different-runner pair raises with both runner names in the message; unequal (case_id, trial) coverage raises via the rule extracted from PR #48; a missing cost_usd reports null plus the count of unreported rows rather than 0; and a runner reporting no token counts yields null, not 0.
score output is byte-identical before and after the _coverage_errors extraction on the same input.

What

The change is confined to one module plus its docs and tests. measure reads the responses JSONL, groups rows by condition, sums the token and cost fields already present on each row, and prints JSON in the same shape the existing score branch prints. The pairing rule score already enforces is extracted into a shared helper so both summaries apply one definition of comparability instead of two copies that can drift.

Demo

measure subcommand demo

Simulated demo: an animated walkthrough, not a screen recording. The token and cost figures shown are illustrative example values, not a measured run of this repository.

Prior attempts on this issue

Issue #4 has two earlier closed-unmerged PRs, neither of which attempted the measurement work: #11 (refine ADHD-friendly response behavior) was closed once #10 landed disable-model-invocation and settled the activation-scope question, and #24 (opt-in always-on mode via a SessionStart hook) was closed shortly after opening. Both were about skill activation behaviour. This change does not revisit either approach; it adds a read-only aggregation command over data the harness already records.

Refs #4

Adds a measure subcommand that aggregates the usage and cost_usd fields
run_evals.py already records on every response row, reporting per-condition
totals and the candidate-versus-baseline delta.

Applies the same comparability rule the release gate uses: conditions are
compared only when they cover identical (case_id, trial) sets and came from
the same runner. Mixing runners or partial coverage raises rather than
emitting a number that looks like a measurement.

Refs ayghri#4
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant