Benchmarks on whether this reduces performance of the model

[hugobiais]

This plugin honestly looks great, but I'm a bit reluctant when it comes to installing it because I'm concerned that I'll see a decrease in performance with it (since sometimes, the quality of the response of a model can be linked to the number of tokens it produces).

Would anyone be down to run benchmarks with this skill and see if we see and decline in performance or not?

Caveman does it: https://github.com/juliusbrussee/caveman

[ayghri]

@hugobiais We can define how to measure the performance and I'm ready to test it out.

[isatimur]

Implemented the reproducible evaluation foundation in #9:

• 14 cases across direct answers, agent autonomy, debugging, explanations, destructive actions, ambiguity, progress, output contracts, planning, and medical boundaries
• weighted correctness/autonomy/actionability/safety/concision rubric
• baseline/candidate/comparator run matrix with three trials
• hard configured budget ceiling, blind scoring, release gates, resumability, and bounded retries
• 8 unit tests plus CI

The first live Claude run produced 22 baseline responses with $6.65 reported spend. It then stopped on the account's weekly usage limit. The partial data is preserved and resumable; no performance claim is being made until both conditions complete under the same model and rubric.

[ayghri]

Update: the eval harness from #9 is now on main, trimmed to evals only. This issue stays open until someone produces baseline-vs-candidate numbers with it.

[mvanhorn]

Opened #95 with a measure subcommand for scripts/run_evals.py.

It aggregates the usage and cost_usd fields the harness already records on every response row and reports per-condition token, cost and response-length totals plus a candidate-versus-baseline delta. It reuses the pairing rule from #48, so conditions are compared only when they cover identical (case_id, trial) sets and came from the same runner; mixing runners or partial coverage raises rather than emitting a number that looks like a measurement.

That covers the cost half of this issue. Producing the actual baseline-vs-candidate numbers is still open, since it needs a chosen model and trial count, so this references the issue rather than closing it.