Skip to content

feat(evals): add multi-turn persistence evaluation - #69

Open
yrooogerg wants to merge 2 commits into
ayghri:mainfrom
yrooogerg:feat/session-eval
Open

feat(evals): add multi-turn persistence evaluation#69
yrooogerg wants to merge 2 commits into
ayghri:mainfrom
yrooogerg:feat/session-eval

Conversation

@yrooogerg

Copy link
Copy Markdown

Why this PR

The existing evaluation suite mainly measures single-turn response quality. i-have-adhd also promises to persist across turns and topic changes, stop suggesting code changes after repeated debugging failures, and exit when the user says normal mode. Those behaviors previously had no automated coverage.

This PR turns the most important multi-turn behavior chain into a reproducible evaluation, reducing the risk of silent regressions when the Skill changes.

What changed

  • Added a six-turn session scenario covering topic switching, repeated failures, and mode exit.
  • Uses Claude Code session resume to verify persistence after the Skill is injected once.
  • Isolates baseline and candidate runs from local Skills, plugins, MCP servers, and memory.
  • Runs deterministic structural checks before an independent blind Semantic Judge.
  • Produces structured results and evidence for every pass criterion.
  • Shares one budget across generation and judging, including cumulative reported failure costs.
  • Adds a short README prompt directing contributors to run relevant evaluations before submitting Skill changes or PRs.

Validation

python3 -m unittest discover -s tests -v
python3 scripts/run_scenario_eval.py validate \
  evals/scenarios/persistence-topic-switch-stop

All 18 tests pass.

Scope

This PR adds one high-value multi-turn scenario and its runner without changing Skill behavior. The scenario contract can support additional multi-turn regression cases later.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant