feat(store): add per-run keying and config-drift guard to EvaluationDataStore (#347)

[lifelmy]

Description

Extends the result-store layer so a case can be evaluated N times with every run kept, and guards against silently mixing results from different experiment configs. This is the store-layer foundation the issue describes as "additive and backward compatible" and the producer side of the multi-run history that #186's trend analyzer consumes.

EvaluationDataStore protocol — load/save gain a defaulted run_index: int = 0, and a new completed_run_count(case_name) -> int reports how many consecutive runs a case already has. run_index=0 preserves the exact single-run behavior, so existing callers need no change.

LocalFileTaskResultStore —

• Per-run layout: each case gets its own subdirectory (results/<case>/run_0.json, run_1.json, ...). completed_run_count counts consecutive runs from 0 and stops at the first gap, so it returns the next index to write when topping up to N.
• Migration: a legacy flat {case_name}.json from the old single-run layout is read as that case's run 0, so existing stores keep working with no migration step.
• Config-drift guard: an optional config_hash (plus overwrite) is recorded in a _manifest.json. Reopening with a different hash raises ConfigDriftError rather than appending incompatible results; overwrite=True discards prior results/manifest and starts fresh. Passing no hash skips the guard entirely.

Scope note (deliberate): this PR stops at the store contract + local implementation, which is self-contained and backward compatible. The num_runs resume loop inside run_evaluations_async and the --num-runs/--overwrite CLI flags from the proposal are intentionally left for a focused follow-up so this change stays small and easy to review — happy to send that next if maintainers prefer it here. The proposed EvaluationDataStore shape (defaulted run_index + completed_run_count) is exactly what that runner change will build on.

Related Issues

Addresses #347 (store-layer portion). Complements #186 (trend analysis consumes multi-run history).

Type of Change

New feature

Testing

hatch test tests/strands_evals/test_local_file_task_result_store.py -> 14 passed (per-run keying, legacy-flat migration, config-drift raise/overwrite/no-guard). hatch test tests/strands_evals/test_local_file_task_result_store.py tests/strands_evals/test_experiment.py -> 131 passed (updated the in-memory DictEvaluationDataStore test double to the new protocol; existing store/experiment behavior unchanged). hatch fmt --formatter --check -> 360 files already formatted; hatch fmt --linter -> all checks passed; mypy -p src via hatch-static-analysis env -> Success, no issues in 235 source files.

• I ran the linter, formatter, type checker, and the affected test suites

Checklist

• I have read the CONTRIBUTING document
• I have reviewed and understand every line of code in this PR. I use a coding assistant to help implement, and I review and take responsibility for the final change.
• My change is focused and reasonably small; I have split the runner/CLI wiring into a follow-up
• I have added tests that prove the feature works
• I have updated the documentation accordingly (public store API changed; happy to add docs if maintainers point me at the source)
• My changes generate no new warnings

---

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[lifelmy]

Task A follow-up: the repository-wide pre-commit test run exposed one stale CLI assertion that still expected the legacy flat c1.json layout. I updated it to assert the new per-run path c1/run_0.json and pushed ae283b3. Verification: focused store/experiment/CLI tests -> 132 passed; pre-commit format, lint, and full unit-test hooks passed; git diff --check passed.

[lifelmy]

Task A base refresh: merged current origin/main (24acea695, CI command pinning) without conflicts and pushed c538646b5 (0 behind / 5 ahead). The evaluation-store diff is unchanged. Verification: store/experiment/CLI suites 166 passed; focused Ruff check/format and git diff --check passed.

[lifelmy]

Task A base refresh: merged current origin/main (9ea55c96d461, README-only) without conflicts and pushed f8e9a468e533 (0 behind / 6 ahead). The evaluation-store diff is unchanged. Verification: store/experiment/CLI suites 166 passed; repository-wide Ruff check/format and git diff --check passed.

[lifelmy]

Task A base refresh: merged the latest main (9de7900ab) into this branch without conflicts and pushed 06db979c7 (0 behind).

Verification on the refreshed head:

• uv run pytest -q tests/strands_evals/test_experiment.py tests/strands_evals/test_local_file_task_result_store.py tests/strands_evals/cli/test_run.py → 173 passed
• uv run ruff check . → passed
• uv run ruff format --check . → 362 files already formatted
• git diff --check refs/remotes/origin/main...HEAD → passed

I use a coding assistant to help implement, and I review and take responsibility for the final change.