feat(evaluators): add FuzzyEquals deterministic evaluator

[lifelmy]

Description

Adds FuzzyEquals, a configurable deterministic evaluator for comparing structured outputs where exact matching is too brittle. Addresses strands-agents/harness-sdk#3727.

The existing deterministic evaluators (Equals, Contains, StartsWith) only do exact matching, which fails for SQL result sets, JSON objects, tabular data, and numeric computations where row/key order, numeric precision, or whitespace shouldn't decide a pass. FuzzyEquals compares actual_output against an expected value recursively across lists and dicts, applying configurable knobs at every level.

Knobs (matching the issue proposal)

• ignore_order — treat lists as multisets so element order does not matter
• ignore_key_order — dict key/value pairs must match regardless of insertion order (default True)
• numeric_tolerance — relative tolerance for numeric values (0.01 accepts +/-1%)
• case_sensitive — case-insensitive string comparison when False
• ignore_whitespace — strip and collapse internal whitespace before comparing
• subset_mode — pass if the expected value is contained in actual (actual ⊇ expected)
• type_coercion — numeric/boolean strings compare equal to numbers/booleans ("42" == 42)

from strands_evals.evaluators.deterministic import FuzzyEquals

evaluator = FuzzyEquals(
    ignore_order=True,
    numeric_tolerance=0.01,
    case_sensitive=False,
    ignore_whitespace=True,
    type_coercion=True,
)

Follows the existing Evaluator[InputT, OutputT] pattern and returns an EvaluationOutput with a detailed diff path in the reason field (e.g. at $.rows[0].total: numbers differ ...). A module sentinel distinguishes an explicit value=None from the "compare against expected_output" default — something Equals cannot currently do.

Testing

• hatch fmt --formatter --check — 362 files already formatted
• hatch fmt --linter — all checks passed
• mypy -p src (hatch-static-analysis env) — Success: no issues found in 236 source files
• pytest tests/strands_evals/evaluators/deterministic/ — 114 passed (44 new FuzzyEquals tests, no regressions)

Notes

I use a coding assistant to help implement, and I review and take responsibility for the final change.

Marking as Draft for maintainer feedback on the API surface and whether the recursive implementation (vs. the deepdiff alternative mentioned in the issue) is preferred before finalizing.

Related Issue

strands-agents/harness-sdk#3727

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[lifelmy]

Task A base refresh: merged current origin/main (24acea695, CI command pinning) without conflicts and pushed fa2e85879 (0 behind / 4 ahead). The feature diff is unchanged. Verification: fuzzy evaluator suite 44 passed; focused Ruff check/format and git diff --check passed.

[lifelmy]

Task A base refresh: merged current origin/main (9ea55c96d461, README-only) without conflicts and pushed f5c8994890fe (0 behind / 5 ahead). The fuzzy evaluator diff is unchanged. Verification: fuzzy evaluator suite 44 passed; repository-wide Ruff check/format and git diff --check passed.

[lifelmy]

Task A base refresh: merged the latest main (9de7900ab) into this branch and pushed db6f2a417 (0 behind).

This was a real export conflict in src/strands_evals/evaluators/__init__.py; I resolved it by preserving this PR's FuzzyEquals export alongside upstream's new StructuredOutputReport / StructuredOutputSimilarity exports.

Verification on the refreshed head:

• uv run pytest -q tests/strands_evals/evaluators/deterministic/test_fuzzy.py tests/strands_evals/evaluators/deterministic/test_structured_output.py → 44 passed, 188 skipped
• uv run ruff check . → passed
• uv run ruff format --check . → 364 files already formatted
• git diff --check refs/remotes/origin/main...HEAD → passed

I use a coding assistant to help implement, and I review and take responsibility for the final change.