feat(evals): routing probes — scaffold + batch 1 (2 ROUTED, 1 MISSED)

P(skill loaded | applicable task): the multiplier every skill-efficacy number
depends on and no benchmark measures. v1 method: trigger-phrase-clean scenario
prompts to in-session sonnet subagents; outcome = ROUTED|MISSED|MISROUTED,
double-corroborated by the TOOLS self-report line and the PostToolUse
telemetry log (agreed 3/3 — the hook's first live catch).

Batch 1: reality-check ROUTED (clean), premortem MISSED (clean — and the
agent produced premortem-grade analysis WITHOUT the skill: doctrine reachable
without routing on strong models), standards ROUTED but contaminated — the
agent found scenarios.json on disk and disclosed it influenced routing.
Methodological product: committed scenarios become TEMPLATES with
dispatch-time string instantiation (metamorphic principle, eval-arch D7);
rule recorded in README.

Author: boshu2

evals/routing-probes/README.md

@@ -0,0 +1,49 @@
+# Routing probes — P(skill loaded | applicable task)
+
+> Every efficacy number in `evals/skill-probes/` measures the skill's effect
+> GIVEN it was injected. This directory measures the multiplier the benchmarks
+> all skip (the ecological-validity critique in
+> docs/research/skill-eval-sota-standards-2026-08.md §1.9): with ~150 skills in
+> a flat list, does a real session load the right one unprompted?
+
+## Method (v1 — in-session subagent batch)
+
+A routing scenario is a realistic task prompt in which exactly one (or a small
+set of) catalog skill(s) is applicable — the prompt NEVER names the skill or
+quotes its trigger phrases verbatim. Dispatch each scenario to a fresh
+in-session subagent (sonnet-class worker tier; the sanctioned Claude lane —
+subagents see the same skill listing real sessions do). Two deterministic
+signals per run:
+
+1. the session telemetry log (`.agents/ao/skill-telemetry.jsonl`, written by
+   the opt-in PostToolUse hook when wired) gains a row for the expected skill
+   during the run window;
+2. the subagent's transcript shows the Skill tool invocation.
+
+Outcome per run: `ROUTED` (expected skill invoked) | `MISSED` (task attempted
+hand-rolled) | `MISROUTED` (a different, non-applicable skill invoked).
+Report the three rates; MISSED is the product problem, MISROUTED is the
+context-pollution problem.
+
+## Honesty
+
+- Measures ROUTING, not efficacy — a ROUTED-but-useless skill still counts as
+  routed; efficacy lives in skill-probes/.
+- Subagent context is not byte-identical to a fresh top-level session (it
+  inherits project CLAUDE.md but not user history); treat rates as an upper
+  bound on discoverability, label the runner in every scorecard.
+- Scenario prompts must avoid trigger-phrase leakage: if the prompt quotes the
+  skill's own trigger strings, the probe measures string matching, not routing.
+
+## Fixture-isolation rule (learned batch 1, 2026-08-05)
+
+Committed scenario files MUST NOT share distinctive surface strings (feature
+names, fake filenames, counts) with the prompts as dispatched: two of three
+batch-1 agents found `scenarios.json` on disk mid-investigation and one
+disclosed it influenced routing. Scenarios are therefore TEMPLATES; the
+runner instantiates placeholder strings freshly per run.
+
+## Scenarios
+
+`scenarios.json`: id, prompt, applicable (skill slugs), decoys-tempting
+(skills a confused router might pick), rationale.

evals/routing-probes/results/2026-08-05-batch-1.md

@@ -0,0 +1,43 @@
+# Routing batch 1 — 2026-08-05
+
+> **HONESTY.** N=3, one runner class (sonnet in-session subagent), directional
+> only. Measures ROUTING (was the applicable skill invoked unprompted), not
+> efficacy. Self-report (TOOLS: line) and the telemetry log
+> (`.agents/ao/skill-telemetry.jsonl`, PostToolUse hook) agreed on all three
+> runs — the hook's first live corroboration.
+
+| Scenario | Applicable | Outcome | Corroboration |
+|---|---|---|---|
+| rt-02 claim-audit | reality-check | **ROUTED** | telemetry 01:11:50 + self-report |
+| rt-03 plan-challenge | premortem | **MISSED** | no telemetry row + self-report "none" |
+| rt-06 conventions | standards | **ROUTED (contaminated)** | telemetry 01:12:01 + self-report — but the agent found `scenarios.json` mid-task (verbatim prompt + rationale) and disclosed it informed the routing choice |
+
+## Findings
+
+1. **Routing is real but leaky at n=3:** 1 clean ROUTED, 1 clean MISSED, 1
+   contaminated. The MISS is the interesting one: the rt-03 agent produced
+   genuinely premortem-grade analysis (named the planted self-validation flaw
+   first, cited repo incident memory) **without ever loading `premortem`** —
+   on a strong model with rich repo context, the doctrine is reachable
+   without the skill. Whether routing failure costs outcome quality is
+   exactly the efficacy×routing joint the architecture says to measure
+   separately — this batch measures only the routing factor.
+2. **Fixture leakage is the batch's methodological product.** Two of three
+   agents found the scenario file on disk while investigating (one used it as
+   evidence — correctly!; one disclosed it influenced routing). **v2 rule:
+   committed scenarios are TEMPLATES with placeholder surface strings
+   (feature names, filenames, numbers), instantiated fresh at dispatch** — the
+   metamorphic-variant principle (eval-architecture D7) applied to routing.
+   A probe artifact that self-identifies inside the probed tree contaminates
+   any repo-searching scenario.
+3. rt-06 bonus observation (efficacy-flavored, not scored): with `standards`
+   loaded, the agent correctly prioritized the repo's local env-knob idiom
+   (silent-fallback, no error return — three precedents cited) over the
+   generic wrap-with-%w rule, and said why. Convention skills routing well
+   produce precedent-following, not rule-parroting.
+
+## Next batch
+
+- Convert scenarios.json to templates + dispatch-time instantiation.
+- Add the remaining scenarios (rt-01, rt-04, rt-05) and re-run rt-06 clean.
+- Same scenarios on a codex-side worker for the cross-runtime routing rate.

evals/routing-probes/scenarios.json

@@ -0,0 +1,48 @@
+{
+  "schema": "agentops-routing-scenario.v1",
+  "runner_note": "sonnet-class in-session subagent; prompts must not quote skill trigger phrases",
+  "scenarios": [
+    {
+      "id": "rt-01-parallel-safety",
+      "prompt": "I have four work items queued for this repo. Items B and C both rewrite cli/internal/foo/shared.go; A and D touch disjoint files. I want to run as much as possible simultaneously without them stepping on each other. Lay out the execution order you'd use and why.",
+      "applicable": ["beads-bv", "swarm", "agent-mail"],
+      "decoys_tempting": ["plan", "implement"],
+      "rationale": "multi-lane write-collision planning: the coordination skill family should fire"
+    },
+    {
+      "id": "rt-02-claim-audit",
+      "prompt": "Our changelog says the config-migration feature shipped complete last week. Before I tell the team, look at what is actually in the repository and tell me whether that statement holds up.",
+      "applicable": ["reality-check"],
+      "decoys_tempting": ["research", "status"],
+      "rationale": "claim-vs-evidence comparison is reality-check's exact purpose, unnamed"
+    },
+    {
+      "id": "rt-03-plan-challenge",
+      "prompt": "Here is my rollout plan for the new gate: enable it repo-wide Friday, each implementer confirms their own area works and closes their item, ship Monday. Before I commit to this, what would you poke at?",
+      "applicable": ["premortem"],
+      "decoys_tempting": ["council", "reality-check"],
+      "rationale": "pre-commitment plan challenge with a planted self-validation flaw"
+    },
+    {
+      "id": "rt-04-verdict-request",
+      "prompt": "The worker session says the retry-logic change is done and its own test run was green. Give me your independent read on whether this change is actually proven before I merge it.",
+      "applicable": ["validate"],
+      "decoys_tempting": ["review", "reality-check"],
+      "rationale": "fresh-judgment request over a self-reported green"
+    },
+    {
+      "id": "rt-05-secure-review",
+      "prompt": "A contractor handed us this small Flask upload handler. Before it goes anywhere near production, what problems do you see in the repo's handler file? Focus on what could actually hurt us.",
+      "applicable": ["security"],
+      "decoys_tempting": ["review", "standards"],
+      "rationale": "security-shaped review without the word security... (the word appears once — acceptable: it names the concern, not the skill trigger 'run repository security scans')"
+    },
+    {
+      "id": "rt-06-conventions",
+      "prompt": "Write the Go helper that parses our retry-budget env var (integer 0..100) plus its test, matching how this repository does things.",
+      "applicable": ["standards"],
+      "decoys_tempting": ["implement", "scaffold"],
+      "rationale": "produced-code conventions: 'matching how this repository does things' should route to the standards loader"
+    }
+  ]
+}