docs(mcp): fuzz + conformance guide, CLI reference, ADR-0015, roadmap v1.2

Author: abhay-codes07

docs/ROADMAP.md

@@ -97,5 +97,15 @@ tools through MCP get offline, byte-identical integration tests (ADR-0014).
 - [x] stdio transport: transparent recording proxy + replay serve-loop
 - [x] `volo mcp record | serve` CLI, `examples/mcp_calc_server.py`, byte-faithful e2e test
 - [x] Docs: README quickstart, docs-site guide, CLI reference
-- [ ] M11: MCP fuzz — scenario operators at the MCP boundary (`volo mcp fuzz`) + conformance
-      report for MCP-server authors
+
+## v1.2.0 — M11: MCP fuzz + conformance ✅
+Goal: adversarial testing at the MCP boundary — "what does my agent do when the server returns
+garbage?" — plus a regression gate for MCP-server authors.
+
+- [x] `volo_mcp.fuzz`: scenario operators applied inside the `{"result": ...}` envelope;
+      handshake/meta steps and recorded protocol errors stay byte-intact; seeded/reproducible
+- [x] `volo mcp fuzz` — one servable mutated recording per operator (resilience, robustness,
+      security, order_sensitivity), `--serve` for a live hostile world, `--report` JSON
+- [x] `volo_mcp.conformance` + `volo mcp conformance` — replay recorded requests against the
+      LIVE server, diff answers (errors included), exit 1 on behavioral change
+- [ ] M12: pytest plugin (`pytest-volo`) — reliability tests as unit tests

docs/adr/0015-mcp-fuzz-and-conformance.md

@@ -0,0 +1,59 @@
+# ADR 0015: MCP fuzz targets result envelopes only; conformance replays against the live server
+
+- Status: accepted
+- Date: 2026-07-03
+
+## Context
+
+M11 brings adversarial testing to the MCP boundary (ADR-0014). Two design questions:
+(1) *what* may the fuzzer mutate in a recorded MCP session, and (2) what does "conformance"
+mean for an MCP-server author using Volo as a regression gate?
+
+## Decision
+
+**Fuzz (`volo_mcp.fuzz`)** reuses the generic `volo-scenarios` operators unchanged, wrapped by
+two MCP-specific rules:
+
+1. **Fuzz targets are only real tool responses** — `tool_call` steps with an `mcp.tool:` prefix
+   whose response is a `{"result": <object>}` envelope. Handshake/meta steps
+   (`mcp:initialize`, `mcp:tools/list`, …) and recorded *protocol errors* are byte-intact in
+   every mutation, so a fuzzed session still boots and error behavior stays authentic.
+2. **Operators run inside the envelope.** The fuzzer extracts the target steps into a
+   sub-recording, unwraps `{"result": X}` → `X`, applies the operator, re-wraps, and merges the
+   steps back into their original positions. Operators therefore mutate the object the agent
+   actually reads (e.g. `corrupt_field` flips `isError`; `prompt_injection` lands inside the
+   content), and the mutated recording replays through `MCPReplayServer` with no special cases.
+
+Default library: `drop_tool_result`, `corrupt_field`, `prompt_injection`, `reorder_steps`
+(failure classes: resilience, robustness, security, order_sensitivity). Excluded:
+`inject_latency` (latency metadata is never served over the wire) and `ambiguous_user_turn`
+(MCP recordings contain no model calls). Mutations are seeded → reproducible in CI.
+
+**Conformance (`volo_mcp.conformance`)** treats a recording as a behavioral contract: every
+recorded request is rebuilt (`messages.request_message`, the inverse of `tool_key`) and sent to
+a freshly spawned live server; each reply is compared to the recorded envelope. Verdicts:
+`identical` / `different` / `no_reply`; anything non-identical fails (exit 1 in the CLI).
+Recorded protocol errors are part of the contract and must reproduce.
+
+## Consequences
+
+- Every fuzz output is itself a valid, servable recording — `volo mcp serve` and future
+  reliability scoring work on hostile worlds with zero extra machinery.
+- Restricting targets to result envelopes means the fuzzer never breaks the transport layer —
+  by design. Malformed-protocol fuzzing (bad JSON-RPC framing, wrong ids) is a separate future
+  concern, closer to the transport tests than to scenario operators.
+- Conformance request reconstruction is lossy for `tools/call` params beyond
+  `name`/`arguments` (deliberately dropped from the cache identity in ADR-0014). Servers keying
+  behavior on exotic params need a fresh recording rather than conformance replay.
+- Conformance compares byte-equal envelopes; servers with legitimately nondeterministic fields
+  (timestamps, ids) will report `different`. A normalization hook is future work if demanded.
+
+## Alternatives considered
+
+- **MCP-specific operator implementations** — rejected: duplicates the M2 taxonomy and forks
+  the failure-class vocabulary the reliability engine already understands.
+- **Fuzzing everything, including handshake/meta and error envelopes** — rejected: trivially
+  broken sessions (failed `initialize`) mask the interesting failures, and mutating a recorded
+  error's shape produces worlds no real server can express.
+- **Conformance via the simulator (replay both sides offline)** — rejected: that tests Volo
+  against itself; the author's question is whether the *live build* still honors the contract.

website/cli.mdx

@@ -54,6 +54,21 @@ return JSON-RPC error `-32042` instead of a hallucination. See [Test MCP servers
 uv run volo mcp serve rec.json
 ```
 
+## `volo mcp fuzz`
+Mutate recorded tool responses with the adversarial scenario operators (seeded, reproducible);
+write one servable mutated recording per operator, or serve one hostile world directly.
+```bash
+uv run volo mcp fuzz rec.json [--seed 0] [--out-dir DIR] [--report fuzz.json]
+uv run volo mcp fuzz rec.json --op prompt_injection --serve
+```
+
+## `volo mcp conformance`
+Replay recorded requests against the LIVE server and diff the answers (recorded errors included).
+Exit 1 if behavior changed — a CI regression gate for MCP-server authors.
+```bash
+uv run volo mcp conformance rec.json [--report conf.json] -- <server command...>
+```
+
 ## `volo scenarios`
 List the default adversarial scenario operators and their failure classes.
 

website/mcp.mdx

@@ -88,9 +88,42 @@ A recorded `tools/list` response is automatically distilled into per-tool **sche
 but whose tool declares a response shape — can be synthesized under the same guarantee:
 **validate or flag, never hallucinate**.
 
-## What's next
+## Fuzz the boundary: what does your agent do when the server misbehaves?
 
-MCP **fuzzing** is the next milestone: Volo's adversarial scenario operators
-(corrupt fields, inject latency, prompt-inject tool outputs…) applied at the MCP boundary, plus a
-conformance report for MCP server authors. Track progress in the
+`volo mcp fuzz` mutates the recorded tool responses with Volo's adversarial scenario operators —
+dropped results, corrupted fields, embedded prompt injections, reordered answers — and writes one
+mutated recording per operator. Each one is a servable "hostile world":
+
+```bash
+uv run volo mcp fuzz calc.json --report fuzz-report.json
+# mcp fuzz: drop_tool_result   [resilience]        1 change(s) -> calc.drop_tool_result.seed0.json
+# mcp fuzz: corrupt_field      [robustness]        1 change(s) -> calc.corrupt_field.seed1.json
+# mcp fuzz: prompt_injection   [security]          1 change(s) -> calc.prompt_injection.seed2.json
+# mcp fuzz: reorder_steps      [order_sensitivity] 1 change(s) -> calc.reorder_steps.seed3.json
+
+# point your agent at one hostile world directly:
+uv run volo mcp fuzz calc.json --op prompt_injection --serve
+```
+
+Mutations are seeded (same seed, same hostile world — reproducible in CI), they only ever touch
+*real tool results*: the `initialize` handshake, `tools/list`, and recorded protocol errors stay
+byte-intact so the session still boots and failure behavior stays authentic.
+
+## For MCP server authors: conformance
+
+A recording is a behavioral contract. `volo mcp conformance` replays every recorded request
+against your **live** server build and diffs the answers — recorded errors included:
+
+```bash
+uv run volo mcp conformance calc.json --report conformance.json -- python my_server.py
+# mcp conformance: ok [002] mcp:tools/list      identical
+# mcp conformance: !! [003] mcp.tool:add        different
+# mcp conformance: 3 identical, 1 different, 0 no-reply -> FAIL   (exit code 1)
+```
+
+Run it in your server's CI to catch behavioral regressions before your users' agents do.
+
+## Roadmap
+
+Track what's next in the
 [roadmap](https://github.com/abhay-codes07/VOLO/blob/main/docs/ROADMAP.md).