Add MCP output size measurement tooling

[Larry Osterman (LarryOsterman)]

Why

MCP clients benefit from understanding the protocol cost of tool discovery and learn-mode responses. This adds a repeatable, opt-in measurement workflow to compare consolidated and namespace server modes.

This tool is written as a test to take advantage of the existing MCP test infrastructure that invokes the MCP tool so that the metrics can be generated based on the actual output of the MCP server.

Approach

• Adds an opt-in stdio integration test that captures initialize, discovery, top-level learn, and per-command learn responses across both modes.
• Writes structured reports and response artifacts, including decoded top learn descriptions and extracted inner-command schemas.
• Adds PowerShell scripts to summarize JSON reports as console, JSON, and Markdown output, and to run build, measurement, and summarization end to end.

Validation

Ran the end-to-end measurement script successfully.

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).
There may be pipelines that require an authorized user to comment /azp run to run.

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Pull request overview

Adds an opt-in measurement workflow to quantify MCP server response sizes (initialize, tools/list discovery, and learn-mode tool/command responses) to help compare protocol cost between consolidated and namespace modes.

Changes:

• Adds an opt-in stdio integration test that starts azmcp in both modes and records response sizes + artifacts.
• Adds a summarization script to compare modes, emit console/JSON/Markdown summaries, and extract learn text / inner-command schemas.
• Adds an end-to-end runner script that builds, runs the measurement test, and then summarizes results.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 6 comments.

File	Description
servers/Azure.Mcp.Server/tests/Azure.Mcp.Server.Tests/Infrastructure/McpOutputSizeTests.cs	New opt-in integration test that runs stdio discovery/learn flows and writes structured measurement artifacts.
eng/scripts/Summarize-McpOutputSizes.ps1	Summarizes measurement report and extracts per-tool learn text and inner-command artifacts.
eng/scripts/Measure-McpOutputSizes.ps1	Orchestrates build → measurement test run → summarization into a single workflow.

Suppressed comments (3)

servers/Azure.Mcp.Server/tests/Azure.Mcp.Server.Tests/Infrastructure/McpOutputSizeTests.cs:191

• JsonDocument should be disposed (it uses pooled buffers). This loop can run for hundreds of tools, so leaving documents undisposed can significantly increase transient memory usage.

                });
                var response = await SendRequestAsync(process, request);
                var document = JsonDocument.Parse(response);
                Assert.True(
                    document.RootElement.TryGetProperty("result", out _),

servers/Azure.Mcp.Server/tests/Azure.Mcp.Server.Tests/Infrastructure/McpOutputSizeTests.cs:246

• JsonDocument should be disposed here as well (pooled buffers). This inner loop can run many times (tools × commands) so it’s worth keeping allocations bounded.

                    });
                    var response = await SendRequestAsync(process, request);
                    var document = JsonDocument.Parse(response);
                    Assert.True(
                        document.RootElement.TryGetProperty("result", out var commandResult),
                        $"The learn response for '{tool}.{command}' did not contain a result.");

servers/Azure.Mcp.Server/tests/Azure.Mcp.Server.Tests/Infrastructure/McpOutputSizeTests.cs:75

• ServerTestOptions is referenced here, but no such type exists in the repo, so the test project won’t compile. Use the same direct server-start argument pattern as other infrastructure tests.

        using var process = Process.Start(new ProcessStartInfo
        {
            FileName = executablePath,
            Arguments = ServerTestOptions.GetServerStartArguments($"server start --mode {mode}"),
            UseShellExecute = false,

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Alan Zimmer (alzimmermsft)]

Will review the CSharp portion after we determine if we want it as a test or a standalone application like many of our other outer loop tooling (like Vally)

[Alan Zimmer (alzimmermsft)]

Long term would be nice to expand this out to support any MCP server in this repo

[Larry Osterman (LarryOsterman)]

Agreed.

[Alan Zimmer (alzimmermsft)]

Do we need $Clean if our implicit behavior is overwriting any previous result we captured?

[Larry Osterman (LarryOsterman)]

I added the -clean because the tool outputs to the same directory all the time and sometimes there are artifacts from previous runs (this happened yesterday while I was preparing for office hours - I had some artifacts from last week which confused me).

So when things change, there can be artifacts left over from previous runs that no longer apply.

[Alan Zimmer (alzimmermsft)]

Does this need to be a separate script? Should the other script and where we land on how this works just write this report always?

[Larry Osterman (LarryOsterman)]

No it doesn't, but in other code reviews people objected to the large size of the scripts so I broke it up into a script runner and an analysis/output generator.

[Larry Osterman (LarryOsterman)]

FWIW, it's now a single script.

[Larry Osterman (LarryOsterman)]

Alan Zimmer (@alzimmermsft) It's now a standalone tool, and a single script :).