[AzureBackup] Fix exception types for correct telemetry classification#2841
Merged
shrja-ms merged 4 commits intoJun 10, 2026
Conversation
Add kql-customer-queries.md (queries 14-24) covering: - Customer identification via P360/C360 cross-cluster joins - Subscription breakdown and customer name resolution - Tool usage ranking and per-customer tool breakdown - Client/editor distribution and version analysis - Daily trend queries for dashboards - Customer aggregation rules and known data quirks Update SKILL.md to reference customer report workflow and document P360_CustomerName usage (vs unreliable CustomerName).
Replace InvalidOperationException with ArgumentException for user-input validation errors (unsupported policy type, DPP update not supported, ambiguous datasource, schedule/retention not applicable). Replace one InvalidOperationException with KeyNotFoundException for job-not-found after FormatException fallback. The ADX telemetry classifier treats InvalidOperationException as McpToolBug. These errors are actually Customer-side (bad input, wrong vault type, ambiguous match) and should classify as Customer (via ArgumentException -> 400) or Customer (via KeyNotFoundException -> 404). Changes: - RsvBackupOperations.cs: 7x InvalidOperationException -> ArgumentException - AzureBackupService.cs: DPP update not supported -> ArgumentException - DppBackupOperations.cs: Job not found fallback -> KeyNotFoundException - PolicyUpdateCommand.cs: Remove InvalidOperationException error handlers - Tests: Update exception types in mocks and assertions
shrja-ms
requested review from
alzimmermsft,
conniey,
saikoumudi,
sandeep-sen,
vukelich,
wbreza and
xirzec
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adjusts exception types in the Azure Backup tool/service layer so user-input validation failures are classified as Customer errors in telemetry (instead of being misclassified as MCP tool bugs), and updates command/test/documentation assets accordingly.
Changes:
- Replace several
InvalidOperationExceptionthrows used for input/validation scenarios withArgumentException(and one case toKeyNotFoundException) to influence telemetry classification. - Simplify
PolicyUpdateCommanderror handling by removingInvalidOperationException-specific branches now that the service layer no longer throws them for expected validation cases. - Update unit/live tests and add documentation + changelog entry covering telemetry/customer reporting guidance.
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tools/Azure.Mcp.Tools.AzureBackup/tests/Azure.Mcp.Tools.AzureBackup.Tests/Policy/PolicyUpdateCommandTests.cs | Updates unit tests to expect ArgumentException for DPP-not-supported and unsupported policy type scenarios. |
| tools/Azure.Mcp.Tools.AzureBackup/tests/Azure.Mcp.Tools.AzureBackup.Tests/AzureBackupCommandTests.cs | Updates live test assertion/comment to expect ArgumentException for DPP policy update not supported. |
| tools/Azure.Mcp.Tools.AzureBackup/src/Services/RsvBackupOperations.cs | Converts several validation/selection errors from InvalidOperationException to ArgumentException for telemetry classification. |
| tools/Azure.Mcp.Tools.AzureBackup/src/Services/DppBackupOperations.cs | Changes the job-not-found-after-FormatException-fallback throw type (see review comment). |
| tools/Azure.Mcp.Tools.AzureBackup/src/Services/AzureBackupService.cs | Throws ArgumentException (instead of InvalidOperationException) when policy update is attempted against a non-RSV vault. |
| tools/Azure.Mcp.Tools.AzureBackup/src/Commands/Policy/PolicyUpdateCommand.cs | Removes InvalidOperationException branches from error message + status code selection. |
| tools/Azure.Mcp.Tools.AzureBackup/skills/azurebackup-telemetry-report/SKILL.md | Expands the telemetry-report skill documentation to include customer/adoption reporting guidance and references. |
| tools/Azure.Mcp.Tools.AzureBackup/skills/azurebackup-telemetry-report/references/kql-customer-queries.md | Adds customer/adoption KQL query reference material. |
| servers/Azure.Mcp.Server/changelog-entries/azurebackup-fix-exception-telemetry-classification.yml | Adds a changelog entry describing the telemetry classification fix. |
…eption ListJobsAsync may return a partial list when it also hits FormatException during enumeration. A null FirstOrDefault result does NOT reliably mean the job is missing — it may exist beyond the enumerator break point. Throw FormatException (not KeyNotFoundException/404) to preserve the SDK-parse-failure semantics. This is a genuine tool limitation (tracked in azure-sdk-for-net#59306), not a 'not found' condition.
alzimmermsft
reviewed
Jun 9, 2026
Move azure-sdk-for-net#59306 reference from the exception message to a code comment. The issue link is not actionable by the user of Azure MCP.
alzimmermsft
approved these changes
Jun 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes telemetry misclassification where user-input validation errors were tagged as \McpToolBug\ instead of \Customer.
The ADX telemetry classifier treats \InvalidOperationException\ as \McpToolBug. Several throws in the Azure Backup service layer use \InvalidOperationException\ for what are actually user-input validation errors (unsupported policy type, DPP not supporting update, ambiguous datasource match, schedule/retention not applicable). These inflate the MCP bug count — e.g., 1 hit on \policy_update\ (beta.16, Jun 5) was classified as a tool bug when it was a user passing an unsupported policy type.
Changes
Telemetry Impact
This change ensures user-facing validation errors are correctly classified as Customer errors in the ADX dashboard and weekly telemetry reports.
Testing
Invoking Livetests
Copilot submitted PRs are not trustworthy by default. Users with \write\ access to the repo need to validate the contents of this PR before leaving a comment with the text /azp run mcp - pullrequest - live. This will trigger the necessary livetest workflows to complete required validation.