Skip to content

feat(vdiff): add summary_only option to vdiff show - #915

Closed
pedroalb wants to merge 5 commits into
slack-22.0from
pedroalb/vdiff-show-summary-only-22
Closed

feat(vdiff): add summary_only option to vdiff show#915
pedroalb wants to merge 5 commits into
slack-22.0from
pedroalb/vdiff-show-summary-only-22

Conversation

@pedroalb

@pedroalb pedroalb commented Aug 18, 2026

Copy link
Copy Markdown

Description

Adds a summary_only option to vdiff show that omits the large per-table row-sample arrays from each target's response, returning the vdiff and per-table state, has_mismatch, rows_compared, and the report's scalar counters.

The per-table report column holds a JSON DiffReport containing both scalar counters (ProcessedRows, MatchingRows, MismatchedRows, ExtraRowsSource, ExtraRowsTarget) and sampled row data in the MismatchedRowsSample, ExtraRowsSourceSample, and ExtraRowsTargetSample arrays. The sample arrays carry actual row data, including large blob/JSON columns, and vdiff show aggregates them across every target shard, so for blob-heavy tables the response can exceed gRPC message limits.

When summary_only is set, each target primary strips just those sample arrays via JSON_REMOVE(vdt.report, '$.MismatchedRowsSample', '$.ExtraRowsSourceSample', '$.ExtraRowsTargetSample') while preserving the scalar counters. This keeps the reported counts accurate (so has_mismatch can never disagree with MismatchedRows) and drops only the unbounded, blob-carrying part of the report, letting callers that only need a continue/cancel decision (e.g. the resharder) avoid transferring the samples. JSON_REMOVE returns NULL when the report is NULL (a LEFT JOIN with no matching vdiff_table row), matching the plain-column behavior.

The option is threaded end to end:

  • VDiffShowRequest.summary_only (vtctld API)
  • forwarded onto each tablet's VDiffRequest.Options.ReportOptions.summary_only before the fan-out, so the target primaries strip the samples at the source
  • getVDiffSummary selects the sample-stripping report expression via a single DRY summary query (only the report select-expression varies)
  • a --summary-only flag on the vtctldclient vdiff show command

The new proto fields are additive (new field numbers, optional), so mixed-version vtctld/vttablet deployments remain safe and default to existing behavior. Note the size reduction only takes effect once the VDiff target primaries run this build, since the sample stripping happens on the tablet.

Note on the design

An earlier revision blanked the whole report to a literal '{}'. That was changed because it (a) made BuildSummary parse zero for every counter, so the summary could show HasMismatch: true alongside MismatchedRows: 0, and (b) turned the LEFT-JOIN NULL report into a non-NULL '{}'. Stripping only the sample arrays with JSON_REMOVE avoids both while still removing the large payload. This matches the approach taken upstream in vitessio#20870.

Related Issue(s)

Internal: reduces VDiffShow response size for blob-heavy keyspaces that were exceeding gRPC message limits during resharding. Upstreamed as vitessio#20870 (issue vitessio#20869).

Testing

Unit test. TestVDiffSummaryQuery (go/vt/vttablet/tabletmanager/vdiff/action_test.go) asserts structurally (not by brittle full-string match):

  • both variants preserve the %a bind placeholders so the query stays parameterized;
  • the non-summary query returns the stored report as-is and does not use JSON_REMOVE;
  • the summary-only query strips exactly the three sample arrays;
  • the two variants are identical after swapping the report select-expression, so no other summary column or clause can silently differ.

MySQL semantics verified on 8.0.46 and 5.7.44 (Docker), including the full summary query against a populated _vt.vdiff/vdiff_table schema:

  • sample arrays removed, all scalar counters preserved;
  • a report with no sample arrays is returned unchanged (JSON_REMOVE of absent paths is a no-op);
  • the LEFT JOIN-with-no-row case returns a NULL report on both versions.

Manual verification on a dev/staging cluster via vtctl-slack VDiff ... show <uuid>:

  • default (text) output: unchanged (state + per-table counts).
  • --format=json: full per-table Reports present (existing behavior).
  • --format=json --summary-only: the sampled-row arrays are omitted while State, HasMismatch, RowsCompared, and the per-table counters remain populated.

Confirmed the --summary-only flag is present and wired through vtctl-slack VDiff show --help.

Checklist

  • "Backport to:" labels have been added if this change should be back-ported to release branches
  • If this change is to be back-ported to previous releases, a justification is included in the PR description
  • Tests were added or are not required
  • Did the new or modified tests pass consistently locally and on CI?
  • Documentation was added or is not required

Deployment Notes

No DB migrations. Roll out to VDiff target primaries (vttablet) to realize the response-size reduction; vtctld/CLI changes are safe in any version-skew order and default to current behavior. The JSON_REMOVE JSON-path functions are available on MySQL 5.7 and 8.0 (verified on both).

Add a summary_only option to vdiff show that omits the per-table report
body (the sample row diffs) from each target's response, returning only
the vdiff and per-table state, has_mismatch, and rows_compared.

When set, each target primary selects a literal empty object in place of
the stored report column, so the (potentially very large) report is never
read from MySQL, never held in tablet memory, and never sent to vtctld.
This lets callers that only need a continue/cancel decision avoid
transferring reports that can exceed gRPC message limits for tables with
large blob/JSON rows.

The option is threaded end to end: VDiffShowRequest.summary_only ->
VDiffReportOptions.summary_only on each tablet request (set before the
fan-out) -> getVDiffSummary selecting the empty report expression. A
--summary-only flag is added to the vtctldclient vdiff show command. The
proto fields are additive so mixed-version vtctld/vttablet deployments
remain safe, defaulting to existing behavior.

Signed-off-by: Pedro Albuquerque <pedro.albuquerque@slack-corp.com>
@pedroalb
pedroalb requested a review from a team as a code owner August 18, 2026 06:53
@github-actions github-actions Bot added this to the v22.0.4 milestone Aug 18, 2026
Regenerate the vtadmin-web TypeScript/JS proto bindings so they include
the new summary_only fields on VDiffShowRequest and VDiffReportOptions,
keeping the Check Make VTAdmin Web Proto CI job green.

Signed-off-by: Pedro Albuquerque <pedro.albuquerque@slack-corp.com>
Replace the reportExprToken placeholder and strings.Replace with two
named query constants built from shared column-list and FROM/WHERE
pieces. This removes the sentinel-in-SQL and runtime string munging
while avoiding duplication of the column list: sqlVDiffSummary and
sqlVDiffSummaryNoReport differ only in the report select-expression
(vdt.report vs '{}'). Behavior is unchanged.

Signed-off-by: Pedro Albuquerque <pedro.albuquerque@slack-corp.com>
…only

The summary-only query previously replaced the whole per-table report with a
literal '{}'. That had two problems: BuildSummary parses the scalar counters
(ProcessedRows, MatchingRows, MismatchedRows, ExtraRows*) out of the report
JSON, so blanking it produced misleading zero counts alongside
has_mismatch=true; and '{}' is non-NULL, unlike vdt.report on a LEFT JOIN with
no matching row.

Strip only the large row-sample arrays via JSON_REMOVE while preserving the
counters. Counts stay accurate, JSON_REMOVE(NULL, ...) returns NULL so the
no-row case matches the plain-column behavior, and the sampled rows (the
unbounded, blob-carrying part that can exceed gRPC message limits) are still
kept out of the response.

Verified on MySQL 8.0.46 and 5.7.44. Mirrors vitessio#20870.

Signed-off-by: Pedro Albuquerque <pedro.albuquerque@slack-corp.com>
The summary_only proto field comments and the --summary-only flag help still
described the earlier '{}'-blanking behavior, implying the per-table difference
counts are dropped. The implementation strips only the report's sampled-row
arrays via JSON_REMOVE and preserves the scalar counters, so the docs were
inaccurate. Update the two proto comments and the CLI flag help to describe the
actual behavior (samples stripped, counters preserved) and regenerate the Go
proto bindings. Comment-only; field name, number, and type are unchanged.

Signed-off-by: Pedro Albuquerque <pedro.albuquerque@slack-corp.com>
@pedroalb pedroalb closed this Aug 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant