feat(vdiff): add summary_only option to vdiff show - #915
Closed
pedroalb wants to merge 5 commits into
Closed
Conversation
Add a summary_only option to vdiff show that omits the per-table report body (the sample row diffs) from each target's response, returning only the vdiff and per-table state, has_mismatch, and rows_compared. When set, each target primary selects a literal empty object in place of the stored report column, so the (potentially very large) report is never read from MySQL, never held in tablet memory, and never sent to vtctld. This lets callers that only need a continue/cancel decision avoid transferring reports that can exceed gRPC message limits for tables with large blob/JSON rows. The option is threaded end to end: VDiffShowRequest.summary_only -> VDiffReportOptions.summary_only on each tablet request (set before the fan-out) -> getVDiffSummary selecting the empty report expression. A --summary-only flag is added to the vtctldclient vdiff show command. The proto fields are additive so mixed-version vtctld/vttablet deployments remain safe, defaulting to existing behavior. Signed-off-by: Pedro Albuquerque <pedro.albuquerque@slack-corp.com>
Regenerate the vtadmin-web TypeScript/JS proto bindings so they include the new summary_only fields on VDiffShowRequest and VDiffReportOptions, keeping the Check Make VTAdmin Web Proto CI job green. Signed-off-by: Pedro Albuquerque <pedro.albuquerque@slack-corp.com>
Replace the reportExprToken placeholder and strings.Replace with two
named query constants built from shared column-list and FROM/WHERE
pieces. This removes the sentinel-in-SQL and runtime string munging
while avoiding duplication of the column list: sqlVDiffSummary and
sqlVDiffSummaryNoReport differ only in the report select-expression
(vdt.report vs '{}'). Behavior is unchanged.
Signed-off-by: Pedro Albuquerque <pedro.albuquerque@slack-corp.com>
…only
The summary-only query previously replaced the whole per-table report with a
literal '{}'. That had two problems: BuildSummary parses the scalar counters
(ProcessedRows, MatchingRows, MismatchedRows, ExtraRows*) out of the report
JSON, so blanking it produced misleading zero counts alongside
has_mismatch=true; and '{}' is non-NULL, unlike vdt.report on a LEFT JOIN with
no matching row.
Strip only the large row-sample arrays via JSON_REMOVE while preserving the
counters. Counts stay accurate, JSON_REMOVE(NULL, ...) returns NULL so the
no-row case matches the plain-column behavior, and the sampled rows (the
unbounded, blob-carrying part that can exceed gRPC message limits) are still
kept out of the response.
Verified on MySQL 8.0.46 and 5.7.44. Mirrors vitessio#20870.
Signed-off-by: Pedro Albuquerque <pedro.albuquerque@slack-corp.com>
The summary_only proto field comments and the --summary-only flag help still
described the earlier '{}'-blanking behavior, implying the per-table difference
counts are dropped. The implementation strips only the report's sampled-row
arrays via JSON_REMOVE and preserves the scalar counters, so the docs were
inaccurate. Update the two proto comments and the CLI flag help to describe the
actual behavior (samples stripped, counters preserved) and regenerate the Go
proto bindings. Comment-only; field name, number, and type are unchanged.
Signed-off-by: Pedro Albuquerque <pedro.albuquerque@slack-corp.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds a
summary_onlyoption tovdiff showthat omits the large per-table row-sample arrays from each target's response, returning the vdiff and per-table state,has_mismatch,rows_compared, and the report's scalar counters.The per-table
reportcolumn holds a JSONDiffReportcontaining both scalar counters (ProcessedRows,MatchingRows,MismatchedRows,ExtraRowsSource,ExtraRowsTarget) and sampled row data in theMismatchedRowsSample,ExtraRowsSourceSample, andExtraRowsTargetSamplearrays. The sample arrays carry actual row data, including large blob/JSON columns, andvdiff showaggregates them across every target shard, so for blob-heavy tables the response can exceed gRPC message limits.When
summary_onlyis set, each target primary strips just those sample arrays viaJSON_REMOVE(vdt.report, '$.MismatchedRowsSample', '$.ExtraRowsSourceSample', '$.ExtraRowsTargetSample')while preserving the scalar counters. This keeps the reported counts accurate (sohas_mismatchcan never disagree withMismatchedRows) and drops only the unbounded, blob-carrying part of the report, letting callers that only need a continue/cancel decision (e.g. the resharder) avoid transferring the samples.JSON_REMOVEreturns NULL when the report is NULL (aLEFT JOINwith no matchingvdiff_tablerow), matching the plain-column behavior.The option is threaded end to end:
VDiffShowRequest.summary_only(vtctld API)VDiffRequest.Options.ReportOptions.summary_onlybefore the fan-out, so the target primaries strip the samples at the sourcegetVDiffSummaryselects the sample-stripping report expression via a single DRY summary query (only the report select-expression varies)--summary-onlyflag on thevtctldclient vdiff showcommandThe new proto fields are additive (new field numbers, optional), so mixed-version vtctld/vttablet deployments remain safe and default to existing behavior. Note the size reduction only takes effect once the VDiff target primaries run this build, since the sample stripping happens on the tablet.
Note on the design
An earlier revision blanked the whole report to a literal
'{}'. That was changed because it (a) madeBuildSummaryparse zero for every counter, so the summary could showHasMismatch: truealongsideMismatchedRows: 0, and (b) turned the LEFT-JOIN NULL report into a non-NULL'{}'. Stripping only the sample arrays withJSON_REMOVEavoids both while still removing the large payload. This matches the approach taken upstream in vitessio#20870.Related Issue(s)
Internal: reduces VDiffShow response size for blob-heavy keyspaces that were exceeding gRPC message limits during resharding. Upstreamed as vitessio#20870 (issue vitessio#20869).
Testing
Unit test.
TestVDiffSummaryQuery(go/vt/vttablet/tabletmanager/vdiff/action_test.go) asserts structurally (not by brittle full-string match):%abind placeholders so the query stays parameterized;JSON_REMOVE;MySQL semantics verified on 8.0.46 and 5.7.44 (Docker), including the full summary query against a populated
_vt.vdiff/vdiff_tableschema:JSON_REMOVEof absent paths is a no-op);LEFT JOIN-with-no-row case returns a NULL report on both versions.Manual verification on a dev/staging cluster via
vtctl-slack VDiff ... show <uuid>:--format=json: full per-tableReportspresent (existing behavior).--format=json --summary-only: the sampled-row arrays are omitted whileState,HasMismatch,RowsCompared, and the per-table counters remain populated.Confirmed the
--summary-onlyflag is present and wired throughvtctl-slack VDiff show --help.Checklist
Deployment Notes
No DB migrations. Roll out to VDiff target primaries (vttablet) to realize the response-size reduction; vtctld/CLI changes are safe in any version-skew order and default to current behavior. The
JSON_REMOVEJSON-path functions are available on MySQL 5.7 and 8.0 (verified on both).