Skip to content

Commit bf09fd7

Browse files
committed
feat(vdiff): add only_summary option to vdiff show
VDiff show returns a per-table diff report body (stored in _vt.vdiff_table.report) alongside the summary state. The report's row-sample arrays (MismatchedRowsSample, ExtraRowsSourceSample, ExtraRowsTargetSample) carry actual sampled row data, including large blob/JSON columns, and vdiff show aggregates them across every target shard. For diffs over tables with large rows this can push the aggregated gRPC response past message-size limits and make vdiff show fail outright, leaving callers that only need progress and the has_mismatch flag with no way to read the summary. Add an only_summary option, threaded from the vtctldclient `--only-summary` flag through VDiffShowRequest and the tablet VDiffReportOptions. When set, the tablet's summary query strips the row-sample arrays from the report via JSON_REMOVE while preserving the scalar counters (ProcessedRows, MatchingRows, MismatchedRows, ExtraRows*), so the summary counts stay accurate and the large sampled rows are neither read into the response nor sent to vtctld. JSON_REMOVE returns NULL when the report is NULL (no joined vdiff_table row), matching the plain-column behavior. All other summary columns are unaffected. The summary query is composed from shared column-list and FROM/WHERE constants so the two variants differ only in the report select-expression. ## AI Disclosure This change was co-authored with Claude Code, which helped with implementation and testing. Signed-off-by: Pedro Albuquerque <pedro.albuquerque@slack-corp.com>
1 parent bd205ba commit bf09fd7

14 files changed

Lines changed: 282 additions & 15 deletions

File tree

changelog/25.0/25.0.0/summary.md

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@
1919
- **[Minor Changes](#minor-changes)**
2020
- **[VReplication](#minor-changes-vreplication)**
2121
- [Default data protection for `_reverse` workflow cancel/complete](#vreplication-reverse-workflow-data-protection)
22+
- [`vdiff show --only-summary` omits the per-table row-sample report](#vreplication-vdiff-only-summary)
2223
- **[VTGate](#minor-changes-vtgate)**
2324
- [Ingress bytes in query LogStats](#vtgate-logstats-ingress-bytes)
2425
- [New controls for cross-keyspace reads](#vtgate-cross-keyspace-reads)
@@ -170,6 +171,16 @@ The `--keep-data` flag help text has been updated to note this default explicitl
170171

171172
See [#19906](https://github.com/vitessio/vitess/pull/19906) for details.
172173

174+
#### <a id="vreplication-vdiff-only-summary"/>`vdiff show --only-summary` omits the per-table row-sample report</a>
175+
176+
`vtctldclient vdiff ... show` now accepts an `--only-summary` flag. When set, the per-table diff report has its sampled-row arrays (`MismatchedRowsSample`, `ExtraRowsSourceSample`, `ExtraRowsTargetSample`) stripped on the tablet before the response is built, while the scalar counters (processed, matching, mismatched, and extra rows) and all other summary fields are preserved.
177+
178+
The sampled-row arrays carry actual row data, including large `BLOB`/`JSON` columns, and `vdiff show` aggregates them across every target shard. For diffs over tables with large rows this could push the aggregated response past gRPC message limits and make `vdiff show` fail outright, leaving no way to read the summary or mismatch state. `--only-summary` lets callers that only need progress and the mismatch state avoid transferring the samples while keeping the reported counts accurate.
179+
180+
The option is exposed as `only_summary` on the `VDiffShowRequest` (vtctld) and `VDiffReportOptions` (tablet) protobuf messages. It is opt-in and backward compatible: without the flag, the full report is returned as before.
181+
182+
See [#20870](https://github.com/vitessio/vitess/pull/20870) for details.
183+
173184
### <a id="minor-changes-vtgate"/>VTGate</a>
174185

175186
#### <a id="vtgate-logstats-ingress-bytes"/>Ingress bytes in query LogStats</a>

go/cmd/vtctldclient/command/vreplication/vdiff/vdiff.go

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -82,8 +82,9 @@ var (
8282
}{}
8383

8484
showOptions = struct {
85-
Arg string
86-
Verbose bool
85+
Arg string
86+
Verbose bool
87+
OnlySummary bool
8788
}{}
8889

8990
stopOptions = struct {
@@ -645,6 +646,7 @@ func commandShow(cmd *cobra.Command, args []string) error {
645646
Workflow: common.BaseOptions.Workflow,
646647
TargetKeyspace: common.BaseOptions.TargetKeyspace,
647648
Arg: showOptions.Arg,
649+
OnlySummary: showOptions.OnlySummary,
648650
})
649651
if err != nil {
650652
return err
@@ -709,6 +711,7 @@ func registerCommands(root *cobra.Command) {
709711
base.AddCommand(resume)
710712

711713
show.Flags().BoolVar(&showOptions.Verbose, "verbose", false, "Show verbose output in summaries")
714+
show.Flags().BoolVar(&showOptions.OnlySummary, "only-summary", false, "Omit the per-table diff report body and return only the vdiff and per-table summary state. Useful for large diffs where the report can exceed gRPC message limits.")
712715
base.AddCommand(show)
713716

714717
stop.Flags().StringSliceVar(&stopOptions.TargetShards, "target-shards", nil, "The target shards to stop the vdiff on; default is all shards.")

go/vt/proto/tabletmanagerdata/tabletmanagerdata.pb.go

Lines changed: 19 additions & 4 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

go/vt/proto/tabletmanagerdata/tabletmanagerdata_vtproto.pb.go

Lines changed: 34 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

go/vt/proto/vtctldata/vtctldata.pb.go

Lines changed: 16 additions & 3 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

go/vt/proto/vtctldata/vtctldata_vtproto.pb.go

Lines changed: 34 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

go/vt/vtctl/workflow/vdiff.go

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -566,6 +566,11 @@ func (s *Server) VDiffShow(ctx context.Context, req *vtctldatapb.VDiffShowReques
566566
Workflow: req.Workflow,
567567
Action: string(vdiff.ShowAction),
568568
ActionArg: req.Arg,
569+
Options: &tabletmanagerdatapb.VDiffOptions{
570+
ReportOptions: &tabletmanagerdatapb.VDiffReportOptions{
571+
OnlySummary: req.GetOnlySummary(),
572+
},
573+
},
569574
}
570575

571576
ts, err := s.buildTrafficSwitcher(ctx, req.TargetKeyspace, req.Workflow)

go/vt/vttablet/tabletmanager/vdiff/action.go

Lines changed: 19 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -115,11 +115,27 @@ func (vde *Engine) PerformVDiffAction(ctx context.Context, req *tabletmanagerdat
115115
return resp, nil
116116
}
117117

118-
func (vde *Engine) getVDiffSummary(vdiffID int64, dbClient binlogplayer.DBClient) (*query.QueryResult, error) {
118+
// vdiffSummaryQuery returns the summary query to run. When onlySummary is true
119+
// it returns the variant that strips the row-sample arrays from the per-table
120+
// report, keeping the scalar counters. The samples are the part that can grow
121+
// very large (they carry sampled row data, including large blob/JSON columns)
122+
// and, fanned out across many target shards, can push the aggregated response
123+
// past gRPC message limits. Stripping them lets callers that only need the
124+
// vdiff/table state, counts and has_mismatch avoid transferring that data while
125+
// keeping the summary counters accurate. All other summary columns are
126+
// unaffected.
127+
func vdiffSummaryQuery(onlySummary bool) string {
128+
if onlySummary {
129+
return sqlVDiffSummaryOnly
130+
}
131+
return sqlVDiffSummary
132+
}
133+
134+
func (vde *Engine) getVDiffSummary(vdiffID int64, dbClient binlogplayer.DBClient, reportOpts *tabletmanagerdatapb.VDiffReportOptions) (*query.QueryResult, error) {
119135
var qr *sqltypes.Result
120136
var err error
121137

122-
query, err := sqlparser.ParseAndBind(sqlVDiffSummary, sqltypes.Int64BindVariable(vdiffID), sqltypes.StringBindVariable(vde.dbName))
138+
query, err := sqlparser.ParseAndBind(vdiffSummaryQuery(reportOpts.GetOnlySummary()), sqltypes.Int64BindVariable(vdiffID), sqltypes.StringBindVariable(vde.dbName))
123139
if err != nil {
124140
return nil, err
125141
}
@@ -336,7 +352,7 @@ func (vde *Engine) handleShowAction(ctx context.Context, dbClient binlogplayer.D
336352
case 1:
337353
row := qr.Named().Row()
338354
vdiffID, _ := row["id"].ToInt64()
339-
summary, err := vde.getVDiffSummary(vdiffID, dbClient)
355+
summary, err := vde.getVDiffSummary(vdiffID, dbClient, req.GetOptions().GetReportOptions())
340356
resp.Output = summary
341357
if err != nil {
342358
return err

go/vt/vttablet/tabletmanager/vdiff/action_test.go

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@ package vdiff
1919
import (
2020
"context"
2121
"fmt"
22+
"strings"
2223
"testing"
2324
"time"
2425

@@ -34,6 +35,47 @@ import (
3435
vtrpcpb "vitess.io/vitess/go/vt/proto/vtrpc"
3536
)
3637

38+
// TestVDiffSummaryQuery asserts that the two summary-query variants differ only
39+
// in how they project the report column: the full variant returns the stored
40+
// report as-is, while the summary-only variant strips just the row-sample
41+
// arrays and keeps the scalar counters. The assertions are structural (split on
42+
// the report select-expression and compare the surrounding column list and
43+
// FROM/WHERE) so harmless SQL formatting changes don't break the test.
44+
func TestVDiffSummaryQuery(t *testing.T) {
45+
const (
46+
fullReportExpr = `vdt.report as report`
47+
summaryReportExpr = `JSON_REMOVE(vdt.report, '$.MismatchedRowsSample', '$.ExtraRowsSourceSample', '$.ExtraRowsTargetSample') as report`
48+
)
49+
50+
full := vdiffSummaryQuery(false)
51+
summary := vdiffSummaryQuery(true)
52+
53+
// Both must be bindable (report/columns aside, the FROM/WHERE has the %a
54+
// placeholders that ParseAndBind fills).
55+
require.Contains(t, full, "%a", "bind placeholders must be preserved for ParseAndBind")
56+
require.Contains(t, summary, "%a", "bind placeholders must be preserved for ParseAndBind")
57+
58+
// Each variant uses its own report select-expression and not the other's.
59+
require.Contains(t, full, fullReportExpr, "full query must select the stored report as-is")
60+
require.NotContains(t, full, "JSON_REMOVE", "full query must not strip the report")
61+
require.Contains(t, summary, summaryReportExpr, "summary-only query must strip the row-sample arrays from the report")
62+
63+
// The summary-only variant must strip the sample arrays but must not touch
64+
// the scalar counters, so the summary counts stay accurate.
65+
for _, sample := range []string{"MismatchedRowsSample", "ExtraRowsSourceSample", "ExtraRowsTargetSample"} {
66+
require.Contains(t, summaryReportExpr, sample, "expected sample array %q to be stripped", sample)
67+
}
68+
69+
// The two variants must be identical everywhere except the report
70+
// select-expression: swapping in the full expression must reproduce the
71+
// full query exactly, so no other column or clause can silently differ.
72+
require.Equal(t,
73+
full,
74+
strings.Replace(summary, summaryReportExpr, fullReportExpr, 1),
75+
"summary-only must differ from the full query only in the report select-expression",
76+
)
77+
}
78+
3779
func TestPerformVDiffAction(t *testing.T) {
3880
ctx, cancel := context.WithTimeout(t.Context(), 30*time.Second)
3981
defer cancel()

0 commit comments

Comments
 (0)