VDiff: save a sample for every drained extra row so reconciliation can match them (#20855) - #930
Open
pedroalb wants to merge 1 commit into
Open
VDiff: save a sample for every drained extra row so reconciliation can match them (#20855)#930pedroalb wants to merge 1 commit into
pedroalb wants to merge 1 commit into
Conversation
…n match them (vitessio#20855) Signed-off-by: Matthias Crauwels <matthias.crauwels@planetscale.com> Signed-off-by: Matt Lord <mattalord@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Matt Lord <mattalord@gmail.com> (cherry picked from commit e5b3f9a) Signed-off-by: Pedro Albuquerque <pedro.albuquerque@slack-corp.com>
pedroalb
force-pushed
the
pedroalb/vdiff-drain-sample-22
branch
from
September 2, 2026 09:07
ce3f507 to
5488456
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport of upstream vitessio#20855 to
slack-22.0Description
When one side's stream ends before the other, the remaining rows on the other side are drained: they are counted as extra rows, but only the first of them gets a row-diff sample saved.
reconcileExtraRows()(vitessio#18585) matches extra rows by comparing the saved samples, so drained rows beyond the first can never be reconciled. When the source and target streams return the same rows in a different order (e.g. because of PK collation differences between the sides), the diff ends with extra rows on both sides that are actually identical, and the sample-less drained rows survive reconciliation as false-positive extras — the vdiff reportsHasMismatch: trueon matching data.This PR makes the drain path save a sample for each drained row, subject to the usual
--max-extra-rows-to-comparecap, the same as for extra rows found in the main compare loop. It also:reconcileExtraRows()'smaxRowsby the number of saved samples rather than the extra-row counts;primitiveExecutor.drain();Test coverage:
TestDiffDrainedRowSamplingdrives the fulldiff()loop for both drain directions plus the sampling cap, andTestDiffDrainStreamErrorinjects a stream error mid-drain, verifies the deferred progress update persists neither counts nor samples for the partially drained rows, then resumes and verifies each drained row is counted exactly once. Both tests fail on the previous code.Backport notes
Cherry-picked from upstream
e5b3f9a6ec, rebased ontoslack-22.0(after #927). Thetable_differ.godrain rewrite andworkflow_differ.goreconcile logic are identical to upstream. The one adaptation is inreport.go: slack-22.0 predates an unrelated upstream change that renders binary sample values as hex, so only theLosslessValueslogic was applied onto the existingaddVal, without that hex rendering (it is orthogonal to this fix — both sides render identically, so sample matching still works). For the same missing dependency, upstream merged this to release-24.0 (vitessio#20943) but closed the release-23.0 backport (vitessio#20942); slack-22.0 is in release-23.0's position.