stream_migrator: skip OnlineDDL stopped streams during SwitchWrites - #20837
stream_migrator: skip OnlineDDL stopped streams during SwitchWrites#20837yushuqin wants to merge 113 commits into
Conversation
Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>
Signed-off-by: GitHub <noreply@github.com> Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> Co-authored-by: frouioui <frouioui@users.noreply.github.com> Co-authored-by: Florent Poinsard <florent.poinsard@outlook.fr>
…own (vitessio#18103) (vitessio#18106) Signed-off-by: Andres Taylor <andres@planetscale.com> Co-authored-by: vitess-bot[bot] <108069721+vitess-bot[bot]@users.noreply.github.com>
vitessio#18098) (vitessio#18115) Signed-off-by: Manan Gupta <manan@planetscale.com> Co-authored-by: vitess-bot[bot] <108069721+vitess-bot[bot]@users.noreply.github.com>
…itessio#18117) Signed-off-by: Manan Gupta <manan@planetscale.com> Co-authored-by: vitess-bot[bot] <108069721+vitess-bot[bot]@users.noreply.github.com>
…vitessio#18123) Signed-off-by: Noble Mittal <noblemittal@outlook.com> Co-authored-by: vitess-bot[bot] <108069721+vitess-bot[bot]@users.noreply.github.com>
Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>
…ease (vitessio#18134) Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>
… without needing to split them (vitessio#18059) (vitessio#18139) Signed-off-by: Manan Gupta <manan@planetscale.com> Co-authored-by: vitess-bot[bot] <108069721+vitess-bot[bot]@users.noreply.github.com>
…itessio#18153) Signed-off-by: Harshit Gangal <harshit@planetscale.com>
…itessio#18156) Signed-off-by: Manan Gupta <manan@planetscale.com> Co-authored-by: vitess-bot[bot] <108069721+vitess-bot[bot]@users.noreply.github.com>
…e` support (vitessio#18148) (vitessio#18157) Signed-off-by: Manan Gupta <manan@planetscale.com> Co-authored-by: vitess-bot[bot] <108069721+vitess-bot[bot]@users.noreply.github.com>
…r definers (vitessio#18142) (vitessio#18158) Signed-off-by: Manan Gupta <manan@planetscale.com> Co-authored-by: vitess-bot[bot] <108069721+vitess-bot[bot]@users.noreply.github.com>
Signed-off-by: Andres Taylor <andres@planetscale.com>
…) (vitessio#18165) Signed-off-by: Andres Taylor <andres@planetscale.com> Co-authored-by: vitess-bot[bot] <108069721+vitess-bot[bot]@users.noreply.github.com>
Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> Co-authored-by: Florent Poinsard <florent.poinsard@outlook.fr>
…ease (vitessio#18170) Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> Co-authored-by: Florent Poinsard <florent.poinsard@outlook.fr>
…o#18186) (vitessio#18192) Signed-off-by: deepthi <deepthi@planetscale.com> Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> Co-authored-by: vitess-bot[bot] <108069721+vitess-bot[bot]@users.noreply.github.com> Co-authored-by: Florent Poinsard <florent.poinsard@outlook.fr>
…o#18197) Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>
…itessio#18199) Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>
…o#18200) Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> Signed-off-by: Andres Taylor <andres@planetscale.com> Co-authored-by: Florent Poinsard <florent.poinsard@outlook.fr> Co-authored-by: Andres Taylor <andres@planetscale.com>
…18177) (vitessio#18189) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: vitess-bot[bot] <108069721+vitess-bot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> Co-authored-by: Florent Poinsard <florent.poinsard@outlook.fr>
…ease (vitessio#18206) Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> Co-authored-by: Florent Poinsard <florent.poinsard@outlook.fr>
Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> Co-authored-by: Florent Poinsard <florent.poinsard@outlook.fr>
Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
* empty commit to test CI Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * update skip logic Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * update skip logic Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * add private repo token to non-template files Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * add private repo token to non-template files Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * remove ci we dont need Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * remove more ci we dont need Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * update un-templated runs-on: Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * remove even more ci we dont need Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * update un-templated runs-on:, part 2 Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * use go@1.22 for old release Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> --------- Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
* [release-22.0] Bump to `v22.0.1-SNAPSHOT` after the `v22.0.0` release (vitessio#18225) Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> Co-authored-by: Florent Poinsard <florent.poinsard@outlook.fr> * [release-22.0] fix: Preserve multi-column TupleExpr in tuple simplifier (vitessio#18216) (vitessio#18220) Signed-off-by: Harshit Gangal <harshit@planetscale.com> Co-authored-by: vitess-bot[bot] <108069721+vitess-bot[bot]@users.noreply.github.com> Co-authored-by: Harshit Gangal <harshit@planetscale.com> * [release-22.0] Properly handle grpc dial errors in the throttler metric aggregation (vitessio#18073) (vitessio#18231) Signed-off-by: Arthur Schreiber <arthurschreiber@github.com> Signed-off-by: Mohamed Hamza <mhamza15@github.com> Co-authored-by: vitess-bot[bot] <108069721+vitess-bot[bot]@users.noreply.github.com> Co-authored-by: Mohamed Hamza <mhamza15@github.com> * [release-22.0] test: TestQueryTimeoutWithShardTargeting fix flaky test (vitessio#18242) (vitessio#18250) Signed-off-by: Harshit Gangal <harshit@planetscale.com> Co-authored-by: vitess-bot[bot] <108069721+vitess-bot[bot]@users.noreply.github.com> * [release-22.0] make sure to give MEMBER OF the correct precedence (vitessio#18237) (vitessio#18245) Signed-off-by: Andres Taylor <andres@planetscale.com> Co-authored-by: Andrés Taylor <andres@planetscale.com> * [release-22.0] Fix evalengine crashes on unexpected types (vitessio#18254) (vitessio#18258) Co-authored-by: vitess-bot[bot] <108069721+vitess-bot[bot]@users.noreply.github.com> * [release-22.0] Fix subquery merging regression introduced in vitessio#11379 (vitessio#18260) (vitessio#18263) Signed-off-by: Arthur Schreiber <arthurschreiber@github.com> Co-authored-by: Andrés Taylor <andres@planetscale.com> * [release-22.0] json array insert test (vitessio#18284) (vitessio#18286) Signed-off-by: Harshit Gangal <harshit@planetscale.com> Co-authored-by: vitess-bot[bot] <108069721+vitess-bot[bot]@users.noreply.github.com> Co-authored-by: Harshit Gangal <harshit@planetscale.com> * [release-22.0] Fix `SET` and `START TRANSACTION` in create procedure statements (vitessio#18279) (vitessio#18293) Signed-off-by: Manan Gupta <manan@planetscale.com> Co-authored-by: vitess-bot[bot] <108069721+vitess-bot[bot]@users.noreply.github.com> * [release-22.0] Fix deadlock in semi-sync monitor (vitessio#18276) (vitessio#18290) Signed-off-by: Manan Gupta <manan@planetscale.com> Co-authored-by: vitess-bot[bot] <108069721+vitess-bot[bot]@users.noreply.github.com> * [release-22.0] Upgrade the Golang version to `go1.24.3` (vitessio#18239) Signed-off-by: GitHub <noreply@github.com> Signed-off-by: Andres Taylor <andres@planetscale.com> Co-authored-by: frouioui <frouioui@users.noreply.github.com> Co-authored-by: Andres Taylor <andres@planetscale.com> * [release-22.0] Atomic Copy: Handle error that was ignored while streaming tables and log it (vitessio#18313) (vitessio#18316) Signed-off-by: Rohit Nayak <rohit@planetscale.com> Co-authored-by: vitess-bot[bot] <108069721+vitess-bot[bot]@users.noreply.github.com> * [release-22.0] fix: handle dml query for None opcode (vitessio#18326) (vitessio#18345) Signed-off-by: Harshit Gangal <harshit@planetscale.com> Co-authored-by: vitess-bot[bot] <108069721+vitess-bot[bot]@users.noreply.github.com> Co-authored-by: Harshit Gangal <harshit@planetscale.com> * [release-22.0] fix: keep LIMIT/OFFSET even when merging UNION queries (vitessio#18361) (vitessio#18363) Signed-off-by: Andres Taylor <andres@planetscale.com> Co-authored-by: vitess-bot[bot] <108069721+vitess-bot[bot]@users.noreply.github.com> Co-authored-by: Andres Taylor <andres@planetscale.com> * [release-22.0] Fix: Deadlock in `Close` and `write` in semi-sync monitor. (vitessio#18359) (vitessio#18368) Signed-off-by: Manan Gupta <manan@planetscale.com> Co-authored-by: vitess-bot[bot] <108069721+vitess-bot[bot]@users.noreply.github.com> * [release-22.0] Upgrade the Golang version to `go1.24.4` (vitessio#18329) Signed-off-by: GitHub <noreply@github.com> Signed-off-by: Manan Gupta <manan@planetscale.com> Co-authored-by: frouioui <frouioui@users.noreply.github.com> Co-authored-by: Manan Gupta <manan@planetscale.com> * [release-22.0] fix version issue when using --mysql-shell-speedup-restore=true (vitessio#18310) (vitessio#18356) Co-authored-by: vitess-bot[bot] <108069721+vitess-bot[bot]@users.noreply.github.com> * [release-22.0] Split workflow with flaky vdiff2 e2e test. Skip flaky Migrate test. (vitessio#18300) (vitessio#18334) Signed-off-by: Rohit Nayak <rohit@planetscale.com> Co-authored-by: vitess-bot[bot] <108069721+vitess-bot[bot]@users.noreply.github.com> * [release-22.0] Throttler: keep watching topo even on error (vitessio#18223) (vitessio#18322) Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> Co-authored-by: vitess-bot[bot] <108069721+vitess-bot[bot]@users.noreply.github.com> * [release-22.0] Code Freeze for `v22.0.1` (vitessio#18374) Signed-off-by: Manan Gupta <manan@planetscale.com> * [release-22.0] Release of `v22.0.1` (vitessio#18375) Signed-off-by: Manan Gupta <manan@planetscale.com> * add private repo config to new CI file Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * `make generate_ci_workflows` Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> --------- Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr> Signed-off-by: Harshit Gangal <harshit@planetscale.com> Signed-off-by: Arthur Schreiber <arthurschreiber@github.com> Signed-off-by: Mohamed Hamza <mhamza15@github.com> Signed-off-by: Andres Taylor <andres@planetscale.com> Signed-off-by: Manan Gupta <manan@planetscale.com> Signed-off-by: GitHub <noreply@github.com> Signed-off-by: Rohit Nayak <rohit@planetscale.com> Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> Co-authored-by: vitess-bot <139342327+vitess-bot@users.noreply.github.com> Co-authored-by: Florent Poinsard <florent.poinsard@outlook.fr> Co-authored-by: vitess-bot[bot] <108069721+vitess-bot[bot]@users.noreply.github.com> Co-authored-by: Harshit Gangal <harshit@planetscale.com> Co-authored-by: Mohamed Hamza <mhamza15@github.com> Co-authored-by: Andrés Taylor <andres@planetscale.com> Co-authored-by: frouioui <frouioui@users.noreply.github.com> Co-authored-by: Manan Gupta <manan@planetscale.com> Co-authored-by: Manan Gupta <guptamanan100@gmail.com>
* support mysql protocol connection attributes (vitessio#18548) Signed-off-by: Michael Demmer <mdemmer@slack-corp.com> * vtgateconn minor enhancements (vitessio#18551) Signed-off-by: Michael Demmer <mdemmer@slack-corp.com> * add DialCustom for overridden runtime behavior Signed-off-by: Michael Demmer <mdemmer@slack-corp.com> --------- Signed-off-by: Michael Demmer <mdemmer@slack-corp.com>
…ERS` (#709) * `slack-22.0`: setup `slack_cross_cell` durability policy + `CODEOWNERS` Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * fix path Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * fix package Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * add `GOPRIVATE` to docker Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * fix private repo in docker builds Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * update docker files Signed-off-by: Tanjin Xu <tanjin.xu@slack-corp.com> * update docker test Signed-off-by: Tanjin Xu <tanjin.xu@slack-corp.com> * fix vitess-addons Signed-off-by: Tanjin Xu <tanjin.xu@slack-corp.com> * undo pre release version change Signed-off-by: Tanjin Xu <tanjin.xu@slack-corp.com> * fix vitess-addon Signed-off-by: Tanjin Xu <tanjin.xu@slack-corp.com> * update vitess-addon Signed-off-by: Tanjin Xu <tanjin.xu@slack-corp.com> * update go version for v19 build Signed-off-by: Tanjin Xu <tanjin.xu@slack-corp.com> * update vitess-addons Signed-off-by: Tanjin Xu <tanjin.xu@slack-corp.com> * update vitess-addons Signed-off-by: Tanjin Xu <tanjin.xu@slack-corp.com> * fix vitess-addon Signed-off-by: Tanjin Xu <tanjin.xu@slack-corp.com> * Revert "fix vitess-addon" This reverts commit 3fc0d3f84bd04a59f9f9e4d7aa26f9ca3dc8b268. * update test Signed-off-by: Tanjin Xu <tanjin.xu@slack-corp.com> * update test Signed-off-by: Tanjin Xu <tanjin.xu@slack-corp.com> * fix help message Signed-off-by: Tanjin Xu <tanjin.xu@slack-corp.com> * Fix regression in v22 around new flag setup (vitessio#18507) Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> Signed-off-by: Tanjin Xu <tanjin.xu@slack-corp.com> * fix help message Signed-off-by: Tanjin Xu <tanjin.xu@slack-corp.com> * fix help message Signed-off-by: Tanjin Xu <tanjin.xu@slack-corp.com> * [release-22.0] CI: Fix `VDiff2` flaky e2e test (vitessio#18494) (vitessio#18526) Signed-off-by: Noble Mittal <noblemittal@outlook.com> Co-authored-by: vitess-bot[bot] <108069721+vitess-bot[bot]@users.noreply.github.com> * skip few ci tests Signed-off-by: Tanjin Xu <tanjin.xu@slack-corp.com> * disable one query in v19 downgrade testing Signed-off-by: Tanjin Xu <tanjin.xu@slack-corp.com> * update test Signed-off-by: Tanjin Xu <tanjin.xu@slack-corp.com> * update test Signed-off-by: Tanjin Xu <tanjin.xu@slack-corp.com> --------- Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> Signed-off-by: Tanjin Xu <tanjin.xu@slack-corp.com> Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> Signed-off-by: Noble Mittal <noblemittal@outlook.com> Co-authored-by: Tim Vaillancourt <tim@timvaillancourt.com> Co-authored-by: Dirkjan Bussink <d.bussink@gmail.com> Co-authored-by: vitess-bot[bot] <108069721+vitess-bot[bot]@users.noreply.github.com>
Review ChecklistHello reviewers! 👋 Please follow this checklist when reviewing this Pull Request. General
Tests
Documentation
New flags
If a workflow is added or modified:
Backward compatibility
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7d42a97d09
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| nonOnlineDDLStopped := make([]*VReplicationStream, 0, len(stoppedStreams)) | ||
| for _, stream := range stoppedStreams { | ||
| if stream.WorkflowType != binlogdatapb.VReplicationWorkflowType_OnlineDDL { | ||
| nonOnlineDDLStopped = append(nonOnlineDDLStopped, stream) |
There was a problem hiding this comment.
Limit the exemption to completed OnlineDDL cutovers
If OnlineDDL setup fails after generateInsertStatement creates its initially Stopped stream but before generateStartStatement runs, this type-only filter treats that orphan as a legitimate cutover artifact even though it does not have the stopped for online DDL cutover message. readSourceStreams then includes the orphan in the full stream set, and the SwitchWrites flow can migrate and eventually restart it on the target shards rather than refusing the reshard as before. Restrict this exemption to OnlineDDL streams carrying the cutover message, matching the legacy path.
Useful? React with 👍 / 👎.
| @@ -452,7 +452,7 @@ func (sm *StreamMigrator) legacyReadSourceStreams(ctx context.Context, cancelMig | |||
| // If so, we request the operator to clean them up, or restart them before going ahead. | |||
| // This allows us to assume that all stopped streams can be safely restarted | |||
| // if we cancel the operation. | |||
| stoppedStreams, err := sm.legacyReadTabletStreams(ctx, source.GetPrimary(), "state = 'Stopped' and message != 'FROZEN'") | |||
| stoppedStreams, err := sm.legacyReadTabletStreams(ctx, source.GetPrimary(), "state = 'Stopped' and message != 'FROZEN' and message != 'stopped for online DDL cutover'") | |||
There was a problem hiding this comment.
Remove cutover artifacts before templatizing streams
When the completed OnlineDDL entry this exemption is meant to permit is present, it is excluded only from the stopped-stream guard; the subsequent unconstrained read still adds it to tabletStreams. OnlineDDL rules match the generated internal shadow-table name and their filter has no in_keyrange, so BuildStreamMigrator passes the entry to templatizeKeyRange, which fails to find that internal table in SourceKeyspaceSchema().Tables and dereferences the nil vtable. Thus the reported recent-OnlineDDL scenario can turn SwitchWrites from a validation error into a panic; exclude these completed cutover entries from the stream set being migrated in both read paths.
Useful? React with 👍 / 👎.
|
Closing — branch was based on slack-22.0 and included unrelated commits. Re-opening with a clean single-commit branch based on main. |
Description
stream_migrator.readSourceStreamsblocks SwitchWrites when it finds vreplication streams instate='Stopped'that aren't'FROZEN'. This is intended to catch orphaned/problematic streams, but it also catches OnlineDDL cutover entries which legitimately remain Stopped (with message "stopped for online DDL cutover") for up to 24h beforegcArtifacts()cleans them up.This causes reshard SwitchWrites to fail if any OnlineDDL ran recently on the source shards.
Fixes #20838
Fix
and message != 'stopped for online DDL cutover'to the stopped-streams queryWorkflowType_OnlineDDLstreams from the stopped checkChecklist