Skip to content

SwitchWrites blocked by OnlineDDL stopped vreplication streams #20838

Description

@yushuqin

Overview

During a reshard SwitchWrites operation, stream_migrator.readSourceStreams blocks if it finds any vreplication stream in state='Stopped' that isn't 'FROZEN'. This is intended to catch orphaned streams, but it also catches OnlineDDL entries that legitimately remain Stopped after a successful cutover.

Root Cause

After a successful OnlineDDL cutover, the executor calls binlogplayer.StopVReplication(id, "stopped for online DDL cutover"), setting the stream to state='Stopped'. These entries remain for up to 24h until gcArtifacts() calls deleteVReplicationEntry().

During this window, if a reshard SwitchWrites is attempted on the same shard, the stream migrator finds these Stopped entries and refuses to proceed with:

cannot migrate until all streams are running: <shard>: <uid>

Impact

Any keyspace that runs OnlineDDL schema changes cannot complete a reshard SwitchWrites for up to 24h after the last OnlineDDL cutover, unless the operator manually cleans up the stopped entries with OnlineDDL <keyspace> cleanup <uuid>.

Reproduction

  1. Run an OnlineDDL migration on a sharded keyspace
  2. Wait for cutover to complete successfully
  3. Attempt a reshard SwitchWrites on the same source shards within 24h
  4. Observe the "cannot migrate until all streams are running" error

Suggested Fix

PR: #20837

Metadata

Metadata

Assignees

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions