Overview
During a reshard SwitchWrites operation, stream_migrator.readSourceStreams blocks if it finds any vreplication stream in state='Stopped' that isn't 'FROZEN'. This is intended to catch orphaned streams, but it also catches OnlineDDL entries that legitimately remain Stopped after a successful cutover.
Root Cause
After a successful OnlineDDL cutover, the executor calls binlogplayer.StopVReplication(id, "stopped for online DDL cutover"), setting the stream to state='Stopped'. These entries remain for up to 24h until gcArtifacts() calls deleteVReplicationEntry().
During this window, if a reshard SwitchWrites is attempted on the same shard, the stream migrator finds these Stopped entries and refuses to proceed with:
cannot migrate until all streams are running: <shard>: <uid>
Impact
Any keyspace that runs OnlineDDL schema changes cannot complete a reshard SwitchWrites for up to 24h after the last OnlineDDL cutover, unless the operator manually cleans up the stopped entries with OnlineDDL <keyspace> cleanup <uuid>.
Reproduction
- Run an OnlineDDL migration on a sharded keyspace
- Wait for cutover to complete successfully
- Attempt a reshard SwitchWrites on the same source shards within 24h
- Observe the "cannot migrate until all streams are running" error
Suggested Fix
PR: #20837
Overview
During a reshard SwitchWrites operation,
stream_migrator.readSourceStreamsblocks if it finds any vreplication stream instate='Stopped'that isn't'FROZEN'. This is intended to catch orphaned streams, but it also catches OnlineDDL entries that legitimately remain Stopped after a successful cutover.Root Cause
After a successful OnlineDDL cutover, the executor calls
binlogplayer.StopVReplication(id, "stopped for online DDL cutover"), setting the stream tostate='Stopped'. These entries remain for up to 24h untilgcArtifacts()callsdeleteVReplicationEntry().During this window, if a reshard SwitchWrites is attempted on the same shard, the stream migrator finds these Stopped entries and refuses to proceed with:
Impact
Any keyspace that runs OnlineDDL schema changes cannot complete a reshard SwitchWrites for up to 24h after the last OnlineDDL cutover, unless the operator manually cleans up the stopped entries with
OnlineDDL <keyspace> cleanup <uuid>.Reproduction
Suggested Fix
PR: #20837