-
Notifications
You must be signed in to change notification settings - Fork 2.4k
Don't manage --unmanaged tablets
#20841
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
6bb77cf
0545ed4
fc59fde
0d536f0
acc0803
179e985
0bf665a
f1a9f58
4c81fec
eb8eee5
1b62c5c
0dd93e2
1e37b9a
f83e70c
d02da86
0354226
a5b49b7
7985996
8d0b3ea
6c87d19
487250f
55d7b4f
5b58bc9
c71856f
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -32,7 +32,11 @@ | |
| - [`EmergencyReparentShard` no longer waits on replicas that cannot win the election](#ers-lagging-relay-log-wait) | ||
| - [`EmergencyReparentShard` can explicitly recover from split brain](#ers-allow-split-brain-promotion) | ||
| - [Reparent candidate ordering now respects partially ordered GTID histories](#reparent-gtid-candidate-ordering) | ||
| - [Reparents refuse shards containing unmanaged tablets](#reparent-unmanaged-tablets) | ||
| - **[VTOrc](#minor-changes-vtorc)** | ||
| - [VTOrc no longer watches `--unmanaged` tablets](#vtorc-unmanaged-tablets) | ||
| - **[VTTablet](#minor-changes-vttablet)** | ||
| - [Unmanaged tablets reject replication and reparent RPCs](#vttablet-unmanaged-rpc-guard) | ||
| - [Consolidator Reject on Waiter Cap](#vttablet-consolidator-reject-on-cap) | ||
| - [Query timeout for state-changing statements on the streaming path](#vttablet-stream-query-timeout) | ||
| - [Query rules now apply to queries on the streaming path](#vttablet-rules-apply-to-streaming) | ||
|
|
@@ -298,8 +302,26 @@ Candidates are now ordered by GTID dominance before the existing promotion-rule, | |
|
|
||
| See [#20579](https://github.com/vitessio/vitess/issues/20579). | ||
|
|
||
| #### <a id="reparent-unmanaged-tablets"/>Reparents refuse shards containing unmanaged tablets</a> | ||
|
|
||
| `EmergencyReparentShard` and `PlannedReparentShard` now error, before issuing any RPC, if a tablet in the shard was started with `--unmanaged`. Vitess cannot revoke writes from or repoint a tablet it does not manage, so ERS cannot guarantee no other tablet still accepts writes, and PRS would promote the new primary and only then report failure. This only affects a shard mixing unmanaged and managed tablets, which is not a supported layout; a tablet record written before this release reports itself as managed, so a shard part-way through an upgrade is unaffected. | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
The claim that this only affects mixed shards is incorrect: AGENTS.md reference: AGENTS.md:L231-L234 Useful? React with 👍 / 👎. |
||
|
|
||
| ### <a id="minor-changes-vtorc"/>VTOrc</a> | ||
|
|
||
| #### <a id="vtorc-unmanaged-tablets"/>VTOrc no longer watches `--unmanaged` tablets</a> | ||
|
|
||
| A `vttablet` started with `--unmanaged` now records that in its topology record, via a new `mysql_mode` field, and VTOrc skips those tablets entirely: never written to its backend, never probed, analysed or recovered. Previously nothing outside the `vttablet` process knew a tablet was unmanaged, so VTOrc, which filters only on `--clusters-to-watch`, would flag one replicating from outside Vitess (`ReplicaIsWritable`, `NotConnectedToPrimary`) and "repair" it, issuing `SetReadOnly` and `SetReplicationSource` against a MySQL it had been told not to manage. | ||
|
|
||
| **Impact**: these tablets disappear from VTOrc's API, UI and tablet counts. The `--clusters-to-watch` exclusions used to carve them out are only safe to remove once every unmanaged `vttablet` and every VTOrc watching them run v25, since a tablet only declares itself unmanaged when it restarts, a record written by an older `vttablet` reads back as managed, and an older VTOrc ignores the field entirely. Restore the exclusions before downgrading either component. | ||
|
|
||
| ### <a id="minor-changes-vttablet"/>VTTablet</a> | ||
|
|
||
| #### <a id="vttablet-unmanaged-rpc-guard"/>Unmanaged tablets reject replication and reparent RPCs</a> | ||
|
|
||
| A `vttablet` started with `--unmanaged` now refuses the tabletmanager RPCs that would change replication on its external MySQL, or promote or demote it, returning `FAILED_PRECONDITION` to every caller. Those are `StopReplication`, `StopReplicationMinimum`, `StartReplication`, `RestartReplication`, `StartReplicationUntilAfter`, `ResetReplication`, `ResetReplicationParameters`, `SetReplicationSource`, `StopReplicationAndGetStatus`, `InitPrimary`, `InitReplica`, `PopulateReparentJournal`, `DemotePrimary`, `UndoDemotePrimary`, `PromoteReplica`, `SetReadOnly` and `SetReadWrite`. Read-only RPCs are unaffected, and `ChangeType` stays available so `TabletExternallyReparented` keeps working, though it now only moves the topology record rather than configuring semi-sync or restarting replication. | ||
|
|
||
| **Impact**: anyone deliberately driving these RPCs against an unmanaged tablet, for example via `vtctldclient`, now gets an error instead of a silent change to the external MySQL. | ||
|
|
||
| #### <a id="vttablet-consolidator-reject-on-cap"/>Consolidator Reject on Waiter Cap</a> | ||
|
|
||
| A new `--consolidator-reject-on-cap` flag (default `false`) has been added to VTTablet. When enabled alongside a non-zero `--consolidator-query-waiter-cap`, queries that would join a consolidated result but exceed the **global** consolidator waiter cap are rejected with a `RESOURCE_EXHAUSTED` error instead of silently falling back to independent MySQL execution. | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -8,6 +8,7 @@ The goals, rules and limitations of ERS. Changes to `go/vt/vtctl/reparentutil` m | |
| - ERS runs while the shard cannot accept writes: it must resolve the availability loss, and every stage is time-bound (`--wait-replicas-timeout` for replica waits, `topo.RemoteOperationTimeout` for primary RPCs) — each added stage extends the total outage. Changes should prioritize reducing points of failure - avoid new RPCs or work that may delay or make ERS more brittle | ||
| - ERS holds the shard lock for its entire run and re-checks it at every phase boundary (`CheckShardLocked`), aborting on loss; it is the only guard against two concurrent reparents. Any new phase must re-check the lock before acting | ||
| - ERS must error if a shard contains a mix of GTID-based and non-GTID-based replication. Their position semantics differ (`Combined` = retrieved+executed for GTID vs. executed-only for non-GTID), so a unified split-brain / most-advanced check across both is unsafe | ||
| - ERS must error if any tablet in the shard reports `TabletMySQLMode_UNMANAGED`, and must do so before issuing its first RPC. Vitess cannot stop replication on, or revoke writes from, a tablet it does not manage, so `haveRevoked` cannot be made truthful for such a shard and a promotion would leave a second writer that cannot be revoked. Unmanaged tablets belong in a keyspace of their own; a tablet record written before `TabletMySQLMode` existed reports `MANAGED`, so a shard part-way through an upgrade is unaffected | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
A shard part-way through an upgrade is not universally unaffected: once an unmanaged vttablet has restarted on v25 and published AGENTS.md reference: AGENTS.md:L240-L245 Useful? React with 👍 / 👎. |
||
| - ERS must never promote a tablet with received-but-unapplied transactions. Replica candidates establish this by completing a relay-log apply wait; a demoted former primary is exempt, since it was not replicating its received and executed positions are the same | ||
| - For non-GTID flavors, ERS must wait on every candidate and fail on any error. The "filter to leading group + short-circuit on first success" optimization is only safe for GTID-based flavors, where `Combined` is distinct from the executed position | ||
| - Candidates at the same received (`Combined`) position converge to identical state once their relay logs are applied, which is what makes short-circuiting the relay-log wait on the first success safe. Leading candidates with incomparable positions are a suspected split brain and must all be waited on — a failed one must not be dropped before the split-brain check sees it. A split-brain promotion override instead waits only for the chosen `--new-primary` to finish applying its relay logs; the discarded leaders are not waited on, since their divergent branches are excluded from the promoted history, so a wedged losing branch cannot block the recovery the override exists for | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This release note says only
EmergencyReparentShardandPlannedReparentShardreject mixed unmanaged shards, but the same commit adds a preflight toInitShardPrimaryLocked, soInitShardPrimarynow also returns an error where it previously proceeded. Operators need that command-level behavior change called out explicitly rather than discovering it during shard initialization.AGENTS.md reference: AGENTS.md:L231-L234
Useful? React with 👍 / 👎.