What did you do?
Wanted to know what happens to a changefeed when one capture is alive as far as PD is concerned — process up, lease renewing, listed in capture list — but cannot actually talk to its peers. It turns out the changefeed's checkpoint stops advancing and never resumes, while the changefeed continues to report state=normal with no error, for as long as the condition lasts. In the test that was over 30 minutes; an earlier run of ours sat like that for about nine hours.
Steps to Reproduce
Cluster: 4 TiCDC captures (new arch), 3 changefeeds to Kafka, 12 tables. One table was pre-split so that it distributes across all captures:
-- ~12,000 regions, mostly empty, so the table exceeds the default region-threshold of 10000
-- (1000-region cap per statement, and each region must span >= 1000 key values)
SPLIT TABLE batches BETWEEN (0) AND (2000000) REGIONS 1000;
SPLIT TABLE batches BETWEEN (2000000) AND (4000000) REGIONS 1000;
-- and so on in 2,000,000-wide steps, twelve statements in total, up to 24000000
That left batches-test with 1,004 spans spread across the four captures (roughly 155-198 dispatchers each). Changefeed settings were left at defaults except enable-table-across-nodes = true
Then, on a capture that held work but was neither the coordinator nor any changefeed's maintainer:
- Restart it, so it rejoins quickly and is seen as an add rather than a departure.
- As soon as it is back, drop only its peer port at the host level, leaving PD, TiKV and Kafka reachable:
iptables -I FORWARD 1 -s -p tcp --dport 8300 -j DROP
iptables -I FORWARD 1 -s -p tcp --sport 8300 -j DROP
iptables -I FORWARD 1 -d -p tcp --dport 8300 -j DROP
Timeline
20:17:47.427 deleting pod ticdc-0
20:17:49.778 maintainer node changed addedCount=1 (the capture rejoins)
20:17:52.536 DROP applied to 192.168.185.191 (port 8300 only)
The important part is that this is a selective failure. Because only 8300 is blocked, the capture keeps renewing its PD lease and stays a cluster member forever. I confirmed that from inside the isolated pod — it still had 7 established connections to PD on 2379, 36 to TiKV on 20160, and 9 to Kafka on 9092 — and it kept writing rows to the sink at about 180/s throughout.
What did you expect to see?
Either the spans on the unreachable capture get rescheduled and the checkpoint carries on, or — if that can't be done safely — the changefeed goes into warning so that changefeed-error-stuck-duration starts counting and somebody gets paged.
What did you see instead?
The checkpoints froze one second before the fault landed and never moved again. Thirty minutes later all three changefeeds still reported state=normal with an empty error field:
batches-test state=normal ckpt=2026-07-27 20:17:46.736 frozen_for=0:30:30 err=none
item-issue-test state=normal ckpt=2026-07-27 20:18:15.635 frozen_for=0:30:01 err=none
item-test state=normal ckpt=2026-07-27 20:17:46.736 frozen_for=0:30:30 err=none
Because the changefeed never leaves normal, changefeed-error-stuck-duration never starts — it isn't that 1800s elapsed and the timer was too slow, the timer was never running.
The maintainers do notice, loudly. This line repeated 53,518 times in 30 minutes (about 1,830/min, roughly every 100ms per changefeed):
[2026/07/27 20:30:00.080 +00:00] [WARN] [maintainer.go:706] ["checkpointTs can not be advanced, since missing capture heartbeat"] [changefeedID=default/item-test] [node=8e2b7486-bafc-44cf-9684-31f169efb5c6] [checkpointTs=467975133104701482] [resolvedTs=467975133104701482]
[2026/07/27 20:30:00.085 +00:00] [WARN] [maintainer.go:706] ["checkpointTs can not be advanced, since missing capture heartbeat"] [changefeedID=default/item-issue-test] [node=8e2b7486-bafc-44cf-9684-31f169efb5c6] [checkpointTs=467975134598922270] [resolvedTs=467975134598922270]
The blocked capture is not silent or crashed — it reports the broken link continuously, naming both ends by ID and address. In the same 30-minute window:
[2026/07/27 20:30:00.030 +00:00] [ERROR] [heartbeat_collector.go:228] ["failed to send heartbeat request message"] [error="ErrorType: ConnectionNotFound, Reason: Stream not ready, local: 8e2b7486-bafc-44cf-9684-31f169efb5c6, localAddr: ticdc-0.ticdc-ng-peer.tidb-cluster.svc:8300, target: 95fd454a-090e-45b1-ab68-e68e4e5c5de9, targetAddr: ticdc-3.ticdc-ng-peer.tidb-cluster.svc:8300"]
[2026/07/27 20:30:00.467 +00:00] [WARN] [metrics_collector.go:369] ["send changefeed metrics to coordinator failed"] [error="ErrorType: ConnectionNotFound, Reason: Stream not ready, local: 8e2b7486-bafc-44cf-9684-31f169efb5c6, localAddr: ticdc-0.ticdc-ng-peer.tidb-cluster.svc:8300, target: 95fd454a-090e-45b1-ab68-e68e4e5c5de9, targetAddr: ticdc-3.ticdc-ng-peer.tidb-cluster.svc:8300"]
[2026/07/27 20:30:02.377 +00:00] [ERROR] [remote_target.go:370] ["Failed to connect to remote target"] [error="ErrorType: ConnectionFailed, Reason: Cannot open bidirectional grpc stream, error: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 192.168.138.27:8300: i/o timeout\""]
[2026/07/27 20:30:09.741 +00:00] [WARN] [event_store.go:1579] ["send broadcast message to coordinator failed"] [error="ErrorType: ConnectionNotFound, Reason: Stream not ready, local: 8e2b7486-bafc-44cf-9684-31f169efb5c6, localAddr: ticdc-0.ticdc-ng-peer.tidb-cluster.svc:8300, target: 95fd454a-090e-45b1-ab68-e68e4e5c5de9, targetAddr: ticdc-3.ticdc-ng-peer.tidb-cluster.svc:8300"]
Note checkpointTs and resolvedTs are the same value in every one of those lines.
The scheduler also knows something is wrong — 1,419 of these, one every 30s per operator, and they never expire:
2026/07/27 20:40:26.146 +00:00] [WARN] [operator_controller.go:268] ["operator is still in running queue"] [changefeedID=default/item-test] [operator=40533823322966678649904880314334846888] [operator="add dispatcher operator: 40533823322966678649904880314334846888, dest:8e2b7486-bafc-44cf-9684-31f169efb5c6"] [timeSinceCreated=22m32.756193537s]
[2026/07/27 20:40:10.370 +00:00] [WARN] [operator_controller.go:268] ["operator is still in running queue"] [changefeedID=default/item-issue-test] [operator=809249526894596825515335929580816404126] [operator="move dispatcher operator: 809249526894596825515335929580816404126, origin:95fd454a-090e-45b1-ab68-e68e4e5c5de9, dest:8e2b7486-bafc-44cf-9684-31f169efb5c6"] [timeSinceCreated=21m2.729721682s]
Everything the three healthy captures logged at WARN or ERROR during the stall (window 20:17:52 → 20:49:06 UTC, 31m14s, port 8300 blocked only):
▎ 55420 WARN maintainer.go:706 | checkpointTs can not be advanced, since missing capture heartbeat
▎ 1463 WARN operator_controller.go:268 | operator is still in running queue
▎ 130 WARN event_store.go:449 | register dispatcher with large startTs lag
▎ 83 WARN region_event_handler.go:404 | resolved ts advance step is too large
▎ 27 WARN region_request_worker.go:280 | region request worker receives a region event for an untracked region
▎ 25 WARN region_request_worker.go:334 | region request worker receives a resolved ts event for an untracked region
▎ 12 WARN message_center.go:175 | remote target error, reset the connection
▎ 8 WARN lock_resolver.go:730 | lock txn not found, lock has expired
▎ 6 ERROR remote_target.go:625 | Error receiving message
▎ 1 WARN metrics_collector.go:326 | slow dispatcher by checkpointTs
▎ 1 WARN maintainer_controller.go:184 | nodeID not match
And from the isolated capture over the same window:
▎ 18372 ERROR heartbeat_collector.go:228 | failed to send heartbeat request message
▎ 1841 WARN metrics_collector.go:369 | send changefeed metrics to coordinator failed
▎ 613 WARN message_center.go:175 | remote target error, reset the connection
▎ 612 ERROR remote_target.go:420 | Failed to connect to remote target
▎ 612 ERROR remote_target.go:370 | Failed to connect to remote target
▎ 184 WARN event_store.go:1579 | send broadcast message to coordinator failed
timeSinceCreated climbs past 22 minutes without anything acting on it. Worth mentioning that the API-driven paths in maintainer/maintainer_controller_helper.go do bound the very same operator types — maxTry := 30 with a 2s sleep, then errors.ErrTimeout ("move table operator is timeout") — so a 60s deadline already exists for MoveTable, MoveSplitTable, SplitTableByRegionCount and MergeTable when a human or the API asks. It just isn't applied when the scheduler issues the same operator itself.
ticdc_owner_checkpoint_ts_lag tracks the truth (and matches what the API reports, 376s vs 377.9s at the time). ticdc_maintainer_checkpoint_ts_lag is stuck on a healthy-looking 12.6s and never moves again.
Separately, ticdc_owner_resolved_ts_lag reported a healthy 4-6.5s for the entire stall.
This combination is actively misleading. "Checkpoint lag climbing, resolved lag healthy" reads like the classic slow-sink signature — resolved runs ahead, checkpoint trails — so the natural conclusion is that Kafka is the bottleneck. It isn't. Both watermarks are equally frozen and the sink is fine.
Versions of the cluster
Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):
TiDB v8.5.5 Enterprise 1fa258b8, TiKV 8.5.5 Enterprise a0234aad, TiCDC v8.5.7-release.1 e7e1f614
Upstream TiKV version (execute tikv-server --version):
TiKV 8.5.5 Enterprise a0234aad
TiCDC version (execute cdc version):
TiCDC v8.5.7-release.1 e7e1f614
What did you do?
Wanted to know what happens to a changefeed when one capture is alive as far as PD is concerned — process up, lease renewing, listed in capture list — but cannot actually talk to its peers. It turns out the changefeed's checkpoint stops advancing and never resumes, while the changefeed continues to report state=normal with no error, for as long as the condition lasts. In the test that was over 30 minutes; an earlier run of ours sat like that for about nine hours.
Steps to Reproduce
Cluster: 4 TiCDC captures (new arch), 3 changefeeds to Kafka, 12 tables. One table was pre-split so that it distributes across all captures:
-- ~12,000 regions, mostly empty, so the table exceeds the default region-threshold of 10000
-- (1000-region cap per statement, and each region must span >= 1000 key values)
SPLIT TABLE batches BETWEEN (0) AND (2000000) REGIONS 1000;
SPLIT TABLE batches BETWEEN (2000000) AND (4000000) REGIONS 1000;
-- and so on in 2,000,000-wide steps, twelve statements in total, up to 24000000
That left batches-test with 1,004 spans spread across the four captures (roughly 155-198 dispatchers each). Changefeed settings were left at defaults except enable-table-across-nodes = true
Then, on a capture that held work but was neither the coordinator nor any changefeed's maintainer:
iptables -I FORWARD 1 -s -p tcp --dport 8300 -j DROP
iptables -I FORWARD 1 -s -p tcp --sport 8300 -j DROP
iptables -I FORWARD 1 -d -p tcp --dport 8300 -j DROP
Timeline
20:17:47.427 deleting pod ticdc-0
20:17:49.778 maintainer node changed addedCount=1 (the capture rejoins)
20:17:52.536 DROP applied to 192.168.185.191 (port 8300 only)
The important part is that this is a selective failure. Because only 8300 is blocked, the capture keeps renewing its PD lease and stays a cluster member forever. I confirmed that from inside the isolated pod — it still had 7 established connections to PD on 2379, 36 to TiKV on 20160, and 9 to Kafka on 9092 — and it kept writing rows to the sink at about 180/s throughout.
What did you expect to see?
Either the spans on the unreachable capture get rescheduled and the checkpoint carries on, or — if that can't be done safely — the changefeed goes into warning so that changefeed-error-stuck-duration starts counting and somebody gets paged.
What did you see instead?
The checkpoints froze one second before the fault landed and never moved again. Thirty minutes later all three changefeeds still reported state=normal with an empty error field:
batches-test state=normal ckpt=2026-07-27 20:17:46.736 frozen_for=0:30:30 err=none
item-issue-test state=normal ckpt=2026-07-27 20:18:15.635 frozen_for=0:30:01 err=none
item-test state=normal ckpt=2026-07-27 20:17:46.736 frozen_for=0:30:30 err=none
Because the changefeed never leaves normal, changefeed-error-stuck-duration never starts — it isn't that 1800s elapsed and the timer was too slow, the timer was never running.
The maintainers do notice, loudly. This line repeated 53,518 times in 30 minutes (about 1,830/min, roughly every 100ms per changefeed):
The blocked capture is not silent or crashed — it reports the broken link continuously, naming both ends by ID and address. In the same 30-minute window:
Note checkpointTs and resolvedTs are the same value in every one of those lines.
The scheduler also knows something is wrong — 1,419 of these, one every 30s per operator, and they never expire:
Everything the three healthy captures logged at WARN or ERROR during the stall (window 20:17:52 → 20:49:06 UTC, 31m14s, port 8300 blocked only):
And from the isolated capture over the same window:
timeSinceCreated climbs past 22 minutes without anything acting on it. Worth mentioning that the API-driven paths in maintainer/maintainer_controller_helper.go do bound the very same operator types — maxTry := 30 with a 2s sleep, then errors.ErrTimeout ("move table operator is timeout") — so a 60s deadline already exists for MoveTable, MoveSplitTable, SplitTableByRegionCount and MergeTable when a human or the API asks. It just isn't applied when the scheduler issues the same operator itself.
ticdc_owner_checkpoint_ts_lag tracks the truth (and matches what the API reports, 376s vs 377.9s at the time). ticdc_maintainer_checkpoint_ts_lag is stuck on a healthy-looking 12.6s and never moves again.
Separately, ticdc_owner_resolved_ts_lag reported a healthy 4-6.5s for the entire stall.
This combination is actively misleading. "Checkpoint lag climbing, resolved lag healthy" reads like the classic slow-sink signature — resolved runs ahead, checkpoint trails — so the natural conclusion is that Kafka is the bottleneck. It isn't. Both watermarks are equally frozen and the sink is fine.
Versions of the cluster
Upstream TiDB cluster version (execute
SELECT tidb_version();in a MySQL client):TiDB v8.5.5 Enterprise 1fa258b8, TiKV 8.5.5 Enterprise a0234aad, TiCDC v8.5.7-release.1 e7e1f614Upstream TiKV version (execute
tikv-server --version):TiKV 8.5.5 Enterprise a0234aadTiCDC version (execute
cdc version):TiCDC v8.5.7-release.1 e7e1f614