Bug Report
What did you do?
Created a changefeed on three TiCDC captures for 32 tables with 6,248 Regions
per table (199,936 record Regions in total). The changefeed used a fresh
start-ts, so the initial scans were inside the default scan-priority lag
threshold.
The bootstrap was tested twice on exact master
d480b05fa79b596204b2e881c7184a5f2902ffa2. The second attempt cold-restarted
all three TiCDC captures before creating the changefeed.
Other controlled settings:
- one TiKV store;
- TiDB
min-ts-interval = 200ms;
- MySQL dry-run sink with 50ms simulated latency;
- no DML workload during bootstrap;
- default TiCDC
pending-region-request-queue-size = 32;
- default TiKV
cdc.incremental-scan-concurrency-limit = 10000.
What did you expect to see?
The initial scans should remain bounded by TiCDC's pending-Region flow control,
finish registering all Regions, and let resolved-ts/checkpoint-ts advance past
the changefeed start-ts.
Historical master 369674c75814928e5a3f1122fc7520c370033571
completed the same 199,936-Region bootstrap in 206.9 seconds.
What did you see instead?
Both exact-master attempts failed to advance past start-ts within 20 minutes.
The controlled second attempt ran from
2026-07-25 04:27:36.633 to 04:47:12.818 CST.
Metrics in that window:
| Metric |
Observed value |
ticdc_subscription_client_requested_region_count{state="pending"} |
aggregate peak 198,320 |
increase(ticdc_kvclient_event_feed_error_count{type="KvIsBusy"}[19m]) |
40.69m / 35.93m / 41.09m on the three captures; about 119.6m total |
Other ticdc_kvclient_event_feed_error_count types |
0 |
| TiKV unresolved CDC Regions |
exactly 10,000 |
| TiKV captured CDC Regions at the end |
about 15,507 |
| LogPuller Region matchers |
about 200,397 |
| EventStore resolved input |
0 |
| Successful Region scan progress during the storm |
about 4-5 Regions/s |
For comparison, two successful builds with the same topology had zero
KvIsBusy, reached about 200,099 captured Regions with zero unresolved
Regions, and peaked at only 89 and 96 aggregate pending Region requests.
At default INFO level, TiCDC records cdc region error only at DEBUG, so the
ServerIsBusy reason is not printed in TiCDC logs. The error counter above is
the direct TiCDC signal. TiKV's 300 MiB log files rotated roughly every
12 seconds during the retry storm, dominated by per-Region incremental-scan
messages such as:
[2026/07/25 04:44:52.763 +08:00] [INFO] [initializer.rs:542]
["cdc has scanned all incremental scan locks"] ... [region_id=562097]
[2026/07/25 04:44:52.764 +08:00] [INFO] [initializer.rs:388]
["cdc async incremental scan finished"] ... [region_id=562097]
TiKV v8.5.5 rejects registrations after the configured/default 10,000
incremental-scan tasks with:
ServerIsBusy("too many pending incremental scans")
Suspected cause
The behavior is introduced by scan-priority commit d8b039fd5 / PR #5480:
- A recent start-ts makes initial subscription scans high priority.
subscription_client.go derives
force := regionTask.Priority() <= forcedPriorityBase.
requestCache.add(..., force=true) bypasses the
pendingCount < maxPendingCount flow-control check.
region_failure_handler.go preserves the inherited high priority for
ServerIsBusy and Congested.
- The rejected task is immediately requeued as high priority, bypasses flow
control again, and has no backoff or jitter.
The deployed TiKV v8.5.5 uses a kvproto version that does not consume the new
ChangeDataRequest.scan_priority field. Therefore the intended TiKV-side
priority behavior is unavailable, while TiCDC's local force and retry
behavior still take effect.
The full-span initialization change in dd07f601b makes the frozen start-ts
visible and correct, but does not create the request storm.
Suggested fix
Priority and local admission bypass should be separate decisions:
- initial subscription scans may carry high scan priority to TiKV, but should
still honor the per-store pending-Region limit;
ServerIsBusy and Congested retries should be demoted behind normal
flow control and use bounded backoff with jitter;
force=true should be reserved for a small, bounded class of recovery or
control work, not all recent-start initial scans.
As a narrow hotfix, demoting ServerIsBusy/Congested retries to
TaskLowPrior should break the positive feedback loop. It is not a complete
fix by itself because the first high-priority bootstrap burst can still bypass
the local pending limit before TiKV returns the first busy response.
Versions
- TiCDC: master
d480b05fa79b596204b2e881c7184a5f2902ffa2
- TiDB: v8.5.5, commit
1fa258b833ff113883beeba40bc130be7ce66610
- TiKV: v8.5.5, commit
a0234aad114258b60e0ec23c17486c79845299be
Bug Report
What did you do?
Created a changefeed on three TiCDC captures for 32 tables with 6,248 Regions
per table (199,936 record Regions in total). The changefeed used a fresh
start-ts, so the initial scans were inside the default scan-priority lag
threshold.
The bootstrap was tested twice on exact master
d480b05fa79b596204b2e881c7184a5f2902ffa2. The second attempt cold-restartedall three TiCDC captures before creating the changefeed.
Other controlled settings:
min-ts-interval = 200ms;pending-region-request-queue-size = 32;cdc.incremental-scan-concurrency-limit = 10000.What did you expect to see?
The initial scans should remain bounded by TiCDC's pending-Region flow control,
finish registering all Regions, and let resolved-ts/checkpoint-ts advance past
the changefeed start-ts.
Historical master
369674c75814928e5a3f1122fc7520c370033571completed the same 199,936-Region bootstrap in 206.9 seconds.
What did you see instead?
Both exact-master attempts failed to advance past start-ts within 20 minutes.
The controlled second attempt ran from
2026-07-25 04:27:36.633to04:47:12.818CST.Metrics in that window:
ticdc_subscription_client_requested_region_count{state="pending"}increase(ticdc_kvclient_event_feed_error_count{type="KvIsBusy"}[19m])ticdc_kvclient_event_feed_error_counttypesFor comparison, two successful builds with the same topology had zero
KvIsBusy, reached about 200,099 captured Regions with zero unresolvedRegions, and peaked at only 89 and 96 aggregate pending Region requests.
At default INFO level, TiCDC records
cdc region erroronly at DEBUG, so theServerIsBusyreason is not printed in TiCDC logs. The error counter above isthe direct TiCDC signal. TiKV's 300 MiB log files rotated roughly every
12 seconds during the retry storm, dominated by per-Region incremental-scan
messages such as:
TiKV v8.5.5 rejects registrations after the configured/default 10,000
incremental-scan tasks with:
Suspected cause
The behavior is introduced by scan-priority commit
d8b039fd5/ PR #5480:subscription_client.goderivesforce := regionTask.Priority() <= forcedPriorityBase.requestCache.add(..., force=true)bypasses thependingCount < maxPendingCountflow-control check.region_failure_handler.gopreserves the inherited high priority forServerIsBusyandCongested.control again, and has no backoff or jitter.
The deployed TiKV v8.5.5 uses a kvproto version that does not consume the new
ChangeDataRequest.scan_priorityfield. Therefore the intended TiKV-sidepriority behavior is unavailable, while TiCDC's local
forceand retrybehavior still take effect.
The full-span initialization change in
dd07f601bmakes the frozen start-tsvisible and correct, but does not create the request storm.
Suggested fix
Priority and local admission bypass should be separate decisions:
still honor the per-store pending-Region limit;
ServerIsBusyandCongestedretries should be demoted behind normalflow control and use bounded backoff with jitter;
force=trueshould be reserved for a small, bounded class of recovery orcontrol work, not all recent-start initial scans.
As a narrow hotfix, demoting
ServerIsBusy/Congestedretries toTaskLowPriorshould break the positive feedback loop. It is not a completefix by itself because the first high-priority bootstrap burst can still bypass
the local pending limit before TiKV returns the first busy response.
Versions
d480b05fa79b596204b2e881c7184a5f2902ffa21fa258b833ff113883beeba40bc130be7ce66610a0234aad114258b60e0ec23c17486c79845299be