Skip to content

logpuller: high-priority bootstrap bypasses region flow control #5748

Description

@asddongmen

Bug Report

What did you do?

Created a changefeed on three TiCDC captures for 32 tables with 6,248 Regions
per table (199,936 record Regions in total). The changefeed used a fresh
start-ts, so the initial scans were inside the default scan-priority lag
threshold.

The bootstrap was tested twice on exact master
d480b05fa79b596204b2e881c7184a5f2902ffa2. The second attempt cold-restarted
all three TiCDC captures before creating the changefeed.

Other controlled settings:

  • one TiKV store;
  • TiDB min-ts-interval = 200ms;
  • MySQL dry-run sink with 50ms simulated latency;
  • no DML workload during bootstrap;
  • default TiCDC pending-region-request-queue-size = 32;
  • default TiKV cdc.incremental-scan-concurrency-limit = 10000.

What did you expect to see?

The initial scans should remain bounded by TiCDC's pending-Region flow control,
finish registering all Regions, and let resolved-ts/checkpoint-ts advance past
the changefeed start-ts.

Historical master 369674c75814928e5a3f1122fc7520c370033571
completed the same 199,936-Region bootstrap in 206.9 seconds.

What did you see instead?

Both exact-master attempts failed to advance past start-ts within 20 minutes.
The controlled second attempt ran from
2026-07-25 04:27:36.633 to 04:47:12.818 CST.

Metrics in that window:

Metric Observed value
ticdc_subscription_client_requested_region_count{state="pending"} aggregate peak 198,320
increase(ticdc_kvclient_event_feed_error_count{type="KvIsBusy"}[19m]) 40.69m / 35.93m / 41.09m on the three captures; about 119.6m total
Other ticdc_kvclient_event_feed_error_count types 0
TiKV unresolved CDC Regions exactly 10,000
TiKV captured CDC Regions at the end about 15,507
LogPuller Region matchers about 200,397
EventStore resolved input 0
Successful Region scan progress during the storm about 4-5 Regions/s

For comparison, two successful builds with the same topology had zero
KvIsBusy, reached about 200,099 captured Regions with zero unresolved
Regions, and peaked at only 89 and 96 aggregate pending Region requests.

At default INFO level, TiCDC records cdc region error only at DEBUG, so the
ServerIsBusy reason is not printed in TiCDC logs. The error counter above is
the direct TiCDC signal. TiKV's 300 MiB log files rotated roughly every
12 seconds during the retry storm, dominated by per-Region incremental-scan
messages such as:

[2026/07/25 04:44:52.763 +08:00] [INFO] [initializer.rs:542]
["cdc has scanned all incremental scan locks"] ... [region_id=562097]
[2026/07/25 04:44:52.764 +08:00] [INFO] [initializer.rs:388]
["cdc async incremental scan finished"] ... [region_id=562097]

TiKV v8.5.5 rejects registrations after the configured/default 10,000
incremental-scan tasks with:

ServerIsBusy("too many pending incremental scans")

Suspected cause

The behavior is introduced by scan-priority commit d8b039fd5 / PR #5480:

  1. A recent start-ts makes initial subscription scans high priority.
  2. subscription_client.go derives
    force := regionTask.Priority() <= forcedPriorityBase.
  3. requestCache.add(..., force=true) bypasses the
    pendingCount < maxPendingCount flow-control check.
  4. region_failure_handler.go preserves the inherited high priority for
    ServerIsBusy and Congested.
  5. The rejected task is immediately requeued as high priority, bypasses flow
    control again, and has no backoff or jitter.

The deployed TiKV v8.5.5 uses a kvproto version that does not consume the new
ChangeDataRequest.scan_priority field. Therefore the intended TiKV-side
priority behavior is unavailable, while TiCDC's local force and retry
behavior still take effect.

The full-span initialization change in dd07f601b makes the frozen start-ts
visible and correct, but does not create the request storm.

Suggested fix

Priority and local admission bypass should be separate decisions:

  • initial subscription scans may carry high scan priority to TiKV, but should
    still honor the per-store pending-Region limit;
  • ServerIsBusy and Congested retries should be demoted behind normal
    flow control and use bounded backoff with jitter;
  • force=true should be reserved for a small, bounded class of recovery or
    control work, not all recent-start initial scans.

As a narrow hotfix, demoting ServerIsBusy/Congested retries to
TaskLowPrior should break the positive feedback loop. It is not a complete
fix by itself because the first high-priority bootstrap burst can still bypass
the local pending limit before TiKV returns the first busy response.

Versions

  • TiCDC: master d480b05fa79b596204b2e881c7184a5f2902ffa2
  • TiDB: v8.5.5, commit 1fa258b833ff113883beeba40bc130be7ce66610
  • TiKV: v8.5.5, commit a0234aad114258b60e0ec23c17486c79845299be

Metadata

Metadata

Assignees

No one assigned

    Labels

    affects-8.5This bug affects the 8.5.x(LTS) versions.severity/majortype/bugThe issue is confirmed as a bug.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions