Maintainer errors are dropped when checkpoint regresses, preventing automatic changefeed recovery

[asddongmen]

What did you do?

We observed a production incident in the new-architecture TiCDC with a Kafka sink.

During a transient connectivity disturbance:

1. One capture lost its etcd session and restarted.
2. The capture membership change caused many table dispatchers of one changefeed to be rescheduled.
3. During rescheduling, the maintainer checkpoint regressed below the checkpoint already stored by the coordinator.
4. At the same time, a dispatcher reported a retryable ErrKafkaAsyncSendMessage.
5. The affected changefeed never entered warning/backoff and did not recover automatically. A manual changefeed restart was required.

All customer, cluster, changefeed, table, network, and DML identifiers below are masked.

Sanitized timeline

The capture restart was caused by the TiCDC process fencing itself after its etcd session ended, rather than by OOM or a Kubernetes eviction:

[2026/07/22 16:30:08.634 +00:00] [WARN] [etcd_watcher.go:70]
["session is disconnected"]
[error="[CDC:ErrEtcdSessionDone]etcd session is done"]

[2026/07/22 16:30:08.672 +00:00] [ERROR] [server.go:<line>]
["cdc server exits with error"]
[error="[CDC:ErrCaptureSuicide]capture suicide"]

The affected changefeed maintainer then observed the old capture leaving and a new capture joining:

[2026/07/22 16:30:13.486 +00:00] [INFO] [maintainer.go:540]
["maintainer node changed"]
[changefeedID=default/<changefeed>]
[addedCount=0] [removedCount=1]
[removedNodes=["<old-capture-id>"]]

[2026/07/22 16:30:15.987 +00:00] [INFO] [maintainer.go:540]
["maintainer node changed"]
[changefeedID=default/<changefeed>]
[addedCount=1] [removedCount=0]
[addedNodes=["<new-capture-id>"]]

The incident queued more than 100 add-dispatcher operators for this changefeed. Around the same time, another capture reported a Kafka send timeout:

[2026/07/22 16:30:15.758 +00:00] [ERROR] [maintainer.go:799]
["dispatcher report an error"]
[changefeedID=default/<changefeed>]
[sourceNode=<capture-id>]
[error="[CDC:ErrKafkaAsyncSendMessage]kafka async send message failed:
        keyspace=default, changefeed=<changefeed>, eventType=dml,
        dmlInfo=<redacted>;
        ErrorInfo:read tcp <ticdc-address>:<port>-><kafka-broker>:9092: i/o timeout"]

Metrics show that the maintainer checkpoint regressed during failover. Values are expressed relative to a redacted base physical timestamp T:

16:30:07  maintainer checkpoint = T + 4.800s
16:30:22  maintainer checkpoint = T          # regressed by 4.800s
           owner checkpoint retained = T + 5.750s

Later:
           maintainer checkpoint = T + 1.800s
           owner checkpoint      = T + 5.750s
           difference            = 3.950s

After the regression, the same Kafka error was reported by the maintainer every five seconds. Log-site grouping found one original dispatcher-manager error followed by 289 maintainer reports. The last report was about 30 minutes after the first one.

There were no backoff.go entries for the affected changefeed. In the same incident, at least six other changefeeds with Kafka errors did reach:

[ERROR] [backoff.go:157] ["changefeed maintainer report an error"]
[WARN]  [backoff.go:220] ["changefeed meets an error, will be stopped"]

Those changefeeds were automatically stopped/recreated and recovered. The affected changefeed remained in normal state while checkpoint lag grew linearly.

A manual changefeed restart closed all dispatcher managers and recreated the sink clients:

[2026/07/22 17:00:18.709 +00:00] [INFO] [dispatcher_manager.go:889]
["closing event dispatcher manager"]
[changefeedID=default/<changefeed>]

[2026/07/22 17:00:41.164 +00:00] [INFO] [dispatcher_manager.go:306]
["dispatcher manager initialized"]
[changefeedID=default/<changefeed>]
[maintainerID=<new-maintainer-id>]
[startTs=<redacted>]

After the restart, checkpoint lag returned to about 3 seconds and no new Kafka async-send error was observed.

What did you expect to see?

A retryable error in MaintainerStatus.Err should be handled even when the incoming checkpoint is lower than the coordinator's last stored checkpoint.

The coordinator should:

1. Keep the stored checkpoint monotonic.
2. Still process the error and transition the changefeed to warning/backoff.
3. Stop and recreate the changefeed automatically.

A safe checkpoint regression can occur during node failover because absent/scheduling spans participate in the maintainer's global checkpoint minimum. It should not disable error recovery.

What did you see instead?

Changefeed.UpdateStatus gates both checkpoint storage and error handling on the incoming checkpoint being monotonic:

if newStatus != nil && newStatus.CheckpointTs >= old.CheckpointTs {
    c.status.Store(newStatus)
    // ...
    return c.backoff.CheckStatus(newStatus)
}

return false, config.StateNormal, nil

When the maintainer checkpoint regresses, the whole status is ignored, including newStatus.Err. As long as subsequent reports remain below the coordinator checkpoint:

• Backoff.CheckStatus is never called.
• The changefeed never enters warning.
• Automatic stop/recreate is never triggered.
• The dispatcher manager keeps re-reporting its first terminal sink error until it is closed manually.

Relevant production-version code:

• checkpoint gate in Changefeed.UpdateStatus
• absent/scheduling spans constrain the maintainer checkpoint
• dispatcher manager resends the first error until close

The same lower-checkpoint early-return behavior is still present on the current master branch at the time of filing.

Suggested fix and regression test

Decouple checkpoint monotonicity from error handling:

• Do not overwrite the stored checkpoint with a lower value.
• Do not discard MaintainerStatus.Err.
• Process the retryable error using the stored checkpoint (or otherwise route it to backoff independently).

Add a test covering:

old stored checkpoint = 200
incoming checkpoint   = 150
incoming Err          = retryable sink error

expected:
- stored checkpoint remains 200
- state changes to warning
- retry/backoff starts

This is related by symptom to #3952, but the root cause is different. In #3952 the error reached warning/backoff and recovery failed later during maintainer re-add. In this incident the error is dropped before backoff starts.

Versions of the cluster

Upstream TiDB cluster version:

v8.5 series (exact binary version was not separately collected)

Upstream TiKV version:

v8.5 series (exact binary version was not separately collected)

TiCDC version:

Release Version: v8.5.7-release.1
Git Commit Hash: e7e1f6142308f893acbd2aea8ee57af49a53e5fc