Skip to content

Conversation

@Supreetha-V
Copy link

@Supreetha-V Supreetha-V commented Nov 26, 2025

Summary:

This PR addresses the issue where the MySQL CDC connector silently fails when encountering
reconnection errors (e.g., after credential rotation), ignoring configured connect.timeout
and connect.max-retries parameters and entering an infinite retry loop.

Problem:

When MySQL credentials are rotated or connection issues occur during binlog reading:

  • The connector enters an infinite retry loop, spawning new threads on each attempt
  • The configured connect.timeout and connect.max-retries parameters are ignored for reconnection
  • The job stops consuming data without throwing an error or restarting
  • No actionable logs are produced, making debugging difficult

Logs from before the change:

2025-12-05 17:06:56,069 INFO io.debezium.connector.mysql.MySqlStreamingChangeEventSource [] - Stopped reading binlog after 0 events, last recorded offset: {transaction_id=null, ts_sec=1764954381, file=mysql-bin.000003, pos=139087, gtids=6d2eb014-d1fb-11f0-9e96-0242ac130006:1-17,af6ffbc8-7459-11f0-bd64-0242ac130005:1-729, server_id=1, event=1} 
2025-12-05 17:06:57,109 INFO org.apache.flink.cdc.connectors.mysql.source.reader.MySqlSourceReader [] - Binlog offset for tables [local_agr.application_setting, local_agr.asset, local_agr.asset_attribute] on checkpoint 9: {transaction_id=null, ts_sec=1764954296, file=mysql-bin.000003, pos=138668, kind=SPECIFIC, gtids=6d2eb014-d1fb-11f0-9e96-0242ac130006:1-16,af6ffbc8-7459-11f0-bd64-0242ac130005:1-729, row=0, event=0, server_id=1} 
2025-12-05 17:07:27,733 INFO io.debezium.util.Threads [] - Creating thread debezium-mysqlconnector-mysql_binlog_source-binlog-client 
2025-12-05 17:07:57,113 INFO org.apache.flink.cdc.connectors.mysql.source.reader.MySqlSourceReader [] - Binlog offset for tables [local_agr.application_setting, local_agr.asset, local_agr.asset_attribute] on checkpoint 10: {transaction_id=null, ts_sec=1764954296, file=mysql-bin.000003, pos=138668, kind=SPECIFIC, gtids=6d2eb014-d1fb-11f0-9e96-0242ac130006:1-16,af6ffbc8-7459-11f0-bd64-0242ac130005:1-729, row=0, event=0, server_id=1} 
2025-12-05 17:08:27,791 INFO io.debezium.util.Threads [] - Creating thread debezium-mysqlconnector-mysql_binlog_source-binlog-client 

Changes:

  • Added binlog.fail-on-reconnection-error configuration option (default: false for backward compatibility)
  • When enabled, the binlog reader now properly respects connect.timeout and connect.max-retries
    during reconnection attempts
  • Added specific handling for AuthenticationException to detect credential rotation issues
  • The job now fails fast with a descriptive error when reconnection limits are exceeded,
    allowing Flink's restart strategy to handle recovery

@Supreetha-V Supreetha-V marked this pull request as draft November 27, 2025 11:25
@Supreetha-V Supreetha-V marked this pull request as ready for review December 9, 2025 07:46
@Supreetha-V Supreetha-V marked this pull request as draft December 9, 2025 07:48
@github-actions github-actions bot added the docs Improvements or additions to documentation label Dec 9, 2025
@Supreetha-V Supreetha-V changed the title Fix credential rotation handling in MySQL cdc connector [FLINK-38788] [cdc connector mysql]: Fix silent failure during reconnection after credential rotation Dec 9, 2025
@Supreetha-V Supreetha-V marked this pull request as ready for review December 9, 2025 13:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Improvements or additions to documentation mysql-cdc-connector

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant