mysqlcdc: checkpoint at transaction boundary#4396
Conversation
|
Commits Review LGTM |
21624c0 to
ea70436
Compare
|
Commits Review LGTM |
Ensure we only checkpoint at the transaction boundary as opposed to the row boundary. This way we don't risk recovering mid transaction in the event of a crash.
ea70436 to
a6446f9
Compare
|
Commits Review LGTM |
|
Commits Review LGTM |
The problem:
Currently we checkpoint at a row level. A MySQL binlog looks like the following, with
go-mysqlkeeping an internal buffer of events keyed by the table ID that's set in theTABLE_MAP_EVENT:If the connector restarts mid transaction then it's unable to recover as it no longer has the table id and we see the above error.
This change:
This changes listens to a sentient event (
XID_EVENT) that exists after every transaction is flushed. Ensuring we checkpoint at this position ensures we capture at clean transaction boundaries and only recover at the beginning of the transaction.Attn: This does mean that in the event of a connector crash, it's possible that we redeliver already processed messages as we start from the beginning of the transaction (satisfying at-least-once delivery).