Skip to content

postgres_cdc: support "before" data in change events for UPDATE and DELETE events - #4555

Merged
josephwoodward merged 3 commits into
mainfrom
jw/postgrebefore
Jun 26, 2026
Merged

postgres_cdc: support "before" data in change events for UPDATE and DELETE events#4555
josephwoodward merged 3 commits into
mainfrom
jw/postgrebefore

Conversation

@josephwoodward

@josephwoodward josephwoodward commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

This change introduces a new before metadata property for update and delete change events which contains the state of the changed row before the commit.

It updates the existing implementation to reuse decoding of PostgreSQL's WAL tuples between new and old values.

Integration tests passing.

Proof of Work

metadata.before is now on update and delete (but not insert) messages when REPLICA IDENTITY FULL is set:

image

@claude

claude Bot commented Jun 25, 2026

Copy link
Copy Markdown

Commits

  1. postgres_cdc: Add commit_ts_ms to insert, update and delete events — the message after system: must start lowercase and use imperative mood per the enforced commit format (e.g. add, not Add). Should be postgres_cdc: add commit_ts_ms to insert, update and delete events. The other two commits (update integration test..., support "before" data...) are correctly formatted.

Review

Adds commit_ts_ms and before metadata to the postgres_cdc input. The commit-time capture/reset around BEGIN/COMMIT, the before decoding for update (via OldTuple) and delete, and the integration test coverage all look correct and consistent with the existing tuple-decoding patterns. The new message.CommitTime assignment is safely after the existing message == nil guard.

LGTM

@claude

claude Bot commented Jun 26, 2026

Copy link
Copy Markdown

Commits
LGTM

Review
This PR adds a before metadata field to postgres_cdc for update and delete events and refactors the per-operation tuple decoding into a shared decodeTuple helper. The change keeps the flat fleet message shape (additive top-level metadata, raw row body untouched — compliant with CDC standard §5.2.1), correctly resolves unchanged TOAST columns against the old tuple under REPLICA IDENTITY FULL, and is covered by the updated integration test for both update and delete paths. The delete path now errors on unknown column data types (previously a silent no-op), but this only makes it consistent with the insert/update paths and pgoutput never emits other types for these tuples, so it is not a regression.

LGTM

@claude

claude Bot commented Jun 26, 2026

Copy link
Copy Markdown

Commits
LGTM

Review
This PR exposes pre-change row state via a new before metadata key for postgres_cdc update/delete events, and refactors the per-operation tuple-decoding loops into a shared decodeTuple helper.

Reviewed for correctness, CDC §5 conformance, and project Go patterns. Findings:

  • before is emitted as flat top-level metadata (not a nested before/after/source/op envelope), so it conforms with §5.2.1; the body remains the raw row. The internal/plugins/cdctest gate only checks config field names, which this PR does not touch.
  • The delete path's default: case changed from lenient (empty) to returning an error, but the stream negotiates proto_version '1' (text mode), so column data types are only n/u/t — the error branch is unreachable, not a regression.
  • decodeTuple adds an idx >= len(rel.Columns) bounds guard, making it strictly safer than the original loops.
  • For deletes, Data and BeforeData share the same map, but Data is JSON-marshaled to bytes before becoming the body, so there is no aliasing concern.
  • New behavior is covered by added integration-test assertions.

LGTM

@josephwoodward
josephwoodward marked this pull request as ready for review June 26, 2026 11:02
@josephwoodward josephwoodward changed the title postgres_cdc: support "before" data in change events for update and deletes postgres_cdc: support "before" data in change events for update and delete events Jun 26, 2026
@josephwoodward josephwoodward changed the title postgres_cdc: support "before" data in change events for update and delete events postgres_cdc: support "before" data in change events for UPDATE and DELETE events Jun 26, 2026
@claude

claude Bot commented Jun 26, 2026

Copy link
Copy Markdown

Commits
LGTM

Review
Adds a before metadata field carrying the pre-change row for postgres_cdc update/delete events, and refactors the three inlined pgoutput tuple-decode loops into a shared decodeTuple helper.

Reviewed the refactor against the original insert/update/delete loops (behavior-preserving, with an added bounds guard), the new flat before metadata key (consistent with the existing flat schema metadata — body stays flat, no nested envelope, conforms to the §5.2 fleet shape), and the integration test assertions (insert→update→delete flow correctly checks before.name for both update and delete). No issues found.

LGTM

@claude

claude Bot commented Jun 26, 2026

Copy link
Copy Markdown

Commits
LGTM

Review
Focused change adding a before metadata field to postgres_cdc update/delete events, backed by a refactor that consolidates tuple decoding into a single decodeTuple helper. The before key is emitted as flat top-level metadata (not a nested before/after envelope), conforming to the CDC flat-message standard (§5.2.1). The update branch correctly limits the TOAST fallback to OldTupleType == 'O' (REPLICA IDENTITY FULL), and the new helper adds a defensive bounds check that the original per-branch loops lacked. Integration test coverage asserts the pre-change values for both update and delete. Commit messages conform to the system: message imperative convention.

LGTM

@josephwoodward
josephwoodward merged commit 17988cb into main Jun 26, 2026
10 of 12 checks passed
@josephwoodward
josephwoodward deleted the jw/postgrebefore branch June 26, 2026 12:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants