Skip to content

feat: Implement ReadN #276

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 24 commits into from
Apr 30, 2025
Merged

feat: Implement ReadN #276

merged 24 commits into from
Apr 30, 2025

Conversation

raulb
Copy link
Member

@raulb raulb commented Apr 25, 2025

Description

Since the introduction of end-to-end batching on the Connector SDK, (part of the Connector SDK 0.12.0), source connectors can add support to batching by implementing the ReadN method. This wasn't a breaking change since the SDK will fallback to the traditional Read method as it can be spotted in the logs when starting Conduit.

Example:

2025-04-24T19:21:21+00:00 INF source does not support batch reads, falling back to single reads component=plugin connector_id=postgres-to-kafka:postgres-source plugin_name=builtin:postgres plugin_type=source

This pull-request implements ReadN while still maintaining support to Read method for those who wish to use this connector in an older version of Conduit. This is probably not necessary considering Postgres is a builtin connector.

To really utilize the advantage of this, it is recommended to run Conduit with the flag that enables a new upcoming architecture via --preview.pipeline-arch-v2. More information on this blog post.

Running benchmarks

Inserting 20M records using Benchi using CDC:

  • WIth Read: 142824.47 msg/s
  • With ReadN: 153107.18 msg/s (previously )

This is a 7,2% better on CDC.

Quick checks:

  • I have followed the Code Guidelines.
  • There is no other pull request for the same update/change.
  • I have written unit tests.
  • I have made sure that the PR is of reasonable size and can be easily reviewed.

@raulb raulb self-assigned this Apr 25, 2025
@raulb raulb marked this pull request as ready for review April 25, 2025 13:26
@raulb raulb requested a review from a team as a code owner April 25, 2025 13:26
@raulb raulb moved this from Triage to Ready for review in Conduit Main Apr 25, 2025
@raulb raulb requested a review from lyuboxa April 25, 2025 21:02
Copy link
Member

@lovromazgon lovromazgon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few cleanups, otherwise looks good 👍

@github-project-automation github-project-automation bot moved this from Ready for review to Reviewed (ready to ship) in Conduit Main Apr 30, 2025
@raulb raulb requested review from hariso and lovromazgon April 30, 2025 14:03
@raulb raulb enabled auto-merge (squash) April 30, 2025 14:30
@raulb raulb merged commit e468f39 into main Apr 30, 2025
3 checks passed
@raulb raulb deleted the read-n branch April 30, 2025 14:31
@github-project-automation github-project-automation bot moved this from Reviewed (ready to ship) to Done in Conduit Main Apr 30, 2025
@raulb raulb mentioned this pull request Apr 30, 2025
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

4 participants