fix: advance bookmark for streams that return no records#15
Closed
jchumtong wants to merge 1 commit into
Closed
Conversation
Streams that emit zero rows over the sync window never set a replication_key value, so their bookmark stayed empty and they re-scanned from start_date on every run. Advance the bookmark to the last fully-scanned date regardless of record count. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
tap-branchran a full scan fromstart_dateevery day for any stream that produces little or no data (eo_impression,eo_commerce_event,eo_custom_event,eo_web_session_start), while the high-volume streams (eo_click/eo_install/eo_open/eo_reinstall) synced incrementally as expected.Confirmed from a production run's committed state in S3:
Root cause
The replication-key value is only ever derived from
record["export_date"]on emitted rows. A stream that returns zero records across the window has no row to read it from, so its bookmark stays{}. On the next runget_starting_timestampfalls back tostart_dateand the tap re-requests every date from the Branch API again — a full scan daily (and the bulk of the ~54 min runtime).Note this is not fixable via
is_sorted: that flag only controls when/how a value read from records is committed; with no records there is nothing to bookmark in either mode.What
In
BranchExportStream.get_records, after the date loop, advance the bookmark to the last fully-scanned date even when the window produced no records, viaincrement_state. State plumbing, replication key, and the four working streams are unchanged.Verification
Against the installed
singer-sdk, the newincrement_statecall produces a progress marker that the SDK's end-of-syncfinalize_state_progress_markerspromotes to:{"replication_key": "export_date", "replication_key_value": "2026-06-21T00:00:00.000000+00:00"}— byte-for-byte the shape the working streams already write to S3.
Notes
start <= end) and floored to midnight, so the boundary day is re-fetched next run andprimary_keys=("id",)dedupes the overlap.increment_statefromsinger_sdk.helpers._state(private path, standard for taps) — worth a glance on the next SDK major bump.🤖 Generated with Claude Code