Skip to content

fix: advance bookmark for streams that return no records#15

Closed
jchumtong wants to merge 1 commit into
mainfrom
jakob/fix-tap-branch-state-set
Closed

fix: advance bookmark for streams that return no records#15
jchumtong wants to merge 1 commit into
mainfrom
jakob/fix-tap-branch-state-set

Conversation

@jchumtong

Copy link
Copy Markdown

Why

tap-branch ran a full scan from start_date every day for any stream that produces little or no data (eo_impression, eo_commerce_event, eo_custom_event, eo_web_session_start), while the high-volume streams (eo_click/eo_install/eo_open/eo_reinstall) synced incrementally as expected.

Confirmed from a production run's committed state in S3:

"bookmarks": {
  "eo_click":       {"replication_key": "export_date", "replication_key_value": "2026-06-20T00:00:00..."},
  "eo_install":     {"replication_key": "export_date", "replication_key_value": "2026-06-20T00:00:00..."},
  "eo_open":        {"replication_key": "export_date", "replication_key_value": "2026-06-20T00:00:00..."},
  "eo_reinstall":   {"replication_key": "export_date", "replication_key_value": "2026-06-20T00:00:00..."},
  "eo_commerce_event":    {},
  "eo_custom_event":      {},
  "eo_impression":        {},
  "eo_web_session_start": {}
}

Root cause

The replication-key value is only ever derived from record["export_date"] on emitted rows. A stream that returns zero records across the window has no row to read it from, so its bookmark stays {}. On the next run get_starting_timestamp falls back to start_date and the tap re-requests every date from the Branch API again — a full scan daily (and the bulk of the ~54 min runtime).

Note this is not fixable via is_sorted: that flag only controls when/how a value read from records is committed; with no records there is nothing to bookmark in either mode.

What

In BranchExportStream.get_records, after the date loop, advance the bookmark to the last fully-scanned date even when the window produced no records, via increment_state. State plumbing, replication key, and the four working streams are unchanged.

Verification

Against the installed singer-sdk, the new increment_state call produces a progress marker that the SDK's end-of-sync finalize_state_progress_markers promotes to:

{"replication_key": "export_date", "replication_key_value": "2026-06-21T00:00:00.000000+00:00"}

— byte-for-byte the shape the working streams already write to S3.

Notes

  • No data loss from Branch's ~1-day export lag: the window is inclusive (start <= end) and floored to midnight, so the boundary day is re-fetched next run and primary_keys=("id",) dedupes the overlap.
  • Imports increment_state from singer_sdk.helpers._state (private path, standard for taps) — worth a glance on the next SDK major bump.

🤖 Generated with Claude Code

Streams that emit zero rows over the sync window never set a
replication_key value, so their bookmark stayed empty and they
re-scanned from start_date on every run. Advance the bookmark to the
last fully-scanned date regardless of record count.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@jchumtong jchumtong closed this Jun 23, 2026
@jchumtong jchumtong deleted the jakob/fix-tap-branch-state-set branch June 23, 2026 14:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant