-
Notifications
You must be signed in to change notification settings - Fork 11.7k
[indexer-alt-framework] remove skip_watermark and standardize first_checkpoint behavior
#24192
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
c9da228 to
1f06f38
Compare
first_checkpoint behavior
first_checkpoint behaviorskip_watermark and standardize first_checkpoint behavior
deea9bd to
6cc8646
Compare
|
@emmazzz @amnn this hopefully makes it easier to review the desired behavior involving
We can remove the consistency checks |
6cc8646 to
9310861
Compare
7232278 to
4d8009d
Compare
…he starting ingestion checkpoint, and each pipeline's checkpoint to start processing from. rework tests , and remove --skip-watermark from doc
4d8009d to
35a62e5
Compare
Description
This PR standardizes behavior around
--first-checkpointand its effects onA. The checkpoint to start ingesting from
B. The checkpoint for a pipeline to resume processing from
--first-checkpointoverrides this behavior, and tells the indexer that these pipelines are to process checkpoints from--first-checkpointinstead.Test plan
New tests
Release notes
Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required.
For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates.