Skip to content

Conversation

@wlmyng
Copy link
Contributor

@wlmyng wlmyng commented Nov 6, 2025

Description

This PR standardizes behavior around --first-checkpoint and its effects on
A. The checkpoint to start ingesting from
B. The checkpoint for a pipeline to resume processing from

  • When an indexer starts, it will always start ingesting from the next checkpoint after the smallest committed checkpoint across all pipelines added to it.
  • Pipelines without a watermark are by default to be backfilled from genesis.
  • Providing --first-checkpoint overrides this behavior, and tells the indexer that these pipelines are to process checkpoints from --first-checkpoint instead.

Test plan

New tests


Release notes

Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required.

For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates.

  • Protocol:
  • Nodes (Validators and Full nodes):
  • gRPC:
  • JSON-RPC:
  • GraphQL:
  • CLI:
  • Rust SDK:

@vercel
Copy link

vercel bot commented Nov 6, 2025

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Preview Comments Updated (UTC)
sui-docs Ready Ready Preview Comment Nov 12, 2025 7:38pm
2 Skipped Deployments
Project Deployment Preview Comments Updated (UTC)
multisig-toolkit Ignored Ignored Preview Nov 12, 2025 7:38pm
sui-kiosk Ignored Ignored Preview Nov 12, 2025 7:38pm

@wlmyng wlmyng temporarily deployed to sui-typescript-aws-kms-test-env November 6, 2025 00:05 — with GitHub Actions Inactive
@wlmyng wlmyng force-pushed the indexer-framework-align-first-checkpoint-usage branch from c9da228 to 1f06f38 Compare November 6, 2025 00:07
@wlmyng wlmyng temporarily deployed to sui-typescript-aws-kms-test-env November 6, 2025 00:07 — with GitHub Actions Inactive
@github-actions github-actions bot added the Type: Documentation Improvements or additions to documentation label Nov 6, 2025
@wlmyng wlmyng changed the title remove skip_watermark, which allows us to standardize how to update t… [indexer-alt-framework] remove skip_watermark and standardize first_checkpoint behavior Nov 6, 2025
@wlmyng wlmyng temporarily deployed to sui-typescript-aws-kms-test-env November 6, 2025 01:02 — with GitHub Actions Inactive
@wlmyng wlmyng temporarily deployed to sui-typescript-aws-kms-test-env November 6, 2025 01:06 — with GitHub Actions Inactive
@wlmyng wlmyng changed the title [indexer-alt-framework] remove skip_watermark and standardize first_checkpoint behavior [indexer-alt-framework] remove skip_watermark and standardize first_checkpoint behavior Nov 6, 2025
@wlmyng wlmyng force-pushed the indexer-framework-align-first-checkpoint-usage branch from deea9bd to 6cc8646 Compare November 6, 2025 01:31
@wlmyng wlmyng temporarily deployed to sui-typescript-aws-kms-test-env November 6, 2025 01:31 — with GitHub Actions Inactive
@wlmyng wlmyng marked this pull request as ready for review November 6, 2025 01:31
@wlmyng wlmyng temporarily deployed to sui-typescript-aws-kms-test-env November 6, 2025 01:31 — with GitHub Actions Inactive
@wlmyng
Copy link
Contributor Author

wlmyng commented Nov 6, 2025

@emmazzz @amnn this hopefully makes it easier to review the desired behavior involving --first-checkpoint.

For each pipeline, we calculate the next checkpoint to ingest as either the checkpoint after its watermark, or if that doesn't exist, then the provided first checkpoint, and if that doesn't exist, then 0 (genesis).
The checkpoint that ingestion starts as is the minimum across all the next checkpoints calculated this way (per-pipeline).

We can remove the consistency checks first_checkpoint <= committer_watermark because the ingestion piece will always be the minimum among the next checkpoint to process.

@wlmyng wlmyng force-pushed the indexer-framework-align-first-checkpoint-usage branch from 6cc8646 to 9310861 Compare November 12, 2025 18:44
@wlmyng wlmyng temporarily deployed to sui-typescript-aws-kms-test-env November 12, 2025 18:44 — with GitHub Actions Inactive
@wlmyng wlmyng temporarily deployed to sui-typescript-aws-kms-test-env November 12, 2025 18:45 — with GitHub Actions Inactive
@wlmyng wlmyng force-pushed the indexer-framework-align-first-checkpoint-usage branch from 7232278 to 4d8009d Compare November 12, 2025 18:46
@wlmyng wlmyng temporarily deployed to sui-typescript-aws-kms-test-env November 12, 2025 18:46 — with GitHub Actions Inactive
…he starting ingestion checkpoint, and each pipeline's checkpoint to start processing from. rework tests

, and remove --skip-watermark from doc
@wlmyng wlmyng force-pushed the indexer-framework-align-first-checkpoint-usage branch from 4d8009d to 35a62e5 Compare November 12, 2025 19:35
@wlmyng wlmyng temporarily deployed to sui-typescript-aws-kms-test-env November 12, 2025 19:36 — with GitHub Actions Inactive
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Type: Documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants