Skip to content

Restore durable ExtraIndexer checkpoint recovery - #2531

Open
a-shannon wants to merge 3 commits into
ergoplatform:masterfrom
a-shannon:fix/extra-indexer-catch-up-retry
Open

Restore durable ExtraIndexer checkpoint recovery#2531
a-shannon wants to merge 3 commits into
ergoplatform:masterfrom
a-shannon:fix/extra-indexer-catch-up-retry

Conversation

@a-shannon

@a-shannon a-shannon commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

This continues #2465 on master. Its deleted v6.0.4 base prevents reopening the earlier PR. The updated contribution retains its operational policy and addresses the pending-save boundary described below.

ExtraIndexer checkpoints persist the exact indexed header together with forward rows in one database batch. Restart validates checkpoint provenance and terminal mappings instead of assigning the current best header to existing rows. Catch-up follows the selected valid full-block chain and retains the immediate applied-event recovery from #2476.

Failed forward saves retain their buffers and block both indexing entry points until persistence succeeds. This also protects segmented rows whose parent references have already changed. Rollback persists a marker before modifying rows and propagates failures through its final checkpoint. A failed rollback stops only ExtraIndexer; the node continues running, and the next history startup rebuilds the optional index.

Existing nonempty schema-6 extra indexes rebuild once under schema 7. Interrupted, malformed or inconsistent checkpoints also rebuild. Valid schema-7 checkpoints survive restart. Rebuild duration depends on index size and hardware and has not been benchmarked.

Validation: JDK 8, Scala 2.12.20; 30 indexer tests pass on the final candidate, with eight unchanged storage-test results reused. Coverage includes actual non-genesis indexing and reopen, selected full-chain identity, atomic row/checkpoint writes, cache consistency, retry after failed segmented writes, and isolated rollback write-phase failures followed by restart. Independent source and follow-up review completed.

This updates the existing recovery contribution against the current base. It changes the optional derived index, with no consensus or UTXO-state change.

A separate commit includes the already reviewed V2 summary/cache dependency from #2511. Its five files match that contribution. Fourteen focused history, cache, serialization and synchronizer tests pass on this indexer branch, with independent composition review. Those fixtures use the current history/storage code with the optional index disabled; the indexer recovery evidence above remains separate. Review and integrate #2511 first.

The shared test-only follow-up polls selected-header agreement and records bounded rollback observations. Nine observation contracts and independent review pass. All eight CI checks pass at 342e718e705d55603f015e74b84ae0cd4d7b9f5d. This successful composition does not retrospectively isolate the cause of the previous selected-header disagreement.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant