Restore durable ExtraIndexer checkpoint recovery - #2531
Open
a-shannon wants to merge 3 commits into
Open
Conversation
a-shannon
marked this pull request as ready for review
September 7, 2026 01:45
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This continues #2465 on
master. Its deletedv6.0.4base prevents reopening the earlier PR. The updated contribution retains its operational policy and addresses the pending-save boundary described below.ExtraIndexer checkpoints persist the exact indexed header together with forward rows in one database batch. Restart validates checkpoint provenance and terminal mappings instead of assigning the current best header to existing rows. Catch-up follows the selected valid full-block chain and retains the immediate applied-event recovery from #2476.
Failed forward saves retain their buffers and block both indexing entry points until persistence succeeds. This also protects segmented rows whose parent references have already changed. Rollback persists a marker before modifying rows and propagates failures through its final checkpoint. A failed rollback stops only ExtraIndexer; the node continues running, and the next history startup rebuilds the optional index.
Existing nonempty schema-6 extra indexes rebuild once under schema 7. Interrupted, malformed or inconsistent checkpoints also rebuild. Valid schema-7 checkpoints survive restart. Rebuild duration depends on index size and hardware and has not been benchmarked.
Validation: JDK 8, Scala 2.12.20; 30 indexer tests pass on the final candidate, with eight unchanged storage-test results reused. Coverage includes actual non-genesis indexing and reopen, selected full-chain identity, atomic row/checkpoint writes, cache consistency, retry after failed segmented writes, and isolated rollback write-phase failures followed by restart. Independent source and follow-up review completed.
This updates the existing recovery contribution against the current base. It changes the optional derived index, with no consensus or UTXO-state change.
A separate commit includes the already reviewed V2 summary/cache dependency from #2511. Its five files match that contribution. Fourteen focused history, cache, serialization and synchronizer tests pass on this indexer branch, with independent composition review. Those fixtures use the current history/storage code with the optional index disabled; the indexer recovery evidence above remains separate. Review and integrate #2511 first.
The shared test-only follow-up polls selected-header agreement and records bounded rollback observations. Nine observation contracts and independent review pass. All eight CI checks pass at
342e718e705d55603f015e74b84ae0cd4d7b9f5d. This successful composition does not retrospectively isolate the cause of the previous selected-header disagreement.