Order chunk metadata and snapshot scheduling with rollback - #8243
Open
Amaury Chamayou (achamayou) wants to merge 1 commit into
Open
Conversation
Amaury Chamayou (achamayou)
force-pushed
the
achamayou-order-chunk-metadata-rollback
branch
from
August 31, 2026 07:50
c4e377b to
07deb7f
Compare
This was referenced Aug 31, 2026
Ledger chunk sizes were appended, and snapshot scheduling rolled back, outside the lock guarding the rollback epoch. A transaction could therefore restore chunk metadata for an entry a concurrent view change had already discarded, leaving the chunker permanently ahead of the store and skewing every later chunk boundary. Take the version lock for both the rollback and the append, and skip the append when the batch's rollback epoch or view no longer holds. A rollback can only discard a batch's writes by truncating, which moves the epoch on; a rollback that does not truncate may still move the view, which consensus rejects - so both are checked. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 75d99c5d-6efa-4048-8032-8c78b97208d9
Amaury Chamayou (achamayou)
force-pushed
the
achamayou-order-chunk-metadata-rollback
branch
from
August 31, 2026 12:29
07deb7f to
66282cf
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why this is necessary
LedgerChunkerdecides where ledger chunks end, andSnapshotterdecides where snapshots are taken. Both accumulate state that outlives the transaction which contributes to it, and both are reset byStore::rollback()when a view change discards uncommitted work.Neither was ordered against the commit path.
Store::commit()appended an entry''s size to the chunker after releasingversion_lock, so this interleaving was possible:The chunker is now permanently one entry ahead of the store. Its counter is independent of the store''s version, so the offset never self-corrects: every subsequent entry is recorded against the wrong version,
get_unchunked_size()sums the wrong range, and chunk boundaries drift from what the ledger actually contains.Snapshotter::rollback()had the same shape, able to interleave with a commit recording a committable index.What changes
Both are updated under
version_lock, the same lock the rollback takes, and the append is skipped when the batch''s rollback epoch or view no longer holds.The two conditions are complementary rather than redundant, which is worth stating because it is not obvious:
versionand therefore incrementsrollback_count;This invariant is recorded as a comment at the check, so a future change to rollback semantics cannot silently invalidate it.
Why this is minimal
Only the ordering changes. No new locks are introduced and no state is added: the epoch and view compared were already captured by
Store::commit()for its existinglast_replicatedguard.Moving the chunker and snapshotter calls inside the existing
version_locksection is what makes them atomic with respect torollback_count, which is the property the check depends on. Leaving them outside and adding a separate lock would reintroduce the same window.Performance
This is the one change in the stack with a cost on the commit path: one
version_lockacquire/release per entry in a batch. The lock is short and usually uncontended, but it scales with batch size. If it shows up in benchmarks the check can be hoisted out of the loop, since the epoch cannot change mid-loop without the whole batch failing anyway.Lock ordering is unchanged in shape:
commit_lock -> version_lock -> chunker_lock.commit_lockis only ever taken at the top ofStore::commit(), beforeversion_lock, andLedgerChunkeris a leaf that holds no store reference, so no cycle is introduced.Testing
kv_testgains "Chunk metadata is not restored by a batch a rollback discarded". It drives the race deterministically and without threads, using aPendingTxthat performs the rollback inline -Store::commit()invokes it in exactly the window concerned.Verified that the test fails without the fix (the chunker ends one entry ahead of the store) and passes with it.
Labelled
run-long-test.Review stack
These five PRs come from one investigation and are stacked; review and merge in order.
All were found by an interleaving-exploration harness built over the real KV, consensus and history stack (draft #8238). The harness itself is deliberately not included here; these PRs carry only the fixes and the single-threaded regression tests that pin them.