Skip to content

An out-of-crank savepoint rollback leaves the run-queue cache ahead of the database #1071

Closed

Description

@sirtimid

Found reviewing #1021 (sirtimid/crank-rollback-integrity).

What

KernelStore.rollbackSavepoint is a bare pass-through to kdb.rollbackSavepoint, so the out-of-crank path #1021 formalises gets none of the cache reverting that revertStateBeneathRollback now does for the crank path — and the out-of-crank block writes through exactly the caches that matter.

Failure scenario

  1. A peer sends deliver/message. Inside the turn, #handleRemoteDeliver → KernelQueue.enqueueSend → kernelStore.enqueueRun → ctx.runQueue.enqueue, which does incCounter(head): cached head goes N → N+1 in memory and in the DB, and the item is written at queue.run.N.
  2. releaseSavepoint(savepointName) throws — disk full at COMMIT, the very case the rest of fix: keep a crank's store work inside one transaction #1021 is about — so the driver discards the whole transaction and RemoteHandle's catch calls rollbackSavepoint.
  3. The DB is back at head = N with no queue.run.N row. The in-memory head still says N+1, and runQueueLengthCache is one too high.
  4. The next enqueue writes queue.run.N+1 and sets head = N+2, leaving a permanent hole at N.
  5. When the run loop's dequeueRun reaches tail N, provideStoredQueue.dequeue calls kv.getRequired('queue.run.N') and throws no record matching key 'queue.run.N'. Run loop dead, and everything behind the hole stranded.

setRemoteHighestReceivedSeq throwing reaches the same place.

The same rollback also leaves ctx.maybeFreeKrefs holding the krefs RemoteManager → persistPeerRestart → forgetEndpointImports added; that half is mostly benign, since the restored refcounts make collectGarbage skip them.

Pre-existing in kind — handleRemoteMessage rolled back this way on main too. But #1021 adds the remedy for the sibling path and promotes this one to a first-class transaction, so leaving it bare is the gap a reader would least expect.

Suggested fix

Give KernelStore.rollbackSavepoint the cache half of revertStateBeneathRollback: refreshRunQueue(), runQueueLengthCache = -1, refreshCachedValues(). If the candidate set should be restored too, have createSavepoint stash new Set(context.maybeFreeKrefs) keyed by savepoint name for rollbackSavepoint to put back.

Activity

  1. self-assigned this
    on Sep 15, 2026
  2. added a commit that references this issue on Sep 18, 2026
    54d3ee1
  3. added a commit that references this issue on Sep 22, 2026
    f67ec77
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions