op-reth committed blocks later become unreadable, leaving local gap and blocking safe derivation

[jcortejoso]

Summary

A Celo mainnet celo-reth / op-reth datadir developed a local gap where blocks 67211634 through 67211637 became unreadable even though logs show they were received, executed, added, and committed earlier. The unsafe chain continued past the gap and matched Forno at later blocks, but op-node could not advance safe/finalized because the engine could not read the next safe block after 67211633.

This appears to be a storage/provider/static-file consistency issue, not a canonical chain divergence.

Observed local gap

On the affected op-reth RPC:

• 67211633 was present.
• 67211634 through 67211637 returned null.
• 67211638 was present.

Forno had the missing blocks, and local 67211638.parentHash matched Forno block 67211637, so the local node did not appear to diverge from Celo mainnet. Instead, the local store had an unreadable gap inside an otherwise canonical chain.

Known hashes from the investigation:

• local/Forno 67211633: 0x96d6fe1e1b81f43e786cde3d44f4c0d5c9e0da930fb4e49611b45cc89d06f3bc
• Forno 67211634: 0x2fbd551dea2d53f20df1966a339d417291a207161f906a935c61f506a3f1fa06
• Forno 67211635: 0xbae1c8dd32ad6b250e42c95d34ac2067646a131cab3c5d66478ab06c8a3d4a53
• Forno 67211636: 0x66593291d1ced88758edb74b3be6015c4c768ec08f8cf6938a63edbd7744fda4
• Forno 67211637: 0xb1cdd7ef255a42f555169ca41d7cab87e4a93e7e2cd18dbe6009a3df45533f4f
• local/Forno 67211638: 0xd022d7c4517d3c76f2b70a81f7baf793a8ae091d41af2ea00e5861a4cb2d97b1

Logs indicate the missing blocks were committed

The missing block timestamps were approximately:

• 67211634: 2026-05-18T13:53:12Z
• 67211635: 2026-05-18T13:53:13Z
• 67211636: 2026-05-18T13:53:14Z
• 67211637: 2026-05-18T13:53:15Z

op-reth logs showed each of these blocks being received, added, and committed:

• 67211634: received 13:53:12.035324Z, added 13:53:12.072262Z, committed 13:53:12.074077Z
• 67211635: received 13:53:13.043232Z, added 13:53:13.162668Z, committed 13:53:13.164519Z
• 67211636: received 13:53:14.037280Z, added 13:53:14.146090Z, committed 13:53:14.147863Z
• 67211637: received 13:53:15.034068Z, added 13:53:15.047042Z, committed 13:53:15.048456Z

op-node also inserted/processed those payloads around the same timestamps.

Safe derivation symptom

Later, op-node advanced safe to 67211633 and then immediately reset because it could not read the next block from the engine:

Deriver system is resetting
err="expected engine was synced and had unsafe block to reconcile, but cannot find the block: not found"

The first reset happened around 2026-05-18T13:56:37Z, shortly after op-node recorded safe head 67211633. This lines up with the first missing block being 67211634.

Static-file/provider symptoms

A db get against static-file headers also reported an inconsistent static-file segment:

Error: File is in an inconsistent state

The affected segment was the headers segment covering 67000000_67499999.

A later recovery attempt with a Celo-aware stage command made it past the earlier CIP-64 transaction decoding problem (tracked separately in #180), but failed during Execution unwind because header 67211634 is missing:

2026-05-19T15:54:57.421615Z  INFO Unwinding{stage=Execution}: Starting unwind from=67293475 to=67211633 bad_block=None
2026-05-19T16:02:41.526782Z ERROR shutting down due to error
Error: database integrity error occurred: no header found for Number(67211634)

Caused by:
    no header found for Number(67211634)

Location:
    /usr/local/cargo/git/checkouts/reth-e231042ee7db3fb7/d6324d6/crates/cli/commands/src/stage/unwind.rs:74:9

This reinforces that the local database/static files are missing the header for 67211634, despite earlier commit logs.

Host/process signals checked

Grafana/host checks did not show an obvious OOM, process restart, or filesystem device error around the incident window:

• op-reth/op-node process start timestamps appeared constant around the failure window.
• node_vmstat_oom_kill did not indicate an OOM event.
• node_filesystem_device_error was zero for /var/lib/op-reth and other checked mounts.

ExEx relevance

The node had proofs-history ExEx enabled, and the ExEx later logged Missing block 67211634, but that looks like an additional detector rather than the root cause. Similar behavior may have been observed on a Celo Sepolia node without ExEx enabled, so this should probably be investigated first as a storage/static-file/provider consistency issue independent of ExEx.

Impact

The node can continue serving/holding later unsafe blocks that match the canonical chain, but op-node safe/finalized derivation cannot cross the missing local block. Recovery by unwind is also blocked because the execution unwind expects the missing header.

Questions / investigation points

• How can op-reth log a block as committed and later be unable to read its header by number?
• Can static-file production or pruning leave a small header/body gap while later headers remain available?
• Is there a known failure mode where static-file segment metadata/indexes become inconsistent without a process restart/OOM/filesystem error?
• Should stage unwind have a repair path for this kind of partial static-file/header gap?
• Is this specific to Celo primitives/storage routing, or inherited from upstream reth static-file behavior?

[jcortejoso]

Current status (consolidated 2026-05-20)

Replacing earlier comments on this issue with a single up-to-date status, since several theories from the morning have been ruled out by today's investigation. Original issue body (May-18 unreadable-gap incident) is left as-is for the historical record.

TL;DR

• The original failure mode (the May-18 four-block unreadable gap at 67,211,634–67,211,637) has not recurred since PR refactor: replace Cip64Storage VecDeque with single-slot Option #172 (single-slot Cip64Storage) + PR feat(celo-reth): wire up --proofs-history ExEx (closes #174) #175 (debug_assert relaxation for re-execution paths) were deployed.
• A different but related failure mode appeared today, 2026-05-20, on the same testbed (fsn-1 running sha-44b69f7): op-reth rejects ~0.6% of op-node-derived payloads as INVALID, all of which contain CIP-64 transactions, wedging safe/finalized indefinitely at the EL-sync anchor (block 67,387,242).
• The downstream symptom in op-node is identical — Deriver system is resetting: cannot find the block: not found reset loop — but the upstream cause is state-root divergence, not static-file unreadability.
• An executor-code diff between op-rs's kona-executor and celo-kona-reth's BlockExecutor finds no divergence: both share alloy-celo-evm and celo-revm, and the CIP-64 EVM handler in crates/celo-revm/src/handler.rs is the same code path from both sides. The divergence is therefore not in code, it's in either state or image-version.

Where the original-gap fix landed

• de1fc9d (refactor: replace Cip64Storage VecDeque with single-slot Option #172) replaced the Cip64Storage VecDeque with a single-slot Option. The VecDeque was the mechanism by which an unpaired transact_raw (e.g. from the ExEx re-execution path) could silently mis-attribute CIP-64 receipt data to a later transaction, which was the most plausible cause of the receipts-root divergence + downstream static-file inconsistency observed on May 18.
• 0398203 (PR feat(celo-reth): wire up --proofs-history ExEx (closes #174) #175) demoted the panic-on-slot-reuse to debug_assert! so release builds don't crash when the ExEx re-execution path naturally hits the slot a second time without an intervening build_receipt.
• fsn-1 has run the patched image (sha-44b69f7) since 2026-05-20 11:25 UTC with --proofs-history enabled. No static-file inconsistency, no MDBX gap, no ExEx panics. Forward-fill is clean. The original op-reth committed blocks later become unreadable, leaving local gap and blocking safe derivation #182 mechanism appears resolved.

New failure mode observed today

On fsn-1 starting 2026-05-20 ~15:35 UTC, op-reth begins responding INVALID to specific op-node-derived payloads:

reth_consensus_engine_beacon_new_payload_messages          22996
reth_consensus_engine_beacon_new_payload_valid             22859   (99.4%)
reth_consensus_engine_beacon_new_payload_invalid             136   (0.6%)
reth_consensus_engine_beacon_forkchoice_updated_invalid     1404
reth_consensus_engine_beacon_invalid_headers_unique_inserts   12

12 unique invalid blocks, all in 67,376,073–67,381,283. Every one contains at least one type-0x7b (CIP-64) transaction. Witnesses (pre-state tries) preserved on fsn-1 at /var/lib/op-reth/data/invalid_block_hooks/witness/{number}_{hash}.witness.re_executed.json.

Concrete divergence at block 67,381,283:

• Canonical hash (op-reth's chain): 0xdeea824473db701a42dec3e21e07198630f85c8ccab34180118d7a86873480d4
• INVALID hash (op-node derivation): 0xfcf002da6978546a78f353c7c42378380d87a958bb1777680cda1e6ab7b43ad0

Once one INVALID block is cached, op-node's safedb retains the hash, and on every restart op-node tries to advance via that hash → INVALID → reset → loop. Recovery requires wiping /var/lib/op-node/safedb, which we have not yet done pending team review.

Executor diff (the "Likely next step" from prior comment)

Pulled celo-org/celo-kona@0398203 (current main) and mapped the two executor lineages:

	Lineage A: crates/kona/executor/	Lineage B: crates/celo-reth/
Consumers	celo-client, execution-verifier (stateless fault-proof validation)	live op-reth binary on fsn-{0,1}
EVM crate	alloy-celo-evm (default features)	alloy-celo-evm (+ optional_no_base_fee)
Handler crate	celo-revm (default features)	celo-revm (+ alloy-op-evm, reth, optional_no_base_fee)
CIP-64 debit_gas_fees	crates/celo-revm/src/handler.rs:325–393	same code path
CIP-64 credit_gas_fees	crates/celo-revm/src/handler.rs:190–269	same code path
Receipt builder	CeloAlloyReceiptBuilder — with-bloom envelope	CeloRethReceiptBuilder — bloomless (reth recomputes bloom at serialization)

The feature-flag deltas don't change CIP-64 execution behavior:

• reth feature: adds a reth_evm::TransactionEnv trait impl on CeloTransaction only (crates/celo-revm/src/transaction.rs:126–141). No behavior change.
• optional_no_base_fee: passthrough to revm's CfgEnv::disable_base_fee. Only affects execution if that flag is actually set on the CfgEnv for a consensus payload, which shouldn't happen.
• alloy-op-evm: dep-only (gates the dep:alloy-op-evm import).

Cip64Storage is Arc<Mutex<>>-shared; both lineages wire the EVM and receipt builder onto the same instance correctly. The doc comment in crates/alloy-celo-evm/src/cip64_storage.rs:23–39 explicitly notes that misuse of this storage causes receipts-root divergence specifically, not state-root.

Bottom line: there is no executor-code divergence in this repo that could produce a state-root mismatch on a CIP-64 block. The divergence must lie outside the executor code itself.

Working theories, ranked

1. Stale state from the EL-sync handoff (most likely). fsn-1's MDBX/state at the snap-sync anchor (block 67,387,242) was populated by snap-sync from peers, not by executing every prior block locally. If snap-sync delivered a fee-currency contract storage state that's slightly off from canonical (e.g., one slot of one ERC-20 fee-currency contract at version N-1 from the snap-sync target), every CIP-64 tx that reads that slot via the fee-currency exchange-rate oracle would compute a different base_fee_in_erc20 → different gas_cost/effective_gas_price → different ERC-20 balances → divergent state root.

   This matches all the observations:

   • Only fsn-1 (the snap-synced node) is affected; fsn-0 (PVC-restored, never snap-synced) is not.
   • Failure clusters on CIP-64-bearing blocks only (0.6% of newPayloads).
   • The bug appeared day-1 post-EL-sync completion, not after a code change.
   • 99.4% of newPayloads are still VALID — only blocks whose execution actually reads the divergent state slot fail.

2. Receipts-root via bloom serialization. CeloRethReceiptBuilder (Lineage B) returns bloomless CeloReceipt variants and reth recomputes bloom at serialization. If reth's recomputation sees a different logs Vec than with_bloom() would have hashed in Lineage A (e.g., due to subtly different inclusion/exclusion of CIP-64 logs_pre/logs_post), receipts_root diverges and op-reth returns INVALID without state being involved. Worth a focused look at crates/celo-reth/src/receipt.rs and reth's BlockBody envelope encoding paths.

3. Image-level regression between sha-94b112b (fsn-0) and sha-44b69f7 (fsn-1). Neither SHA resolves as a git ref in this repo (git rev-parse returns the literal string back); they're docker tags from the build pipeline. Action: someone with access to the celo-kona-reth image build pipeline should map both tags to git commits and git diff between them on crates/alloy-celo-evm, crates/celo-revm, crates/celo-reth/src to rule out a regression. Candidate change points by inspection of git log --all:

   • de1fc9d — refactor: replace Cip64Storage VecDeque with single-slot Option (refactor: replace Cip64Storage VecDeque with single-slot Option #172)
   • 0398203 — fix(alloy-celo-evm): tolerate Cip64Storage slot reuse (current HEAD)
   • The unmerged karl/per-evm-cip-64-storage branch with several Cip64Storage rewiring commits

Smallest reproducer

Block 67,381,283 — canonical 0xdeea8244…480d4, INVALID 0xfcf002da…b43ad0. Witness preserved.

1. Load the witness file at /var/lib/op-reth/data/invalid_block_hooks/witness/67381283_0xfcf002da6978546a78f353c7c42378380d87a958bb1777680cda1e6ab7b43ad0.witness.re_executed.json into execution-verifier or celo-client.
2. Execute block 67,381,283 from the witness — capture: each CIP-64 tx's Cip64Info (debit_gas_used, credit_gas_used, base_fee_in_erc20, logs_pre/post), final state root, final receipts root, computed logs_bloom.
3. Compare against the canonical block at the same height (eth_getBlockByHash 0xdeea8244…480d4).
4. The first field that differs identifies the layer:
   • State root differs only → state-input bug (theory Configure Renovate #1 most likely)
   • Receipts root differs only → bloom serialization or logs merging bug (theory Add CeloTxEnv #2)
   • Both differ → execution divergence in revm (would require deeper investigation)
   • gas-used differs → CIP-64 gas accounting bug

Operational state on fsn-1

• Image: sha-44b69f7, ExEx forward-filling cleanly (no panics, no static-file inconsistency).
• op-node restarted at 2026-05-20 19:28 CEST; restart did not clear the loop because op-node's on-disk safedb still references the cached invalid hash. Reset rate climbed 1,645 → 17,982+ within a couple minutes.
• Canonical chain advancing fine via p2p gossip (current ~67,397,xxx). eth_call / eth_getProof unaffected. Only safe/finalized derivation is stuck.
• Pending operator action: mv /var/lib/op-node/safedb /var/lib/op-node/safedb.staletest.<ts> && podman restart op-node to break the loop (preserving safedb for debugging). Holding pending team review since fsn-1 is also the testbed for --proofs-history validation work.

Suggested next steps

• (maintainers' call) Probably close this issue since the original gap mechanism is resolved by PR refactor: replace Cip64Storage VecDeque with single-slot Option #172/feat(celo-reth): wire up --proofs-history ExEx (closes #174) #175, and open a separate issue for the CIP-64 state-root divergence — different failure mode (divergence vs unreadability), different reproducer (witness-based replay), different scope (EL-sync-only).
• (action item) Map sha-94b112b and sha-44b69f7 to git commits and diff crates/{alloy-celo-evm,celo-revm,celo-reth/src} between them to rule out theory Add CeloPrecompiles #3.
• (action item, lower priority) Run the witness reproducer on a clean kona-executor build to confirm theory Configure Renovate #1 vs Add CeloTxEnv #2.

Happy to file the new issue myself if maintainers prefer to keep this one scoped to the original gap.