[Node Operator Question] Pectra update causes missing trie node

[yorickdowne]

Are you running the most up to date node software?

• Yes

Did you check the documentation?

• Yes

Did you check for duplicate questions?

• Yes

Issue Description

We see an issue where op-geth shows missing trie node and op-node restarts continuously, when updating an existing full node to Pectra-ready releases. Instructions in release notes were followed.

Steps to Reproduce

Start with a deployment of op-geth 1.101411.8 and op-node 1.10.3.

Shut it down and observe op-geth logs. See that buffer_layers is less than 128 and take note of the block height

Update to op-geth 1.101500.0 and op-node 1.11.1, and start again

op-geth might show missing trie node errors. if it does, op-node will continuously restart

At that point a debug.setHead to the safe block height recorded earlier might help - but in my testing on base-sepolia, it did not.

This has happened to multiple Chainlink NOs. We don't have a solution other than resyncing from snapshot, if the debug.setHead doesn't do the trick.

I have another base-sepolia and so can attempt to reproduce this.

Which component is affected?

• op-geth
• op-node
• Other

Expected vs. Actual Behavior

Expected: Upgrade is successful if all steps are followed diligently

Actual: Not always, upgrade may fail

Environment Details

Linux, Docker, 32 GiB+ RAM, NVMe, 6+ cores

Software versions see reproduction steps

Protocol Information

Various. base-sepolia, optimism-mainnet - we've seen this on more than one chain

Node Logs

No response

Troubleshooting Attempts

debug.setHead, see reproduction steps

Additional Information

No response

Feedback

No response

[jsvisa]

debug.setHead will set the head block to an old block, and then resync the gap blocks from p2p. So after setHead, does the op-geth resynced again?

BTW, are you running the op-geth in archive mode with hash schema?

[yorickdowne]

No, debug.setHead to the block shown during shutdown didn't fix this for us. Other NOs have reported success.

It's intermittent. Our second base sepolia had a different pattern - op-geth was fine. We ended up resyncing it anyway because of op-node wanting a lengthy L1 walk back by hash after every start

These are not archive nodes, they are full nodes. Path afaik.

[jsvisa]

yeah, debug.setHead for full node will missing trie nodes if the block gap > 128 blocks.

[jsvisa]

If you want to use debug.setHead, I think you'd better use hash schema + archive mode.

[Chomtana]

You could also resync with snap sync. It might be the fastest solution for a full node.

https://docs.optimism.io/operators/node-operators/management/snap-sync

[yorickdowne]

Resyncing works. Given that this is happening to multiple node operators: Ideal would be not to have to resync and find root cause, so we can upgrade safely

[opfocus]

We will pay attention to this issue and try to report it.
Since there are currently no relevant best practices in docs and other locations, this is the best answer we can give.
If you don't mind, please close this question and mark if you think there is an acceptable answer.