[OP Stack Deployment] How to recover the L2 chain: finalized and safe block are stuck

[lesterli]

Did you check the documentation?

• Yes

Did you check for duplicate questions?

• Yes

Issue Description

We deployed a local OP devnet: L1: 1 geth node, 1 lighthouse bn node, 1 vc node; L2: 1 op-geth node, 1 op-node, 1 op-proposer, 1 op-batcher.
Now both the finalized block and safe block are stuck at block 374898. After checking the logs, we found two main issues:

1. The OP node failed to publish a newly created block. From the L1 Beacon node log, it indicated an L1 re-org occurred at block 124667 (slot 124671), we guess it caused validation failed for the L2 block 374899 due to its parent block 374898 having an L1 origin of 124677. Here is the re-org log from the L1 Beacon node.

Sep 26 07:17:49.515 INFO Beacon chain re-org                     reorg_distance: 23, new_slot: 154961, new_head: 0xa64a15a92052385f10e5949465b131b26d2550ce979593ee86e883b750a0e711, previous_slot: 124671, previous_head: 0x07e5170ba7b21162b94d16d21b38a9a76e6ff5a83b8b6b45b773fd5a29a71e0b, service: beacon

2. The OP node failed to fetch blobs during the reset process. While attempting to fetch blobs for historical blocks, it throws the error because the blobs of the block 124553 (slot 124557) were pruned on L1.

How can we recover this OP L2 chain?

Logs

1. the validation failed from the op-node log:

t=2024-09-26T09:58:03+0000 lvl=info msg="Started sequencing new block" parent=0x84da9213b65b37e18ddfd77d7359aa30b0decd6d7e597eea3d79a0daf00ce64f:374898 l1Origin=0xfaf047735ee34be93c9985bab299a5c9dd0a9f0b08a9c6bf0792e7ad819a7433:124667
t=2024-09-26T09:58:03+0000 lvl=info msg="Scheduled sequencer action" delta=-11.53µs
t=2024-09-26T09:58:03+0000 lvl=info msg="Sequencer sealed block" payloadID=0x03a16fc06a0a1667 block=0xdf7c823c43c390c2e3b621b63f78c7a0030e5d965e48aaa453a5c265a2ca97d6:374899 parent=0x84da9213b65b37e18ddfd77d7359aa30b0decd6d7e597eea3d79a0daf00ce64f:374898 txs=1 time=1727155049
t=2024-09-26T09:58:03+0000 lvl=info msg="Publishing signed execution payload on p2p" id=0xdf7c823c43c390c2e3b621b63f78c7a0030e5d965e48aaa453a5c265a2ca97d6:374899
t=2024-09-26T09:58:03+0000 lvl=info msg="Inserted block" hash=0xdf7c823c43c390c2e3b621b63f78c7a0030e5d965e48aaa453a5c265a2ca97d6 number=374899 state_root=0x977469e2f123261a422ca36d87c9c7d56ccdc8c7d9b31f10a73faf64abd0ad25 timestamp=1727155049 parent=0x84da9213b65b37e18ddfd77d7359aa30b0decd6d7e597eea3d79a0daf00ce64f prev_randao=0x702f11844b6ca1683c7659739b19f028dd7e5e150b57ce292286e9b93406425e fee_recipient=0x4200000000000000000000000000000000000011 txs=1 last_in_span=false derived_from=0x0000000000000000000000000000000000000000000000000000000000000000:0
t=2024-09-26T09:58:03+0000 lvl=info msg="Sequencer inserted block" block=0xdf7c823c43c390c2e3b621b63f78c7a0030e5d965e48aaa453a5c265a2ca97d6:374899 parent=0x84da9213b65b37e18ddfd77d7359aa30b0decd6d7e597eea3d79a0daf00ce64f:374898
t=2024-09-26T09:58:03+0000 lvl=warn msg="payload is too old" topic=blocksV3 timestamp=1727155049
t=2024-09-26T09:58:03+0000 lvl=warn msg="failed to publish newly created block" id=0xdf7c823c43c390c2e3b621b63f78c7a0030e5d965e48aaa453a5c265a2ca97d6:374899 hash=0xdf7c823c43c390c2e3b621b63f78c7a0030e5d965e48aaa453a5c265a2ca97d6 err="validation failed"

2. the derivation failed error: failed to fetch blobs in op-node log:

t=2024-09-26T13:43:47+0000 lvl=warn msg="failed to notify engine driver of L1 finalized block change" err="context deadline exceeded"
t=2024-09-26T13:43:48+0000 lvl=info msg="Walking back L1Block by hash" curr=0xfaf047735ee34be93c9985bab299a5c9dd0a9f0b08a9c6bf0792e7ad819a7433:124667 next=0x7212927f94ee1db39d8f24edaa85db53671fbed3963a06aef9727e18dfdd9513:124666 l2block=0x93280b4f341afa11ddb9fb0950e6247a375f0277b83b05a60e2a64a77f1e4f59:373997
...
t=2024-09-26T13:43:48+0000 lvl=info msg="Walking back L1Block by hash" curr=0x39fa70f1e28ae852afcf6380d0f5b55bb3362cb56db261e5c57ca9a146013571:124664 next=0xa9f9d18558d0d1377128f574d4b925c159141215abb23a796ba5589bb6a4a45a:124663 l2block=0x7e76171ff0625d8af6e3e0066854c3d7f31293c7600ae71370987c461a2548b8:373989
t=2024-09-26T13:43:48+0000 lvl=info msg="Hit finalized L2 head, returning immediately" unsafe=0x9dabdd618c09c580d6f568e160bb5b0c53fc4db66a01f4ecddac0143b282c4cb:476339 safe=0x7e76171ff0625d8af6e3e0066854c3d7f31293c7600ae71370987c461a2548b8:373989 finalized=0x7e76171ff0625d8af6e3e0066854c3d7f31293c7600ae71370987c461a2548b8:373989 unsafe_origin=0xbaed7eedf0770166f1305b0b7193a9728badc8db2f5b1acb3911562100c5b875:128479 safe_origin=0xa9f9d18558d0d1377128f574d4b925c159141215abb23a796ba5589bb6a4a45a:124663
t=2024-09-26T13:43:48+0000 lvl=info msg="Engine reset confirmed, sequencer may continue" next=true
t=2024-09-26T13:43:48+0000 lvl=info msg="Scheduled sequencer action" delta=1.999931781s
t=2024-09-26T13:43:48+0000 lvl=info msg="New L1 safe block" l1_safe=0xbaed7eedf0770166f1305b0b7193a9728badc8db2f5b1acb3911562100c5b875:128479
t=2024-09-26T13:43:48+0000 lvl=info msg="New L1 finalized block" l1_finalized=0x2b53c44f7276ffafeddb7eeb23c3a302e782ec1df97dc48cfc8a4d3ab83f177c:128471
t=2024-09-26T13:43:48+0000 lvl=info msg="Rewinding derivation-pipeline L1 traversal to handle reset"
t=2024-09-26T13:43:49+0000 lvl=info msg="completed reset of derivation pipeline" origin=0x3f8abe3687a090c9fd53281f247c532f022ce424cffd4bdd5197340489b5d7cb:124543
t=2024-09-26T13:43:49+0000 lvl=info msg="New L1 safe block" l1_safe=0xbaed7eedf0770166f1305b0b7193a9728badc8db2f5b1acb3911562100c5b875:128479
t=2024-09-26T13:43:49+0000 lvl=info msg="New L1 finalized block" l1_finalized=0x2b53c44f7276ffafeddb7eeb23c3a302e782ec1df97dc48cfc8a4d3ab83f177c:128471
t=2024-09-26T13:43:49+0000 lvl=info msg="New L1 finalized block" l1_finalized=0x2b53c44f7276ffafeddb7eeb23c3a302e782ec1df97dc48cfc8a4d3ab83f177c:128471
t=2024-09-26T13:43:49+0000 lvl=info msg="Reset of L1Retrieval done" origin=0x3f8abe3687a090c9fd53281f247c532f022ce424cffd4bdd5197340489b5d7cb:124543
t=2024-09-26T13:43:49+0000 lvl=info msg="New L1 safe block" l1_safe=0xbaed7eedf0770166f1305b0b7193a9728badc8db2f5b1acb3911562100c5b875:128479
...
t=2024-09-26T13:43:49+0000 lvl=info msg="New L1 safe block" l1_safe=0xfa6934c3bc58e3fe32e77c9caa894e1b94530585f0536edddd516620be66d210:128486
t=2024-09-26T13:43:49+0000 lvl=warn msg="L1 head signal indicates a possible L1 re-org" old_l1_head=0x7b0f8818fa57095417f1a88580c372aae70c98644c2c49543f76ae90ad2e5f7a:128496 new_l1_head_parent=0xb4f3b8de8602aee9b7d90349d2a5b4fa0bd7c36ed3de46cca068d275305f2682 new_l1_head=0xf5d0122b83477d2dbf9226ff6a766daf01f3b1acc64a5913d6def8313a5c1b29:128515
t=2024-09-26T13:43:49+0000 lvl=info msg="New L1 safe block" l1_safe=0xfa6934c3bc58e3fe32e77c9caa894e1b94530585f0536edddd516620be66d210:128486
...
t=2024-09-26T13:43:49+0000 lvl=info msg="Advancing bq origin" origin=0xfbfb16900778ee75c39897b1281e95d7f3eeaa6eb977cb3252372bf1f4d2d887:124544 originBehind=true
t=2024-09-26T13:43:49+0000 lvl=info msg="Advancing bq origin" origin=0xd121c359b817f391db2f58f56af3eb948589dc987e1348b7924468894b9b2eb5:124545 originBehind=true
...
t=2024-09-26T13:43:49+0000 lvl=warn msg="Engine temporary error" err="derivation failed: temp: failed to fetch blobs: failed to get blob sidecars for L1BlockRef 0xb679121a666393289a4bbbaf22479457b2939b4022b7becf95d3b9efd9b60f6f:124553: failed to fetch blob sidecars for slot 124557 block 0xb679121a666393289a4bbbaf22479457b2939b4022b7becf95d3b9efd9b60f6f:124553: #returned blobs(0) != #requested blobs(1)"
t=2024-09-26T13:43:50+0000 lvl=info msg="Started sequencing new block" parent=0x9dabdd618c09c580d6f568e160bb5b0c53fc4db66a01f4ecddac0143b282c4cb:476339 l1Origin=0x98a188132678635dc80fa204def8ddd907e71a84e78f251cfe9cd93edebafc11:128480

Additional Information

No response

Feedback

No response

[imtipi]

The OP node failed to fetch blobs during the reset process. While attempting to fetch blobs for historical blocks, it throws the error because the blobs of the block 124553 (slot 124557) were pruned on L1.

use rpc that has non-pruned blobs,example: quicknode

[lesterli]

It is a local private devnet 😂

[Chomtana]

How are you launching a private devnet? By make devnet-up?

However, since devnet rely on only one node, we may not be able to recover pruned blobs.

[lesterli]

Yes, it is too bad, but thank you.

[bap2pecs]

How are you launching a private devnet? By make devnet-up?

However, since devnet rely on only one node, we may not be able to recover pruned blobs.

but all tx data is still on the L2 node. so technically we can still derive the canonical chain from it?

[Chomtana]

I think if an L2 experiences a reorg, the transaction data for every reorged block would also be removed.

[bap2pecs]

I think if an L2 experiences a reorg, the transaction data for every reorged block would also be removed.

ah I see. so the txs data is removed from L2 node and since L1 blob was pruned, there is no way to recover