[OP Stack Deployment] L2OutputOracleProxy.latestBlockNumber()(uint256) is stuck

[askucher]

Did you check the documentation?

• Yes

Did you check for duplicate questions?

• Yes

Issue Description

Op stack version: v1.9.1 without alt-da
Smart Contract version: beta.1
Status: Successfully tested deposits and withdrawals, and working well for a while.
Block time: 1 sec
Settlement layer: Base Sepolia

Everything was functioning smoothly for weeks, but recently the op-proposer has become stuck while attempting to update the L2OutputOracleProxy.latestBlockNumber()(uint256).

The system is generating extensive logs similar to the following:

elapsed" currentBlockNumber=728596 nextBlockNumber=728640
5|op-propo | t=2024-12-18T14:38:21+0000 lvl=debug msg="Proposer submission interval has not elapsed" currentBlockNumber=728596 nextBlockNumber=728640
5|op-propo | t=2024-12-18T14:38:33+0000 lvl=debug msg="Proposer submission interval has not elapsed" currentBlockNumber=728596 nextBlockNumber=728640

5|op-proposer | t=2024-12-18T14:38:45+0000 lvl=debug msg="Proposer submission interval has not elapsed" currentBlockNumber=728596 nextBlockNumber=728640

The last transaction occurred two days ago (view on Sepolia BaseScan), and there is no clear indication of the cause, as other logs do not show any issues.

We need to understand how to generally debug this situation and determine how to resolve it.

---

Logs

sync-status

######## Dfine ports sepolia #########

.envrc inherited
------------------------------------------------------------
------------------------------------------------------------
------------------------------------------------------------
SECTION: sync_status
------------------------------------------------------------
------------------------------------------------------------
------------------------------------------------------------
{
  "current_l1": {
    "hash": "0xc8323950e8754a77184c16dbe2d787c4dd84efe496541459efb506028f68dddd",
    "number": 19383380,
    "parentHash": "0x446e34e36896c0458c28c3099a20df6ff94914f2f21acb0681e4d7e3e3339abf",
    "timestamp": 1734535048
  },
  "current_l1_finalized": {
    "hash": "0xfefb10c49a3cfdd3277cce9ae2a74b091f5c74c93da2b0a540c3a0d98ff2cafb",
    "number": 19382722,
    "parentHash": "0xf88bfc3e6750ae66383122a25e212a3ae4662cb650367feb567a229d7ccdbf1a",
    "timestamp": 1734533732
  },
  "head_l1": {
    "hash": "0x3248691ddb37847ef74f6cff80264690ce2dcfd21466c4171c135d508b9664fa",
    "number": 19383384,
    "parentHash": "0x7f487ee042dbe041abedece87cbcfffc175b564340a0cbef991e9ccb3a46139e",
    "timestamp": 1734535056
  },
  "safe_l1": {
    "hash": "0x575a6aecd661d41412fec37a3d5bad415859917a2ecca7e7ba40aab477136070",
    "number": 19383303,
    "parentHash": "0xb899c7ddf90e99fea7c2ad58677e876ebb43b9c1e4d5651c056b5dc2c676e0e8",
    "timestamp": 1734534894
  },
  "finalized_l1": {
    "hash": "0xfefb10c49a3cfdd3277cce9ae2a74b091f5c74c93da2b0a540c3a0d98ff2cafb",
    "number": 19382722,
    "parentHash": "0xf88bfc3e6750ae66383122a25e212a3ae4662cb650367feb567a229d7ccdbf1a",
    "timestamp": 1734533732
  },
  "unsafe_l2": {
    "hash": "0xd231bef986346125ec1c16954dfe1d183e44c5562d9b91d70913f23b401ff139",
    "number": 816980,
    "parentHash": "0x29d518c76b204afddab1e5e4f762fa56ff26fe301bc504995ed883123c43f058",
    "timestamp": 1734535064,
    "l1origin": {
      "hash": "0x0ff64048854fe6be9477de6b6cf61b8d354599e24a42b671380263e480acfeb7",
      "number": 19383297
    },
    "sequenceNumber": 0
  },
  "safe_l2": {
    "hash": "0x33e2c2aaf87d7505bac8389df4582ec968a9aabede9f7d60b54b79a2e9a2e3c0",
    "number": 816963,
    "parentHash": "0xbc92297f49b6c8b208340327a4cc8237b4ad8d3f4bc0400d2df102f86608b054",
    "timestamp": 1734535030,
    "l1origin": {
      "hash": "0x8dc99abd78fe6f4c4c1996766945ddf516fa2f2d02c435635274a6c03dde9430",
      "number": 19383280
    },
    "sequenceNumber": 0
  },
  "finalized_l2": {
    "hash": "0x6ecbc8f0b770b56a34b8c4f110a4e8a823a551abb56c9a787e0038aa1fbefdeb",
    "number": 728596,
    "parentHash": "0xfa20f1bed22a462f264b983e6858c1e405d41c0aefb6a89feacdcaa3a28549a2",
    "timestamp": 1734358296,
    "l1origin": {
      "hash": "0x51bbfff50561fb48714b47aa9d6ddacc4a4c9e88fd6e225a77e763e485896f6b",
      "number": 19294919
    },
    "sequenceNumber": 0
  },
  "pending_safe_l2": {
    "hash": "0x33e2c2aaf87d7505bac8389df4582ec968a9aabede9f7d60b54b79a2e9a2e3c0",
    "number": 816963,
    "parentHash": "0xbc92297f49b6c8b208340327a4cc8237b4ad8d3f4bc0400d2df102f86608b054",
    "timestamp": 1734535030,
    "l1origin": {
      "hash": "0x8dc99abd78fe6f4c4c1996766945ddf516fa2f2d02c435635274a6c03dde9430",
      "number": 19383280
    },
    "sequenceNumber": 0
  }
}

op-proposer

root@chain-testnet-base:/opt/app/op_stack_deployer# pm2 logs op-proposer
[TAILING] Tailing last 15 lines for [op-proposer] process (change the value with --lines option)
/root/.pm2/logs/op-proposer-error.log last 15 lines:
/root/.pm2/logs/op-proposer-out.log last 15 lines:
5|op-propo | t=2024-12-18T14:35:45+0000 lvl=debug msg="Proposer submission interval has not elapsed" currentBlockNumber=728596 nextBlockNumber=728640
5|op-propo | t=2024-12-18T14:35:57+0000 lvl=debug msg="Proposer submission interval has not elapsed" currentBlockNumber=728596 nextBlockNumber=728640
5|op-propo | t=2024-12-18T14:36:09+0000 lvl=debug msg="Proposer submission interval has not elapsed" currentBlockNumber=728596 nextBlockNumber=728640
5|op-propo | t=2024-12-18T14:36:21+0000 lvl=debug msg="Proposer submission interval has not elapsed" currentBlockNumber=728596 nextBlockNumber=728640
5|op-propo | t=2024-12-18T14:36:33+0000 lvl=debug msg="Proposer submission interval has not elapsed" currentBlockNumber=728596 nextBlockNumber=728640
5|op-propo | t=2024-12-18T14:36:45+0000 lvl=debug msg="Proposer submission interval has not elapsed" currentBlockNumber=728596 nextBlockNumber=728640
5|op-propo | t=2024-12-18T14:36:57+0000 lvl=debug msg="Proposer submission interval has not elapsed" currentBlockNumber=728596 nextBlockNumber=728640
5|op-propo | t=2024-12-18T14:37:09+0000 lvl=debug msg="Proposer submission interval has not elapsed" currentBlockNumber=728596 nextBlockNumber=728640
5|op-propo | t=2024-12-18T14:37:21+0000 lvl=debug msg="Proposer submission interval has not elapsed" currentBlockNumber=728596 nextBlockNumber=728640
5|op-propo | t=2024-12-18T14:37:33+0000 lvl=debug msg="Proposer submission interval has not elapsed" currentBlockNumber=728596 nextBlockNumber=728640
5|op-propo | t=2024-12-18T14:37:45+0000 lvl=debug msg="Proposer submission interval has not elapsed" currentBlockNumber=728596 nextBlockNumber=728640
5|op-propo | t=2024-12-18T14:37:57+0000 lvl=debug msg="Proposer submission interval has not elapsed" currentBlockNumber=728596 nextBlockNumber=728640
5|op-propo | t=2024-12-18T14:38:09+0000 lvl=debug msg="Proposer submission interval has not elapsed" currentBlockNumber=728596 nextBlockNumber=728640
5|op-propo | t=2024-12-18T14:38:21+0000 lvl=debug msg="Proposer submission interval has not elapsed" currentBlockNumber=728596 nextBlockNumber=728640
5|op-propo | t=2024-12-18T14:38:33+0000 lvl=debug msg="Proposer submission interval has not elapsed" currentBlockNumber=728596 nextBlockNumber=728640

5|op-proposer  | t=2024-12-18T14:38:45+0000 lvl=debug msg="Proposer submission interval has not elapsed" currentBlockNumber=728596 nextBlockNumber=728640

op-batcher . Also batcher works fine because we can see result here https://sepolia.basescan.org/address/0x256DE50807FDf545d5D311e1a8932C6AB8489Da8

/root/.pm2/logs/op-batcher-error.log last 15 lines:
/root/.pm2/logs/op-batcher-out.log last 15 lines:
4|op-batch | losed" id=7919d0a611a5e9bb0e2a904985cc3fd9 blocks_pending=0 num_frames=1 input_bytes=82 output_bytes=121 oldest_l1_origin=0xb1af667b51133e876607d52c12cd843efe68cb3816db948be4aa8abd5702734a:19382124 l1_origin=0xb1af667b51133e876607d52c12cd843efe68cb3816db948be4aa8abd5702734a:19382124 oldest_l2=0x2c536565d245ff2a5e97e92f8e6f4ef1177e4ef289352653694bae36f5a5e008:815807 latest_l2=0x2c536565d245ff2a5e97e92f8e6f4ef1177e4ef289352653694bae36f5a5e008:815807 full_reason="channel full: max channel duration reached" compr_ratio=1.475609756097561 latest_l1_origin=0xb1af667b51133e876607d52c12cd843efe68cb3816db948be4aa8abd5702734a:19382124
4|op-batch | t=2024-12-18T14:39:12+0000 lvl=info msg="Building Calldata transaction candidate" size=122
4|op-batch | t=2024-12-18T14:39:24+0000 lvl=info msg="Transaction confirmed" service=batcher tx=0x5430c18ae7c450bc818b758d307d078c1d86db84171f7c8418961c25734d3cb2 block=0xedf8306963bee046b90809924ed08fe6ea439c4a2e4ca7f1e8c8d04b89b0b500:19382233 effectiveGasPrice=1000000459
4|op-batch | t=2024-12-18T14:39:24+0000 lvl=info msg="Handling receipt" id="{id:[{chID:[86 131 125 95 185 244 220 105 224 108 244 92 162 41 227 67] frameNumber:0}] isCancel:false isBlob:false}"
4|op-batch | t=2024-12-18T14:39:24+0000 lvl=info msg="Transaction confirmed" tx_id=56837d5fb9f4dc69e06cf45ca229e343:0 tx=0x5430c18ae7c450bc818b758d307d078c1d86db84171f7c8418961c25734d3cb2 block=0xedf8306963bee046b90809924ed08fe6ea439c4a2e4ca7f1e8c8d04b89b0b500:19382233
4|op-batch | t=2024-12-18T14:39:24+0000 lvl=info msg="Channel is fully submitted" id=56837d5fb9f4dc69e06cf45ca229e343 min_inclusion_block=19382233 max_inclusion_block=19382233
4|op-batch | t=2024-12-18T14:39:24+0000 lvl=warn msg="Last submitted block lagged behind L2 safe head: batch submission will continue from the safe head now" last=0x2c536565d245ff2a5e97e92f8e6f4ef1177e4ef289352653694bae36f5a5e008:815807 safe=0x1fee0750dd70c65afb6808918d8c6ae15a8a888513964600e79131f5a5115e99:815815
4|op-batch | t=2024-12-18T14:39:24+0000 lvl=warn msg="Error calculating L2 block range" err="L2 safe head ahead of L2 unsafe head"
4|op-batch | t=2024-12-18T14:39:24+0000 lvl=info msg="Publishing transaction" service=batcher tx=0x885a8835ec00bb152033b61e2014739d6e47946b0fc51efb5d4f525cbc591f7a nonce=96713 gasTipCap=1000000000 gasFeeCap=3000000000 gasLimit=22820 tx=0x885a8835ec00bb152033b61e2014739d6e47946b0fc51efb5d4f525cbc591f7a
4|op-batch | t=2024-12-18T14:39:24+0000 lvl=info msg="Transaction successfully published" service=batcher tx=0x885a8835ec00bb152033b61e2014739d6e47946b0fc51efb5d4f525cbc591f7a nonce=96713 gasTipCap=1000000000 gasFeeCap=3000000000 gasLimit=22820 tx=0x885a8835ec00bb152033b61e2014739d6e47946b0fc51efb5d4f525cbc591f7a
4|op-batch | t=2024-12-18T14:39:25+0000 lvl=warn msg="Found L2 reorg" block_number=815816
4|op-batch | t=2024-12-18T14:39:25+0000 lvl=info msg="Channel manager is closing"
4|op-batch | t=2024-12-18T14:39:25+0000 lvl=info msg="Channel is in-flight and will need to be submitted after close" id=7919d0a611a5e9bb0e2a904985cc3fd9 confirmed=0 pending=1
4|op-batch | t=2024-12-18T14:39:25+0000 lvl=info msg="Reviewed all pending channels on close" remaining=1

4|op-batcher  | t=2024-12-18T14:39:36+0000 lvl=info msg="Transaction confirmed" service=batcher tx=0x885a8835ec00bb152033b61e2014739d6e47946b0fc51efb5d4f525cbc591f7a block=0x068bd380c277d6547ffc443fff1ee3cb16851d54d7db07d2a0e8fe31cf2626f1:19382239 effectiveGasPrice=1000000460
4|op-batcher  | t=2024-12-18T14:39:36+0000 lvl=info msg="Clearing state"
4|op-batcher  | t=2024-12-18T14:39:36+0000 lvl=info msg="Handling receipt" id="{id:[{chID:[121 25 208 166 17 165 233 187 14 42 144 73 133 204 63 217] frameNumber:0}] isCancel:false isBlob:false}"
4|op-batcher  | t=2024-12-18T14:39:36+0000 lvl=info msg="Transaction confirmed" tx_id=7919d0a611a5e9bb0e2a904985cc3fd9:0 tx=0x885a8835ec00bb152033b61e2014739d6e47946b0fc51efb5d4f525cbc591f7a block=0x068bd380c277d6547ffc443fff1ee3cb16851d54d7db07d2a0e8fe31cf2626f1:19382239
4|op-batcher  | t=2024-12-18T14:39:36+0000 lvl=info msg="Channel is fully submitted" id=7919d0a611a5e9bb0e2a904985cc3fd9 min_inclusion_block=19382239 max_inclusion_block=19382239
4|op-batcher  | t=2024-12-18T14:39:36+0000 lvl=info msg="Clearing state with safe L1 origin" origin=0x19539a77ccf58d8a6a0a50c0f7c0b158d30547a242c75c74ec5c83b83878ac9c:19382138
4|op-batcher  | t=2024-12-18T14:39:36+0000 lvl=info msg="State cleared"
4|op-batcher  | t=2024-12-18T14:39:36+0000 lvl=info msg="Starting batch-submitter work at safe-head" safe=0x0dd30409830642719594dfcff305df214f35b501bfdc5475ebe9427e81997462:815821
4|op-batcher  | t=2024-12-18T14:39:36+0000 lvl=info msg="Added L2 block to local state" block=0xea4550286e4f80df5192314cee0bef3850debb2c62af00e00b29e2fe2383778a:815822 tx_count=2 time=1734532748
4|op-batcher  | t=2024-12-18T14:39:36+0000 lvl=info msg="Added L2 block to local state" block=0x7c7851ec6a8e4b7185fa60f2d93a35487809e5f35f730dccd2f45152a4b9a5a9:815823 tx_count=3 time=1734532750
4|op-batcher  | t=2024-12-18T14:39:37+0000 lvl=info msg="Created channel" id=115136579e9a36ad3901d1db999d5dc7 l1Head=0xfc5c9e791bffaf0115759eed5c684c3365875371f013d68cef1784f03f76abf8:19382244 l1OriginLastClosedChannel=0x19539a77ccf58d8a6a0a50c0f7c0b158d30547a242c75c74ec5c83b83878ac9c:19382138 blocks_pending=2 batch_type=0 compression_algo=zlib target_num_frames=1 max_frame_size=119999 use_blobs=false
4|op-batcher  | t=2024-12-18T14:39:37+0000 lvl=info msg="Channel closed" id=115136579e9a36ad3901d1db999d5dc7 blocks_pending=0 num_frames=1 input_bytes=613 output_bytes=489 oldest_l1_origin=0x4316c736ffac0cd68d0d4d791b053d140238a886a4e6c3e4a29233e4c7116647:19382139 l1_origin=0x7646966deb3096b7d97e640514f9b81be49cc70bc4f9e5bddc01e900b91559b8:19382140 oldest_l2=0xea4550286e4f80df5192314cee0bef3850debb2c62af00e00b29e2fe2383778a:815822 latest_l2=0x7c7851ec6a8e4b7185fa60f2d93a35487809e5f35f730dccd2f45152a4b9a5a9:815823 full_reason="channel full: max channel duration reached" compr_ratio=0.797716150081566 latest_l1_origin=0x7646966deb3096b7d97e640514f9b81be49cc70bc4f9e5bddc01e900b91559b8:19382140
4|op-batcher  | t=2024-12-18T14:39:37+0000 lvl=info msg="Building Calldata transaction candidate" size=490
4|op-batcher  | t=2024-12-18T14:39:37+0000 lvl=info msg="Added L2 block to local state" block=0x6d581a7dd221917e9e20f77a29a61579604d1114c5899053d09ba236f8ce86c0:815824 tx_count=2 time=1734532752
4|op-batcher  | t=2024-12-18T14:39:37+0000 lvl=info msg="Added L2 block to local state" block=0xd8a215666a9ec4c9124ae897b045f641ad0122b439675c7a3c59225e1d89abd6:815825 tx_count=1 time=1734532754
4|op-batcher  | t=2024-12-18T14:39:37+0000 lvl=info msg="Created channel" id=dfcdb4ea2c69af74d4ae3f96568c2b77 l1Head=0xfc5c9e791bffaf0115759eed5c684c3365875371f013d68cef1784f03f76abf8:19382244 l1OriginLastClosedChannel=0x7646966deb3096b7d97e640514f9b81be49cc70bc4f9e5bddc01e900b91559b8:19382140 blocks_pending=2 batch_type=0 compression_algo=zlib target_num_frames=1 max_frame_size=119999 use_blobs=false
4|op-batcher  | t=2024-12-18T14:39:37+0000 lvl=info msg="Channel closed" id=dfcdb4ea2c69af74d4ae3f96568c2b77 blocks_pending=0 num_frames=1 input_bytes=313 output_bytes=332 oldest_l1_origin=0x3aee866f2f14c084e949adc1e02e722cacc2e1cb33b1475a78fec5e47c7dd336:19382141 l1_origin=0xc7415c2de26975ce3398ab3d719f1b4b87a488d8ca2966a744a38dcb1fa37a82:19382142 oldest_l2=0x6d581a7dd221917e9e20f77a29a61579604d1114c5899053d09ba236f8ce86c0:815824 latest_l2=0xd8a215666a9ec4c9124ae897b045f641ad0122b439675c7a3c59225e1d89abd6:815825 full_reason="channel full: max channel duration reached" compr_ratio=1.060702875399361 latest_l1_origin=0xc7415c2de26975ce3398ab3d719f1b4b87a488d8ca2966a744a38dcb1fa37a82:19382142
4|op-batcher  | t=2024-12-18T14:39:37+0000 lvl=info msg="Building Calldata transaction candidate" size=333
4|op-batcher  | t=2024-12-18T14:39:37+0000 lvl=info msg="Publishing transaction" service=batcher tx=0x8a54efa169b1e81be71d337b823ed3c25f20ab2f7b242fd1c42e0dc3f26990d1 nonce=96714 gasTipCap=1000000000 gasFeeCap=3000000000 gasLimit=28744 tx=0x8a54efa169b1e81be71d337b823ed3c25f20ab2f7b242fd1c42e0dc3f26990d1
4|op-batcher  | t=2024-12-18T14:39:37+0000 lvl=info msg="Transaction successfully published" service=batcher tx=0x8a54efa169b1e81be71d337b823ed3c25f20ab2f7b242fd1c42e0dc3f26990d1 nonce=96714 gasTipCap=1000000000 gasFeeCap=3000000000 gasLimit=28744 tx=0x8a54efa169b1e81be71d337b823ed3c25f20ab2f7b242fd1c42e0dc3f26990d1

op-node

[TAILING] Tailing last 15 lines for [op-node] process (change the value with --lines option)
/root/.pm2/logs/op-node-error.log last 15 lines:
/root/.pm2/logs/op-node-out.log last 15 lines:
8|op-node  | =info msg="Scheduled sequencer action" delta=-3.049µs
8|op-node  | t=2024-12-18T14:41:40+0000 lvl=info msg="Sequencer sealed block" payloadID=0x039f4633c69dd525 block=0x3027e35db2ca5232810672189dfd886e409d6354a4d12f689eb7914d7a29b953:815897 parent=0x13a12676a7197c0211eeab6cfb41df52169c356fd96c13d0daa68f56f87528ab:815896 txs=1 time=1734532898
8|op-node  | t=2024-12-18T14:41:40+0000 lvl=info msg="Inserted block" hash=0x3027e35db2ca5232810672189dfd886e409d6354a4d12f689eb7914d7a29b953 number=815897 state_root=0x4ce704e7920113bd7b3dbbe98f8691796d0848ce50593634315c37eafff46f7f timestamp=1734532898 parent=0x13a12676a7197c0211eeab6cfb41df52169c356fd96c13d0daa68f56f87528ab prev_randao=0xce9d99522a993f5b6e5a6e3aa3d0fa5f195530e81248c22c270026c6ecc64e2a fee_recipient=0x4200000000000000000000000000000000000011 txs=1 last_in_span=false derived_from=0x0000000000000000000000000000000000000000000000000000000000000000:0
8|op-node  | t=2024-12-18T14:41:40+0000 lvl=info msg="Sequencer inserted block" block=0x3027e35db2ca5232810672189dfd886e409d6354a4d12f689eb7914d7a29b953:815897 parent=0x13a12676a7197c0211eeab6cfb41df52169c356fd96c13d0daa68f56f87528ab:815896
8|op-node  | t=2024-12-18T14:41:40+0000 lvl=info msg="Scheduled sequencer action" delta=-2.503µs
8|op-node  | t=2024-12-18T14:41:40+0000 lvl=info msg="Started sequencing new block" parent=0x3027e35db2ca5232810672189dfd886e409d6354a4d12f689eb7914d7a29b953:815897 l1Origin=0x4c8909fb16f1506ef57e83a2dce9599ccc0dd62e121ae060639cc913157b140d:19382215
8|op-node  | t=2024-12-18T14:41:40+0000 lvl=info msg="Scheduled sequencer action" delta=-7.778µs
8|op-node  | t=2024-12-18T14:41:40+0000 lvl=info msg="Sequencer sealed block" payloadID=0x031213b715965221 block=0x16a3a814be8a49e4a0bafebb34e4b3c2076499700c23444c345d1dbd8da11873:815898 parent=0x3027e35db2ca5232810672189dfd886e409d6354a4d12f689eb7914d7a29b953:815897 txs=1 time=1734532900
8|op-node  | t=2024-12-18T14:41:40+0000 lvl=info msg="Inserted block" hash=0x16a3a814be8a49e4a0bafebb34e4b3c2076499700c23444c345d1dbd8da11873 number=815898 state_root=0x2fc1a1745608cb94b652bd878c3b19a1ae5b148aa455e683543e3b2e3189186f timestamp=1734532900 parent=0x3027e35db2ca5232810672189dfd886e409d6354a4d12f689eb7914d7a29b953 prev_randao=0xce9d99522a993f5b6e5a6e3aa3d0fa5f195530e81248c22c270026c6ecc64e2a fee_recipient=0x4200000000000000000000000000000000000011 txs=1 last_in_span=false derived_from=0x0000000000000000000000000000000000000000000000000000000000000000:0
8|op-node  | t=2024-12-18T14:41:40+0000 lvl=info msg="Sequencer inserted block" block=0x16a3a814be8a49e4a0bafebb34e4b3c2076499700c23444c345d1dbd8da11873:815898 parent=0x3027e35db2ca5232810672189dfd886e409d6354a4d12f689eb7914d7a29b953:815897
8|op-node  | t=2024-12-18T14:41:40+0000 lvl=info msg="Scheduled sequencer action" delta=-1.909µs
8|op-node  | t=2024-12-18T14:41:40+0000 lvl=info msg="Started sequencing new block" parent=0x16a3a814be8a49e4a0bafebb34e4b3c2076499700c23444c345d1dbd8da11873:815898 l1Origin=0xb32710d2f9985d928432c76383fc519b3ea88eaea19d19dc0af8f708efc1ca15:19382216
8|op-node  | t=2024-12-18T14:41:40+0000 lvl=info msg="Scheduled sequencer action" delta=1.093551893s

8|op-node  | t=2024-12-18T14:41:41+0000 lvl=info msg="Sequencer sealed block" payloadID=0x03cd5497df2a1471 block=0x8108850a7c093833764fc67f5f14cb868c6b6485d375b624ab4b0b42c6d2276f:815899 parent=0x16a3a814be8a49e4a0bafebb34e4b3c2076499700c23444c345d1dbd8da11873:815898 txs=1 time=1734532902
8|op-node  | t=2024-12-18T14:41:41+0000 lvl=info msg="Inserted block" hash=0x8108850a7c093833764fc67f5f14cb868c6b6485d375b624ab4b0b42c6d2276f number=815899 state_root=0x717f2c21bec7cc5753975fd76db098be6b68b5045a50316c6bd12b4cbb3a9198 timestamp=1734532902 parent=0x16a3a814be8a49e4a0bafebb34e4b3c2076499700c23444c345d1dbd8da11873 prev_randao=0x472d25fef8653d9dd2e3b973c6b2588e919358bdd4e79fb32da06225de1c34ef fee_recipient=0x4200000000000000000000000000000000000011 txs=1 last_in_span=false derived_from=0x0000000000000000000000000000000000000000000000000000000000000000:0
8|op-node  | t=2024-12-18T14:41:41+0000 lvl=info msg="Sequencer inserted block" block=0x8108850a7c093833764fc67f5f14cb868c6b6485d375b624ab4b0b42c6d2276f:815899 parent=0x16a3a814be8a49e4a0bafebb34e4b3c2076499700c23444c345d1dbd8da11873:815898
8|op-node  | t=2024-12-18T14:41:41+0000 lvl=info msg="Scheduled sequencer action" delta=44.128561ms
8|op-node  | t=2024-12-18T14:41:42+0000 lvl=info msg="Started sequencing new block" parent=0x8108850a7c093833764fc67f5f14cb868c6b6485d375b624ab4b0b42c6d2276f:815899 l1Origin=0x1bc606c2bfaf35f0c564cb31b7bc4493b702635c1649ba5639f1cddcfc5d5275:19382217
8|op-node  | t=2024-12-18T14:41:42+0000 lvl=info msg="Scheduled sequencer action" delta=1.607824031s

Full log:
pm2_logs_op-node (1).txt

op-geth

[TAILING] Tailing last 15 lines for [op-geth] process (change the value with --lines option)
/root/.pm2/logs/op-geth-out.log last 15 lines:
/root/.pm2/logs/op-geth-error.log last 15 lines:
3|op-geth  | WARN [12-18|14:42:50.464] Served eth_sendRawTransaction            conn=172.71.146.218:9852 reqid=3 duration="45.381µs" err="max priority fee per gas higher than max fee per gas"
3|op-geth  | INFO [12-18|14:42:50.723] Starting work on payload                 id=0x03f74b430134c296
3|op-geth  | INFO [12-18|14:42:50.723] Updated payload                          id=0x03f74b430134c296 number=815,933 hash=e714c9..86734e txs=1    withdrawals=0 gas=46606      fees=0               root=f32e2d..2a0ef9 elapsed="555.034µs"
3|op-geth  | INFO [12-18|14:42:50.723] Stopping work on payload                 id=0x03f74b430134c296 reason=delivery     elapsed="692.634µs"
3|op-geth  | INFO [12-18|14:42:50.726] Imported new potential chain segment     number=815,933 hash=e714c9..86734e blocks=1 txs=1    mgas=0.047  elapsed=2.032ms      mgasps=22.927   snapdiffs=1.82MiB    triedirty=0.00B
3|op-geth  | INFO [12-18|14:42:50.728] Chain head was updated                   number=815,933 hash=e714c9..86734e root=f32e2d..2a0ef9 elapsed="441.798µs"
3|op-geth  | WARN [12-18|14:42:51.122] Served eth_sendRawTransaction            conn=172.71.147.2:42726 reqid=3 duration="43.758µs" err="max priority fee per gas higher than max fee per gas"
3|op-geth  | INFO [12-18|14:42:51.691] Starting work on payload                 id=0x035900c1b4c555a7
3|op-geth  | INFO [12-18|14:42:51.691] Updated payload                          id=0x035900c1b4c555a7 number=815,934 hash=997b67..923156 txs=1    withdrawals=0 gas=46606      fees=0               root=8a7ca3..681714 elapsed="462.548µs"
3|op-geth  | WARN [12-18|14:42:51.904] Served eth_sendRawTransaction            conn=172.71.146.219:34026 reqid=3 duration="41.803µs" err="max priority fee per gas higher than max fee per gas"
3|op-geth  | INFO [12-18|14:42:51.951] Stopping work on payload                 id=0x035900c1b4c555a7 reason=delivery     elapsed=259.954ms
3|op-geth  | INFO [12-18|14:42:51.953] Imported new potential chain segment     number=815,934 hash=997b67..923156 blocks=1 txs=1    mgas=0.047  elapsed=1.810ms      mgasps=25.740   snapdiffs=1.82MiB    triedirty=0.00B
3|op-geth  | INFO [12-18|14:42:51.954] Chain head was updated                   number=815,934 hash=997b67..923156 root=8a7ca3..681714 elapsed="432.059µs"
3|op-geth  | INFO [12-18|14:42:52.343] Starting work on payload                 id=0x0370daf586fada21
3|op-geth  | INFO [12-18|14:42:52.344] Updated payload                          id=0x0370daf586fada21 number=815,935 hash=cbd275..85c382 txs=1    withdrawals=0 gas=49406      fees=0               root=eb8aa9..8fd1cb elapsed="657.936µs"

3|op-geth  | WARN [12-18|14:42:52.695] Served eth_sendRawTransaction            conn=172.71.146.10:20222 reqid=3 duration="56.189µs" err="max priority fee per gas higher than max fee per gas"
3|op-geth  | WARN [12-18|14:42:53.463] Served eth_sendRawTransaction            conn=172.71.147.7:63710 reqid=3 duration="45.227µs" err="max priority fee per gas higher than max fee per gas"
3|op-geth  | INFO [12-18|14:42:53.951] Stopping work on payload                 id=0x0370daf586fada21 reason=delivery     elapsed=1.607s
3|op-geth  | INFO [12-18|14:42:53.954] Imported new potential chain segment     number=815,935 hash=cbd275..85c382 blocks=1 txs=1    mgas=0.049  elapsed=2.560ms      mgasps=19.294   snapdiffs=1.83MiB    triedirty=0.00B
3|op-geth  | INFO [12-18|14:42:53.955] Chain head was updated                   number=815,935 hash=cbd275..85c382 root=eb8aa9..8fd1cb elapsed="445.123µs"
3|op-geth  | WARN [12-18|14:42:54.105] Served eth_sendRawTransaction            conn=172.71.147.2:42726 reqid=3 duration="60.661µs" err="max priority fee per gas higher than max fee per gas"
3|op-geth  | INFO [12-18|14:42:54.362] Starting work on payload                 id=0x03b54f2a2284ef53
3|op-geth  | INFO [12-18|14:42:54.363] Updated payload                          id=0x03b54f2a2284ef53 number=815,936 hash=d6ec3c..49090d txs=1    withdrawals=0 gas=46606      fees=0               root=574cd4..0deb55 elapsed="487.075µs"
3|op-geth  | WARN [12-18|14:42:54.759] Served eth_sendRawTransaction            conn=172.71.146.10:20222 reqid=3 duration="42.106µs" err="max priority fee per gas higher than max fee per gas"
3|op-geth  | WARN [12-18|14:42:55.422] Served eth_sendRawTransaction            conn=172.71.146.248:21130 reqid=3 duration="55.521µs" err="max priority fee per gas higher than max fee per gas"

Additional Information

we have restarted the op-node to let it traverse the history and may resolve this issue

Feedback

No response

[askucher]

Analysis.

1. This block is not reoganized and finalized on Base: https://sepolia.basescan.org/block/19294919

"finalized_l2": {
    "hash": "0x6ecbc8f0b770b56a34b8c4f110a4e8a823a551abb56c9a787e0038aa1fbefdeb",
    "number": 728596,
    "parentHash": "0xfa20f1bed22a462f264b983e6858c1e405d41c0aefb6a89feacdcaa3a28549a2",
    "timestamp": 1734358296,
    "l1origin": {
      "hash": "0x51bbfff50561fb48714b47aa9d6ddacc4a4c9e88fd6e225a77e763e485896f6b",
      "number": 19294919
    },
    "sequenceNumber": 0
  },

So, not a reorg problem

2. Restart of op-node, op-proposer did not help
3. Batcher works normally
4. Op proposer does not provide any additional details --log.level="DEBUG" is enabled and --poll-interval="12s", --allow-non-finalized=false so I have assumption that op-proposer is waiting for block finalization to send it on l1 but block is never finalized

[askucher]

Progress

1. When we set op-proposer --allow-non-finalized=true then the issue is solved but the problem is not resolved.
2. Conclusion that issue in default finalizer. And it's really strange that op-proposer does not provide the reason in debug mode
3. We set --allow-non-finalized=false to check can it go further.
4. Hope to get the normal and resultive support from the optimism dev team because it's a security question of the optimism core functionality
   Update: it does not work when --allow-non-finalized=false. Because finalized blocks are stuck

[opfocus]

So the main problem you have is that finalized_l2 stops growing, right?
When deploying L3, you may be able to run your own L2 base sepolia node to improve the stability of its finalized signal.