[OP Stack Deployment] L2 finalized height stuck at 0

[stevenkhong]

Did you check the documentation?

• Yes

Did you check for duplicate questions?

• Yes

Issue Description

After running the chain for a few days, op-node reports some questionable behaviors:

1. No finalized L2 blocks (see sync status output). safe height is somewhat consistently 25.3k blocks away from latest block.
2. Replica nodes only sync up to safe height, and seemingly unaware of the latest blocks produced by the Sequencer node. optimism_syncStatus returns the safe height=unsafe height.

Custom configurations: v1.7.7, Alt-DA (Avail) with Plasma enabled.
L1 Batch Inbox: 0xff00000000000000000000090000000000016481
L2 Explorer: https://sepolia.pivotalscan.xyz/
Setup: 1 sequencer with 3 replicas.
Replica Sync: syncmode=full, gcmode=archive, p2p enabled

Logs

No errors/warnings found in the recent logs.

Additional Information

Sequencer `optimism_syncStatus` output

{
  "current_l1": {
    "hash": "0xf3a8bbdb34d2dbf2d6e8bbe169fb9881eb5de10683d363ded366d2fdb766bcb2",
    "number": 6396664,
    "parentHash": "0x91a9579b208f1442d5fe06a34d05d31e5e6615a93e0f72b8c12fa03ed1195d6d",
    "timestamp": 1722244632
  },
  "current_l1_finalized": {
    "hash": "0x0000000000000000000000000000000000000000000000000000000000000000",
    "number": 0,
    "parentHash": "0x0000000000000000000000000000000000000000000000000000000000000000",
    "timestamp": 0
  },
  "head_l1": {
    "hash": "0x3f6dc1734c7a0a95b5d3bcd0b7e2d8dc782c67e3772f60c53577af754fa13614",
    "number": 6396668,
    "parentHash": "0x2f5ff13c396f2c8dd451336744e7e2775d73e10152177b8dd47b147b771894be",
    "timestamp": 1722244692
  },
  "safe_l1": {
    "hash": "0x103792ee6934426efc092442f446aea72c15706e31ab2b36ebf2668858692a6b",
    "number": 6396617,
    "parentHash": "0xc795ad3bb4932d8ff507b981a6f9363f126707a85cf899618db4f84502a3a722",
    "timestamp": 1722243936
  },
  "finalized_l1": {
    "hash": "0xc5857e8b7290243ef20b4eaf5d607e574062d146ba19b20b34cdcb08d57a46bd",
    "number": 6396590,
    "parentHash": "0x1138b42c85f9f1ea2ae670f2acbb439832c47047bc426b21637d3d12db7ff246",
    "timestamp": 1722243540
  },
  "unsafe_l2": {
    "hash": "0x1587376a9ff7bdfcaa1922ae068045278094bfa9532915e968353c520a66a5ce",
    "number": 207897,
    "parentHash": "0x2326f5e03c8393f57c1efb2ac4e7cd532c168d115da7d12b2c42180435f5de24",
    "timestamp": 1722244710,
    "l1origin": {
      "hash": "0x91a9579b208f1442d5fe06a34d05d31e5e6615a93e0f72b8c12fa03ed1195d6d",
      "number": 6396663
    },
    "sequenceNumber": 2
  },
  "safe_l2": {
    "hash": "0x25f495230ed98542310ab4a35e080c767428036b6f089e4958250f19b0766f21",
    "number": 182645,
    "parentHash": "0x22ec16819acb597fd5ddc88083a1e0944d115167de07d1c74ba3dd0345c94dc2",
    "timestamp": 1722194206,
    "l1origin": {
      "hash": "0xcf2e476353facb43fdbb9ca0ae6ca7603e920bf222fd7b53358a15eea14e9fa8",
      "number": 6393064
    },
    "sequenceNumber": 17
  },
  "finalized_l2": {
    "hash": "0x862270f876fe66ee51e13e03ff1f6e422598ec72a875888b95df1098832bc055",
    "number": 0,
    "parentHash": "0x0000000000000000000000000000000000000000000000000000000000000000",
    "timestamp": 1721828916,
    "l1origin": {
      "hash": "0x563eb7bc438f354b559f80b023ea49dc63a56572897695a7853d2f4001e25f9f",
      "number": 6367882
    },
    "sequenceNumber": 0
  },
  "pending_safe_l2": {
    "hash": "0x25f495230ed98542310ab4a35e080c767428036b6f089e4958250f19b0766f21",
    "number": 182645,
    "parentHash": "0x22ec16819acb597fd5ddc88083a1e0944d115167de07d1c74ba3dd0345c94dc2",
    "timestamp": 1722194206,
    "l1origin": {
      "hash": "0xcf2e476353facb43fdbb9ca0ae6ca7603e920bf222fd7b53358a15eea14e9fa8",
      "number": 6393064
    },
    "sequenceNumber": 17
  }
}

Deploy config

{
  "l1StartingBlockTag": "0x563eb7bc438f354b559f80b023ea49dc63a56572897695a7853d2f4001e25f9f",

  "l1ChainID": 11155111,
  "l2ChainID": 16481,
  "l2BlockTime": 2,
  "l1BlockTime": 12,

  "maxSequencerDrift": 600,
  "sequencerWindowSize": 3600,
  "channelTimeout": 300,

  "p2pSequencerAddress": "0xC1203e49f4F88800FEa5ffbdff38703ba8e32b01",
  "batchInboxAddress": "0xff00000000000000000000090000000000016481",
  "batchSenderAddress": "0xC121671D2Aa2be0bFa81759B2AC639f648888A34",

  "l2OutputOracleSubmissionInterval": 120,
  "l2OutputOracleStartingBlockNumber": 0,
  "l2OutputOracleStartingTimestamp": 1721828916,

  "l2OutputOracleProposer": "0xc122a79192Ae2b84EAcFCA2A7aDbcD36d8259F1A",
  "l2OutputOracleChallenger": "0xc12EEca02bce41107F9c75E92761FD913186c6De",

  "finalizationPeriodSeconds": 12,

  "proxyAdminOwner": "0xc12EEca02bce41107F9c75E92761FD913186c6De",
  "baseFeeVaultRecipient": "0xc12EEca02bce41107F9c75E92761FD913186c6De",
  "l1FeeVaultRecipient": "0xc12EEca02bce41107F9c75E92761FD913186c6De",
  "sequencerFeeVaultRecipient": "0xc12EEca02bce41107F9c75E92761FD913186c6De",
  "finalSystemOwner": "0xc12EEca02bce41107F9c75E92761FD913186c6De",
  "superchainConfigGuardian": "0xc12EEca02bce41107F9c75E92761FD913186c6De",

  "baseFeeVaultMinimumWithdrawalAmount": "0x8ac7230489e80000",
  "l1FeeVaultMinimumWithdrawalAmount": "0x8ac7230489e80000",
  "sequencerFeeVaultMinimumWithdrawalAmount": "0x8ac7230489e80000",
  "baseFeeVaultWithdrawalNetwork": 0,
  "l1FeeVaultWithdrawalNetwork": 0,
  "sequencerFeeVaultWithdrawalNetwork": 0,

  "gasPriceOracleOverhead": 0,
  "gasPriceOracleScalar": 1000000,

  "enableGovernance": true,
  "governanceTokenSymbol": "PLUS",
  "governanceTokenName": "Plus",
  "governanceTokenOwner": "0xc12EEca02bce41107F9c75E92761FD913186c6De",

  "l2GenesisBlockGasLimit": "0x1c9c380",
  "l2GenesisBlockBaseFeePerGas": "0x3b9aca00",
  "l2GenesisRegolithTimeOffset": "0x0",

  "eip1559Denominator": 50,
  "eip1559DenominatorCanyon": 250,
  "eip1559Elasticity": 6,

  "l2GenesisEcotoneTimeOffset": "0x0",
  "l2GenesisDeltaTimeOffset": "0x0",
  "l2GenesisCanyonTimeOffset": "0x0",

  "systemConfigStartBlock": 0,

  "requiredProtocolVersion": "0x0000000000000000000000000000000000000000000000000000000000000000",
  "recommendedProtocolVersion": "0x0000000000000000000000000000000000000000000000000000000000000000",

  "faultGameAbsolutePrestate": "0x03c7ae758795765c6664a5d39bf63841c71ff191e9189522bad8ebff5d4eca98",
  "faultGameMaxDepth": 44,
  "faultGameClockExtension": 0,
  "faultGameMaxClockDuration": 600,
  "faultGameGenesisBlock": 0,
  "faultGameGenesisOutputRoot": "0x0000000000000000000000000000000000000000000000000000000000000000",
  "faultGameSplitDepth": 14,
  "faultGameWithdrawalDelay": 604800,

  "preimageOracleMinProposalSize": 1800000,
  "preimageOracleChallengePeriod": 86400,

  "proofMaturityDelaySeconds": 604800,
  "disputeGameFinalityDelaySeconds": 302400,
  "respectedGameType": 0,

  "useFaultProofs": true,
  "usePlasma": true,

  "daCommitmentType": "GenericCommitment",
  "daChallengeWindow": 160,
  "daResolveWindow": 160,
  "daBondSize": 1000000,
  "daResolverRefundPercentage": 0
}

Rollup config

{
  "genesis": {
    "l1": {
      "hash": "0x563eb7bc438f354b559f80b023ea49dc63a56572897695a7853d2f4001e25f9f",
      "number": 6367882
    },
    "l2": {
      "hash": "0x862270f876fe66ee51e13e03ff1f6e422598ec72a875888b95df1098832bc055",
      "number": 0
    },
    "l2_time": 1721828916,
    "system_config": {
      "batcherAddr": "0xc121671d2aa2be0bfa81759b2ac639f648888a34",
      "overhead": "0x0000000000000000000000000000000000000000000000000000000000000000",
      "scalar": "0x00000000000000000000000000000000000000000000000000000000000f4240",
      "gasLimit": 30000000
    }
  },
  "block_time": 2,
  "max_sequencer_drift": 600,
  "seq_window_size": 3600,
  "channel_timeout": 300,
  "l1_chain_id": 11155111,
  "l2_chain_id": 16481,
  "regolith_time": 0,
  "canyon_time": 0,
  "delta_time": 0,
  "ecotone_time": 0,
  "batch_inbox_address": "0xff00000000000000000000090000000000016481",
  "deposit_contract_address": "0xb52f393f73f202e1b2c8935de700f49a64deef14",
  "l1_system_config_address": "0x83088e66fc3e89b2f446f1c0b3f864e4fb068d8e",
  "protocol_versions_address": "0x0000000000000000000000000000000000000000",
  "plasma_config": {
    "da_challenge_contract_address": "0x0000000000000000000000000000000000000000",
    "da_commitment_type": "GenericCommitment",
    "da_challenge_window": 160,
    "da_resolve_window": 160
  },
  "da_challenge_contract_address": "0x0000000000000000000000000000000000000000"
}

Run Configs

Sequencer Consensus Client Config

op-node \
  --l2=http://localhost:8551 \
  --l2.jwt-secret=/secret/path/to/jwt.txt \
  --sequencer.enabled \
  --sequencer.l1-confs=5 \
  --verifier.l1-confs=4 \
  --rollup.config=/not/so/secret/path/to/rollup.json \
  --rpc.addr=0.0.0.0 \
  --rpc.port=8547 \
  --rpc.enable-admin \
  --p2p.sequencer.key=0xbeefbeefbeefbeef… \
  --p2p.listen.ip=123.123.123.123 \
  --p2p.listen.tcp=9003 \
  --p2p.listen.udp=9003 \
  --p2p.advertise.ip=123.123.123.123 \
  --l1=https://eth-sepolia.g.alchemy.com/v2/MyAlchemyToken \
  --l1.beacon=https://foo-bar.ethereum-sepolia.quiknode.pro/MyQuikNodeToken/ \
  --l1.rpckind=alchemy \
  --l1.trustrpc=true \
  --l1.beacon.ignore=true \
  --plasma.enabled=true \
  --plasma.da-server=http://localhost:8000

Replica Consensus Client Config

op-node \
  --l2=http://localhost:8551 \
  --l2.jwt-secret=/secret/path/to/jwt.txt \
  --sequencer.l1-confs=5 \
  --verifier.l1-confs=4 \
  --rollup.config=/not/so/secret/path/to/rollup.json \
  --rpc.addr=0.0.0.0 \
  --rpc.port=8547 \
  --rpc.enable-admin \
  --p2p.listen.ip=0.0.0.0 \
  --p2p.static=/ip4/123.123.123.123/tcp/9003/p2p/16UPeerIdString… \
  --p2p.listen.tcp=9003 \
  --p2p.listen.udp=9003 \
  --l1=https://eth-sepolia.g.alchemy.com/v2/MyAlchemyToken \
  --l1.rpckind=alchemy \
  --l1.beacon=https://foo-bar.ethereum-sepolia.quiknode.pro/MyQuikNodeToken/ \
  --plasma.enabled=true \
  --plasma.da-server=http://123.123.123.123:8000

Edits:
2024-07-30: added deploy config and rollup json.
2024-08-02: added op-node run configs for sequencer and replica.

Feedback

No response

[sbvegan]

This is interesting that the safe block is being marked and the finalized is not. Can you please share your:

• deployment config: the JSON file you used to deploy your L1 contracts
• rollup config: the file you generated after deployment

[stevenkhong]

Added the deployment config and rollup config to the original message. I used the default config generated with the Getting Started tutorial, modified the governance token, and added DA configs with reference to Avail documentation.

[Chomtana]

Replica nodes only sync up to safe height, and seemingly unaware of the latest blocks produced by the Sequencer node.

This is because you have p2p.disable on sequencer node and yet to setup p2p.static on the replica node?

[stevenkhong]

We do have setup p2p.static on replica nodes. I added to the original message the run configs for both sequencer and replica consensus client nodes, maybe I am doing something wrong there. 🤔
Btw, you have a reference to another case where the safe latest height on replica nodes do not sync up?

[Chomtana]

Can you try pinging the IP and port from the replica to see if it's reachable?

https://docs.optimism.io/builders/node-operators/configuration/base-config#configuring-peer-to-peer-networking

[opfocus]

Have you observed whether the op-node log indicates that L1 may be reorganized or dropped batch?
I think this is a problem in the derivation process, derive logic:
https://specs.optimism.io/protocol/derivation.html#batch-queue