[OP Stack Deployment] op-batcher creates excess L1 transactions when catching up

[seromenho]

Did you check the documentation?

• Yes

Did you check for duplicate questions?

• Yes

Issue Description

I’m deploying a custom OP Stack chain using alt-da, where transaction data is not posted to L1.
Under normal circumstances, the batcher is submitting a transaction approximately every 90 minutes,. However, if the chain is down for a while (e.g., 24h), once it’s restarted the batcher posts a much higher number of transactions (e.g., 87+) in rapid succession until it “catches up” with L2.
This behavior is surprising since during a 24h period I would expect around 16 transactions, assuming 90 minutes per batch. Instead, the catch-up phase appears to ignore the normal pacing and floods L1 with batch transactions.
My current suspicion is that this is related to missing backpressure or throttling behavior. However, I am not certain how --throttle-threshold=0 (disables throttling) influences this, especially with my short --poll-interval=1s.
My questions:
Why does the batcher behave differently after downtime?
Is this expected?
Are there recommended defaults for throttling when using alt-da?
Is there a mechanism similar to “window pressure” that triggers this acceleration?
What I’m now considering doing:
As seen in the Batcher Sequencer Throttling documentation, definitely enable throttling with a non-zero --throttle-threshold and

 --throttle-threshold=1_000_000 \
  --throttle-tx-size=300 \
  --throttle-block-size=21_000 \
  --throttle-always-block-size=130_000

in hopes of enforcing a steadier pacing of batch production. But I would like some confirmation that this is the right direction.

What else could be the solution for this issue?

Thank you

Logs

OP Logs during catchup

Jul 15 17:56:28 op-stack-ovh-181190 op-batcher msg="Throttling loop is DISABLED due to 0 throttle-threshold. This should not be disabled in prod."
Jul 15 17:56:30 op-stack-ovh-181190 op-batcher msg="safe head above newest block in state, clearing channel manager state" syncStatus.headL1=5c98f8..92a64e:8770775 syncStatus.currentL1=4275ba..c9162c:8770360 syncStatus.localSafeL2=6763b9..d00da3:2924044 syncStatus.safeL2=6763b9..d00da3:2924044 syncStatus.unsafeL2=849060..665317:2924045 syncActions="SyncActions{blocksToPrune: 0, channelsToPrune: 0, clearState: 1d0c22..6ecb07:8766760, blocksToLoad: [2924045, 2924045]}" newestBlockInState=107727..a8c332:2924039
Jul 15 17:56:30 op-stack-ovh-181190 op-batcher msg="safe head above newest block in state, clearing channel manager state" syncStatus.headL1=5c98f8..92a64e:8770775 syncStatus.currentL1=c38f4d..d28216:8770361 syncStatus.localSafeL2=982aba..27aab0:2924048 syncStatus.safeL2=982aba..27aab0:2924048 syncStatus.unsafeL2=982aba..27aab0:2924048 syncActions="SyncActions{blocksToPrune: 0, channelsToPrune: 0, clearState: 5e30df..a64724:8766761, blocksToLoad: nil}" newestBlockInState=849060..665317:2924045
Jul 15 17:56:32 op-stack-ovh-181190 op-batcher msg="safe head above newest block in state, clearing channel manager state" syncStatus.headL1=5c98f8..92a64e:8770775 syncStatus.currentL1=422de6..c7ba51:8770363 syncStatus.localSafeL2=1de9a3..7ac49e:2924061 syncStatus.safeL2=1de9a3..7ac49e:2924061 syncStatus.unsafeL2=1de9a3..7ac49e:2924061 syncActions="SyncActions{blocksToPrune: 0, channelsToPrune: 0, clearState: 61d652..f0dbfc:8766763, blocksToLoad: nil}" newestBlockInState=f0d54a..daaaa6:2924052
Jul 15 17:56:34 op-stack-ovh-181190 op-batcher msg="safe head above newest block in state, clearing channel manager state" syncStatus.headL1=5c98f8..92a64e:8770775 syncStatus.currentL1=f549a9..7065e2:8770365 syncStatus.localSafeL2=7ee7ad..aa7202:2924075 syncStatus.safeL2=7ee7ad..aa7202:2924075 syncStatus.unsafeL2=7ee7ad..aa7202:2924075 syncActions="SyncActions{blocksToPrune: 0, channelsToPrune: 0, clearState: c7622f..e54322:8766765, blocksToLoad: nil}" newestBlockInState=bf7f80..a5af15:2924069
Jul 15 17:56:36 op-stack-ovh-181190 op-batcher msg="safe head above newest block in state, clearing channel manager state" syncStatus.headL1=5c98f8..92a64e:8770775 syncStatus.currentL1=7ead17..63ca0d:8770367 syncStatus.localSafeL2=986d7a..0807e0:2924087 syncStatus.safeL2=986d7a..0807e0:2924087 syncStatus.unsafeL2=986d7a..0807e0:2924087 syncActions="SyncActions{blocksToPrune: 0, channelsToPrune: 0, clearState: 6ef090..6bcf3f:8766767, blocksToLoad: nil}" newestBlockInState=6ebc34..20fc38:2924081
Jul 15 17:56:41 op-stack-ovh-181190 op-batcher msg="safe head above newest block in state, clearing channel manager state" syncStatus.headL1=507a4f..b6c445:8770776 syncStatus.currentL1=ec9bcb..8f0b0e:8770372 syncStatus.localSafeL2=2a9fb5..89b115:2924116 syncStatus.safeL2=2a9fb5..89b115:2924116 syncStatus.unsafeL2=3e00ad..8e344c:2924117 syncActions="SyncActions{blocksToPrune: 0, channelsToPrune: 0, clearState: 47da40..ea0767:8766772, blocksToLoad: [2924117, 2924117]}" newestBlockInState=61b804..2836a5:2924111
Jul 15 17:56:45 op-stack-ovh-181190 op-batcher msg="Found L2 reorg" block_number=2924135
Jul 15 17:56:45 op-stack-ovh-181190 op-batcher msg="error loading blocks, clearing state and waiting for node sync" err="adding L2 block to state: block does not extend existing chain"
Jul 15 17:56:51 op-stack-ovh-181190 op-batcher msg="transaction from unknown channel marked as confirmed" id=69827bd0ae41764db996e6d99a3158ee:0
Jul 15 18:03:12 op-stack-ovh-181190 op-batcher msg="safe head above newest block in state, clearing channel manager state" syncStatus.headL1=e3975c..017b36:8770808 syncStatus.currentL1=023dcd..7e9146:8770781 syncStatus.localSafeL2=6ac469..8a5bda:2926588 syncStatus.safeL2=6ac469..8a5bda:2926588 syncStatus.unsafeL2=2fbdde..640f96:2926589 syncActions="SyncActions{blocksToPrune: 0, channelsToPrune: 0, clearState: fec885..c89b2f:8767181, blocksToLoad: [2926589, 2926589]}" newestBlockInState=fd72c2..378f9c:2926577
Jul 15 18:03:12 op-stack-ovh-181190 op-batcher msg="safe head above newest block in state, clearing channel manager state" syncStatus.headL1=e3975c..017b36:8770808 syncStatus.currentL1=fcd126..66db6c:8770782 syncStatus.localSafeL2=2a9068..8f9a04:2926594 syncStatus.safeL2=2a9068..8f9a04:2926594 syncStatus.unsafeL2=b6a6d9..f0bd01:2926595 syncActions="SyncActions{blocksToPrune: 0, channelsToPrune: 0, clearState: cbf32a..39222d:8767182, blocksToLoad: [2926595, 2926595]}" newestBlockInState=2fbdde..640f96:2926589
Jul 15 18:03:13 op-stack-ovh-181190 op-batcher msg="safe head above newest block in state, clearing channel manager state" syncStatus.headL1=145e2a..319b26:8770809 syncStatus.currentL1=77ceef..476f32:8770783 syncStatus.localSafeL2=33ad0a..6e71d0:2926601 syncStatus.safeL2=33ad0a..6e71d0:2926601 syncStatus.unsafeL2=33ad0a..6e71d0:2926601 syncActions="SyncActions{blocksToPrune: 0, channelsToPrune: 0, clearState: 0fb81e..fdfb49:8767183, blocksToLoad: nil}" newestBlockInState=b6a6d9..f0bd01:2926595
Jul 15 18:03:20 op-stack-ovh-181190 op-batcher msg="Found L2 reorg" block_number=2926655
Jul 15 18:03:20 op-stack-ovh-181190 op-batcher msg="error loading blocks, clearing state and waiting for node sync" err="adding L2 block to state: block does not extend existing chain"
Jul 15 18:03:22 op-stack-ovh-181190 op-batcher msg="transaction from unknown channel marked as confirmed" id=955d97d122c83d33d8139d6194cbac92:0
Jul 15 18:03:35 op-stack-ovh-181190 op-batcher msg="transaction from unknown channel marked as confirmed" id=d2400ecee91bc42b8abb7cb4ca50d5e9:0

Additional Information

Batcher version: op-batcher 1.14.0
Other component versions:
op-geth: v1.101511.0
op-node: 1.13.4
Batcher command:

op-batcher \
  --throttle-threshold=0 \
  --l2-eth-rpc=http://localhost:8545/ \
  --max-channel-duration=1800 \
  --rollup-rpc=http://localhost:9545/ \
  --poll-interval=1s \
  --sub-safety-margin=6 \
  --num-confirmations=1 \
  --safe-abort-nonce-too-low-count=3 \
  --resubmission-timeout=30s \
  --rpc.addr=0.0.0.0 \
  --rpc.port=8548 \
  --rpc.enable-admin \
  --l1-eth-rpc=$L1_RPC_URL \
  --private-key=$GS_BATCHER_PRIVATE_KEY \
  --wait-node-sync=true \
  --altda.enabled=true \
  --altda.da-service=true \
  --altda.da-server=http://localhost:9999/ \
  --log.level=WARN'

Feedback

No response

[imtipi]

disables throttling cause tx submission unlimited

and if you get downtime,try this

Sequencer Window Expiry
If the sequencer window expires (for example, due to extended batcher downtime), some manual intervention is required to help the chain recover. By restarting the sequencer with SEQUENCER_RECOVER=true, or by calling the optimism_setRecoverMode admin API method with the boolean parameter set to true, the sequencer's behavior is changed. While recover mode is enabled, the tx pool is disabled and the l1 origin is progressed steadily: this means blocks are sequenced which are identical to those produced by verifiers under autoderivation. To enable the quickest recovery, the batcher should be configured for singular (not span) batches. After some time, the l1 origin of the l2 safe head will once again catch up close to the l1 head. Then, the recover mode should be disabled and the chain is back in a normal state.

[opfocus]

If your problem has been solved, please remember to close this discussion. If not, please add a comment.