feat(consensus): BAL apply metrics, shadow root comparison + journal-bypassing parallel bulk apply (opt-in)

[LukaszRozmej]

Changes

BAL-driven parallel state root workstream: observability, a zero-risk shadow mode, and the journal-bypassing bulk applier itself (opt-in). No behavior change on the canonical path with default config.

• Timing metrics for the parallel-path BAL apply pipeline, following the existing StateRootTimeSink pattern (CacheLinePaddedLong, gated by ExecutionMetricsFlag):

  • BalWarmupWaitTime — time iteration 0 spends joining the HintBal prefetch
  • BalApplyTime / BalStateRootTime — the journal walk vs the Commit + RecalculateStateRoot tail of ApplyStateChanges
  • BalRootReadyLagTime — wall-clock from the last tx-execution worker finishing to root ready, instrumented entirely inside ParallelBlockValidationTransactionsExecutor via contention-free per-tx completion timestamps (no shared counter on the parallel hot path). This is the number that says whether root work or execution binds newPayload latency on BAL blocks.

• BalPostState (Nethermind.Core): pure static helper computing an account's post-block state from parent account + ReadOnlyAccountChanges — last-change-wins per field with parent fallback, EIP-158 touched-totally-empty ⇒ absent. The unit-testable core the bulk applier below builds on.

• Shadow-mode root comparison behind Blocks.ParallelBalStateRootShadow (default false, hidden from docs): when enabled on the parallel BAL path, an independent read-only env at the parent state applies the BAL post-values (accounts via BalPostState, last-value slot writes, code inserts), computes the state root, and compares with the canonical result. The comparison is queued off the block-processing thread (and only after BAL validation passes), so shadow-enabled canaries don't pollute the timing baselines above. BalShadowRootComparisons counts completed comparisons — the soak's positive signal; mismatch ⇒ BalShadowRootMismatches, swallowed failures ⇒ BalShadowRootFailures; the error logs are sampled after the first occurrences so peer-fed blocks can't flood them. The canonical result is never affected.

• Journal-bypassing parallel BAL bulk apply behind Blocks.ParallelBalBulkApply (default true on this branch so benchmarks and every parallel-BAL test exercise it; hidden from docs) — the optimization the metrics above measure. Iteration 0 currently replays the BAL through the journaled world-state operations (CreateAccountIfNotExists/AddToBalance/SetNonce/InsertCode/Set + Commit), paying per-operation journal and bookkeeping costs before the root can be computed. With the flag on, WorldState.BulkApplyBal (new IBalBulkWorldState seam) instead reads each parent account through the scope (warm from the BAL read warmup), derives the post-block account via BalPostState, and writes straight into the backend scope's StartWriteBatch — the same bulk seam WorldState.Commit funnels into, minus the journal — with per-account storage batches filled and disposed in parallel (storage-root computation happens inside each dispose; the outer batch reconciles the roots into the account leaves). Correctness notes:

  • Block-level change tracking is preserved: bulk-applied accounts (and the batch's storage-root fixups) are recorded via StateProvider.TrackBulkAppliedState with Before == After, so GetAccountChanges (TxPool cache invalidation) matches the journaled path and a later FlushToTree skips them instead of clobbering reconciled storage roots.
  • Absent post-block accounts (EIP-158 sweep) go through Set(null); their row's slot writes are skipped, mirroring the journaled commit's sweep. Zero slot values become empty bytes, i.e. slot deletes, matching StorageTree semantics.
  • Dispatch is BlockAccessListManager.ApplyBlockStateChanges (interface default = journaled replay); --Blocks.ParallelBalBulkApply=false restores byte-for-byte today's path and is the benchmark A/B baseline.

Purpose: quantify where block latency goes on BAL blocks (warmup wait / apply / root tail), de-risk the bulk applier by soaking the BAL-derived root reconstruction on devnets with zero consensus exposure, and let benchmarks A/B the applier itself — bulk is the branch default, --Blocks.ParallelBalBulkApply=false is the journaled baseline (pair with --Blocks.ParallelBalStateRootShadow=true so the shadow's journaled replay cross-checks the bulk root block by block).

Types of changes

What types of changes does your code introduce?

• Bugfix (a non-breaking change that fixes an issue)
• New feature (a non-breaking change that adds functionality)
• Breaking change (a change that causes existing functionality not to work as expected)
• Optimization
• Refactoring
• Documentation update
• Build-related changes
• Other: Description

Testing

Requires testing

• Yes
• No

If yes, did you write tests?

• Yes
• No

Notes on testing

• BalPostStateTests (11): creation, parent fallbacks, last-change-wins, EIP-158 on/off, code change with unchanged nonce, null-code normalization, reads-only rows.
• BalBulkApplyTests (20): 9 parity scenarios × {flat, trie} scope providers assert the bulk-applied root equals the journaled root and the account-change sets match (created account, balance/nonce-only, storage overwrite/zero-out/create, code deploy, EIP-158 sweep with and without storage, storage-only change, multi-account mix, reads-only row), plus manager dispatch tests.
• BalShadowStateRootTests (9): shadow/canonical root equivalence over real tries (shared TrieStore mirroring production) for account creation, balance-only, storage overwrite/zero-out/create, code deploy, EIP-158 sweep; mismatch reported without throwing; flag-off fully inert (env never created); throwing env factory and corrupt BAL never crash. A capturing logger plus the BalShadowRootComparisons/BalShadowRootFailures deltas assert the shadow actually ran, guarding against vacuous passes.
• Regression: BlockAccessList* + BlockProcessorTests 79/79, Eip7928Tests 204/204, StandardConfigTests + metrics-description tests, Nethermind.Consensus.Test ProcessingStats/SlowBlock 17/17 — all green.
• Next step once merged: enable the shadow flag on glamsterdam devnet canaries; target is BalShadowRootComparisons > 0 with 0 mismatches and 0 failures over ≥1 week — with ParallelBalBulkApply also enabled, the shadow's journaled replay cross-checks the bulk applier block by block — before the bulk applier's default is flipped on the canonical path.

Documentation

Requires documentation update

• Yes
• No

Requires explanation in Release Notes

• Yes
• No

Remarks

Part of the BAL benchmark performance workstream (#12681, #12682). The bulk applier ships here behind Blocks.ParallelBalBulkApply (default on for this branch's benchmark/soak phase — revisit the default before merge if a longer soak is wanted); an optional persist-tail overlap PR will follow, gated on the benchmark, metrics, and shadow-soak results from this PR.

🤖 Generated with Claude Code

[claude]

Claude finished @LukaszRozmej's task in 9m 52s —— View job

---

Code review — PR #12683

• Gather context (diff, changed files, repo rules)
• Review BalPostState correctness
• Review shadow root path (BlockAccessListManager.StateRoot.cs)
• Review metrics + executor instrumentation
• Review config & tests
• Post review

---

Verdict

The design is sound and the "no behaviour change on the default path" claim holds up under reading: the only edits to canonical code are MetricsTimer wrappers around unchanged logic, a moved-but-identical CreateParentStateHeader, one extra Dispose, and the new metric/config surface. I traced the shadow applier against the canonical ApplyStateChanges semantics (EIP-158 sweep vs. BalPostState returning null + DeleteAccount, last-value slot writes, WithoutLeadingZeros zero-out ⇒ slot delete, storage skipped for post-absent accounts) and found no root-divergence bug — post-Cancun the "account with storage but empty account fields" case that would break the continue at StateRoot.cs:82 isn't reachable. Production uses _readOnlyTrieStore (WorldStateManager.CreateResettableWorldState), so the shadow's Commit/RecalculateStateRoot cannot leak nodes into the canonical store. No Critical or High findings.

Three Medium items, all about the instrumentation rather than consensus correctness — and two of them undercut the PR's own stated goals, which is why I'd like them addressed or explicitly waived before merge.

Critical: 0 · High: 0 · Medium: 3 · Low: 6

Medium

1. ParallelBlockValidationTransactionsExecutor.cs:41-45 — the lag tracking is not opt-in, and its counter is unpadded. ExecutionMetricsFlag.IsActive is a compile-time constant (Metrics.cs:38-46, true unless NO_EXEC_METRICS), so contrary to the code comment and the PR description this runs in every default release build: one Interlocked.Decrement per transaction on a long sharing a cache line with _balRootReadyAt, which the latency-critical iteration-0 BAL-apply worker writes concurrently. The repo's own idiom for this is CacheLinePaddedLong; better still, drop the shared counter and take the max of per-tx completion timestamps after the loop.

2. BlockAccessListManager.StateRoot.cs:36-72 — the soak can pass vacuously. BalShadowRootMismatches is the only signal, and every no-op path is silent: !ParallelExecutionEnabled, bal is null, parentStateRoot is null, canonicalRoot is null, readOnlyTxProcessingEnvFactory is null (Debug only), and the catch-all. "0 mismatches over ≥1 week" is therefore indistinguishable from "the shadow never ran once" — and PR2 is gated on that number. Add a BalShadowRootComparisons counter (bumped only after an actual comparison) and a failure counter, so the criterion becomes comparisons > 0 && mismatches == 0 && failures == 0. Both Error logs are also unbounded per block, reachable via newPayload with invalid blocks.

3. BlockAccessListManager.StateChanges.cs:109 — the shadow is synchronous on the critical path and confounds the metrics it ships with. Per block on the processing thread: a full BAL replay, a second Commit + RecalculateStateRoot, and an Account + two Hash256 per changed account (StateRoot.cs:88-90 materialises the AccountStruct it already has). The plan is to enable this on the same devnet canaries that report BalApplyTime / BalStateRootTime / BalRootReadyLagTime — those baselines will be measured on a node carrying an extra state root and a fresh GC load. Move it off-thread, or document that shadow nodes must not be the timing baseline. It also currently runs before ValidateBlockAccessList, so the work is spent on blocks about to be rejected.

Low

4. BalPostState.cs:42 — CodeChange.CodeHash is 0x000…0 for null code, not Keccak.OfAnEmptyString (CodeChange.cs:18), so it never trips the EIP-158 check and diverges from InsertCode. Unreachable from a decoded BAL today, but this helper is billed as PR2's canonical core. Same comment carries the AccountStruct/ValueHash256 signature suggestion for PR2.

5. BlockAccessListManager.cs:30-35 — the partial-file index in the class <summary> is stale. It's an explicit map and no longer lists BlockAccessListManager.StateRoot.cs. Suggested line:

   ///   * BlockAccessListManager.StateRoot.cs             — shadow BAL state root, apply timing sinks
   

   Also, BalWarmupWaitTimeSink and BalApplyTimeSink live in a file named StateRoot.cs but measure the HintBal join and the journal walk — consider a name that covers the apply pipeline, or move those two sinks next to what they instrument.

6. BlockAccessListManager.StateChanges.cs:37-79 — diff churn. Wrapping the loop in using (MetricsTimer<BalApplyTimeSink> _ = new()) re-indents ~30 unchanged lines, so a reviewer has to diff-read the whole BAL replay to confirm nothing moved. Extracting the loop into a private static void ApplyBalToState(...) would make the timing change a 3-line diff — AGENTS.md calls out reviewer fatigue as the second-order concern after correctness.

7. BalRootReadyLagTime description says "validation worker", but the drain point is the last tx execution worker (IncrementalValidationWorkItem completes separately). Also, since the metric contributes 0 when the root wins the race, a dashboard cannot distinguish "no lag" from "not sampled" — a companion sample count would fix both readings.

8. Test fidelity: SharedStoreEnvFactory uses the writable TrieStore, while production goes through _readOnlyTrieStore via CreateResettableWorldState. The comment says it mirrors production, but the read-only-store commit/root path — the one that actually runs on a node — is never exercised. Wrapping the store read-only in the factory would close that.

9. Shadow_does_not_throw_for_corrupt_bal is close to vacuous. With applyCanonical: false and canonicalRootOverride: TestItem.KeccakF, MismatchDelta == 1 holds for any shadow output, so the assertion message ("must surface as a mismatch report") overclaims what is verified. The Assert.DoesNotThrow is the real content — which the test name already says.

10. No test covers ReportBalRootReadyLag. The lag path has a small state machine (txCount == 0 stamping the drain point before the loop; a worker that throws never decrementing, so nothing is reported) and no coverage.

Notes

• dotnet build / dotnet test are not permitted in this environment, so I did not compile or run the suites — the review is static. I did verify the things most likely to break a build: InternalsVisibleTo("Nethermind.Blockchain.Test") exists on Nethermind.Consensus (needed for the internal RunShadowStateRootComparison), IReadOnlyTxProcessorSource : IDisposable (needed for DisposeAndNull), ConfigItemAttribute.HiddenFromDocs exists, Account implements value equality (the BalPostStateTests assertions depend on it), and the AccountChangesBuilder overloads the tests use are all present.
• BalShadowStateRootTests has no [Parallelizable] and nothing else in the tree touches BalShadowRootMismatches, so the static-counter delta reads are safe from cross-fixture interference.
• Config plumbing is correct: DefaultValue = "false" matches the implicit false on BlocksConfig, and HiddenFromDocs is right for a diagnostics flag.

Nothing here blocks the design; items 1-3 are the ones I'd want resolved (or waived with rationale) since two of them affect whether the PR delivers the measurement it exists to produce.
· branch perf/bal-apply-metrics-shadow-root

[claude]

Medium — unpadded contended counter on the parallel hot path, and it is not actually opt-in.

ExecutionMetricsFlag.IsActive is a compile-time constant (Nethermind.Evm/Metrics.cs:38-46 — true unless NO_EXEC_METRICS is defined). So the comment "captured per block only when detailed metrics are on" is not accurate: in every default release build, OnBalTxWorkerFinished() performs an unconditional Interlocked.Decrement on _balRemainingTxWorkers once per transaction, on the parallel newPayload path.

Two consequences:

1. _balRemainingTxWorkers is contended by all tx workers while _balRootReadyAt is written concurrently by the iteration-0 BAL-apply worker — the three longs share a cache line, so the atomic RMW invalidates the line the latency-critical BAL apply thread writes to. Depending on CLR field layout they may also share a line with the _receiptsTracerPool / _gasResultPool / _txExecutionOrder references that every worker reads. The repo already has the idiom for this (CacheLinePaddedLong, used throughout Metrics.cs) — please use it here, or drop the shared counter entirely: stamping a completion timestamp into the existing per-tx slot and taking the max after the loop is contention-free.

2. Since the gate is compile-time, there is no way for an operator to turn this measurement off short of a custom build — worth stating explicitly, because the PR's own framing suggests it is conditional.

Also note the metric's description says "last parallel BAL validation worker finishing", but the drain point is the last tx execution worker (IncrementalValidationWorkItem completion is separate). Worth aligning the wording so dashboards are not misread.

Fix this →

[claude]

Medium — the soak can pass vacuously; there is no positive signal that the shadow ever ran.

BalShadowRootMismatches is the only metric. Every path that makes the shadow a no-op is silent or near-silent:

• !ParallelExecutionEnabled (line 38) — returns without a word. Enable the flag on a node where Blocks.ParallelExecution happens to be off and you get "0 mismatches" forever.
• bal is null / parentStateRoot is null / canonicalRoot is null (line 50) — silent.
• readOnlyTxProcessingEnvFactory is null (line 54) — Debug only, which is off at default log levels.
• the catch (line 68) logs at Error, but that log is the only trace; nothing counts it.

The stated acceptance criterion is "0 mismatches over ≥1 week before PR2 adopts the bulk applier on the canonical path". As written, an unnoticed early-return or a per-block exception produces exactly the same observable as a clean soak — and PR2 would then ship on the strength of a measurement that never happened. Please add a BalShadowRootComparisons counter (incremented only after a root was actually computed and compared) and a BalShadowRootFailures counter in the catch, so the soak conclusion is comparisons > 0 && mismatches == 0 && failures == 0.

Related: the catch (Exception ex) { _logger.Error(...) } and the mismatch _logger.Error(...) are both unbounded — one per block. A peer feeding invalid blocks via newPayload can drive a sustained error-log stream on a shadow-enabled node. Rate-limiting or Warn for the mismatch case would be safer.

Fix this →

[claude]

Medium — the shadow runs synchronously on the block-processing critical path, and it confounds the very metrics this PR adds.

SetBlockAccessList is called from BlockProcessor.ProcessBlock after ComputeStateRoot, i.e. squarely inside newPayload latency. On a shadow-enabled node each block now pays, on the processing thread:

• a full BAL replay onto a fresh read-only world state,
• a second Commit + RecalculateStateRoot over the whole changed set,
• an Account + two Hash256 allocations per changed account (ComputeShadowStateRoot line 88-90 materialises AccountStruct into a class just to call BalPostState.Compute), so thousands of short-lived allocations per mainnet-size block.

The plan is to enable this flag on the same devnet canaries that are supposed to report BalApplyTime / BalStateRootTime / BalRootReadyLagTime. Those numbers will be measured on a node whose block processing has an extra state-root computation and a fresh GC load bolted on — which is the opposite of what PR1 is for. Either run the comparison off the processing thread (the shadow env is independent of stateProvider, so this looks feasible), or document clearly that shadow-enabled nodes must not be the ones producing the timing baseline.

Ordering nit while here: RunShadowStateRootComparison is invoked before ValidateBlockAccessList, so the shadow work is also spent on blocks that are about to be rejected. Moving it after validation costs nothing and removes that.

Fix this →

[claude]

Low — CodeChange.CodeHash is the zero hash for null code, not Keccak.OfAnEmptyString.

CodeChange.CodeHash is code is null ? default : ValueKeccak.Compute(code) (CodeChange.cs:18), and default(ValueHash256) is 0x000…0. So a code change carrying null code yields codeHash == Hash256.Zero here, which (a) never trips the EIP-158 empty check below and (b) diverges from the canonical path, where InsertCode(address, null, spec) would either throw or settle on Keccak.OfAnEmptyString. Zero-length code is fine (ValueKeccak.Compute([]) is Keccak.OfAnEmptyString); it is specifically null that is inconsistent.

A decoded BAL should never carry null code, so this is not currently reachable — but BalPostState is explicitly billed as the "unit-testable core for the future bulk applier (PR2)", where it will run on the canonical path over peer-supplied data. Worth either normalising (changes.CodeChanges[^1].Code is null ? Keccak.OfAnEmptyString : …) or stating the precondition in <remarks>.

Also, forward-looking (Low): the signature takes and returns Account (a class), and ComputeShadowStateRoot has to allocate an Account plus two Hash256 from the AccountStruct it already read just to call it. For PR2's canonical bulk applier that is an allocation per changed account per block. Consider static bool TryCompute(in AccountStruct parent, bool parentExists, ReadOnlyAccountChanges changes, IReleaseSpec spec, out AccountStruct post) now, while there are no callers to migrate — .agents/rules/coding-style.md asks for low-allocation patterns on paths like this.

[github-actions]

EVM Opcode Benchmark Diff

Aggregated runs: base=3, pr=3
Noisy rerun opcodes: PUSH1, PUSH15

No significant regressions or improvements detected.

[LukaszRozmej]

Review feedback addressed in 71b921b:

• Contended lag counter → per-tx completion timestamps (contention-free per-slot stores; drain point = max after the loop joins). Comment fixed to state the tracking is compile-time (NO_EXEC_METRICS), metric description now says tx execution worker.
• Vacuous soak → added BalShadowRootComparisons (positive signal, incremented only after an actual compare) and BalShadowRootFailures (catch path). Soak criterion is now comparisons > 0 && mismatches == 0 && failures == 0 (PR body updated). Mismatch/failure logs sampled after the first occurrences so they can't flood.
• Shadow on the critical path → queued off the block-processing thread, only after ValidateBlockAccessList/ValidateStructuralEquivalence pass; gates + parent root captured synchronously; runs serialized over the shared env. Timing baselines on shadow-enabled canaries stay clean.
• Null-code CodeChange → normalized to Keccak.OfAnEmptyString in BalPostState.Compute + regression test (EIP-158 on/off).

Deferred deliberately: the AccountStruct-based TryCompute signature (PR2's canonical-path concern — will migrate there with the bulk applier) and the per-account allocations in ComputeShadowStateRoot (diagnostics-only, now off-thread).

🤖 Generated with Claude Code

[github-actions]

EXPB Benchmark Comparison

Run: View workflow run

superblocks

Scenario: nethermind-flat-superblocks-perf-bal-apply-metrics-shadow-root-delay0s

Client Processing (SSE)

Metric	PR	Master (cached)	Delta
AVG (ms)	888.55	862.51	+3.02%
MEDIAN (ms)	845.4	825.3	+2.44%
P90 (ms)	1030.4	1018.5	+1.17%
P95 (ms)	1228.3	1192.6	+2.99%
P99 (ms)	2863.1	2837.6	+0.90%
MIN (ms)	560.9	572.7	-2.06%
MAX (ms)	2863.1	2837.6	+0.90%

K6 TTFB

Metric	PR	Master (cached)	Delta
AVG (ms)	1696.95	1546.70	+9.71%
MEDIAN (ms)	1125.38	1072.64	+4.92%
P90 (ms)	3281.41	3214.31	+2.09%
P95 (ms)	3577.31	3505.32	+2.05%
P99 (ms)	4734.76	3664.81	+29.20%
MIN (ms)	701.77	669.70	+4.79%
MAX (ms)	7448.68	4861.33	+53.22%

realblocks

Scenario: nethermind-flat-realblocks-perf-bal-apply-metrics-shadow-root-delay0s

Client Processing (SSE)

Metric	PR	Master (cached)	Delta
AVG (ms)	20.11	20.42	-1.52%
MEDIAN (ms)	17.3	17.4	-0.57%
P90 (ms)	33.0	33.4	-1.20%
P95 (ms)	38.9	40.2	-3.23%
P99 (ms)	84.9	82.2	+3.28%
MIN (ms)	0.2	0.2	+0.00%
MAX (ms)	200.4	187.4	+6.94%

K6 TTFB

Metric	PR	Master (cached)	Delta
AVG (ms)	24.05	24.43	-1.56%
MEDIAN (ms)	20.45	20.89	-2.11%
P90 (ms)	37.11	37.39	-0.75%
P95 (ms)	43.14	44.80	-3.71%
P99 (ms)	90.12	87.07	+3.50%
MIN (ms)	0.68	0.70	-2.86%
MAX (ms)	437.96	436.40	+0.36%

fusaka

Scenario: nethermind-flat-fusaka-perf-bal-apply-metrics-shadow-root-delay0s

Client Processing (SSE)

Metric	PR	Master (cached)	Delta
AVG (ms)	33.30	33.26	+0.12%
MEDIAN (ms)	29.2	29.0	+0.69%
P90 (ms)	53.7	52.9	+1.51%
P95 (ms)	63.1	64.5	-2.17%
P99 (ms)	116.7	115.4	+1.13%
MIN (ms)	4.4	4.7	-6.38%
MAX (ms)	249.6	254.7	-2.00%

K6 TTFB

Metric	PR	Master (cached)	Delta
AVG (ms)	41.73	41.45	+0.68%
MEDIAN (ms)	35.89	35.89	+0.00%
P90 (ms)	64.31	64.25	+0.09%
P95 (ms)	75.26	77.85	-3.33%
P99 (ms)	134.52	127.56	+5.46%
MIN (ms)	6.04	5.96	+1.34%
MAX (ms)	918.64	583.40	+57.46%

[kamilchodola]

Benchmark A/B — state-actor suite (state-actor-glamsterdam-devnet-7-full)

Ran the state-actor EEST benchmarks (our reproduction of the ethpandaops production suite: 557 GB flat synthetic state, 422M accounts) with default config — note Blocks.ParallelBalBulkApply defaults to true, so this measures the journal-bypassing bulk applier on the parallel BAL path.

side	image	stateful (454)	compute (2,040)
base	master-3181e53 (= merge-base, #12681 included)	31048503540	31052363733
PR	pr-12683	31048989298	31055363877

All 2,494 tests pass on both images. Overall medians are neutral (stateful −0.6%, compute −0.1%), but the deltas are strongly structured:

Regressions — account-creation-heavy BAL blocks (consistent, well outside noise)

test family	suite	n	Δ MGas/s
bloatnet/test_create (create2 spam + immediate access)	stateful	18	−21% (every variant down; −32…−35% for 32B contracts, −9…−17% for 1KB)
bloatnet/test_call (call_value_to_empty)	stateful	2	−35%
scenario/contract_creation	compute	8	−33%
scenario/auth_transaction (EIP-7702)	compute	16	−36%
scenario/ether_transfers (to empty)	compute	14	−6%

Cross-run calibration: the create category moves only ±1–2% between independent runs of the same image, so −21% is unambiguous. The common denominator is new/mutated account identity — contract creations, 7702 delegations, touched-empty accounts — pointing at the bulk applier's account-creation path being slower than the journaled replay it bypasses.

Wins

• bloatnet/test_sstore +4.6% (n=60) — existing-slot write-heavy blocks, the bulk batch's intended case
• compute instruction +2.0% average

(eip7928/deploy_then_interact shows +20% here, but an A/A control shows that family swings ±50% run-to-run under parallel execution — not readable at n=1.)

Suggestion

The applier is a net win on storage-write-heavy blocks but pays ~20–36% on creation/authorization-heavy ones. Worth either optimizing the creation path before it ships default-on, or flipping ParallelBalBulkApply default to false until then (numbers above are with the PR's default true).