Journaling: Azure append-blob WAL replay is paced per committed block server-side — ~5.3 s recovery for 1,000 journaled ops over WAN (curl-reproducible; checkpointing never engages)

[danielmeza]

Description

We evaluated Microsoft.Orleans.Journaling (10.2.1-preview.1.alpha.1) as the persistence layer for a POS system (silo at a retail station, Azure Storage in another region, ~113 ms WAN RTT). The write path is excellent — flat ~1 RTT per op at every state size, and one blob append per WriteStateAsync() even when a commit spans four durable structures (verified with an HttpPipelinePolicy counting requests: 3,021 ops → 3,021 appends, 0 anomalies).

The problem is recovery: replaying the WAL on reactivation is paced per committed append-block on the Azure service side, so recovery time grows with the number of journaled operations rather than WAL bytes, and no client-side change can fix it. A grain with ~1,000 journaled ops (~0.5 MB WAL) takes 5.3 s median to reactivate over our WAN link — while the same bytes as a single-block blob download in <200 ms.

We root-caused it outside Orleans entirely (see repro): the WAL blob left by the run has 1,006 committed blocks, and a raw curl of that blob transfers at ~97 KB/s with 0.48 s TTFB (5.7–5.9 s total) — statistically identical to the Orleans replay time. AzureBlobJournalStorage.ReadAsync does a single DownloadStreamingAsync (our pipeline policy confirms exactly 1 GET per replay), so the Orleans reader is fine; the storage layout (one tiny block per operation) is what makes recovery O(ops) × ~5.7 ms server-side.

Two aggravating observations:

1. Checkpoint compaction never engaged across ~1,006 ops in our runs — replay cost grew linearly to the full op count, so medium-lived grains sit on the un-compacted path with default settings.
2. Azurite does not reproduce the pacing (same replay: 0.8 s), so local testing hides this completely — it only appears against real Azure Storage.

Reproduction Steps

Minimal, no Orleans required:

1. Create an append blob on a real storage account and commit ~1,000 small blocks (~450–500 B each — e.g., loop AppendBlockAsync, or just let a DurableGrain journal 1,000 ops).
2. curl the blob from a client ≥100 ms away. We measure ~97 KB/s / 5.7–5.9 s for 566 KB in 1,006 blocks, on a link that downloads a same-size single-block blob in <200 ms. (Does not reproduce against Azurite.)

Orleans-level repro (what we actually ran):

1. DurableGrain with [FromKeyedServices] IDurableDictionary<Guid, TLine> (~450 B per entry when serialized); silo with AddAzureBlobJournalStorage() and JournaledStateManagerOptions.JournalFormatKey = "orleans-binary" (JSON default is not the variable — replay is block-paced, not byte-paced).
2. Point the BlobServiceClient at a storage account ≥100 ms RTT away.
3. Sequentially: 1,000 × (mutate + await WriteStateAsync()).
4. DeactivateOnIdle() + IManagementGrain.ForceActivationCollection(TimeSpan.Zero); verify the activation is gone.
5. Call any read method and time reactivation.

Expected behavior

Replay is bandwidth-bound: ~RTT + WAL-bytes/bandwidth (≈0.2–0.3 s for 0.5 MB on this link), or checkpointing keeps the replay tail short for grains with a few hundred ops.

Actual behavior

Pooled p50 over ≥3 full repetitions per cell, verified deactivation before each timed reactivation:

Cell (binary format)	rehydrate p50	per-run values	GETs observed
Real Azure, ~102 journaled ops	557 ms	454 / 558 / 1,143	1 per replay
Real Azure, ~1,006 ops (~0.5 MB WAL)	5,309 ms	4,607 / 5,309 / 6,460	1 per replay
Azurite via 100 ms-shaped gateway, same 1,006 ops	810 ms	712 / 810 / 1,162	1 per replay

Linear in op count (~5.3 ms/op over WAN), environment-sensitive (0.8 ms/op local) — a CPU-bound fold would be environment-independent.

Reference points from the same harness, same link, same op count (N=1,000): a hand-rolled event log on Azure Table (JournaledGrain + CustomStorage, one entity per event, replay = 1–2 partition range queries) rehydrates in 882 ms; whole-state grain storage (Table) in 692 ms; blob grain storage in 358 ms.

Write path for contrast (all good): addline p50 114.9–116.4 ms flat at N=100→1,000 real Azure; 120.8 ms on a saturated 3 Mbit shared uplink. First append of a fresh grain: ~355–362 ms (container/WAL creation + manifest — fine, worth documenting).

Regression?

No — first evaluation of the alpha (10.2.1-preview.1.alpha.1 on the 10.2.1-preview.1 runtime). A B-cell recheck on 10.2-preview vs our archived 10.0 numbers drifted only 1.4–3.5 %, so the comparison isn't a runtime artifact.

Known Workarounds

None client-side (the pacing is service-side per block). Design-level directions that would fix it:

• Coalesce appends — batch nearby commits into one AppendBlock; 1,006 ops in ~100 blocks would already cut replay ~10×.
• Checkpoint by op count with an aggressive default (e.g., every 100–500 ops): checkpoints are block blobs written in one shot and read at wire speed; today's defaults never fired across 1,006 ops.
• Periodically compact the WAL itself (rewrite as a single-block blob / fold into the checkpoint).
• A Table-backed journal provider would sidestep block pacing entirely (partition range queries return 1,000 entities/page at wire speed — our CustomStorage-on-Table implementation of the same data shape replays the same ops in 882 ms) and would help money-tier scenarios standardized on Table + ETag; today AzureBlobJournalStorageOptions is the only shipped provider.

Configuration

• Microsoft.Orleans.{Server,Sdk,Persistence.AzureStorage} 10.2.1-preview.1; Microsoft.Orleans.Journaling{,.AzureStorage} 10.2.1-preview.1.alpha.1
• .NET 10.0.8, macOS arm64 (M4 Pro), ServerGC, single silo (UseLocalhostClustering)
• Azure StorageV2 Standard_LRS (southcentralus), client RTT ≈ 113 ms, ~100 Mbit effective bandwidth
• Journal format: orleans-binary

Other information

• Minor DX note: switching to the JSON format (the 10.2 default) throws JsonTypeInfo metadata for type 'System.String' was not provided at grain activation unless every journaled payload type is registered in a source-generated JsonSerializerContext via UseJsonJournalFormat(...). Reasonable for AOT, but the default format failing on first use for non-trivial state is rough onboarding — a friendlier error (or reflection fallback in non-AOT apps) would help.
• The one-append-per-op batching across multiple injected durable structures is the API's best property — please keep it.
• Happy to share the benchmark project (~500 lines, self-contained: counting pipeline policy, verified-deactivation harness) and the raw per-operation CSVs (85k+ samples across 4 persistence designs) on request.

[danielmeza]

We'd like to contribute an Azure Table-backed IJournalStorage to Orleans.Journaling.AzureStorage as an alternative to the append-blob provider, addressing the replay pacing above.

Sketch: one partition per JournalId; each AppendAsync chunk is one entity (binary properties chunked at 64 KiB; oversized appends span rows within a single EGT, ≤100 entities / 4 MiB), and every batch includes an ETag-guarded header row for optimistic concurrency (InconsistentStateException on mismatch). ReplaceAsync writes a new generation prefix and flips the header pointer in one guarded operation; ReadAsync streams partition range-query pages to the consumer (1,000 rows/page, wire-speed — the 1,006-op journal above replays in under 1 s with this layout in our benchmark prototype vs 5.3 s from the append blob); IsCompactionRequested derives from row count.

If this direction works for you we'll open a draft PR (implementation + tests mirroring AzureBlobJournalStorageTests/StorageStreamingTests against Azurite). Happy to follow your lead on packaging, naming, and the checkpoint/manifest translation.