Batch small worker output uploads into BatchUpdateBlobs

[erneestoc]

Problem

The worker uploads every output file as its own ByteStream.Write stream.
Each stream carries a fixed setup cost (measured ~1.6ms on the read-side
investigation; ~30µs/blob even on zero-RTT loopback), so actions with many
small outputs pay it N times. This is the write-side twin of the merged
read coalescing (#2540).

Measured (M-series, real ByteStream+CAS servers over TCP, byte-verified)

Stream-per-blob (64-wide, today's shape) vs BatchUpdateBlobs batches
(3MiB cap, 256B/entry overhead — same constants as #2540):

blob size	stream-per-blob	batched	ratio
4KiB ×2048	62ms (30µs/blob)	4ms (1µs/blob)	15.5×
32KiB ×512	18ms (35µs/blob)	5ms (9µs/blob)	3.6×
256KiB ×128	14ms (109µs/blob)	10ms (78µs/blob)	1.4×

Zero-RTT loopback floor; real network RTT widens the gap (each avoided
stream saves a full setup round trip).

Mechanism

• StoreDriver::update_many(Vec<(StoreKey, Bytes)>) with a default
  loop-over-update_oneshot implementation — zero behavior change for
  stores that don't override — plus StoreOptimizations::SubscribesToUpdateMany
  so call sites can gate batching work.
• Opt-in GrpcSpec.experimental_write_batching (max_blob_size_bytes
  default 128KiB, max_batch_bytes default 3MiB). When set,
  GrpcStore::update_many packs small blobs into BatchUpdateBlobs
  requests (per-entry overhead charge, ambient digest function, digest
  dedup within a call) reusing the existing retried batch_update_blobs.
  Per-entry status isolation: retryable per-entry errors fall back to the
  streaming path; non-retryable errors propagate.
• FastSlowStore::update_many forwards batching to the slow tier when it
  advertises support, registering the same in-flight-slow-write guards as
  update() so concurrent has() visibility is preserved.
• Worker call site: running_actions_manager collects output files
  ≤128KiB into a SmallFileBatcher (32MiB in-flight cap) and publishes
  them with one update_many per action; large files keep the existing
  streaming/whole-file path. Only active when the store chain advertises
  SubscribesToUpdateMany — behavior is byte-identical otherwise.

Trade-offs / notes

• Batched small outputs are written to the fast tier via update_oneshot
  rather than update_with_whole_file's rename-without-fsync, trading a
  local fast-tier write for N-1 avoided slow-tier streams. The flag is
  opt-in and aimed at remote (gRPC) slow tiers where stream setup dominates.
• Composes with the ByteStream write-dedup PR Add opt-in join-the-flight ByteStream write dedup #2591: the two features
  partition upload traffic (this PR reroutes small worker outputs off
  ByteStream; Add opt-in join-the-flight ByteStream write dedup #2591 dedups concurrent ByteStream streams). Batches and
  streams for the same digest are reconciled by existing store-level dedup;
  no double-single-flight is built.

Tests

• New grpc_write_batching_test.rs: 8 tests incl. batch packing, per-entry
  retryable fallback, non-retryable propagation, threshold routing, digest
  dedup, FastSlow chain E2E against a fake gRPC CAS.
• New worker test small_outputs_batched_when_store_subscribes: real
  action execution, asserts exactly one update_many with deduped items.
• Suites: cargo store/worker/util all green; cargo check --tests --workspace clean; bazel store+config+worker+util 62/62 with clippy
  pedantic + nightly rustfmt aspects.

Review-round fixes (pre-PR, 8-area adversarial review)

• CacheMetricsStore now forwards update_many — it forwards
  optimized_for, and the store factory wraps every backend in it, so
  without this the default production composition advertised batching but
  dispatched the per-blob loop (regression test:
  cache_metrics_wrapper_preserves_batching).
• Construction-time error_if!: max_batch_bytes must cover
  max_blob_size_bytes + 256B overhead, otherwise an over-budget
  single-entry request bypasses the per-entry streaming fallback
  (regression test: rejects_blob_threshold_larger_than_batch_budget).
• All other reviewed areas verified clean: per-entry status matched by
  digest; FastSlow guard lifecycle cancel-safe; deferred-upload window
  closes before the ActionResult is stored (AC write happens after
  upload_results); ambient digest-function propagation matches every
  existing GrpcStore RPC; packing always makes progress; drop-mid-flush is
  RAII-clean.

Second review round (8-angle adversarial review, pre-PR)

Fixed:

• Whole-RPC batch failures now fall back to per-item streaming (an
  intermediary's gRPC message-size limit could reject a 3MiB batch that
  individual streams would survive; previously this hard-failed the action).
  Regression test: whole_rpc_failure_falls_back_to_streaming.
• Batch chunks dispatch concurrently (buffer_unordered(4)) and the
  oversized/fallback path uploads with width-32 concurrency, restoring the
  old per-file overlap (a lowered max_blob_size_bytes no longer routes
  mid-size outputs into a serial loop).
• The fast-tier arm of FastSlowStore::update_many writes with width-16
  concurrency instead of the serial default loop; a redundant full-Vec
  clone removed.
• The 256-byte per-entry overhead is one shared constant across read
  batcher, write packer, and constructor validation; the worker's 128KiB
  small-file threshold now references the exported config default so the
  two cannot drift.

Known limitations (documented in code):

• In-flight-write guards are held for the whole batch, so a concurrent
  has() on a batch member blocks until the batch settles (coarser than
  update(); window shrunk by concurrent dispatch; per-chunk completion is
  a possible follow-up requiring an API change).
• Up to 32MiB of small-output bytes buffered per action (flushed in-place
  beyond that); no cross-action global budget yet.
• Batch metrics in CacheMetricsStore are all-or-nothing on partial
  failures (accepted approximation).
• Wrappers that forward optimized_for(SubscribesToUpdateMany) must also
  forward update_many — contract now documented on the optimization enum;
  wrappers using the default false are safe (feature quietly inactive
  behind them).

🤖 Generated with Claude Code

https://claude.ai/code/session_01UXVtatcR9YMecBiu9RjwpC

---

This change is 

Review update (2026-07-22)

The batching path was hardened after adversarial review:

• Batch RPC deadlines now use the configured rpc_timeout_s; timeout failures retry and then fall back to streaming.
• Missing, malformed, invalid, or omitted per-entry statuses are treated as malformed and fall back per item; non-retryable item errors still propagate.
• Worker small-output reads are capped at expected size plus one byte, preventing an unexpectedly growing output from being read without bound.
• Added regression coverage for malformed/omitted statuses, timeout fallback, and bounded worker reads.

Focused Cargo and Bazel store/worker suites, formatting checks, and git diff --check pass. The feature remains opt-in and is still intended primarily for many-small-output actions.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
nativelink	Ready	Preview, Comment	Jul 22, 2026 10:57pm
nativelink-aidm	Ready	Preview, Comment	Jul 22, 2026 10:57pm