Skip to content

feat: offload large payloads via external storage - #1699

Closed
jmaeagle99 wants to merge 3 commits into
mainfrom
extstore-integration
Closed

jmaeagle99 wants to merge 3 commits into
mainfrom
extstore-integration

Conversation

@jmaeagle99

Copy link
Copy Markdown
Contributor

What was changed

External storage now works end to end in the Rust SDK, behind the experimental feature. Configure a driver with DataConverter::with_external_storage, and any payload at or above the size threshold is handed to the driver and replaced in history by a small reference. References are swapped back for their payloads before anything reads them.

  • Workers offload the payloads in workflow and activity completions after the codec runs, and retrieve them from activations and activity tasks before the codec runs.
  • Clients offload what they send (workflow, signal, query and update arguments, schedules, async activity completions and heartbeats) and retrieve what they read (results, query and update results, history, memos, schedules and standalone activities).
  • With no storage configured, a reference fails with TMPRL1105 instead of reaching application code as though it were the application's own data.
  • Errors. Drivers return StorageDriverError, which any error converts into with ?. The SDK reports ExternalStorageError, which every client error type now carries as its own variant. Configuration problems are ExternalStorageConfigError. To publish the runtime error beside the driver contracts, temporalio_common::data_converters is now a module of its own that re-exports everything it did before.
  • Docs. The user-facing API is documented rather than hidden, still behind experimental, with its current limitations listed on ExternalStorage.

Why?

Enable evaluation of external storage feature in the Rust SDK. Configuration lands together with the worker and client paths.

Out of scope

These are deliberately left for follow-up changes and are listed as limitations in the docs:

  • Heartbeat details recorded from inside an activity are not offloaded yet.
  • Drivers receive no target (workflow, activity or Nexus operation); it is always None.
  • The concurrency limits are not enforced, so their settings stay hidden for now.
  • References are not passed through the payload codec, and no marker is attached to them.
  • No metrics, task failure causes, or worker heartbeat reporting yet.

Checklist

  1. How was this tested:

Integration tests with an in-memory driver: a workflow passes a large value through an activity and returns it, and the raw history is checked for references in place of the workflow input, the activity input and the result; an update sends and returns a large value through the client and worker; and a driver that fails to store surfaces as the client's ExternalStorage error. Unit tests cover the runner's error reporting and its behavior with no storage configured. The existing data converter, update, query, schedule, async activity and standalone activity integration suites pass, along with cargo lint, cargo test-lint and cargo +nightly fmt.

  1. Any docs updates needed? Rustdoc only. A changelog fragment is included.

@jmaeagle99
jmaeagle99 force-pushed the extstore-integration branch from 97dd5a2 to ce8b9cd Compare October 26, 2026 20:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant