Replay buffered observational-memory results at production's step in Mastra memory replay

.agents/skills/kitaru-adapter-development/SKILL.md

@@ -62,4 +62,6 @@ For Python adapters, run the focused adapter tests, then the plugin workspace fo
 
 For TypeScript adapters, run the affected package's test, typecheck, lint, and build scripts. Run the root `pnpm test`, `pnpm typecheck`, `pnpm lint`, and `pnpm pack:check` when shared primitives, workspace metadata, or packaging changes.
 
+The Mastra adapter is tested against every Mastra release set the memory replay factory accepts. `MEMORY_REPLAY_TESTED_VERSIONS` in `packages/mastra/src/memory-replay-versions.ts` lists those sets (`@mastra/core`, the `@mastra/memory` release built against it, and the `@mastra/pg` release the PostgreSQL tests use), and `assertMemoryReplayVersions` accepts exactly its core and memory pairs. `packages/mastra/` installs the oldest set. Each private `packages/mastra-compat/<major.minor>/` package installs one other set and runs every `packages/mastra` test against it through `packages/mastra-compat/shared/`; the 1.71 package also holds the Mastra compatibility suite. To add a Mastra release, check which memory and pg releases were built against it (their published `devDependencies`), copy a compat package with those pins, run its suite including PostgreSQL, then add the table row and the adapter's `@mastra/memory` peer version. `packages/mastra/test/tested-versions.test.ts` fails when the table, the test packages, the installed versions and the peer range disagree, and the packed-tarball smoke reads the table from the built adapter. Dependabot ignores these pins; see `.github/dependabot.yml`. When you raise the supported `@mastra/core` ceiling for the rest of the adapter, move the compatibility suite to the newest compat package in the same change.
+
 Use live provider or framework tests only when their credentials and external side effects are explicitly in scope.

.claude/skills/kitaru-adapter-development/SKILL.md

@@ -62,4 +62,6 @@ For Python adapters, run the focused adapter tests, then the plugin workspace fo
 
 For TypeScript adapters, run the affected package's test, typecheck, lint, and build scripts. Run the root `pnpm test`, `pnpm typecheck`, `pnpm lint`, and `pnpm pack:check` when shared primitives, workspace metadata, or packaging changes.
 
+The Mastra adapter is tested against every Mastra release set the memory replay factory accepts. `MEMORY_REPLAY_TESTED_VERSIONS` in `packages/mastra/src/memory-replay-versions.ts` lists those sets (`@mastra/core`, the `@mastra/memory` release built against it, and the `@mastra/pg` release the PostgreSQL tests use), and `assertMemoryReplayVersions` accepts exactly its core and memory pairs. `packages/mastra/` installs the oldest set. Each private `packages/mastra-compat/<major.minor>/` package installs one other set and runs every `packages/mastra` test against it through `packages/mastra-compat/shared/`; the 1.71 package also holds the Mastra compatibility suite. To add a Mastra release, check which memory and pg releases were built against it (their published `devDependencies`), copy a compat package with those pins, run its suite including PostgreSQL, then add the table row and the adapter's `@mastra/memory` peer version. `packages/mastra/test/tested-versions.test.ts` fails when the table, the test packages, the installed versions and the peer range disagree, and the packed-tarball smoke reads the table from the built adapter. Dependabot ignores these pins; see `.github/dependabot.yml`. When you raise the supported `@mastra/core` ceiling for the rest of the adapter, move the compatibility suite to the newest compat package in the same change.
+
 Use live provider or framework tests only when their credentials and external side effects are explicitly in scope.

.github/dependabot.yml

@@ -417,7 +417,8 @@ updates:
       # users are told is unsupported. Treat that PR as the prompt for a
       # deliberate range extension like #1092, not something to merge as is.
       # Separate groups keep a Mastra range question from holding up an `ai`
-      # patch.
+      # patch. `@mastra/*` version updates are ignored below, so for Mastra
+      # that range extension always starts by hand.
       vercel-ai:
         patterns:
           - ai
@@ -426,8 +427,10 @@ updates:
           - minor
           - patch
 
-      # `zod` is declared only by the Mastra adapter and its example, where it
-      # defines the tool schemas Mastra validates.
+      # `zod` is declared only by the Mastra adapter, its test packages and its
+      # example, where it defines the tool schemas Mastra validates. The
+      # `@mastra/*` version updates this group would carry are ignored below;
+      # the pattern stays so they never land in the catch-all either.
       mastra:
         patterns:
           - "@mastra/*"
@@ -465,6 +468,29 @@ updates:
       # package builds do not accidentally depend on APIs absent from Node 22.
       - dependency-name: "@types/node"
         update-types: ["version-update:semver-major"]
+      # The Mastra memory replay factory accepts only the release sets in
+      # MEMORY_REPLAY_TESTED_VERSIONS (packages/mastra/src/memory-replay-versions.ts),
+      # and packages/mastra plus each packages/mastra-compat/<x.y> package pins
+      # one set so CI runs the full suite on it. Dependabot writes one version
+      # into every package.json that declares a dependency, and nothing here
+      # can scope an update to one workspace member: `ignore` takes only names
+      # and versions, `dependency-type` classifies a name across all members
+      # (the examples make `@mastra/core` a production dependency), and a
+      # second npm block inside this pnpm workspace fails as misconfigured. So
+      # any `@mastra/*` update would move every pin at once, as the cooldown-
+      # limited #1222 did when it moved `@mastra/core` from 1.71.0 down to 1.67.0.
+      #
+      # Each Mastra package is built against one core release (memory 1.31.0
+      # does not load on core 1.67.0), so all of `@mastra/*` moves by hand:
+      # validate a new release in a new compat package, add its row to the
+      # table, and bump the examples then. Listing every update type ignores
+      # version updates only; security updates still open, and CI fails them
+      # until their release set is tested.
+      - dependency-name: "@mastra/*"
+        update-types:
+          - version-update:semver-major
+          - version-update:semver-minor
+          - version-update:semver-patch
     cooldown:
       default-days: 7
       semver-major-days: 14

.github/workflows/ci.yml

@@ -72,7 +72,12 @@ jobs:
       - run: pnpm run generate:check
       - run: pnpm run lint
       - run: pnpm run typecheck
-      - run: pnpm run test:built
+      - name: Test TypeScript packages, including Mastra memory on PostgreSQL
+        env:
+          POSTGRES_PASSWORD: password
+        run: >-
+          KITARU_TEST_MASTRA_POSTGRES_URL="postgres://postgres:${POSTGRES_PASSWORD}@localhost:5433/postgres"
+          pnpm run test:built
       - name: Report TypeScript coverage
         run: |
           set -euo pipefail

.gitignore

@@ -9,6 +9,7 @@ helm/Chart.lock
 # Root TypeScript workspace
 /node_modules/
 /packages/*/node_modules/
+/packages/mastra-compat/*/node_modules/
 /examples/typescript/mastra_support_triage/node_modules/
 /examples/typescript/mastra_adaptive_conversation/node_modules/
 /examples/typescript/vercel_ai_support_triage/node_modules/

biome.json

@@ -4,6 +4,7 @@
     "includes": [
       "packages/core/**",
       "packages/mastra/**",
+      "packages/mastra-compat/**",
       "packages/vercel-ai/**",
       "examples/typescript/mastra_support_triage/**",
       "examples/typescript/mastra_adaptive_conversation/**",

changelog.d/1178.added.md

@@ -0,0 +1,10 @@
+- Add opt-in Mastra streamed replay with isolated native thread memory. `createMemoryReplayAgent()` from `@zenml-io/kitaru-mastra/memory` records starting working and observational memory, observation and reflection outputs including failed attempts, effective actor requests, controlled files, and memory changes. Replay reuses those recorded outputs, matched to calls by phase and input, without reading or writing production memory or, by default, calling the observer or reflector model; working-memory updates run live against the isolated replay store. Requires a tested Mastra core and memory pair, from core 1.67.0 with memory 1.30.0 through core 1.71.0 with memory 1.32.1 (the adapter docs list each pair and its `@mastra/pg` release), and a Kitaru server newer than 0.27.1; existing history-only recordings must be recorded again to use this path.
+- The Kitaru server tracks whether a recorded Mastra memory session can be replayed in its `mastra_replay_state` metadata (`pending`, `eligible`, or `ineligible` with a `mastra_replay_reason`). Replaying a pending, ineligible, or incomplete baseline returns HTTP 409 with a `mastra_replay_<reason>` code, and CLI and MCP error details include that reason and the baseline's `session_id`. An experiment run records each refused baseline as a failed replay and still runs the others. A pending session closed without a replay decision, such as a plain `failed` update after a recorder stopped mid-turn, is stored as `ineligible` with reason `abandoned` (failed) or `unfinalized` (completed).
+- Mastra memory replay coordinates every writer of a thread and resource through a `MastraExclusiveMemoryAccess` lease; deployments with more than one server writing to the same threads must supply a shared lease backend that expires leases of dead holders. Only overlapping turns become ineligible; a reply that starts while the previous turn's buffered observational-memory work is still running leaves that previous turn eligible when the backend supports the optional `markFinalizing()` lease method, and the reply is ineligible with `earlier_turn_finalizing`, as is each further reply that starts while the previous reply's memory work is still running. A `markUnsafeWrite()` marker lasts until the application calls `resetAfterQuiescence()`.
+- Mastra memory replay `files` can be a function that returns the attachment URLs for each call. URLs in file and image parts of the input and thread history need no declaration: with a `resolveFile`, Kitaru captures each one when a processor resolves it, so `files` is optional for a processor that resolves attachments through `resolveFile`, a turn that sends a new attachment URL stays eligible with `files: []`, and older attachments the processor never reads cost no downloads. A message attachment that Mastra does not send to the model, because the message also holds a file part, does not make a turn ineligible. A turn whose input or history URL reaches the model as a URL instead of bytes is ineligible with `file_url_sent_to_model`. A turn whose processor downloads an input or history URL with its own client instead of `resolveFile` is ineligible with `file_url_undeclared`. Recorded Mastra input, thread history, and evidence never keep URL credentials: an input attachment that Kitaru captured is recorded as a `kitaru-file://` reference, and every other URL keeps its text with its credentials (download tokens, signatures, API keys, userinfo) replaced by `REDACTED`.
+- A Mastra memory replay baseline turn does not wait on a slow or unresponsive Kitaru server. Evidence uploads run in the background in order; the turn waits at most `sessionSetupWaitMs` (default 2 seconds) for its session and `fileCaptureWaitMs` (default 10 seconds) for declared files, then answers natively and records why it is not replayable. A native fallback reuses the file downloads Kitaru already started instead of fetching the URLs again.
+- Mastra memory replay sends the model the same request production sent, even when the recorded input was stored in PostgreSQL `jsonb`, which re-sorts object keys, and even when the replay runs days after the baseline. Replay restores the recorded key order and evaluates observational-memory date labels and `activateAfterIdle` at the recorded turn time.
+- A Mastra memory replay baseline whose native answer succeeded but whose recording cannot be replayed is stored as a `completed` session with its answer, marked `ineligible` with a specific `mastra_replay_reason` such as `replay_input_too_large`, `credential_key_unsupported`, `om_config_unsupported`, `memory_lease_conflict`, `om_tape_incomplete`, or `server_rejected_finalization`. `onRecordingError` receives the same code as `reason`, including for turns that ran natively because recording could not be set up (`stage: "setup"`). Failed provider calls keep their HTTP status and its category, such as `HTTP 429: rate limited`, but not the provider's message, which can echo request content and credentials, and each failed call is recorded as one model node.
+- Mastra memory replay stores each captured file once as a content-addressed Kitaru blob after the native answer, instead of inlining base64 bytes in the replay input, so attachments no longer count against the 16 MiB replay input limit. One file may use the whole 16 MiB per-turn file limit. A process that already stored a file does not upload it again on later turns, replay tasks are granted read access to the blobs their replay input names, and the server checks that each named blob holds the recorded length and hash before marking a turn eligible. Inline attachment content in thread history that matches a captured file is replaced by its reference before Kitaru copies or size-checks the history, so it neither counts toward the 16 MiB replay input limit nor slows the start of the turn. A turn whose files cannot be stored is ineligible with `file_store_failed`, and a replay of a turn whose file blob was deleted is refused with `mastra_replay_file_missing`. Replay inputs recorded with inline file bytes still replay. When a processor writes a captured file's bytes into a message and Mastra saves them in thread history, later replay inputs, memory-change evidence and model-request evidence name the file by its reference instead of repeating the bytes, and replay writes the exact bytes back into the restored history.
+- A Mastra memory replay whose actor takes a step production never took, so that observational memory needs a blocking observation or reflection with no recorded output, fails as diverged by default. The failed replay records an `om_unanswered_call` span naming that call, and closes its session before its stream ends. Set `missingObservationalMemoryResults: "live"` on `createMemoryReplayAgent()` to call the observer or reflector model for those calls only; each live call is recorded as an `llm_call` node and the replay session reports `mastra_om_live_calls` in its metadata.
+- Mastra memory replay records application data as it is by default, whatever its keys are called: tool arguments and results, messages, working and observational memory, the captured request context, configuration, and model-request and memory-change evidence. This keeps a turn with a field such as an opaque `resultToken` handle replayable, but it also means a real `apiKey` or `client_secret` that an application puts in those places is stored in Kitaru. Kitaru always keeps `authorization`, `proxy-authorization`, `cookie`, `set-cookie`, `headers`, and `abortSignal` keys and Mastra's authentication token out of recordings and marks such a turn `credential_key_unsupported`, redacts credentials in URLs, and never stores a provider's error message. To mask other keys, pass `isSecretKey(key)` to `createMemoryReplayAgent()`: `isSecretKey: isCredentialKeyName`, exported from `@zenml-io/kitaru-mastra/memory`, treats names such as `token`, `password`, or `accessToken` as credentials, and `nonSecretKeys` lists keys that always record as they are. The history-only `KitaruAgent` and the other TypeScript adapters still redact credential-looking keys by name.

changelog.d/1178.fixed.md

@@ -0,0 +1,4 @@
+- A failed agent task or replay whose process exits without writing a result now reports its result session's status and error, such as a Mastra replay divergence, instead of a generic "exited successfully without writing a result" message. A task token may now list the sessions its own task produced.
+- The history-only Mastra `KitaruAgent` and the Vercel AI adapter now treat compound credential key names such as `access_token`, `clientSecret`, `refresh_token`, or `x-api-key`, and such names followed by a format word such as `secret_value` or `privateKeyPem`, like `token` or `password`: recorded tool and model evidence redacts their values. A Mastra memory replay agent created with `isSecretKey: isCredentialKeyName` marks a turn that holds one anywhere in its recorded data ineligible with `credential_key_unsupported`. Plural counters such as `max_tokens` and pagination names such as `pageToken` stay unchanged.
+- Recorded provider metadata from the TypeScript adapters now hides values under `headers`, `abortSignal`, and the other transport keys, not only under credential-named keys.
+- The Mastra memory replay agent no longer stores a provider's error message: the `error` part Mastra saves on a failed turn's assistant message is recorded with its message replaced by `[redacted]`, in memory-change evidence and in later turns' replay input.

changelog.d/1229.changed.md

@@ -0,0 +1 @@
+- The Mastra memory replay factory (`createMemoryReplayAgent()`) accepts every Mastra release set its full test suite passes on, instead of only `@mastra/core` 1.67.0 with `@mastra/memory` 1.30.0: core 1.67.0 through 1.71.0, each with the `@mastra/memory` release built against it (1.30.0, 1.31.0, 1.31.0, 1.32.0, 1.32.1). Install core and memory from the same row of the table in the adapter docs, which also names the tested `@mastra/pg` release for each. Any other combination, including a supported core with another row's memory, still fails closed with `version_mismatch`, and the error now lists the supported pairs. The adapter's `@mastra/memory` peer range lists the same releases.

changelog.d/mastra-om-buffer-timing.fixed.md

@@ -0,0 +1 @@
+- A Mastra memory replay (`createMemoryReplayAgent()`) no longer observes at a step where production did not. With a slow production observer, the instant replay used to start extra buffering rounds while the recorded round would still have been running. Each extra round split later steps into more stored messages and raised OM's pending token count, so a turn that ended just under `messageTokens` crossed it only in replay and the actor's last prompt differed. Baselines now record how many actor steps had started when each OM output arrived and how long the call took, and replay returns a matching buffered output at that step, or after that duration at the latest. Baselines recorded earlier replay as before.