Replay buffered observational-memory results at production's step in Mastra memory replay

changelog.d/1230.fixed.md

@@ -0,0 +1 @@
+- A Mastra memory replay (`createMemoryReplayAgent()`) no longer observes at a step where production did not. With a slow production observer, the instant replay used to start extra buffering rounds while the recorded round would still have been running. Each extra round split later steps into more stored messages and raised OM's pending token count, so a turn that ended just under `messageTokens` crossed it only in replay and the actor's last prompt differed. Baselines now record how many actor steps had started when each OM output arrived and how long the call took, and replay returns a matching buffered output at that step, or after that duration at the latest. Baselines recorded earlier replay as before.

docs/book/adapters/mastra.md

@@ -206,7 +206,7 @@ The factory creates a fresh native agent for each invocation. A baseline uses yo
 
 Replay never calls `sourceMemory()` and never writes to your source storage. It calls an observer or reflector model only when you opt in with `missingObservationalMemoryResults: "live"`, described below.
 
-Each replay OM call takes the unused recorded output with the same phase (observer or reflector), model method, and input. The input comparison ignores message times, dates, generated ids, and how an attachment is held: a declared URL, its captured reference, its downloaded bytes, or its content inline as base64 text or a data URL. Replayed OM models accept captured references and network URLs, so Mastra never downloads a file for them. Mastra also counts an attachment's tokens from its URL, sometimes by asking the provider, and those counts decide when OM observes. A baseline therefore records the tokens OM counted for each attachment, declared, from thread history, or held inline as bytes by a processor, and replay reuses them instead of counting the captured reference or calling the provider. An attachment a replay counts without a recorded count, such as new inline content, takes Mastra's local estimate, so replay never asks the provider to count tokens. Mastra's number of OM calls depends on timing: a slow production observer merges buffering rounds that an instant replay makes separately, and it covers messages the actor produced while it ran. A buffered call (async observation or reflection) whose input matches no unused output therefore gets no result, because another window's output could describe messages the replay has not produced yet. Its messages stay in the actor's context, as they did in production while the observer ran, and a later buffered call usually matches the recorded window. A blocking call whose input matches no unused output takes the next unused output of its phase, and replay records an `om_input_mismatch` span. Its `calls` attribute lists each such call with its phase, model method, `recorded_ordinal` (the tape position of the output it took), and `replay_call` (its position among the replay's OM calls). A blocking call after its phase's recorded outputs are used up fails replay with `KITARU_REPLAY_DIVERGED:mastra_om_call_order`, because an empty observation would drop the observed messages from the actor's context. So does a blocking call whose phase has no recorded output at all; a buffered call of such a phase gets no result and counts as a surplus call. The failed replay records an `om_unanswered_call` span that names the call: its phase, model method, `replay_call`, and `cause` (`no_recorded_result` when production made no call of that phase and method, or `recorded_results_used_up`). By default, no OM call reaches a provider. To let such a replay finish instead, set `missingObservationalMemoryResults: "live"` on `createMemoryReplayAgent`. A blocking call with no recorded output then calls the observer or reflector model that `resolveModel` returns for the recorded identity, with captured files sent as their recorded bytes, and recorded outputs still answer every other call. Buffered calls never go live. Each live call is recorded as an `llm_call` node named `om_observer_live_call` or `om_reflector_live_call`, and the replay session reports how many ran in `metadata.mastra_om_live_calls`, because part of its memory no longer comes from what production observed. Replay reports the other departures in an `om_call_divergence` span and in the session's `metadata.mastra_om_divergence` counts: `input_mismatches` (blocking calls that took another input's output), `surplus_calls` (buffered calls after their phase's outputs were used up, or of a phase with none), `unused_results`, and `live_calls` (blocking calls the live model answered). A baseline also records failed OM attempts, so a turn whose observer succeeded after Mastra retried it stays eligible, and its replay serves the successful output directly. When a failed blocking observation or an input processor's `abort()` ends the Mastra stream with a tripwire, the session still closes and the lease is released: a baseline becomes `ineligible` and a replay fails. A replay closes its session before its stream ends, so the replay process can exit as soon as it has read the stream. Reusing recorded outputs lets you compare actor instruction/model changes, but does not measure how a fresh observer or reflector would respond to the changed conversation.
+Each replay OM call takes the unused recorded output with the same phase (observer or reflector), model method, and input. The input comparison ignores message times, dates, generated ids, and how an attachment is held: a declared URL, its captured reference, its downloaded bytes, or its content inline as base64 text or a data URL. Replayed OM models accept captured references and network URLs, so Mastra never downloads a file for them. Mastra also counts an attachment's tokens from its URL, sometimes by asking the provider, and those counts decide when OM observes. A baseline therefore records the tokens OM counted for each attachment, declared, from thread history, or held inline as bytes by a processor, and replay reuses them instead of counting the captured reference or calling the provider. An attachment a replay counts without a recorded count, such as new inline content, takes Mastra's local estimate, so replay never asks the provider to count tokens. Mastra's number of OM calls depends on timing: a slow production observer merges buffering rounds that an instant replay makes separately, and it covers messages the actor produced while it ran. A buffered call (async observation or reflection) whose input matches no unused output therefore gets no result, because another window's output could describe messages the replay has not produced yet. Its messages stay in the actor's context, as they did in production while the observer ran, and a later buffered call usually matches the recorded window. A buffered call whose input matches returns its output only once the actor has started as many steps as it had when production's output arrived, or once production's call duration has passed. Mastra starts no new buffering round while one runs, and each round seals the messages it covers, which raises the pending token count that decides when OM observes; an instant output would let replay start rounds production never ran, so a turn that ended just under the observation threshold would observe only in replay. A baseline recorded before this timing was kept returns matching buffered outputs at once. A blocking call whose input matches no unused output takes the next unused output of its phase, and replay records an `om_input_mismatch` span. Its `calls` attribute lists each such call with its phase, model method, `recorded_ordinal` (the tape position of the output it took), and `replay_call` (its position among the replay's OM calls). A blocking call after its phase's recorded outputs are used up fails replay with `KITARU_REPLAY_DIVERGED:mastra_om_call_order`, because an empty observation would drop the observed messages from the actor's context. So does a blocking call whose phase has no recorded output at all; a buffered call of such a phase gets no result and counts as a surplus call. The failed replay records an `om_unanswered_call` span that names the call: its phase, model method, `replay_call`, and `cause` (`no_recorded_result` when production made no call of that phase and method, or `recorded_results_used_up`). By default, no OM call reaches a provider. To let such a replay finish instead, set `missingObservationalMemoryResults: "live"` on `createMemoryReplayAgent`. A blocking call with no recorded output then calls the observer or reflector model that `resolveModel` returns for the recorded identity, with captured files sent as their recorded bytes, and recorded outputs still answer every other call. Buffered calls never go live. Each live call is recorded as an `llm_call` node named `om_observer_live_call` or `om_reflector_live_call`, and the replay session reports how many ran in `metadata.mastra_om_live_calls`, because part of its memory no longer comes from what production observed. Replay reports the other departures in an `om_call_divergence` span and in the session's `metadata.mastra_om_divergence` counts: `input_mismatches` (blocking calls that took another input's output), `surplus_calls` (buffered calls after their phase's outputs were used up, or of a phase with none), `unused_results`, and `live_calls` (blocking calls the live model answered). A baseline also records failed OM attempts, so a turn whose observer succeeded after Mastra retried it stays eligible, and its replay serves the successful output directly. When a failed blocking observation or an input processor's `abort()` ends the Mastra stream with a tripwire, the session still closes and the lease is released: a baseline becomes `ineligible` and a replay fails. A replay closes its session before its stream ends, so the replay process can exit as soon as it has read the stream. Reusing recorded outputs lets you compare actor instruction/model changes, but does not measure how a fresh observer or reflector would respond to the changed conversation.
 
 The following binding uses a process-local store. Supply your existing public memory storage domain and its complete configuration for a persistent application:
 

packages/mastra/README.md

@@ -170,7 +170,7 @@ The factory creates a fresh native agent for each invocation. A baseline uses yo
 
 Replay never calls `sourceMemory()` and never writes to your source storage. It calls an observer or reflector model only when you opt in with `missingObservationalMemoryResults: "live"`, described below.
 
-Each replay OM call takes the unused recorded output with the same phase (observer or reflector), model method, and input. The input comparison ignores message times, dates, generated ids, and how an attachment is held: a declared URL, its captured reference, its downloaded bytes, or its content inline as base64 text or a data URL. Replayed OM models accept captured references and network URLs, so Mastra never downloads a file for them. Mastra also counts an attachment's tokens from its URL, sometimes by asking the provider, and those counts decide when OM observes. A baseline therefore records the tokens OM counted for each attachment, declared, from thread history, or held inline as bytes by a processor, and replay reuses them instead of counting the captured reference or calling the provider. An attachment a replay counts without a recorded count, such as new inline content, takes Mastra's local estimate, so replay never asks the provider to count tokens. Mastra's number of OM calls depends on timing: a slow production observer merges buffering rounds that an instant replay makes separately, and it covers messages the actor produced while it ran. A buffered call (async observation or reflection) whose input matches no unused output therefore gets no result, because another window's output could describe messages the replay has not produced yet. Its messages stay in the actor's context, as they did in production while the observer ran, and a later buffered call usually matches the recorded window. A blocking call whose input matches no unused output takes the next unused output of its phase, and replay records an `om_input_mismatch` span. Its `calls` attribute lists each such call with its phase, model method, `recorded_ordinal` (the tape position of the output it took), and `replay_call` (its position among the replay's OM calls). A blocking call after its phase's recorded outputs are used up fails replay with `KITARU_REPLAY_DIVERGED:mastra_om_call_order`, because an empty observation would drop the observed messages from the actor's context. So does a blocking call whose phase has no recorded output at all; a buffered call of such a phase gets no result and counts as a surplus call. The failed replay records an `om_unanswered_call` span that names the call: its phase, model method, `replay_call`, and `cause` (`no_recorded_result` when production made no call of that phase and method, or `recorded_results_used_up`). By default, no OM call reaches a provider. To let such a replay finish instead, set `missingObservationalMemoryResults: "live"` on `createMemoryReplayAgent`. A blocking call with no recorded output then calls the observer or reflector model that `resolveModel` returns for the recorded identity, with captured files sent as their recorded bytes, and recorded outputs still answer every other call. Buffered calls never go live. Each live call is recorded as an `llm_call` node named `om_observer_live_call` or `om_reflector_live_call`, and the replay session reports how many ran in `metadata.mastra_om_live_calls`, because part of its memory no longer comes from what production observed. Replay reports the other departures in an `om_call_divergence` span and in the session's `metadata.mastra_om_divergence` counts: `input_mismatches` (blocking calls that took another input's output), `surplus_calls` (buffered calls after their phase's outputs were used up, or of a phase with none), `unused_results`, and `live_calls` (blocking calls the live model answered). A baseline also records failed OM attempts, so a turn whose observer succeeded after Mastra retried it stays eligible, and its replay serves the successful output directly. When a failed blocking observation or an input processor's `abort()` ends the Mastra stream with a tripwire, the session still closes and the lease is released: a baseline becomes `ineligible` and a replay fails. A replay closes its session before its stream ends, so the replay process can exit as soon as it has read the stream. Reusing recorded outputs lets you compare actor instruction/model changes, but does not measure how a fresh observer or reflector would respond to the changed conversation.
+Each replay OM call takes the unused recorded output with the same phase (observer or reflector), model method, and input. The input comparison ignores message times, dates, generated ids, and how an attachment is held: a declared URL, its captured reference, its downloaded bytes, or its content inline as base64 text or a data URL. Replayed OM models accept captured references and network URLs, so Mastra never downloads a file for them. Mastra also counts an attachment's tokens from its URL, sometimes by asking the provider, and those counts decide when OM observes. A baseline therefore records the tokens OM counted for each attachment, declared, from thread history, or held inline as bytes by a processor, and replay reuses them instead of counting the captured reference or calling the provider. An attachment a replay counts without a recorded count, such as new inline content, takes Mastra's local estimate, so replay never asks the provider to count tokens. Mastra's number of OM calls depends on timing: a slow production observer merges buffering rounds that an instant replay makes separately, and it covers messages the actor produced while it ran. A buffered call (async observation or reflection) whose input matches no unused output therefore gets no result, because another window's output could describe messages the replay has not produced yet. Its messages stay in the actor's context, as they did in production while the observer ran, and a later buffered call usually matches the recorded window. A buffered call whose input matches returns its output only once the actor has started as many steps as it had when production's output arrived, or once production's call duration has passed. Mastra starts no new buffering round while one runs, and each round seals the messages it covers, which raises the pending token count that decides when OM observes; an instant output would let replay start rounds production never ran, so a turn that ended just under the observation threshold would observe only in replay. A baseline recorded before this timing was kept returns matching buffered outputs at once. A blocking call whose input matches no unused output takes the next unused output of its phase, and replay records an `om_input_mismatch` span. Its `calls` attribute lists each such call with its phase, model method, `recorded_ordinal` (the tape position of the output it took), and `replay_call` (its position among the replay's OM calls). A blocking call after its phase's recorded outputs are used up fails replay with `KITARU_REPLAY_DIVERGED:mastra_om_call_order`, because an empty observation would drop the observed messages from the actor's context. So does a blocking call whose phase has no recorded output at all; a buffered call of such a phase gets no result and counts as a surplus call. The failed replay records an `om_unanswered_call` span that names the call: its phase, model method, `replay_call`, and `cause` (`no_recorded_result` when production made no call of that phase and method, or `recorded_results_used_up`). By default, no OM call reaches a provider. To let such a replay finish instead, set `missingObservationalMemoryResults: "live"` on `createMemoryReplayAgent`. A blocking call with no recorded output then calls the observer or reflector model that `resolveModel` returns for the recorded identity, with captured files sent as their recorded bytes, and recorded outputs still answer every other call. Buffered calls never go live. Each live call is recorded as an `llm_call` node named `om_observer_live_call` or `om_reflector_live_call`, and the replay session reports how many ran in `metadata.mastra_om_live_calls`, because part of its memory no longer comes from what production observed. Replay reports the other departures in an `om_call_divergence` span and in the session's `metadata.mastra_om_divergence` counts: `input_mismatches` (blocking calls that took another input's output), `surplus_calls` (buffered calls after their phase's outputs were used up, or of a phase with none), `unused_results`, and `live_calls` (blocking calls the live model answered). A baseline also records failed OM attempts, so a turn whose observer succeeded after Mastra retried it stays eligible, and its replay serves the successful output directly. When a failed blocking observation or an input processor's `abort()` ends the Mastra stream with a tripwire, the session still closes and the lease is released: a baseline becomes `ineligible` and a replay fails. A replay closes its session before its stream ends, so the replay process can exit as soon as it has read the stream. Reusing recorded outputs lets you compare actor instruction/model changes, but does not measure how a fresh observer or reflector would respond to the changed conversation.
 
 The following binding uses a process-local store. Supply your existing public memory storage domain and its complete configuration for a persistent application:
 

packages/mastra/src/attachment-tokens.ts

@@ -52,7 +52,7 @@ function getAttachmentData(part: unknown): unknown {
   return data instanceof URL ? data.href : data;
 }
 
-function isCount(value: unknown): value is number {
+export function isCount(value: unknown): value is number {
   return typeof value === "number" && Number.isFinite(value) && value >= 0;
 }