Problem
The speculative VAD/STT pipeline can reopen and supersede revisions of one logical user turn. Internally, RealtimeService can replace the previous speculative user message with the latest cumulative transcript. That replacement is not currently represented equivalently on the Realtime wire.
A generic Realtime client only knows standard event identity:
- transcription events are grouped by
item_id;
conversation.item.input_audio_transcription.completed is terminal for that item;- a different
item_id represents a different user item;
- there is no standard
supersedes, transcript-retraction, or in-place transcript-replacement event.
When speech resumes after an earlier revision has already emitted completed, the server can emit another cumulative completion under a new item ID while replacing the earlier message only in its internal chat history. A normal client then retains multiple independent user turns even though the LLM sees one revised turn.
For example, one utterance with pauses can appear on the wire as:
item_A completed: "The quick brown fox"
item_B completed: "The quick brown fox jumps over the lazy dog"
item_C completed: "The quick brown fox jumps over the lazy dog ... returns home"
The bundled/local renderers can visually mask this by replacing the current user display, but a protocol client keyed by item_id cannot infer that B supersedes A or that C supersedes B.
Protocol constraints
The OpenAI Realtime lifecycle associates input_audio_buffer.speech_started and speech_stopped with the user message item created from that speech segment. Incremental transcript deltas add text to one (item_id, content_index), and the completed event provides that committed item's final transcript.
There is no standard server event for revising an already completed input transcription in place. Deleting and recreating an item is possible only as an explicit conversation-history operation and also requires handling any response already based on the deleted item.
Scope
- Define when a speculative turn revision becomes externally committed as a Realtime user item.
- Keep speculative revisions internal until the server is willing to expose a permanent item, or explicitly synchronize any replacement through standard conversation lifecycle events.
- Ensure the client-visible user-item history and the server/LLM chat history converge.
- Define behavior when speech resumes:
- before the current speculative revision has committed;
- after an input item has emitted transcription completion;
- after assistant generation has started or produced committed output.
- Preserve cancellation and response ordering when a still-uncommitted revision is superseded.
- Add protocol-level tests using only standard events and item IDs, without relying on bundled-client knowledge of internal turn revisions.
Acceptance criteria
- A generic Realtime client that maintains history solely from standard events ends with the same committed user-turn history as the server/LLM.
- Superseded, uncommitted revisions do not publish terminal user items, terminal transcription events, or accepted assistant output.
- Once a user item emits
conversation.item.input_audio_transcription.completed, its transcript is not silently replaced only in server-side chat state.
- Continuing speech after an externally committed item is either represented as a genuine new item containing only the new turn/segment, or the earlier item is explicitly removed/replaced with a fully synchronized lifecycle.
- One long utterance with several Smart Turn/VAD pauses is covered end to end, including a pause long enough for final STT and a later continuation.
- Tests assert item identity, event ordering, client history, internal chat history, and assistant cancellation/commit behavior.
Non-goals
Related
Problem
The speculative VAD/STT pipeline can reopen and supersede revisions of one logical user turn. Internally,
RealtimeServicecan replace the previous speculative user message with the latest cumulative transcript. That replacement is not currently represented equivalently on the Realtime wire.A generic Realtime client only knows standard event identity:
item_id;conversation.item.input_audio_transcription.completedis terminal for that item;item_idrepresents a different user item;supersedes, transcript-retraction, or in-place transcript-replacement event.When speech resumes after an earlier revision has already emitted
completed, the server can emit another cumulative completion under a new item ID while replacing the earlier message only in its internal chat history. A normal client then retains multiple independent user turns even though the LLM sees one revised turn.For example, one utterance with pauses can appear on the wire as:
The bundled/local renderers can visually mask this by replacing the current user display, but a protocol client keyed by
item_idcannot infer that B supersedes A or that C supersedes B.Protocol constraints
The OpenAI Realtime lifecycle associates
input_audio_buffer.speech_startedandspeech_stoppedwith the user message item created from that speech segment. Incremental transcript deltas add text to one(item_id, content_index), and the completed event provides that committed item's final transcript.There is no standard server event for revising an already completed input transcription in place. Deleting and recreating an item is possible only as an explicit conversation-history operation and also requires handling any response already based on the deleted item.
Scope
Acceptance criteria
conversation.item.input_audio_transcription.completed, its transcript is not silently replaced only in server-side chat state.Non-goals
Related