Mike/ait token streaming OpenAI sdk #3074
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
mschristensen
force-pushed
the
mike/AIT-token-streaming-OpenAI-SDK
branch
from
a06f5d1 to
52b2e41
Compare
mschristensen
force-pushed
the
mike/AIT-token-streaming-OpenAI-SDK
branch
from
52b2e41 to
3411525
Compare
mschristensen
force-pushed
the
mike/AIT-token-streaming-OpenAI-SDK
branch
from
3411525 to
b7116ee
Compare
Uses numbered steps in a tutorial-like format. Simplifies the code and descriptions of the event streaming model, restricting to the relevant context for this tutorial. Improves copy and code for readability.
Indicates use of the message per token pattern.
paddybyers
approved these changes
Jan 7, 2026
| </Code> | ||
|
|
||
| <Aside data-type="note"> | ||
| This guide uses version 4.x of the OpenAI SDK. Some details of interacting with the OpenAI SDK may diverge from those given here if using a different major version. |
Member
There was a problem hiding this comment.
Suggested change
| This guide uses version 4.x of the OpenAI SDK. Some details of interacting with the OpenAI SDK may diverge from those given here if using a different major version. | |
| This guide uses version 4.x of the OpenAI SDK. Some details of interacting with the OpenAI SDK may differ from those given here if using a different major version. |
| </Code> | ||
|
|
||
| <Aside data-type="note"> | ||
| This is only a representative example for a simple "text in, text out" use case and may not reflect the exact sequence of events that you observe from the OpenAI API. It also does not describe response generation errors or refusals. For complete details on all event types and their properties, see [OpenAI Streaming events](https://platform.openai.com/docs/api-reference/responses-streaming/response). |
Member
There was a problem hiding this comment.
Suggested change
| This is only a representative example for a simple "text in, text out" use case and may not reflect the exact sequence of events that you observe from the OpenAI API. It also does not describe response generation errors or refusals. For complete details on all event types and their properties, see [OpenAI Streaming events](https://platform.openai.com/docs/api-reference/responses-streaming/response). | |
| This is only an illustrative example for a simple "text in, text out" use case and may not reflect the exact sequence of events that you observe from the OpenAI API. It also does not describe response generation errors or refusals. For complete details on all event types and their properties, see [OpenAI Streaming events](https://platform.openai.com/docs/api-reference/responses-streaming/response). |
| - Publishes a `stop` event when the response completes | ||
|
|
||
| <Aside data-type="note"> | ||
| Ably messages are published without `await` to maximize throughput. Ably maintains message ordering even without awaiting each publish. For more information, see [Publishing tokens](/docs/ai-transport/features/token-streaming/message-per-token#publishing). |
Member
There was a problem hiding this comment.
How do we recommend checking for error responses from the publish?
Contributor
Author
There was a problem hiding this comment.
There's a separate ticket to address this: https://ably.atlassian.net/browse/AIT-238
|
|
||
| ### Publishing concurrent responses <a id="multiple-publishers"/> | ||
|
|
||
| The implementation uses `responseId` in message extras to correlate tokens with their originating response. This enables multiple publishers to stream different responses concurrently on the same channel, with each subscriber correctly tracking all responses independently. |
Member
There was a problem hiding this comment.
Suggested change
| The implementation uses `responseId` in message extras to correlate tokens with their originating response. This enables multiple publishers to stream different responses concurrently on the same channel, with each subscriber correctly tracking all responses independently. | |
| The implementation uses `responseId` in message `extras` to correlate tokens with their originating response. This enables multiple publishers to stream different responses concurrently on the same channel, with each subscriber correctly tracking all responses independently. |
(Should we be quoting extras, name etc with backticks?)
rainbowFi
reviewed
Jan 7, 2026
|
|
||
| This guide shows you how to stream AI responses from OpenAI's [Responses API](https://platform.openai.com/docs/api-reference/responses) over Ably using the [message-per-token pattern](/docs/ai-transport/features/token-streaming/message-per-token). Specifically, it implements the [explicit start/stop events approach](/docs/ai-transport/features/token-streaming/message-per-token#explicit-events), which publishes each response token as an individual message, along with explicit lifecycle events to signal when responses begin and end. | ||
|
|
||
| Using Ably to distribute tokens from the OpenAI SDK enables you to broadcast AI responses to thousands of concurrent subscribers with reliable message delivery and ordering guarantees. This approach decouples your AI inference from client connections, enabling you to scale independently and handle reconnections gracefully while ensuring each client receives the complete response stream with all tokens delivered in order. |
Contributor
There was a problem hiding this comment.
Suggested change
| Using Ably to distribute tokens from the OpenAI SDK enables you to broadcast AI responses to thousands of concurrent subscribers with reliable message delivery and ordering guarantees. This approach decouples your AI inference from client connections, enabling you to scale independently and handle reconnections gracefully while ensuring each client receives the complete response stream with all tokens delivered in order. | |
| Using Ably to distribute tokens from the OpenAI SDK enables you to broadcast AI responses to thousands of concurrent subscribers with reliable message delivery and ordering guarantees, ensuring that each client receives the complete response stream with all tokens delivered in order. This approach decouples your AI inference from client connections, enabling you to scale each independently and handle reconnections gracefully. |
|
|
||
| - [`response.created`](https://platform.openai.com/docs/api-reference/responses-streaming/response/created): Signals the start of a response. Contains `response.id` to correlate subsequent events. | ||
|
|
||
| - [`response.output_item.added`](https://platform.openai.com/docs/api-reference/responses-streaming/response/output_item/added): Indicates a new output item. If `item.type === "message"` the item contains model response text; other types may be specified, such as `"reasoning"` for internal reasoning tokens. The `item.id` can be used to filter which tokens to stream. The `output_index` indicates the position of this item in the response's output array. |
Contributor
There was a problem hiding this comment.
The
item.idcan be used to filter which tokens to stream.
I don't quite get this line. I think you're saying "Theitem.idwill be present on on all events relating to this item, so you can use it to filter which tokens are streamed to the clients" but there's probably a better way of saying it.
rainbowFi
approved these changes
Jan 7, 2026
3 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR refactors the OpenAI SDK guide for the message-per-token streaming pattern.
Checklist