[AIT-251] Add an overview page for AI Transport #3094
Conversation
Link to the pending `/ai-transport` overview page.
Add intro describing the pattern, its properties, and use cases.
Includes continuous token streams, correlating tokens for distinct responses, and explicit start/end events.
Splits each token streaming approach into distinct patterns and shows both the publish and subscribe side behaviour alongside one another.
Includes hydration with rewind and hydration with persisted history + untilAttach. Describes the pattern for handling in-progress live responses with complete responses loaded from the database.
Add doc explaining streaming tokens with appendMessage and update compaction allowing message-per-response history.
Unifies the token streaming nav for token streaming after rebase.
Refines the intro copy in message-per-response to have structural similarity with the message-per-token page.
Refine the Publishing section of the message-per-response docs. - Include anchor tags on title - Describe the `serial` identifier - Align with stream pattern used in message-per-token docs - Remove duplicate example
Refine the Subscribing section of the message-per-response docs. - Add anchor tag to heading - Describes each action upfront - Uses RANDOM_CHANNEL_NAME
Refine the rewind section of the message-per-response docs. - Include description of allowed rewind paameters - Tweak copy
Refines the history section for the message-per-response docs. - Adds anchor to heading - Uses RANDOM_CHANNEL_NAME - Use message serial in code snippet instead of ID - Tweaks copy
Fix the hydration of in progress responses via rewind by using the responseId in the extras to correlate messages with completed responses loaded from the database.
Fix the hydration of in progress responses using history by obtaining the timestamp of the last completed response loaded from the database and paginating history forwards from that point.
Removes the headers/metadata section, as this covers the specific semantics of extras.headers handling with appends, which is better addressed by the (upcoming) message append pub/sub docs. Instead, a callout is used to describe header mixin semantics in the appropriate place insofar as it relates to the discussion at hand.
Update the token streaming with message per token docs to include a callout describing resume behaviour in case of transient disconnection.
Fix the message per token docs headers to include anchors and align with naming in the message per response page.
Adds an overview page for a Sessions & Identity section which describes the channel-oriented session model and its benefits over the traditional connection-oriented model. Describes how identity relates to session management and how this works in the context of channel-oriented sessions. Shows how to use identified clients to assign a trusted identity to users and obtain this identity from the agent side. Shows how to use Ably capabilities to control which operations authenticated users can perform on which channels. Shows how to use authenticated user claims to associated a role or other attribute with a user. Updates the docs to describe how to handle authentication, capabilities, identity and roles/attributes for agents separately from end users. Describes how to use presence to mark users and agents as online/offline. Includes description of synthetic leaves in the event of abrupt disconnection. Describe how to subscribe to presence to see who is online, and take action when a user is offline across all devices. Add docs for resuming user and agent sessions, linking to hydration patterns for different token streaming approaches for user resumes and describing agent resume behaviour with message catch up.
Adds a guide for using the OpenAI SDK to consume streaming events from the Responses API and publish them over Ably using the message per token pattern.
- Uses a further-reading callout instead of note - Removes repeated code initialising Ably client (OpenAI client already instantiated)
Adds an anchor tag to the "Client hydration" heading
Similar to the open ai message per token guide, but using the message per response pattern with appends.
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
|
||
| AI Transport runs on Ably's fault-tolerant and highly-available platform. The platform enables data to be streamed between all internet-connected devices at low latencies across the globe. Its elastic global infrastructure delivers enterprise-scale messaging that effortlessly scales to meet demand. | ||
|
|
||
| Drop AI Transport into your applications to transform unreliable HTTP connections into reliable, stateful AI experiences that keep users engaged. |
There was a problem hiding this comment.
I personally don't think we should be making our argument based on HTTP streams being unreliable. Generally they're not. The key benefits of AIT vs streamed HTTP are:
- they're stateful;
- they're bidirectional.
We should of course additionally talk about resumability etc, but if HTTP reliability was the only concern, there wouldn't be that much incentive to use AIT.
There was a problem hiding this comment.
I've reworded it away from HTTP streams entirely; e1e19ed
|
|
||
| Token streaming is the core of how LLMs deliver their responses to users. Tokens are progressively streamed to users from your LLM so that users don't need to wait for a complete response before seeing any output. | ||
|
|
||
| If you stream tokens over brittle HTTP then any interruption to the connection means that all tokens transmitted during the interruption are lost. That might be if a user switches tabs, temporarily loses network connectivity or if their browser crashes - all common scenarios that should be handled gracefully for industry-leading user experiences. |
|
|
||
| ### Enterprise controls <a id="enterprise"/> | ||
|
|
||
| Ably's platform provides [integrations](/docs/platform/integrations) and functionality to ensure that your applications always exceed the requirements of enterprise environments. Whether that's [message auditing](/docs/platform/integrations/streaming), [client identification](/docs/auth/identified-clients) or [RBAC](/docs/auth/capabilities). |
There was a problem hiding this comment.
| Ably's platform provides [integrations](/docs/platform/integrations) and functionality to ensure that your applications always exceed the requirements of enterprise environments. Whether that's [message auditing](/docs/platform/integrations/streaming), [client identification](/docs/auth/identified-clients) or [RBAC](/docs/auth/capabilities). | |
| Ably's platform provides [integrations](/docs/platform/integrations) and functionality to ensure that your applications always exceed the requirements of enterprise environments. Whether that's [message auditing](/docs/platform/integrations/streaming), [client identification](/docs/auth/identified-clients) or [fine-grained authorization](/docs/auth/capabilities). |
I'm not that comfortable describing it as RBAC, because it's not role-based
There was a problem hiding this comment.
Changed to 'granular access controls' 5ae55a7
There was a problem hiding this comment.
Personally I think "granular" is ambiguous - in some contexts it means coarse-grained, and fine-grained in others, so I try to avoid it
|
|
||
| Regardless of whether you're building with OpenAI, Anthropic or Vercel, AI Transport enables you to add a realtime delivery layer to your application, providing the infrastructure required to deliver modern, stateful AI experiences to users. | ||
|
|
||
| AI Transport runs on Ably's fault-tolerant and highly-available platform. The platform enables data to be streamed between all internet-connected devices at low latencies across the globe. Its elastic global infrastructure delivers enterprise-scale messaging that effortlessly scales to meet demand. |
There was a problem hiding this comment.
Perhaps keywords here can include links to platform docs, e.g.:
- /docs/platform/architecture/fault-tolerance
- /docs/platform/architecture/latency
- /docs/platform/architecture/platform-scalability
|
|
||
|  | ||
|
|
||
| ## Get started |
There was a problem hiding this comment.
We will have JS guides for OpenAI and Anthropic for the release at the end of this week, which we can link to.
Python translations will follow shortly, along with Vercel and LangChain guides.
There was a problem hiding this comment.
I've put these in as placeholders, as wasn't sure of the final list before release.
|
|
||
| ## Features | ||
|
|
||
| AI Transport provides a range of features that extend the existing functionality of Ably's Pub/Sub product to enable you to deliver reliable, stateful AI experiences, that provide the first class UX that is expected from modern applications. |
There was a problem hiding this comment.
AIT is not just Pub/Sub, as LiveObjects will be used for some state sync requirements, and (eventually) we expect to have support in Chat for conversational agents (although this is a bit further out)
There was a problem hiding this comment.
Yes, but I couldn't find anything using LO yet in the docs I was working from. Happy to add if there will be some docs on this before release.
|
|
||
| Token streaming is the core of how LLMs deliver their responses to users. Tokens are progressively streamed to users from your LLM so that users don't need to wait for a complete response before seeing any output. | ||
|
|
||
| If you stream tokens over brittle HTTP then any interruption to the connection means that all tokens transmitted during the interruption are lost. That might be if a user switches tabs, temporarily loses network connectivity or if their browser crashes - all common scenarios that should be handled gracefully for industry-leading user experiences. |
There was a problem hiding this comment.
Or refreshes the page
|
|
||
| ### State hydration <a id="hydration"/> | ||
|
|
||
| AI Transport benefits from all the same functionality of Ably's platform. One of the key features for AI Transport is message history; enabling users and agents to retrieve the state of their conversation at any point in time. |
There was a problem hiding this comment.
In addition to message history, synced state via LiveObjects will be a key part of the story (although the state sync stuff isnt yet documented, but is high priority for us)
There was a problem hiding this comment.
Will this exist before release?
|
|
||
| [Read more about state hydration](/docs/ai-transport/features/token-streaming/message-per-response#hydration). | ||
|
|
||
| ### Status-aware cost controls <a id="cost"/> |
There was a problem hiding this comment.
I think the value prop is more than just cost control - it more generally enables async workloads, i.e. a user can go away, the agent continues work, can notify the user via push notifications when done, and the user can return at any point to hydrate and resume the latest session state
There was a problem hiding this comment.
Updated that section, but not sure if it has too much overlap with state hydration or not now: 77d9ca6
|
|
||
| Take a look at some example code running in-browser of the sorts of features you can build with AI Transport underpinning your applications: | ||
|
|
||
| <Tiles> |
There was a problem hiding this comment.
We have an example for message per token (in PR) and message per response (coming soon), but these examples are not model-specific (they use a mock LLM)
There was a problem hiding this comment.
I put this in as a placeholder too (same as the getting started guides). Will they both be ready for release?
rainbowFi
left a comment
rainbowFi
left a comment
There was a problem hiding this comment.
A couple of minor copy edits for readability
|
|
||
| ## Features | ||
|
|
||
| AI Transport provides a range of features that extend the existing functionality of Ably's Pub/Sub product to enable you to deliver reliable, stateful AI experiences, that provide the first class UX that is expected from modern applications. |
There was a problem hiding this comment.
| AI Transport provides a range of features that extend the existing functionality of Ably's Pub/Sub product to enable you to deliver reliable, stateful AI experiences, that provide the first class UX that is expected from modern applications. | |
| AI Transport provides a range of features that extend the existing functionality of Ably's Pub/Sub product to enable you to deliver reliable, stateful AI experiences, that provide the first class UX your users expect from modern applications. |
|
|
||
| ### State hydration <a id="hydration"/> | ||
|
|
||
| AI Transport benefits from all the same functionality of Ably's platform. One of the key features for AI Transport is message history; enabling users and agents to retrieve the state of their conversation at any point in time. |
There was a problem hiding this comment.
| AI Transport benefits from all the same functionality of Ably's platform. One of the key features for AI Transport is message history; enabling users and agents to retrieve the state of their conversation at any point in time. | |
| AI Transport benefits from all the functionality of Ably's platform. One of the key features for AI Transport is message history; enabling users and agents to retrieve the state of their conversation at any point in time. |
mschristensen
force-pushed
the
AIT-129-AIT-Docs-release-branch
branch
from
7821a8f to
426203f
Compare
Description
This PR adds an overview page for AI Transport.
Checklist