Context
PR #4861 publishes chat events synchronously to keep delta, sources, and done events in the correct order. This prevents done from arriving before the last parts of the answer.
Problem
Publishing an event can wait for Redis or for a realtime client. Because we now wait after every delta, a slow publish also pauses the stream coming from the RAG server.
Example
A user has a slow websocket connection while a long answer is being generated. The answer produces more than 100 delta events, but the client cannot read them fast enough. The subscriber buffer fills, followed by the topic buffer. The next Hub.Publish call then waits until the client catches up or disconnects.
While it waits, cozy-stack stops reading the response from the RAG server. The answer appears frozen and the rag-query worker remains busy.
Possible solution
Add a bounded, ordered queue for each chat stream. Reading from the RAG server and publishing realtime events would be separate, while events would still be sent in order.
We should also define what happens when the queue is full or the request is cancelled, and add metrics for queue size and publish time.
Related: #4861
Context
PR #4861 publishes chat events synchronously to keep delta, sources, and done events in the correct order. This prevents done from arriving before the last parts of the answer.
Problem
Publishing an event can wait for Redis or for a realtime client. Because we now wait after every delta, a slow publish also pauses the stream coming from the RAG server.
Example
A user has a slow websocket connection while a long answer is being generated. The answer produces more than 100 delta events, but the client cannot read them fast enough. The subscriber buffer fills, followed by the topic buffer. The next Hub.Publish call then waits until the client catches up or disconnects.
While it waits, cozy-stack stops reading the response from the RAG server. The answer appears frozen and the rag-query worker remains busy.
Possible solution
Add a bounded, ordered queue for each chat stream. Reading from the RAG server and publishing realtime events would be separate, while events would still be sent in order.
We should also define what happens when the queue is full or the request is cancelled, and add metrics for queue size and publish time.
Related: #4861