change ChatCompletionChunk to align with "OpenAI Chat Completions str… #3003
Conversation
…eaming API"
When stream_options: {"include_usage": true} is included, choices is None only for the last chunk, and usage is always None except for the last chunk.
Signed-off-by: Wang, Yi A <[email protected]>
|
details please refer discussion in vllm-project/vllm#12394 |
Signed-off-by: Wang, Yi A <[email protected]>
|
What's broken otherwise ? Our baseline is if it breaks the OpenAI client we try to fix it (depending on how much it's tying to internals it's sometimes hard to be ISO compatible), and otherwise we try to minimize the amount of bits we send (that are not useful) and therefore we don't send what's not needed. We don't look that much as the actual payload they send/receive as they tend to change them. |
|
Closing this as TGI doesn't need to output None all the time. The OpenAI client will put them back, and adding unecessary values (and there's already some that are still there) is just wasted bandwidth. |
|
This has been merged in a different form in the follow-up PR: #3007 The core idea is the same, but lots of testing code was modified to make the tests more consistent a checking for the actual behavior of this flag (plus simplifying some internal structures). |
…eaming API"
When stream_options: {"include_usage": true} is included, choices is None only for the last chunk, and usage is always None except for the last chunk.
see https://cookbook.openai.com/examples/how_to_stream_completions#4-how-to-get-token-usage-data-for-streamed-chat-completion-response
@OlivierDehaene OR @Narsil