Summary
When a client requests stream_options.include_usage on a streaming chat completion, the final SSE chunk does not contain a usage field. This breaks benchmarking tools and observability pipelines that rely on per-request token accounting from the streaming response.
Affected versions
- vllm-swift
v0.6.0
- vLLM
0.19.1
Reproduction
curl -N -s -X POST http://127.0.0.1:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "<your model>",
"messages": [{"role":"user","content":"Hello"}],
"max_tokens": 20,
"stream": true,
"stream_options": {"include_usage": true}
}'
Expected (per OpenAI spec): a final SSE data: chunk before data: [DONE] containing a usage object with prompt_tokens, completion_tokens, total_tokens.
Observed: the stream ends with data: [DONE] directly; no usage chunk is emitted. Clients that need streaming throughput numbers cannot recover them without re-tokenizing manually or making a non-streaming follow-up call.
Impact
- Benchmark scripts (e.g., a
bench.py that tracks prefill / decode tok/s) cannot distinguish prompt vs completion tokens from the stream.
- Observability tools that aggregate per-request usage from streaming responses report zero or undefined.
A workaround is to scrape the engine's periodic loggers.py throughput line, but that is averaged and not per-request.
Reported by @yaanfpv.
Summary
When a client requests
stream_options.include_usageon a streaming chat completion, the final SSE chunk does not contain ausagefield. This breaks benchmarking tools and observability pipelines that rely on per-request token accounting from the streaming response.Affected versions
v0.6.00.19.1Reproduction
Expected (per OpenAI spec): a final SSE
data:chunk beforedata: [DONE]containing ausageobject withprompt_tokens,completion_tokens,total_tokens.Observed: the stream ends with
data: [DONE]directly; no usage chunk is emitted. Clients that need streaming throughput numbers cannot recover them without re-tokenizing manually or making a non-streaming follow-up call.Impact
bench.pythat tracks prefill / decode tok/s) cannot distinguish prompt vs completion tokens from the stream.A workaround is to scrape the engine's periodic
loggers.pythroughput line, but that is averaged and not per-request.Reported by @yaanfpv.