Skip to content

stream_options.include_usage not honored in chat completion streaming responses #19

Description

@yaanfpv

Summary

When a client requests stream_options.include_usage on a streaming chat completion, the final SSE chunk does not contain a usage field. This breaks benchmarking tools and observability pipelines that rely on per-request token accounting from the streaming response.

Affected versions

  • vllm-swift v0.6.0
  • vLLM 0.19.1

Reproduction

curl -N -s -X POST http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "<your model>",
    "messages": [{"role":"user","content":"Hello"}],
    "max_tokens": 20,
    "stream": true,
    "stream_options": {"include_usage": true}
  }'

Expected (per OpenAI spec): a final SSE data: chunk before data: [DONE] containing a usage object with prompt_tokens, completion_tokens, total_tokens.

Observed: the stream ends with data: [DONE] directly; no usage chunk is emitted. Clients that need streaming throughput numbers cannot recover them without re-tokenizing manually or making a non-streaming follow-up call.

Impact

  • Benchmark scripts (e.g., a bench.py that tracks prefill / decode tok/s) cannot distinguish prompt vs completion tokens from the stream.
  • Observability tools that aggregate per-request usage from streaming responses report zero or undefined.

A workaround is to scrape the engine's periodic loggers.py throughput line, but that is averaged and not per-request.

Reported by @yaanfpv.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions