Skip to content

authorized_hits metric not emitted for SSE streaming responses #373

Description

@noyitz

Description

The authorized_hits metric (token count) is only emitted for non-streaming responses. When the upstream provider returns an SSE (Server-Sent Events) streaming response (e.g., Content-Type: text/event-stream), the Wasm plugin does not parse the SSE stream to extract usage.total_tokens from the final event, so the ratelimit-report to Limitador never includes the token count.

Impact

  • authorized_hits{user, model, subscription} — only populated for non-streaming requests
  • authorized_calls{user, subscription} — works correctly for all requests (streaming and non-streaming)
  • Billing/cost dashboards based on authorized_hits undercount token usage for streaming-heavy workloads (e.g., Claude Code, OpenAI Codex which always use streaming)

Reproduction

  1. Configure an ExternalModel pointing to Anthropic API with stream: true
  2. Send a request through the gateway
  3. Check Limitador metrics — authorized_calls increments but authorized_hits does not
  4. Send the same request with stream: false — both metrics increment

Expected Behavior

The Wasm plugin should parse SSE responses to extract token usage from the final event:

  • Anthropic: event: message_delta → data: {"usage": {"output_tokens": N}}
  • OpenAI Responses API: event: response.completed → data: {"response": {"usage": {"total_tokens": N}}}
  • OpenAI Chat Completions: final chunk with usage field when stream_options.include_usage: true

Workaround

We implemented SSE parsing in the IPP (ext_proc) response handler to extract usage data from streaming responses. See:

Environment

  • Kuadrant Wasm shim: v0.12.1
  • Limitador: v2.3.0
  • OpenShift: 4.19
  • MaaS: v0.1.0 (kustomize mode)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    • Status
      No status

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions