Skip to content

feat: track out of budget, rate limits, message too large errors explicitly#114

Merged
subpath merged 1 commit intomainfrom
feat-track-rejections-explicitly-genai-3942
Mar 12, 2026
Merged

feat: track out of budget, rate limits, message too large errors explicitly#114
subpath merged 1 commit intomainfrom
feat-track-rejections-explicitly-genai-3942

Conversation

@subpath
Copy link
Collaborator

@subpath subpath commented Mar 12, 2026

Jira ticket: https://mozilla-hub.atlassian.net/browse/GENAI-3942
Dashboard: https://yardstick.mozilla.org/d/aonbrtc/firefox-ai-executive-dashboard-server-side?folderUid=ff4lo7wllzldsb&orgId=1&from=now-7d&to=now&timezone=browser&var-query0=&var-environment=prod&var-query0-2=&var-datasource=adpvtjmrxoc1sb&showCategory=Histogram

What's new:

Prometheus metric to track completion rejections with service-type, model-name and rejection errors

QA:

  • unit tests old and new ✅
  • Local QA:
    Metrics after the Out Of Budget event:
.....
# TYPE mlpa_chat_completions_with_tools_total counter
# HELP mlpa_chat_tool_calls_per_completion Distribution of tool calls per completion.
# TYPE mlpa_chat_tool_calls_per_completion histogram
# HELP mlpa_chat_requests_with_tools_total Number of chat requests that included a tools payload.
# TYPE mlpa_chat_requests_with_tools_total counter
# HELP mlpa_chat_request_rejections_total Number of chat requests rejected due to budget, rate limit, or payload size.
# TYPE mlpa_chat_request_rejections_total counter
mlpa_chat_request_rejections_total{model="openai/gpt-4o",reason="budget_exceeded",service_type="ai"} 1.0
# HELP mlpa_chat_request_rejections_created Number of chat requests rejected due to budget, rate limit, or payload size.
# TYPE mlpa_chat_request_rejections_created gauge
mlpa_chat_request_rejections_created{model="openai/gpt-4o",reason="budget_exceeded",service_type="ai"} 1.773301188783722e+09

@subpath subpath merged commit 250270d into main Mar 12, 2026
1 check passed
@subpath subpath deleted the feat-track-rejections-explicitly-genai-3942 branch March 12, 2026 13:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants