feat(metrics): support otel and console metric export #1204

codefromthecrypt · 2025-09-16T02:53:44Z

Description

This extends otel ENV config to metrics, when enabled exports instead of a prometheus listener. This allows metrics in otel native systems like elastic stack, otel tui and otherwise, without a prometheus pump. This also allows you to do ad-hoc configuration by using console config.

Here's an example of standalone using otel-tui https://github.com/ymtdzzz/otel-tui:

Updated cmd/aigw examples:

Add otel-tui service to show native otel w/o prometheus
Use profile-based env file selection (.env.otel.${COMPOSE_PROFILES:-console})
Set console as default for immediate debugging without external dependencies

Documentation:

Document separate OTLP endpoints for traces/metrics and exporter types

Phoenix users: Set OTEL_METRICS_EXPORTER=none as Phoenix only supports traces

Fixes #1100

codecov-commenter · 2025-09-16T03:01:03Z

Codecov Report

❌ Patch coverage is 82.08955% with 12 lines in your changes missing coverage. Please review.
✅ Project coverage is 79.36%. Comparing base (0b97426) to head (84fbab7).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
internal/metrics/metrics.go	77.77%	8 Missing and 4 partials ⚠️

❌ Your project status has failed because the head coverage (79.36%) is below the target coverage (86.00%). You can increase the head coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1204      +/-   ##
==========================================
- Coverage   79.38%   79.36%   -0.03%     
==========================================
  Files          88       89       +1     
  Lines       10067    10131      +64     
==========================================
+ Hits         7992     8040      +48     
- Misses       1712     1725      +13     
- Partials      363      366       +3

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

codefromthecrypt · 2025-09-16T04:24:46Z

Tested with ElasticStack here, so I think we're good elastic/observability-examples#86

codefromthecrypt · 2025-09-16T05:39:11Z

triple checked the README everything is smooth

codefromthecrypt · 2025-09-16T05:47:17Z

make test-extproc is flakey.. trying to figure out why (sometimes passes)

codefromthecrypt · 2025-09-16T06:18:47Z

taking to draft because I think it isn't exposing prom by default as it was before. even if things work I suspect we want to default to that and make sure everything works exactly as before unless you set an override

codefromthecrypt · 2025-09-16T07:21:57Z

hopefully last change fixes the flake where other tests making a request were required to have the one that flaked pass (race condition)

codefromthecrypt · 2025-09-16T07:46:42Z

Something else is flakey not sure if I will be able to solve in this PR

mathetake

Reviewed the code and seems straightforward vs unlocking a new use case which is exciting!

mathetake · 2025-09-16T16:31:13Z

2025/09/16 07:22:47 httptest.Server blocked in Close after 5 seconds, waiting for connections:
  *net.TCPConn 0xc000610650 127.0.0.1:40452 in state active
panic: test timed out after 10m0s
	running tests:
		TestOtelOpenAIChatCompletions_propagation (9m37s)

this seems flaky @codefromthecrypt

codefromthecrypt · 2025-09-16T21:47:59Z

@mathetake refactored the prom metrics test to not be flakey TestOtelOpenAIChatCompletions_propagation I'm racing locally and if can find something will do another PR

codefromthecrypt · 2025-09-17T01:45:34Z

ok 🤞

codefromthecrypt · 2025-09-17T01:46:12Z

internal/metrics/metrics_test.go

+}
+
+// TestNewMetricsFromEnv_NetworkExporters tests OTLP and other network-based exporters.
+// We CANNOT use synctest here because it creates a "bubble" where goroutines are isolated


ps I intentionally left this in as it is something I will discuss in the golang meetup tomorrow ;)

This extends otel ENV config to metrics, when enabled exports instead of a prometheus listener. This allows metrics in otel native systems like elastic stack, otel tui and otherwise, without a prometheus pump. This also allows you to do ad-hoc configuration by using `console` config. Updated cmd/aigw examples: - Add otel-tui service to show native otel w/o prometheus - Use profile-based env file selection (.env.otel.${COMPOSE_PROFILES:-console}) - Set console as default for immediate debugging without external dependencies Documentation: - Document separate OTLP endpoints for traces/metrics and exporter types Phoenix users: Set OTEL_METRICS_EXPORTER=none as Phoenix only supports traces Signed-off-by: Adrian Cole <adrian@tetrate.io>

Signed-off-by: Adrian Cole <adrian@tetrate.io>

codefromthecrypt · 2025-09-17T02:10:14Z

tests/extproc/vcr/prometheus_metrics_test.go

+// TestPrometheusMetrics verifies that metrics are properly exported via Prometheus
+// when processing a chat completion request. This test uses the default configuration
+// which exposes metrics on the /metrics endpoint.
+func TestPrometheusMetrics(t *testing.T) {


@hustxiayang fyi once this lands, feel free to refactor it or rename it to TestPromOpenAIChatCompletions so you can have a second TestPromOpenAIEmbeddings that verifies the token things aren't there (or remain not there). another way is to make this a table test, but the parameters may be difficult enough to warrant a separate test for embeddings. not sure.. anyway hope this base test helps!

**Description** This extends otel ENV config to metrics, when enabled exports instead of a prometheus listener. This allows metrics in otel native systems like elastic stack, otel tui and otherwise, without a prometheus pump. This also allows you to do ad-hoc configuration by using `console` config. Here's an example of standalone using otel-tui https://github.com/ymtdzzz/otel-tui: <img width="720" height="296" alt="tui-metrics" src="https://github.com/user-attachments/assets/6640e6ee-c87c-4586-bce7-13d6ca837c96" /> <img width="2096" height="1014" alt="tui-trace" src="https://github.com/user-attachments/assets/0feb6ed2-87e0-4b43-9c96-7f09493ec365" /> Updated cmd/aigw examples: - Add otel-tui service to show native otel w/o prometheus - Use profile-based env file selection (.env.otel.${COMPOSE_PROFILES:-console}) - Set console as default for immediate debugging without external dependencies Documentation: - Document separate OTLP endpoints for traces/metrics and exporter types Phoenix users: Set OTEL_METRICS_EXPORTER=none as Phoenix only supports traces Fixes envoyproxy#1100 --------- Signed-off-by: Adrian Cole <adrian@tetrate.io> Signed-off-by: Erica Hughberg <erica.sundberg.90@gmail.com>

codefromthecrypt force-pushed the otel-metrics branch 3 times, most recently from 1108236 to 0c83d4f Compare September 16, 2025 04:16

codefromthecrypt mentioned this pull request Sep 16, 2025

add Envoy AI Gateway (aigw) to inference-platforms elastic/observability-examples#86

Merged

codefromthecrypt marked this pull request as ready for review September 16, 2025 04:24

codefromthecrypt requested a review from a team as a code owner September 16, 2025 04:24

codefromthecrypt marked this pull request as draft September 16, 2025 06:18

codefromthecrypt marked this pull request as ready for review September 16, 2025 07:53

mathetake approved these changes Sep 16, 2025

View reviewed changes

mathetake self-requested a review September 16, 2025 16:31

mathetake self-assigned this Sep 16, 2025

codefromthecrypt force-pushed the otel-metrics branch from d671be5 to b0fd33d Compare September 16, 2025 21:45

codefromthecrypt commented Sep 17, 2025

View reviewed changes

codefromthecrypt added 7 commits September 17, 2025 11:54

fuzz

5c616b0

Signed-off-by: Adrian Cole <adrian@tetrate.io>

service_name

47f0d2c

Signed-off-by: Adrian Cole <adrian@tetrate.io>

fix flake

1670887

Signed-off-by: Adrian Cole <adrian@tetrate.io>

fix prom

ad08389

Signed-off-by: Adrian Cole <adrian@tetrate.io>

refactor

0f113b5

Signed-off-by: Adrian Cole <adrian@tetrate.io>

polish

84fbab7

Signed-off-by: Adrian Cole <adrian@tetrate.io>

codefromthecrypt force-pushed the otel-metrics branch from 9cddcb8 to 84fbab7 Compare September 17, 2025 01:54

codefromthecrypt mentioned this pull request Sep 17, 2025

refactor: separate embedding metrics to remove metrics which does not make sense for embeddings #1206

Closed

codefromthecrypt commented Sep 17, 2025

View reviewed changes

mathetake approved these changes Sep 17, 2025

View reviewed changes

mathetake merged commit 58a130b into envoyproxy:main Sep 17, 2025
28 checks passed

This was referenced Sep 24, 2025

📝 Add guide documentation (58a130b9) missBerg/ai-gateway#2

Closed

📝 Add guide documentation (58a130b9) missBerg/ai-gateway#29

Closed

📝 Add guide documentation (58a130b9) missBerg/ai-gateway#33

Closed

feat(metrics): support otel and console metric export #1204

feat(metrics): support otel and console metric export #1204

Uh oh!

Conversation

codefromthecrypt commented Sep 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented Sep 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

codefromthecrypt commented Sep 16, 2025

Uh oh!

codefromthecrypt commented Sep 16, 2025

Uh oh!

codefromthecrypt commented Sep 16, 2025

Uh oh!

codefromthecrypt commented Sep 16, 2025

Uh oh!

codefromthecrypt commented Sep 16, 2025

Uh oh!

codefromthecrypt commented Sep 16, 2025

Uh oh!

mathetake left a comment

Choose a reason for hiding this comment

Uh oh!

mathetake commented Sep 16, 2025

Uh oh!

codefromthecrypt commented Sep 16, 2025

Uh oh!

codefromthecrypt commented Sep 17, 2025

Uh oh!

codefromthecrypt Sep 17, 2025

Choose a reason for hiding this comment

Uh oh!

codefromthecrypt Sep 17, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codefromthecrypt commented Sep 16, 2025 •

edited

Loading

codecov-commenter commented Sep 16, 2025 •

edited

Loading