Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions inference-platforms/aigw/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,10 @@
This shows how to use [Envoy AI Gateway][docs] to proxy Ollama, accessible via an
OpenAI compatible API.

Envoy AI Gateway [YAML configuration](ai-gateway-local.yaml) is processed and run
by `aigw`, which launches an Envoy proxy to handle requests. OpenTelemetry support
Envoy AI Gateway is automatically configured by OpenAI and OpenTelemetry
environment variables read by `aigw run`, such as `OPENAI_API_KEY`.

`aigw run` launches an Envoy proxy to handle requests. OpenTelemetry support
for GenAI metrics and traces is handled directly in the `aigw` (go) binary.

OpenTelemetry traces produced by Envoy AI Gateway follow the [OpenInference specification][openinference].
Expand Down Expand Up @@ -32,7 +34,7 @@ Once Envoy AI Gateway is running, use [uv][uv] to make an OpenAI request via
[chat.py](../chat.py):

```bash
uv run --exact -q --env-file env.local ../chat.py
OPENAI_BASE_URL=http://localhost:1975/v1 uv run --exact -q --env-file env.local ../chat.py
```

## Notes
Expand Down
113 changes: 0 additions & 113 deletions inference-platforms/aigw/ai-gateway-local.yaml

This file was deleted.

4 changes: 1 addition & 3 deletions inference-platforms/aigw/docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,12 +21,10 @@ services:
env_file:
- env.local
environment:
- OPENAI_HOST=host.docker.internal
- OPENAI_BASE_URL=http://host.docker.internal:11434/v1
- OTEL_EXPORTER_OTLP_ENDPOINT=http://host.docker.internal:4318
ports:
- "1975:1975" # OpenAI compatible endpoint at /v1
extra_hosts: # localhost:host-gateway trick doesn't work with aigw
- "host.docker.internal:host-gateway"
volumes:
- ./ai-gateway-local.yaml:/config.yaml:ro
command: ["run", "/config.yaml"]
3 changes: 2 additions & 1 deletion inference-platforms/aigw/env.local
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
OPENAI_BASE_URL=http://localhost:1975/v1
# Override default ENV variables for Ollama
OPENAI_BASE_URL=http://localhost:11434/v1
OPENAI_API_KEY=unused
CHAT_MODEL=qwen3:0.6B

Expand Down