Skip to content

kserve: a served model reports the API interfaces its runtime registered #144

Description

@teemow

Problem

Nothing in model-manager's API says which API interfaces a served model answers. Model, LoadedModel and Preset carry capabilities (the preset's free-form tags: chat, tools, reasoning, …) and format (the string vLLM); the agent endpoint hardcodes Provider: "OpenAI" and BaseURL + "/v1" (internal/backend/kserve/served.go). A caller — the portal's Models pages through muster, an agent picking a model, a person pointing a CLI at it — cannot see that a model served on llm-d answers OpenAI chat completions, legacy completions, the Responses API and Anthropic Messages with count_tokens, or that an embeddings preset answers /v1/embeddings and no chat interface at all. Clients differ exactly here (Claude Code speaks Messages, most CLIs chat completions, newer ones Responses), so the guess is made on the wrong side.

The interface set is not a preset property and must not be typed per preset. It follows the runtime version (vLLM serves /v1/responses since 0.10.0, /v1/messages since 0.11.1, /v1/messages/count_tokens since 0.17.0) and the model's task class: since vLLM 0.16.0 the OpenAI-compatible server registers a family's routers only when the engine's supported tasks include it — a generate model gets chat completions, completions, Responses and Messages, a pooling model gets embeddings, score and rerank and no /v1/chat/completions (a request to it is a plain 404). GET /v1/models (id, root, max_model_len, permission) and GET /version say nothing about tasks; GET /openapi.json (FastAPI's default, absent only with --disable-fastapi-docs) lists exactly the routes this server registered for this model. The llm-d runtime the platform ships is well past 0.16.

Proposed solution

When an LLMInferenceService turns Ready, the kserve backend reads the running server once, at the origin the agent endpoint uses (the workload Service, port 8000): GET /version for the runtime version and GET /openapi.json for the route list. The route list is the interface list. model-manager maps the known paths onto the platform's interface vocabulary — agentgateway's format names, so the same list can later describe the model to a gateway — and reports:

runtime: {name: vllm, version: "0.23.0"}
interfaces:
  - {type: Completions, path: /v1/chat/completions}   # legacy /v1/completions implied
  - {type: Responses, path: /v1/responses}
  - {type: Messages, path: /v1/messages}
  - {type: AnthropicTokenCount, path: /v1/messages/count_tokens}
  # a pooling preset: {type: Embeddings, path: /v1/embeddings}

on LoadedModel (and on Model while it is loaded), in list_loaded_models, get_model, GET /api/v1/loaded and api/openapi.yaml. Preset gets no interface field: a preset states no interfaces. The read happens once per Ready transition and is kept with the loaded model. A server that answers no /openapi.json (a runtime started with --disable-fastapi-docs) or is older than 0.16 reports an empty list and a reason naming why — nothing is inferred from a version table.

Whether a registered interface works for a given request is a launch-time fact of the preset — tool calling needs --enable-auto-tool-choice --tool-call-parser, thinking blocks a --reasoning-parser, chat a template — and stays with the preset (giantswarm/agent-platform#313); this issue reports what the server registered, not what a request will get back.

The read needs the serving namespace's network policy to admit model-manager and the presets to keep FastAPI's docs on: giantswarm/agent-platform#602.

Acceptance criteria

  • list_loaded_models on a Ready llm-d model carries runtime.version (vLLM's) and interfaces with Completions, Responses, Messages and AnthropicTokenCount for a generate preset; an embeddings preset lists Embeddings and no Completions.
  • The interfaces come from the server's route list; a server whose /openapi.json is unreachable or absent reports an empty list and a reason — no per-version table, nothing invented.
  • list_models shows the same for a loaded model; list_presets carries no interface field.
  • Unit tests over recorded openapi.json documents of a generate and a pooling server; a lab proof in the agentlab serving slice with the fields on the wire through muster.
  • The API reference and the MCP tool descriptions name the field and its vocabulary; the vocabulary is agentgateway's format names.

Epic: giantswarm/giantswarm#37590. Related: giantswarm/agent-platform#602 (the policies and the docs flag), giantswarm/backstage#2495 (the Models pages), #145 (the gateway model built from this list), giantswarm/agent-platform#313 (the parsers stay preset facts).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions