Skip to content

fix(openai-compat): surface availability state for OpenAI-compatible backends #149

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
petersimmons1972 wants to merge 1 commit into
base: main
Choose a base branch
from
Open

fix(openai-compat): surface availability state for OpenAI-compatible backends #149

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion internal/adapter/converter/unified_converter.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,9 +74,15 @@ func (c *UnifiedConverter) ConvertToFormat(models []*domain.UnifiedModel, filter
func (c *UnifiedConverter) convertModel(model *domain.UnifiedModel) UnifiedModelData {
availability := make([]EndpointStatus, 0, len(model.SourceEndpoints))
for _, ep := range model.SourceEndpoints {
// Use GetEffectiveState() rather than ep.State directly: the lifecycle
// unifier updates the typed ModelState field, while ep.State (the legacy
// string) is only set at discovery time and never transitions. Reading
// the effective state ensures health-driven transitions surface in the
// API response. GetEffectiveState() also normalises legacy string values
// ("loaded", "not-loaded", "available") to the typed enum.
availability = append(availability, EndpointStatus{
Endpoint: ep.EndpointName, // Use endpoint name instead of URL
State: ep.State,
State: string(ep.GetEffectiveState()),
Comment on lines +77 to +85
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot May 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Keep availability filtering aligned with effective state

convertModel now uses effective state, but matchesAvailabilityFilter still checks legacy state (ep.State) at Line 172. This can produce inconsistent /olla/models?available=... results versus the availability.state returned in the payload.

Suggested fix 
 func matchesAvailabilityFilter(model *domain.UnifiedModel, available *bool) bool {
 	if available == nil {
 		return true
 	}

 	isAvailable := false
 	for _, ep := range model.SourceEndpoints {
-		if ep.State == "loaded" {
+		if string(ep.GetEffectiveState()) == "loaded" {
 			isAvailable = true
 			break
 		}
 	}
 	return *available == isAvailable
 }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
// Use GetEffectiveState() rather than ep.State directly: the lifecycle
// unifier updates the typed ModelState field, while ep.State (the legacy
// string) is only set at discovery time and never transitions. Reading
// the effective state ensures health-driven transitions surface in the
// API response. GetEffectiveState() also normalises legacy string values
// ("loaded", "not-loaded", "available") to the typed enum.
availability = append(availability, EndpointStatus{
Endpoint: ep.EndpointName, // Use endpoint name instead of URL
State: ep.State,
State: string(ep.GetEffectiveState()),
func matchesAvailabilityFilter(model *domain.UnifiedModel, available *bool) bool {
if available == nil {
return true
}
isAvailable := false
for _, ep := range model.SourceEndpoints {
if string(ep.GetEffectiveState()) == "loaded" {
isAvailable = true
break
}
}
return *available == isAvailable
}
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@internal/adapter/converter/unified_converter.go` around lines 77 - 85,
convertModel builds availability using ep.GetEffectiveState() but
matchesAvailabilityFilter still reads the legacy ep.State, causing inconsistent
filtering vs the returned availability.State; update matchesAvailabilityFilter
to use ep.GetEffectiveState() (or the normalized typed value it returns) instead
of ep.State so the /olla/models?available=... filter aligns with the
availability entries produced by convertModel; locate matchesAvailabilityFilter
and change its state-check logic to call ep.GetEffectiveState() (or compare
against the same enum/string used when creating EndpointStatus) so both
filtering and payload use the same effective state source.

})
}
// OLLA-85: [Unification] Models with different digests fail to unify correctly.
Expand Down
9 changes: 9 additions & 0 deletions internal/adapter/registry/profile/parsers.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -225,6 +225,15 @@ func (p *openAIParser) Parse(data []byte) ([]*domain.ModelInfo, error) {
Name: model.ID,
Type: model.Object, // always "model" in openai responses
LastSeen: now,
// OpenAI-compatible /v1/models has no size or state fields. For these
// backends (vllm, llama.cpp, Infinity, etc.) the presence of a model
// in the discovery response IS the availability signal — these servers
// only list models that are loaded and ready to serve. Without a
// non-zero Size, MapModelState() falls through to "unknown" and the
// model never appears available in the unified /olla/models response.
// Set a sentinel size of 1 to signal "loaded / available, exact size
// not reported by this backend".
Size: 1,
}

// openai is stingy with metadata
Expand Down