Make the model scanner work on machines that are not the author's

An audit across realistic model sets found the discovery layer filled every role
on the author's box and left roles empty nearly everywhere else. Three causes.

Undeclared size was treated as disqualifying. /v1/models states no parameter
count, so every OpenAI-compatible and cloud model arrived with param_b=None
against a rule reading `pb is not None and pb >= 18` — GPT-4o was Execution-only
while a local 8B held the same role. The asymmetry was accidental: three lines
above, unknown tool_calling is already ELIGIBLE-but-unproven, and _score gives an
unknown size no credit, so a measured model still outranks a maybe. A test pins
that ordering.

No threshold was loosened. A measured 8B still cannot be Council, because it
genuinely is not council-class and saying otherwise would be the flattering
answer. role_basis now records whether a role was held on measured grounds or
merely undeclared ones — a guess and a measurement should not be presented as
the same claim.

The role fallback was capability-blind: asked for Vision with no vision model
installed, it returned a code model, because the chain fell back to Execution
regardless of what was asked. That is not degradation — a text model handed an
image answers wrongly or errors. Vision and Embedding now refuse rather than
substitute, so a caller can say "no vision model installed" instead of relaying
confident nonsense. Quality-tier substitutions are still permitted and now
carry `substituted`, `requested_role`, and a reason; previously the only signal
was comparing the returned role against one the caller had to remember asking for.

A refusal was filed as an absence. The probes sent no Authorization header, so a
runtime behind `vllm serve --api-key` answered 401 and was reported offline —
sending the operator to restart a server that was running and had simply refused
an unauthenticated request. The key now travels with the probe, an HTTP error
proves reachability, and 401/403 reports authorized:false with the remedy.

15 new tests, including a property test that renaming a model to "frobozz-42x"
cannot change its classification. 687 tests pass.

Author: ElmatadorZ

README.md

@@ -251,7 +251,7 @@ badge is red, the claim that this works is not currently true.
 
 ```bash
 cd backend
-python -m pytest -q          # 672 tests
+python -m pytest -q          # 687 tests
 ```
 
 ---

backend/main.py

@@ -7263,9 +7263,13 @@ def _runtime_extra_probes():
     out = []
     try:
         conn = sqlite3.connect(DB_PATH); c = conn.cursor()
-        for cid, name, base, at in c.execute(
-                "SELECT id,name,base_url,api_type FROM connections"):
-            out.append({"runtime": name or cid, "url": base, "api_type": at or "ollama"})
+        # The api_key travels with the probe: a runtime behind `--api-key` used to
+        # answer 401 and be filed as offline, which sent the operator to restart a
+        # server that was running and had merely refused an unauthenticated request.
+        for cid, name, base, at, key in c.execute(
+                "SELECT id,name,base_url,api_type,api_key FROM connections"):
+            out.append({"runtime": name or cid, "url": base,
+                        "api_type": at or "ollama", "api_key": key or None})
         conn.close()
     except Exception:
         pass

backend/runtime_registry.py

@@ -28,6 +28,11 @@
 _COUNCIL_MIN_B = 30.0   # council = the heaviest models
 
 
+# Roles a text model physically CANNOT stand in for. Substituting here does not
+# degrade quality — it produces confident nonsense, so the router must refuse.
+HARD_CAPABILITY_ROLES = ("Vision", "Embedding")
+
+
 def classify(model: Dict[str, Any]) -> List[str]:
     """Capability → roles. A model may hold several roles. No model names used."""
     roles: List[str] = []
@@ -48,12 +53,21 @@ def classify(model: Dict[str, Any]) -> List[str]:
     if tools is not False and small:
         roles.append("Execution")
 
-    # Reasoning: explicit thinking capability OR a mid/large model.
-    if thinking or (pb is not None and pb >= _REASON_MIN_B):
+    # Reasoning / Council: size is the signal, but an UNDECLARED size is not a
+    # disqualification. /v1/models states no parameter count, so every
+    # OpenAI-compatible and cloud model arrives with param_b=None — and excluding
+    # those meant a frontier hosted model could never hold a reasoning role while
+    # a local 8B could. That is backwards.
+    #
+    # This mirrors the decision already made for tool_calling three lines up:
+    # unknown stays eligible, and `_score` gives it no size credit, so a model
+    # whose size is KNOWN to qualify always outranks one that merely might.
+    # Nothing is lost either way — an empty role already fell back to Execution
+    # and returned the same small model, just without saying so.
+    undeclared = pb is None
+    if thinking or undeclared or pb >= _REASON_MIN_B:
         roles.append("Reasoning")
-
-    # Council: the heaviest models.
-    if pb is not None and pb >= _COUNCIL_MIN_B:
+    if undeclared or pb >= _COUNCIL_MIN_B:
         roles.append("Council")
 
     if not roles:
@@ -78,6 +92,19 @@ def flatten(scan_results: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
             rec["api_type"] = rt.get("api_type") or m.get("api_type")
             rec["online"] = rt.get("online", True)
             rec["roles"] = classify(rec)
+            # Say on what grounds it qualified. A role held because the size is
+            # undeclared is not the same claim as one held because the size was
+            # read and met the bar, and the operator-facing registry should not
+            # present them as if they were.
+            rec["role_basis"] = ("declared-capability" if rec.get("param_b") is not None
+                                 or rec.get("thinking") or rec.get("vision")
+                                 or rec.get("embedding")
+                                 else "undeclared-size (eligible, unproven)")
+            # A runtime that answered but refused us is NOT offline, and saying
+            # "offline" would send the operator to restart a server that is running.
+            if rt.get("authorized") is False:
+                rec["authorized"] = False
+                rec["unavailable_reason"] = rt.get("reason") or "authentication required"
             out.append(rec)
     return out
 

backend/runtime_router.py

@@ -49,13 +49,28 @@ def route_with_registry(task_or_role: str, registry: Dict[str, Any],
                         is_role: bool = False) -> Dict[str, Any]:
     """Pick the best model for the task/role from a prebuilt registry. Pure."""
     role = task_or_role if is_role else task_to_role(task_or_role)
+    asked = role
     ranked = (registry.get("rankings") or {}).get(role, [])
+    substituted = False
     if not ranked:
-        # graceful fallback: any Execution model, else any model at all
+        # Vision and Embedding are physical capabilities, not quality tiers. A
+        # text model handed an image does not answer worse — it answers wrongly,
+        # or errors. Refuse instead of substituting; the caller can then say
+        # "no vision model installed" rather than relay confident nonsense.
+        try:
+            from runtime_registry import HARD_CAPABILITY_ROLES
+        except Exception:
+            HARD_CAPABILITY_ROLES = ("Vision", "Embedding")
+        if role in HARD_CAPABILITY_ROLES:
+            return {"role": role, "model": None, "runtime": None, "url": None,
+                    "api_type": None, "endpoint": None,
+                    "reason": f"no model declares the {role} capability — "
+                              f"substituting a text model here would fabricate, not degrade"}
+        # Quality-tier roles may fall back, but the substitution is REPORTED.
         for fb in ("Execution", "Reasoning", "Utility"):
             ranked = (registry.get("rankings") or {}).get(fb, [])
             if ranked:
-                role = fb
+                role, substituted = fb, True
                 break
     if not ranked:
         return {"role": role, "model": None, "runtime": None, "url": None,
@@ -64,9 +79,18 @@ def route_with_registry(task_or_role: str, registry: Dict[str, Any],
     api = top.get("api_type", "ollama")
     url = (top.get("url") or "").rstrip("/")
     endpoint = url + ("/api/chat" if api == "ollama" else "/chat/completions")
-    return {"role": role, "model": top["id"], "runtime": top["runtime"],
-            "url": url, "api_type": api, "endpoint": endpoint,
-            "score": top.get("score"), "alternatives": ranked[1:4]}
+    out = {"role": role, "model": top["id"], "runtime": top["runtime"],
+           "url": url, "api_type": api, "endpoint": endpoint,
+           "score": top.get("score"), "alternatives": ranked[1:4]}
+    if substituted:
+        # The caller asked for one class of model and is getting another. Comparing
+        # `role` against what you asked only works if you remember what you asked,
+        # so state it outright.
+        out["requested_role"] = asked
+        out["substituted"] = True
+        out["reason"] = (f"no {asked} model available — substituted the best {role} "
+                         f"model; expect lower quality for a {asked} task")
+    return out
 
 
 # ── health monitor (Phase 7) ──────────────────────────────────────────────────

backend/runtime_scanner.py

@@ -20,6 +20,7 @@
 
 import json
 import re
+import urllib.error
 import urllib.request
 from typing import Any, Dict, List, Optional
 
@@ -35,22 +36,48 @@
 
 
 # ── http helpers ──────────────────────────────────────────────────────────────
-def _get(url: str, timeout: float = 3.0) -> Optional[Any]:
+# A runtime may sit behind a key: `vllm serve --api-key`, LM Studio with auth on,
+# or any hosted OpenAI-compatible endpoint. Sending no Authorization header meant
+# such a runtime answered 401 and was filed as OFFLINE — indistinguishable from
+# "not running", so the operator was told to start a server that was already up.
+# The key travels with the probe; a refusal is now reported as a refusal.
+def _headers(api_key: Optional[str] = None) -> Dict[str, str]:
+    h = {"Content-Type": "application/json"}
+    if api_key:
+        h["Authorization"] = f"Bearer {api_key}"
+    return h
+
+
+def _request(url: str, timeout: float, api_key: Optional[str] = None,
+             body: Optional[dict] = None) -> Dict[str, Any]:
+    """Returns {data, status, unauthorized, reachable}. Never raises.
+
+    `unauthorized` is the point: it separates "the server said no" from "there was
+    no server", which the previous bare `except: return None` collapsed into one.
+    """
+    data = json.dumps(body).encode() if body is not None else None
+    req = urllib.request.Request(url, data=data, headers=_headers(api_key))
     try:
-        with urllib.request.urlopen(url, timeout=timeout) as r:
-            return json.loads(r.read().decode("utf-8", "replace"))
+        with urllib.request.urlopen(req, timeout=timeout) as r:
+            return {"data": json.loads(r.read().decode("utf-8", "replace")),
+                    "status": getattr(r, "status", 200),
+                    "unauthorized": False, "reachable": True}
+    except urllib.error.HTTPError as e:
+        # An HTTP error means something answered — the runtime IS reachable.
+        return {"data": None, "status": e.code,
+                "unauthorized": e.code in (401, 403), "reachable": True}
     except Exception:
-        return None
+        return {"data": None, "status": None, "unauthorized": False,
+                "reachable": False}
 
 
-def _post(url: str, body: dict, timeout: float = 6.0) -> Optional[Any]:
-    try:
-        req = urllib.request.Request(url, data=json.dumps(body).encode(),
-                                     headers={"Content-Type": "application/json"})
-        with urllib.request.urlopen(req, timeout=timeout) as r:
-            return json.loads(r.read().decode("utf-8", "replace"))
-    except Exception:
-        return None
+def _get(url: str, timeout: float = 3.0, api_key: Optional[str] = None) -> Optional[Any]:
+    return _request(url, timeout, api_key)["data"]
+
+
+def _post(url: str, body: dict, timeout: float = 6.0,
+          api_key: Optional[str] = None) -> Optional[Any]:
+    return _request(url, timeout, api_key, body=body)["data"]
 
 
 # ── pure capability parsers (unit-tested) ─────────────────────────────────────
@@ -92,8 +119,8 @@ def caps_from_ollama_show(show: dict) -> Dict[str, Any]:
 
 
 # ── per-runtime model discovery ───────────────────────────────────────────────
-def _ollama_models(base: str) -> List[Dict[str, Any]]:
-    tags = _get(base.rstrip("/") + "/api/tags")
+def _ollama_models(base: str, api_key: Optional[str] = None) -> List[Dict[str, Any]]:
+    tags = _get(base.rstrip("/") + "/api/tags", api_key=api_key)
     out: List[Dict[str, Any]] = []
     if not tags:
         return out
@@ -107,7 +134,7 @@ def _ollama_models(base: str) -> List[Dict[str, Any]]:
             "context": None, "tool_calling": None, "vision": None,
             "thinking": None, "embedding": None, "api_type": "ollama",
         }
-        show = _post(base.rstrip("/") + "/api/show", {"model": mid})
+        show = _post(base.rstrip("/") + "/api/show", {"model": mid}, api_key=api_key)
         if show:
             c = caps_from_ollama_show(show)
             rec.update({k: c[k] for k in ("context", "tool_calling", "vision",
@@ -121,13 +148,13 @@ def _ollama_models(base: str) -> List[Dict[str, Any]]:
     return out
 
 
-def _openai_models(base: str) -> List[Dict[str, Any]]:
-    data = _get(base.rstrip("/") + "/models")
+def _openai_models(base: str, api_key: Optional[str] = None) -> List[Dict[str, Any]]:
+    data = _get(base.rstrip("/") + "/models", api_key=api_key)
     out: List[Dict[str, Any]] = []
     if not data:
         return out
     # llama.cpp exposes richer metadata at /props (single loaded model)
-    props = _get(base.rstrip("/") + "/props") or {}
+    props = _get(base.rstrip("/") + "/props", api_key=api_key) or {}
     pctx = None
     try:
         pctx = int((props.get("default_generation_settings") or {}).get("n_ctx")
@@ -152,20 +179,40 @@ def _openai_models(base: str) -> List[Dict[str, Any]]:
 
 
 def scan_runtime(probe: Dict[str, str]) -> Dict[str, Any]:
-    """Probe one runtime → {runtime,url,online,api_type,models}. Never raises."""
+    """Probe one runtime → {runtime,url,online,api_type,models,...}. Never raises.
+
+    `probe` may carry an `api_key`; a runtime that answers 401/403 is reported as
+    online-but-unauthorized rather than offline, because telling the operator to
+    restart a server that is running and simply refused the request wastes their
+    time and hides the real fix.
+    """
     url, api = probe["url"], probe.get("api_type", "openai")
+    key = probe.get("api_key") or None
+    unauthorized = False
     try:
         if api == "ollama":
-            models = _ollama_models(url)
-            online = _get(url.rstrip("/") + "/api/tags") is not None
+            probe_r = _request(url.rstrip("/") + "/api/tags", 3.0, key)
+            models = _ollama_models(url, key) if not probe_r["unauthorized"] else []
         else:
-            models = _openai_models(url)
-            online = (_get(url.rstrip("/") + "/models") is not None
-                      or _get(url.rstrip("/") + "/health") is not None)
+            probe_r = _request(url.rstrip("/") + "/models", 3.0, key)
+            if not probe_r["reachable"]:
+                probe_r = _request(url.rstrip("/") + "/health", 3.0, key)
+            models = _openai_models(url, key) if not probe_r["unauthorized"] else []
+        unauthorized = bool(probe_r["unauthorized"])
+        # Reachable is the honest signal: a refusal proves something is listening.
+        online = probe_r["reachable"] or bool(models)
     except Exception:
         models, online = [], False
-    return {"runtime": probe["runtime"], "url": url, "api_type": api,
-            "online": bool(online or models), "models": models}
+    out: Dict[str, Any] = {
+        "runtime": probe["runtime"], "url": url, "api_type": api,
+        "online": bool(online), "models": models,
+    }
+    if unauthorized:
+        out["authorized"] = False
+        out["reason"] = ("the runtime is reachable but refused the request (401/403) — "
+                         "it needs an API key. Add one to this connection; it is "
+                         "running, not down.")
+    return out
 
 
 def scan(extra_probes: Optional[List[Dict[str, str]]] = None,