docs: ADR + plan for OpenAI-compatible remote inference - #24
Conversation
Adds an opt-in backend that runs inference against any OpenAI-compatible endpoint — localhost (Ollama, LM Studio, llama.cpp, vLLM, Infinity, MLX) or internet-facing (OpenRouter, Venice, Jina) — instead of in the browser. Documentation only, following the IMPROVEMENT_PLAN.md precedent. No source changes. The design embeds media directly rather than captioning it, so the remote path is a peer of the nomic/sapiens2 image-feature-extraction backends rather than a variant of chrome-ai. A second pipeline covers VLMs and video LLMs, which emit text rather than vectors. Only downscaled JPEG thumbnails are uploaded, and only after an explicit per-host consent gate. Two findings shape the design and are recorded in the ADR: - There is no single wire format for image embeddings. Five request shapes are needed to cover the named targets, so the format becomes a probeable configuration dimension rather than a hard-coded body. - Ollama's embedding endpoints are text-only (ollama#5304, open since June 2024), so it is served by the VLM pipeline, not the direct one. The UI must say so rather than surfacing an HTTP 400. The plan also catalogues nine latent defects the change would hit — among them isDownloadError() claiming API 401s and opening the HuggingFace upload modal, embedText() skipping L2 normalisation, and zero vectors outranking dissimilar items in cosine search — and lands them as a behaviour-preserving first milestone. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WGZ5syBsRr54E7Cce6HthJ
CI status: the two red checks are pre-existing, not from this PR
MegaLinter ❌ — 4 failing linters, all pre-existing
The two linters that actually touch this PR's files both pass with zero errors. gitleaks flags Worth noting it is a real, still-valid-looking credential sitting in public history. Fixing it properly means rotating the Unsplash key and either rewriting history or adding a grype / osv-scanner / trivy all report the same two dependency advisories: Cloudflare Pages ❌Fails at Verification on this branch
No action taken. If you'd like, I can open a follow-up PR for the gitleaks allowlist and the two dependency bumps. Generated by Claude Code |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2c7306cc48
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| - Cache prefix from `currentCachePrefix()` (B2), namespaced on a synchronous FNV-1a hash of | ||
| `host | pipeline | model | embedderId | dim | wireFormat`. Deliberately **not** `crypto.subtle.digest`, | ||
| which is async and would force `currentCachePrefix()` and all its callers to become promises. Include | ||
| `dim` even though the model implies it — providers change dimensions, and that is the one mismatch | ||
| producing garbage rather than an error (B3). |
There was a problem hiding this comment.
The proposed namespace omits the describe prompt, maximum image width, and video frame count even though each changes the media or caption being embedded. After a user changes one of these settings and reloads, readCachedEmbeddings() will therefore reuse vectors generated from the old inputs, making the setting appear ineffective and producing stale clustering/search results. Include every vector-affecting input in the namespace, as well as the full normalized base URL rather than only its host.
Useful? React with 👍 / 👎.
| Changing base URL, model or pipeline **after** a load must trigger the same page reload as `#model-select` | ||
| (`:2827-2834`), because the cache namespace changes underneath `state.vectors`. |
There was a problem hiding this comment.
Pipeline B exposes an embedding-source switch between remote embeddings and local Nomic, but this reload requirement names only base URL, model, and pipeline. If the source is changed after media has been embedded, state.vectors remains in the old embedding space while subsequent queries use the newly selected embedder; when both happen to have the same dimension, the dimension guard cannot detect the mismatch and search returns plausible but meaningless rankings. Treat embedding-source changes as reload-required too.
Useful? React with 👍 / 👎.
| - **Embedding space**: index-time and query-time vectors must come from the same embedder with the same | ||
| prompt prefixes. Any change of embedder, model, or dimension must change the IndexedDB cache namespace |
There was a problem hiding this comment.
Requiring index-time and query-time vectors to use the same prompt prefixes contradicts the existing Nomic contract: indexing uses search_document: at src/app.ts:1590, while querying uses search_query: at src/app.ts:627, and the new plan explicitly preserves that distinction. Following this guide literally would place one side in the wrong task space and degrade semantic search; describe the invariant as using the same embedder with that model's required role-specific prefix scheme, not identical prefixes.
Useful? React with 👍 / 👎.
Three fixes from automated review, all correct. Cache namespace: the first draft covered host, pipeline, model, embedder, dim and wire format, omitting the describe prompt, image size and video frame count — each of which changes what actually gets embedded. Leaving them out is what makes a setting look broken: change it, reload, and readCachedEmbeddings() serves vectors built from the old input. The rule is now "every input that changes the resulting vector", and the namespace keys on the full normalized base URL rather than the host alone, since two paths on one host can be different services behind a gateway. This reverses the earlier position that excluded the prompt on chrome-ai precedent. That precedent is a bug rather than a contract, and it is cheaper to be wrong about locally. Namespacing is non-destructive, so the cost of including a setting is one re-embed and reverting restores the prior cache for free. Reload triggers: the embedding-source switch (remote /embeddings vs local nomic-embed-text) was missing, and it is the most dangerous omission — nomic-embed-text is 768-d and so are many remote models, so flipping it after a run leaves state.vectors in the old space while queries use the new one at matching width, which the dimension guard cannot detect. Embedding-space invariant: "the same prompt prefixes" contradicted nomic's contract, where the prefixes deliberately differ by role (search_document: when indexing, search_query: when querying). Reworded to require the same embedder under that model's own role-prefix scheme. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WGZ5syBsRr54E7Cce6HthJ
Deploying media-clusterer with
|
| Latest commit: |
246321a
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://e95085db.media-clusterer.pages.dev |
| Branch Preview URL: | https://claude-openai-compatible-inf.media-clusterer.pages.dev |
|
All three review points were correct and are fixed in P1 — namespace every embedding-affecting setting. Right, and the omission was the interesting kind: a setting left out of the namespace doesn't fail loudly, it just appears to do nothing. The rule is now "every input that changes the resulting vector," with a table covering the describe prompt, This reverses my earlier reasoning, which excluded the prompt on the grounds that One imprecision left in deliberately, and noted as such: P1 — reload when the embedding source changes. The sharpest of the three. P2 — role-specific prefixes. Correct, my wording was wrong. "The same prompt prefixes" reads as identical, when nomic's contract is that they deliberately differ by role. Reworded to require the same embedder under that model's own role-prefix scheme — differing by role for nomic, absent on both sides for a model without one, never mixed across sides. Verified on Still docs-only — no source changes. Generated by Claude Code |
The merge of main in 246321a kept both header rows and both delimiter rows from the two branches instead of merging the data rows under one header, so #23's header and its row of dashes rendered as ordinary table rows. Drops the duplicates, leaving one header, one delimiter, and the 0001/0002 rows. Compact pipe style, which is what this repo's markdownlint MD060 enforces. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WGZ5syBsRr54E7Cce6HthJ
2 participants
Why
Every embedding is computed in the browser, and that caps three things at once.
Nothing happens until a large download finishes. The default
sapiens2-fp16is 229 MB (src/app.ts:206). Throughput is capped by the local GPU — and silently falls back to WASM when WebGPU is missing (src/app.ts:1175-1181). The model ceiling is the device ceiling: purpose-built multimodal retrieval models and video LLMs can't run in a tab at all.customModelHost(src/types.ts:61) looks like it addresses this and doesn't — it redirects model-file downloads to a HuggingFace mirror. Inference still runs locally.Meanwhile Ollama, LM Studio, llama.cpp, vLLM, Infinity and MLX all expose an OpenAI-compatible API on localhost, and OpenRouter, Venice and Jina expose the same shape over the internet. A user running any of them has capable inference sitting idle with no way to point the app at it.
What this adds
Documentation only. No source changes, following the
IMPROVEMENT_PLAN.mdprecedent.docs/adr/0002-openai-compatible-remote-inference.md— the decision.REMOTE_INFERENCE_PLAN.md— eight-milestone plan, wire formats, tests, risks, verification.docs/adr/README.md— the ADR convention (see the note on PR docs: ADR + plan for a small video language model; ci: force MegaLinter auto-fix #23 below).AGENT.md— Remote AI Mode, plus a secrets rule and an embedding-space invariant.The decision, briefly
One new
ModelVariant,openai, with two pipelines behind a single client, settings block, and consent gate:directPOST {baseUrl}/embeddingsvlmPOST {baseUrl}/chat/completions→ embed the captionPipeline A embeds the image itself, making the remote path a peer of
nomic/sapiens2rather than a variant ofchrome-ai. It also gives a genuinely shared image/text embedding space, which makes semantic search better-founded than today's caption-mediatedchrome-aisearch.Only downscaled JPEG thumbnails are uploaded — 384 px at q0.8, roughly 25–40 KB.
createImageBitmap(blob, { resizeWidth })does the downscale during decode, so a 48 MP original is never decoded at full size.Two research findings that shape the design
There is no single wire format for image embeddings.
/v1/chat/completionswith animage_urlpart is universal;/v1/embeddingswith an image is not:input: [{ content: [{ type: 'image_url', image_url: { url } }] }]messages: [...]— chat-shapedinput: [{ image: '<b64>' }]input: ['<data URI>']{ content: 'Image: [img-1]', image_data: [...] }— and reported unreliableSo the format becomes a probeable configuration dimension rather than a hard-coded body. That's the main reason this is larger than "add a
fetchcall".Ollama cannot do direct image embeddings (ollama#5304, open since June 2024). It runs VLMs fine, so it's fully served by pipeline B — but the UI has to say so rather than surfacing an HTTP 400.
On the plus side, OpenRouter's
/v1/embeddings/modelscurrently lists three image-capable models, one of them free (nvidia/llama-nemotron-embed-vl-1b-v2:free), which makes the whole direct path verifiable at zero cost.google/gemini-embedding-2accepts video.Nine latent defects this surfaces
None are introduced here; all are load-bearing for the feature, and the plan lands them as a behaviour-preserving M0. Three set the shape of the design:
isDownloadError()(src/modelFallback.ts:111-124) matchesunauthorizedand bare status codes, so an API 401 would open the HuggingFace model-upload fallback modal.embedText()(src/app.ts:633) never callsl2normalize(), relying on Transformers.jsnormalize: true— butsearchByCosine()is a raw dot product and remote responses aren't guaranteed unit-norm.0, which is above every genuinely dissimilar item. Failed embeddings currently rank high.Trade-offs stated in the ADR
README.md:15andindex.html:745promise "no uploads, no server" unconditionally. That has to be rewritten. Watering it down is a real cost, accepted deliberately rather than hidden.sessionStorageoption and by saying so in the UI; not eliminated.Secret hygiene
captureConsoleIntegration({ levels: ['error'] })(src/sentry.ts:15) ships everyconsole.errorto a remote sink, and Sentry breadcrumbs record fetch URLs. So: key inlocalStorage['mc_openai_key'], never instateormc_settings;Authorizationheader only, never a query string; redaction at error-construction; noconsole.erroron this path.One assumption flagged for day-one confirmation
OpenRouter's embeddings docs show image inputs as
https://URLs. Local files can't be public URLs, so pipeline A on OpenRouter needsdata:URIs to be accepted — inferred from the adjacent chat-completions and rerank endpoints documenting data-URI support, not observed on/embeddings. Onecurlsettles it. If it fails, pipeline A still stands on Jina/vLLM/Infinity and OpenRouter users take pipeline B; nothing else changes.Note on overlap with #23
#23 also adds
docs/adr/README.mdand numbers its ADR0001. The convention text here is identical, so the only merge conflict should be the one-row index table. Whichever lands second adds its row.REMOTE_INFERENCE_PLAN.mdM6 (multi-frame video) overlapsVIDEO_LM_PLAN.mdM3 — both need the sameextractVideoFrames(file, n)generalisation. Noted in both directions; whichever lands first owns the function.Verification
npm run type-check— cleannpm test— 9 files, 76 tests passed (proving no source was touched)markdownlint-cli2 --config .markdownlint.json— 0 issues on all new filessrc/file.ts:linecitations checked programmatically for range, and every citation taken from exploration rather than direct reading was spot-checked against the code it claimsgit diff --statshows only the four documentation files.Generated by Claude Code